Part 1 – Overlap in the assignment of kids to Gifted and Talented (GT) education In these

exercises you are going to replicate results from the lecture and estimate the effect of following a
gifted and talented program (gt) on students’ math and language scores in secondary education.
The data are drawn from Booij et al. (2016). In Part 1 you will try and figure out if you could
apply a Regression Control (RC) design, as you did in previous CLabs 3.1 and 3.2.
1. Open the data and familiarize yourself using desc and sum.
Contains data from C:\Users\Minza Mangi\Downloads\gta_data.dta
obs: 3,057
vars: 9
size: 220,104

storage display value

variable name type format label variable label

id double %12.0g
cohort double %12.0g cohort
male double %12.0g
age double %12.0g
ist_norm double %12.0g IST pre-test distance to cutoff
cito double %12.0g End of primary school exam CITO score
resmath double %12.0g Average math score grade 2 to 6
reslang double %12.0g Average language score grade 2 to 6
__gt double %12.0g Treated child (GT)

. sum

Variable Obs Mean Std. Dev. Min Max

id 3057 1575.147 907.7054 1 3127

cohort 3057 2003.974 3.768187 1998 2010
male 3057 .5407262 .4984201 0 1
age 3057 12.15911 .4890858 9.655914 14.75806
ist_norm 3057 -12.99607 15.57155 -70 43

cito 3057 547.4874 2.484096 518 550

resmath 3057 6.607289 1.147726 3 9.925139
reslang 3057 6.77146 .7965957 3.995833 9.487708
__gt 3057 .210664 .4078469 0 1

Visualize the data using hist XXX, frac name(hist_XXX) where XXX is the name of the
variable that you are looking at. Check out all variables (except id, of course). What
variables do you think are realized prior to, and what are realized post of the GT progam?
 .1
.08
.06
Fraction
.04
.02
0

2 4 6 8 10
Average math score grade 2 to 6
.15
.1
Fraction
.05
0

4 6 8 10
Average language score grade 2 to 6
 .4
.3
Fraction
.2 .1
0

520 530 540 550

End of primary school exam CITO score
.6
.4
Fraction
.2
0

0 .2 .4 .6 .8 1
male
 .2
.15
Fraction
.1.05
0

10 11 12 13 14 15
age
.1
.08.06
Fraction
.04
.02
0

-100 -50 0 50
IST pre-test distance to cutoff
 .08
.06
Fraction
.04 .02
0

1995 2000 2005 2010

cohort
.8
.6
Fraction
.4.2
0

0 .2 .4 .6 .8 1
Treated child (GT)
The age, male, cohort and ist_norm are prior to GT program, whereas average math score,
average language score and _gt are after the GT program.

2. To estimate the effect of the gt we will ultimately have to compare individuals that get
the program
(gt=1) to those that don’t (gt=0). Type
Summary statistics: mean
by categories of: __gt (Treated child (GT))

__gt id cohort male age ist_norm cito resmath reslang __gt

0 1576.70 2003.98 0.52 12.20 -18.77 547.21 6.41 6.69 0.00

1 1569.31 2003.95 0.64 12.02 8.65 548.54 7.36 7.07 1.00

Total 1575.15 2003.97 0.54 12.16 -13.00 547.49 6.61 6.77 0.21

The above summary statistics shows that individuals who get the program have the
highest average score in math and language, as compared to those who didn’t.

3. Observing imbalance (prior differences between the groups) is not a good sign, but not
necessarily a problem. Imbalance with respect to gender, for example, only poses a big problem
if it predicts math and/or language scores. A simple way to check this is to do a regression using
prior variables as predictors. For simplicity we will neglect cohort and only look at math:
. reg resmath male age ist cito, r

Linear regression Number of obs = 3057

F( 4, 3052) = 137.69
Prob > F = 0.0000
R-squared = 0.1708
Root MSE = 1.0458

Robust
resmath Coef. Std. Err. t P>|t| [95% Conf. Interval]

male .0162051 .0381654 0.42 0.671 -.0586273 .0910376

age -.1639968 .0421439 -3.89 0.000 -.2466301 -.0813636
ist_norm .0241678 .0013369 18.08 0.000 .0215465 .0267891
cito .0704671 .0093546 7.53 0.000 .0521252 .0888091
_cons -29.6732 5.113683 -5.80 0.000 -39.69982 -19.64659

Is gender predictive?

The above regression results shows that gender is not significant predictor math. Its p-
value is 0.671 which is greater than alpha of 0.05.
So, is imbalance with respect to gender a problem? It is not always easy to interpret the
coefficients from a regression. The coefficient of age, for example, 𝛽𝛽̂ 𝑎ge = −.1639968 and
significant (𝑡𝑡 − 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 = −3.89). What does it mean? It means that when a student’s age is 1
year higher, her predicted grade is 0.16 lower. That seems a lot, but is it? That depends on how
big a 1 year change is.

Recall from q1 that 𝑆𝑆𝑆(𝑎𝑎𝑎𝑎𝑎𝑎) = 0.48. A 1 unit change in age is more than 2 ×
𝑆𝑆𝑆(𝑎𝑎𝑎𝑎𝑎𝑎). If ±2 × 𝑆𝑆𝑆(𝑎𝑎𝑎𝑎𝑎𝑎) around the mean contains about 95% of the data, 2 ×
𝑆𝑆𝑆𝑆(𝑎𝑎𝑎𝑎𝑎𝑎) must contain about 47.5%. So if your age is 1 year higher, you “overtake” about
47.5% of students and go from being one of the youngest to one of the oldest. A big change.
Also, you can look at the histogram of age to see if you think a 1 year age difference is large or
not given this sample.

The historgram below shows that a 1 year age difference is not larger for this sample. The
mean age is 12.15 with +(-) 0.89 standard deviation which also depicts from histogram.
1.5
1
Density
.5
0

10 11 12 13 14 15
age

4. A handy way to make the size of the coefficients comparable is to scale it by

𝑆D(𝑎ge) /𝑆𝑆𝑆𝑆(𝑟emath) *𝛽age=𝛽𝑠dage . The interpretation then becomes: “if age is 1 SD
higher, how many SD higher is resmath expected to be?” You can get this number quickly by
using the beta option reg resmath male age ist cito, r beta
. reg resmath male age ist cito, r beta

Linear regression Number of obs = 3057

F( 4, 3052) = 137.69
Prob > F = 0.0000
R-squared = 0.1708
Root MSE = 1.0458

Robust
resmath Coef. Std. Err. t P>|t| Beta

male .0162051 .0381654 0.42 0.671 .0070374

age -.1639968 .0421439 -3.89 0.000 -.0698847
ist_norm .0241678 .0013369 18.08 0.000 .3278925
cito .0704671 .0093546 7.53 0.000 .1525166
_cons -29.6732 5.113683 -5.80 0.000 .

𝑆D(𝑎ge) /𝑆D(𝑟emath) *𝛽age=𝛽𝑠dage

0.489) /1.15 *(-0.1639)=𝛽𝑠dage
𝛽𝑠dage=-0.06988

If age is 1 SD higher than average score in math will be 6.98% less.

What does this reveal? Is the coefficient of age really that big, or are some of the others bigger?

No, it shows that coefficient of age is not so much higher its impact is only 6.98 % whereas
impact of IST pre trest distance is 32.7% on average math scores

5. The philosophy of doing a controlled comparison is to find subgroups of units that are
comparable in terms of other observable characteristics (cohort, male, age, ist, cito) and compare
gt=0 and gt=1 people within that subgroup, rather than all units. This requires “overlap”: within
each subgroup we need treated (gt=1) and control (gt=0) units. From q4 we know that imbalance
with respect to ist_norm and cito would be most problematic because these are very predictive of
math grades. Lets check if we have overlap with respect to these variables:
 550
End of primary school exam CITO score
520 530 540

-100 -50 0 50
IST pre-test distance to cutoff

Control Treated (GT)

Do we have overlap? Can we use the RC design here? What would the scatter have looked like if
gt had been randomly assigned?

Yes there is problem of overlapping we are unable to distinguish perceive number of points
whether they are in control group or treatment.

Part 2 – Regression Discontinuity design: Comparing kids around the cutoff In part 2 of these
exercises you are going to replicate results from the lecture and use a regression discontinuity
design to tackle the problem of “no overlap”. The previous question showed that there are no
control units with ist_norm≥0, and no treated units with ist_norm
 10
Average math score grade 2 to 6
4 62 8

-100 -50 0 50
IST pre-test distance to cutoff

Control Treated (GT)

By giving different colors and shapes to the scatter points in the sample gt==0 and gt==1, we can
clearly see the overlap problem. We can also see, however, that there are many units around
ist_norm=0, the cutoff value at which treatment status discretely changes from 0 to 1. 7. From
the scatter in q6 it is hard – if not impossible – to see if students score higher after the cutoff. To
see we can add regression lines to the plot
 10
8
6
4
2

-100 -50 0 50
IST pre-test distance to cutoff

Average math score grade 2 to 6 Average math score grade 2 to 6

Fitted values Fitted values

The above graph clearly shows that there is a discontinuity in average math scores at point
ist_norm=0.The cutoff point is crucial in ensuring that treatment and control groups do not
overlap. Traditional regression models are unreliable in estimating intervention impact
under these circumstances, but RD analysis provides reliable estimates when there is no
overlap in assignment. This limited use of RD analysis still holds promise for numerous
therapeutic applications.

8. The lines in the plot show a jump (discontinuity) at point ist_norm=0. From the graph it is
difficult to see exactly how large the jump is, and we cannot tell if it is significant. For that we
need a regression. First we quietly run a regression without controls for future reference and
store it as a; then we do the RD regression and store it as b:
Linear regression Number of obs = 3057
F( 3, 3053) = 176.29
Prob > F = 0.0000
R-squared = 0.1567
Root MSE = 1.0545

Robust
resmath Coef. Std. Err. t P>|t| [95% Conf. Interval]

__gt .2974139 .079824 3.73 0.000 .1408996 .4539282

ist_norm .0202418 .0017761 11.40 0.000 .0167594 .0237242

c.__gt#c.ist_norm .0115797 .0063593 1.82 0.069 -.0008893 .0240486

_cons 6.786604 .0410018 165.52 0.000 6.70621 6.866998

The above regression analysis shows that (discontinuity) at point ist_norm=0 Is significant
and there is not a large the jump the coefficient of ist_norm is 0.0202, this means 2.2%
students score higher math course after the cutoff.

9. The graph from q7 does not clearly show how the data fits the line(s). To see that we have plot
the data using a smaller number of bins. The program grrd does that (check out the code in
progs.do if you want to learn about it):
. grrd resmath

Source SS df MS Number of obs = 3057

F( 3, 3053) = 189.08
Model 630.752567 3 210.250856 Prob > F = 0.0000
Residual 3394.8373 3053 1.11196767 R-squared = 0.1567
Adj R-squared = 0.1559
Total 4025.58987 3056 1.31727417 Root MSE = 1.0545

resmath Coef. Std. Err. t P>|t| [95% Conf. Interval]

ist_norm .0202418 .0018597 10.88 0.000 .0165955 .0238882

1.gt .2974139 .0750786 3.96 0.000 .1502043 .4446236

gt#c.ist_norm
1 .0115797 .0057695 2.01 0.045 .0002672 .0228921

_cons 6.786604 .0409828 165.60 0.000 6.706247 6.86696

(option xb assumed; fitted values)

(2 missing values generated)
(3055 real changes made)
(28 missing values generated)

The above analysis shows that there is significant impact of student around ist_norm=0
for math skills prior to GT program (b1= 0.0202418 & p-value =0.000) and after GT
program (b2= 0.0115797 & p-value =0.045) for mat scores.

10. The graph makes is visually clear that there is a discontinuity in average math scores at point
ist_norm=0. If it is reasonable to assume that students around the cutoff are similar in all aspects
other than gt assignment, we can conclude that the difference that we see is due to the program.
We have some additional prior variables (cohort, male, age, cito) that we can include to see if the
result changes:
. eststo c: reg resmath gt ist_norm c.gt#c.ist_norm age male cito i.cohort, r

Linear regression Number of obs = 3057

F( 18, 3038) = 47.65
Prob > F = 0.0000
R-squared = 0.2310
Root MSE = 1.0094

Robust
resmath Coef. Std. Err. t P>|t| [95% Conf. Interval]

gt .2976529 .0763997 3.90 0.000 .1478525 .4474532

ist_norm .0167683 .0017821 9.41 0.000 .013274 .0202626

c.gt#c.ist_norm .0114248 .0059921 1.91 0.057 -.0003241 .0231737

age -.1095607 .0407904 -2.69 0.007 -.1895403 -.029581

male -.0102603 .0370716 -0.28 0.782 -.0829482 .0624277
cito .0776459 .0089153 8.71 0.000 .0601653 .0951266

cohort
1999 -.1774812 .0916062 -1.94 0.053 -.3570976 .0021353
2000 .0991205 .0885403 1.12 0.263 -.0744844 .2727254
2001 .0050158 .0985941 0.05 0.959 -.188302 .1983337
2002 -.2836387 .0975594 -2.91 0.004 -.4749278 -.0923496
2003 -.0853821 .0977454 -0.87 0.382 -.277036 .1062717
2004 .0211138 .0902265 0.23 0.815 -.1557974 .198025
2005 .0432867 .0944868 0.46 0.647 -.1419779 .2285513
2006 -.1448801 .0971173 -1.49 0.136 -.3353024 .0455422
2007 .1553589 .0911375 1.70 0.088 -.0233384 .3340563
2008 .6665464 .0937586 7.11 0.000 .4827098 .8503831
2009 .4678772 .1015936 4.61 0.000 .2686781 .6670764
2010 .4537919 .0889739 5.10 0.000 .2793367 .6282471

_cons -34.52609 4.873045 -7.09 0.000 -44.08089 -24.97129

The above regression results shows that students around the cutoff are not similar after
incorporating age, male, cito and cohort ads control variables because beta coefficient of
c.ist_norm (0.1143) has p-value 0.57 which is insignificant at 5% confidence level.

Look at columns (1) – (3) and explain the differences and similarities. Do you believe it is
credible to assume that students around ist_norm=0 are similar?
. esttab a b c, b(a2) se nogap star(* 0.10 ** 0.05 *** 0.01)

(1) (2) (3)

resmath resmath resmath

__gt 0.95*** 0.30***

(0.051) (0.080)
ist_norm 0.020*** 0.017***
(0.0018) (0.0018)
c.__gt#c.i~m 0.012*
(0.0064)
gt 0.30***
(0.076)
c.gt#c.ist~m 0.011*
(0.0060)
age -0.11***
(0.041)
male -0.010
(0.037)
cito 0.078***
(0.0089)
1998.cohort 0
(.)
1999.cohort -0.18*
(0.092)
2000.cohort 0.099
(0.089)
2001.cohort 0.0050
(0.099)
2002.cohort -0.28***
(0.098)
2003.cohort -0.085
(0.098)
2004.cohort 0.021
(0.090)
2005.cohort 0.043
(0.094)
2006.cohort -0.14
(0.097)
2007.cohort 0.16*
(0.091)
2008.cohort 0.67***
(0.094)
2009.cohort 0.47***
(0.10)
2010.cohort 0.45***
(0.089)
_cons 6.41*** 6.79*** -34.5***
(0.021) (0.041) (4.87)

N 3057 3057 3057

Standard errors in parentheses

* p<0.10, ** p<0.05, *** p<0.01

tabstat *, by(gt) format(%6.2f)

The above analysis shows that students around ist_norm=0 are not similar in terms of
gender and cohort years of 2000, 2001, 2003,2004 ,2005 & 2006)
11. Finally, we can repeat the analysis for language grades qui eststo d: reg reslang gt ist_norm
c.gt#c.ist_norm age male cito i.cohort,r grrd reslang esttab a b c d, b(a2) se nogap star(* 0.10 **
0.05 *** 0.01) Do we see an effect on language skills as well? How large?
. grrd reslang

Source SS df MS Number of obs = 3057

F( 3, 3053) = 52.81
Model 95.6641153 3 31.8880384 Prob > F = 0.0000
Residual 1843.56586 3053 .603853869 R-squared = 0.0493
Adj R-squared = 0.0484
Total 1939.22998 3056 .634564784 Root MSE = .77708

reslang Coef. Std. Err. t P>|t| [95% Conf. Interval]

ist_norm .0079921 .0013704 5.83 0.000 .005305 .0106792

1.gt .160463 .0553268 2.90 0.004 .0519814 .2689446

gt#c.ist_norm
1 -.0001287 .0042516 -0.03 0.976 -.0084651 .0082076

_cons 6.841757 .030201 226.54 0.000 6.78254 6.900973

(option xb assumed; fitted values)

(2 missing values generated)
(3055 real changes made)

The above analysis shows that there is also significant impact of student around
ist_norm=1 for language skill prior to GT program (b1= 0.0079921 & p-value =0.000).
Whereas, there is insignificant impact of student around ist_norm=1 for language skill
after GT program (b2= 0.001287 & p-value =0.976).
It is noticeable that results prior to GRT program are significant for both math and
language scores, Whereas after GT program it is significant for math and insignificant for
language skills score.
. esttab a b c d, b(a2) se nogap star(* 0.10 ** 0.05 *** 0.01)

(1) (2) (3) (4)

resmath resmath resmath reslang

__gt 0.95*** 0.30***

(0.051) (0.080)
ist_norm 0.020*** 0.017*** 0.0047***
(0.0018) (0.0018) (0.0013)
c.__gt#c.i~m 0.012*
(0.0064)
gt 0.30*** 0.14***
(0.076) (0.055)
c.gt#c.ist~m 0.011* 0.0041
(0.0060) (0.0043)
age -0.11*** -0.048
(0.041) (0.029)
male -0.010 -0.35***
(0.037) (0.027)
cito 0.078*** 0.077***
(0.0089) (0.0068)
1998.cohort 0 0
(.) (.)
1999.cohort -0.18* -0.12*
(0.092) (0.071)
2000.cohort 0.099 -0.052
(0.089) (0.071)
2001.cohort 0.0050 0.074
(0.099) (0.076)
2002.cohort -0.28*** -0.056
(0.098) (0.075)
2003.cohort -0.085 0.12*
(0.098) (0.074)
2004.cohort 0.021 0.049
(0.090) (0.069)
2005.cohort 0.043 -0.014
(0.094) (0.075)
2006.cohort -0.14 -0.037
(0.097) (0.071)
2007.cohort 0.16* 0.0061
(0.091) (0.069)
2008.cohort 0.67*** 0.085
(0.094) (0.071)
2009.cohort 0.47*** 0.066
(0.10) (0.080)
2010.cohort 0.45*** 0.14*
(0.089) (0.074)
_cons 6.41*** 6.79*** -34.5*** -34.4***
(0.021) (0.041) (4.87) (3.74)

N 3057 3057 3057 3057

Standard errors in parentheses

* p<0.10, ** p<0.05, *** p<0.01

.
Do we see an effect on language skills as well? How large?