Activity 3 LIVE: Randomized Experiment Balance and Estimation in Gerber et al. 2008

Activity 3 LIVE: Randomized Experiment Balance and Estimation in Gerber et al. 2008#

2026-09-22

# imports we'll need
import numpy as np
import pandas as pd

First, let’s load in the data at ~/COMSC-341CD/data/gerber2008.csv and take a look at the first few rows via DataFrame.head().

# TODO uncomment to load data
vote_df = pd.read_csv("~/COMSC-341CD/data/gerber2008.csv")
vote_df.head()

	sex	yob	treatment	voted	hh_size	g2000	g2002	g2004	p2000	p2002	p2004
0	male	1941	Civic Duty	No	2	yes	yes	yes	no	yes	No
1	female	1947	Civic Duty	No	2	yes	yes	yes	no	yes	No
2	male	1951	Hawthorne	Yes	3	yes	yes	yes	no	yes	No
3	female	1950	Hawthorne	Yes	3	yes	yes	yes	no	yes	No
4	female	1982	Hawthorne	Yes	3	yes	yes	yes	no	yes	No

# TODO examining the top and bottom of the dataframe
vote_df.tail()

	sex	yob	treatment	voted	hh_size	g2000	g2002	g2004	p2000	p2002	p2004
344079	male	1943	Control	Yes	2	yes	yes	yes	no	yes	Yes
344080	male	1955	Control	No	2	yes	yes	yes	yes	no	Yes
344081	female	1958	Control	No	2	yes	no	yes	no	no	No
344082	male	1937	Control	Yes	2	yes	yes	yes	yes	yes	Yes
344083	female	1949	Control	Yes	2	yes	yes	yes	no	no	Yes

# TODO giving a summary of the dataframe
vote_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 344084 entries, 0 to 344083
Data columns (total 11 columns):
 #   Column     Non-Null Count   Dtype 
---  ------     --------------   ----- 
 0   sex        344084 non-null  object
 1   yob        344084 non-null  int64 
 2   treatment  344084 non-null  object
 3   voted      344084 non-null  object
 4   hh_size    344084 non-null  int64 
 5   g2000      344084 non-null  object
 6   g2002      344084 non-null  object
 7   g2004      344084 non-null  object
 8   p2000      344084 non-null  object
 9   p2002      344084 non-null  object
 10  p2004      344084 non-null  object
dtypes: int64(2), object(9)
memory usage: 28.9+ MB

# TODO get the column names
vote_df.columns

Index(['sex', 'yob', 'treatment', 'voted', 'hh_size', 'g2000', 'g2002',
       'g2004', 'p2000', 'p2002', 'p2004'],
      dtype='object')

# TODO get the number of rows and columns
vote_df.shape

(344084, 11)

# TODO return the data types of each column
vote_df.dtypes

sex          object
yob           int64
treatment    object
voted        object
hh_size       int64
g2000        object
g2002        object
g2004        object
p2000        object
p2002        object
p2004        object
dtype: object

The columns we have are:

sex: the respondent’s sex
yob: the respondent’s year of birth
treatment: the treatment the respondent received
voted: our outcome: whether the respondent voted in the 2006 primary election
hh_size: the size of the respondent’s household
g2000, g2002, g2004: whether the respondent voted in the 2000, 2002, and 2004 general election
p2000, p2002, p2004: whether the respondent voted in the 2000, 2002, and 2004 primary election

1. Column generation#

Let’s exmaine the treatment column by running .unique() on it:

# TODO show the unique values of the treatment column

These two options correspond to the \(T=0\) and \(T=1\) interventions we discussed earlier:

\[\begin{split} T = \begin{cases} 1 & \text{mail stating that their household turnout would be publicized to their neighbors} \\ 0 & \text{mail reminding them that voting is a civic duty} \end{cases} \end{split}\]

Let’s generate an additional column that corresponds to the respondent’s age in 2006.

# TODO generate the 'age' column
vote_df['age'] = 2006 - vote_df['yob']
vote_df[['age', 'yob']]

	age	yob
0	65	1941
1	59	1947
2	55	1951
3	56	1950
4	24	1982
...	...	...
344079	63	1943
344080	51	1955
344081	48	1958
344082	69	1937
344083	57	1949

344084 rows × 2 columns

2. Balance check#

One way we can check whether exchangeability is plausible is to check whether the distribution of data characteristics are the same between the treatment and control groups.

Let’s now check the balance of the various characteristics between the \(T=1\) ('Neighbors') and \(T=0\) ('Civic Duty') groups.

To do this, we can call the groupby(column) method on the dataframe and then use the mean() method:

# example usage of groupby and mean
df.groupby('treat_column')['continuous_column'].mean()

For the continuous variables of age and hh_size, let’s compute the mean for each group of the treatment column.

# TODO your code for the grouped mean of age
vote_df.groupby('treatment')['age'].mean()

treatment
Civic Duty    49.659035
Control       49.813546
Hawthorne     49.704795
Neighbors     49.852936
Self          49.792506
Name: age, dtype: float64

# TODO your code for the grouped mean of hh_size
vote_df.groupby('treatment')['hh_size'].mean()

treatment
Civic Duty    2.189126
Control       2.183667
Hawthorne     2.180138
Neighbors     2.187770
Self          2.180805
Name: hh_size, dtype: float64

Let’s also check the balance of the categorical variables across the two treatment groups.

For categoricalvariables, we can use the value_counts() method with normalize=True along with .groupby() to compute the proportions for each individual characteristic:

# example usage of value_counts and groupby
df.groupby('treat_column')['categorical_column'].value_counts(normalize=True)

For the each of the categorical variables of ['sex', 'g2000', 'g2002', 'g2004', 'p2000', 'p2002', 'p2004'], let’s compute and print the proportion of respondents who have the characteristic for each treatment group:

vote_df.groupby('treatment')['g2000'].value_counts(normalize=True)

treatment   g2000
Civic Duty  yes      0.841724
            no       0.158276
Control     yes      0.843377
            no       0.156623
Hawthorne   yes      0.844414
            no       0.155586
Neighbors   yes      0.841653
            no       0.158347
Self        yes      0.840389
            no       0.159611
Name: proportion, dtype: float64

categorical_vars = ['sex', 'g2000', 'g2002', 'g2004', 'p2000', 'p2002', 'p2004']

# loop over all the categorical variables
for column in categorical_vars:
    # TODO your code for grouped value counts
    print(vote_df.groupby('treatment')[column].value_counts(normalize=True))

treatment   sex   
Civic Duty  female    0.500183
            male      0.499817
Control     male      0.501059
            female    0.498941
Hawthorne   male      0.500995
            female    0.499005
Neighbors   female    0.500065
            male      0.499935
Self        male      0.500419
            female    0.499581
Name: proportion, dtype: float64
treatment   g2000
Civic Duty  yes      0.841724
            no       0.158276
Control     yes      0.843377
            no       0.156623
Hawthorne   yes      0.844414
            no       0.155586
Neighbors   yes      0.841653
            no       0.158347
Self        yes      0.840389
            no       0.159611
Name: proportion, dtype: float64
treatment   g2002
Civic Duty  yes      0.811110
            no       0.188890
Control     yes      0.810895
            no       0.189105
Hawthorne   yes      0.812952
            no       0.187048
Neighbors   yes      0.811340
            no       0.188660
Self        yes      0.811476
            no       0.188524
Name: proportion, dtype: float64
treatment   g2004
Civic Duty  yes      1.0
Control     yes      1.0
Hawthorne   yes      1.0
Neighbors   yes      1.0
Self        yes      1.0
Name: proportion, dtype: float64
treatment   p2000
Civic Duty  no       0.746428
            yes      0.253572
Control     no       0.748116
            yes      0.251884
Hawthorne   no       0.749634
            yes      0.250366
Neighbors   no       0.748802
            yes      0.251198
Self        no       0.748888
            yes      0.251112
Name: proportion, dtype: float64
treatment   p2002
Civic Duty  no       0.611152
            yes      0.388848
Control     no       0.610626
            yes      0.389374
Hawthorne   no       0.605670
            yes      0.394330
Neighbors   no       0.613413
            yes      0.386587
Self        no       0.608090
            yes      0.391910
Name: proportion, dtype: float64
treatment   p2004
Civic Duty  No       0.600555
            Yes      0.399445
Control     No       0.599661
            Yes      0.400339
Hawthorne   No       0.596770
            Yes      0.403230
Neighbors   No       0.593335
            Yes      0.406665
Self        No       0.597519
            Yes      0.402481
Name: proportion, dtype: float64

Are there any differences between the \(T=1\) and \(T=0\) groups in terms of all of these characteristics, or do they seem approximatelycomparable?

Your response: TODO

3. Causal effect estimation#

Finally, let’s estimate the ATE for the voted outcome. From what we looked at in class, because this is a randomized experiment, we can estimate the ATE by computing the difference in means between the \(T=1\) and \(T=0\) groups for the voted outcome. For this particular dataset, that means:

\[\begin{split} \begin{align*} \text{ATE} = E[Y(1) - Y(0)] \xrightarrow[]{\text{Identification}} \; &E[Y | T=1] - E[Y | T=0]\\ E[Y | T=1] - E[Y | T=0] \; \xrightarrow[]{\text{Estimation}} \; &\hat{E}[Y | T=1] - \hat{E}[Y | T=0]\\ =&\hat{P}(\text{voted='Yes'} \mid \text{treatment='Neighbors'}) - \hat{P}(\text{voted='Yes'} \mid \text{treatment='Civic Duty'}) \end{align*} \end{split}\]

Tip

Try computing the two proportions separately and then subtracting them.

We can compute the proportion of respondents who voted ‘Yes’ in each treatment group by using the .value_counts() method with normalize=True and indexing on the 'Yes' category.

vote_df.loc[vote_df['treatment'] == 'Civic Duty']['voted'].value_counts(normalize=True)

voted
No     0.685462
Yes    0.314538
Name: proportion, dtype: float64

# TODO your code for estimating the ATE
estimated_ATE = 0.377 - 0.314
print(estimated_ATE)

0.063

Write down an interpretation of this estimate as a causal quantity.

Your response: we see a 6.3 percent increase in the probability of voting in the treatment group of neighbors vs civic duty.

References#

Gerber, Alan S., Donald P. Green, and Christopher W. Larimer, 2008, Replication Materials for “Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment.” http://hdl.handle.net/10079/c7507a0d-097a-4689-873a-7424564dfc82. ISPS Data Archive.
Full article: https://isps.yale.edu/sites/default/files/publication/2012/12/ISPS08-001.pdf