Activity 14: Panel data

Activity 14: Panel data#

2025-04-15

import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf

Part 1#

With panel data, instead of having a single row per unit in our dataframes, we have potentially multiple datapoints per unit across time. Given that:

  • December: \(t=1\)

  • March: \(t=2\)

  • June: \(t=3\)

We have 3 datapoints for each town. The “Post-treatment period?” column is a binary variable that is 1 if the datapoint is in the post-treatment period and 0 otherwise.

Finish populating the markdown table below with the correct values:

Unit

Time

Outcome

Post-treatment period?

South Hadley

1

100

0

South Hadley

2

90

0

South Hadley

3

70

1

TODO

We can use pandas MultiIndex to represent the multiple indices needed for panel data. The pd.set_index() can take a list of columns to use as the new index.

traffic_df = pd.DataFrame(
    {
        'town': ['South Hadley', 'South Hadley', 'South Hadley', 'Hadley', 'Hadley', 'Hadley'],
        'time': [1, 2, 3, 1, 2, 3],
        'outcome': [100, 90, 70, 80, 70, 60],
        "post_treatment": [0, 0, 1, 0, 0, 1]
    }
)

# TODO set the index to be the['town', 'time'] columns
#traffic_df = traffic_df.set_index(["TODO"])

display(traffic_df)

# note that time and town are no longer columns
display(traffic_df.columns)
town time outcome post_treatment
0 South Hadley 1 100 0
1 South Hadley 2 90 0
2 South Hadley 3 70 1
3 Hadley 1 80 0
4 Hadley 2 70 0
5 Hadley 3 60 1 
Index(['town', 'time', 'outcome', 'post_treatment'], dtype='object')

The multi-index is now, where the first level (level=0) is the town and the second level (level=1) is the time.

With a multi-index, the .loc method can take a tuple that specifies an index to retrieve:

# selects all the South Hadley datapoints
display(traffic_df.loc["South Hadley"])

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[3], line 2
      1 # selects all the South Hadley datapoints
----> 2 display(traffic_df.loc["South Hadley"])

File ~/Github/comsc341-cd/venv/lib/python3.11/site-packages/pandas/core/indexing.py:1191, in _LocationIndexer.__getitem__(self, key)
   1189 maybe_callable = com.apply_if_callable(key, self.obj)
   1190 maybe_callable = self._check_deprecated_callable_usage(key, maybe_callable)
-> 1191 return self._getitem_axis(maybe_callable, axis=axis)

File ~/Github/comsc341-cd/venv/lib/python3.11/site-packages/pandas/core/indexing.py:1431, in _LocIndexer._getitem_axis(self, key, axis)
   1429 # fall thru to straight lookup
   1430 self._validate_key(key, axis)
-> 1431 return self._get_label(key, axis=axis)

File ~/Github/comsc341-cd/venv/lib/python3.11/site-packages/pandas/core/indexing.py:1381, in _LocIndexer._get_label(self, label, axis)
   1379 def _get_label(self, label, axis: AxisInt):
   1380     # GH#5567 this will fail if the label is not present in the axis.
-> 1381     return self.obj.xs(label, axis=axis)

File ~/Github/comsc341-cd/venv/lib/python3.11/site-packages/pandas/core/generic.py:4301, in NDFrame.xs(self, key, axis, level, drop_level)
   4299             new_index = index[loc]
   4300 else:
-> 4301     loc = index.get_loc(key)
   4303     if isinstance(loc, np.ndarray):
   4304         if loc.dtype == np.bool_:

File ~/Github/comsc341-cd/venv/lib/python3.11/site-packages/pandas/core/indexes/range.py:417, in RangeIndex.get_loc(self, key)
    415         raise KeyError(key) from err
    416 if isinstance(key, Hashable):
--> 417     raise KeyError(key)
    418 self._check_indexing_error(key)
    419 raise KeyError(key)

KeyError: 'South Hadley'

# selects the row for South Hadley at time 1
display(traffic_df.loc[("South Hadley", 1)])

# equivalently, we can chain the `.loc` method to filter different levels of the multi-index
display(traffic_df.loc["South Hadley"].loc[1])

To select rows based on the second level of the multi-index, we can use pd.xs, which takes a cross-section of the DataFrame:

# Select all rows where the second level of the multi-index (time) equals 1
traffic_df.xs(1, level=1)

Write a line of code to select the Hadley datapoint at time 3, and submit your answer to pollEverywhere:

Part 2#

Run the cell below to load the organ donation data. The dataframe has the following columns:

  • State: the state name

  • Quarter: the quarter of data

  • Quarter_Num: the quarter number

  • Rate: the organ donation registration rate

organ_df = pd.read_csv("~/COMSC-341CD/data/organ_donations.csv")

Since the data is quarterly and begins in 2010 Q4, the first post-treatment period (after July 2011) is 2011 Q3, which corresponds to Quarter_Num = 4. Create the following columns to prepare the data for a difference-in-differences analysis:

  • is_california: a binary variable indicating whether the state is California

  • post_treatment: a binary variable indicating whether the quarter is after 2011 Q3 (Quarter_Num >= 4)

  • is_treated: a binary variable indicating whether the state is California AND the quarter is after 2011 Q3

# TODO: Create the columns
organ_df['is_california'] = None
organ_df['post_treatment'] = None
organ_df['is_treated'] = None

Like we did in part 1, set the index to be the ['State', 'Quarter_Num'] columns.

# TODO set the multi-index
#organ_df.set_index(TODO)

Finally, let’s visually evaluate the parallel trends assumption by plotting the rate against the quarter number in the pre-treatment period.

# TODO select the dataframe for the pre-treatment period
organ_df_pre = None


# TODO plot an sns.pointplot using organ_df_pre of 'Rate' against 'Quarter_Num', with 'is_california' as the hue
# sns.pointplot()

Does there appear to be any clear violations of the parallel trends assumption?

Part 3#

We just discussed the following formula for using regression to compute a difference-in-differences estimate:

\[ Y = \beta_0 + \beta_1 (\text{treated group}) + \beta_2 (\text{after treatment}) + \beta_3 (\text{treated group} \times \text{after treatment}) + \epsilon \]

Write the formula in terms of the variables in the organ_df dataframe we created in part 2. The outcome of interest is Rate, while the treated group is California.

# TODO your code here
formula = ''

# did_model = smf.ols(TODO)
# did_results = did_model.fit()
# print(did_results.params)

What is your ATT estimate of the effect of active choice vs opt-in on California organ donation rates?

Acknowledgements#

This activity is derived from Nick Huntington-Klein’s analysis of Kessler and Roth (2014) in Chapter 18 of The Effect.

Contents