Project 3 Part 2: Analysis 💉

Instrumental Variables

—TODO your name here

Collaboration Statement

• TODO brief statement on the nature of your collaboration.

• TODO your collaborator’s names here

Part 2 Rubric

Section	Points
Question and Design: McDonald et al.	1
Estimation	1
Interpretation	1
Reflection	0.5
Total	3.5 pts

Notebook and function imports

Tip

If you click on the vertical blue bar on the left of a cell, you can collapse the code which can help organize the notebook as you work through the project.

If you have tested your implementation in Part 1 against the autograder, you would have generated a file called proj3_functions.py. Let’s now import those functions into this notebook for use in Part 2.

If you are running this notebook on the JupyterHub allocated for the course:

1. Open the file browser by going to the menu bar “View -> File Browser”

2. Navigate to comsc341cd.github.io/projects/, you should see your proj3_analysis.ipynb file in that folder

3. Click on the upload button in the upper right and upload the proj3_functions.py file to this directory

4. Run the following cell to import the functions

Show code cell content Hide code cell content

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import statsmodels.formula.api as smf
import ipywidgets as widgets

from ipywidgets import interact_manual

from proj3_functions import tsls_with_covariates, complier_covariate_mean, wald_estimator, intent_to_treat, prop_never_takers, prop_always_takers, prop_compliers

rng = np.random.RandomState(42)

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 10
      6 import ipywidgets as widgets
      8 from ipywidgets import interact_manual
---> 10 from proj3_functions import tsls_with_covariates, complier_covariate_mean, wald_estimator, intent_to_treat, prop_never_takers, prop_always_takers, prop_compliers
     12 rng = np.random.RandomState(42)

ModuleNotFoundError: No module named 'proj3_functions'

Tip

If your widget from part 1 is rendering after you run the above cell, you can remove it by commenting out the @interact_manual decorator in your proj3_functions.py file and restarting the kernel for this notebook.

Like Project 2, we have provided implementations of bootstrap_ci, bootstrap_dfs to estimate the confidence interval and bootstrap distribution of our IV estimators.

Show code cell content Hide code cell content

def bootstrap_ci(bootstrap_values, alpha=0.05):
    """
    Computes the confidence interval using the percentile method.

    Args:
        bootstrap_values (list[float]): the bootstrapped values
        alpha (float): the significance level, defaults to 0.05

    Returns:
        list[float]: the confidence interval [lower, upper]
    """
    lower = np.percentile(bootstrap_values, alpha * 100)
    upper = np.percentile(bootstrap_values, (1 - alpha) * 100)
    return [lower, upper]


def bootstrap_dfs(df, n_bootstraps=1000):
    """
    Bootstraps the dataframe `n_bootstraps` times.

    Args:
        df (pd.DataFrame): the dataframe to bootstrap
        n_bootstraps (int): the number of bootstraps to generate

    Returns:
        list[pd.DataFrame]: a list of bootstrapped dataframes
    """
    bootstrap_dfs = []

    for i in range(n_bootstraps):
        bootstrap_dfs.append(df.sample(frac=1, replace=True))

    return bootstrap_dfs

McDonald et al. Vaccine Study

McDonald et al. (1992) was an early “nudge” study that used computerized reminders sent to physicians to encourage them to give flu vaccines to their patients. It has since been frequently used to study the relationship between intent-to-treat (ITT) and complier-based effects.

Prior Knowledge

Below is a motivation and description of the study design adapted and sourced from Hirano et al. (2000):

Because of epidemiologic evidence of increased morbidity related to influenza, experimental evidence of serologic efficacy (ie, antibody production) of the influenza vaccine, and observational studies suggesting improved outcomes in vaccinated patients, health officials in most countries recommend annual influenza vaccination for elderly persons and other people at high risk of influenza. However, no randomized controlled trials of the effects of the influenza vaccination on pulmonary (ie, lung and respiratory-related) morbidity in high-risk adults have been published (McDonald et al., 1992). One reason for this is that demonstrated efficacy for some subpopulations raises ethical barriers against performing randomized controlled trials on other subpopulations, which would require withholding vaccination from some subjects. One way around this impasse is to perform a randomized trial of an intervention that increases the use of influenza vaccine in one group of patients without affecting the use of influenza vaccine in another group. McDonald et al. (1992) exploited this idea to study influenza vaccine efficacy in reducing morbidity in high-risk adults, using a computer-generated reminder for flu shots. The study was conducted over a 3-year period (1978–1980) in an academic primary-care practice affiliated with a large urban public teaching hospital. Physicians in the practice were randomly assigned to either an intervention or a control group at the beginning of the study. Since physicians at the clinic each cared for a fixed group of patients, their patients were similarly classified. During the study period, physicians in the intervention group received a computer-generated reminder when a patient with a scheduled appointment was eligible for the influenza vaccine under U.S. Public Health Service Criteria.

We analyse this study using 2861 individuals observed in 1980, a particularly severe flu epidemic season. For each person \(i\) we observe: a binary variable \(Z_i\) , the ‘assignment’ or ‘encouragement’, equal to one if patient \(i\)’s physician received a reminder letter indicating that the patient was eligible to receive the influenza vaccine under U.S. Public Health Service Criteria and zero otherwise; a binary variable \(T_i\) , the ‘treatment’, equal to 1 if person \(i\) received the vaccine and 0 otherwise; a binary outcome \(Y_i\) , equal to 1 if person \(i\) subsequently experienced a flu-related hospitalization during the winter, which we define as being hospitalized for respiratory problems, and 0 otherwise.

flu_df = pd.read_csv('~/COMSC-341CD/data/flu_iv.csv')

The primary variables of interest in the dataset are as follows:

• \(Z\) assign: 1 if physician was randomly assigned to the intervention group, 0 if control

• \(T\) receive: 1 if a patient received the flu shot, 0 if not

• \(Y\) outcome: 1 if patient was hospitalized for with flu-related complications, 0 if not

We additionally have the following set of covariates:

• age: age of the patient

• female: 1 if patient is female, 0 if male

• copd: 1 if patient has a chronic obstructive pulmonary disease (COPD) like emphysema or chronic bronchitis, 0 if not

• white: 1 if patient is white, 0 if not

• renal: 1 if patient has renal disease, 0 if not

• dm: 1 if patient has diabetes, 0 if not

• heartd: 1 if patient has heart disease, 0 if not

• liverd: 1 if patient has liver disease, 0 if not

1. Question and Design

In the markdown cell below, provide a description of the “question” and “design” steps of the causal roadmap for this study (~1-2 brief paragraphs). You may format your response however you like, but make sure to include the following:

• 1.1: An interpretation of the three causal quantities that we could potentially estimate from this study design

• 1.2: A brief discussion of the covariates of age and copd, and why they may be relevant confounders of the treatment and outcome variables in the study

• 1.3: An assessment of the validity of the IV assumptions of relevance, exclusion restriction, and instrument unconfoundedness

• 1.4: A DAG of the study design embedded as an image using markdown syntax![](image.png), including the covariates of age and copd

Our study design choice is an instrumental variable design. In an encouragement study analyzed as an IV design, we have potentially multiple causal quantities of interest that we can estimate:

• Intent-to-treat (ITT)

  • \(E[Y|Z=1] - E[Y|Z=0]\)

  • Interpretation in the context of the study: TODO

• Average treatment effect (ATE)

  • \(E[Y(1) - Y(0)]\)

  • Interpretation in the context of the study: TODO

• Local average treatment effect (LATE)

  • \(E[Y(1) - Y(0) \mid \text{complier}]\)

  • Interpretation in the context of the study: TODO

TODO 1.2 confounding discussion

TODO 1.3 IV assumption assessment

2. Estimation

Next, perform analysis and estimation of causal quantities of the flu dataset. Your analysis should include the following:

• 2.1: A table of descriptive statistics for age, copd, female and white grouped by actual treatment receive, along with a brief discussion of what you observe.

• 2.2: A point estimate of the intent-to-treat effect of assign on outcome using your intent_to_treat function, along with a bootstrapped confidence interval with 1,000 bootstraps (which is the default number of bootstraps in the bootstrap_dfs function).

• 2.3: A point estimate of the ATE/LATE (recall that the ATE and LATE in an IV design are the same) using your tsls_with_covariates function, along with a bootstrapped confidence interval for the following covariate sets:

  • no covariates

  • all covariates

• 2.4: the proportion of compliers, never takers, and always takers in the sample

• 2.5: A table of mean/percentages for age, female, copd, and white among:

  • the entire sample

  • compliers, by using your complier_covariate_mean function

  • always takers, by selecting the subset of units that are always takers

  • never takers, by selecting the subset of units that are never takers

Tip

You can refer back to the way we formatted tables and confidence intervals in Project 1, as well as how you computed bootstrap confidence intervals in both Projects 1 and Project 2.

Remember that the point estimate (treatment effect) should be estimated using the original dataset, while the confidence interval should be estimated using the bootstrapped distribution.

# TODO your code here for 2.1-2.3 estimation and analysis

covariate_subset = ['age', 'female', 'copd', 'white']
all_covariates = ['age', 'female', 'copd', 'white', 'renal', 'dm', 'heartd', 'liverd']

# TODO your code here for 2.4-2.5 compliance status proportions

2.1 Baseline descriptive statistics

Covariate	\(T=1\)	\(T=0\)
mean age	TODO
% female
% copd
% white

2.2 Intent-to-treat (ITT) estimate

Estimate	95% CI
TODO

2.3 Instrumental variable estimates

Model	Estimate	95% CI
No covariates	TODO
All covariates

TODO 2.3 ATE/LATE with and without covariates

TODO 2.4 compliance status proportions

• always takers

• never takers

• compliers

2.5 Covariates by compliance status

Covariate	Overall	Complier	Always Taker	Never Taker
mean age
% female
% copd
% white

3. Interpretation

Interpret your estimation results from part 2:

• 3.1: Compare the ITT and IV point estimates (ignoring confidence intervals for now). How do they differ in terms of sign and magnitude? Why might this difference exist (think about what \(Z\) and \(T\) are for this study)?

• 3.2: You should find that both IV estimates have confidence intervals that include 0. What does this tell us about the flu vaccine’s effect on hospitalizations in this study? How should we interpret these results, and what are the limitations of drawing strong conclusions from this single study? Some things you can consider:

  • the sample size and sample representativeness

  • the choice to define the outcome as flu-related hospitalization

  • the choice of instrument and covariates

• 3.3: Looking at your table of covariate means among different compliance types (compliers, always-takers, never-takers), select one covariate and compliance type that shows a notable difference from the overall mean and speculate on why we might observe this difference. For example, “The mean age of compliers is higher/lower than the overall mean, which could be due to to xyz.”

• 3.4: Read the excerpt from Imbens (2014) below on interpreting the LATE. Discuss whether the LATE seems to be a useful quantity to report for this study. Some things you can consider:

  • whether healthcare systems should care about effects specifically for compliers versus the general population

  • how the difference between compliers and non-compliers might inform future intervention design

  • what ethical considerations arise when we focus on treatment effects for only a subset of the population (see the proportion of compliers in the sample in 2.4)

  • how you might communicate these results and limitations to non-technical decision makers and the general public

Do We Care About the Local Average Treatment Effect?

The local average treatment effect is an unusual [causal quantity]. It is an average effect of the treatment for a subpopulation that cannot be identified in the sense that there are no units whom we know for sure to belong to this subpopulation, although there are some units whom we know do not belong to it. A more conventional approach is to start an analysis by clearly articulating the object of interest, say the average effect of a treatment for a well-defined population. There may be challenges in obtaining credible estimates of this object of interest, and along the way one may make more or less credible assumptions, but typically the focus remains squarely on the originally specified object of interest.

Here, the approach appears to be quite different. We did not articulate explicitly what the target [causal quantity] was. In the McDonald et al. influenza-vaccine application a natural quantity might be the [average treatment effect] of the vaccine. Then, apparently more or less by accident, the definition of the compliance types led us to focus on the average effects for compliers. In this example, the compliers were defined by the response in terms of the receipt of the vaccine to the receipt of the letter. It appears difficult to argue that this is a substantially interesting group, and in fact no attempt was made to do so.

This type of example has led distinguished researchers both in economics and in statistics to question whether and why one should care about the local average treatment effect. The economist Angus Deaton writes “I find it hard to make any sense of the LATE [local average treatment effect]”. Judea Pearl similarly wonders “Realizing that the population averaged treatment effect (ATE) is not identifiable in experiments marred by noncompliance, they have shifted attention to a specific response type (i.e., compliers) for which the causal effect was identifiable, and presented the latter [the local average treatment] as an approximation for ATE. However, most authors in this category do not state explicitly whether their focus on a specific stratum (the compliers) is motivated by mathematical convenience, mathematical necessity (to achieve identification) or a genuine interest in the [group] under analysis”…Let me attempt to clear up this confusion. An instrumental variables analysis is an analysis in a second-best setting. It would have been preferable if one had been able to carry out a well-designed randomized experiment. However, such an experiment was not carried out, and we have noncompliance. As a result, we cannot answer all the questions we might have wanted to ask. Specifically, if the noncompliance is substantial, we are limited in the questions we can answer credibly and precisely. Ultimately, there is only one subpopulation we can credibly (point-)identify the average effect of the treatment for, namely, the compliers.

It may be useful to draw an analogy. Suppose a researcher is interested in evaluating a medical treatment and suppose a randomized experiment had been carried out to estimate the average effect of this new treatment. However, the population of the randomized experiment included only [college students], and the researcher is interested in the average effect for the entire population…What should the researcher do? I would argue that the researcher should report the results for the [college students], and acknowledge the limitation of the results for the original question of interest. Similarly, in the instrumental variables I see the limitation of the results to the compliers as one that was unintended, but driven by the lack of identification for other subpopulations given the design of the study. This limitation should be acknowledged, but one should not drop the analysis simply because the original [causal quantity] cannot be identified. Note that our case with instrumental variables is slightly worse than in the given example, because we cannot actually identify all individuals with certainty as compliers.

TODO your responses:

3.1:

3.2:

3.3:

3.4:

4. Reflection

• 4.1. How much time did you spend on this assignment?

• 4.2. Were there any parts of the assignment that you found particularly challenging?

• 4.3. What is one thing you have a better understanding of after completing this assignment and going through the class content?

• 4.4. Do you have any follow-up questions about concepts that you’d like to explore further?

TODO your responses:

4.1:

4.2:

4.3:

4.4:

Typesetting and submission

In addition to the usual .ipynb file submission, we’ll also typeset this notebook using Jupyter Book, which will create a static HTML site that you can view locally or upload to GitHub Pages. The site is configured through the _config.yml file:

title: 'Project 3 Analysis 💉'
author: 'TODO'
home_page_in_navbar: false
# Optionally uncomment this if you want to use a custom theme
# sphinx:
#   config:
#     html_theme: sphinx_rtd_theme

Open this file by going to the menu bar “View -> File Browser” and navigating to the comsc341cd.github.io/projects/ directory. Then open the _config.yml and add your name to the author field.

Then, open a terminal by navigating to the File Browser and clicking “New -> Terminal”. Run the following commands to build the site and package it up into a .zip file:

# change directory to the projects folder
cd ~/comsc341cd.github.io/projects/

# builds the site, may take ~30-50 seconds to run since it's re-running all the cells
jb build proj3_analysis_solution.ipynb

# packages the site into a zip file
zip -r project3_analysis.zip _build/

You can then view your site by downloading the zip file, unzipping it, and opening the _build/html/proj3_analysis.html file in your browser. If you’ve updated the _config.yml file, you should see your name in the footer of the site.

How to submit

Once you’ve confirmed that your notebook is being rendered as a static site correctly, submit both your proj3_analysis.ipynb and proj3_analysis.zip files to Gradescope.

Optional explorations

To clean up your rendered notebook, feel free to delete the questions and prompts after you’ve completed the assignment.

If you’d like to change the theming of your rendered site, you can do so by uncommenting the # sphinx: section and changing the html_theme to one of the other themes. For example, if you’d like to use the “readthedocs” theme, uncomment the # sphinx: section and change the html_theme to sphinx_rtd_theme. You can see a gallery of the different themes at sphinx-themes.org.

You can also experiment with different typesetting content options available in Jupyter Book, such as margin comments and references: https://jupyterbook.org/en/stable/content/index.html