Causal Inference
Hey students! š Welcome to one of the most important concepts in public policy research - causal inference! This lesson will teach you how researchers figure out whether policies actually work or if they just look like they work. By the end of this lesson, you'll understand the difference between correlation and causation, learn about various research designs that help us make causal claims, and discover the common pitfalls that can lead us astray. Think of yourself as a detective šµļø - we're going to learn how to find the real culprits behind policy outcomes!
Understanding Causation vs. Correlation
Let's start with the foundation, students. You've probably heard the phrase "correlation doesn't imply causation" before, but what does this really mean in the world of public policy?
Imagine your city implements a new after-school program, and test scores go up the following year. Did the program cause the improvement? Not necessarily! Maybe parents became more involved in their children's education that year, or perhaps the school district hired better teachers. This is the fundamental challenge of causal inference - separating the true effect of a policy from all the other factors that might influence outcomes.
In research terms, we call the policy or intervention we're studying the treatment. The outcome we're measuring (like test scores) is the dependent variable. The tricky part is that we can never observe what would have happened to the same group of people if they hadn't received the treatment - this hypothetical scenario is called the counterfactual.
Real-world example: When researchers studied whether job training programs help unemployed workers find jobs, they found that participants had higher employment rates than non-participants. But here's the catch - people who sign up for job training might be more motivated to find work anyway! This selection bias could make the program look more effective than it actually is.
Experimental Designs: The Gold Standard
The most reliable way to establish causation is through randomized controlled trials (RCTs) šÆ. In an RCT, researchers randomly assign some people to receive the treatment (treatment group) and others to not receive it (control group). Because assignment is random, the two groups should be similar in all ways except for the treatment.
Think of it like flipping a coin to decide who gets the new policy intervention. This randomness is crucial because it eliminates selection bias - the problem we mentioned earlier where certain types of people might be more likely to participate.
A famous example is the Moving to Opportunity experiment conducted in the 1990s. The U.S. Department of Housing and Urban Development randomly selected some low-income families to receive housing vouchers that helped them move to lower-poverty neighborhoods. By comparing these families to those who didn't receive vouchers, researchers could measure the true causal effects of neighborhood quality on children's outcomes.
The results? Children who moved to better neighborhoods when they were young (under age 13) had significantly higher earnings as adults - about 31% more than the control group! This is powerful evidence that neighborhood environment causally affects life outcomes.
However, RCTs aren't always possible or ethical. You can't randomly assign some cities to have high pollution levels just to study health effects! This is where quasi-experimental designs come in.
Quasi-Experimental Designs: Creative Solutions
When randomization isn't possible, researchers use quasi-experimental designs that try to mimic the logic of experiments using naturally occurring variation. Let me walk you through the most important ones, students.
Natural Experiments occur when some external force creates random or as-good-as-random assignment to treatment. For example, researchers studied the effects of military service on earnings by using the Vietnam War draft lottery. Since draft numbers were randomly assigned, men with low numbers (who were drafted) could be compared to those with high numbers (who weren't) to measure the causal effect of military service.
Regression Discontinuity Design exploits arbitrary cutoffs in policy rules. Imagine a scholarship program that gives money to all students with GPAs of 3.5 or higher. Students with GPAs just above and just below 3.5 are very similar, but only those above get the scholarship. By comparing these nearly identical groups, researchers can estimate the causal effect of financial aid.
A real example: researchers studied whether smaller class sizes improve student achievement by looking at a rule in Israel where classes with more than 40 students must be split. Schools with 40 students had one class, while schools with 41 students had two classes of about 20 each. This arbitrary rule created a natural experiment showing that smaller classes do indeed improve test scores.
Difference-in-Differences compares changes over time between treatment and control groups. Let's say State A implements a new education policy in 2020, but State B doesn't. We compare how outcomes changed in State A versus State B from before to after 2020. This design controls for factors that affect both states equally and factors that are constant within each state over time.
Instrumental Variables use a third variable (the instrument) that affects treatment but only affects the outcome through its effect on treatment. For example, researchers studying the effects of education on earnings might use compulsory schooling laws as an instrument. These laws force some people to stay in school longer, but they only affect earnings through their impact on education levels.
Common Threats to Causal Inference
Even with careful research designs, several threats can undermine our ability to make causal claims. Let me help you spot these potential problems, students! šØ
Selection Bias occurs when the treatment and control groups differ in important ways beyond the treatment itself. We touched on this earlier with the job training example. People who choose to participate in programs might be systematically different from those who don't.
Omitted Variable Bias happens when we fail to account for important factors that affect both treatment assignment and outcomes. For instance, if wealthier neighborhoods are more likely to implement new policing strategies AND have lower crime rates for other reasons (better schools, more social services), we might incorrectly attribute all the crime reduction to the policing strategy.
Reverse Causality is when the outcome actually causes the treatment, rather than the other way around. For example, if we observe that areas with more police have more crime, we might wrongly conclude that police cause crime. In reality, crime probably causes cities to hire more police!
Measurement Error can bias results if our measures of treatment or outcomes are inaccurate. If some people in the "control" group actually received the treatment but we don't know it, our estimates will be biased toward zero.
External Validity concerns whether results from one context apply to others. A job training program that works well in one city might not work in another city with different economic conditions or demographics.
Advanced Identification Strategies
Modern policy research employs sophisticated strategies to strengthen causal inference. Propensity Score Matching tries to create comparable treatment and control groups by matching individuals with similar probabilities of receiving treatment based on observed characteristics.
Synthetic Control Methods create artificial control groups by combining multiple untreated units to match the treated unit's pre-treatment characteristics. This approach was famously used to study the economic effects of German reunification by creating a "synthetic West Germany" from other developed countries.
Event Study Designs examine how outcomes evolve before and after policy implementation, helping researchers test whether effects appear exactly when expected and whether pre-trends were parallel between groups.
Conclusion
Causal inference is the cornerstone of evidence-based policymaking, students. While establishing causation is challenging, the research designs we've explored - from randomized experiments to creative quasi-experimental approaches - provide powerful tools for separating true policy effects from mere coincidence. Remember that good research design is about more than just fancy statistical techniques; it's about careful thinking, understanding context, and being honest about limitations. As you encounter policy debates in the future, you'll now be equipped to ask the right questions: How do we know this policy caused these outcomes? What alternative explanations might exist? The ability to think causally will make you a more informed citizen and a better policy analyst! š
Study Notes
ā¢ Causal inference: Determining whether a policy or intervention actually causes observed outcomes, not just correlates with them
ā¢ Treatment: The policy, program, or intervention being studied
ā¢ Counterfactual: What would have happened to the treatment group if they hadn't received the treatment
ā¢ Randomized Controlled Trial (RCT): Gold standard design where participants are randomly assigned to treatment or control groups
ā¢ Selection bias: When treatment and control groups differ systematically beyond the treatment itself
ā¢ Natural experiment: When external forces create random or quasi-random assignment to treatment
ā¢ Regression discontinuity: Exploits arbitrary cutoffs in policy rules to compare similar individuals on either side of the threshold
ā¢ Difference-in-differences: Compares changes over time between treatment and control groups: $$\text{Effect} = (\text{Treatment}_{after} - \text{Treatment}_{before}) - (\text{Control}_{after} - \text{Control}_{before})$$
ā¢ Instrumental variables: Uses a third variable that affects treatment but only affects outcomes through treatment
ā¢ Omitted variable bias: Bias from failing to control for factors that affect both treatment and outcomes
ā¢ Reverse causality: When the outcome actually causes the treatment rather than vice versa
ā¢ External validity: Whether results from one study apply to other contexts or populations
ā¢ Internal validity: Whether the study design allows for valid causal inferences within the study context