Module 4 · Intermediate · ~50 min

🔬 Research Workflows: Data & Analysis

The core of the course. Real, practical workflows for cleaning data, running analyses, debugging code, and generating outputs — with Stata, R, and Python examples throughout.

In this module

  1. The research workflow
  2. Data cleaning
  3. Running analysis
  4. Debugging code
  5. Generating tables
  6. Best practices: the verification habit
  7. Exercise: full pipeline

The research workflow

A typical quantitative research project moves through these stages, and Claude Code can help at every one:

Research pipeline

📥
Raw Data
Messy, inconsistent
🧹
Clean
Fix, merge, reshape
📊
Analyse
Regressions, tests
📋
Output
Tables, figures
✍️
Write
Paper, memo

Data cleaning

Data cleaning is where researchers spend the most time — and where Claude Code shines. Here are the most common tasks and how to prompt for them:

Merging datasets

> I have two datasets: hh_baseline.dta (household baseline survey)
  and hh_followup.dta (same households 2 years later). Both have
  hh_id as the household identifier. Merge them into
  hh_panel.dta and:
  - Flag households in baseline but not follow-up (attrition)
  - Flag households in follow-up but not baseline (new entrants)
  - Create an "attrition" dummy variable
  Save the merge report to logs/merge_report.txt

Fixing inconsistent data

> In hh_survey.dta, the district variable has inconsistent spelling:
  "Addis Ababa", "addis ababa", "A. Ababa", "Addis-Ababa".
  Standardise all variations to "Addis Ababa" (title case, no hyphens).
  Show me all unique district names before and after so I can verify.

Handling outliers

> In the income variable (inc_monthly_usd), some values look implausibly
  high (above 50,000) and some are negative. Don't delete them —
  instead, create:
  - A flag variable "inc_outlier" = 1 for these observations
  - A winsorized version "inc_wins" capped at the 99th percentile
  Summarise both original and winsorized versions so I can compare.

Merging datasets

> I have two tibbles: baseline_df (loaded from baseline.csv) and
  followup_df (from followup.csv). Both have hh_id as the key.
  Write an R script that:
  - Joins them with dplyr::full_join
  - Creates attrition and new_entrant indicator columns
  - Prints a summary of the merge quality
  - Saves the result to data/clean/panel_data.rds

Fixing messy column names

> Load survey_data.csv. The column names are a mess:
  "Household ID", "Monthly Income (USD)", "# of Children", etc.
  Use janitor::clean_names() to fix them, then show me the
  before/after comparison, and save to data/clean/survey_clean.rds

Missing value analysis

> For my panel dataset, create a missing value report:
  - Which variables have more than 10% missing?
  - Is missingness random or correlated with treatment status?
  - Create a visualisation using naniar::vis_miss()
  Save the report as output/missing_report.html

Merging datasets

> I have baseline.csv and endline.csv. Both have 'hh_id' as the key.
  Write a pandas script that:
  - Merges them with indicator='_merge'
  - Creates attrition and new_entrant columns
  - Prints a merge quality report (N matched, N left-only, N right-only)
  - Saves to data/clean/panel.parquet

Standardising categories

> The 'occupation' column in survey.csv has many variants of the same
  category: "farmer", "Farmer", "FARMER", "small farmer", "farm worker".
  Create a mapping dictionary and standardise to 5 clean categories.
  Show the before/after distribution with value_counts().

Running analysis

🧠
Your job: the identification strategy. Claude's job: the implementation. You decide whether an instrumental variables approach is valid. You decide whether the parallel trends assumption holds. Claude implements the estimator you choose. Never let Claude choose your empirical strategy. 
> Run my main specification:
  OLS regression of log(fert_kg_ha + 1) on insurance_take,
  controlling for: hh_size, land_ha, age_head, educ_head.
  Include village fixed effects and cluster standard errors at village level.
  Use reghdfe. Store estimates as "main_ols".
  Then run the same specification but instrument insurance_take
  with rainfall_sd using ivreg2. Store as "main_iv".
  Export both to a single LaTeX table using estout.
> Using my panel_data.rds, run the following:
  1. OLS with fixest::feols: log(fert + 0.01) ~ insurance | village_id,
     clustering by village_id. Store as mod1.
  2. IV with fixest::feiv: same outcome, instrument insurance with rainfall_sd,
     same FE and clustering. Store as mod2.
  3. Export both models to a LaTeX table using modelsummary.
     Include N, R2, FE indicator, and cluster info in the footer.
> Using panel.parquet, run with linearmodels:
  1. Pooled OLS: log(fert+0.01) ~ insurance + hh_size + land_ha
     with entity fixed effects (PanelOLS), clustering by village_id
  2. IV equivalent using IV2SLS, instrument insurance with rainfall_sd
  Print results and export to output/tables/main_results.tex (LaTeX format)

Debugging code

One of Claude Code's most valuable skills is debugging. When something breaks:

🐛
The best debugging prompt Don't just paste the error. Give Claude: (1) the error message, (2) the code that produced it, (3) what you expected it to do. Claude will diagnose and fix it much faster with all three pieces. 
> I'm getting this error in analysis.do at line 47:
  "variable rainfall_sd not found"

  Here's the relevant code (lines 40–55):
  [paste your code]

  What I expect: rainfall_sd should be in the merged dataset.
  The merge was done in clean_data.do. Can you figure out what went wrong?

Common debugging requests

Generating tables

> Create a summary statistics table comparing treatment vs. control groups.
  Variables: age_head, educ_head, hh_size, land_ha, inc_monthly_usd,
             fert_kg_ha (baseline), dist_market_km
  For each variable show: mean and SD for control, mean and SD for treatment,
  and the p-value of a t-test for difference.
  Add stars: * p<0.10, ** p<0.05, *** p<0.01
  Export to output/tables/balance_table.tex (estout format)
  And output/tables/balance_table.xlsx (for sharing with co-authors)
> Create a balance table from panel_data.rds.
  Use modelsummary::datasummary_balance() with the insurance_take variable.
  Variables: age_head, educ_head, hh_size, land_ha, inc_monthly_usd,
             fert_kg_ha_baseline
  Export to: output/tables/balance.tex and output/tables/balance.html
  The LaTeX version should have a caption and label for the paper.

Best practices: the verification habit

The most important habit when using Claude Code for analysis is always asking Claude to verify its own work. After every significant task: 

> You just cleaned the dataset. Before I use it, please:
  1. Check that no observations were unexpectedly dropped
     (compare N before and after)
  2. Verify the key identifier (hh_id) is unique with no duplicates
  3. Check for any variables that are now all-missing
  4. Make sure the treatment variable still has the same distribution
     as the raw data
  Tell me if anything looks wrong.
✅
The "sanity check" prompt After any data operation, ask: "Before I proceed, what could have gone wrong with what you just did? Are there edge cases that might have been handled incorrectly?" This catches 80% of errors before they compound.
🎯

Module 4 Exercise: Full pipeline

Use Claude Code to run a mini-analysis pipeline on a real or toy dataset. Here's a concrete task:

Download the PSID wage dataset (a classic labour economics dataset) or use your own data. Then ask Claude to:

  1. Load the data and create a summary statistics table
  2. Regress log wages on education, experience, and experience-squared — with robust standard errors
  3. Produce a coefficient plot showing the education return
  4. Export the regression table to LaTeX
  5. Write a 2-paragraph description of the results suitable for a paper

After each step, verify the outputs look correct before moving to the next.

Stata or R or Python ~35 min
← Previous 📋 CLAUDE.md Next module ✍️ Writing & LaTeX →