A Metacase in economics

From 2013, central inspection teams entered China's government departments in five waves. Lai and Li (2024) ask whether competition for those departments' civil-service posts changed afterwards. The task reproduces one result as three numbers: the coefficient (how large the effect is), its standard error (how uncertain) and its p-value (whether significant).

The first difficulty is that it cannot simply be run. The task supplies only the authors' data; the matching Stata script is withheld, and there is no Stata in the container. The agent must read the three source programs behind the command, one over four thousand lines, and rebuild them within 1%.

The second is an incomplete specification: the task does not say which covariates, the other factors a regression controls for, to include. In the authors' script, none gives −66.33, the reference; five give −69.35, 4.6% away, which scores 0.

01Context & evidence

Scoping the data, and marking what is open.

One data file, one paragraph of instructions, no original script. First the agent settles what the data can estimate, and which settings are stated or open.

  1. Data3,246 department-year observations on 717 departments, 2012–2017, with 30 labelled variables, taken from the authors' public replication package.
  2. VariablesThe outcome comp is the applicant-to-hire ratio (labelled AHR); the treatment inspectmain turns 1 once a leading department is inspected. Three more variables mark the wave, or never inspected.
  3. Treatment and sample87 of the 119 leading departments were inspected in five cohorts, 2013–2017, giving 180 department-years after inspection. Never-inspected departments form the control group.
  4. Four settingsDepartment and year fixed effects (removing each one's own average), cohorts by first inspection year, never-inspected controls, and clustering by department (errors may correlate within a department).
  5. Open items and constraintsThree numbers and a Python script that reproduces them. No authors' script, no Stata, 30 minutes; the task does not say which column's covariates to use.

Kept in the Metacase as Goal · Evidence · Constraints
what was known at the decision point

3,246department-year observations

02Prior practice & strategy

Method from public source, specification from the task.

The estimation command the task names is a public, general-purpose program, and every choice is kept on record.

  1. Rules and practiceThe command eventstudyinteract is the Sun–Abraham estimator: one effect per cohort, weighted by its share of treated observations. Unlike plain two-way fixed effects, it never uses inspected departments as controls.
  2. Looking for the scriptNot in the container. GitHub, Google, Bing, Harvard Dataverse, DataCite and Crossref yield two replication packages for other studies, neither with the script; the agent declines both.
  3. Back to the sourceAt 24 seconds it fetches the source of eventstudyinteract, then of reghdfe, on which it depends; a fetch of avar fails. Both are general-purpose programs with nothing specific to this study.
  4. Choice of specificationThe stated settings include no covariates, yet the task mentions covariates “per table column”. None gives −66.33; other sets give other numbers, and the agent picks none without support.
  5. A note left behindThe script's header and a separate note state that this is the baseline without covariates, and that the covariates of the script's columns could not be verified.

Kept in the Metacase as Practice · Options · Trade-offs
the options weighed, and the reasons on record

2public source files obtained

03Actions & execution

Estimator rebuilt in Python, then checked another way.

gpt-6-astra runs in the Terminus-2 harness for about eight minutes: 40 model requests and 52 tool calls, all logged.

  1. Source readingThe combined effect weights five cohort effects by share (14, 8, 65, 79 and 14 treated observations); its variance adds a term for estimating those shares to the regression's clustered variance.
  2. Sample definitionSeventy-six departments appear once, one of them treated. reghdfe drops such singletons, leaving 3,170 observations on 641 departments; the shares must be computed before the drop.
  3. Degrees of freedomFrom the source of reghdfe it confirms the small-sample correction: department effects are nested in clusters and not counted again; six year effects count as five parameters.
  4. Independent checkRerun with explicit department and year dummies, the five cohort coefficients agree with the first result to numerical precision.
  5. FailureOne outside lookup hangs and holds the terminal, so the queued commands that write the files never run. The agent interrupts it, finds both files missing and writes them again.

Kept in the Metacase as Steps · Tools · Intermediate work
every step, in order

52tool calls

04Outcomes & verification

Out-of-loop verification: three numbers compared.

Harbor adds the grader and the reference only after the agent stops. The grader calls no model; it compares the result file with the reference.

  1. Verified figuresCoefficient −66.3269, standard error 25.0531. At the result file's full precision, the coefficient matches the reference to 11 decimal places and the standard error to nine.
  2. Basis of the p-valueThe agent uses a t distribution with 640 degrees of freedom (641 departments less one): 0.00831. The reference, 0.00811, matches the normal; the gap of 0.0002 is within 0.01.
  3. Decision ruleCoefficient and standard error within 1% relative error (the difference as a share of the reference), the p-value within 0.01. All three pass: 1; otherwise 0.
  4. The weight of a specificationAfterwards, column 2's five covariates from the authors' public script, which the agent never obtained, were run through its program: −69.35, standard error 22.68, 4.6% and 9.5% off. Score: 0.

Kept in the Metacase as Deliverable · Result · Checks · Limits
the result, and the limits of the check

1.000grader verdict, one pilot run

05Reflection & reuse

Reviewing the specification, and where replication stops.

The record breaks one number into the settings it depends on, so a later attempt can check each of them rather than compare results.

  1. What heldEstimator, fixed effects, clustering and singletons follow the public source, the control cohort follows the task, and a second method confirms the result. No covariate added without support; what is unverified is stated.
  2. The four settings in the task
    What they show
    Fixed effects, cohort variable, control cohort and clustering are stated; covariates get one phrase, “per table column”.
    Why it matters
    Different covariates give different numbers, here 4.6% apart; picking a set without support decides for the researcher.
    This run
    Reported without covariates, and stated as unverified.
  3. Variable labels in the data
    What they show
    Three variables mark each department's wave, and whether it was never inspected.
    Why it matters
    With staggered treatment, the not-yet-treated are treated later and cannot be controls throughout; wrong controls turn timing gaps between cohorts into effects.
    This run
    Never-inspected departments as the control cohort.
  4. Source of eventstudyinteract
    What it shows
    The variance of the combined effect has two terms: the clustered variance of the cohort effects and that of the estimated shares.
    Why it matters
    The first alone gives 24.89, not 25.05: the variance is understated and the interval too narrow.
    This run
    Both terms included.
  5. Default setting of reghdfe
    What it shows
    The 76 departments observed once are dropped by default; the shares are computed before that.
    Why it matters
    Dropping sets the sample (3,246 or 3,170), the clusters (717 or 641) and the correction; in the wrong order, the weights change too.
    This run
    Shares first, then dropped as in the source.
  6. Basis of the p-value
    What it shows
    The agent uses a t distribution with 640 degrees of freedom; the reference agrees with the normal distribution.
    Why it matters
    For the same t statistic, −2.647, the two bases differ by 0.0002; a report should state its distribution and degrees of freedom.
    This run
    The script states its basis; the gap is within tolerance.
  7. What failedThe p-value's basis differs from the reference and passes only on tolerance. The authors' script was never found, so the baseline rests on the task's settings alone. The estimate stood at 185 seconds; five more minutes went on searching, and a stalled lookup nearly left the files unwritten.
  8. Conditions and signatureApplies where public code defines the estimator and the task leaves the specification open. Signature: open specification, source comparison, numerical replication, stating what was not verified.

Kept in the Metacase as What held · What failed · When it applies
what the next attempt can borrow or change

Watch the run in full

econ-0157 · gpt-6-astra · Terminus-2 2.0.0 · 4 min 59 sec, with English captions
The grader, in full

Reward = 1 when all three comparisons pass; otherwise 0.

The grader sees two things: the result file the agent wrote (the only thing scored) and the reference values. It checks only that the reproducing script exists, and tries to run it only when the result file is missing.

3 comparisons · all must pass
ComparedReferenceValue in this runErrorToleranceVerdict
Coefficient−66.326901−66.3269010.00% relativeRelative error of at most 1%Pass
Standard error25.05311625.0531160.00% relativeRelative error of at most 1%Pass
p-value0.0081100.0083100.0002 absoluteAbsolute error of at most 0.01Pass
4 diagnostics · recorded, not scored
DiagnosticMeaningThis run
Partial replicationCoefficient and standard error both within 5%Yes
Coefficient onlyCoefficient within 5%Yes
DirectionSame sign as the referenceYes
SignificanceSame sign, and the p-value in the same band (1%, 5%, 10% or not significant)Yes
One data set, two specifications: with and without covariates
SpecificationCovariatesCoefficientStandard errorDifference from the referenceThis run
Column 1 (reference)None−66.3325.05Coefficient 0.0%, standard error 0.0%Reported by the agent
Column 2Five−69.3522.68Coefficient 4.6%, standard error 9.5%Recomputed afterwards with the same program