A Metacase in society

A cold-chain warehouse operator has given frontline supervisors at three hubs six weeks of incident-handling training; a fourth hub, deferred, kept its operating records for the same weeks as the reference. On 8 June a joint meeting decides acceptance and payment for the provider's three work packages, 816,000 yuan in all, on evidence closing 31 May; the agent prepares the recommendations.

The difficulty is that none of the 17 files is settled: a name in the wrong role, three missing sign-outs, two practice attempts, disputed observations, a repair dated a day apart. The provider still rates all three as passed, because the material was “uploaded”.

It must deliver a workbook that can be recomputed and a decision memo, telling “never learned” apart from “never had the chance”. Pending stays pending.

01Context & evidence

Evidence by status; three packages to decide.

Joint lead for learning effectiveness and procurement acceptance. Training ended 15 March, observation 24 May; every status stays as recorded.

  1. EvidenceEight detail files, five workbooks, four documents: 4,940 test responses, 755 attendance events, 341 opportunity records, 1,145 observations, 720 tickets and 96 hub-weeks.
  2. Open itemsA status and amount for each package: A at 244,800 yuan; B at 367,200 in four equal evidence units; C at 204,000 in two equal validation units.
  3. ConstraintSettlement rules v1.1: “uploaded” is not “reviewed”. Pending, estimated and awaiting-approval items keep their status, and the role cannot approve payment.
  4. Data boundaryPersonal learning data serves evaluation, coaching and acceptance only, never discipline, reassignment, pay cuts or dismissal.

Kept in the Metacase as Goal · Evidence · Constraints
what was known at the decision point

17source files

02Prior practice & strategy

Measures by contract; opportunity apart from ability.

Two contract thresholds per package, plus amount rules and the pre-review view. How the agent chose its measures is kept only in its step record.

  1. Thresholds for ADelivery verified; effective learning time of at least 85% of the planned 1,620 minutes, for at least 90% of supervisors.
  2. Thresholds for B and CB: delayed-test coverage of 90%, and 75% behaviour attainment among those with two verified incidents. C: positive net gains on two core metrics, a usable close-out index, no high-risk item closed.
  3. Candidate treatmentsThe provider claims full acceptance; the 29 May pre-review leans to conditional A, and partial B and C by evidence unit. After recomputing each threshold, the agent adopts the latter and splits amounts by unit.
  4. Choice of measureThree measures of behaviour attainment, all computed: mean of all reviewed items, 73 of 73; mean of the last round, 67; all five last-round ratings at 3 or more, 53. It adopts the second.
  5. Comparison designThe deferred fourth hub is the same-period reference: a trained hub's change minus the reference hub's is its net change, which removes shocks common to all hubs.

Kept in the Metacase as Practice · Options · Trade-offs
the options weighed, and the reasons on record

6contract thresholds

03Actions & execution

Each source recomputed, then two files built.

gemini-3.8-flash runs in the DeepSeek Harness for about 19 minutes: 86 model requests and 85 tool calls, all logged.

  1. Evidence gatheringIt reads the four documents, then every sheet of the workbooks and detail files, checking each status field; it starts building files about seven minutes in.
  2. Recomputation82 of 83 reach the 1,377-minute threshold, one at 1,170; three people's 360 pending minutes are held out, leaving 1,440 each. Mean test scores: 59.06, 94.07, 90.27.
  3. StratificationWith two verified incidents as the bar: 73 can be evaluated, 8 lack opportunity, and 2 are restricted by leave or transfer.
  4. FailureThe memo script halts on taking the first row of an empty table. The agent finds the cause, rewrites it and regenerates the file.
  5. Read-backIt reads both files back: sheet count, dimensions, key figures. Ten sheets, but only three cells hold formulas; the rest are typed values.

Kept in the Metacase as Steps · Tools · Intermediate work
every step, in order

85tool calls

04Outcomes & verification

Out-of-loop verification: 34 criteria, judged singly.

Harbor adds the rubric to the same container only after the agent stops. RewardKit runs one session per criterion on another model; nothing flows back.

  1. Acceptance decisionsEvery package status and amount is judged correct: A conditional, 220,320 yuan for approval and 24,480 retained; B three of four units, 275,400; C one of two, 102,000.
  2. A measure at odds with the rubricBehaviour attainment is reported as 67 of 73; the rubric's measure, a mean of 3.0 over all reviewed items, gives 73. Both clear the 75% threshold, so B stands.
  3. Not shown directlyEach trained hub's net change is missing; only a pooled figure appears, 4.33 points on the response-SLA rate. By hub: 6.5, 4.5 and 2.0. One subtraction was all it took.
  4. Attribution and traceabilityThe memo says the data “irrefutably prove” the training's value, rather than a descriptive association; with no source index, conclusions cannot be traced to source files.
  5. Score structureOf 28 positive criteria, 19 met; none of six penalties triggered. 79 of 106: one pilot run's score, not accuracy or a ranking.

Kept in the Metacase as Deliverable · Result · Checks · Limits
the result, and the limits of the check

0.745rubric score, one pilot run

05Reflection & reuse

Reviewing every disposition, and the limits of reuse.

The record ties each loss to what the agent saw and did, so a later attempt starts from a specific failure rather than a score.

  1. What heldAll 13 criteria on scope, attendance, tests, opportunity strata, observation records and kept status are met; every package status and amount is right, and no penalty applies.
  2. Supervisor scope and shift roster
    What it shows
    84 names, one a warehouse coordinator the list marks as an error.
    Why it matters
    Scope comes before every rate; one extra person dilutes them all.
    This run
    Excluded; 83 in scope.
  3. Attendance, make-up and exemption events
    What it shows
    Three people each have 360 minutes without a sign-out, pending verification.
    Why it matters
    Pending time is not confirmed time; restating it puts unverified minutes into the basis for payment.
    This run
    Held out; all three still clear 1,377 minutes. With sign-outs unverified, A is conditional.
  4. Three-point situational test responses
    What it shows
    Two people each made a void practice attempt the day before the delayed test; the provider calls it closed, but on 28 May it reverted to pending a spot check.
    Why it matters
    Coverage and scores count formal attempts only; practice changes both.
    This run
    Practice excluded; 79 valid delayed tests.
  5. Incident opportunity log
    What it shows
    Ten people have one verified incident each; two are restricted by leave or transfer.
    Why it matters
    No opportunity is not no ability; the data rules forbid scoring it as zero or failing.
    This run
    73 evaluable, 8 lacking opportunity, 2 restricted; the matching penalty not triggered.
  6. On-site behaviour coding
    What it shows
    18 of 1,145 observations are disputed, all on “identification”, for six supervisors.
    Why it matters
    Until resolved, the evidence unit cannot pass; the amount rules say the same.
    This run
    Kept pending; B's behaviour-transfer unit deferred.
  7. Weekly hub operating metrics
    What they show
    All four hubs' figures for the week of 18 May are estimates.
    Why it matters
    An estimate is not a measurement; keeping or dropping the week changes the net change.
    This run
    Kept as estimates, with a recomputation excluding that week.
  8. Equipment, weather and cargo disruptions
    What they show
    A chiller repair's end date differs by a day; the pre-review notes that one hub's gain overlaps the repair.
    Why it matters
    A day decides whether a gain came before or after the repair, and so its attribution; the rubric asks for a ±1-day recheck and separate heat, cargo and terminal weeks.
    This run
    Placed at SYN-HUB-01 where the rubric names SYN-HUB-03; none of the four treatments done.
  9. What failedAnalysis and argument lost all 6 points, hub-level net changes all 9: the figures were right but not laid out for a decision-maker, and the limits went unstated. No self-check would catch that.
  10. Package A · training preparation and delivery10 / 10
    Provider's claim
    244,800 yuan, self-rated as accepted.
    Agent's disposition
    Conditional acceptance; 220,320 yuan for approval, 24,480 retained.
    What decided it
    Sessions held, 82 of 83 meet the hours; three sign-outs pending, so 90% and 10% by the amount rules.
  11. Package B · evidence of retention and transfer17 / 17
    Provider's claim
    367,200 yuan, self-rated as accepted.
    Agent's disposition
    Three of four units pass; 275,400 for approval, 91,800 deferred.
    What decided it
    Delayed-test coverage of 79 of 83; the behaviour-transfer unit waits on the disputed observations.
  12. Package C · operational validation and close-out13 / 13
    Provider's claim
    204,000 yuan, self-rated as accepted.
    Agent's disposition
    One of two units passes; 102,000 for approval, 102,000 deferred.
    What decided it
    Operational validation passes; the close-out unit does not and is deferred.
  13. Sample, learning and opportunity39 / 39
    Starting point
    84 names, 755 attendance events, 4,940 responses, 341 opportunity records.
    Agent's disposition
    83 in scope, 82 of 83 meet the hours, means 59.06, 94.07 and 90.27, strata 73, 8 and 2.
    What decided it
    All 13 related criteria met; pending and void records keep their status.
  14. Behaviour attainment0 / 3
    Starting point
    A 75% threshold, on a mean of 3.0 over reviewed items.
    Agent's disposition
    Last round only: 67 of 73, with six marked as not attaining.
    What decided it
    The rubric's measure gives 73 of 73; the agent computed all three and chose another.
  15. Operational comparison0 / 9
    Starting point
    Four metrics for four hubs, at baseline and in observation.
    Agent's disposition
    Each hub's own change, and one pooled net change for the three.
    What decided it
    The rubric wants each hub's net change shown; pooling hides 6.5, 4.5 and 2.0 points.
  16. Attribution and responsibility0 / 6
    Starting point
    The pre-review asks for no single attribution.
    Agent's disposition
    States it for one hub only.
    What decided it
    Coaching and behaviour not limited to association, responsibilities not split; the four disruption treatments missing.
  17. Traceability and closure0 / 9
    Starting point
    Conclusions must trace back to people, incidents, hub-weeks and procurement rules.
    Agent's disposition
    Ten sheets, no source index; a closure plan of three actions.
    What decided it
    No source index or five source chains; no controlled drill or overdue rule in the closure plan.
  18. Conditions and signatureApplies where evidence has mixed status, decisions follow contract rules unit by unit, and data use is bounded. Signature: evidence status, choice of measure, bounded attribution, recommendation without approval.

Kept in the Metacase as What held · What failed · When it applies
what the next attempt can borrow or change

Watch the run in full

HR-T3-114 · gemini-3.8-flash · DeepSeek Harness 0.2.2-rc.1 · 5 min 1 sec, with English captions
The expert rubric, in full

Reward = max(0, points earned − penalties) ÷ 106. This task has no red lines.

The judge sees three things: the two files the agent handed in (the only thing scored), the task's inputs to check every claim against, and a reference solution that calibrates structure and scale but is never an answer key.

Key: R numbers are rubric criteria; A, B and C are the three work packages; SYN-HUB-01 to SYN-HUB-03 are the three trained hubs and SYN-HUB-04 the deferred reference hub.

28 checks to earn · 106 points · all pass or fail
CheckWhat it checksReadsPointsThis run
① Scope and sample · 9 points · this run 9
R384 names on the authorisation list, one excluded for the wrong role, 83 in scopeWorkbook3Met · +3
R4A supervisor on approved leave from week 3 of observation: restricted, outside the behaviour denominator, learning records keptWorkbook3Met · +3
R33A supervisor transferred before the first observation: likewise restricted, learning records keptWorkbook3Met · +3
② Attendance and learning time · 12 points · this run 12
R5Effective learning time recomputed from verified attendance: a threshold of 1,377 minutes, 82 of 83 meeting it, the one below at 1,170Workbook3Met · +3
R6First supervisor with a missing sign-out: 360 minutes shown as pending and not countedWorkbook3Met · +3
R34Second such supervisor, same requirementWorkbook3Met · +3
R35Third such supervisor, same requirementWorkbook3Met · +3
③ Three-point tests · 9 points · this run 9
R783 valid tests before and after training, 79 delayed, with the four missing supervisors namedWorkbook3Met · +3
R8Formal, valid attempts only, stating that two void practice attempts are outside scores and coverageBoth3Met · +3
R9Mean scores of about 59.1, 94.1 and 90.3, a gain of about 35.0 from before to after, each to ±0.1Either3Met · +3
④ Opportunity and behaviour · 9 points · this run 6
R10Strata by two or more verified incidents: 73 evaluable, 8 lacking opportunity, 2 outside observation; no zero or failing mark for lack of opportunityWorkbook3Met · +3
R11Behaviour attainment of 73 of 73 among the evaluable, on a mean of at least 3.0 across reviewed itemsWorkbook3Not met · 0
R121,145 observations, 1,127 reviewed and 18 pending, with the six supervisors concerned namedWorkbook3Met · +3
⑤ Operational comparison and attribution · 15 points · this run 0
R14First trained hub's change net of the deferred hub, shown directly: response-SLA rate +6.5 points, isolation completeness +8.2, repeat incidents −1.1, loss per 10,000 order lines −260 yuanWorkbook3Not met · 0
R15Second trained hub: +4.5, +5.7 and −0.7 points, −170 yuanWorkbook3Not met · 0
R16Third trained hub: +2.0, +2.7 and −0.3 points, −70 yuanWorkbook3Not met · 0
R17Net improvement not all attributed to training: the chiller repair's end date kept pending and rechecked at ±1 day, extreme-heat weeks separated or excluded, sensitive cargo stratified, the terminal-upgrade week separatedBoth3Not met · 0
R13Coaching and behaviour limited to a descriptive association; the provider's remedial duties listed apart from the company's duties to provide opportunity, scheduling, coaching and reviewMemo3Not met · 0
⑥ Package status and amounts · 40 points · this run 40
R20Package A accepted conditionally, because sessions were delivered but three sign-outs are pending; not stated as fully acceptedBoth7Met · +7
R21Package B accepted in part, three of four equal evidence unitsBoth10Met · +10
R22Package C accepted in part, one of two equal validation unitsBoth10Met · +10
R23Package A: 220,320 yuan for approval, 24,480 retained, 0 deferred, summing to the contract amountBoth3Met · +3
R24Package B: 275,400 yuan for approval, 91,800 deferred, 0 retainedBoth7Met · +7
R25Package C: 102,000 yuan for approval, 102,000 deferred, 0 retainedBoth3Met · +3
⑦ General: structure, status, traceability, closure · 12 points · this run 3
R1The workbook covers 11 functions: definitions, scope check, learning path, opportunity strata, behaviour evidence, operational comparison, disruption limits, stage-gate decisions, payment split, closure tracking and a source indexWorkbook3Not met · 0
R18Pending sign-outs, void practice attempts, pending incident links, observer disagreements, the repair-time gap and the provider's “uploaded” claims are not restated as verified, valid or closedWorkbook3Met · +3
R19The source index sets up at least five chains that can be rechecked, each with its named keys and the 31 May cutoffWorkbook3Not met · 0
R26The closure plan lists at least four actions, each with an owner, a deadline, closing evidence and what happens when overdueBoth3Not met · 0
6 penalties · up to −44 · only deducted, never added
CheckWhat gets penalisedPointsThis run
R27Deliverables containing a real person's name, contact details, ID number, address or usable credentials−7Not triggered
R28Deliverables embedding unauthorised commercial reports, third-party images, brand material or protected attachments−3Not triggered
R29A package's acceptance status contradicting itself in a way that changes the action taken, with no version or correction note−7Not triggered
R30Stating a recommendation for approval as payment made, or as final approval by the authorised body−10Not triggered
R31Recommending that personal test, observation or coaching results be used for discipline, reassignment, pay cuts or dismissal−10Not triggered
R32Judging supervisors who lacked opportunity as failing on behaviour, not improved or not transferring training−7Not triggered
One threshold, three measures: R11, behaviour attainment
Contract threshold: at least 75% of the 73 supervisors with two or more verified incidents.
MeasureAttainingShareAgainst the thresholdThis run
Mean of at least 3.0 across all reviewed behaviour items (the rubric's measure)73 of 73100%AboveComputed, not adopted
Mean of at least 3.0 in the last round67 of 7391.8%AboveAdopted
All five ratings at least 3 in the last round53 of 7372.6%BelowComputed, not adopted