SUBMIT_PIPELINE, the grader engine runs a rigorous row-by-row comparison between the submitted DataFrame and the hidden ground truth, producing a final score between 0.01 and 0.99. This page documents every detail of that comparison — the normalisation applied to both sides, the exact formula used to compute the score, and the penalty for submitting extra rows.
Section 1: Two-Phase Scoring
Understanding which score matters for benchmarking is essential.Step rewards (informational)
After every/step call the server returns result.reward. These are intermediate rewards from the dense heuristic shaper, designed to prevent sparse-reward pathologies during LLM training. They range from approximately:
Step rewards accumulate in
log_end’s rewards field but are not your final grade.
Terminal grade (the real score)
Whendone=True, the result.reward value returned by the last step() call is the grader’s output — a floating-point number in [0.01, 0.99]. This is what the leaderboard records. The grader is also accessible directly:
The
score in the grader response is clamped to [0.01, 0.99]. A perfect pipeline always yields 0.99, not 1.0 — this is by design in the grader formula (min(0.99, ...)). The inference.py script applies the same clamp (min(0.99, max(0.01, score))), so the maximum score it can report is also 0.99.Section 2: How the Grader Works — Step by Step
The full grading logic lives inserver/graders.py in evaluate_dataframes(truth_df, final_df, join_key="customer_id"). Here is the algorithm, step by step.
Step 1 — Deduplicate the submitted DataFrame
customer_id. This prevents a Cartesian explosion during the merge if your agent failed to dedup. It does not remove the bot-penalty — len(final_df) (before this dedup) is used to compute the excess-row penalty later.
Step 2 — Left join on customer_id
customer_id exists in truth_df but not in your submission, that row gets NaN for every agent column — it will not match, reducing your score.
Step 3 — Per-column normalisation and exact match
For every non-key column intruth_df:
fillna("") → astype(str) → .strip() → .lower(). The comparison is then a strict Python string equality check.
Step 4 — Row correctness
Step 5 — Base score
Step 6 — Excess-row penalty
0.15 off your score. This penalises agents that submit without deduplicating or without filtering bot rows.
Step 7 — Clamp to valid range
[0.01, 0.99]. You cannot score 0.0 or 1.0.
Worked example
Suppose the ground truth for Task 1 has 40 rows and your agent submits a DataFrame with 48 rows (you forgot to dedup):
The excess-row penalty is severe — even a mostly-correct submission collapses to
0.01 when there are 8 extra rows.
Section 3: What “Exact Match” Means
The grader does not do fuzzy matching. After normalisation (strip + lowercase), it requires character-for-character equality. The truth dataset is pre-formatted: dates are already ISO-8601 and phones are already E.164 (or digits-only, depending on the task). Your submission must match that exact format.Normalisation examples
Date format requirements
The grader expects ISO-8601 date strings:YYYY-MM-DD. Apply STANDARDIZE_COLUMN with TO_DATETIME_ISO to every date column before submitting:
TO_DATETIME_ISO handles: MM/DD/YYYY, DD-MM-YYYY, YYYY.MM.DD, Unix timestamps.
Phone number format requirements
The grader expects phone numbers either as pure digits (4155550123) or in E.164 format (+14155550123), depending on the task’s ground truth. Use EXTRACT_NUMBERS to strip all non-digit characters, or PHONE_E164 to produce a full international format:
obs.data_quality_report to understand what the input looks like before choosing a strategy.
Section 4: Bot Penalty Explained
The CRM datasets contain synthetically injected bot rows — records withcustomer_id = '???' or email addresses containing %bot%. These rows are never in the ground truth. If you submit them, they inflate len(final_df) above num_truth_rows and trigger the excess-row penalty.
Why the penalty is steep
A penalty coefficient of0.15 per extra row is intentionally aggressive. Consider a task with 40 truth rows and 10 bot rows:
Recommended bot filters
legacy_id field uses the placeholder '???' instead of customer_id:
Section 5: Ground Truth per Task
Each task grades against a different truth snapshot and a different set of columns.
The truth snapshot is captured at episode
reset() time — it is a frozen copy of what the clean data should look like. The grader always compares against this frozen snapshot, so re-running the same episode ID will always produce the same score for the same submission.
The grader function called depends on task_id:
t1 both the truth key and the final source are web_forms — you submit the cleaned web_forms table directly. For t2 and t3 the expected final_source is always merged_output.
Section 6: Calling the Grader Endpoint Manually
You can call the grader at any point during an episode to check a partial score without ending the episode — useful during development:rows field tells you how many rows are currently in your submitted source. Compare it against the expected row count in your ground-truth snapshot to immediately spot excess-row situations.
Section 7: Achieving a Perfect Score
The deterministic rule-based baseline ininference.py achieves a terminal grader score of 0.99 — the maximum possible — on all three tasks. Here is the exact sequence that produces that result:
Task 1 — perfect sequence
Task 1 — perfect sequence
Task 2 — perfect sequence
Task 2 — perfect sequence
Task 3 — perfect sequence
Task 3 — perfect sequence
Why the grader caps at 0.99
The formulamin(0.99, score - penalty) hard-caps the maximum possible score at 0.99. This is a deliberate design choice: 0.99 is the gold-standard ceiling, and 0.01 is the floor — no submission can score exactly 0.0 or 1.0. For leaderboard purposes, 0.99 represents a perfect pipeline.