Skip to main content
The CRM Data Pipeline Environment uses a two-phase scoring system. During an episode, the server emits a dense heuristic reward after every step to guide the agent toward correct behaviour. When the agent calls SUBMIT_PIPELINE, the grader engine runs a rigorous row-by-row comparison between the submitted DataFrame and the hidden ground truth, producing a final score between 0.01 and 0.99. This page documents every detail of that comparison — the normalisation applied to both sides, the exact formula used to compute the score, and the penalty for submitting extra rows.

Section 1: Two-Phase Scoring

Understanding which score matters for benchmarking is essential.

Step rewards (informational)

After every /step call the server returns result.reward. These are intermediate rewards from the dense heuristic shaper, designed to prevent sparse-reward pathologies during LLM training. They range from approximately: Step rewards accumulate in log_end’s rewards field but are not your final grade.

Terminal grade (the real score)

When done=True, the result.reward value returned by the last step() call is the grader’s output — a floating-point number in [0.01, 0.99]. This is what the leaderboard records. The grader is also accessible directly:
The score in the grader response is clamped to [0.01, 0.99]. A perfect pipeline always yields 0.99, not 1.0 — this is by design in the grader formula (min(0.99, ...)). The inference.py script applies the same clamp (min(0.99, max(0.01, score))), so the maximum score it can report is also 0.99.

Section 2: How the Grader Works — Step by Step

The full grading logic lives in server/graders.py in evaluate_dataframes(truth_df, final_df, join_key="customer_id"). Here is the algorithm, step by step.

Step 1 — Deduplicate the submitted DataFrame

Before any comparison, the grader silently deduplicates your submission on customer_id. This prevents a Cartesian explosion during the merge if your agent failed to dedup. It does not remove the bot-penalty — len(final_df) (before this dedup) is used to compute the excess-row penalty later.

Step 2 — Left join on customer_id

A left join is used, keeping all rows from the ground truth. If a customer_id exists in truth_df but not in your submission, that row gets NaN for every agent column — it will not match, reducing your score.

Step 3 — Per-column normalisation and exact match

For every non-key column in truth_df:
Both sides are normalised identically: fillna("") → astype(str) → .strip() → .lower(). The comparison is then a strict Python string equality check.

Step 4 — Row correctness

A row is marked correct only if every single column matches. One mismatched column in a row makes the entire row incorrect.

Step 5 — Base score

The base score is the fraction of ground-truth rows that were perfectly reproduced.

Step 6 — Excess-row penalty

If your submitted DataFrame contains more rows than the ground truth, each extra row costs 0.15 off your score. This penalises agents that submit without deduplicating or without filtering bot rows.

Step 7 — Clamp to valid range

The final score is clamped to [0.01, 0.99]. You cannot score 0.0 or 1.0.

Worked example

Suppose the ground truth for Task 1 has 40 rows and your agent submits a DataFrame with 48 rows (you forgot to dedup): The excess-row penalty is severe — even a mostly-correct submission collapses to 0.01 when there are 8 extra rows.

Section 3: What “Exact Match” Means

The grader does not do fuzzy matching. After normalisation (strip + lowercase), it requires character-for-character equality. The truth dataset is pre-formatted: dates are already ISO-8601 and phones are already E.164 (or digits-only, depending on the task). Your submission must match that exact format.

Normalisation examples

NaN does not normalise to "". After astype(str), Python’s float("nan") becomes the string "nan", which does not equal "". Always use HANDLE_MISSING with DROP_ROW or FILL_VALUE to eliminate nulls before submitting — do not rely on the grader’s fillna("") to rescue unfilled nulls in your data.

Date format requirements

The grader expects ISO-8601 date strings: YYYY-MM-DD. Apply STANDARDIZE_COLUMN with TO_DATETIME_ISO to every date column before submitting:
Common input formats that TO_DATETIME_ISO handles: MM/DD/YYYY, DD-MM-YYYY, YYYY.MM.DD, Unix timestamps.

Phone number format requirements

The grader expects phone numbers either as pure digits (4155550123) or in E.164 format (+14155550123), depending on the task’s ground truth. Use EXTRACT_NUMBERS to strip all non-digit characters, or PHONE_E164 to produce a full international format:
Profile the source first and check obs.data_quality_report to understand what the input looks like before choosing a strategy.

Section 4: Bot Penalty Explained

The CRM datasets contain synthetically injected bot rows — records with customer_id = '???' or email addresses containing %bot%. These rows are never in the ground truth. If you submit them, they inflate len(final_df) above num_truth_rows and trigger the excess-row penalty.

Why the penalty is steep

A penalty coefficient of 0.15 per extra row is intentionally aggressive. Consider a task with 40 truth rows and 10 bot rows:
A perfect clean pipeline fails completely if bots are not removed. Always filter them.
For Task 3 with the legacy source, the legacy_id field uses the placeholder '???' instead of customer_id:

Section 5: Ground Truth per Task

Each task grades against a different truth snapshot and a different set of columns. The truth snapshot is captured at episode reset() time — it is a frozen copy of what the clean data should look like. The grader always compares against this frozen snapshot, so re-running the same episode ID will always produce the same score for the same submission. The grader function called depends on task_id:
Note that for t1 both the truth key and the final source are web_forms — you submit the cleaned web_forms table directly. For t2 and t3 the expected final_source is always merged_output.

Section 6: Calling the Grader Endpoint Manually

You can call the grader at any point during an episode to check a partial score without ending the episode — useful during development:
The rows field tells you how many rows are currently in your submitted source. Compare it against the expected row count in your ground-truth snapshot to immediately spot excess-row situations.
Call the grader mid-episode after each major transformation (dedup, SQL filter, merge) to track score progression before SUBMIT_PIPELINE. This is the fastest debugging loop — tweak a SQL query, call the grader, and see the delta immediately.

Section 7: Achieving a Perfect Score

The deterministic rule-based baseline in inference.py achieves a terminal grader score of 0.99 — the maximum possible — on all three tasks. Here is the exact sequence that produces that result:
All 8 steps complete in well under the 15-step budget.
13 steps — still within the 15-step budget.
Task 3 uses a single complex SQL to merge all three sources rather than separate UNION steps, staying within the 15-step budget with room to spare.

Why the grader caps at 0.99

The formula min(0.99, score - penalty) hard-caps the maximum possible score at 0.99. This is a deliberate design choice: 0.99 is the gold-standard ceiling, and 0.01 is the floor — no submission can score exactly 0.0 or 1.0. For leaderboard purposes, 0.99 represents a perfect pipeline.