Skip to main content
The CRM Pipeline environment uses a carefully designed dense heuristic reward system to guide autonomous agents through the multi-step data cleaning process. Understanding how rewards are calculated — and crucially, what they do and do not measure — is essential for building an agent that optimizes for genuine data quality rather than gaming the reward signal.

Why Dense Rewards Matter for Data Engineering Agents

In a sparse reward setting, the agent receives a signal only at the very end of the episode — when it submits the pipeline and the grader compares its output against the hidden truth. For a typical Task 1 episode with 40+ rows and 5 columns, the space of possible cleaning strategies is enormous. An agent operating under sparse rewards would need to randomly explore this space for hundreds of episodes before receiving any useful signal. This is computationally intractable for LLM-based agents operating with API budget constraints. Dense rewards solve this by providing immediate feedback after every action. When your agent correctly identifies that an email column contains uppercase values and applies LOWERCASE_STRIP, it receives a small positive reward proportional to how many values were actually changed. When it makes a mistake (referencing a nonexistent column, issuing a malformed SQL query), it receives a negative reward immediately. This creates a continuous learning signal that guides the agent toward productive sequences of actions. The key design insight is that the dense rewards are heuristic proxies — they measure likely improvement without consulting the hidden ground truth. They are designed to be aligned with true quality improvement in expectation, but they are not the true score.
The step rewards you accumulate during an episode are completely separate from the grader score (0.01–0.99) produced at SUBMIT_PIPELINE time. A high cumulative step reward does not guarantee a high grader score. A low cumulative step reward does not preclude a high grader score. The grader is the true and final measure of pipeline quality.

Complete Step Reward Schedule


Progressive Rewards and Systematic Cleaning

The reward structure is intentionally designed to encourage a systematic cleaning sequence rather than random action selection. Notice the progression:
  1. Profiling earns +0.02 — rewarding the agent for gathering information before acting. An agent that blindly standardizes without profiling misses this reward.
  2. Standardization scales with impact — the reward for STANDARDIZE_COLUMN is computed as min(0.05, 0.01 + improvement_ratio × 0.1), where improvement_ratio is the fraction of rows that actually changed. A column that was already clean earns only +0.01; a heavily dirty column earns up to +0.05. This encourages the agent to standardize the most problematic columns first.
  3. Missing value handling rewards proportionally — the reward for HANDLE_MISSING scales with the null density of the column. Fixing a column with 11 nulls out of 40 rows earns more than fixing a column with 1 null. This encourages the agent to prioritize high-null columns.
  4. Deduplication reward is proportional to yield — (removed / start_len) × 0.2 means a dataset with many duplicates yields high reward when cleaned. If a source has few duplicates, deduplication earns little reward, implicitly signaling it may not be the priority.
  5. SQL execution earns +0.1 — the highest individual step reward (excluding the terminal submission reward), reflecting the high value of complex multi-source operations that SQL enables.

The Premature Submission Penalty

Attempting SUBMIT_PIPELINE when step_count < MIN_STEPS_BEFORE_SUBMIT (where MIN_STEPS_BEFORE_SUBMIT = 3) results in a -0.15 penalty and done=False. The episode continues. This is the largest per-step penalty in the system, designed to strongly discourage agents from submitting immediately after reset. The -0.15 penalty costs more than the combined reward of roughly 7 successful PROFILE_SOURCE calls, creating a significant incentive to perform meaningful cleaning work.
The step count is incremented before the action handler runs. This means that even the premature submission attempt itself counts toward the step total. If your agent submits prematurely on step 2 (earning -0.15), it now has step_count = 2 — one more valid action is sufficient to satisfy the minimum before re-submitting.

Terminal Reward: Schema Match Heuristic

When SUBMIT_PIPELINE succeeds (i.e., step_count >= 3), the environment immediately computes a fast heuristic reward based solely on column presence:
For Task 1 with 5 target columns and a final DataFrame that contains all 5:
For a final DataFrame that is missing the phone column:
This is the step reward embedded in the terminal observation. It does not reflect the grader score.

The Grader: True Terminal Scoring

The grader in server/graders.py computes the real evaluation score. It is triggered server-side when SUBMIT_PIPELINE succeeds, and it compares your final DataFrame against the hidden ground truth using row-level, column-level exact matching.

Grader Algorithm (Step by Step)

Step 1: Deduplication guard The grader enforces uniqueness on the join key before evaluating, to prevent a Cartesian explosion if the agent failed to deduplicate:
This means the grader silently deduplicates your submission before scoring. However, duplicate rows are not “free” — they are penalized by the row count penalty in Step 5 (because len(final_df) is measured before this deduplication). Step 2: Left join truth onto agent output
A left join from truth to agent output means:
  • Every row in truth_df is represented in the merged result
  • If a customer_id exists in truth but not in your output, that row has all _agent columns as NaN
  • If a customer_id exists in your output but not in truth, it does not appear in the merged result (but counts toward the row count penalty)
Step 3: Column-level string normalization and comparison For each non-key column in truth_df:
The grader normalizes both sides with .fillna("").astype(str).str.strip().str.lower() before comparing. This means:
  • NaN values in both truth and agent become "" and match each other
  • Trailing whitespace is stripped on both sides
  • Case differences are ignored at the comparison stage
However, since the hidden truth already stores values in the exact target format (ISO 8601 dates, E.164 phones, lowercase emails), an agent that fails to standardize will produce mismatches here. Step 4: Row-level correctness
A row is counted as correct only if every single non-key column matches. A row where 4 out of 5 columns are correct is scored as 0 for that row. Step 5: Row count penalty for leftover junk
If your final DataFrame has more rows than the truth, you are penalized 0.15 per extra row. This is the mechanism that punishes agents for failing to remove bot rows and duplicate entries. For example: if truth has 40 rows but your final DataFrame has 48 rows (8 bot rows not removed), your penalty is 8 × 0.15 = 1.2. This can devastate an otherwise good score. Step 6: Final score clamping
The grader score is always in the range [0.01, 0.99]. A perfect clean pipeline scores 0.99 (never 1.0 by design). An empty or severely broken submission scores 0.01.

Complete Grader Formula


Reward Accumulation: A Sample Episode

The following table traces the reward accumulation for a well-executed Task 1 episode, illustrating how step rewards build up alongside the terminal grader score. In this scenario, the step reward (+0.61) and the grader score (0.94) are high and aligned — because the systematic cleaning approach that earns step rewards genuinely improves data quality. However, the two numbers are computed completely independently.

Why Extra Rows Are Penalized So Heavily

The 0.15 penalty per extra row is deliberately steep. Consider a scenario where an agent submits a final DataFrame with all 40 truth rows correctly cleaned plus 8 bot rows that were not removed:
A perfect cleaning job on the real rows but a failure to remove bots produces the minimum possible score. This reflects the real-world consequence: a CRM database polluted with bot and duplicate records is not a clean database, regardless of how well the legitimate rows are formatted. The penalty structure ensures that bot removal is not optional — it is a hard requirement for a high score.
The most reliable way to remove bot rows is via EXECUTE_SQL with an explicit filter. Bot rows in this environment have predictable signatures:
  • customer_id LIKE 'BOT%' — all bot IDs begin with “BOT”
  • phone = '+00000000000' — all-zero phone number
  • signup_date = '1970-01-01' — Unix epoch date
A single SQL query can filter on any combination of these conditions before submitting.

Step Rewards vs. Grader Score: Key Differences

In research settings, cumulative step reward is useful as a proxy metric during training — it provides a signal after every episode without requiring ground truth access. The grader score is the authoritative number for benchmarking and comparison across agent implementations.