> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Understanding Grading and Scoring in the CRM Pipeline

> Learn how the CRM Pipeline grader evaluates your agent's output against the hidden ground truth, including the row-match formula and excess-row penalty.

The CRM Data Pipeline Environment uses a two-phase scoring system. During an episode, the server emits a dense heuristic reward after every step to guide the agent toward correct behaviour. When the agent calls `SUBMIT_PIPELINE`, the grader engine runs a rigorous row-by-row comparison between the submitted DataFrame and the hidden ground truth, producing a final score between `0.01` and `0.99`. This page documents every detail of that comparison — the normalisation applied to both sides, the exact formula used to compute the score, and the penalty for submitting extra rows.

## Section 1: Two-Phase Scoring

Understanding which score matters for benchmarking is essential.

### Step rewards (informational)

After every `/step` call the server returns `result.reward`. These are **intermediate rewards** from the dense heuristic shaper, designed to prevent sparse-reward pathologies during LLM training. They range from approximately:

| Event | Reward |
| - | - |
| Successful `STANDARDIZE_COLUMN` | `+0.03` to `+0.05` |
| Successful `DEDUPLICATE` | `+0.05` |
| Successful `EXECUTE_SQL` | `+0.03` to `+0.04` |
| Invalid action (error in feedback) | `-0.05` to `-0.1` |
| Premature `SUBMIT_PIPELINE` | `-0.5` |

Step rewards accumulate in `log_end`'s `rewards` field but are **not your final grade**.

### Terminal grade (the real score)

When `done=True`, the `result.reward` value returned by the last `step()` call is the grader's output — a floating-point number in `[0.01, 0.99]`. This is what the leaderboard records. The grader is also accessible directly:

```bash theme={null}
curl -X POST "http://localhost:8080/grader/{episode_id}?final_source=web_forms&task_id=t1"
```

```json theme={null}
{
  "episode_id": "3f8a1c22-b4e2-4d07-9a11-0f7e2c3d1a00",
  "task_id": "t1",
  "score": 0.95,
  "rows": 40
}
```

> [!NOTE]
>
> The `score` in the grader response is clamped to `[0.01, 0.99]`. A perfect pipeline always yields `0.99`, not `1.0` — this is by design in the grader formula (`min(0.99, ...)`). The `inference.py` script applies the same clamp (`min(0.99, max(0.01, score))`), so the maximum score it can report is also `0.99`.

## Section 2: How the Grader Works — Step by Step

The full grading logic lives in `server/graders.py` in `evaluate_dataframes(truth_df, final_df, join_key="customer_id")`. Here is the algorithm, step by step.

### Step 1 — Deduplicate the submitted DataFrame

```python theme={null}
eval_df = final_df.drop_duplicates(subset=["customer_id"])
```

Before any comparison, the grader silently deduplicates your submission on `customer_id`. This prevents a Cartesian explosion during the merge if your agent failed to dedup. **It does not remove the bot-penalty** — `len(final_df)` (before this dedup) is used to compute the excess-row penalty later.

### Step 2 — Left join on customer\_id

```python theme={null}
merged = pd.merge(
    truth_df,
    eval_df,
    on="customer_id",
    suffixes=("_truth", "_agent"),
    how="left",
)
```

A **left join** is used, keeping all rows from the ground truth. If a `customer_id` exists in `truth_df` but not in your submission, that row gets `NaN` for every agent column — it will not match, reducing your score.

### Step 3 — Per-column normalisation and exact match

For every non-key column in `truth_df`:

```python theme={null}
s_truth = merged[col_truth].fillna("").astype(str).str.strip().str.lower()
s_agent = merged[col_agent].fillna("").astype(str).str.strip().str.lower()
merged[f"{col}_match"] = (s_truth == s_agent)
```

Both sides are normalised identically: `fillna("")` → `astype(str)` → `.strip()` → `.lower()`. The comparison is then a strict Python string equality check.

### Step 4 — Row correctness

```python theme={null}
match_cols = [f"{col}_match" for col in truth_df.columns if col != "customer_id"]
merged["row_correct"] = merged[match_cols].all(axis=1)
```

A row is marked **correct** only if **every single column** matches. One mismatched column in a row makes the entire row incorrect.

### Step 5 — Base score

```python theme={null}
correct_rows = merged["row_correct"].sum()
score = correct_rows / num_truth_rows
```

The base score is the fraction of ground-truth rows that were perfectly reproduced.

### Step 6 — Excess-row penalty

```python theme={null}
penalty = max(0, len(final_df) - num_truth_rows) * 0.15
```

If your submitted DataFrame contains **more rows than the ground truth**, each extra row costs `0.15` off your score. This penalises agents that submit without deduplicating or without filtering bot rows.

### Step 7 — Clamp to valid range

```python theme={null}
return min(0.99, max(0.01, score - penalty))
```

The final score is clamped to `[0.01, 0.99]`. You cannot score `0.0` or `1.0`.

### Worked example

Suppose the ground truth for Task 1 has **40 rows** and your agent submits a DataFrame with **48 rows** (you forgot to dedup):

| Metric | Value |
| - | - |
| Ground truth rows | 40 |
| Submitted rows | 48 |
| Correctly matched rows | 37 |
| Base score | 37 / 40 = **0.925** |
| Excess rows | 48 − 40 = 8 |
| Penalty | 8 × 0.15 = **1.20** |
| Raw final | 0.925 − 1.20 = **−0.275** |
| Clamped final | max(0.01, −0.275) = **0.01** |

The excess-row penalty is severe — even a mostly-correct submission collapses to `0.01` when there are 8 extra rows.

## Section 3: What "Exact Match" Means

The grader does **not** do fuzzy matching. After normalisation (strip + lowercase), it requires character-for-character equality. The truth dataset is pre-formatted: dates are already ISO-8601 and phones are already E.164 (or digits-only, depending on the task). Your submission must match that exact format.

### Normalisation examples

| Your value | Truth value | After normalisation | Match? |
| - | - | - | - |
| `"JOHN@GMAIL.COM"` | `"john@gmail.com"` | both → `"john@gmail.com"` | ✅ |
| `"  john  "` | `"john"` | both → `"john"` | ✅ |
| `None` | `""` | both → `""` | ✅ |
| `"NaN"` | `""` | `"nan"` vs `""` | ❌ |
| `"2023/05/15"` | `"2023-05-15"` | `"2023/05/15"` vs `"2023-05-15"` | ❌ |
| `"(415) 555-0123"` | `"4155550123"` | differ | ❌ |
| `"+14155550123"` | `"+14155550123"` | both identical | ✅ |

> [!WARNING]
>
> `NaN` does **not** normalise to `""`. After `astype(str)`, Python's `float("nan")` becomes the string `"nan"`, which does not equal `""`. Always use `HANDLE_MISSING` with `DROP_ROW` or `FILL_VALUE` to eliminate nulls before submitting — do not rely on the grader's `fillna("")` to rescue unfilled nulls in your data.

### Date format requirements

The grader expects ISO-8601 date strings: `YYYY-MM-DD`. Apply `STANDARDIZE_COLUMN` with `TO_DATETIME_ISO` to every date column before submitting:

```python theme={null}
CRMPipelineAction(
    action_type=PipelineActionType.STANDARDIZE_COLUMN,
    source="web_forms",
    column="signup_date",
    standardization_strategy=StandardizationStrategy.TO_DATETIME_ISO,
)
```

Common input formats that `TO_DATETIME_ISO` handles: `MM/DD/YYYY`, `DD-MM-YYYY`, `YYYY.MM.DD`, Unix timestamps.

### Phone number format requirements

The grader expects phone numbers either as pure digits (`4155550123`) or in E.164 format (`+14155550123`), depending on the task's ground truth. Use `EXTRACT_NUMBERS` to strip all non-digit characters, or `PHONE_E164` to produce a full international format:

```python theme={null}
# Strip to digits only — matches "4155550123" in truth
standardization_strategy=StandardizationStrategy.EXTRACT_NUMBERS

# International E.164 — matches "+14155550123" in truth
standardization_strategy=StandardizationStrategy.PHONE_E164
```

Profile the source first and check `obs.data_quality_report` to understand what the input looks like before choosing a strategy.

## Section 4: Bot Penalty Explained

The CRM datasets contain synthetically injected bot rows — records with `customer_id = '???'` or email addresses containing `%bot%`. These rows are never in the ground truth. If you submit them, they inflate `len(final_df)` above `num_truth_rows` and trigger the excess-row penalty.

### Why the penalty is steep

A penalty coefficient of `0.15` per extra row is intentionally aggressive. Consider a task with 40 truth rows and 10 bot rows:

```
score      = 40/40 = 1.00   (all real rows matched)
penalty    = 10 × 0.15 = 1.50
final      = max(0.01, 1.00 - 1.50) = 0.01
```

A perfect clean pipeline fails completely if bots are not removed. Always filter them.

### Recommended bot filters

```python theme={null}
# Via EXECUTE_SQL — most reliable approach
CRMPipelineAction(
    action_type=PipelineActionType.EXECUTE_SQL,
    query="""
        SELECT * FROM web_forms
        WHERE customer_id != '???'
          AND email NOT LIKE '%bot%'
          AND customer_id IS NOT NULL
    """,
    output_table="web_forms_clean",
)
```

For Task 3 with the legacy source, the `legacy_id` field uses the placeholder `'???'` instead of `customer_id`:

```python theme={null}
query="""
    SELECT REPLACE(legacy_id, 'OLD-', 'CUST_') AS customer_id,
           contact_email AS email,
           home_phone    AS phone
    FROM legacy_db
    WHERE legacy_id != '???'
      AND contact_email NOT LIKE '%bot%'
"""
```

## Section 5: Ground Truth per Task

Each task grades against a different truth snapshot and a different set of columns.

| Task | Truth key | Graded columns | Source names |
| - | - | - | - |
| `t1` | `web_forms` | `customer_id`, `name`, `email`, `signup_date`, `phone` | `web_forms` |
| `t2` | `merged_output` | `customer_id`, `name`, `email`, `signup_date`, `phone` | `web_forms`, `legacy_db` |
| `t3` | `merged_output` | `customer_id`, `email`, `phone` | `salesforce`, `web_leads`, `legacy_db` |

The truth snapshot is captured at episode `reset()` time — it is a frozen copy of what the clean data should look like. The grader always compares against this frozen snapshot, so re-running the same episode ID will always produce the same score for the same submission.

The grader function called depends on `task_id`:

```python theme={null}
# From server/graders.py
def grade_task_1(env): return _grade(env, truth_key="web_forms",      final_source="web_forms")
def grade_task_2(env): return _grade(env, truth_key="merged_output",  final_source="merged_output")
def grade_task_3(env): return _grade(env, truth_key="merged_output",  final_source="merged_output")
```

Note that for `t1` both the truth key and the final source are `web_forms` — you submit the cleaned `web_forms` table directly. For `t2` and `t3` the expected `final_source` is always `merged_output`.

## Section 6: Calling the Grader Endpoint Manually

You can call the grader at any point during an episode to check a partial score without ending the episode — useful during development:

```bash theme={null}
curl -X POST \
  "http://localhost:8080/grader/{episode_id}?final_source=web_forms&task_id=t1" \
  -H "Content-Type: application/json"
```

```json theme={null}
{
  "episode_id": "3f8a1c22-b4e2-4d07-9a11-0f7e2c3d1a00",
  "task_id": "t1",
  "score": 0.95,
  "rows": 40
}
```

The `rows` field tells you how many rows are currently in your submitted source. Compare it against the expected row count in your ground-truth snapshot to immediately spot excess-row situations.

> [!TIP]
>
> Call the grader mid-episode after each major transformation (dedup, SQL filter, merge) to track score progression before `SUBMIT_PIPELINE`. This is the fastest debugging loop — tweak a SQL query, call the grader, and see the delta immediately.

## Section 7: Achieving a Perfect Score

The deterministic rule-based baseline in `inference.py` achieves a terminal grader score of `0.99` — the maximum possible — on all three tasks. Here is the exact sequence that produces that result:

> [accordion] Task 1 — perfect sequence
>
> ```
> 1. PROFILE_SOURCE        source=web_forms
> 2. STANDARDIZE_COLUMN    source=web_forms  column=email        strategy=LOWERCASE_STRIP
> 3. STANDARDIZE_COLUMN    source=web_forms  column=name         strategy=LOWERCASE_STRIP
> 4. STANDARDIZE_COLUMN    source=web_forms  column=phone        strategy=EXTRACT_NUMBERS
> 5. STANDARDIZE_COLUMN    source=web_forms  column=signup_date  strategy=TO_DATETIME_ISO
> 6. DEDUPLICATE           source=web_forms  strategy=EXACT_EMAIL
> 7. EXECUTE_SQL           query="SELECT * FROM web_forms WHERE customer_id != '???' AND email NOT LIKE '%bot%'"
>                          output_table=web_forms_clean
> 8. SUBMIT_PIPELINE       final_source=web_forms_clean
> ```
>
> All 8 steps complete in well under the 15-step budget.

> [accordion] Task 2 — perfect sequence
>
> ```
> 1–4.  Standardize web_forms (email, name, phone, signup_date)
> 5.    EXECUTE_SQL  filter bots from web_forms  → web_forms_clean
> 6–9.  Standardize legacy_db  (email, name, phone, signup_date)
> 10.   EXECUTE_SQL  filter bots from legacy_db  → legacy_db_clean
> 11.   EXECUTE_SQL  UNION ALL web_forms_clean + legacy_db_clean → merged_output
> 12.   DEDUPLICATE  source=merged_output  strategy=EXACT_EMAIL
> 13.   SUBMIT_PIPELINE  final_source=merged_output
> ```
>
> 13 steps — still within the 15-step budget.

> [accordion] Task 3 — perfect sequence
>
> ```
> 1.  STANDARDIZE_COLUMN  source=salesforce  column=email          strategy=LOWERCASE_STRIP
> 2.  STANDARDIZE_COLUMN  source=salesforce  column=phone          strategy=EXTRACT_NUMBERS
> 3.  STANDARDIZE_COLUMN  source=web_leads   column=email          strategy=LOWERCASE_STRIP
> 4.  STANDARDIZE_COLUMN  source=web_leads   column=phone          strategy=EXTRACT_NUMBERS
> 5.  STANDARDIZE_COLUMN  source=legacy_db   column=contact_email  strategy=LOWERCASE_STRIP
> 6.  STANDARDIZE_COLUMN  source=legacy_db   column=home_phone     strategy=EXTRACT_NUMBERS
> 7.  EXECUTE_SQL  3-way UNION with bot filter + GROUP BY email → merged_output
> 8.  SUBMIT_PIPELINE  final_source=merged_output
> ```
>
> Task 3 uses a single complex SQL to merge all three sources rather than separate UNION steps, staying within the 15-step budget with room to spare.

### Why the grader caps at 0.99

The formula `min(0.99, score - penalty)` hard-caps the maximum possible score at `0.99`. This is a deliberate design choice: `0.99` is the gold-standard ceiling, and `0.01` is the floor — no submission can score exactly `0.0` or `1.0`. For leaderboard purposes, `0.99` represents a perfect pipeline.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.