> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# POST /grader/{episode_id} — Score a Completed Episode

> POST /grader/{episode_id} compares your agent's final DataFrame against hidden ground truth and returns a normalized score from 0.01 to 0.99.

`POST /grader/{episode_id}` is the evaluation endpoint that scores your agent's submitted pipeline output against the hidden ground truth snapshot that was locked in server memory at the time `/reset` was called. The grader performs a left-join on `customer_id`, checks every non-key column for exact match (case-insensitive, whitespace-stripped), and computes a normalized accuracy score clamped to the range `[0.01, 0.99]`. A penalty is applied for excess rows beyond the truth row count.

Call this endpoint manually after your pipeline is complete — once `/reset` has been called and a `SUBMIT_PIPELINE` action has been executed with `done = true`.

## Path Parameters

**episode\_id** `string` *required*

The UUID of the episode to grade. This is the value returned in
`observation.last_action_feedback` after `/reset` (format:
`"Environment reset natively for task t1. Episode UUID: {uuid}"`),
and also available from `GET /state` as `episode_id`.

## Query Parameters

**final\_source** `string` *required*

The name of the source DataFrame to grade — the same value you passed as
`final_source` in your `SUBMIT_PIPELINE` action. For tasks `t1`, this is
typically `"web_forms"`. For tasks `t2` and `t3`, this is typically
`"merged_output"`.

**task\_id** `string` *required*

The task identifier. Must be one of `"t1"`, `"t2"`, or `"t3"`. This
controls which truth key the grader loads from the stored snapshot:
`"web_forms"` for `t1`, or `"merged_output"` for `t2` and `t3`.

## Request

All parameters are passed as URL query strings. No request body is needed.

```bash theme={null}
curl -X POST "http://localhost:8080/grader/3f8a1b2c-d4e5-6f7a-8b9c-0d1e2f3a4b5c?final_source=web_forms&task_id=t1"
```

For Task 2 or Task 3:

```bash theme={null}
curl -X POST "http://localhost:8080/grader/7c3e9d1a-f2b4-5e6d-9a8b-7c0d1e2f3a4b?final_source=merged_output&task_id=t3"
```

## Response

```json theme={null}
{
  "episode_id": "3f8a1b2c-d4e5-6f7a-8b9c-0d1e2f3a4b5c",
  "task_id": "t1",
  "score": 0.95,
  "rows": 40
}
```

## Response Fields

**episode\_id** `string`

The episode UUID echoed back from the path parameter. Use this to correlate
grader results with specific reset calls in multi-episode evaluation runs.

**task\_id** `string`

The task identifier echoed back from the query parameter (`t1`, `t2`, or
`t3`). Confirms which truth schema was used during evaluation.

**score** `float`

The normalized accuracy score for the agent's submitted DataFrame. Range is
always `[0.01, 0.99]`. See the Scoring Formula and Score Interpretation
sections below for how this value is computed and what it means.

**rows** `integer`

The number of rows in the agent's submitted DataFrame at the time of
grading. Compare this against the truth row count to detect whether your
pipeline produced extra rows (bots not removed, duplicates retained) or
fewer rows than expected (over-aggressive dropping).

## Error Responses

**404 — No truth snapshot found**

Returned when `episode_id` does not match any cached ground truth snapshot.
This occurs when `/reset` was never called, the server was restarted
(clearing in-memory state), or you supplied an incorrect UUID.

```json theme={null}
{
  "detail": "No underlying truth snapshot found for episode_id='3f8a1b2c-...'. Ensure /reset is called explicitly before invoking the grader engine."
}
```

**400 — Invalid task\_id**

Returned when `task_id` is not one of the recognized values (`t1`, `t2`, `t3`).

```json theme={null}
{
  "detail": "Unrecognized task_id constraint 't4'."
}
```

**400 — No final DataFrame submitted**

Returned when the agent has not yet executed a `SUBMIT_PIPELINE` action
that set `done = true`. The `env.final_df` pointer is `None`.

```json theme={null}
{
  "detail": "The active agent has not materialized a final dataframe for submission."
}
```

**500 — Truth key missing from snapshot**

An internal state error indicating the truth snapshot exists but does not
contain the expected truth key for the given `task_id`. This should not
occur in normal operation.

```json theme={null}
{
  "detail": "Target truth key 'merged_output' is missing from the state snapshot."
}
```

## Scoring Formula

The grader executes the following algorithm in `evaluate_dataframes()` from `server/graders.py`:

**1. Deduplicate agent output on `customer_id`**

Duplicate `customer_id` values in the agent's DataFrame are dropped before comparison to prevent Cartesian explosion during the merge.

**2. Left-join truth against agent on `customer_id`**

```python theme={null}
merged = pd.merge(truth_df, agent_df, on="customer_id",
                  suffixes=('_truth', '_agent'), how='left')
```

Every truth row is preserved. Agent rows without a matching `customer_id` in the truth are excluded.

**3. Column-by-column exact match**

For every non-key column in the truth schema, both the truth value and the agent value are lowercased and whitespace-stripped, then compared for exact equality:

```python theme={null}
s_truth = merged[col_truth].fillna("").astype(str).str.strip().str.lower()
s_agent = merged[col_agent].fillna("").astype(str).str.strip().str.lower()
merged[f"{col}_match"] = (s_truth == s_agent)
```

A row is marked **correct** only if **all** column comparisons are `True`.

**4. Compute base score**

```python theme={null}
score = correct_rows / num_truth_rows
```

**5. Apply excess-row penalty**

Each row in the agent's output beyond the truth row count incurs a `-0.15` penalty per extra row:

```python theme={null}
penalty = max(0, len(agent_df) - num_truth_rows) * 0.15
```

**6. Clamp to valid range**

```python theme={null}
return min(0.99, max(0.01, score - penalty))
```

> [!NOTE]
>
> Because the grader uses case-insensitive, stripped exact-string matching,
> your pipeline must produce values that match the ground truth format
> precisely after normalization. Dates must be `YYYY-MM-DDTHH:MM:SS` (ISO 8601
> via `TO_DATETIME_ISO`). Phones must be `+{digits}` with no separators
> (E.164 via `EXTRACT_NUMBERS`). Emails must be fully lowercase with no
> surrounding whitespace.

## Score Interpretation

| Score Range | Interpretation |
| - | - |
| `0.95 – 0.99` | Near-perfect output. All columns are correctly formatted for the vast majority of rows. Minor residual issues on edge-case inputs (malformed dates that could not be parsed, etc.). |
| `0.80 – 0.95` | Good cleaning with some columns still imperfect. Likely a missed `STANDARDIZE_COLUMN` pass or leftover bot rows not filtered by `EXECUTE_SQL`. |
| `0.50 – 0.80` | Partial cleaning. A significant fraction of rows have at least one column in the wrong format. Review `data_quality_report` and re-run standardization steps. |
| `0.01 – 0.50` | Poor output. Core cleaning steps were skipped, wrong column names were used, or the wrong `final_source` was submitted. |

## Calling the Grader Programmatically

```python theme={null}
import requests, re

# 1. Reset and capture episode_id
reset_resp = requests.post("http://localhost:8080/reset", json={}).json()
feedback = reset_resp["observation"]["last_action_feedback"]
episode_id = re.search(r"Episode UUID: ([^\s]+)", feedback).group(1)
task_id = requests.get("http://localhost:8080/state").json()["task_id"]

# 2. … run your cleaning pipeline via POST /step …

# 3. Submit
submit_resp = requests.post("http://localhost:8080/step", json={
    "action_type": "SUBMIT_PIPELINE",
    "final_source": "web_forms"
}).json()

# 4. Grade
if submit_resp["done"]:
    grade = requests.post(
        f"http://localhost:8080/grader/{episode_id}",
        params={"final_source": "web_forms", "task_id": task_id}
    ).json()
    print(f"Score: {grade['score']} on {grade['rows']} rows")
```

> [!WARNING]
>
> Ground truth snapshots are held in server process memory and are **not**
> persisted to disk. If the server process restarts between `/reset` and your
> grader call, the snapshot is lost and the grader will return a 404. Always
> complete a full episode (reset → step → submit → grade) within a single
> uninterrupted server session.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.