> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# POMDP Model: How the Environment Simulates Data Engineering

> Learn how the CRM Pipeline models data cleaning as a Partially Observable Markov Decision Process with episodic state, transitions, and terminal scoring.

The CRM Data Pipeline Environment does not simply expose a CRUD API over DataFrames. It formalizes the entire data cleaning workflow as a **Partially Observable Markov Decision Process (POMDP)** — a mathematically rigorous decision-making framework that models what the agent can and cannot see, how its actions change the world, and how progress is measured. Understanding this model is essential before you write a single line of agent logic.

## What Is a POMDP?

A Markov Decision Process (MDP) defines an agent's world as a tuple **(S, A, T, R, γ)**: a set of states S, a set of actions A, a transition function T(s' | s, a), a reward function R(s, a), and a discount factor γ. In a **fully observable** MDP, the agent receives the true state s at every time step and can make optimal decisions with complete information.

A POMDP extends this model by introducing a critical constraint: the agent **cannot directly observe the true state**. Instead, it receives an **observation** o drawn from an observation function O(o | s, a) — a noisy, partial, or filtered view of the underlying world. The agent must maintain a **belief state** (a probability distribution over possible true states) and update it as it collects new observations.

Real-world enterprise data engineering maps onto a POMDP almost perfectly:

* **The true state is hidden.** The ground truth — what the clean, correct CRM data actually looks like — is captured at episode start and never shown to the agent. The agent cannot access or compare against it during the episode.
* **Observations are partial.** The agent sees only a preview of the current DataFrame, a quality report it requested, and feedback from its last action. It never sees a "diff" against truth.
* **Actions are irreversible mutations.** A `STANDARDIZE_COLUMN` call transforms the DataFrame in-place. An incorrect strategy cannot be automatically undone.
* **Rewards are shaped but imperfect.** Step rewards reflect heuristic quality improvement, not ground-truth correctness. Only the terminal grader compares against truth.

## The Full Hidden State

The **true hidden state** of the environment at any point in an episode consists of:

1. **The ground truth DataFrames** — immutable snapshots captured at `reset()` time. These are generated once by `data_generator.py` (using `Faker` with `Faker.seed(None)` for per-episode randomness) and never modified. The agent never reads from this store directly.

2. **The live mutable sources** — the dictionary of DataFrames held inside the environment instance. Every action that succeeds mutates one or more DataFrames here. This is the agent's actual working surface.

3. **The step counter** — `CRMPipelineState.step_count`, incremented on every call to `step()`, including erroneous ones. It gates the `SUBMIT_PIPELINE` action.

4. **The task configuration** — the schema target, conflict rules, and task ID, set at `reset()` time from the task data bundle. These define the objective and the evaluation criteria.

5. **The final submitted DataFrame** — populated only when `SUBMIT_PIPELINE` is processed. The grader reads from this.

The agent can never directly inspect items 1, 3, 4, or 5. It can only infer the state of item 2 by calling `VIEW_SOURCE` and `PROFILE_SOURCE`.

## The Observation Space (What the Agent Actually Sees)

After every call to `/reset` or `/step`, the environment builds and returns a `CRMPipelineObservation` object. This object is your agent's **entire window into the world**. It contains:

| Field | Updated By | What It Tells the Agent |
| - | - | - |
| `current_task_objective` | Set at reset, static | The natural-language description of the task |
| `schema_target` | Set at reset, static | The target column names and format descriptions |
| `available_sources` | Updated on merge/SQL | Which DataFrames currently exist |
| `current_view` | `VIEW_SOURCE` | A 3-row markdown preview of one DataFrame |
| `data_quality_report` | `PROFILE_SOURCE` | Null counts and dtypes for one DataFrame |
| `last_action_feedback` | Every step | Human-readable result of the last action |
| `reward` | Every step | The step reward just earned |
| `done` | `SUBMIT_PIPELINE` | Whether the episode has terminated |
| `conflict_rules` | Set at reset, static | Per-column source priority (Task 3 only) |

Notice what is **absent**: the agent never sees the ground truth, the full DataFrame (only 3 rows via VIEW\_SOURCE), the cumulative score, or how many rows differ from truth. The agent must build its own belief about data quality through systematic profiling.

## Why This Is Genuinely Partially Observable

The partial observability is not a superficial design choice. It reflects three distinct structural gaps between what the agent sees and what exists in the world:

**Gap 1: VIEW\_SOURCE shows only 3 rows.** The environment calls `df.head(3).to_markdown()` — not the full DataFrame. A dataset of 60 rows after bot injection might have problems concentrated in rows 40–55 that the agent never sees in a raw preview. The agent must use `PROFILE_SOURCE` to get statistical summaries, then form a belief about the distribution of problems.

**Gap 2: PROFILE\_SOURCE is per-source.** The quality report is generated only for the source named in the action. If the agent fails to profile `legacy_db` before merging it, it receives no null-count or dtype information about that source. Uninspected sources are blind spots.

**Gap 3: Truth is completely hidden.** Even after `SUBMIT_PIPELINE`, the agent receives only a schema-match heuristic reward (not a row-by-row diff). It cannot backtrack. This forces the agent to adopt a systematic cleaning strategy rather than reactive patching.

## POMDP Cycle Diagram

```mermaid theme={null}
flowchart TD
    A([Episode Start\nreset called]) --> B[Environment generates\nfresh DataFrames + truth]
    B --> C[UUID assigned\nTruth snapshot stored\nstep_count = 0]
    C --> D[Initial Observation O₀\nreturned to agent]
    D --> E{Agent selects\naction Aₜ}
    E --> F[POST /step\nCRMPipelineAction payload]
    F --> G[Environment executes\naction handler]
    G --> H[DataFrame mutated\nor error caught]
    H --> I[Heuristic reward Rₜ\ncalculated]
    I --> J[New Observation Oₜ₊₁\nbuilt from updated state]
    J --> K{done == true?}
    K -- No --> E
    K -- Yes --> L[Grader compares\nfinal_df vs truth_df]
    L --> M[Terminal score\n0.01 – 0.99 returned]
    M --> N([Episode End])

    style A fill:#1e3a8a,color:#fff,stroke:#3b82f6
    style N fill:#064e3b,color:#fff,stroke:#10b981
    style L fill:#7c2d12,color:#fff,stroke:#f97316
    style K fill:#312e81,color:#fff,stroke:#818cf8
```

## Episode Lifecycle in Detail

### Phase 1: Reset

When your agent calls `POST /reset`, the environment's `reset()` method executes the following sequence:

1. Reads `TASK_ID` from the environment variable (defaults to `"t1"`).
2. Calls `get_task_data(task_id)` from `data_generator.py`, which generates fresh, randomized DataFrames using `Faker`. Every episode produces **different data** because `Faker.seed(None)` is used.
3. Deep-copies all source DataFrames into the internal sources dictionary. This prevents Pandas in-place mutations from leaking across parallel episodes.
4. Generates a fresh UUID as the `episode_id`.
5. Stores a **second deep copy** of all source DataFrames as the hidden truth snapshot, keyed by `episode_id`. This snapshot is immutable for the episode's lifetime — it is what the grader will compare against.
6. Initializes `CRMPipelineState(episode_id=episode_id, step_count=0, task_id=task_id)`.
7. Returns the initial `CRMPipelineObservation` with `done=False`, `reward=0.0`.

```python theme={null}
# The truth isolation pattern:
# 1. At reset time, a deep copy of source DataFrames is stored as the immutable truth snapshot
# 2. The agent's actions operate on a second, independent copy of those DataFrames
# 3. The grader retrieves the truth snapshot at submission time using the episode's UUID

episode_id = str(uuid.uuid4())
# Deep copy for truth — never touched by agent actions
truth_snapshot = {k: v.copy() for k, v in task_data["hidden_truth"].items()}
# Deep copy for the agent's working surface — mutated by every action
working_sources = {k: v.copy() for k, v in task_data["sources"].items()}
```

### Phase 2: The Step Loop

Each call to `POST /step` increments `step_count` first (before any action logic runs), then dispatches to the appropriate handler. The increment-first design means that even a failed action counts toward your step budget. After handler execution, the observation is assembled from the current state of the sources dictionary, last feedback string, quality report, and current view.

The step loop continues until `done=True` is returned.

### Phase 3: SUBMIT\_PIPELINE and Terminal Grading

When your agent submits `SUBMIT_PIPELINE`, two things happen:

1. **Minimum step check**: If `step_count < MIN_STEPS_BEFORE_SUBMIT` (which is **3**), the submission is rejected: `reward = -0.15`, `done = False`. The episode continues and the agent must perform more cleaning work.

2. **Schema match heuristic reward**: If the step count requirement is met, the environment immediately computes a fast heuristic reward based on column presence:

```python theme={null}
schema_match_ratio = len([c for c in self._schema_target if c in df.columns]) / max(1, len(self._schema_target))
reward = 0.2 * schema_match_ratio
```

This rewards the agent for having the right column names in its output, but does **not** check values. The deep value-level comparison happens in the grader.

3. **Grader execution**: The appropriate grader (`grade_task_1`, `grade_task_2`, or `grade_task_3`) calls `evaluate_dataframes(truth_df, final_df)`. This is a row-level, column-level exact-match comparison against the hidden truth. See the [Reward System](https://ritika-fa61c43f.mintlify.site/concepts/reward-system.md) page for the complete grader formula.

## The MIN\_STEPS\_BEFORE\_SUBMIT Constraint

The constant `MIN_STEPS_BEFORE_SUBMIT = 3` exists to prevent degenerate agent strategies. Without it, a naive agent could call `POST /reset` and immediately `SUBMIT_PIPELINE` with the raw, uncleaned data. The minimum step requirement enforces at least minimal engagement with the data: your agent must perform **at least 3 steps** before a submission is accepted.

> [!NOTE]
>
> The step count includes **all** actions, not just successful cleaning operations. A `VIEW_SOURCE` call counts. An action that returns an error counts. The minimum is a floor on engagement, not a floor on quality.

In practice, a well-performing agent will execute far more than 3 steps. A complete Task 1 pipeline typically requires: `PROFILE_SOURCE` → `STANDARDIZE_COLUMN` (email) → `STANDARDIZE_COLUMN` (dates) → `STANDARDIZE_COLUMN` (phone) → `HANDLE_MISSING` → `DEDUPLICATE` → `SUBMIT_PIPELINE` — seven steps minimum.

## State Isolation: Preventing Cross-Episode Leakage

In a multi-agent or load-tested deployment scenario, multiple episodes may be running concurrently. The environment uses UUID-keyed isolation to ensure:

* **Truth isolation**: The ground truth snapshot for each episode is written once at reset time and read only by the grader for that specific episode. No other episode's truth is accessible.
* **DataFrame isolation**: The `{k: v.copy() for k, v in ...}` deep copy at reset time means that Pandas in-place mutations (like `df.drop_duplicates(inplace=True)`) inside action handlers never touch the truth snapshot.

```python theme={null}
# Truth isolation: written once at reset, keyed by episode UUID
# The grader retrieves the correct truth snapshot at SUBMIT_PIPELINE time
# using the episode's UUID — ensuring concurrent agents never cross-contaminate

episode_id = str(uuid.uuid4())
truth_snapshot = {k: v.copy() for k, v in task_data["hidden_truth"].items()}
# grader access: env.get_episode_truth() → truth_snapshot[episode_id]
```

## CRMPipelineState Fields

The `CRMPipelineState` model tracks the minimal state metadata needed to manage the episode. It is available via `GET /state`.

**episode\_id** `Optional[str]` (default: None)

A UUID v4 string assigned at `reset()` time. It is used to key the truth snapshot and the active environment reference for grading. `None` before the first reset.

**step\_count** `int` (default: 0)

Increments at the **start** of every `step()` call, before the action handler runs. Used to enforce the `MIN_STEPS_BEFORE_SUBMIT = 3` constraint. Never decrements.

**task\_id** `str` (default: t1)

One of `"t1"`, `"t2"`, or `"t3"`. Read from the `TASK_ID` environment variable at reset time. Determines which data generator runs, which grader is selected, and which schema target is loaded.

## Task Difficulty and POMDP Complexity

The three tasks escalate the complexity of the POMDP's hidden state in distinct ways:

| Task | Sources | Hidden Complexity | Key Challenge |
| - | - | - | - |
| **t1 (Easy)** | `web_forms` (1 source) | Single DataFrame with formatting noise + bot rows | Standardize formats, detect bots, deduplicate |
| **t2 (Medium)** | `web_forms` + `legacy_db` (2 sources) | Two DataFrames, overlapping customers, severe nulls | Merge on key, resolve duplicates across sources |
| **t3 (Hard)** | `salesforce` + `web_leads` + `legacy_db` (3 sources) | Three DataFrames with conflicting column names, bots in all 3, per-column conflict rules | 3-way join, conflict resolution via `conflict_rules`, bot filtering via SQL |

In Task 3, the `conflict_rules` field in the observation provides explicit per-column source priority (`prefer_salesforce` for email, `prefer_web_leads` for phone). The hidden truth was constructed by applying these rules — meaning the grader expects the agent to have honored them precisely.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.