Skip to main content
The CRM Data Pipeline Environment does not simply expose a CRUD API over DataFrames. It formalizes the entire data cleaning workflow as a Partially Observable Markov Decision Process (POMDP) — a mathematically rigorous decision-making framework that models what the agent can and cannot see, how its actions change the world, and how progress is measured. Understanding this model is essential before you write a single line of agent logic.

What Is a POMDP?

A Markov Decision Process (MDP) defines an agent’s world as a tuple (S, A, T, R, γ): a set of states S, a set of actions A, a transition function T(s’ | s, a), a reward function R(s, a), and a discount factor γ. In a fully observable MDP, the agent receives the true state s at every time step and can make optimal decisions with complete information. A POMDP extends this model by introducing a critical constraint: the agent cannot directly observe the true state. Instead, it receives an observation o drawn from an observation function O(o | s, a) — a noisy, partial, or filtered view of the underlying world. The agent must maintain a belief state (a probability distribution over possible true states) and update it as it collects new observations. Real-world enterprise data engineering maps onto a POMDP almost perfectly:
  • The true state is hidden. The ground truth — what the clean, correct CRM data actually looks like — is captured at episode start and never shown to the agent. The agent cannot access or compare against it during the episode.
  • Observations are partial. The agent sees only a preview of the current DataFrame, a quality report it requested, and feedback from its last action. It never sees a “diff” against truth.
  • Actions are irreversible mutations. A STANDARDIZE_COLUMN call transforms the DataFrame in-place. An incorrect strategy cannot be automatically undone.
  • Rewards are shaped but imperfect. Step rewards reflect heuristic quality improvement, not ground-truth correctness. Only the terminal grader compares against truth.

The Full Hidden State

The true hidden state of the environment at any point in an episode consists of:
  1. The ground truth DataFrames — immutable snapshots captured at reset() time. These are generated once by data_generator.py (using Faker with Faker.seed(None) for per-episode randomness) and never modified. The agent never reads from this store directly.
  2. The live mutable sources — the dictionary of DataFrames held inside the environment instance. Every action that succeeds mutates one or more DataFrames here. This is the agent’s actual working surface.
  3. The step counter — CRMPipelineState.step_count, incremented on every call to step(), including erroneous ones. It gates the SUBMIT_PIPELINE action.
  4. The task configuration — the schema target, conflict rules, and task ID, set at reset() time from the task data bundle. These define the objective and the evaluation criteria.
  5. The final submitted DataFrame — populated only when SUBMIT_PIPELINE is processed. The grader reads from this.
The agent can never directly inspect items 1, 3, 4, or 5. It can only infer the state of item 2 by calling VIEW_SOURCE and PROFILE_SOURCE.

The Observation Space (What the Agent Actually Sees)

After every call to /reset or /step, the environment builds and returns a CRMPipelineObservation object. This object is your agent’s entire window into the world. It contains: Notice what is absent: the agent never sees the ground truth, the full DataFrame (only 3 rows via VIEW_SOURCE), the cumulative score, or how many rows differ from truth. The agent must build its own belief about data quality through systematic profiling.

Why This Is Genuinely Partially Observable

The partial observability is not a superficial design choice. It reflects three distinct structural gaps between what the agent sees and what exists in the world: Gap 1: VIEW_SOURCE shows only 3 rows. The environment calls df.head(3).to_markdown() — not the full DataFrame. A dataset of 60 rows after bot injection might have problems concentrated in rows 40–55 that the agent never sees in a raw preview. The agent must use PROFILE_SOURCE to get statistical summaries, then form a belief about the distribution of problems. Gap 2: PROFILE_SOURCE is per-source. The quality report is generated only for the source named in the action. If the agent fails to profile legacy_db before merging it, it receives no null-count or dtype information about that source. Uninspected sources are blind spots. Gap 3: Truth is completely hidden. Even after SUBMIT_PIPELINE, the agent receives only a schema-match heuristic reward (not a row-by-row diff). It cannot backtrack. This forces the agent to adopt a systematic cleaning strategy rather than reactive patching.

POMDP Cycle Diagram

Episode Lifecycle in Detail

Phase 1: Reset

When your agent calls POST /reset, the environment’s reset() method executes the following sequence:
  1. Reads TASK_ID from the environment variable (defaults to "t1").
  2. Calls get_task_data(task_id) from data_generator.py, which generates fresh, randomized DataFrames using Faker. Every episode produces different data because Faker.seed(None) is used.
  3. Deep-copies all source DataFrames into the internal sources dictionary. This prevents Pandas in-place mutations from leaking across parallel episodes.
  4. Generates a fresh UUID as the episode_id.
  5. Stores a second deep copy of all source DataFrames as the hidden truth snapshot, keyed by episode_id. This snapshot is immutable for the episode’s lifetime — it is what the grader will compare against.
  6. Initializes CRMPipelineState(episode_id=episode_id, step_count=0, task_id=task_id).
  7. Returns the initial CRMPipelineObservation with done=False, reward=0.0.

Phase 2: The Step Loop

Each call to POST /step increments step_count first (before any action logic runs), then dispatches to the appropriate handler. The increment-first design means that even a failed action counts toward your step budget. After handler execution, the observation is assembled from the current state of the sources dictionary, last feedback string, quality report, and current view. The step loop continues until done=True is returned.

Phase 3: SUBMIT_PIPELINE and Terminal Grading

When your agent submits SUBMIT_PIPELINE, two things happen:
  1. Minimum step check: If step_count < MIN_STEPS_BEFORE_SUBMIT (which is 3), the submission is rejected: reward = -0.15, done = False. The episode continues and the agent must perform more cleaning work.
  2. Schema match heuristic reward: If the step count requirement is met, the environment immediately computes a fast heuristic reward based on column presence:
This rewards the agent for having the right column names in its output, but does not check values. The deep value-level comparison happens in the grader.
  1. Grader execution: The appropriate grader (grade_task_1, grade_task_2, or grade_task_3) calls evaluate_dataframes(truth_df, final_df). This is a row-level, column-level exact-match comparison against the hidden truth. See the Reward System page for the complete grader formula.

The MIN_STEPS_BEFORE_SUBMIT Constraint

The constant MIN_STEPS_BEFORE_SUBMIT = 3 exists to prevent degenerate agent strategies. Without it, a naive agent could call POST /reset and immediately SUBMIT_PIPELINE with the raw, uncleaned data. The minimum step requirement enforces at least minimal engagement with the data: your agent must perform at least 3 steps before a submission is accepted.
The step count includes all actions, not just successful cleaning operations. A VIEW_SOURCE call counts. An action that returns an error counts. The minimum is a floor on engagement, not a floor on quality.
In practice, a well-performing agent will execute far more than 3 steps. A complete Task 1 pipeline typically requires: PROFILE_SOURCE → STANDARDIZE_COLUMN (email) → STANDARDIZE_COLUMN (dates) → STANDARDIZE_COLUMN (phone) → HANDLE_MISSING → DEDUPLICATE → SUBMIT_PIPELINE — seven steps minimum.

State Isolation: Preventing Cross-Episode Leakage

In a multi-agent or load-tested deployment scenario, multiple episodes may be running concurrently. The environment uses UUID-keyed isolation to ensure:
  • Truth isolation: The ground truth snapshot for each episode is written once at reset time and read only by the grader for that specific episode. No other episode’s truth is accessible.
  • DataFrame isolation: The {k: v.copy() for k, v in ...} deep copy at reset time means that Pandas in-place mutations (like df.drop_duplicates(inplace=True)) inside action handlers never touch the truth snapshot.

CRMPipelineState Fields

The CRMPipelineState model tracks the minimal state metadata needed to manage the episode. It is available via GET /state.
Optional[str]
default:"None"
A UUID v4 string assigned at reset() time. It is used to key the truth snapshot and the active environment reference for grading. None before the first reset.
int
default:"0"
Increments at the start of every step() call, before the action handler runs. Used to enforce the MIN_STEPS_BEFORE_SUBMIT = 3 constraint. Never decrements.
str
default:"t1"
One of "t1", "t2", or "t3". Read from the TASK_ID environment variable at reset time. Determines which data generator runs, which grader is selected, and which schema target is loaded.

Task Difficulty and POMDP Complexity

The three tasks escalate the complexity of the POMDP’s hidden state in distinct ways: In Task 3, the conflict_rules field in the observation provides explicit per-column source priority (prefer_salesforce for email, prefer_web_leads for phone). The hidden truth was constructed by applying these rules — meaning the grader expects the agent to have honored them precisely.