What Is a POMDP?
A Markov Decision Process (MDP) defines an agent’s world as a tuple (S, A, T, R, γ): a set of states S, a set of actions A, a transition function T(s’ | s, a), a reward function R(s, a), and a discount factor γ. In a fully observable MDP, the agent receives the true state s at every time step and can make optimal decisions with complete information. A POMDP extends this model by introducing a critical constraint: the agent cannot directly observe the true state. Instead, it receives an observation o drawn from an observation function O(o | s, a) — a noisy, partial, or filtered view of the underlying world. The agent must maintain a belief state (a probability distribution over possible true states) and update it as it collects new observations. Real-world enterprise data engineering maps onto a POMDP almost perfectly:- The true state is hidden. The ground truth — what the clean, correct CRM data actually looks like — is captured at episode start and never shown to the agent. The agent cannot access or compare against it during the episode.
- Observations are partial. The agent sees only a preview of the current DataFrame, a quality report it requested, and feedback from its last action. It never sees a “diff” against truth.
- Actions are irreversible mutations. A
STANDARDIZE_COLUMNcall transforms the DataFrame in-place. An incorrect strategy cannot be automatically undone. - Rewards are shaped but imperfect. Step rewards reflect heuristic quality improvement, not ground-truth correctness. Only the terminal grader compares against truth.
The Full Hidden State
The true hidden state of the environment at any point in an episode consists of:-
The ground truth DataFrames — immutable snapshots captured at
reset()time. These are generated once bydata_generator.py(usingFakerwithFaker.seed(None)for per-episode randomness) and never modified. The agent never reads from this store directly. - The live mutable sources — the dictionary of DataFrames held inside the environment instance. Every action that succeeds mutates one or more DataFrames here. This is the agent’s actual working surface.
-
The step counter —
CRMPipelineState.step_count, incremented on every call tostep(), including erroneous ones. It gates theSUBMIT_PIPELINEaction. -
The task configuration — the schema target, conflict rules, and task ID, set at
reset()time from the task data bundle. These define the objective and the evaluation criteria. -
The final submitted DataFrame — populated only when
SUBMIT_PIPELINEis processed. The grader reads from this.
VIEW_SOURCE and PROFILE_SOURCE.
The Observation Space (What the Agent Actually Sees)
After every call to/reset or /step, the environment builds and returns a CRMPipelineObservation object. This object is your agent’s entire window into the world. It contains:
Notice what is absent: the agent never sees the ground truth, the full DataFrame (only 3 rows via VIEW_SOURCE), the cumulative score, or how many rows differ from truth. The agent must build its own belief about data quality through systematic profiling.
Why This Is Genuinely Partially Observable
The partial observability is not a superficial design choice. It reflects three distinct structural gaps between what the agent sees and what exists in the world: Gap 1: VIEW_SOURCE shows only 3 rows. The environment callsdf.head(3).to_markdown() — not the full DataFrame. A dataset of 60 rows after bot injection might have problems concentrated in rows 40–55 that the agent never sees in a raw preview. The agent must use PROFILE_SOURCE to get statistical summaries, then form a belief about the distribution of problems.
Gap 2: PROFILE_SOURCE is per-source. The quality report is generated only for the source named in the action. If the agent fails to profile legacy_db before merging it, it receives no null-count or dtype information about that source. Uninspected sources are blind spots.
Gap 3: Truth is completely hidden. Even after SUBMIT_PIPELINE, the agent receives only a schema-match heuristic reward (not a row-by-row diff). It cannot backtrack. This forces the agent to adopt a systematic cleaning strategy rather than reactive patching.
POMDP Cycle Diagram
Episode Lifecycle in Detail
Phase 1: Reset
When your agent callsPOST /reset, the environment’s reset() method executes the following sequence:
- Reads
TASK_IDfrom the environment variable (defaults to"t1"). - Calls
get_task_data(task_id)fromdata_generator.py, which generates fresh, randomized DataFrames usingFaker. Every episode produces different data becauseFaker.seed(None)is used. - Deep-copies all source DataFrames into the internal sources dictionary. This prevents Pandas in-place mutations from leaking across parallel episodes.
- Generates a fresh UUID as the
episode_id. - Stores a second deep copy of all source DataFrames as the hidden truth snapshot, keyed by
episode_id. This snapshot is immutable for the episode’s lifetime — it is what the grader will compare against. - Initializes
CRMPipelineState(episode_id=episode_id, step_count=0, task_id=task_id). - Returns the initial
CRMPipelineObservationwithdone=False,reward=0.0.
Phase 2: The Step Loop
Each call toPOST /step increments step_count first (before any action logic runs), then dispatches to the appropriate handler. The increment-first design means that even a failed action counts toward your step budget. After handler execution, the observation is assembled from the current state of the sources dictionary, last feedback string, quality report, and current view.
The step loop continues until done=True is returned.
Phase 3: SUBMIT_PIPELINE and Terminal Grading
When your agent submitsSUBMIT_PIPELINE, two things happen:
-
Minimum step check: If
step_count < MIN_STEPS_BEFORE_SUBMIT(which is 3), the submission is rejected:reward = -0.15,done = False. The episode continues and the agent must perform more cleaning work. - Schema match heuristic reward: If the step count requirement is met, the environment immediately computes a fast heuristic reward based on column presence:
- Grader execution: The appropriate grader (
grade_task_1,grade_task_2, orgrade_task_3) callsevaluate_dataframes(truth_df, final_df). This is a row-level, column-level exact-match comparison against the hidden truth. See the Reward System page for the complete grader formula.
The MIN_STEPS_BEFORE_SUBMIT Constraint
The constantMIN_STEPS_BEFORE_SUBMIT = 3 exists to prevent degenerate agent strategies. Without it, a naive agent could call POST /reset and immediately SUBMIT_PIPELINE with the raw, uncleaned data. The minimum step requirement enforces at least minimal engagement with the data: your agent must perform at least 3 steps before a submission is accepted.
The step count includes all actions, not just successful cleaning operations. A
VIEW_SOURCE call counts. An action that returns an error counts. The minimum is a floor on engagement, not a floor on quality.PROFILE_SOURCE → STANDARDIZE_COLUMN (email) → STANDARDIZE_COLUMN (dates) → STANDARDIZE_COLUMN (phone) → HANDLE_MISSING → DEDUPLICATE → SUBMIT_PIPELINE — seven steps minimum.
State Isolation: Preventing Cross-Episode Leakage
In a multi-agent or load-tested deployment scenario, multiple episodes may be running concurrently. The environment uses UUID-keyed isolation to ensure:- Truth isolation: The ground truth snapshot for each episode is written once at reset time and read only by the grader for that specific episode. No other episode’s truth is accessible.
- DataFrame isolation: The
{k: v.copy() for k, v in ...}deep copy at reset time means that Pandas in-place mutations (likedf.drop_duplicates(inplace=True)) inside action handlers never touch the truth snapshot.
CRMPipelineState Fields
TheCRMPipelineState model tracks the minimal state metadata needed to manage the episode. It is available via GET /state.
Optional[str]
default:"None"
A UUID v4 string assigned at
reset() time. It is used to key the truth snapshot and the active environment reference for grading. None before the first reset.int
default:"0"
Increments at the start of every
step() call, before the action handler runs. Used to enforce the MIN_STEPS_BEFORE_SUBMIT = 3 constraint. Never decrements.str
default:"t1"
One of
"t1", "t2", or "t3". Read from the TASK_ID environment variable at reset time. Determines which data generator runs, which grader is selected, and which schema target is loaded.Task Difficulty and POMDP Complexity
The three tasks escalate the complexity of the POMDP’s hidden state in distinct ways:
In Task 3, the
conflict_rules field in the observation provides explicit per-column source priority (prefer_salesforce for email, prefer_web_leads for phone). The hidden truth was constructed by applying these rules — meaning the grader expects the agent to have honored them precisely.