> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Action Space: All 8 Pipeline Operations Explained

> Reference guide for all 8 CRMPipelineAction types — required fields, reward values, JSON examples, and enum values for every pipeline operation.

Actions are expressed as `CRMPipelineAction` Pydantic objects sent as JSON to `POST /step`. Each action type maps to a concrete data engineering operation inside the environment. The environment dispatches on the `action_type` field, routes to the corresponding handler method, mutates the in-memory Pandas DataFrames, computes a heuristic step reward, and returns a fresh `CRMPipelineObservation`. All fields not required by a given action type should be omitted or set to `null` — the Pydantic model treats them as `Optional`.

The base action structure is minimal by design:

```json theme={null}
{
  "action_type": "<PipelineActionType>",
  // additional fields depending on action type
}
```

Every field beyond `action_type` is optional at the model level, but individual handlers will raise a `ValueError` (caught as a `-0.05` penalty) if required fields are absent for that specific operation. The full field reference is at the bottom of this page.

***

## The 8 Action Types

> [accordion] VIEW\_SOURCE — Preview a DataFrame
>
> **What it does:** Calls `df.head(3).to_markdown()` on the named source and stores the result in the current view. This is a **read-only** action — it does not mutate any DataFrame. Use it to understand the raw structure of a source before deciding which transformations to apply.
>
> **Required fields:**
>
> * `source` — the name of the DataFrame to preview (must exist in `available_sources`)
>
> **Optional fields:** None
>
> **Returns in observation:**
>
> * `current_view` — a markdown-formatted table of the first 3 rows
> * `last_action_feedback` — `"Viewing top 3 rows of <source>"`
> * `data_quality_report` — **not updated** by this action
>
> **Reward:** `0.0` — purely informational, no reward signal
>
> **When to use it:** Call `VIEW_SOURCE` at the start of an episode to understand column names, example values, and obvious formatting problems. Also useful after a `MERGE_SOURCES` or `EXECUTE_SQL` operation to verify the shape of a newly created DataFrame.
>
> ```json theme={null}
> {
>   "action_type": "VIEW_SOURCE",
>   "source": "web_forms"
> }
> ```
>
> **Example feedback:**
>
> ```
> Viewing top 3 rows of web_forms
> ```
>
> **Example current\_view output:**
>
> ```
> |    | customer_id   | name              | email                   | signup_date   | phone             |
> |----|---------------|-------------------|-------------------------|---------------|-------------------|
> |  0 | CUST_1000     |   ALICE JOHNSON   | ALICE.JOHNSON@GMAIL.COM | 2023/04/12    | (415) 555-0192    |
> |  1 | CUST_1001     | bob smith         | bob@example.com         | ??/??/????    | Invalid Number    |
> |  2 | BOT10293      | Newsletter Bot    | promo99@bulk-mail.co    | 1970-01-01    | +00000000000      |
> ```
>
> > [!NOTE]
> >
> > `VIEW_SOURCE` shows only 3 rows. For datasets of 40–70 rows, those 3 rows may not be representative of the full distribution of data quality problems. Always follow up with `PROFILE_SOURCE` to get statistical summaries across all rows.

> [accordion] PROFILE\_SOURCE — Generate a Quality Report
>
> **What it does:** Computes null counts and data types for every column in the named source DataFrame, then formats the results as a markdown-style report. This updates the `data_quality_report` field in the observation. Like `VIEW_SOURCE`, this is **read-only** and does not mutate the DataFrame.
>
> **Required fields:**
>
> * `source` — the name of the DataFrame to profile
>
> **Optional fields:** None
>
> **Returns in observation:**
>
> * `data_quality_report` — a per-column report in the format `- **<col>**: type=<dtype>, nulls=<count>`
> * `last_action_feedback` — `"Profiled <source>"`
>
> **Reward:** `+0.02` — rewarded for gathering information before acting
>
> **When to use it:** Call `PROFILE_SOURCE` on every source at the start of a task. This is the primary mechanism for overcoming the environment's partial observability — null counts reveal which columns need `HANDLE_MISSING`, and dtype values reveal which columns need `STANDARDIZE_COLUMN`. Without profiling, your agent is operating blind.
>
> ```json theme={null}
> {
>   "action_type": "PROFILE_SOURCE",
>   "source": "web_forms"
> }
> ```
>
> **Example data\_quality\_report:**
>
> ```
> - **customer_id**: type=object, nulls=0
> - **name**: type=object, nulls=2
> - **email**: type=object, nulls=8
> - **signup_date**: type=object, nulls=5
> - **phone**: type=object, nulls=11
> ```
>
> **Example feedback:**
>
> ```
> Profiled web_forms
> ```
>
> > [!TIP]
> >
> > Profile every source before any transformation. In Task 3, the three sources (`salesforce`, `web_leads`, `legacy_db`) have different column names — `legacy_db` uses `legacy_id`, `contact_email`, and `home_phone` instead of the standard names. Profiling reveals this schema mismatch before you attempt a merge.

> [accordion] STANDARDIZE\_COLUMN — Apply a Deterministic Format Transform
>
> **What it does:** Applies one of four deterministic string transformation strategies to a single column in a named source. The transformation is applied **in-place** to the specified column. The reward is heuristically calculated based on how many values were actually changed (a proxy for how dirty the column was).
>
> **Required fields:**
>
> * `source` — the DataFrame containing the column
> * `column` — the column to transform
> * `standardization_strategy` — one of the four strategies below
>
> **Optional fields:** None
>
> **Reward:** `min(0.05, 0.01 + improvement_ratio × 0.1)` on success (minimum `+0.01`, maximum `+0.05`), `-0.05` on exception (e.g., column not found, source not found)
>
> **Strategies in depth:**
>
> **`LOWERCASE_STRIP`** — Converts all string values to lowercase and strips leading/trailing whitespace. Implemented as `df[col].astype(str).str.lower().str.strip()`. Use this on `email` and `name` columns where values may be uppercase or padded.
>
> ```json theme={null}
> {
>   "action_type": "STANDARDIZE_COLUMN",
>   "source": "web_forms",
>   "column": "email",
>   "standardization_strategy": "LOWERCASE_STRIP"
> }
> ```
>
> **`TO_DATETIME_ISO`** — Parses dates using `pd.to_datetime(errors='coerce')` and formats them as `'%Y-%m-%dT00:00:00'`. Unparseable values (like `"??/??/????"` or `"2023-13-12"`) become empty strings via `.fillna("")`. Use this on `signup_date` columns.
>
> ```json theme={null}
> {
>   "action_type": "STANDARDIZE_COLUMN",
>   "source": "web_forms",
>   "column": "signup_date",
>   "standardization_strategy": "TO_DATETIME_ISO"
> }
> ```
>
> **`EXTRACT_NUMBERS`** — Strips all non-digit characters from phone values using `re.sub(r'\D+', '', s)`, then prepends a `+` prefix. Also handles extension stripping — values containing `ext` or `x` are split at that boundary before digit extraction. Values that produce no digits return an empty string. Use this on phone columns with mixed formats like `"(415) 555-0192"` or `"555-1234 ext. 42"`.
>
> ```json theme={null}
> {
>   "action_type": "STANDARDIZE_COLUMN",
>   "source": "web_forms",
>   "column": "phone",
>   "standardization_strategy": "EXTRACT_NUMBERS"
> }
> ```
>
> **`PHONE_E164`** — Defined in the `StandardizationStrategy` enum but **not separately implemented** in the current handler. The standardization dispatch has no branch for `PHONE_E164`, so calling it returns a minimal reward (`+0.01`) without transforming any values. For phone normalization, use `EXTRACT_NUMBERS` instead — it provides the full digit-extraction and `+` prefix logic.
>
> ```json theme={null}
> {
>   "action_type": "STANDARDIZE_COLUMN",
>   "source": "web_forms",
>   "column": "phone",
>   "standardization_strategy": "EXTRACT_NUMBERS"
> }
> ```
>
> **Example feedback:**
>
> ```
> Standardized web_forms.email using LOWERCASE_STRIP
> ```
>
> > [!WARNING]
> >
> > The reward for `STANDARDIZE_COLUMN` is **capped at 0.05** regardless of how many rows were changed. The formula is `min(0.05, 0.01 + improvement_ratio × 0.1)`. Calling the same standardization twice on a column that is already clean yields the minimum reward (`+0.01`) because `improvement_ratio` will be near zero.

> [accordion] HANDLE\_MISSING — Fill or Drop Null Values
>
> **What it does:** Resolves null (NaN) values in a specific column of a named source using one of four strategies. The `DROP_ROW` and `FILL_VALUE` strategies are fully implemented and return proportional rewards. The `FILL_MEAN` and `FILL_MODE` strategies are defined in the enum but not yet implemented in the handler — submitting them raises an internal exception that is caught as a **`-0.05` penalty**. Use `FILL_VALUE` or `DROP_ROW` for reliable null handling.
>
> **Required fields:**
>
> * `source` — the DataFrame containing the column
> * `column` — the column with null values to handle
> * `missing_strategy` — one of the four strategies (`FILL_VALUE` and `DROP_ROW` are implemented)
>
> **Optional fields:**
>
> * `fallback_value` — required only when `missing_strategy` is `FILL_VALUE`. Ignored for all other strategies.
>
> **Reward:** `+0.01` to `+0.05` on success (proportional to null fraction resolved), `-0.05` on exception or unimplemented strategy
>
> **Strategies in depth:**
>
> **`FILL_VALUE`** *(implemented)* — Fills all null values in the column with the literal string provided in `fallback_value`, as `df[col].fillna(action.fallback_value, inplace=True)`. Use this to fill missing phone numbers with `""` (empty string) or missing emails with a sentinel value.
>
> ```json theme={null}
> {
>   "action_type": "HANDLE_MISSING",
>   "source": "web_forms",
>   "column": "phone",
>   "missing_strategy": "FILL_VALUE",
>   "fallback_value": ""
> }
> ```
>
> **`DROP_ROW`** *(implemented)* — Drops all rows where the specified column is null using `df.dropna(subset=[col], inplace=True)`. This is a destructive operation — rows are permanently removed from the episode's working DataFrame. Use with caution: dropping rows with missing emails may remove legitimate customers, reducing your grader score.
>
> ```json theme={null}
> {
>   "action_type": "HANDLE_MISSING",
>   "source": "web_forms",
>   "column": "email",
>   "missing_strategy": "DROP_ROW"
> }
> ```
>
> **`FILL_MEAN`** *(not yet implemented — returns `-0.05`)* — Defined in the enum but the handler has no branch for this strategy. Submitting it raises an `UnboundLocalError` caught as a `-0.05` penalty.
>
> **`FILL_MODE`** *(not yet implemented — returns `-0.05`)* — Defined in the enum but the handler has no branch for this strategy. Submitting it raises an `UnboundLocalError` caught as a `-0.05` penalty.
>
> **Reward formulas:**
>
> * `DROP_ROW`: `max(0.01, (null_count_before / (len(df_after_drop) + null_count_before)) × 0.1)`
> * `FILL_VALUE`: `max(0.01, (null_count_before / len(df)) × 0.05)`
>
> **Example feedback:**
>
> ```
> Handled 11 missing values in web_forms.phone
> ```
>
> > [!WARNING]
> >
> > The `DROP_ROW` strategy removes entire rows from the DataFrame. If a legitimate customer has a null phone number, dropping on `phone` removes that customer from your final output. The grader performs a left-join from truth to your submission — missing truth rows count as incorrect rows, directly reducing your score.

> [accordion] DEDUPLICATE — Remove Duplicate Records
>
> **What it does:** Identifies and removes duplicate rows from a named source using one of two strategies. Operates in-place and saves the deduplicated DataFrame back to the sources store. The reward is proportional to the fraction of rows removed (i.e., how many duplicates were found).
>
> **Required fields:**
>
> * `source` — the DataFrame to deduplicate
> * `deduplication_strategy` — one of the two strategies below
>
> **Optional fields:** None
>
> **Reward:** `(removed / start_len) × 0.2` — proportional to duplicate density, maximum `0.2` if all rows were duplicates. Returns `-0.05` on exception.
>
> **Strategies in depth:**
>
> **`EXACT_EMAIL`** — Calls `df.drop_duplicates(subset=["email"], inplace=True)`. Keeps the first occurrence of each unique email address and removes all subsequent rows with the same email. This is appropriate when `email` is the reliable unique identifier. Note that rows with null emails are treated as a single group by Pandas' `drop_duplicates` — the first null-email row is kept.
>
> ```json theme={null}
> {
>   "action_type": "DEDUPLICATE",
>   "source": "web_forms",
>   "deduplication_strategy": "EXACT_EMAIL"
> }
> ```
>
> **`FUZZY_NAME_PHONE`** — A soft-match deduplication strategy. Creates two temporary columns: `_tmp_name` (lowercase-stripped name) and `_tmp_phone` (digits-only phone). Drops duplicates on the combined key `["_tmp_name", "_tmp_phone"]`, then removes the temporary columns. This catches cases where the same person appears twice with slightly different formatting (e.g., `"  Alice Johnson  "` vs `"alice johnson"`) but the same normalized phone.
>
> ```json theme={null}
> {
>   "action_type": "DEDUPLICATE",
>   "source": "web_forms",
>   "deduplication_strategy": "FUZZY_NAME_PHONE"
> }
> ```
>
> **Example feedback:**
>
> ```
> Deduplicated web_forms, removed 12 rows.
> ```
>
> > [!TIP]
> >
> > Bot rows injected by the data generator have recognizable signatures: `customer_id` values beginning with `"BOT"`, `signup_date` of `"1970-01-01"`, and phone `"+00000000000"`. While `DEDUPLICATE` can remove duplicate bots, it won't remove unique bot rows. Use `EXECUTE_SQL` with a `WHERE customer_id NOT LIKE 'BOT%'` filter to eliminate them entirely.

> [accordion] MERGE\_SOURCES — Join Two DataFrames
>
> **What it does:** Performs an outer join between two named source DataFrames on a specified key column, resolves column conflicts according to the `conflict_rule`, and stores the result as `"merged_output"` in the sources store. The join uses `pd.merge(df1, df2, on=key, how="outer", suffixes=('_s1', '_s2'))` and then reconciles every pair of conflicting columns.
>
> **Required fields:**
>
> * `source` — the first (primary) DataFrame
> * `source2` — the second DataFrame to join
> * `join_key` — the column name shared by both DataFrames to join on
> * `output_table` — the name to use for the merged result in `available_sources`
>
> **Optional fields:**
>
> * `conflict_rule` — one of `PREFER_S1`, `PREFER_S2`, or `COALESCE` (default behavior if omitted is `COALESCE`)
>
> **Reward:** `+0.05` on success, `-0.05` if the join key is missing from either source
>
> **Conflict rules:**
>
> Both DataFrames are normalized on the join key before merging (`astype(str).str.lower().str.strip()`), which enables fuzzy key matching.
>
> After the outer join, any column that existed in both DataFrames (and thus received `_s1` / `_s2` suffixes) is reconciled:
>
> * **`PREFER_S1`** — Takes the value from the first source; falls back to the second if null: `merged[s1_col].combine_first(merged[s2_col])`
> * **`PREFER_S2`** — Takes the value from the second source; falls back to the first if null: `merged[s2_col].combine_first(merged[s1_col])`
> * **`COALESCE`** — Takes the first non-null value from either source (functionally equivalent to `PREFER_S1` in this implementation): `merged[s1_col].combine_first(merged[s2_col])`
>
> ```json theme={null}
> {
>   "action_type": "MERGE_SOURCES",
>   "source": "web_forms",
>   "source2": "legacy_db",
>   "join_key": "customer_id",
>   "output_table": "merged_output",
>   "conflict_rule": "PREFER_S1"
> }
> ```
>
> **Example feedback:**
>
> ```
> Merged web_forms and legacy_db into 'merged_output' using customer_id.
> ```
>
> > [!NOTE]
> >
> > `MERGE_SOURCES` always writes to `"merged_output"` in the sources store, regardless of the `output_table` value. The `output_table` field is accepted by the model but the handler hardcodes the key `"merged_output"`. After a merge, `"merged_output"` will appear in your `available_sources` list. For a 3-way merge in Task 3, use `EXECUTE_SQL` to perform a custom 3-table join instead.
>
> > [!WARNING]
> >
> > Before merging, ensure that both sources use the **same column name** for the join key. In Task 3, `legacy_db` uses `legacy_id` and `contact_email` instead of `customer_id` and `email`. You must rename or transform those columns (via `EXECUTE_SQL`) before `MERGE_SOURCES` can join on `customer_id`.

> [accordion] EXECUTE\_SQL — Run a SELECT Query Over DataFrames
>
> **What it does:** Loads all current DataFrames into an in-memory SQLite database (one table per source, named by its key in the sources store), executes your `SELECT` query via `pd.read_sql_query()`, stores the result DataFrame under `output_table`, and closes the connection. This is the most powerful action in the space — it enables arbitrary filtering, joining, column renaming, and aggregation.
>
> **Required fields:**
>
> * `query` — a valid SQLite `SELECT` statement
> * `output_table` — the name to register the result under in `available_sources`
>
> **Optional fields:** None
>
> **Reward:** `+0.1` on success, `-0.1` on SQL error or security violation
>
> **Security restrictions:** The query is uppercased and checked for forbidden keywords before execution. Queries that do not begin with `SELECT ` are blocked. The following keywords also block execution: `DROP `, `DELETE `, `UPDATE `, `INSERT `, `ALTER `, `CREATE `, `TRUNCATE `.
>
> ```json theme={null}
> {
>   "action_type": "EXECUTE_SQL",
>   "query": "SELECT * FROM web_forms WHERE customer_id NOT LIKE 'BOT%'",
>   "output_table": "web_forms_clean"
> }
> ```
>
> **Example — Filter bots from a source:**
>
> ```json theme={null}
> {
>   "action_type": "EXECUTE_SQL",
>   "query": "SELECT * FROM web_forms WHERE customer_id NOT LIKE 'BOT%' AND phone != '+00000000000'",
>   "output_table": "web_forms_no_bots"
> }
> ```
>
> **Example — 3-way merge for Task 3:**
>
> ```json theme={null}
> {
>   "action_type": "EXECUTE_SQL",
>   "query": "SELECT sf.customer_id, sf.email, COALESCE(wl.phone, sf.phone) as phone FROM salesforce sf LEFT JOIN web_leads wl ON LOWER(TRIM(sf.customer_id)) = LOWER(TRIM(wl.customer_id)) WHERE sf.customer_id NOT LIKE 'BOT%'",
>   "output_table": "merged_output"
> }
> ```
>
> **Example — Rename legacy\_db columns and filter:**
>
> ```json theme={null}
> {
>   "action_type": "EXECUTE_SQL",
>   "query": "SELECT REPLACE(legacy_id, 'OLD-', 'CUST_') as customer_id, LOWER(TRIM(contact_email)) as email, home_phone as phone FROM legacy_db WHERE legacy_id NOT LIKE 'BOT%'",
>   "output_table": "legacy_normalized"
> }
> ```
>
> **Example feedback (success):**
>
> ```
> Executed SQL successfully. Wrote 42 rows to 'web_forms_clean'.
> ```
>
> **Example feedback (error):**
>
> ```
> SQL Error: no such table: salesfore
> ```
>
> **Example feedback (blocked):**
>
> ```
> Blocked Action: Only SELECT queries are allowed for security.
> ```
>
> > [!TIP]
> >
> > `EXECUTE_SQL` is the recommended approach for Task 3's 3-way merge. It gives you full control over join type, column selection, conflict resolution logic (via `COALESCE`), and bot filtering — all in a single step. The SQLite table names inside the query must exactly match the keys in `available_sources`.
>
> > [!WARNING]
> >
> > The column type conversion `clean_df[col].astype(str)` is applied to all object-dtype columns before loading into SQLite. This means null values become the string `"None"` inside SQLite. Use `WHERE column != 'None'` rather than `WHERE column IS NOT NULL` when filtering null-origin values in SQL.

> [accordion] SUBMIT\_PIPELINE — Finalize and Trigger Grading
>
> **What it does:** Signals the end of the episode. If the `step_count` is below `MIN_STEPS_BEFORE_SUBMIT` (3), the submission is rejected with a penalty and the episode continues. Otherwise, `done=True` is set, the final DataFrame is captured from the named source, a schema-match heuristic reward is computed, and the grader engine is triggered server-side to produce the terminal score.
>
> **Required fields:**
>
> * `final_source` — the name of the DataFrame in `available_sources` that represents your cleaned output
>
> **Optional fields:** None
>
> **Reward (premature submission, step\_count \< 3):** `-0.15`, `done=False`
>
> **Reward (successful submission):** `0.2 × schema_match_ratio` where `schema_match_ratio = len([c for c in schema_target if c in final_df.columns]) / max(1, len(schema_target))`. Maximum possible step reward is `+0.20`.
>
> **Terminal behavior:** `done=True` is returned in the observation. The episode is over — no further `step()` calls will produce meaningful state changes.
>
> ```json theme={null}
> {
>   "action_type": "SUBMIT_PIPELINE",
>   "final_source": "web_forms"
> }
> ```
>
> For Task 2 or 3 where a merged output was produced:
>
> ```json theme={null}
> {
>   "action_type": "SUBMIT_PIPELINE",
>   "final_source": "merged_output"
> }
> ```
>
> **Example feedback (success):**
>
> ```
> Pipeline submitted with final source: merged_output.
> ```
>
> **Example feedback (premature):**
>
> ```
> Early submission blocked. You must perform at least 3 cleaning steps (Profile, Standardize, etc.) before submitting.
> ```
>
> > [!NOTE]
> >
> > The `final_source` you name must exist in `available_sources` at the time of submission. If the DataFrame does not exist, the grader receives an empty DataFrame and returns `0.01` (the minimum score floor).
>
> > [!WARNING]
> >
> > The schema-match heuristic reward (`0.2 × ratio`) only checks whether the **column names** in your final DataFrame match those in `schema_target`. It does not check values. A DataFrame with the right column names but all wrong values will earn `+0.20` in step reward but a near-zero grader score. The grader is the true measure of quality.

***

## Full Action Field Reference

The following table lists every field in `CRMPipelineAction` along with which action types require or use it.

| Field | Type | Required By | Description |
| - | - | - | - |
| `action_type` | `PipelineActionType` | **All** | The operation to execute |
| `source` | `Optional[str]` | VIEW\_SOURCE, PROFILE\_SOURCE, STANDARDIZE\_COLUMN, HANDLE\_MISSING, DEDUPLICATE, MERGE\_SOURCES | Primary DataFrame name |
| `source2` | `Optional[str]` | MERGE\_SOURCES | Secondary DataFrame for join |
| `column` | `Optional[str]` | STANDARDIZE\_COLUMN, HANDLE\_MISSING | Column to transform |
| `standardization_strategy` | `Optional[StandardizationStrategy]` | STANDARDIZE\_COLUMN | `TO_DATETIME_ISO`, `LOWERCASE_STRIP`, `EXTRACT_NUMBERS`, `PHONE_E164` |
| `missing_strategy` | `Optional[MissingStrategy]` | HANDLE\_MISSING | `FILL_VALUE` or `DROP_ROW` (implemented); `FILL_MEAN`, `FILL_MODE` (enum-only, return `-0.05`) |
| `deduplication_strategy` | `Optional[DeduplicationStrategy]` | DEDUPLICATE | `FUZZY_NAME_PHONE`, `EXACT_EMAIL` |
| `fallback_value` | `Optional[str]` | HANDLE\_MISSING (`FILL_VALUE` only) | Literal fill value for nulls |
| `join_key` | `Optional[str]` | MERGE\_SOURCES | Shared column to join on |
| `conflict_rule` | `Optional[ConflictRule]` | MERGE\_SOURCES | `PREFER_S1`, `PREFER_S2`, `COALESCE` |
| `final_source` | `Optional[str]` | SUBMIT\_PIPELINE | DataFrame name for grading |
| `query` | `Optional[str]` | EXECUTE\_SQL | Raw SQLite SELECT statement |
| `output_table` | `Optional[str]` | EXECUTE\_SQL, MERGE\_SOURCES | Target alias for result DataFrame |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.