> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Generator: How Messy CRM Data Is Procedurally Created

> Deep-dive into the data_generator.py chaos engine — how base truth is generated, corrupted, and used to create realistic messy CRM benchmark datasets.

The `data_generator.py` module is the core of the benchmark's difficulty system. Every time the environment resets, it procedurally generates a fresh batch of messy CRM data by following a strict four-phase pipeline: it creates clean ground-truth records, feeds them through a chaos engine that corrupts every column, injects realistic-looking bot rows, and then shuffles the combined result into a random order. Because this pipeline runs at reset time with unseeded randomness (`Faker.seed(None)`, `random.seed(None)`), no two episodes are identical — your agent must handle the full distribution of corruption, not a fixed dataset.

***

## Generation Pipeline Overview

Every episode follows the same four-phase pipeline. The diagram below shows how clean ground-truth rows are corrupted, augmented with bots, and exposed to the agent — while the original clean rows are stored separately for grading.

```mermaid theme={null}
flowchart TD
    A["create_base_truth(N)\nclean rows: customer_id, name,\nemail, signup_date, phone"] --> B["correlate_truth()\nPhase 2: chaos engine"]
    A --> T["GLOBAL_TRUTH_STORE\n(hidden — grader only)"]

    subgraph chaos ["Phase 2 — correlate_truth() chaos steps"]
        B --> C1["Step 1: Missing Data\nemail=None 20%\nphone=None 20%"]
        C1 --> C2["Step 2: String Chaos\nname: padded or UPPER 50%\nemail: UPPER 50%"]
        C2 --> C3["Step 3: Date Chaos\n??/??/???? · 2023-13-12 · None · slash"]
        C3 --> C4["Step 4: Phone Chaos\next suffix · (xxx) · Invalid Number · raw digits"]
        C4 --> C5["Step 5: Fuzzy Dup Injection\n25% chance → extra row, email=None"]
    end

    C5 --> D["messy_rows list"]
    B2["generate_bot_row() × K\nBOT prefix · epoch date · +00000000000"] --> D

    D --> E["random.shuffle()"]
    E --> F["pd.DataFrame()\nstored in available_sources\nagent reads this"]

    F -->|"SUBMIT_PIPELINE"| G["Grader: compare\nfinal_df vs GLOBAL_TRUTH_STORE"]
    T --> G
```

The agent **never has direct access** to `GLOBAL_TRUTH_STORE`. It can only interact with the messy DataFrames exposed through `available_sources` in the observation. The grader compares your submitted `merged_output` (or final source) against the hidden truth snapshot to compute accuracy.

***

## Phase 1 — create\_base\_truth(size)

`create_base_truth(size)` generates `size` clean, fully-populated rows using the Faker library. This is the pristine ground truth before any corruption is applied.

```python theme={null}
def create_base_truth(size=50):
    truth = []
    for i in range(size):
        truth.append({
            "customer_id": f"CUST_{1000 + i}",   # e.g. CUST_1000, CUST_1001 ...
            "name":        fake.name(),             # e.g. "Jane Doe"
            "email":       fake.email().lower(),    # e.g. "jane.doe@example.com"
            "signup_date": fake.date_this_decade().isoformat(),  # e.g. "2022-07-14"
            "phone":       "+" + fake.msisdn()      # e.g. "+15551234567" (E.164)
        })
    return truth
```

Every field is fully populated. Emails are already lowercased. Phones already carry the `+` E.164 prefix. Dates are valid ISO 8601. This is the ideal target schema your pipeline should converge toward.

### Base truth row counts by task

| Task | `create_base_truth` size |
| - | - |
| `t1` | 40 rows |
| `t2` | 40 rows (from t1) + 20 additional rows for `legacy_db` |
| `t3` | 60 rows |

***

## Phase 2 — correlate\_truth() — The Chaos Engine

`correlate_truth(truth_list)` iterates over every clean row and applies five independent corruption steps. Each step is probabilistic and operates on a different column family. Understanding the exact probability thresholds and mutation shapes is essential for writing the correct cleaning strategy.

```python theme={null}
def correlate_truth(truth_list):
    messy = []
    for r in truth_list:
        dirty = dict(r)
        # Steps 1–5 applied here ...
        messy.append(dirty)
        # Step 5 may also append a second fuzzy-duplicate row
    return messy
```

### Step 1 — Missing Data Chaos

Two independent coin flips remove the email and phone values.

```python theme={null}
if random.random() > 0.8: dirty["email"] = None   # 20% chance
if random.random() > 0.8: dirty["phone"] = None   # 20% chance
```

On average, **1 in 5 rows** will have no email and **1 in 5** will have no phone. These two events are independent, so roughly 4 % of rows lose both. Rows that lose their email become effectively un-joinable unless a fuzzy name/phone strategy is used.

### Step 2 — String Chaos

Name and email receive casing and whitespace mutations separately:

```python theme={null}
# Name: 50% whitespace padding OR 50% full uppercase
if dirty["name"]:
    dirty["name"] = f"  {dirty['name']}  " if random.random() > 0.5 \
                    else dirty["name"].upper()

# Email (if not already None): 50% full uppercase
if dirty["email"]:
    dirty["email"] = dirty["email"].upper() if random.random() > 0.5 \
                     else dirty["email"]
```

A name like `"Jane Doe"` becomes either `"  Jane Doe  "` (whitespace) or `"JANE DOE"` (uppercase). An email like `"jane.doe@gmail.com"` becomes `"JANE.DOE@GMAIL.COM"` with 50 % probability.

**Cleaning prescription:** Apply `LOWERCASE_STRIP` to `email` and `name` before any join operation.

### Step 3 — Date Chaos (Five Branches)

`signup_date` is replaced according to a single random draw against four nested thresholds:

```python theme={null}
date_rand = random.random()
if   date_rand > 0.85: dirty["signup_date"] = "??/??/????"     # 15% — unparseable
elif date_rand > 0.70: dirty["signup_date"] = "2023-13-12"     # 15% — invalid month 13
elif date_rand > 0.55: dirty["signup_date"] = None             # 15% — missing entirely
else:                  dirty["signup_date"] = r["signup_date"][:10].replace("-", "/")
#                                                               # 55% — slash-separated
```

The probability breakdown across all rows in a source:

| Outcome | Approximate Frequency | Example Value |
| - | - | - |
| Completely unparseable | \~15 % | `"??/??/????"` |
| Invalid calendar date | \~15 % | `"2023-13-12"` |
| Missing (`None`) | \~15 % | `None` |
| Slash-separated (parseable) | \~55 % | `"2023/07/14"` |

> [!NOTE]
>
> The `TO_DATETIME_ISO` standardization strategy handles all four cases automatically. Parseable slash-separated dates are converted to ISO 8601. The three invalid/missing cases are mapped to an empty string `""`.

### Step 4 — Phone Chaos (Four Branches)

Phone mutation only applies to rows where `dirty["phone"]` is not `None` after Step 1. The raw digits are first extracted by stripping the `+` prefix, then one of three mutations is applied:

```python theme={null}
if dirty["phone"]:
    raw_phone = dirty["phone"].replace("+", "")   # e.g. "15551234567"
    prand = random.random()
    if   prand > 0.85: dirty["phone"] = raw_phone + " ext. " + str(random.randint(10, 999))
    elif prand > 0.70: dirty["phone"] = f"({raw_phone[:3]}) {raw_phone[3:6]}-{raw_phone[6:]}"
    elif prand > 0.55: dirty["phone"] = "Invalid Number"
    # else: phone stays as raw_phone (no + prefix) — ~55% of non-None phones
```

| Probability Range | Mutation | Example Output |
| - | - | - |
| `prand > 0.85` (\~15 %) | Extension suffix | `"15551234567 ext. 42"` |
| `0.70 < prand ≤ 0.85` (\~15 %) | US parenthetical format | `"(155) 512-34567"` |
| `0.55 < prand ≤ 0.70` (\~15 %) | Literal junk string | `"Invalid Number"` |
| `prand ≤ 0.55` (\~55 %) | Digits only, no `+` | `"15551234567"` |

**Cleaning prescription:** Use `EXTRACT_NUMBERS` first (strips `ext.` and parenthetical formatting, but note the environment's implementation also handles the `ext` pattern), then apply `PHONE_E164` to re-attach the `+` prefix.

### Step 5 — Deduplication Chaos (Fuzzy Duplicate Injection)

After appending the corrupted row to `messy`, the engine has a **25 % chance** of also appending a second, fuzzy-duplicate row:

```python theme={null}
if random.random() > 0.75:          # 25% chance
    dup = dict(dirty)
    # Change first name to a different random first name, keep last name
    if r["name"] and len(r["name"].split()) > 1:
        dup["name"] = fake.first_name() + " " + r["name"].split()[-1]
    dup["email"] = None             # Duplicate has no email — harder to deduplicate
    messy.append(dup)
```

The duplicate row deliberately has `email = None`, which makes `EXACT_EMAIL` deduplication alone insufficient. The duplicate is detectable by matching the last name plus phone number — exactly what `FUZZY_NAME_PHONE` is designed for. Across a 40-row truth list, you can expect roughly **10 extra fuzzy-duplicate rows** after this step.

***

## Phase 3 — generate\_bot\_row()

Bot rows are injected after `correlate_truth()` completes. Each bot row is constructed with fixed structural anomalies:

```python theme={null}
def generate_bot_row():
    local  = fake.user_name() + str(random.randint(1, 99))   # e.g. "john42"
    domain = random.choice(_BOT_DOMAINS)
    return {
        "customer_id": f"BOT{random.randint(10000, 99999)}",  # BOT prefix
        "name":        random.choice(_BOT_NAMES),              # fixed list
        "email":       f"{local}@{domain}",                   # bulk-mail domain
        "signup_date": "1970-01-01",                          # Unix epoch
        "phone":       "+" + "0" * 11                         # +00000000000
    }
```

### Bot constants

```python theme={null}
_BOT_DOMAINS = ["promo-alerts.net", "click-rewards.xyz", "freewinners.info", "bulk-mail.co"]
_BOT_NAMES   = ["Support Team", "Newsletter Bot", "Promo Sender", "Deals Alert"]
```

Bot rows do **not** go through `correlate_truth()`. They are appended directly to the messy list before the shuffle.

***

## Phase 4 — Shuffle and DataFrame Construction

After all rows (corrupted real + bots) are assembled, the list is shuffled uniformly at random and converted to a Pandas DataFrame:

```python theme={null}
random.shuffle(messy_rows)
pd.DataFrame(messy_rows)
```

The shuffle ensures that bot rows and duplicate rows are not conveniently positioned at the end of the DataFrame — your agent must handle them appearing anywhere.

***

## Task-Specific Generation Flows

> [tab] Task 1 — Easy
>
> ```
> create_base_truth(40)
>         ↓
> correlate_truth()          ← ~10 fuzzy-dup rows added here
>         ↓
> + 8 × generate_bot_row()
>         ↓
> random.shuffle()
>         ↓
> {"web_forms": DataFrame}
> ```
>
> **Hidden truth:** `{"web_forms": DataFrame(original 40 clean rows)}`
>
> Expected messy DataFrame size before cleaning: **\~58 rows** (40 base + \~10 fuzzy duplicates + 8 bots).

> [tab] Task 2 — Medium
>
> ```
> generate_easy_task()                      ← same as t1
>         ↓
> create_base_truth(20)                     ← second independent truth batch
>         ↓
> correlate_truth()                         ← ~5 fuzzy-dup rows
>         ↓
> + 5 × generate_bot_row()
>         ↓
> random.shuffle()
>         ↓
> {"web_forms": ..., "legacy_db": DataFrame}
> ```
>
> **Hidden truth:** `{"merged_output": concat(web_forms_truth, legacy_truth).drop_duplicates(subset=["email"])}`
>
> The two truth batches are deduped on `email` before being stored, so the grader expects a merged result without any cross-source duplicates.

> [tab] Task 3 — Hard
>
> ```
> create_base_truth(60)
>         ↓
> ┌───────────────────────────────────────────────────┐
> │  For each truth row:                              │
> │   salesforce ← {customer_id, email, sf_phone}    │  (70% have phone)
> │   web_leads  ← {wl_cid?, email, fresh_phone?}    │  (60% have phone)
> │   legacy     ← {OLD-NNNN, UPPERCASE_EMAIL, phone} │
> └───────────────────────────────────────────────────┘
>         ↓ sample fractions
> df_sf  = salesforce.sample(frac=0.90)
> df_wl  = web_leads.sample(frac=0.80)
> df_leg = legacy.sample(frac=0.85)
>         ↓ inject 5 bots per source
> + 5 × bot into each of the 3 DataFrames
>         ↓ shuffle each independently
> {"salesforce": ..., "web_leads": ..., "legacy_db": ...}
> ```
>
> **Hidden truth:** `{"merged_output": resolved per CONFLICT_RULES}`
>
> * `customer_id` → from `salesforce` (prefer\_salesforce)
> * `email` → from `salesforce` (prefer\_salesforce)
> * `phone` → `web_leads` fresh phone when available, else `salesforce` phone (prefer\_web\_leads)

***

## Hidden Truth vs. Messy Input

The ground truth is captured **before** `correlate_truth()` runs and stored in `GLOBAL_TRUTH_STORE` keyed by the episode UUID. Your agent has no direct access to it. The only signal you receive is the reward feedback from the grader when you submit.

```
GLOBAL_TRUTH_STORE[episode_id] ← immutable snapshot of clean rows
                                   captured at reset time
                                   
self._sources[source_name]     ← mutable messy DataFrames
                                   your agent reads and transforms these
```

When you call `SUBMIT_PIPELINE`, the environment retrieves your designated `final_source` DataFrame and compares it row-by-row against the hidden truth. Your cleaning pipeline must reconstruct the truth as closely as possible purely from the observable messy data.

> [!WARNING]
>
> Because randomness is unseeded at the start of each episode, you cannot hard-code expected row counts or specific corruption values. Your pipeline logic must be robust to the full probability distribution described above.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.