Skip to main content
Every episode in the CRM Data Pipeline Environment exposes one or more named data sources through the available_sources field of the observation. Each source is a distinct in-memory Pandas DataFrame with its own column schema, data quality characteristics, and corruption patterns introduced by the chaos engine. Understanding which sources appear in which tasks — and exactly how their data is distorted — is the prerequisite for building a reliable cleaning pipeline.

Source Availability by Task

The three task tiers progressively introduce more sources and more complex inter-source conflicts. Use this table as a quick reference when deciding which sources to load and clean for a given episode.
At the start of each episode the environment sets available_sources in the observation to exactly these source names. Pass the exact name string (e.g., "web_forms") as the source field in every action payload.

web_forms

web_forms is the primary source in Tasks 1 and 2. It is produced by running create_base_truth() through the full correlate_truth() chaos engine, then appending synthetic bot rows. Expect severe corruption across every column.

Schema

Date Chaos — signup_date

The chaos engine applies one of four mutations to every signup_date value based on a uniform random draw:
The TO_DATETIME_ISO standardization strategy handles all four cases: it parses valid formats to YYYY-MM-DD, and maps "??/??/????", invalid months, and None to an empty string "".

Phone Chaos — phone

When a phone value is present (80 % of rows), the engine applies one of three mutations:

salesforce

salesforce appears only in Task 3. It is the canonical source for email and customer_id according to the task’s CONFLICT_RULES. The data generator samples 90 % of the base truth rows into Salesforce, so up to 10 % of real customers may be absent from this source.

Schema

salesforce does not go through correlate_truth(). Its data is cleaner than web_forms, but it still contains bot rows and can have None phone values.

legacy_db

legacy_db appears in both Task 2 and Task 3, but its schema differs between the two tasks.

Task 2 Schema

In Task 2, legacy_db is generated from a second independent call to create_base_truth(20) followed by the full correlate_truth() chaos engine. Its schema is identical to web_forms:

Task 3 Schema

In Task 3, legacy_db is generated directly from the base truth using a different mapping that preserves old enterprise conventions. It is sampled at 85 % of the 60 base rows.
In Task 3, legacy_db has no customer_id column — only legacy_id. You cannot join it to other sources on customer_id directly. Join via LOWER(contact_email) matching against salesforce.email or web_leads.email instead.

web_leads

web_leads appears only in Task 3. It contains fresher phone data than Salesforce, making it the preferred source for the phone column per CONFLICT_RULES. The generator samples 80 % of base truth rows, so 20 % of customers may be absent.

Schema

The CONFLICT_RULES for Task 3 set "phone": "prefer_web_leads". When building your SQL merge, use COALESCE(web_leads.phone, salesforce.phone) to implement this rule — web_leads phone takes precedence, falling back to Salesforce when the web_leads value is missing.

Bot Injection Patterns

Both the easy/medium tasks and the hard task inject synthetic bot rows into every source. Bots are designed to look plausible at a glance — valid email format, reasonable-looking names — but carry several identifying signals.

Bot Identifiers

Bot Counts per Task

Filtering Bots in SQL

Use the known bot domains in a WHERE clause to exclude bot rows during SQL-based merges:
For legacy_db in Task 3 where emails are uppercase, apply LOWER() first:
You can also filter bots using customer_id LIKE 'BOT%' or phone = '+00000000000' or signup_date = '1970-01-01'. Combining multiple signals in a single WHERE clause produces the most robust removal.