Skip to main content
The CRM Data Pipeline benchmark exposes three tasks of escalating complexity, each spawned fresh on every /reset call using Faker with a randomised seed. No two episodes share the same data — your agent must generalise a cleaning strategy, not memorise a specific dataset. The difficulty progression mirrors real enterprise data engineering escalation: a single messy source, then two sources with conflicting records, then a three-way merge with divergent schemas and per-column conflict resolution rules.
Calling SUBMIT_PIPELINE before completing at least 3 action steps in any task incurs a hard penalty of -0.15 and the episode does not end. The minimum-steps gate (MIN_STEPS_BEFORE_SUBMIT = 3) is enforced in server/environment.py regardless of how clean your data already is. Always run at least one PROFILE_SOURCE and one STANDARDIZE_COLUMN before submitting.

Task Comparison at a Glance

Task 1: Web Forms Normalization

Task 1 confronts your agent with a single source — web_forms — that has been heavily corrupted by the correlate_truth() function in server/data_generator.py. The source starts with 40 real customer rows whose ground-truth values are clean, ISO-formatted, E.164-formatted, and lowercase. Before your agent sees them, correlate_truth() runs every row through a randomised corruption pipeline and appends 8 bot rows. Your agent must undo all of that corruption.

Input Data Characteristics

The web_forms DataFrame your agent receives contains approximately 40 real rows (some will have fuzzy duplicates appended, making the raw row count higher) plus 8 synthetic bot rows injected at the end and then shuffled. The following corruptions are applied stochastically per row:Missing Data (20% probability per field)
Approximately 20% of rows will have email set to None and independently ~20% will have phone set to None.String Chaos (50% probability per field)
Names gain leading and trailing whitespace, or are uppercased entirely. Emails are uppercased in roughly half of all rows.Date Chaos (tiered random)Phone Chaos (tiered random, only when phone is non-null)Fuzzy Duplicate Injection (25% probability per row)After corrupting a row, correlate_truth() rolls a 25% chance to append a near-duplicate: the last name is kept, the first name is replaced with a random Faker first name, and the email field is set to None. These fuzzy duplicates test your agent’s DEDUPLICATE action.

Bot Row Schema

Bot domains are: promo-alerts.net, click-rewards.xyz, freewinners.info, bulk-mail.co.

Ground Truth Target Schema

The grader compares each column by lowercasing and stripping both the truth value and your submitted value before checking equality, so case differences in name will not fail rows. However, signup_date and phone must match the exact format — 2023/01/15 will not match 2023-01-15T00:00:00.

Scoring

Your final score is computed by evaluate_dataframes(truth_df, agent_df) in server/graders.py:
Leaving even one bot row in your submission costs you 0.15 of score. Failing to deduplicate fuzzy rows means extra rows are penalised.
1

Profile the source

Earns +0.02. The quality report will show null counts per column and data types, giving your agent the context it needs to prioritise transformations.
2

Standardize email (lowercase + strip)

Calls df["email"].str.lower().str.strip(). Earns up to +0.05 based on how many values were uppercase.
3

Standardize name (strip whitespace)

4

Standardize phone (extract to E.164)

The EXTRACT_NUMBERS handler strips extension suffixes, removes all non-digit characters, and prepends +. An all-zeros phone (+00000000000) from a bot row will survive this step but will be filtered in the SQL step.
5

Standardize signup_date to ISO 8601

Uses pd.to_datetime(errors='coerce') — invalid dates like ??/??/???? and 2023-13-12 become NaT and are then written as "".
6

Deduplicate by email

Calls df.drop_duplicates(subset=["email"]). Fuzzy duplicates that have email = None are retained but will not match real customers in the grader join. Filter them next.
7

Filter bots and nulls via SQL

Earns +0.10. This is your bot-removal step. Alternatively, filter on customer_id NOT LIKE 'BOT%'.
8

Submit the pipeline

Earns 0.2 × schema_match_ratio as a step reward. The full terminal score comes from the grader.