correlate_truth() function appends a fuzzy-duplicate row with a 25 % probability for every base truth record — which means a 40-row truth list can produce roughly 10 additional duplicate rows before bot injection even begins. These duplicates have subtly different names but the same last name, and deliberately have email = None to resist naive email-matching. If you submit without deduplicating, the grader penalizes you proportionally for every extra row beyond the expected truth count. Eliminating duplicates early and cleanly is essential to maximizing your episode score.
Why Duplicates Are Hard to Remove
The chaos engine constructs duplicates specifically to defeat simple deduplication:- The duplicate row has a different first name (generated by
fake.first_name()), so exact-name matching fails. - The duplicate row has
email = None, so email-based deduplication cannot match it to the original. - The duplicate inherits all other corruption from the original row — same uppercase casing, same whitespace, same phone chaos — making it visually indistinguishable from a legitimate record.
- Last name (shared between original and duplicate)
- Phone number (shared when both rows had a non-None phone after Step 1 chaos)
FUZZY_NAME_PHONE strategy exists.
The DEDUPLICATE Action
Send aDEDUPLICATE action with a source and deduplication_strategy:
VIEW_SOURCE on the same source will reflect the reduced row count.
EXACT_EMAIL
EXACT_EMAIL calls DataFrame.drop_duplicates(subset=["email"]) on the target source. It keeps the first occurrence of each email value and drops all subsequent rows with the same email.
Behavior
- Match criterion: Two rows are duplicates if and only if their
emailvalues are identical strings. - Null handling: Rows with
NoneorNaNin theemailcolumn are not considered duplicates of each other — Pandas’drop_duplicatestreatsNaNas unique. AllNone-email rows are kept. - Order sensitivity: Keeps the first occurrence (lowest integer index in the DataFrame). Because the DataFrame is shuffled at reset time, “first” is effectively random with respect to the original truth order.
JSON Action
Expected Feedback
N is the number of rows dropped.
Reward
The reward is proportional to how many rows were removed:(10/50) × 0.2 = 0.04.
When to Use It
- Task 1
- Task 2
- Task 3
EXACT_EMAIL is your primary deduplication tool in Task 1. After standardizing email with LOWERCASE_STRIP, all casing inconsistencies are resolved, making exact matching reliable. Run the sequence:STANDARDIZE_COLUMN→email→LOWERCASE_STRIPDEDUPLICATE→EXACT_EMAILHANDLE_MISSING→email→DROP_ROW(remove bot rows and None-email duplicates)
The 25 % fuzzy duplicates have
email = None, so EXACT_EMAIL alone does not remove them. You will still have those duplicate rows after this step. Follow up with FUZZY_NAME_PHONE to catch the remaining ones.FUZZY_NAME_PHONE
FUZZY_NAME_PHONE is a name-and-phone-based deduplication strategy designed to catch the chaos engine’s fuzzy duplicates that evade EXACT_EMAIL. The environment implements it by:
- Creating a temporary
_tmp_namecolumn:name.astype(str).str.lower().str.strip() - Creating a temporary
_tmp_phonecolumn:phone.astype(str).str.replace(r'\D+', '', regex=True)(digits only) - Calling
drop_duplicates(subset=["_tmp_name", "_tmp_phone"]) - Dropping the two temporary columns
Behavior
- Match criterion: Two rows are duplicates if their normalized (lowercase, stripped) name and their digit-only phone are both identical.
- Name normalization: Lowercases and strips whitespace. This means
" JANE DOE "and"jane doe"are treated as the same name. - Phone normalization: Strips all non-digits.
"+15551234567","15551234567", and"(155) 512-34567"all normalize to"15551234567". - Limitation: Because duplicates have a different first name, they will not match on full normalized name unless the last-name-only signal happens to match after normalization. For best results, apply
LOWERCASE_STRIPtonamebefore runningFUZZY_NAME_PHONE.
JSON Action
When to Use It
UseFUZZY_NAME_PHONE as a second deduplication pass after EXACT_EMAIL, targeting rows that had email = None in the chaos engine’s fuzzy duplicates. It is also useful when email data is heavily missing (>30 % null) and you need an alternative matching signal.
Recommended Deduplication Order
1
Standardize email first
Always run
LOWERCASE_STRIP on email before any deduplication. Without it, "JANE.DOE@GMAIL.COM" and "jane.doe@gmail.com" are treated as different emails, and EXACT_EMAIL misses the duplicate entirely.2
Run EXACT_EMAIL deduplication
Remove all rows where the normalized email is an exact duplicate of an earlier row.
3
Standardize name before fuzzy dedup
Normalize names to lowercase/stripped before running fuzzy matching so the temporary
_tmp_name comparison works correctly.4
Run FUZZY_NAME_PHONE for residual duplicates
Catch the chaos engine’s fuzzy duplicates (different first name,
email = None) that survived the EXACT_EMAIL pass.5
Drop remaining None-email rows
After deduplication, rows with
email = None that were not matched as duplicates are likely either bot rows (which should be removed by other means) or legitimate customers with missing emails. Decide whether to drop or fill based on your task’s schema requirements.Grader Penalty for Extra Rows
The grader measures the accuracy of your submitted DataFrame against the hidden truth. Extra rows — rows in your submission that have no corresponding truth record — subtract from your score. The penalty formula is:18 × 0.15 = 2.7 points — a substantial fraction of the total possible score.