> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CRM Deduplication Strategies: Exact and Fuzzy Matching

> Use EXACT_EMAIL and FUZZY_NAME_PHONE deduplication strategies to eliminate duplicate CRM records inserted by the data generator's chaos engine.

Deduplication is one of the highest-impact steps in your CRM pipeline. The chaos engine's `correlate_truth()` function appends a fuzzy-duplicate row with a 25 % probability for every base truth record — which means a 40-row truth list can produce roughly **10 additional duplicate rows** before bot injection even begins. These duplicates have subtly different names but the same last name, and deliberately have `email = None` to resist naive email-matching. If you submit without deduplicating, the grader penalizes you proportionally for every extra row beyond the expected truth count. Eliminating duplicates early and cleanly is essential to maximizing your episode score.

***

## Why Duplicates Are Hard to Remove

The chaos engine constructs duplicates specifically to defeat simple deduplication:

1. The duplicate row has a **different first name** (generated by `fake.first_name()`), so exact-name matching fails.
2. The duplicate row has `email = None`, so email-based deduplication cannot match it to the original.
3. The duplicate inherits all other corruption from the original row — same uppercase casing, same whitespace, same phone chaos — making it visually indistinguishable from a legitimate record.

The only reliable signals available for matching these duplicates are:

* **Last name** (shared between original and duplicate)
* **Phone number** (shared when both rows had a non-None phone after Step 1 chaos)

This is precisely why the `FUZZY_NAME_PHONE` strategy exists.

***

## The DEDUPLICATE Action

Send a `DEDUPLICATE` action with a `source` and `deduplication_strategy`:

```json theme={null}
{
  "action_type": "DEDUPLICATE",
  "source": "<source_name>",
  "deduplication_strategy": "<STRATEGY>"
}
```

The environment mutates the source DataFrame in place, removing duplicate rows, and returns a feedback string and a proportional reward. After deduplication, `VIEW_SOURCE` on the same source will reflect the reduced row count.

***

## EXACT\_EMAIL

`EXACT_EMAIL` calls `DataFrame.drop_duplicates(subset=["email"])` on the target source. It keeps the first occurrence of each email value and drops all subsequent rows with the same email.

### Behavior

* **Match criterion:** Two rows are duplicates if and only if their `email` values are identical strings.
* **Null handling:** Rows with `None` or `NaN` in the `email` column are **not** considered duplicates of each other — Pandas' `drop_duplicates` treats `NaN` as unique. All `None`-email rows are kept.
* **Order sensitivity:** Keeps the first occurrence (lowest integer index in the DataFrame). Because the DataFrame is shuffled at reset time, "first" is effectively random with respect to the original truth order.

### JSON Action

```json theme={null}
{
  "action_type": "DEDUPLICATE",
  "source": "web_forms",
  "deduplication_strategy": "EXACT_EMAIL"
}
```

### Expected Feedback

```
"Deduplicated web_forms, removed N rows."
```

where `N` is the number of rows dropped.

### Reward

The reward is proportional to how many rows were removed:

```
reward = (removed_rows / original_row_count) × 0.2
```

A source with 10 duplicates out of 50 rows returns `(10/50) × 0.2 = 0.04`.

### When to Use It

> [tab] Task 1
>
> `EXACT_EMAIL` is your primary deduplication tool in Task 1. After standardizing `email` with `LOWERCASE_STRIP`, all casing inconsistencies are resolved, making exact matching reliable. Run the sequence:
>
> 1. `STANDARDIZE_COLUMN` → `email` → `LOWERCASE_STRIP`
>
> 2. `DEDUPLICATE` → `EXACT_EMAIL`
>
> 3. `HANDLE_MISSING` → `email` → `DROP_ROW` (remove bot rows and None-email duplicates)
>
> > [!NOTE]
> >
> > The 25 % fuzzy duplicates have `email = None`, so `EXACT_EMAIL` alone does not remove them. You will still have those duplicate rows after this step. Follow up with `FUZZY_NAME_PHONE` to catch the remaining ones.

> [tab] Task 2
>
> In Task 2 you need to deduplicate both `web_forms` and `legacy_db` independently before merging. Run `EXACT_EMAIL` on both sources after email standardization. The hidden truth for Task 2 is `drop_duplicates(subset=["email"])` applied to the concatenated truth lists — so exact email deduplication directly matches what the grader expects.

> [tab] Task 3
>
> In Task 3, `EXACT_EMAIL` is less relevant because the three sources use different schemas and you are joining rather than deduplicating within a single source. However, if you use SQL to build `merged_output`, you may still see duplicate rows if the same customer appears in `salesforce` multiple times (unlikely but possible). After your merge SQL, consider running `EXACT_EMAIL` on `merged_output` as a final safety pass.

***

## FUZZY\_NAME\_PHONE

`FUZZY_NAME_PHONE` is a name-and-phone-based deduplication strategy designed to catch the chaos engine's fuzzy duplicates that evade `EXACT_EMAIL`. The environment implements it by:

1. Creating a temporary `_tmp_name` column: `name.astype(str).str.lower().str.strip()`
2. Creating a temporary `_tmp_phone` column: `phone.astype(str).str.replace(r'\D+', '', regex=True)` (digits only)
3. Calling `drop_duplicates(subset=["_tmp_name", "_tmp_phone"])`
4. Dropping the two temporary columns

### Behavior

* **Match criterion:** Two rows are duplicates if their normalized (lowercase, stripped) name **and** their digit-only phone are both identical.
* **Name normalization:** Lowercases and strips whitespace. This means `"  JANE DOE  "` and `"jane doe"` are treated as the same name.
* **Phone normalization:** Strips all non-digits. `"+15551234567"`, `"15551234567"`, and `"(155) 512-34567"` all normalize to `"15551234567"`.
* **Limitation:** Because duplicates have a **different first name**, they will not match on full normalized name unless the last-name-only signal happens to match after normalization. For best results, apply `LOWERCASE_STRIP` to `name` before running `FUZZY_NAME_PHONE`.

### JSON Action

```json theme={null}
{
  "action_type": "DEDUPLICATE",
  "source": "legacy_db",
  "deduplication_strategy": "FUZZY_NAME_PHONE"
}
```

### When to Use It

Use `FUZZY_NAME_PHONE` as a second deduplication pass after `EXACT_EMAIL`, targeting rows that had `email = None` in the chaos engine's fuzzy duplicates. It is also useful when email data is heavily missing (>30 % null) and you need an alternative matching signal.

> [!WARNING]
>
> `FUZZY_NAME_PHONE` can produce false positives if two distinct customers happen to share the same normalized name and the same phone number (e.g., a father/son with the same last name and shared household phone). Use it selectively on sources where you have strong reason to believe duplicates exist, and verify the row count afterward with `VIEW_SOURCE`.

***

## Recommended Deduplication Order

**1. Standardize email first**

Always run `LOWERCASE_STRIP` on `email` before any deduplication. Without it, `"JANE.DOE@GMAIL.COM"` and `"jane.doe@gmail.com"` are treated as different emails, and `EXACT_EMAIL` misses the duplicate entirely.

```json theme={null}
{
  "action_type": "STANDARDIZE_COLUMN",
  "source": "web_forms",
  "column": "email",
  "standardization_strategy": "LOWERCASE_STRIP"
}
```

**2. Run EXACT\_EMAIL deduplication**

Remove all rows where the normalized email is an exact duplicate of an earlier row.

```json theme={null}
{
  "action_type": "DEDUPLICATE",
  "source": "web_forms",
  "deduplication_strategy": "EXACT_EMAIL"
}
```

**3. Standardize name before fuzzy dedup**

Normalize names to lowercase/stripped before running fuzzy matching so the temporary `_tmp_name` comparison works correctly.

```json theme={null}
{
  "action_type": "STANDARDIZE_COLUMN",
  "source": "web_forms",
  "column": "name",
  "standardization_strategy": "LOWERCASE_STRIP"
}
```

**4. Run FUZZY\_NAME\_PHONE for residual duplicates**

Catch the chaos engine's fuzzy duplicates (different first name, `email = None`) that survived the `EXACT_EMAIL` pass.

```json theme={null}
{
  "action_type": "DEDUPLICATE",
  "source": "web_forms",
  "deduplication_strategy": "FUZZY_NAME_PHONE"
}
```

**5. Drop remaining None-email rows**

After deduplication, rows with `email = None` that were not matched as duplicates are likely either bot rows (which should be removed by other means) or legitimate customers with missing emails. Decide whether to drop or fill based on your task's schema requirements.

```json theme={null}
{
  "action_type": "HANDLE_MISSING",
  "source": "web_forms",
  "column": "email",
  "missing_strategy": "DROP_ROW"
}
```

***

## Grader Penalty for Extra Rows

The grader measures the accuracy of your submitted DataFrame against the hidden truth. Extra rows — rows in your submission that have no corresponding truth record — subtract from your score. The penalty formula is:

```
extra_row_penalty = max(0, len(final_df) - num_truth_rows) × 0.15
```

This means every single extra row beyond the expected truth count costs **0.15 points**. With 10 fuzzy duplicates and 8 bot rows in a Task 1 dataset (18 total extra rows), failing to deduplicate and remove bots costs `18 × 0.15 = 2.7 points` — a substantial fraction of the total possible score.

> [!WARNING]
>
> The extra-row penalty is asymmetric: submitting too many rows is penalized at 0.15 per row, but removing too many real rows (under-counting) affects your recall metric separately. The optimal strategy is precision-first deduplication followed by careful bot removal, not aggressive `DROP_ROW` on all uncertain records.

***

## Deduplication Strategy Quick Reference

| Strategy | Match Criterion | Null Handling | Best For |
| - | - | - | - |
| `EXACT_EMAIL` | Exact string equality on `email` | `None` emails are kept | Post-email-standardization cleanup in Tasks 1 and 2 |
| `FUZZY_NAME_PHONE` | Normalized name + digit-only phone | `None` phones normalize to `""` and match each other | Catching chaos-engine fuzzy duplicates with `email = None` |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.