Skip to main content
Standardization is the first concrete transformation step your pipeline should execute after profiling a source. The environment’s STANDARDIZE_COLUMN action accepts a source, a column, and a standardization_strategy, then applies a deterministic in-place mutation to every value in that column within the in-memory DataFrame. All four strategies are idempotent — running the same strategy twice on an already-clean column is safe. Each strategy was designed to undo a specific corruption pattern from the chaos engine, so choosing the right one for the right column is the difference between a clean merge and a silently broken join key.

The STANDARDIZE_COLUMN Action

To apply a standardization strategy, send an action payload with action_type set to STANDARDIZE_COLUMN along with the target source, column, and standardization_strategy:
A successful call returns a reward between 0.01 and 0.05 (proportional to how many dirty values were corrected) and sets last_action_feedback to:
Standardization mutates the DataFrame in place within the environment’s memory. You do not need to pass the data back — subsequent VIEW_SOURCE or EXECUTE_SQL calls will see the cleaned values immediately.

Strategy Reference

TO_DATETIME_ISO

TO_DATETIME_ISO normalizes date strings into ISO 8601 format (YYYY-MM-DD). Under the hood the environment calls pd.to_datetime(col, errors='coerce') and then formats the result as YYYY-MM-DDT00:00:00, using an empty string "" for any value that could not be parsed.Designed for: signup_date

What it handles

JSON action payload

When to use it

Apply TO_DATETIME_ISO to signup_date in every source that passed through correlate_truth(). For Task 1 and Task 2 this means web_forms and legacy_db. Do not apply it to salesforce or web_leads in Task 3 — those sources do not have a signup_date column.
After running TO_DATETIME_ISO, use HANDLE_MISSING with FILL_VALUE (value "") or DROP_ROW on signup_date to handle the rows that produced empty strings. Whether to drop or fill depends on whether signup_date is required in your final output schema.

Column-to-Strategy Mapping


Cleaning phone numbers reliably requires three sequential steps. Run them in this order:
1

Strip extensions and formatting with EXTRACT_NUMBERS

This removes ext. NNN suffixes, strips parentheses and hyphens from US format numbers, and converts "Invalid Number" strings to "".
2

Enforce E.164 format with PHONE_E164

This prepends + to any digit-only string that is missing it, ensuring every valid phone number follows the +{country_code}{number} shape.
3

Handle remaining empty and null values

After the two standardization steps, values that were "Invalid Number", None, or unparseable will be empty strings "". Decide whether to drop those rows or fill them:
Or to drop rows with no phone:

Reward Feedback

The environment calculates a dense heuristic reward for every STANDARDIZE_COLUMN call based on how many values were in a dirty state before the transformation. The formula is:
This means a source with 80 % dirty values in a column gives a reward of approximately 0.01 + (0.80 × 0.1) = 0.09, capped at 0.05. A source that is already clean gives the minimum reward of 0.01. Use this signal as a heuristic indicator of how much corruption was present — though it does not reflect whether your cleaning strategy was semantically correct.