> ## Documentation Index
> Fetch the complete documentation index at: https://ritika-fa61c43f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Running the Baseline Inference Script on CRM Tasks

> Execute the built-in inference.py script to run an LLM agent or rule-based baseline across all three CRM cleaning tasks and evaluate scores.

The `inference.py` script is the out-of-the-box evaluator bundled with the CRM Data Pipeline Environment. It drives an LLM-powered agent (or falls back to a deterministic rule-based baseline) through all three CRM cleaning tasks — `t1`, `t2`, and `t3` — in sequence, then prints a final score for each task plus a computed average. You do not need to write any agent code to run it; it works immediately once your environment variables are set and the server is running.

## Prerequisites

Before running inference, confirm the following are installed and available in your shell:

* **Python 3.12+** with `uv` (the project uses `uv` as the package runner)
* **`openenv-core`**: `pip install openenv-core`
* **`python-dotenv`**, **`openai`**, **`requests`**, **`pandas`** — all installed automatically via `uv`

The script reads credentials exclusively from environment variables. It never accepts hardcoded keys. Three variables control the LLM backend:

| Variable | Required | Default | Purpose |
| - | - | - | - |
| `HF_TOKEN` or `API_KEY` | Yes (for LLM mode) | — | Authentication token for the inference provider |
| `API_BASE_URL` | No | `https://router.huggingface.co/v1` | OpenAI-compatible endpoint base URL |
| `MODEL_NAME` | No | `Qwen/Qwen2.5-72B-Instruct` | Model identifier passed to the chat completions API |
| `OPENENV_BASE_URL` | No | `http://127.0.0.1:8080` | Address of the running CRM Pipeline server |

> [!NOTE]
>
> If neither `HF_TOKEN` nor `API_KEY` is set, the `openai.OpenAI` client is initialized with `api_key=None`. The script will still run — every LLM call will fail and the rule-based fallback will handle all steps automatically. This is intentional design: the fallback achieves the maximum grader score of `0.99` on all three tasks without any LLM.

## Running Inference

**1. Set environment variables**

Export your credentials in the terminal where you will run the script. The `API_BASE_URL` and `MODEL_NAME` values shown below are the defaults; you may omit them if you want to use Qwen2.5-72B via the HuggingFace Router.

```bash theme={null}
export HF_TOKEN=your_huggingface_token
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
```

Alternatively, create a `.env` file in the repo root — the script calls `load_dotenv()` at startup and will pick it up automatically without overriding any already-exported variables:

```bash theme={null}
# .env
HF_TOKEN=your_huggingface_token
API_BASE_URL=https://router.huggingface.co/v1
MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
```

**2. Start the FastAPI server**

Open a **second terminal** and launch the environment server. The server must be running before you start inference — it binds to `http://127.0.0.1:8080` by default.

```bash theme={null}
uv run -m server.app
```

You should see Uvicorn startup output confirming the server is listening. Keep this terminal open for the entire inference run. If you need a different port, set `OPENENV_BASE_URL` in the inference terminal:

```bash theme={null}
export OPENENV_BASE_URL=http://127.0.0.1:9090
uv run -m server.app --port 9090
```

**3. Run the inference script**

In your first terminal (with credentials exported), run:

```bash theme={null}
uv run python inference.py
```

The script iterates over `["t1", "t2", "t3"]`, runs one full episode for each task, then prints aggregate results. Total runtime with the rule-based fallback is typically under 10 seconds. With a live LLM, runtime depends on API latency and the number of retries.

**4. Inspect the output**

Inference emits structured log lines to stdout. A typical successful run looks like this:

```text theme={null}
[START] task=t1 env=crm_data_pipeline model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action={"action_type":"PROFILE_SOURCE","source":"web_forms"} reward=0.03 done=false error=null
[STEP] step=2 action={"action_type":"STANDARDIZE_COLUMN","source":"web_forms","column":"email","standardization_strategy":"LOWERCASE_STRIP"} reward=0.04 done=false error=null
[STEP] step=3 action={"action_type":"STANDARDIZE_COLUMN","source":"web_forms","column":"name","standardization_strategy":"LOWERCASE_STRIP"} reward=0.04 done=false error=null
[STEP] step=4 action={"action_type":"STANDARDIZE_COLUMN","source":"web_forms","column":"phone","standardization_strategy":"EXTRACT_NUMBERS"} reward=0.04 done=false error=null
[STEP] step=5 action={"action_type":"STANDARDIZE_COLUMN","source":"web_forms","column":"signup_date","standardization_strategy":"TO_DATETIME_ISO"} reward=0.04 done=false error=null
[STEP] step=6 action={"action_type":"DEDUPLICATE","source":"web_forms","deduplication_strategy":"EXACT_EMAIL"} reward=0.05 done=false error=null
[STEP] step=7 action={"action_type":"EXECUTE_SQL","query":"SELECT * FROM web_forms WHERE customer_id != '???' AND email NOT LIKE '%bot%'","output_table":"web_forms_clean"} reward=0.04 done=false error=null
[STEP] step=8 action={"action_type":"SUBMIT_PIPELINE","final_source":"web_forms_clean"} reward=0.99 done=true error=null
[END] success=true steps=8 score=0.990 rewards=0.03,0.04,0.04,0.04,0.04,0.05,0.04,0.99
[DEBUG] Final Score [t1]: 0.9900
...
[DEBUG] === Final Scores ===
[DEBUG] {"t1": 0.99, "t2": 0.99, "t3": 0.99, "average": 0.99}
```

The final JSON line is your benchmark result. A score of `0.99` is the maximum the grader awards — it means every row in your submitted DataFrame matched the hidden ground truth exactly.

## Understanding the Log Format

Every line emitted by `inference.py` follows one of four structured prefixes:

| Prefix | When emitted | Key fields |
| - | - | - |
| `[START]` | Beginning of each task episode | `task`, `env`, `model` |
| `[STEP]` | After every `client.step()` call | `step`, `action` (JSON), `reward`, `done`, `error` |
| `[END]` | After the episode completes or exhausts steps | `success`, `steps`, `score`, `rewards` (comma-separated) |
| `[DEBUG]` | Internal diagnostics — fallback notices, API errors, final scores | freeform |

Step rewards are the **intermediate** reward signal from the server's dense heuristic — they are useful for debugging but are **not** your final grade. Your real score is the terminal grader output, which you see in the `[END]` line's `score` field and the `[DEBUG] Final Score` line.

## How the Episode Loop Works

The `run_task(task_id, use_llm=True)` function implements the full POMDP interaction loop. Understanding it helps you debug unexpected behavior and write better custom agents.

```
run_task("t1", use_llm=True)
│
├── Set TASK_ID env var → CRMDataPipelineEnvClient.reset()
│     └── Server initialises episode, returns first observation
│
├── Loop (up to MAX_STEPS_PER_TASK = 15):
│   │
│   ├── build_user_prompt(obs, steps_remaining, task_id)
│   │     └── Formats objective, sources, schema, view, quality report, feedback
│   │
│   ├── call_gpt_with_retry(prompt, max_retries=3)
│   │     ├── Calls openai_client.chat.completions.create(...)
│   │     ├── On RateLimitError: exponential backoff (1s → 2s → 4s)
│   │     └── On failure: returns None
│   │
│   ├── validate_action(payload)
│   │     ├── Checks action_type is in PipelineActionType enum
│   │     └── Constructs CRMPipelineAction(**payload) — returns None on failure
│   │
│   ├── If action is None → build_smart_fallback(obs, step, task_id)
│   │
│   ├── client.step(action) → StepResult(observation, reward, done)
│   │
│   └── If done=True → break
│
└── score = result.reward (clamped to [0.01, 0.99])
```

The `MAX_STEPS_PER_TASK` constant is set to `15`, which is sufficient headroom even for the complex three-way SQL join in Task 3. If your LLM agent exceeds 15 steps without submitting, the loop terminates and the episode is scored based on whatever state the pipeline was left in.

## The Fallback Mechanism

`call_gpt_with_retry` uses **exponential backoff** across up to 3 attempts. After each failed attempt the code sleeps `2 ** attempt` seconds before retrying:

* Attempt 1 (attempt index 0) — immediate call; on `RateLimitError` sleeps **1 s** (`2^0`)
* Attempt 2 (attempt index 1) — on `RateLimitError` sleeps **2 s** (`2^1`)
* Attempt 3 (attempt index 2) — on `RateLimitError` sleeps **4 s** (`2^2`)
* If all three fail — returns `None`

When `validate_action` receives `None` or an invalid payload, the code immediately falls back to `build_smart_fallback(obs, step, task_id)`, which returns the next action from a fully hardcoded, per-task action plan:

```python Task 1 Fallback Plan theme={null}
[
    {"action_type": "PROFILE_SOURCE", "source": "web_forms"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms", "column": "email",       "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms", "column": "name",        "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms", "column": "phone",       "standardization_strategy": "EXTRACT_NUMBERS"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms", "column": "signup_date", "standardization_strategy": "TO_DATETIME_ISO"},
    {"action_type": "DEDUPLICATE",         "source": "web_forms", "deduplication_strategy": "EXACT_EMAIL"},
    {"action_type": "EXECUTE_SQL", "query": "SELECT * FROM web_forms WHERE customer_id != '???' AND email NOT LIKE '%bot%'", "output_table": "web_forms_clean"},
    {"action_type": "SUBMIT_PIPELINE", "final_source": "web_forms_clean"},
]
```

```python Task 2 Fallback Plan theme={null}
[
    # Clean web_forms
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms",  "column": "email",       "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms",  "column": "name",        "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms",  "column": "phone",       "standardization_strategy": "EXTRACT_NUMBERS"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_forms",  "column": "signup_date", "standardization_strategy": "TO_DATETIME_ISO"},
    {"action_type": "EXECUTE_SQL", "query": "SELECT * FROM web_forms WHERE customer_id != '???' AND email NOT LIKE '%bot%'", "output_table": "web_forms_clean"},
    # Clean legacy_db
    {"action_type": "STANDARDIZE_COLUMN", "source": "legacy_db",  "column": "email",       "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "legacy_db",  "column": "name",        "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "legacy_db",  "column": "phone",       "standardization_strategy": "EXTRACT_NUMBERS"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "legacy_db",  "column": "signup_date", "standardization_strategy": "TO_DATETIME_ISO"},
    {"action_type": "EXECUTE_SQL", "query": "SELECT * FROM legacy_db WHERE customer_id != '???' AND email NOT LIKE '%bot%'", "output_table": "legacy_db_clean"},
    # Merge and dedup
    {"action_type": "EXECUTE_SQL", "query": "SELECT * FROM web_forms_clean UNION ALL SELECT * FROM legacy_db_clean", "output_table": "merged_output"},
    {"action_type": "DEDUPLICATE", "source": "merged_output", "deduplication_strategy": "EXACT_EMAIL"},
    {"action_type": "SUBMIT_PIPELINE", "final_source": "merged_output"},
]
```

```python Task 3 Fallback Plan theme={null}
[
    # Standardize all three sources
    {"action_type": "STANDARDIZE_COLUMN", "source": "salesforce", "column": "email",         "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "salesforce", "column": "phone",         "standardization_strategy": "EXTRACT_NUMBERS"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_leads",  "column": "email",         "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "web_leads",  "column": "phone",         "standardization_strategy": "EXTRACT_NUMBERS"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "legacy_db",  "column": "contact_email", "standardization_strategy": "LOWERCASE_STRIP"},
    {"action_type": "STANDARDIZE_COLUMN", "source": "legacy_db",  "column": "home_phone",    "standardization_strategy": "EXTRACT_NUMBERS"},
    # 3-way SQL merge with bot filtering
    {
        "action_type": "EXECUTE_SQL",
        "query": """
            SELECT MAX(customer_id) as customer_id, email, MAX(phone) as phone
            FROM (
                SELECT customer_id, email, phone FROM salesforce
                    WHERE customer_id != '???' AND email NOT LIKE '%bot%'
                UNION ALL
                SELECT customer_id, email, phone FROM web_leads
                    WHERE customer_id IS NOT NULL AND email NOT LIKE '%bot%'
                UNION ALL
                SELECT REPLACE(legacy_id, 'OLD-', 'CUST_') AS customer_id,
                       contact_email AS email, home_phone AS phone
                FROM legacy_db WHERE legacy_id != '???' AND contact_email NOT LIKE '%bot%'
            )
            GROUP BY email
        """,
        "output_table": "merged_output"
    },
    {"action_type": "SUBMIT_PIPELINE", "final_source": "merged_output"},
]
```

The `RuleBasedBaseline` class is a thin wrapper around `build_smart_fallback` — it holds the `task_id` and calls the fallback on every `act(obs, step)` invocation. It achieves the maximum grader score of `0.99` on all three tasks deterministically.

## Using a Different LLM Model

`inference.py` is compatible with any OpenAI API-compatible provider. Change the target by updating three environment variables before launching the script.

```bash OpenAI GPT-4o theme={null}
export API_BASE_URL=https://api.openai.com/v1
export API_KEY=sk-your-openai-key
export MODEL_NAME=gpt-4o
```

```bash Anthropic (via proxy) theme={null}
export API_BASE_URL=https://api.anthropic.com/v1
export API_KEY=sk-ant-your-key
export MODEL_NAME=claude-3-5-sonnet-20241022
```

```bash Local Ollama theme={null}
export API_BASE_URL=http://localhost:11434/v1
export API_KEY=ollama
export MODEL_NAME=llama3.1:8b
```

```bash vLLM Self-Hosted theme={null}
export API_BASE_URL=http://your-vllm-host:8000/v1
export API_KEY=token-abc123
export MODEL_NAME=meta-llama/Meta-Llama-3.1-70B-Instruct
```

> [!WARNING]
>
> The script uses `response_format={"type": "json_object"}` in the LLM call, which requires the model to support JSON mode. Not all models or providers honour this parameter. If your model does not support it, `json.JSONDecodeError` will be caught and the fallback will engage automatically — but you will lose LLM-driven actions for every step.

## Running Only a Single Task

By default the script runs all three tasks. To run only one task, restrict the task list in the `__main__` block — `run_task` sets `TASK_ID` programmatically before calling `reset()`, so no server restart is needed:

```python theme={null}
# Edit the __main__ block in inference.py to run one task
if __name__ == "__main__":
    results = {}
    for task_id in ["t2"]:            # <-- restrict to one task
        results[task_id] = run_task(task_id, use_llm=True)
    print(results)
```

Alternatively, call `run_task` directly from a Python script or REPL:

```python theme={null}
from inference import run_task
score = run_task("t3", use_llm=False)   # use_llm=False forces pure rule-based
print(f"Score: {score:.4f}")
```

## Running in Pure Rule-Based Mode

Pass `use_llm=False` to bypass all LLM calls entirely. This is the fastest way to establish a `0.99` baseline and verify that your server setup is working correctly:

```python theme={null}
# In inference.py __main__ block, change use_llm to False:
for task_id in ["t1", "t2", "t3"]:
    results[task_id] = run_task(task_id, use_llm=False)
```

```bash theme={null}
uv run python inference.py
# Expected output:
# [DEBUG] === Final Scores ===
# [DEBUG] {"t1": 0.99, "t2": 0.99, "t3": 0.99, "average": 0.99}
```

> [!TIP]
>
> Use rule-based mode (`use_llm=False`) as a sanity check whenever you change the server code or Docker image. If the rule-based baseline drops below `0.99`, something in the environment is broken — not your LLM.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.