inference.py script is the out-of-the-box evaluator bundled with the CRM Data Pipeline Environment. It drives an LLM-powered agent (or falls back to a deterministic rule-based baseline) through all three CRM cleaning tasks — t1, t2, and t3 — in sequence, then prints a final score for each task plus a computed average. You do not need to write any agent code to run it; it works immediately once your environment variables are set and the server is running.
Prerequisites
Before running inference, confirm the following are installed and available in your shell:- Python 3.12+ with
uv(the project usesuvas the package runner) openenv-core:pip install openenv-corepython-dotenv,openai,requests,pandas— all installed automatically viauv
If neither
HF_TOKEN nor API_KEY is set, the openai.OpenAI client is initialized with api_key=None. The script will still run — every LLM call will fail and the rule-based fallback will handle all steps automatically. This is intentional design: the fallback achieves the maximum grader score of 0.99 on all three tasks without any LLM.Running Inference
1
Set environment variables
Export your credentials in the terminal where you will run the script. The Alternatively, create a
API_BASE_URL and MODEL_NAME values shown below are the defaults; you may omit them if you want to use Qwen2.5-72B via the HuggingFace Router..env file in the repo root — the script calls load_dotenv() at startup and will pick it up automatically without overriding any already-exported variables:2
Start the FastAPI server
Open a second terminal and launch the environment server. The server must be running before you start inference — it binds to You should see Uvicorn startup output confirming the server is listening. Keep this terminal open for the entire inference run. If you need a different port, set
http://127.0.0.1:8080 by default.OPENENV_BASE_URL in the inference terminal:3
Run the inference script
In your first terminal (with credentials exported), run:The script iterates over
["t1", "t2", "t3"], runs one full episode for each task, then prints aggregate results. Total runtime with the rule-based fallback is typically under 10 seconds. With a live LLM, runtime depends on API latency and the number of retries.4
Inspect the output
Inference emits structured log lines to stdout. A typical successful run looks like this:The final JSON line is your benchmark result. A score of
0.99 is the maximum the grader awards — it means every row in your submitted DataFrame matched the hidden ground truth exactly.Understanding the Log Format
Every line emitted byinference.py follows one of four structured prefixes:
Step rewards are the intermediate reward signal from the server’s dense heuristic — they are useful for debugging but are not your final grade. Your real score is the terminal grader output, which you see in the
[END] line’s score field and the [DEBUG] Final Score line.
How the Episode Loop Works
Therun_task(task_id, use_llm=True) function implements the full POMDP interaction loop. Understanding it helps you debug unexpected behavior and write better custom agents.
MAX_STEPS_PER_TASK constant is set to 15, which is sufficient headroom even for the complex three-way SQL join in Task 3. If your LLM agent exceeds 15 steps without submitting, the loop terminates and the episode is scored based on whatever state the pipeline was left in.
The Fallback Mechanism
call_gpt_with_retry uses exponential backoff across up to 3 attempts. After each failed attempt the code sleeps 2 ** attempt seconds before retrying:
- Attempt 1 (attempt index 0) — immediate call; on
RateLimitErrorsleeps 1 s (2^0) - Attempt 2 (attempt index 1) — on
RateLimitErrorsleeps 2 s (2^1) - Attempt 3 (attempt index 2) — on
RateLimitErrorsleeps 4 s (2^2) - If all three fail — returns
None
validate_action receives None or an invalid payload, the code immediately falls back to build_smart_fallback(obs, step, task_id), which returns the next action from a fully hardcoded, per-task action plan:
RuleBasedBaseline class is a thin wrapper around build_smart_fallback — it holds the task_id and calls the fallback on every act(obs, step) invocation. It achieves the maximum grader score of 0.99 on all three tasks deterministically.
Using a Different LLM Model
inference.py is compatible with any OpenAI API-compatible provider. Change the target by updating three environment variables before launching the script.
Running Only a Single Task
By default the script runs all three tasks. To run only one task, restrict the task list in the__main__ block — run_task sets TASK_ID programmatically before calling reset(), so no server restart is needed:
run_task directly from a Python script or REPL:
Running in Pure Rule-Based Mode
Passuse_llm=False to bypass all LLM calls entirely. This is the fastest way to establish a 0.99 baseline and verify that your server setup is working correctly: