Create script
You ask the agent to write score_results.py.
Workflows
model evaluation in machine learning
03
Score test results with exact match
You write a short Python script to score your benchmark results. Pairwise LM creates the file, runs it in Shell, and shows the exact output. You review the diff before accepting the changes in Ask first mode.
Review diff
You check the diff in the chat card.
Run command
The agent runs the script in Shell and you read the real exit code.
Check output
You read the accuracy percentage from the output.
Example scoring script
Write a script to score these results.
- write_file
score_results.pysaved - Shell
python score_results.pyexit 0 - read_file
score_results.pyread - grep
correct2 matches
waiting for your OK before the next stepRejectAccept
score_results.py
results = [
("q1", "yes", "yes"),
("q2", "no", "maybe"),
("q3", "ok", "ok"),
]
correct = 0
for q, exp, ans in results:
if exp == ans:
correct += 1
print(correct)
print(len(results))This script scores made-up rows with exact match and prints the accuracy percentage.
Evaluate machine learning model locally
Write script
Ask the agent to create your evaluation script and it uses write_file to generate score_results.py for model evaluation in machine learning and review the diff. Review the diff. The agent uses edit_file to adjust the logic inside the script. You see the changes displayed clearly in the chat card. Accept the changes to match your specific benchmark needs exactly. Use Shell to run the script on your Windows PC. We are just saturating benchmarks with repetitive tests today because the agent runs the tools. Read the results to confirm the script runs without errors.
Check results
You verify the output by reading the file, and the agent shows the exact match count so you can compare it with your expected results. Check the method. Use web_search to read the published method for accuracy and correctness. The agent uses http_get to read the official documentation page. You review the source code to ensure the scoring logic is correct. Run the script again with different settings and compare the new results. Document the method and keep the raw outputs to ensure reproducibility. You review the diff before accepting changes when you evaluate machine learning model results for llm translation benchmark.
