2.9.26
Released 2 Nov 2026
Official Pairwise LM website
LLM Benchmarks run locally on Windows
Pairwise LM runs your scripts to build an llm leaderboard, and version 2.9.26 helps you measure performance on your local hardware. Your private table.
Score the CSV file with exact match and print the accuracy percentage.
- write_file
score_results.pysaved - edit_file
score_results.py+3 - Shell
python score_results.pyexit 0 - read_file
score_results.py8 lines - grep
accuracy0 matches
waiting for your OK before the next stepRejectAccept
34 MB
Standard installer size
25 MB
Single .exe file
13
Interface languages
11
File manipulation tools
3
Auto-run, Ask first, Plan only
LLM Benchmarks you control
llm benchmarks require strict reproducibility and precise scoring, so you define the tasks and the scoring rules before running your own evaluation scripts locally. Check results. Every action appears in the chat as a card with its real result to verify the process. Version 2.9.26 supports this fully and runs the tools itself.
Run your evaluation script
edit_file applies changes with a visible diff, so you review every modification before it is saved, and Shell executes the script on your PC to return the real output and exit code. Precise work. write_file creates the evaluation script while read_file displays the full content of the generated file for your review. grep searches for specific patterns in the text to help you locate relevant data within the workspace. Pairwise LM version 2.9.26 handles llm evaluation for your code with precision and supports ai model benchmark tasks. These benchmarks that measure propensity to agree are essential for accurate assessment.
Tool trace
write_file score_results.py → saved
edit_file score_results.py → +24
Shell python score_results.py → exit 0Measure your own hardware
Shell executes terminal commands to run your build and testing workflows while read_file retrieves the saved output for your records and analysis of the benchmark results. Edit files. grep searches the log for specific tokens and filters noise from the data. edit_file changes the script and shows a diff before you approve the change. The current version 2.9.26 supports these tools on Windows 10/11. Check the results carefully to ensure everything is correct and ready for review.
Tool trace
Shell gpu_monitor --tokens_per_sec --first_token --peak_memory --model local_model --repeats 3
read_file results/throughput_metrics.csv
grep "peak_memory" results/throughput_metrics.csvCompare models pairwise
Pairwise comparison shows which model is better, so write_file saves both answers for later analysis and Shell runs your comparison script to count the wins across many prompts. Run it. Use write_file to save the outputs for later analysis by the user. Shell executes the script and returns the exit code for verification. Count the wins for each model across many prompts to determine the leader. Version 2.9.26 supports this workflow for accurate benchmarking.
Tool trace
write_file model_a.txt → saved
write_file model_b.txt → saved
Shell python compare.py → exit 0Build an LLM Leaderboard
The agent reads the result files and calculates the mean and the spread so that you review the table in the chat and verify the accuracy of the scores. Check the numbers. Interface in 13 languages for global users to access easily. read_file reads the data, and grep searches for specific patterns within the output. This builds your own ranking for ai model comparison with high precision. Shell runs the command to process the results and verify the exit code.
Tool trace
read_file results.csv → 50 rows
Shell python stats.py → exit 0
grep mean stats.py → 1 matchHow it differs from a typical chat assistant
Pairwise LM version 2.9.26 runs your commands and shows the real result on a card, while a chat assistant only suggests actions. Full control.
| Typical chat assistant | Pairwise LM | |
|---|---|---|
| Action | Runs the tools itself | Runs the tool itself |
| Result | Shows text summary | Shows real output card |
| Control | Runs automatically | Ask first or Auto-run |
| Files | Works on text you paste | Reads and writes files |
How to start
- Download the 34 MB setup program for Windows 10 or 11.
- Run the setup program (about 34 MB) and follow the installer.
- Open the app and select your model endpoint or local model.
- Send a prompt and watch the agent run the tools for you.
Who uses it and when
Researchers write their own scoring logic and run it on their hardware to keep the raw outputs next to the scores. Count files. Run 5 tests on your local machine today to verify accuracy.
Engineers run their own benchmarks on local GPUs and the agent captures the latency and memory usage for ai model comparison on their hardware. They review carefully. The output appears as a card with real results and details.
Students use it to learn about llm benchmarks, and they write simple scripts to score answers while the agent runs the scripts and shows the results. This helps. They see the exact commands used and understand the process.
Frequently asked questions
What are llm benchmarks?
A benchmark is a fixed set of tasks together with a scoring rule, so different models can be compared on the same work, and you define the tasks. Fix settings. Record versions for reproducibility in your evaluation scripts.
Is lmbenchmarks.com the official Pairwise LM website?
Yes, lmbenchmarks.com is the official Pairwise LM website, and the download button provides the Windows setup program (about 34 MB) which you should verify by comparing its SHA-256 checksum with the published manifest. Check the hash. IP logging occurs on visits to the static site for downloads.
How does Pairwise LM help with assessing model quality?
Pairwise LM is an autonomous agent that runs your evaluation scripts through Shell and edits them with a diff, so you can compare model answers side-by-side. It works locally. It shows real results in the chat for every action.
What is a GPU AI benchmark?
A GPU AI benchmark is timing your own workload, so you measure tokens per second and peak memory on your PC or over SSH on a server you own. Real metrics. Yes, it measures tokens per second and peak memory on your hardware.
Does the agent run on Windows?
Yes, Pairwise LM runs on Windows 10 and 11, and the setup program is 34 MB while a portable .exe is 25 MB. No account needed. The interface is in 13 languages so you can use it easily.
How do I verify the download?
The SHA-256 checksum serves as a unique digital fingerprint for the installer file, and you verify it by running the certutil command to compare the result with the published manifest. Match the checksum. This confirms the file matches the manifest for safe installation.
Which public benchmarks do you trust?
A benchmark is a fixed set of tasks together with a scoring rule, so different models can be compared on the same work by you. Trust yours. You control the prompts and the settings for every test.
Run your benchmarks today
Download Pairwise LM version 2.9.26 for Windows 10 and 11, because the setup program is 34 MB and the portable version is 25 MB. Run it.
Download Pairwise LM for Windows ↗Site navigation
Every page of this website is available in 20 languages.
Product details ↗ Model configuration ↗ Workflow examples ↗ Download installer ↗ FAQ and support ↗ Privacy policy ↗
