Benchmark your prompt from your coding agent
The OpenMark MCP server lets Cursor, Claude Code, Codex or any MCP client write a test from your actual prompt and data, run it on 100+ models with real API calls, and pick the cheapest model that passes. One OpenMark key, no provider accounts.
Setup
-
Create a key
Sign in at openmark.ai/ui (free, 50 credits), open the account menu and choose API keys (MCP). The key is shown once.
-
Add the server to your client
Claude Code:
claude mcp add --transport http openmark https://openmark.ai/mcp \ --header "Authorization: Bearer om_YOUR_KEY"
Cursor (
~/.cursor/mcp.json, or.cursor/mcp.jsonin a project):{ "mcpServers": { "openmark": { "url": "https://openmark.ai/mcp", "headers": { "Authorization": "Bearer om_YOUR_KEY" } } } }Some directories show the same block with
"type": "streamable-http"added. Both forms point at the same server.Codex (CLI, IDE extension and app share
~/.codex/config.toml). Codex reads the key from an environment variable:[mcp_servers.openmark] url = "https://openmark.ai/mcp" bearer_token_env_var = "OPENMARK_API_KEY"
Then set
OPENMARK_API_KEYto your key (macOS / Linux:export OPENMARK_API_KEY=om_YOUR_KEYin your shell profile; Windows:setx OPENMARK_API_KEY om_YOUR_KEY, then restart Codex).VS Code (GitHub Copilot agent mode,
.vscode/mcp.json):{ "servers": { "openmark": { "type": "http", "url": "https://openmark.ai/mcp", "headers": { "Authorization": "Bearer om_YOUR_KEY" } } } }OpenClaw (
~/.openclaw/openclaw.json; setOPENMARK_API_KEYin the environment as for Codex, then check it withopenclaw mcp doctor openmark --probe):{ "mcp": { "servers": { "openmark": { "url": "https://openmark.ai/mcp", "transport": "streamable-http", "headers": { "Authorization": "Bearer ${OPENMARK_API_KEY}" } } } } }Any other client: remote server over Streamable HTTP at
https://openmark.ai/mcpwith the headerAuthorization: Bearer om_YOUR_KEY. Which snippet to use:- Cursor JSON: Cursor, and most clients configured with an
mcpServersJSON file. - Claude Code command: Claude Code in the terminal or IDE.
- Codex TOML: Codex CLI, Codex IDE extension and Codex app.
- OpenClaw JSON: OpenClaw agents, with or without the OpenMark Router plugin.
- Claude.ai and Claude Desktop connectors don't accept a custom key header yet. Use Claude Code instead.
- Cursor JSON: Cursor, and most clients configured with an
-
Ask your agent
Use OpenMark to benchmark the prompt in src/extract_invoice.py on 6 models, with 5 realistic invoices from tests/fixtures as test cases. Recommend the cheapest model within 2 points of the best score, then watch the task.
Tools
| Tool | What it does |
|---|---|
| get_task_format | The task YAML format and all 18 scoring modes, plus the full Task Guide on demand (weighted scoring, chained tests, documents, examples), so the agent writes a valid, fair test. |
| validate_task | Checks the task the way OpenMark will (structure, scoring settings, your plan's test limit) before anything runs. Free. |
| list_models | Current model ids and prices, and which models accept images, so the agent never guesses a stale model name. |
| prepare_attachments | A one-hour upload link for images, PDFs or documents your tests need. No API key in the command. |
| estimate_cost | The expected credits for a run, split per model, and what your balance needs to start. Nothing runs. |
| run_benchmark | Saves the task to your account and runs it on the chosen models. Spends your credits, like in the app; an optional budget (max_credits) stops it from starting if the quote is higher. |
| get_results | Score, stability, cost per run and latency for every model, the tests each model missed with its answer, and the best pick for accuracy, cost, speed or consistency. |
| export_openclaw_csv | The routing file for the OpenMark Router plugin (the same file as the app's Export > OpenClaw), so OpenClaw sends that kind of task to the winner. Free. |
| watch_task | Re-tests the task when new or cheaper models ship (each check uses credits at the normal rate) and emails you only if one beats the winner. |
| get_account | Your plan, credit balance and limits, and what each plan would unlock. |
Writing the tests
With the MCP server, your agent designs the test: it writes the task from your real prompt and data, instead of OpenMark's task builder. It reads the same reference as the app's Task Guide through get_task_format: every field, all 18 scoring modes and their options, weighted scoring on several criteria, chained tests, documents, and the test-design rules OpenMark's own builder follows. validate_task then checks the task for free before any credits are spent.
To review what your agent wrote, open the Task Guide.
Choosing the winner
Your agent writes the test and picks the models to compare; OpenMark runs them and scores the answers. Tell the agent what matters most and it asks for that ranking:
| Priority | Pick |
|---|---|
| Accuracy (default) | The highest score. Ties go to the cheaper model, then the faster one: the same order as the results table in the app. |
| Cost | The cheapest model within 2 points of the top score (you can change the margin). |
| Speed | The fastest model within that margin. |
| Consistency | The model whose score varies least between runs, within that margin. |
Every result also lists the pick for each priority, and the tests each model got wrong with its answer, so the agent can tell you why the others lost. Scoring is deterministic (exact match, patterns, numbers, JSON checks), never an LLM judge.
Run settings
The same settings as the app's Configure & Run panel, as optional arguments to run_benchmark and estimate_cost. Ask in plain words ("cap answers at 300 tokens", "use the thorough preset") and the agent passes them. Run estimate_cost with and without a setting to see what it changes.
| Setting | Default | What it does |
|---|---|---|
| preset | standard | standard: 2 passes, snappy timeouts. thorough: 3 passes, patient timeouts. |
| stability_runs | 2 | Passes per model, 1 to 5. Two or more show which models are consistent. |
| max_tokens | 500 | Output limit per answer, 16 to 16,000, or 0 for no limit. A higher limit raises the credits needed to start a run, so keep the default unless the results flag answers that came back empty or near the limit; then re-run those models with more room. |
| temperature_discovery | off | Finds each model's best temperature for the task. Costs about 6 times as much. |
| temp_preference | low | With temperature discovery: which temperature wins a tie. |
| timeout_profile | snappy | Wait per answer: snappy 45 s, patient 90 s, or custom (custom_timeout, 120 to 600 s). |
| fail_fast | on | Stops a model after repeated errors to save credits. Turn it off to let every model finish, e.g. to diagnose failures. |
Images and documents
Tests can include images, PDFs and documents. The agent calls prepare_attachments, uploads the files to the one-hour link it gets back (no API key needed), names them in each test, and runs the benchmark. Images go to models that accept images. A prompt like this works:
Use OpenMark to find the cheapest model that identifies the mushroom species in the 8 photos in ./samples. Expected answers are in ./samples/labels.csv.
| Files | Limits |
|---|---|
| Images | PNG, JPG, WebP, GIF, BMP. Up to 25 MB and 24 megapixels each; large images are scaled down to 2,048 px on the long side. Sent only to models that accept images. |
| PDFs | Up to 25 MB and 400 pages each. The text is extracted; the first 50 pages are used. |
| Documents | TXT, Markdown, CSV, JSON, DOCX, XLSX / XLS, RTF. The text is extracted; spreadsheets up to 500,000 cells. |
| Per upload | Up to 10 files and 25 MB in total. Up to 20 files per task. |
| Storage | Free 25 MB, Pro 2 GB, Expert 5 GB per account. |
Watching a task
After a run, the agent can call watch_task with the model it recommended. When new models ship or prices drop, OpenMark re-tests the task on up to four of them, next to your pick, and emails you only if one beats it: about the same score for at most 70% of the cost, or at least 5 points better.
Each check uses credits from your balance at the normal rate, and only runs when there is something new to test. The agent is told the most a check can cost when it starts the watch. If your balance runs low, the check waits and you get one email; it runs the morning after you top up.
Routing in OpenClaw
With the OpenMark Router plugin, OpenClaw sends each message to the model that won your benchmark for that kind of task. Connect OpenClaw to this server (setup above) and one request does the whole loop:
Benchmark my three most common tasks on OpenMark (support replies, invoice extraction, meeting summaries) on 6 models each, then save the routing files for the OpenMark router.
The agent writes and runs the tests, calls export_openclaw_csv for each finished job, and saves the files in the router's benchmarks folder (~/.openclaw/workspace/plugins/openmark-router/benchmarks/). The router picks them up within 5 minutes.
Use it from scripts or CI
No agent needed: the server is plain HTTPS with JSON (JSON-RPC 2.0), no session, so a script or CI job can call any tool with your key. This is the supported way to use OpenMark from code.
curl -s https://openmark.ai/mcp \
-H "Authorization: Bearer $OPENMARK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/call",
"params": {"name": "get_account", "arguments": {}}}'
Every tool works the same way: its name and arguments go in params. The reply holds the text an agent would read in result.content[0].text, and the same data as JSON in result.structuredContent. To check and price a task kept in your repo (both free):
for TOOL in validate_task estimate_cost; do
jq -n --rawfile yaml task.yaml --arg tool "$TOOL" \
'{jsonrpc: "2.0", id: 1, method: "tools/call",
params: {name: $tool, arguments: {task_yaml: $yaml, models: ["gpt-5-mini", "gemini-2.5-flash"]}}}' |
curl -s https://openmark.ai/mcp \
-H "Authorization: Bearer $OPENMARK_API_KEY" \
-H "Content-Type: application/json" -d @- |
jq -r '.result.content[0].text'
done
run_benchmark takes the same arguments and returns a job_id; poll get_results about once a minute until structuredContent.status is completed (or failed). Keys can run benchmarks but can't open billing or change your plan.
Why not a generic model-picker?
Catalog and leaderboard MCP servers tell you which model is good in general. OpenMark scores models on the prompt and inputs your code actually uses.
Exact-match, pattern, numeric and JSON checks, run more than once per model. No LLM judge.
Measured from the tokens each model used on your task. On our hard SQL benchmark, nine models tied at 100% with a 92x price spread.
Watch a task once and OpenMark re-tests it as new models ship, so your pick doesn't go stale.
FAQ
What gets sent to OpenMark?
Only the task the agent writes: prompts, test inputs and expected outputs. It is saved privately in your account, like any task you create in the app. Your code itself is not sent.
What does it cost?
The MCP server is free; benchmark runs and watch checks spend OpenMark credits exactly like in the app. Credits are the models' API list price times 3, at 100 credits per $1, the same for every model and only for the tokens used. A six-model text comparison with 5 tests on budget and mid-range models takes about 35 credits; images, more tests and flagships like Gemini 3.1 Pro cost more. Ask the agent to run estimate_cost first and it shows the price per model. New accounts get 50 credits, and packs start at $5.
What do paid plans add?
Any credit pack unlocks flagship models. Pro and Expert add monthly credits, more tests per task, more saved and watched tasks, and more models in parallel. Ask the agent to run get_account to see your limits.
Can I revoke a key?
Yes, any time from the same API keys screen. It stops working immediately.
Try it on your own prompt
Create a free account, make a key, and ask your agent which model your code should use.
Get a free key