Benchmark your prompt from your coding agent

The OpenMark MCP server lets Cursor, Claude Code, Codex or any MCP client write a test from your actual prompt and data, run it on 100+ models with real API calls, and pick the cheapest model that passes. One OpenMark key, no provider accounts.

Setup

  1. Create a key

    Sign in at openmark.ai/ui (free, 50 credits), open the account menu and choose API keys (MCP). The key is shown once.

  2. Add the server to your client

    Claude Code:

    claude mcp add --transport http openmark https://openmark.ai/mcp \
      --header "Authorization: Bearer om_YOUR_KEY"

    Cursor (~/.cursor/mcp.json, or .cursor/mcp.json in a project):

    {
      "mcpServers": {
        "openmark": {
          "url": "https://openmark.ai/mcp",
          "headers": { "Authorization": "Bearer om_YOUR_KEY" }
        }
      }
    }

    Some directories show the same block with "type": "streamable-http" added. Both forms point at the same server.

    Codex (CLI, IDE extension and app share ~/.codex/config.toml). Codex reads the key from an environment variable:

    [mcp_servers.openmark]
    url = "https://openmark.ai/mcp"
    bearer_token_env_var = "OPENMARK_API_KEY"

    Then set OPENMARK_API_KEY to your key (macOS / Linux: export OPENMARK_API_KEY=om_YOUR_KEY in your shell profile; Windows: setx OPENMARK_API_KEY om_YOUR_KEY, then restart Codex).

    VS Code (GitHub Copilot agent mode, .vscode/mcp.json):

    {
      "servers": {
        "openmark": {
          "type": "http",
          "url": "https://openmark.ai/mcp",
          "headers": { "Authorization": "Bearer om_YOUR_KEY" }
        }
      }
    }

    OpenClaw (~/.openclaw/openclaw.json; set OPENMARK_API_KEY in the environment as for Codex, then check it with openclaw mcp doctor openmark --probe):

    {
      "mcp": {
        "servers": {
          "openmark": {
            "url": "https://openmark.ai/mcp",
            "transport": "streamable-http",
            "headers": { "Authorization": "Bearer ${OPENMARK_API_KEY}" }
          }
        }
      }
    }

    Any other client: remote server over Streamable HTTP at https://openmark.ai/mcp with the header Authorization: Bearer om_YOUR_KEY. Which snippet to use:

    • Cursor JSON: Cursor, and most clients configured with an mcpServers JSON file.
    • Claude Code command: Claude Code in the terminal or IDE.
    • Codex TOML: Codex CLI, Codex IDE extension and Codex app.
    • OpenClaw JSON: OpenClaw agents, with or without the OpenMark Router plugin.
    • Claude.ai and Claude Desktop connectors don't accept a custom key header yet. Use Claude Code instead.
  3. Ask your agent

    Use OpenMark to benchmark the prompt in src/extract_invoice.py on 6 models,
    with 5 realistic invoices from tests/fixtures as test cases. Recommend the
    cheapest model within 2 points of the best score, then watch the task.

Tools

ToolWhat it does
get_task_formatThe task YAML format and all 18 scoring modes, plus the full Task Guide on demand (weighted scoring, chained tests, documents, examples), so the agent writes a valid, fair test.
validate_taskChecks the task the way OpenMark will (structure, scoring settings, your plan's test limit) before anything runs. Free.
list_modelsCurrent model ids and prices, and which models accept images, so the agent never guesses a stale model name.
prepare_attachmentsA one-hour upload link for images, PDFs or documents your tests need. No API key in the command.
estimate_costThe expected credits for a run, split per model, and what your balance needs to start. Nothing runs.
run_benchmarkSaves the task to your account and runs it on the chosen models. Spends your credits, like in the app; an optional budget (max_credits) stops it from starting if the quote is higher.
get_resultsScore, stability, cost per run and latency for every model, the tests each model missed with its answer, and the best pick for accuracy, cost, speed or consistency.
export_openclaw_csvThe routing file for the OpenMark Router plugin (the same file as the app's Export > OpenClaw), so OpenClaw sends that kind of task to the winner. Free.
watch_taskRe-tests the task when new or cheaper models ship (each check uses credits at the normal rate) and emails you only if one beats the winner.
get_accountYour plan, credit balance and limits, and what each plan would unlock.

Writing the tests

With the MCP server, your agent designs the test: it writes the task from your real prompt and data, instead of OpenMark's task builder. It reads the same reference as the app's Task Guide through get_task_format: every field, all 18 scoring modes and their options, weighted scoring on several criteria, chained tests, documents, and the test-design rules OpenMark's own builder follows. validate_task then checks the task for free before any credits are spent.

To review what your agent wrote, open the Task Guide.

Choosing the winner

Your agent writes the test and picks the models to compare; OpenMark runs them and scores the answers. Tell the agent what matters most and it asks for that ranking:

PriorityPick
Accuracy (default)The highest score. Ties go to the cheaper model, then the faster one: the same order as the results table in the app.
CostThe cheapest model within 2 points of the top score (you can change the margin).
SpeedThe fastest model within that margin.
ConsistencyThe model whose score varies least between runs, within that margin.

Every result also lists the pick for each priority, and the tests each model got wrong with its answer, so the agent can tell you why the others lost. Scoring is deterministic (exact match, patterns, numbers, JSON checks), never an LLM judge.

Run settings

The same settings as the app's Configure & Run panel, as optional arguments to run_benchmark and estimate_cost. Ask in plain words ("cap answers at 300 tokens", "use the thorough preset") and the agent passes them. Run estimate_cost with and without a setting to see what it changes.

SettingDefaultWhat it does
presetstandardstandard: 2 passes, snappy timeouts. thorough: 3 passes, patient timeouts.
stability_runs2Passes per model, 1 to 5. Two or more show which models are consistent.
max_tokens500Output limit per answer, 16 to 16,000, or 0 for no limit. A higher limit raises the credits needed to start a run, so keep the default unless the results flag answers that came back empty or near the limit; then re-run those models with more room.
temperature_discoveryoffFinds each model's best temperature for the task. Costs about 6 times as much.
temp_preferencelowWith temperature discovery: which temperature wins a tie.
timeout_profilesnappyWait per answer: snappy 45 s, patient 90 s, or custom (custom_timeout, 120 to 600 s).
fail_fastonStops a model after repeated errors to save credits. Turn it off to let every model finish, e.g. to diagnose failures.

Images and documents

Tests can include images, PDFs and documents. The agent calls prepare_attachments, uploads the files to the one-hour link it gets back (no API key needed), names them in each test, and runs the benchmark. Images go to models that accept images. A prompt like this works:

Use OpenMark to find the cheapest model that identifies the mushroom species
in the 8 photos in ./samples. Expected answers are in ./samples/labels.csv.
FilesLimits
ImagesPNG, JPG, WebP, GIF, BMP. Up to 25 MB and 24 megapixels each; large images are scaled down to 2,048 px on the long side. Sent only to models that accept images.
PDFsUp to 25 MB and 400 pages each. The text is extracted; the first 50 pages are used.
DocumentsTXT, Markdown, CSV, JSON, DOCX, XLSX / XLS, RTF. The text is extracted; spreadsheets up to 500,000 cells.
Per uploadUp to 10 files and 25 MB in total. Up to 20 files per task.
StorageFree 25 MB, Pro 2 GB, Expert 5 GB per account.

Watching a task

After a run, the agent can call watch_task with the model it recommended. When new models ship or prices drop, OpenMark re-tests the task on up to four of them, next to your pick, and emails you only if one beats it: about the same score for at most 70% of the cost, or at least 5 points better.

Each check uses credits from your balance at the normal rate, and only runs when there is something new to test. The agent is told the most a check can cost when it starts the watch. If your balance runs low, the check waits and you get one email; it runs the morning after you top up.

Routing in OpenClaw

With the OpenMark Router plugin, OpenClaw sends each message to the model that won your benchmark for that kind of task. Connect OpenClaw to this server (setup above) and one request does the whole loop:

Benchmark my three most common tasks on OpenMark (support replies, invoice
extraction, meeting summaries) on 6 models each, then save the routing files
for the OpenMark router.

The agent writes and runs the tests, calls export_openclaw_csv for each finished job, and saves the files in the router's benchmarks folder (~/.openclaw/workspace/plugins/openmark-router/benchmarks/). The router picks them up within 5 minutes.

Use it from scripts or CI

No agent needed: the server is plain HTTPS with JSON (JSON-RPC 2.0), no session, so a script or CI job can call any tool with your key. This is the supported way to use OpenMark from code.

curl -s https://openmark.ai/mcp \
  -H "Authorization: Bearer $OPENMARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/call",
       "params": {"name": "get_account", "arguments": {}}}'

Every tool works the same way: its name and arguments go in params. The reply holds the text an agent would read in result.content[0].text, and the same data as JSON in result.structuredContent. To check and price a task kept in your repo (both free):

for TOOL in validate_task estimate_cost; do
  jq -n --rawfile yaml task.yaml --arg tool "$TOOL" \
    '{jsonrpc: "2.0", id: 1, method: "tools/call",
      params: {name: $tool, arguments: {task_yaml: $yaml, models: ["gpt-5-mini", "gemini-2.5-flash"]}}}' |
  curl -s https://openmark.ai/mcp \
    -H "Authorization: Bearer $OPENMARK_API_KEY" \
    -H "Content-Type: application/json" -d @- |
  jq -r '.result.content[0].text'
done

run_benchmark takes the same arguments and returns a job_id; poll get_results about once a minute until structuredContent.status is completed (or failed). Keys can run benchmarks but can't open billing or change your plan.

Why not a generic model-picker?

Your prompt, not a public benchmark

Catalog and leaderboard MCP servers tell you which model is good in general. OpenMark scores models on the prompt and inputs your code actually uses.

Deterministic scoring

Exact-match, pattern, numeric and JSON checks, run more than once per model. No LLM judge.

Real costs

Measured from the tokens each model used on your task. On our hard SQL benchmark, nine models tied at 100% with a 92x price spread.

It keeps checking

Watch a task once and OpenMark re-tests it as new models ship, so your pick doesn't go stale.

FAQ

What gets sent to OpenMark?

Only the task the agent writes: prompts, test inputs and expected outputs. It is saved privately in your account, like any task you create in the app. Your code itself is not sent.

What does it cost?

The MCP server is free; benchmark runs and watch checks spend OpenMark credits exactly like in the app. Credits are the models' API list price times 3, at 100 credits per $1, the same for every model and only for the tokens used. A six-model text comparison with 5 tests on budget and mid-range models takes about 35 credits; images, more tests and flagships like Gemini 3.1 Pro cost more. Ask the agent to run estimate_cost first and it shows the price per model. New accounts get 50 credits, and packs start at $5.

What do paid plans add?

Any credit pack unlocks flagship models. Pro and Expert add monthly credits, more tests per task, more saved and watched tasks, and more models in parallel. Ask the agent to run get_account to see your limits.

Can I revoke a key?

Yes, any time from the same API keys screen. It stops working immediately.

Try it on your own prompt

Create a free account, make a key, and ask your agent which model your code should use.

Get a free key