📋 YAML Task Schema

All the fields you can use in a task YAML file.

task_name: my_evaluation_task

display_name: "My Evaluation Task"

description: "Tests mathematical reasoning with basic arithmetic"

tests:
  - id: test_1
    prompt: "What is 2 + 2?"
    expected: 4
    scoring:
      mode: exact_match
  - id: test_2  
    prompt: "Calculate 15 * 3"
    expected: 45
    points: 2

attachments_root: "./documents"

enable_prompt_placeholders: true

task_defaults:
  temperature: 0.2

Test-Level Fields

Fields you can use for each test in the tests array:

tests:
  - id: complex_task
    prompt: "Solve this complex problem..."
    expected: "Solution"
    points: 5  # This test is worth 5 points

temperature: 0.2  # Override task default

🎯 Scoring Modes

How to score model responses - pick the right tool for your evaluation needs!

💰 budget

📌 code

📦 json

📋 list

🔢 numeric

⚡ ops

📊 performance

✍️ text

🔗 Pipeline Variables

Chain your tests together! Export values from one test and use them in the next - perfect for multi-step workflows.

Overview

How to chain results between tests.

Exporting Values

Use the exports field in a test to capture values:

Copytests:
  - id: test_1
    prompt: "What is 15 + 27?"
    expected: 42
    exports:
      - name: result1
        mode: raw  # Capture full response

Using Exported Values

Reference exported values in subsequent tests with {var:name}:

Copytests:
  - id: test_2
    prompt: "Multiply {var:result1} by 3"
    expected: 126

Common Mistakes

# WRONG:
{{exports.test_1.result1}}  # Don't use
{{exports.result1}}         # Don't use
{{result1}}                 # Don't use

# CORRECT:
{var:result1}               # Use this!

📄 Document Placeholders

Attach PDFs, text files, spreadsheets, and more to your tests. Document content is automatically included when benchmarking.

The {document_content} Placeholder

Use {document_content} to control exactly where attachment text appears in your prompt. Without it, content is added automatically:

Copyenable_prompt_placeholders: true
tests:
  - id: doc_analysis
    prompt: |
      Analyze this document and extract the main topic:
      
      {document_content}
    attachments:
      - report.pdf
    expected: "Financial analysis"

Supported File Types

Text-Extractable Documents

• Text files: .md, .txt
• Documents: .docx, .pdf, .rtf
• Data files: .csv, .json
• Spreadsheets: .xls, .xlsm, .xlsx

Auto-Readable Text Files

• Code files: .c, .java, .js, .py, .ts, .tsx
• Web files: .css, .htm, .html, .xml
• Config files: .json, .yaml, .yml
• Other text: .csv, .log, .md, .mjs, .tsv, .txt

Vision Model Attachments

• Images: .jpg, .jpeg, .png, .webp, .gif, .bmp (requires vision-capable model)

🚀 Complete Examples

Minimal Valid Task

Copytask_name: simple_math
display_name: "Simple Math Test"
tests:
  - prompt: "What is 2 + 2?"
    expected: 4

Chained Math with Pipeline Variables

Copytask_name: chained_math
display_name: Chained Math Example
description: Demonstrates pipeline variables
tests:
  - id: addition
    prompt: "Calculate 15 + 27. Return ONLY the number."
    expected: 42
    scoring:
      mode: exact_match
    exports:
      - name: sum_result
        mode: raw
  
  - id: multiplication
    prompt: "Multiply {var:sum_result} by 3. Return ONLY the number."
    expected: 126
    scoring:
      mode: numeric_from_text

Vision Model Evaluation

Copytask_name: vision_emotional_detection
display_name: Vision - Emotion Detection
description: Evaluate model ability to detect emotions in images
attachments_root: test_data/images

tests:
  - id: happy_detection
    prompt: |
      Look at the attached image and answer with only the word "happy" or "sad".
    expected: happy
    points: 10
    attachments:
      - family_photo.jpg
    scoring:
      mode: case_insensitive

  - id: sad_detection
    prompt: |
      Look at the attached image and answer with only the word "happy" or "sad".
    expected: sad
    points: 10
    attachments:
      - memorial_photo.jpg
    scoring:
      mode: case_insensitive
      
  - id: count_people
    prompt: |
      Count the number of people in the attached image.
      Return ONLY the number (digits only).
    expected: 4
    points: 20
    attachments:
      - group_photo.png
    scoring:
      mode: numeric_from_text
      tolerance: 0

Document Extraction with Scoring

Copytask_name: excel_data_analysis
display_name: Excel Data Analysis
description: Extract and analyze data from spreadsheet
enable_prompt_placeholders: true
attachments_root: test_data/spreadsheets

tests:
  - id: count_items
    prompt: |
      Analyze the attached Excel file.
      Count how many times "apple" appears in the 'Product' column.
      Return ONLY the number.
      
      {document_content}
    expected: 5
    points: 25
    attachments:
      - sales_data.xlsx
    scoring:
      mode: numeric_from_text
      tolerance: 0
      
  - id: sum_values  
    prompt: |
      From the same data, calculate the total revenue for "apple" products.
      Return ONLY the number (no currency symbols).
      
      {document_content}
    expected: 2500
    points: 35
    attachments:
      - sales_data.xlsx
    scoring:
      mode: numeric_tolerance
      tolerance: 0.01
      method: abs

Knowledge Testing with Flexible Scoring

Copytask_name: philosophy_knowledge
display_name: Ancient Philosophy Knowledge Test
description: Test knowledge of ancient Greek philosophers

tests:
  - id: name_philosophers
    prompt: List 5 ancient Greek philosophers. Separate names with commas.
    expected:
      - Socrates
      - Plato
      - Aristotle
      - Epicurus
      - Thales
      - Diogenes
      - Pythagoras
      - Heraclitus
    points: 20
    scoring:
      mode: contains_any
      partial_credit: true
      case_insensitive: true
      
  - id: socratic_method
    prompt: |
      In one sentence, what is the Socratic method?
    expected:
      - questioning
      - dialogue
      - inquiry
      - assumptions
    points: 30
    scoring:
      mode: contains_all
      partial_credit: true
      case_insensitive: true

Multiple Choice Questions

Copytask_name: science_quiz
display_name: Science Knowledge Quiz
description: Multiple choice questions testing scientific knowledge

tests:
  - id: periodic_table_mcq
    prompt: |
      What is the chemical symbol for Gold?
      
      A) Au
      B) Ag
      C) Fe
      D) Cu
      
      Return ONLY the letter of the correct answer.
    expected: A
    points: 10
    scoring:
      mode: exact_match
      case_insensitive: true
      trim_whitespace: true
      
  - id: astronomy_mcq
    prompt: |
      Which planet is known as the "Red Planet"?
      
      A) Venus
      B) Jupiter
      C) Mars
      D) Saturn
      
      Return ONLY the letter of the correct answer.
    expected: C
    points: 10
    scoring:
      mode: exact_match
      case_insensitive: true
      trim_whitespace: true
      
  - id: biology_mcq
    prompt: |
      What is the powerhouse of the cell?
      
      A) Nucleus
      B) Ribosome
      C) Mitochondria
      D) Golgi apparatus
      
      Return ONLY the letter of the correct answer.
    expected: C
    points: 10
    scoring:
      mode: exact_match
      case_insensitive: true
      trim_whitespace: true

💡 Prompt Examples

Examples showing different ways to describe your benchmark tasks. The AI agent will interpret your intent and generate test cases automatically. These examples progress from simple one-liners to more detailed specifications.

Example 1: Just Say What You Want

CopyI want to test if AI models can do basic math.

That's it! The AI agent will generate appropriate math questions and expected answers for you.

Example 2: Add Some Detail

CopyTest if models can add 234 + 567, subtract 89 from 456, and multiply 23 by 4.

Example 3: Organized List Style

CopyCreate a geography quiz with these questions:

1. What's the capital of France? (Answer: Paris)
2. Name the longest river in the world (Answer: Nile or Amazon, accept either)
3. Which continent has the most countries? Give options:
   A) Asia
   B) Africa
   C) Europe
   D) North America
   The correct answer is B

Make the first two worth 15 points each and the multiple choice worth 20 points.

Example 4: Structured Requirements Style

CopyBuild a task to evaluate reading comprehension using an attached article.txt file.

Test 1: Count how many times the word "innovation" appears in the document
Expected: 12 occurrences
Points: 20

Test 2: Extract the main author's name from the first paragraph
Expected: "Dr. Sarah Johnson"
Points: 15

Test 3: List three key technologies mentioned in the article
Expected answers should include at least 3 of: artificial intelligence, quantum computing, biotechnology, renewable energy, nanotechnology
Points: 30

Example 5: Advanced Multi-Step Task

CopyDesign a chained calculation task for testing multi-step reasoning:

Step 1: "A store has 156 apples. If they sell 40% of them, how many are left?"
- Expected: 94 (accept 93.6 with rounding)
- Points: 25
- Save this result for the next step

Step 2: "If each remaining apple costs $0.75, what's the total value of remaining inventory?"
- Expected: 70.50 or 70.5
- Points: 35
- This should use the result from step 1

Step 3: "If apples are packed 6 per box, how many full boxes can be made?"
- Expected: 15
- Points: 40
- Should use the quantity from step 1

Example 6: Expert Technical Specification

CopyComprehensive JSON API response validation task:

test_1: Validate user profile JSON structure
prompt: "Generate a JSON object representing a user profile with fields: name (string), age (integer), email (string), active (boolean)"
expected: JSON schema with required fields and correct types
scoring: json_schema validation
points: 30

test_2: Extract nested values
prompt: "From this JSON: {'company': {'name': 'TechCorp', 'employees': {'count': 250, 'departments': ['Engineering', 'Sales', 'HR']}}}, extract the employee count"
expected: 250
scoring: numeric_from_text with tolerance 0
points: 20

test_3: Array manipulation
prompt: "Given the departments array from the previous JSON, return all departments that start with 'E'"
expected: ["Engineering"]
scoring: exact_match for JSON array
points: 25

🧠 Tips & Best Practices

Practical advice to help you create effective benchmarks and get the most out of your evaluations.

Copy# This works! No scoring mode needed.
tests:
  - id: capital_city
    prompt: "What is the capital of Japan? Answer with the city name only."
    expected: Tokyo

Copy- id: science_mcq
  prompt: |
    Which planet is closest to the Sun?
    
    A) Venus
    B) Mercury
    C) Mars
    D) Earth
    
    Return ONLY the letter.
  expected: B

"Return ONLY the number, no words or units."

Copyscoring:
  mode: exact_match
  case_insensitive: true
  trim_whitespace: true