What to Measure
Evaluate each decision at several levels:
-
Tool selection: Did the agent choose the required tool rather than another tool, an invalid tool, or no tool? For multiple valid tools, define acceptable alternatives in the reference annotation.
-
Argument correctness: Are required fields present, correctly typed, and semantically appropriate? Compare canonicalized arguments and report field-level exact-match or F1 scores.
-
Schema validity: Does the call conform to the tool’s declared JSON Schema?
-
Invocation success: Did execution complete without validation, authentication, dependency, or runtime errors?
-
Task success: Did the overall user goal succeed? This is usually the most important metric, but external service failures must be distinguished from model mistakes.
-
Control behavior: For ambiguous requests, should the agent abstain, ask for clarification, or request approval? Unnecessary calls are errors too.
-
Sequence behavior: In multi-step workflows, check ordering, dependencies, parallel-call handling, state updates, and whether later calls use earlier results.
Evaluation Dataset
Create representative scenarios containing the user request, available tools, agent state, expected calls, acceptable argument variants, expected outcome, failure attribution, and forbidden actions. Include routine cases, ambiguous requests, missing arguments, tool outages, multi-step tasks, and prompt-injection attempts.
Use public function-calling benchmarks for broad comparison, but keep a private, representative test set because public benchmarks can be contaminated and do not represent production traffic. If retrieval is exposed as a tool, evaluate retrieval separately using recall@k, MRR, and nDCG.
Metrics and Review
Use deterministic metrics for tool names, schemas, and arguments. Exact matching is appropriate for identifiers, but domain-specific normalization may be required; for example, "25.00" and 25.0 may represent the same monetary amount.
| Metric |
What it detects |
| Tool selection precision, recall, F1 |
Missing, extra, or incorrect tool choices |
| Argument exact match or field F1 |
Missing, incorrect, or hallucinated parameters |
| Full-call exact accuracy |
Whether both the tool and all canonicalized arguments match |
| Schema-valid call rate |
Structurally invalid tool requests |
| Execution success rate |
Invocation and dependency failures |
| End-to-end task success |
Actual agent usefulness |
| Calls per successful task |
Efficiency and unnecessary action |
| Expected-control accuracy |
Whether the agent abstains, clarifies, or requests approval when required |
Human reviewers are valuable for ambiguous semantic arguments and whether an action matches user intent. An LLM-as-a-judge can help score semantic equivalence, but should not replace schema validation or exact checks for tool names and identifiers; use multiple judges and human calibration for high-impact workflows.
Statistical Rigour
Report sample counts, per-tool results, confidence intervals, and variance across repeated runs. Model providers and open-weight systems such as Llama, Mistral, and Qwen can produce different traces across runs even at low temperature.
Treat repeated runs on the same scenario as clustered observations: use a scenario-level bootstrap or clustered confidence interval rather than treating every run as independent. Compare prompts or models on identical scenarios with paired tests. A benchmark score is evidence about that benchmark—not proof of real-world quality.
Code Example
This provider-independent harness evaluates one tool decision per row. It calculates deterministic metrics and, for independent binary cases, a Wilson interval for task success.
import json
import math
from typing import Any
def canonical(value: Any) -> str:
return json.dumps(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
)
def safe_rate(numerator: int, denominator: int) -> float | None:
return numerator / denominator if denominator else None
def wilson_interval(successes: int, total: int) -> tuple[float, float]:
if total == 0:
raise ValueError("total must be positive")
z = 1.959963984540054
p = successes / total
denominator = 1 + z * z / total
center = (p + z * z / (2 * total)) / denominator
margin = (
z
* math.sqrt((p * (1 - p) + z * z / (4 * total)) / total)
/ denominator
)
return center - margin, center + margin
def evaluate(rows: list[dict[str, Any]]) -> dict[str, Any]:
if not rows:
raise ValueError("rows must not be empty")
selected = 0
argument_cases = 0
argument_exact = 0
call_exact = 0
called_rows = []
task_successes = 0
for row in rows:
expected_tool = row["expected_tool"]
predicted_tool = row["predicted_tool"]
tool_correct = expected_tool == predicted_tool
selected += int(tool_correct)
args_match = False
if expected_tool is not None:
argument_cases += 1
args_match = (
canonical(row["expected_args"])
== canonical(row["predicted_args"])
)
argument_exact += int(args_match)
call_exact += int(tool_correct and args_match)
if predicted_tool is not None:
called_rows.append(row)
if expected_tool is None:
task_success = predicted_tool is None
else:
task_success = bool(
tool_correct
and args_match
and row["schema_valid"]
and row["execution_success"]
)
task_successes += int(task_success)
total = len(rows)
return {
"samples": total,
"tool_selection_accuracy": selected / total,
"argument_exact_accuracy": safe_rate(argument_exact, argument_cases),
"call_exact_accuracy": safe_rate(call_exact, argument_cases),
"schema_validity": safe_rate(
sum(r["schema_valid"] for r in called_rows), len(called_rows)
),
"execution_success_rate": safe_rate(
sum(r["execution_success"] for r in called_rows),
len(called_rows),
),
"task_successes": task_successes,
"task_success_rate": task_successes / total,
}
rows = [
{
"expected_tool": "search_orders",
"expected_args": {"customer_id": "C-1"},
"predicted_tool": "search_orders",
"predicted_args": {"customer_id": "C-1"},
"schema_valid": True,
"execution_success": True,
},
{
"expected_tool": "refund_order",
"expected_args": {"order_id": "O-9", "amount": 25.0},
"predicted_tool": "refund_order",
"predicted_args": {"order_id": "O-9"},
"schema_valid": True,
"execution_success": False,
},
{
"expected_tool": None,
"expected_args": None,
"predicted_tool": None,
"predicted_args": None,
"schema_valid": True,
"execution_success": True,
},
{
"expected_tool": "search_orders",
"expected_args": {"customer_id": "C-1"},
"predicted_tool": "delete_customer",
"predicted_args": {"customer_id": "C-1"},
"schema_valid": True,
"execution_success": True,
},
]
summary = evaluate(rows)
print(summary)
ci = wilson_interval(summary["task_successes"], summary["samples"])
print("Task-success 95% CI:", tuple(round(value, 3) for value in ci))
Safety and Production Practice
Test prompt injection, argument tampering, excessive tool chaining, and unauthorized destructive calls. Enforce tool allowlists, least-privilege credentials, schema validation, approval gates, and sandboxed side effects. Redact sensitive arguments and results from logs.
In production, monitor task success, invalid-call rate, retries, approval overrides, p95 latency, tool cost, and human escalation using sampled or shadowed traces. Version prompts, models, tool schemas, and evaluation data so regressions can be reproduced.