Web Search API Eval

Web Search API Eval

A live, side-by-side benchmark of four web search APIs for AI agents (You.com, Tavily, Exa and Parallel) on 30 agent-style queries across GTM, insurance and finance use cases, plus a buyer memo, a target-account list and a market map built on the results.

By Pallavi Singla · final run Oct 8, 2026

**Live site: https://psinglap.github.io/websearch-api-eval/**

Contents

• The four pages
• Use it in Claude or ChatGPT
• Results at a glance
• Results by use case
• Methodology
• Metrics and the benchmarks they follow
• Developer setup and friction
• Run it yourself
• Limits
• Sources
• Repo layout

The four pages

Use it in Claude or ChatGPT

Ask about these results from inside your AI assistant with the read-only MCP server. It needs no sign-in or API keys and never runs searches.

Server URL: https://websearch-api-eval.psberkeleymba.workers.dev/mcp

Then ask, for example:

• "Which web search API is fastest for a voice agent?"
• "Compare Exa and Parallel on finance."
• "What was question F5 and who got it right?"
• "Show me the overall scorecard chart."
• "How do I run this benchmark on my own queries?"

Answers come from the published results (run Oct 8, 2026) and link back here. Support · Privacy Policy · Terms. Tools, data and deployment are documented in mcp/.

Results at a glance

Headline: You.com is the fastest API tested and the most dependable at returning something fresh: it leads on freshness, reliability and search-only speed. Parallel leads accuracy, with You.com a close second and tied on fully correct queries. Exa returns the most on-topic top 5; Parallel is cheapest per correct answer.

Which API fits which agent (from this test):

Results by use case

Accuracy score (% of max) and search-only client latency p50:

You.com answered all 30 search-only calls in under 1 second; no other provider managed more than 3 of 10 in any use case. Full scorecards, per-query grids and what each API returned are on the Evaluation page.

Methodology

Queries. 30 queries, 10 per use case (GTM G1–G10, Insurance I1–I10, Finance F1–F10). Each is written the way an AI agent calls web search:

Every use case has at least 4 real-time queries; the rest use 24-hour, 48-hour, 7-day or 1-year windows. Some queries expect "none" as the right answer, to test whether an API presents stale pages as current.

Same request for everyone. Each provider gets the exact query string with the same parameters: 10 results, full page content where supported, the freshness window, and domain filters where supported.

Runs. Searches ran in promptfoo with Python providers: 3 runs per query, 90 calls per API. Latency is client-side wall-clock time around the HTTP request only, on a persistent connection after one discarded warm-up call; provider-reported server latency is recorded alongside. Search-only speed comes from a run of first-time queries sent one at a time, so caching doesn't flatter repeats.

Grading. Answers were written and graded from each provider's saved results (top 5), using the published SimpleQA grader rubric for accuracy and FreshQA strict rules for "none" answers and stale results. Answer keys were captured live at run time and archived to the Wayback Machine.

Evidence kept per call. Raw provider JSON, result dates and how each date was found, relevance labels, grader output, and an automatic log of every API anomaly.

Metrics and the benchmarks they follow

With 10 queries per use case, results are reported per query ("won 7 of 10") rather than as percentage-point margins.

Developer setup and friction

Each signup was screen-recorded from sign-in to a copied API key, then to a first successful call.

Every provider worked on the first call. The full timed friction log is on the Evaluation page.

Run it yourself

The full test harness is in eval/: promptfoo configs, Python providers for each API, the assertions behind every metric, ground-truth pullers, answer keys and grades. Metric definitions are in eval/METRICS.md and the query set is in reports/Agent queries.md.

The 30 questions

Every API gets the same search call and parameters. The answer keys (expected answer, source link, Wayback snapshot) are in eval/answer_keys.csv, and the test files built from these questions are in eval/tests/.

GTM agent (developer and company buying signals): 10 questions

Insurance agent (voice and claims, live catastrophe status): 10 questions

Finance agent (analyst Q&A with primary sources): 10 questions

Steps

cd eval
./run.sh setup --playwright   # Python venv + requirements, promptfoo 0.124.0, Chromium
cp .env.example .env          # add your own API keys; .env is git-ignored
./run.sh smoke                # one call per provider to check keys
./run.sh run gtm              # also: insurance, finance
./run.sh analyze              # scorecards and charts into eval/out/

You need your own keys for each search API (YDC_API_KEY, TAVILY_API_KEY, EXA_API_KEY, PARALLEL_API_KEY, plus FIRECRAWL_API_KEY and LINKUP_API_KEY for the six-provider configs) and an SEC_USER_AGENT with your contact email for SEC EDGAR. No keys are stored in this repo. Full steps are in eval/README.md.

Included for audit: answer keys with source and Wayback links (eval/answer_keys.csv), per-query grades (eval/grading/), raw ground-truth snapshots pulled at run time (eval/ground_truth/raw/), the evidence log (eval/evidence/evidence_log.csv) and the friction log (eval/friction_log.csv). Raw provider responses are not included because of their size.

Limits

• 10 queries per use case is a focused test, not a leaderboard; differences of a query or two are within noise.
• Results reflect one run day (Oct 8, 2026); freshness and accuracy depend on what was published that day.
• Prices are vendor list prices as of the run date; cost covers search only, not the answering model.

Sources

• Harnesses and grading: promptfoo · You.com web-search-api-evals · Tavily tavily-search-evals · SimpleQA · FreshLLMs / FreshQA · htmldate evaluation · Wayback Machine Save Page Now
• Third-party benchmarks for comparison: Artificial Analysis Search Index · Openbenchmarks web search
• Market and customer facts on the Target accounts and Search Atlas pages come from public sources (vendor case studies, docs, pricing pages, press) and are linked on each page.

Repo layout