Blog › We lost our own benchmark: web search vs. one API call

We lost our own benchmark: web search vs. one API call

Bosco Labs · 9 November 2026

If you build an agent that onboards suppliers or customers, sooner or later it has to check that a company is still registered, and that it isn't sanctioned or insolvent. We make an API that checks this against official registers, so we wanted to know if an agent needs it at all, or if a decent model with web search gets there by itself.

So we ran a small benchmark, and the first run went badly for us: web search got 23 of 25 companies fully right, our API 17. This post covers how we measured it, the four bugs that run turned up, the rerun, and the cases where our API still answers "unknown".

The setup

We picked 25 companies from 10 countries. Thirteen are large, healthy and well known (Tesco, Novo Nordisk, Apple, Siemens and so on). The other twelve are the cases a check exists for: six are on a sanctions list, and the rest are dissolved, in liquidation or bankrupt (Carillion, Wirecard, Cambridge Analytica's UK company, Getin Noble Bank).

For each company we wrote down three fields: register status (active, winding up or dissolved), sanctioned (on the UK, OFAC SDN or EU list) and insolvent. Each field was checked by hand against an official source, with the URL and the date. Where we couldn't verify a field, it isn't scored. That happened once, for the status of Lightyear's Dutch entity, because the Dutch register has no free access.

Both arms use the same model (Claude Sonnet 5.5, low effort) and get the same question:

• The web arm has Anthropic's server-side web search and web fetch, at most 4 of each, with our own domain blocked.
• The API arm makes one POST to our counterparty-check endpoint with the registration number, LEI or name, and the model answers from the JSON it gets back.

Each answer ends with a small JSON object, and we compare the three fields with the ground truth by exact match. There's no model acting as judge. "Unknown" counts as not correct.

Run 1: 23 to 17, and four bugs

The web arm won the first run clearly. When we went through our misses, four of them were plain bugs on our side.

• UK insolvency only looked at The Gazette notices in our own copy, which starts in 2022. Cambridge Analytica (UK) Limited went into administration in 2018, so it came back with no insolvency at all. The check now reads the company's full insolvency history from Companies House as well.
• Garantex Europe, a sanctioned crypto exchange operator, has been deleted from the Estonian register. Our API answered its registry code with an HTTP 400, so the agent got nothing back, sanctions included. It now returns a report saying the company isn't in the register, and still screens it.
• Camaïeu in France and Wirecard in Germany came back as active, because Sirene and GLEIF still say active while a liquidation runs. An open liquidation notice that carries the company's number is now a fail. Notices that only match on the name, which is all the German register gives you, are flagged for review.
• Sanctions were matched on the name only. Garantex Europe has since been renamed, so its current name matches nothing, while the UK and OFAC entries both carry its Estonian registry code. A listing with the company's registration number or LEI now counts as a hit even when the name differs.

Each fix got a regression test, and we ran the same 25 companies again.

Run 2: 19 to 20

20 against 19 is one company, and with 25 companies and one run each that's noise. Our score went up because we fixed bugs. The web arm's score went down by four with nothing changed on its side. For us that's the most useful number in the whole exercise: on the same 25 companies, two runs of a model with web search were as far apart as the two arms.

Sanctions is the row we'd look at. In run 2 the web arm didn't flag two of the six sanctioned companies. For Petropars UK it ran into the search cap, a fetch was refused, and it answered unknown. For Europäisch-Iranische Handelsbank the run ended in the middle of a search with no answer. In run 1 it called Rosneft Marine (UK) not sanctioned. Over both runs it never flagged a clean company. Some of those misses come from the tools, and an agent in production runs into the same caps and errors, so we counted them.

The score also hides how the web arm got there. In 11 of its 25 answers in run 2 the model said itself that it hadn't checked the sanctions lists directly, and then answered from what it knew. That's usually right for Tesco. It's not something you can show an auditor. The API arm's answers name the list entry and the list version, for example "UK Sanctions List RUS1421, registration number 14850239".

On cost and time it depends on the company. For the 13 well-known ones the web arm was cheaper, about $0.03 against $0.07, because the model mostly answers from memory: 9 of the 25 answers used no search at all. That works for Novo Nordisk and tells you nothing about a small supplier nobody has written about. On the 12 hard cases both arms cost about the same, $0.067 against $0.069, and the web arm took 19 seconds on average against 16, with a slowest answer of 40.

Where our API still says unknown

All five unknowns in the API arm are about insolvency: Novo Nordisk and Maersk (Denmark), Apple and JPMorgan (US), and Garantex Europe (Estonia).

Denmark's CVR register refuses scripted requests unless you sign up for its system access, so we take the Danish status from GLEIF and have no Danish insolvency source. The US has no national company register, and bankruptcy filings sit in PACER, which charges per page. An Estonian company that has been deleted drops out of the register's open data, so there's nothing left to read. In those cases the check returns unverified and the model says unknown. We'd rather it said that than guess "no", and it costs us five points in the table.

What this doesn't tell you

We built the API arm and we wrote the ground truth, the obvious problem with any vendor benchmark. Every field has its source URL and check date, so you can check our work. One label is arguable: Camaïeu still shows as active in Sirene and the liquidation has no closing notice, so we scored it as winding up, while the web arm found a page that says struck off.

The set leans much harder on difficult cases than a normal supplier list would. Both arms ran one model at low effort with capped searches. A stronger setup would cost more and might score differently. The API arm still needs a model to read the result, and that reading counts: for Wirecard the check said "review" and the model concluded winding up from the notices.

Try it yourself

The full tables per company are on our benchmark page. The 25 questions, the ground truth with sources, and both arms' full answers for both runs are on Hugging Face: huggingface.co/datasets/BoscoLabs/counterparty-check-benchmark.

To try the API arm on one company, Garantex Europe for example:

curl https://api.boscolabs.dev/v1/counterparty-check \
  -H "Authorization: Bearer $BOSCO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "14850239", "country": "EE"}'

Which countries are covered for what is on the counterparty-check page, and keys and limits are in the API docs.

If you've run something similar with a different model or more searches, we'd like to see the numbers, especially where they disagree with ours.

Disclosure: this post is by Bosco Labs, which makes counterparty-check. We ran both arms, wrote the ground truth and paid for both runs ($2.78 and $2.92 in total).

Bosco Labs: official-source data for AI agents, as an HTTP API, MCP server and Apify Actors. API docs.