Counterparty Check › Benchmark

Counterparty check benchmark: data tables

Can an AI agent tell whether a company is active, sanctioned or insolvent? We asked the same question about 25 companies twice: once to a model with web search, once to the same model with the result of one counterparty-check call. Run of 2026-11-09, one run per company.

This page holds the tables. How we ran it, the four bugs the first run found and the rerun are on the blog: We lost our own benchmark.

Result

The 25 companies are hand-picked and lean towards hard cases: sanctioned, dissolved or in liquidation. A difference of one or two companies is noise, so the totals are a tie. The row we'd look at is sanctions. In this run the web arm did not flag 2 of the 6 sanctioned companies (it left one open and gave no usable answer on the other); the API arm flagged all 6. In the first run each arm missed one: the web arm called Rosneft Marine (UK) clean, and our API returned an error for Garantex Europe.

On well-known companies the web arm is cheaper and faster, because the model answers from memory (9 of 25 without a single search). That says nothing about a supplier nobody has heard of. On the 12 hard cases both arms cost about the same per company.

The first run

The first run (2026-11-08) went the other way: web search 23/25, counterparty-check 17/25. It showed four bugs in our check. UK insolvency only read a window of Gazette notices. An Estonian company deleted from the register returned an error instead of a report. Companies in liquidation in France and Germany came back as active. And sanctions were matched on the name only, so a listed company that has since been renamed would slip through; a listing that carries the company's registration number now counts as a hit. We fixed all four and ran the same 25 companies again. The web arm went from 23 to 19 with nothing changed on its side, so the spread between two runs of a model with web search is about as large as the gap between the arms.

Where the API still says unknown

All 5 unknown answers in the API arm are about insolvency. Denmark and the US have no insolvency source we can use: the Danish CVR register refuses scripted requests unless you sign up for its system access, and US bankruptcy filings sit in PACER, which is paid per page. A company deleted from the Estonian register is no longer in its open data. In those cases the check says unverified and the model answers unknown, which we count as not correct.

What this does not show

• We built the API arm and we wrote the ground truth. We checked every field by hand on official sources (dated, with URLs); where a register had no free access we used GLEIF, VIES or press, or left the field unscored.
• Both arms use one model (Claude Sonnet 5.5) at low effort, with at most 4 searches and 4 fetches for the web arm. A stronger setting costs more and may score differently.
• Some web-arm misses were the tool, not the model: a search that hit the cap, a run that ended mid-search, a fetch that errored. An agent in production hits those too, so they count.
• Scoring is an exact match on three fields of the JSON at the end of each answer. No model judges the answers.
• The API arm still needs a model to read the result, and its reading counts. For Wirecard the check said review (insolvency notices matched on the name only) and the model concluded winding up from them.
• One label is arguable. Camaïeu still shows as active in Sirene and BODACC has a liquidation with no closing notice, so we scored it winding up; the web arm found a page that says struck off.

Per company

Status, sanctioned, insolvent. n/s = not scored (the ground truth could not be verified); unknown = the answer left the field open.

Ground truth checked 2026-11-08. Model claude-sonnet-5-5. Total spend of the run $2.92. The questions, ground truth with source URLs and both arms' full answers are on Hugging Face. Questions or a company you think we scored wrong: support@boscolabs.dev.