Skip to main content
The Deliverome ProjectSurfaceomev1.3.0

147 genes, five databases, eight models.

SurfaceBench is a 147-protein benchmark assembled from cases where the five public surface-membership databases disagree on surface status. Each protein was manually reviewed and classified as surface, contextually surface, or non-surface. Sonnet 4.6 and Opus 4.8 — with no external tools, working from training-time knowledge plus a short NCBI context block — outperform every gold-standard database on this set. Sonnet 4.6 on that NCBI prompt is the production triage configuration (marked ★ in the table). The table below shows every model × prompt-variant call so you can audit the reasoning per gene.

8 models · headline columns show the NCBI (HGNC + UniProt + NCBI summary) variant·ctx = contextual, ref = refused (the model's safety classifier declined the call)·Click a gene to compare all 5 prompt variants; click a verdict cell to read the agent's reasoning.

147 genesTSV ships verdicts + reason codes + telemetry. Free-text reasoning per cell is on GET /v1/benchmark/matrix.

UniProt
GO
SURFY
CSPA
HPA

Reason · the curator's single ground-truth reason behind each verdict, drawn from the same closed TriageReason vocabulary the triage agent must pick from — so each model's reason is directly comparable to the truth reason. Hover a truncated label for the full text.

DB columns · UniProt · GO · SURFY · CSPA · HPA — the five gating databases that drive candidate-universe membership (same set the homepage catalog shows). Each cell is one source's vote on whether the protein reaches the plasma membrane: filled dot = surface, empty ring = not on the surface in that source.

LLM columns · One compact cell per model on the headline NCBI prompt variant, ordered Haiku 4.5 · Haiku 5.5 · Sonnet 4.6 · Sonnet 5 · Sonnet 5.5 · Opus 4.8 · Opus 5 · Opus 5.5. Every model ran each prompt three times, and each cell shows the majority verdict across those replicates (refused replicates excluded) — the per-cell numbers the paper reports. Each is a verdict pill colored by call (ctx = contextual); pills outlined in maroon disagree with truth (collapsing yes ≡ contextual when scoring correctness). The tinted Sonnet 4.6 ★ column is the production configuration. Hover for the full (model → verdict + reason) tooltip; click a pill to read the agent's reasoning.

Prompt variants · Click a gene to expand the full model × variant grid. Not every model ran every variant: Haiku 4.5 and Sonnet 4.6 ran all four of no context, NCBI, NCBI + web search and NCBI + PubMed; Opus 4.8, Haiku 5.5 and Sonnet 5.5 ran no context and NCBI; Sonnet 5 and Opus 5 ran NCBI only; and Opus 5.5 ran NCBI plus NCBI · low effort — the same prompt at low reasoning effort. Cells with no run on file show a dash.

Refused · A small number of calls were declined by the model's safety classifier and returned no verdict. A refused replicate is left out of its cell's majority; a cell whose replicates all refused shows as refused (ref in the headline columns, with a dashed outline) — distinct from a dash (no run on file) and from errored (a run that returned no valid verdict) — and are not counted as agreeing or disagreeing with truth.