Originalarbeiterreichbar
How Anthropic enables self-service data analytics with Claude
Bericht von Anthropic vom 03.06.2026 über die eigene Datenanalyse mit Claude und Skills. Er nennt Anteile und Treffsicherheit aus den eigenen Prüfläufen des Herstellers.
geprüft 10.11.2026
Worauf sich diese Seite beruft, wörtlich, abgerufen am 12.10.2026:
At Anthropic, 95% of business analytics queries are automated via Claude, with ~95% accuracy in aggregate.
bestätigt 10.11.2026is a folder of markdown the agent reads on demand
bestätigt 10.11.2026exceed 21% on our evals
bestätigt 10.11.2026Adding skills gets these numbers consistently above 95% in aggregate and regularly around 99% in certain domains.
bestätigt 10.11.2026If we can do that, then the resulting execution and SQL becomes trivial.
bestätigt 10.11.2026the agent calls a function and gets one number, the same number every other surface in the company produces
bestätigt 10.11.2026Show your work (filters, inclusions/
bestätigt 10.11.2026exclusions, freshness) for restricted data, return the SQL for the user to run themselves
bestätigt 10.11.2026how fresh the underlying data is, and who owns the model
bestätigt 10.11.2026no deterministic way of proving the correctness
bestätigt 10.11.2026We watched our offline accuracy drift from ~95% at launch to ~65% over a month before we treated this as an engineering problem.
bestätigt 10.11.2026so the PR that changes a model is the same PR that updates the doc describing it
bestätigt 10.11.2026A code-review hook flags any reporting-model change
bestätigt 10.11.2026Roughly 90% of our data-model PRs now include a skill change in the same diff.
bestätigt 10.11.2026a handful of canonical datasets, a few dozen offline evals, and a thin knowledge skill will capture most of the upside
bestätigt 10.11.2026Some of the processes we discussed may be overkill if
bestätigt 10.11.2026curate a small set of canonical, single source-of-truth datasets that are clearly owned, consumption-ready, and discoverable, then aggressively deprecate the near-duplicates
bestätigt 10.11.2026Nearly all data code (i.e., modeling, semantic layer, reference docs, canonical dashboard definitions) lives in a single repo, with CI checks that protect cross-layer integrity.
bestätigt 10.11.2026Our agents are structurally required (by skill instruction) to leverage the semantic layer first
bestätigt 10.11.2026was net-negative on our evals versus a smaller, human-curated layer
bestätigt 10.11.2026we recommend generating the documentation with Claude
bestätigt 10.11.2026giving the agent raw retrieval access to thousands of prior queries moved accuracy by less than a point
bestätigt 10.11.2026narrows the space to a few dozen curated files before a query is ever written
bestätigt 10.11.2026Our reference docs describe tables (grain, scope, and exclusions)
bestätigt 10.11.2026encodes the process a senior analyst would follow: clarify the question, find sources (via the knowledge skill), run the query, and then loop the result through adversarial review sub-agents
bestätigt 10.11.2026It also bundles a dozen reusable analysis patterns (retention curves, rate decomposition, funnel analysis)
bestätigt 10.11.2026the same skill must provide the same answer to questions in Slack, in the IDE, in a dashboard tool, and in standalone agent sessions
bestätigt 10.11.2026On merge, the skill syncs to a plugin marketplace (for IDE users), to cloud-storage blobs (for hosted apps that read a single file), and is served directly as resources over MCP.
bestätigt 10.11.2026increased accuracy by 6% within our eval set, but at the cost of 32% more tokens and 72% higher latency
bestätigt 10.11.2026swapping the adversarial reviewer to a cheaper model to cut latency (it lost most of the accuracy wins, for no real speedup)
bestätigt 10.11.2026One way of addressing this gap is via offline evals, which are simple question / answer pairs.
bestätigt 10.11.2026Dashboard-based evals are auto-generated by Claude (then human validated), covering the most common stakeholder questions.
bestätigt 10.11.2026We also continuously harvest every time a stakeholder corrects the agent in a thread as that correction is a candidate eval.
bestätigt 10.11.2026Pin every eval to a snapshot date, write it against a stable fact table
bestätigt 10.11.2026Every run lands in a warehouse table with the skill version, git SHA, model ID, per-assertion pass/
bestätigt 10.11.2026fail, token count, and wall-clock. until their slice of the eval set clears some threshold (we initially used ~90%)
bestätigt 10.11.2026diminishing returns past a few dozen per topic
bestätigt 10.11.2026Every meaningful skill edit gets a before / after run on the relevant eval slice, with the delta in the PR description.
bestätigt 10.11.2026Each run only takes an hour and replaces a lot of arguments.
bestätigt 10.11.2026A scheduled agent scans stakeholder channels every few hours for similar correction language, drafts a one-line fix to the relevant reference doc, and opens a PR tagged to the domain owner.
bestätigt 10.11.2026Both feed a dashboard reviewed weekly alongside the offline pass rate.
bestätigt 10.11.2026We also regularly prune skill scaffolding as models improve and previous failure modes no longer apply.
bestätigt 10.11.2026members of the Data Science and Data Engineering team
bestätigt 10.11.2026Offline eval accuracy should be ~100%
bestätigt 10.11.2026that ceiling drops with each new model generation
bestätigt 10.11.2026Accuracy moved by less than a point in either direction.
bestätigt 10.11.2026About 80% of the time, yes.
bestätigt 10.11.2026the flip rate was flat
bestätigt 10.11.2026stacking additional rounds of doc refinement past a certain point (we hit three consecutive net-negative iterations
bestätigt 10.11.2026The answer is wrong, but looks plausible and is used without objection.
bestätigt 10.11.2026explicit human sign-off on anything leadership-bound
bestätigt 10.11.2026How important is a correct answer today vs. in the future
bestätigt 10.11.2026How technical is the intended audience of the output
bestätigt 10.11.2026for data scientists who can recognize when an answer is incorrect, you may be more tolerant of errors
bestätigt 10.11.2026How much are you willing to spend for improved accuracy
bestätigt 10.11.2026What is your comfort around access controls and internal data privacy
bestätigt 10.11.2026The most important aspect of ensuring analytics agents are accurate is via strong data foundations
bestätigt 10.11.2026this level of accuracy doesn't tell you your system isn't going to produce a wrong answer
bestätigt 10.11.2026
Semantic Layer im Glossar11 Eine Wissensbasis aufbauen13 Daten auswerten