ResearchAnthropicConfirmed
Story 12 of 15 in the Friday 13 November editionAnthropic publishes 10 safety evaluation design recommendations
Published on 13 November 2026 on Anthropic’s research blog, the guide covers preregistration, offline sandboxes, multi-turn rubrics, LLM grader calibration and statistical confidence intervals.
Edited by Ashutosh Shrivastava
14 Nov 2026, 09:50 IST · 2 min read
Anthropic published 10 recommendations for building and reporting AI safety evaluations on 13 November 2026, drawing on psychology’s replication reforms and its own system-card suites across 16 policy areas and seven languages. The guide covers offline sandboxing after Anthropic paused many internet-enabled evals, causal versus observational designs, appropriate-response rubrics beyond blunt refusal rates, LLM grader calibration and paired sample-size floors.
Study details
- Recommended sample floors
- At least 100 items for complex multi-turn simulations and 1,000 items for single-turn evaluations
- Anthropic single-turn suite scope
- 16 policy areas across 7 languages and multiple product surfaces (94% to 98% API baseline saturation)
- Grader validation rules
- Atomic binary criteria, cross-family judge models, flipped pairwise ordering and blind human agreement
Release details & context
Multi-turn appropriate response rates and wide API confidence bounds
Rather than scoring safety purely by flat refusals, which fail users in self-harm crises or legitimate dual-use biology and cyber queries, Anthropic’s multi-turn simulations fix the opening user turn, vary persona and escalation tactics via a separate simulator model, and fail an entire transcript on a single unsafe turn.
Anthropic noted that multi-turn API confidence intervals can exceed plus or minus 10 percentage points, citing a romance-scam evaluation where a 98% versus 94% gap between Claude Opus 5 and Opus 5.5 proved statistically indistinguishable from sampling noise.
Sources
- PrimaryAnthropic Research: Building strong safety evaluationsanthropic.com
- On XAnthropic (@AnthropicAI) safety evaluations post on Xx.com
Anthropic release timeline (39 launches tracked)
- Anthropic launches Claude Whiteboards, Sheets and generally available Claude Data
Anthropic rolls out multiplayer Claude Whiteboards and Sheets in beta alongside generally available Claude Data across paid plans.
- Anthropic launches roadshow for proposed Nasdaq IPO at $70 to $78 per share
Anthropic opens its Nasdaq IPO roadshow under ticker ANT for 825 million Class A shares at an expected $70 to $78 per share.
- Anthropic rolls out Claude Meetings beta for live Google Meet calls
Anthropic rolls out Claude Meetings in beta on Team and Enterprise plans so Claude joins Google Meet calls to co-create live docs, slides and whiteboards.
- Anthropic opens self-hosted Claude Managed Agents in private beta
Anthropic opens self-hosted Claude Managed Agents in private beta so enterprises can host the agent server, PostgreSQL store and sandboxes themselves.