Skip to content

Awesome AI Safety#

A curated guide to tools, benchmarks and research for evaluating and reducing AI risk. Covers alignment, control, interpretability, application security, formal verification and governance.

Use the evidence at its stated scope. A benchmark score measures sampled behavior. A formal certificate covers an encoded property. A signature authenticates an artifact or statement. None establishes general safety.

Read the website · Curation methodology · October 2026 review · Contribute

Start here#

Your task Start with Check before drawing a conclusion
Evaluate a model or agent Evaluation frameworks Task coverage, elicitation, scorer validity and contamination.
Test prompt injection or harmful requests Red teaming and agent evaluations Attack access, tool permissions, benign utility and repeated trials.
Study a training intervention Alignment and control Baselines, held-out behavior and generalization after further training.
Investigate model computation Interpretability Supported models, intervention evidence and approximation error.
Restrict application behavior Runtime safeguards and security Enforcement boundaries and failure behavior.
Verify a precise property or artifact Formal verification and execution evidence The specification, identity binding and trusted assumptions.
Build a risk-management process Governance and assurance Applicable context, current official guidance and unresolved risks.

Reading labels: Research means a reference implementation or experimental method, with no promise of ongoing maintenance. Historical means archived or superseded. Restricted means public access carries additional use or model terms. Other code entries have documented use instructions and an inspected project license; inclusion is not a security audit. Dataset and model terms can differ from code terms.

Contents#

Evaluation and red teaming#

Evaluation frameworks#

  • Inspect AI - Model and agent evaluations with composable tasks, scorers, tools, sandboxes and inspectable logs.
  • Inspect Evals - Benchmark implementations and documented tasks for Inspect; individual tasks have their own dependencies and terms.
  • LM Evaluation Harness - General language-model benchmark runner with reusable task definitions, including safety-relevant evaluations.
  • METR Public Tasks - Public agent capability tasks, with instructions for running legacy tasks through an Inspect bridge; a subset of METR's evaluation work.

Red teaming#

  • Adversarial Robustness Toolbox - Attacks, defenses and evaluation tools for evasion, poisoning, extraction and inference across supported machine-learning models.
  • PyRIT - Generative AI red-teaming framework with attack strategies, targets, scoring and conversation records.
  • garak - Scanner that probes language models for failure modes such as injection, leakage and harmful generation.
  • promptfoo - Application evaluations and red-team tests with configurable prompts, assertions and attack plugins.
  • HarmBench - Research. Harmful-behavior test cases, attack methods and classifiers for comparing automated red teaming.
  • JailbreakBench - Research. Jailbreak robustness benchmark with shared behaviors, threat models and attack artifacts.
  • StrongREJECT - Research. Jailbreak benchmark and scorers that assess refusal and the usefulness of harmful responses; results depend on the judge.
  • LLM Attacks - Research. Reference implementation of gradient-based adversarial suffix attacks on aligned language models.

Agent evaluations#

  • AgentDojo - Environment for testing indirect prompt injection attacks and defenses in tool-using agents.
  • AgentHarm - Restricted benchmark. Harmful multi-step requests with synthetic tools and benign counterparts; use is limited to improving AI safety and security.
  • Petri Bloom - Research. Generate targeted behavioral scenarios and score model interactions; preserve seed configurations and review judges.
  • Inspect Petri - Research. Auditing agent that explores alignment hypotheses using simulated interactions and rubric-scored transcripts.
  • CyberGym - Vulnerability-analysis and proof-of-concept reproduction tasks on real software; requires substantial local infrastructure and containment.
  • Cybench - Capture-the-flag agent tasks with intermediate subtasks; report guided and unguided results separately.

For an evaluation report, record the threat model, model and harness versions, permissions, budgets, benign utility, scorer validation, failures and uncertainty. See the evaluation checklist.

Alignment and control#

Post-training and preference learning#

Training tools implement optimization procedures. Preference fit and refusal behavior alone do not establish robust alignment.

  • TRL - Language-model post-training library with supervised fine-tuning and preference or reinforcement-learning trainers.
  • OpenRLHF - Distributed reinforcement-learning and preference-training framework built around Ray and vLLM.
  • Alignment Handbook - Training recipes and released artifacts for supervised and preference-based language-model fine-tuning.
  • Direct Preference Optimization - Research. Reference implementation of direct policy optimization from preference comparisons.
  • RewardBench - Reward-model evaluation tasks and scoring code for preference-training research.
  • OpenUnlearning - Compare unlearning methods and metrics on TOFU, MUSE and WMDP; reduced benchmark performance does not establish irreversible deletion.

Control and monitoring#

  • ControlArena - Framework for testing control protocols, monitors and adversarial policies in agent environments with benign and sabotage tasks.
  • Monitorability Evals - Research data. Public evaluation splits and prompt mappings for chain-of-thought monitorability; private splits are omitted and some evaluations are deprecated.

Activation interventions#

  • Representation Engineering - Research. Read and steer language-model behavior using learned representation directions.
  • Circuit Breakers - Research. Training methods that reroute representations associated with harmful behavior in the tested settings.
  • Honest LLaMA - Research. Inference-time activation interventions evaluated for truthfulness on specific models and tasks.

Interpretability#

Libraries and analysis#

  • TransformerLens - Inspect transformer activations and conduct mechanistic-interpretability experiments on supported models.
  • SAELens - Train, load and analyze sparse autoencoders for language-model activations.
  • circuit-tracer - Generate, visualize and intervene on transcoder attribution graphs for supported models; graphs approximate model computation.
  • nnsight - Trace and intervene on neural-network internals, with local and remote execution interfaces.
  • pyvene - Interventions on model representations for activation patching, causal tracing and steering experiments.
  • CircuitsVis - Interactive visualizations for attention patterns and interpretability experiments.

Artifacts and research interfaces#

  • Gemma Scope 2 - Research weights. Sparse autoencoders and transcoders for Gemma 3, distributed under CC BY 4.0; base-model terms remain separate.
  • Neuronpedia - Explore model features, activation examples and attribution graphs through an interactive research interface.
  • Transformer Circuits - Research articles on transformer mechanisms, sparse features and circuit tracing.

Features and graphs are hypotheses about computation. Validate interpretations with interventions and task-level evidence; a readable feature label does not establish a safety mechanism.

Runtime safeguards and security#

Application controls#

  • NeMo Guardrails - Programmable dialog and application rails for language-model systems.
  • Guardrails AI - Schema and custom-validator checks for language-model inputs and outputs; guarantees depend on what each validator actually checks.
  • Llama Guard - Restricted models. Input/output content-safety classifiers; model terms differ from PurpleLlama's evaluation-code license.
  • Inspect Sandboxing - Evaluation execution guide; only work explicitly routed through the sandbox interface runs inside that boundary.
  • OWASP AI Agent Security Cheat Sheet - Guidance on tool permissions, memory separation, approvals and monitoring for agent applications.

Model supply chain#

  • Model Transparency - Sign and verify model-file hashes and signer identity; signatures do not establish benign model behavior.
  • Fickling - Static analysis and loading checks for pickle-based ML artifacts; detection has incomplete coverage.

Benchmarks and datasets#

These are bounded research artifacts. Code, data, model weights and underlying assets may have different licenses. Check the exact release before use. Public tests are vulnerable to contamination, judge errors and coverage gaps.

Harmful content and bias#

  • RealToxicityPrompts - Prompts and annotations for studying toxic language-model continuation.
  • ToxiGen - Historical dataset. Machine-generated text for studying implicit and adversarial hate speech.
  • SafetyBench - English and Chinese multiple-choice questions across safety-related categories; knowledge questions do not measure deployed behavior.
  • BBQ - Question-answering benchmark for social bias under ambiguous and disambiguated contexts.
  • WinoBias - Gender-bias evaluation and data for coreference resolution.
  • CrowS-Pairs - Paired-sentence benchmark for social biases in masked language models; its authors warn it may not reliably measure bias.
  • Winogender - Minimal-pair sentences for gender-bias analysis in coreference resolution.

Truthfulness and grounding#

  • TruthfulQA - Questions designed to elicit commonly held false beliefs; measures truthfulness on this distribution.
  • HaluEval - Generated and human-annotated samples for studying hallucination recognition in language models.
  • Vectara Hallucination Leaderboard - Summarization consistency evaluation using a commercial judge and a particular task setup; rankings are not general hallucination rates.

Capabilities and behavioral probes#

  • WMDP - Hazardous-knowledge multiple-choice questions in biology, chemistry and cybersecurity; a proxy rather than an operational capability test.
  • LABBench2 - Scientific research tasks and an evaluation harness; measures research capabilities, not harmful-use or wet-lab outcomes.
  • CyberSecEval - Cybersecurity evaluation code from PurpleLlama; task suites measure different attack and assistance behaviors.
  • MACHIAVELLI - Text-game benchmark of agent competence and harmful or unethical choices in simulated scenarios.
  • DecodingTrust - Evaluation suite covering multiple trustworthiness dimensions on specified tasks and models.
  • Anthropic Model-Written Evaluations - Historical data. Model-generated evaluations of persona, sycophancy, risk-related behavior and bias, released under CC BY 4.0.

Formal verification and execution evidence#

Neural-network properties#

Formal results apply to specified networks, input domains, numerical semantics and properties. They do not verify unrestricted language-model behavior.

  • alpha-beta-CROWN - Bound propagation and branch-and-bound for specified neural-network properties.
  • auto_LiRPA - Compute certified bounds for supported neural-network computation graphs.
  • Marabou - Solver for neural-network input/output properties expressed through supported interfaces and VNNLIB.
  • NNV - Reachability-based verification of neural networks and controllers; requires MATLAB and explicit system models.
  • ERAN - Research. Abstract-interpretation reference for neural-network robustness certification.
  • VNN-COMP - Competition resources, benchmarks and results for comparing neural-network verifiers.
  • Certified Adversarial Robustness via Randomized Smoothing - Paper. Probabilistic robustness certificates under a specified noise distribution and perturbation norm.

Computation proofs and signed evidence#

A computation proof covers its encoded circuit or program under the proof system's assumptions. Quantization, public/private inputs and model commitments matter. Signed receipts authenticate statements and expose some tampering; check who can observe and attest the underlying event.

  • zkml - Research. Halo2 proof-of-concept for supported ONNX inference computations.
  • Orion - Historical. Archived Cairo implementation of ONNX inference operators for verifiable computation.
  • Signet - Signed agent-action receipts and policy evidence; execution remains an assertion unless independently observed.
  • Phionyx - Research. Experimental governance gates and signed replay evidence; no independent validation of safety effectiveness is established here.
  • Remote Attestation Procedures Architecture - Standard. Roles, evidence and appraisal for remote attestation; measured platform state is evaluated under an explicit policy.

Governance and assurance#

Official frameworks and application security#

  • NIST AI Risk Management Framework - Voluntary framework organized around Govern, Map, Measure and Manage.
  • NIST Generative AI Profile - Generative AI risk-management actions that accompany the AI RMF.
  • NIST SP 800-218A - Secure-development practices for generative AI and foundation-model producers, used with the SSDF.
  • EU AI Act - Commission overview of the risk-based regulatory framework, with links to legislation and current implementation guidance.
  • GPAI Code of Practice - Voluntary guidance supporting general-purpose AI transparency, copyright and systemic-risk obligations.
  • OWASP LLM Top 10 - Application-security risks, attack scenarios and mitigations in the 2026 edition.
  • OWASP Agentic Top 10 - Threats and mitigations for agentic applications in the 2026 edition.
  • AISI Cyber Safety-Case Template - Research template connecting risk models, proxy tasks and evaluation evidence in a structured assurance argument.

Fairness assessment#

Metrics depend on the affected population, decision context and definition of harm. A score alone does not establish sociotechnical fairness.

  • AI Fairness 360 - Dataset and model fairness metrics with mitigation algorithms and examples.
  • Fairlearn - Group-fairness assessment and mitigation for machine-learning systems.
  • Responsible AI Toolbox - Interfaces for model/data exploration, error analysis and fairness assessment.

Research papers and reports#

Years below refer to the first public version. Later revisions may change results. Empirical findings describe their experimental settings; conceptual proposals describe an approach, not demonstrated deployment guarantees.

Foundations and alignment methods#

Year Resource Scope
2012 The Superintelligent Will Argument for instrumental convergence across many possible final goals.
2015 Research Priorities for Robust and Beneficial AI Research agenda spanning technical robustness, control and societal impacts.
2016 Concrete Problems in AI Safety Side effects, reward hacking, scalable supervision, safe exploration and distributional shift.
2017 Deep RL from Human Preferences Learn reward signals from comparisons of agent behavior.
2018 AI Safety via Debate Proposal for adversarial debate judged by weaker overseers.
2018 Scalable Agent Alignment via Reward Modeling Recursive reward-modeling research agenda for scalable supervision.
2019 Risks from Learned Optimization Mesa-optimization and the possibility of objectives differing from the training objective.
2019 Optimal Policies Tend to Seek Power Conditional results on power-seeking in Markov decision processes.
2021 Goal Misgeneralization in Deep RL Experiments where capabilities generalize while the learned goal does not.
2022 InstructGPT Instruction-following language-model training with demonstrations and preference feedback.
2022 Constitutional AI Critique and preference training guided by written principles.
2022 Red Teaming Language Models Human red-teaming study of language-model failure modes.
2023 Direct Preference Optimization Optimize a preference objective directly without training a separate reward model.
2023 Weak-to-Strong Generalization Study stronger models trained using weaker model supervision.
2023 AI Control Evaluate monitoring and editing protocols against intentionally subversive code generation.
2024 Debating with More Persuasive LLMs Weaker judges supervise stronger debaters on information-asymmetric question answering.
2024 Towards Guaranteed Safe AI Proposed certificates relative to explicit world models, specifications and verifiers.

Failure modes and auditing#

Year Resource Scope
2024 Sleeper Agents Intentionally backdoored models can retain triggered behavior after the tested safety-training procedures.
2024 Alignment Faking Experiments on context-dependent behavior when a model is told about its training process.
2024 Sycophancy to Subterfuge Study generalization from specification gaming to reward tampering in constructed environments.
2024 Unlearning or Obfuscating? Benign relearning can recover information suppressed by the tested approximate unlearning methods.
2025 Emergent Misalignment Broad behavioral changes after narrow insecure-code fine-tuning, with model and context dependence.
2025 SAEBench Compare sparse autoencoders across interpretability and application metrics; proxy gains need not improve practical performance.
2026 AuditBench Evaluate auditing tools on models with deliberately implanted hidden behaviors.
2026 Training a Misaligned Reward Seeker Deliberate training on vulnerable reward environments and bounded tests of subsequent reward-seeking behavior.

Behavioral specifications#

  • Claude's Constitution - Published training and behavior specification; stated values do not establish consistent model adherence.

Research groups#

Links lead to primary research feeds or organizational pages. Their inclusion is a route to research, not an endorsement of every claim or policy.

Group Research route
Anthropic Alignment, interpretability and model evaluations.
Google DeepMind Responsibility, safety research and evaluation publications.
OpenAI Safety, alignment and preparedness publications.
Redwood Research AI control and evaluation of deceptive behavior.
ARC Theoretical research on explaining model behavior.
METR Evaluations of autonomous model and agent capabilities.
Center for AI Safety Risk research, evaluations and educational resources.
FAR AI Technical AI-safety research and research support.
Apollo Research Evaluating deceptive behavior and related model risks.
MIRI Research and policy work focused on advanced AI risk.
UK AI Security Institute Frontier AI risk and mitigation research.
EU AI Office General-purpose AI oversight and AI Act implementation.
NIST Measurement, standards and risk-management publications.

Learning resources#

  • BlueDot AI Alignment Curriculum - Public reading curriculum on alignment problems, methods and research directions.
  • ARENA Curriculum - Exercises in interpretability, reinforcement learning, evaluations and alignment science.
  • ML Safety Course - Technical lectures and assignments covering machine-learning safety topics.
  • MATS - Mentored AI-safety research program; admissions and participation terms vary by cohort.
  • AI Safety Camp - Collaborative AI-safety research program with project-based teams.
  • Alignment Forum - Community discussion of alignment research; posts vary in evidence and review status.
  • AI Safety Landscape Map - Community-maintained directory for discovering organizations and projects; check primary sources after discovery.

Contributing#

Propose a resource or correction through a pull request. Include primary-source evidence, license or access terms, reproduction instructions where applicable, and the gap it fills. See CONTRIBUTING.md and the methodology.

License#

The original descriptions and arrangement of this list are dedicated to the public domain under CC0 1.0. Linked resources retain their own licenses and terms.