Follow the gradient: tracing safety degradation back to training data

Problem

Fine-tuning on seemingly harmless instruction data can quietly erode an LLM's safety. Because such data contains no harmful content, toxicity filters and LLM safety judges have difficulty catching it. Existing gradient-based attribution methods were costly and could only be used for small-scale analysis.

Method

The authors built SimpleLoRIF by removing the curvature (Hessian) correction from LoRIF (low-rank influence functions) and limiting backpropagation to the last four transformer blocks. Each layer's gradient is randomly projected and then decomposed to rank 1, so it is stored as 64 floats per layer. Each training example is scored by its gradient similarity to a target set (unsafe responses). The method was applied in two settings. (1) In an emergent misalignment setup, they verified that the gradients of insecure code examples align with those of misaligned responses. (2) On Tulu 3, they removed the top 20% of examples whose gradients were most similar to SafeRLHF unsafe responses, creating SafeTulu.

Results

• All 939,343 Tulu 3 examples were embedded in about one hour on eight H100s. The four-block setting kept a math detection AUC of 0.684 and ran 91.53% faster than the baseline simplified LoRIF.
• Emergent misalignment: in the gradient space of a model fine-tuned on insecure code, the AUC for separating insecure code examples from secure code examples was 0.865 for Mistral and 0.604 for Qwen.
• Training on SafeTulu reduced GCG attack success rates by up to 22% compared with standard training across Qwen, Llama and Olmo. The authors reported that it outperformed LLM-as-a-judge filtering and an existing gradient-based method (He et al.).
• The authors reported that performance on general, coding, math and medical benchmarks was largely maintained or even improved.
• Analysis of the removed data identified less-studied factors that degrade safety, such as tool execution failures, arithmetic word problems and direct coding tasks.

Limitations

The unsafety criterion depends on SafeRLHF responses and GCG attack success rates. The 20% removal ratio is a fixed setting. In the appendix, the authors state that one model's gradient representations do not always transfer to another model's responses. The AUC in the Qwen setting (0.604) is only slightly above chance. The experiments for choosing the number of blocks were run on math data.

Why it matters

This is a practical method for filtering out, at the scale of millions of examples, safety-degrading data that content filters miss during post-training data curation. Using the released SafeTulu and Hugging Face Space, readers can try it themselves with a different removal ratio or target subset.

Key terms

Training data attribution (TDA)

A method for estimating which training examples gave rise to a specific model behavior. It usually compares the gradients that examples induce.

Emergent Misalignment

A phenomenon in which fine-tuning on a narrow set of undesirable data, such as insecure code, leads to broadly harmful behavior even in areas unrelated to coding.

GCG attack

An automated jailbreak technique that appends an optimization-found suffix string to a prompt to bypass a model's safety refusals.

LoRIF

An attribution technique that makes influence function computation feasible even for large LLMs by decomposing projected gradients into low-rank form for storage.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Saket Reddy, Amro Abbas, Anshuman Suri, Ari Morcos, David Schwab, Pratyush Maini · Posted 2026-10-27 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.