Your fine-tuning data can make a model less safe without a single harmful example.

Slide 1 · AI SAFETY · DATA CURATION

Your fine-tuning data can make a model less safe without a single harmful example.

ONE ARROW = ONE EXAMPLE’S PUSH ON THE WEIGHTS

→ aligned with unsafe responses

GCG JAILBREAK SUCCESS · QWEN3-1.7B

−22%

relative drop, 47.7% → 37.0%, after removing 20% of training data by gradient

THE PAPER

Follow the Gradient: Tracing Safety Degradation Back to Training Data

Reddy, Abbas, Suri, Morcos, Schwab, Maini · DatologyAI, UIUC

THE IDEA

Judge data by how it changes the model, not by what it says.

Slide 2 · THE PROBLEM

Filters read the content. The damage is in the training effect.

Fine-tuning on ordinary instruction data is known to erode safety alignment, yet the responsible examples look benign.

FLAGGED BY THE GRADIENT FILTER

MATHS SOLUTION

First code attempt fails, Python traceback, model carries on

CODING TASK

“Write a Python function that…”: short, immediately compliant

WORD PROBLEM

Quantities carried through prose: “each centre will receive…”

Content filter / LLM judge

Sees maths, code, chat. Nothing harmful, so it keeps them.

LLM-JUDGE FILTERING · QWEN3-1.7B · GCG SUCCESS

47.7% → 47.3% Gradient view

Asks: does this example push the weights the same way unsafe responses do?

Slide 3 · THE METHOD · SIMPLELORIF

A cheap gradient fingerprint for every training example

1 Embed

Gradient fingerprint for 939,343 Tulu 3 examples + 43,469 unsafe PKU-SafeRLHF responses

2 Score

Similarity to centroids of the unsafe- response gradients

3 Remove

Drop the top 20% most unsafe-aligned → SafeTulu

4 Train

Fine-tune on the other 80%; test safety & capability

WHERE GRADIENTS ARE TAKEN

Last 4 blocks backprop here only

Earlier blocks forward only

WHAT MAKES IT SCALE

• Simplifies LoRIF: no curvature correction; only the ranking matters

• 32 × 32 random projections per layer, stored rank-1: 64 floats / layer

• Embedding model: Qwen3-0.6B

COST

~1 h

to embed all of Tulu 3 on 8×H100 GPUs

Slide 4 · KEY RESULT · SAFETY

Gradient filtering cut jailbreak success for all four models

Each filter removes 20% of Tulu 3. Llama 3.1 8B starts low (11.7 → 7.0). Source: Table 1, arXiv 2610.37481.

GCG ATTACK SUCCESS ON JAILBREAKBENCH · % · LOWER IS BETTER · MEAN OF 3 SEEDS

Baseline (full Tulu 3)

Random 20% removed

LLM-as-a-judge

He et al. (2024)

SafeTulu (gradient)

0% 10% 20% 30% 40% 50% Qwen3 1.7B 47.7 52.0 47.3 46.3 37.0 Llama 3.2 1B 38.0 40.0 39.0 35.3 30.3 OLMo 2 1B 48.3 48.0 46.3 49.0 41.3 Llama 3.1 8B 11.7 9.7 8.7 9.3 7.0

Slide 5 · KEY RESULT · CAPABILITY

Safer at little capability cost, and the removed data is surprising

QWEN3-1.7B · BASELINE → SAFETULU

GSM8K 52.6 → 58.1 IFEval 37.2 → 46.1 MATH-500 26.3 → 32.7 MMLU-Pro 19.8 → 22.4

Other models mostly flat; a few small dips (e.g. OLMo 2 1B MMLU-Pro 14.4 → 13.6).

OVER-REPRESENTED IN REMOVED DATA

1.71×

Maths solutions with tool- execution failures / Python tracebacks

1.74×

“Write a Python function” tasks

↑

“Narrative arithmetic”: quantities carried through prose

Ratios from the paper’s removed-vs-kept analysis. Length alone doesn’t explain it.

Authors’ hypothesis: direct, immediately compliant answers may reinforce a general habit of compliance. They call this suggestive, not causal.

Slide 6 · TAKEAWAY

Safety-aware curation has to look at training dynamics, not just text.

CRITICAL PERSPECTIVE

• Preprint, not yet peer-reviewed
• Mostly 1–2B models (+ one 8B); small models still at 30–41% GCG success
• “Unsafe” defined by one reference set (PKU- SafeRLHF); sycophancy etc. untested
• Validation AUC varies: 0.865 (Mistral-Small- 24B) vs 0.604 (Qwen-Coder)

DISCUSSION

Should gradient-based safety audits become a standard step before fine- tuning, or is it cheaper to re-align afterwards?

REFERENCES

arxiv.org/abs/2610.37481

huggingface.co/datasets/DatologyAI/SafeTulu

huggingface.co/spaces/DatologyAI/SafeTulu-Space