ToxicChat is a benchmark of roughly 10,000 real user prompts taken from the Vicuna chatbot demo. Each prompt carries a binary toxicity label and a binary jailbreaking label. It was built to show that toxicity in conversations with a chatbot looks different from toxicity on social media. A ToxicChat score tells you how well a classifier separates harmful user inputs from benign ones on English chatbot traffic of that kind. It says nothing about what your own model writes back, about your users or about your content policy. Read with that boundary in mind, it is one of the more useful public checks for a customer-facing moderation layer. Read without it, it is easy to over-trust.
The authors give the domain gap as the reason the benchmark exists. Social media users typically post their views directly. Chatbot users often ask questions or give instructions, so toxicity sits implicitly inside seemingly friendly requests. The paper reports that the moderation models and APIs it tested “fail to deliver good quality” on ToxicChat, pointing to a large domain discrepancy.
How the ToxicChat dataset was built and labelled from real chatbot queries
The source data is a random sample of conversations collected in April from the Vicuna demo. Before labelling, the authors removed noisy or non-informative content, non-English inputs and personally identifiable information. Emails, phone numbers and addresses were replaced with generic placeholders, so the syntax of each prompt stayed intact.
Annotators gave each prompt a binary toxicity label, and the final label needed a strict majority: at least three annotators had to agree. A trial round of 720 prompts reached 96.11% inter-annotator agreement and a 7.22% toxicity rate. The trial also turned up prompts that try to trick the chatbot into producing toxic output while looking harmless. The authors call these “jailbreaking” queries and gave them a label of their own. If your main exposure is that kind of input, our explainer on jailbreak safety for RAG systems, chatbots and agents covers it from the application side.
Labelling the full set used a human-AI collaborative scheme. On the trial data, the authors found that no single score threshold cleanly separated toxic from non-toxic text, but a high model confidence generally did mean high toxicity. They therefore used Perspective API as a pre-filter and treated any text scoring below 1e-1.43 as non-toxic. On the trial set, that filter missed one toxic example out of 48. The scheme removed around 60% of the human annotation workload while keeping the labels reliable. In practice only part of the dataset carries human labels, and a field marks which part (see the fields below).
After checking and removing unsuitable data, the paper reports 10,166 examples with a 7.10% toxicity rate and a 1.75% jailbreaking rate. Those rates are low, and that is deliberate. The authors aimed for a set that follows the real distribution of toxicity in user-AI conversations. A classifier that flags almost nothing can still post high accuracy on such a set. The paper scores it with precision, recall and F1.
Where to download ToxicChat, and which version and split to use
ToxicChat is published as a dataset on Hugging Face. It comes in two configurations, toxicchat1123 and toxicchat0124, and each one is split into a train set and a test set of about 5.08k rows each. The original paper describes a random 50/50 train/test split. The version name gives the update date, so 0124 is the January 2024 release.
Version 0124 fixes annotation errors found in 1123. The maintainers ran a model error analysis, had two annotators check each fix, and ended with label changes on 1.28% of toxicity labels and 0.34% of jailbreaking labels. They also added 20 human-annotated examples. The figures in the table below come from the dataset README. They differ slightly from the 7.10% and 1.75% in the paper, so always quote a rate together with its version.
| Statistic (dataset README) | toxicchat1123 | toxicchat0124 |
|---|---|---|
| User prompts | 10,165 | 10,165 |
| Human annotations | 5,634 | 5,654 |
| Toxic examples | 7.18% | 7.33% |
| Jailbreaking examples | 1.78% | 2.01% |
Each row has toxicity and jailbreaking as 0/1 integers and a human_annotation boolean. Each row also stores the OpenAI Moderation category scores for the prompt. The maintainers recommend the latest version for both training and evaluation, and ask that models only be compared on the same version. That second point matters more than it sounds. Published work does not all use the same version: GradSafe, for example, reports results on the official 1123 test set. A score from a 1123 paper and a score from a 0124 run are not directly comparable.
Precision, recall, jailbreaking recall and AUPRC on ToxicChat
The paper puts the main goal plainly: catch toxic text without over-flagging non-toxic text. It reports precision, recall and F1 on the human-annotated part of the test set. It adds jailbreaking recall, which is the share of jailbreaking prompts the model flags as toxic. Later work on moderation models, including Llama Guard and GradSafe, reports area under the precision-recall curve (AUPRC). AUPRC summarises the trade-off for the unsafe class across all thresholds and helps when choosing a threshold for a particular use case.
On the authors’ evaluation, moderation APIs and models fine-tuned on other toxicity datasets fall well behind a model trained on the benchmark’s own training split, both on toxicity and on jailbreaking text. The README reports an F1 of 0.8221 for its ToxicChat-T5-large baseline. For OpenAI Moderation (updated 25 January 2024, at a threshold of 0.02) it reports 0.6141. The WildGuard authors found that removing real-world user-LLM data from their training mix cost 10.3 F1 points on ToxicChat. A 2026 study gives prompt-level F1 baselines on the benchmark of 0.33 for Llama-3.1-8B, 0.18 for Qwen2.5, 0.62 for Phi-4 and 0.78 for Gemma.
Prompt-based adaptation counts for a lot. Llama Guard was evaluated by putting the benchmark’s taxonomy into its prompt, and it beat every other method on AUPRC even though none of the models had been trained on that data. Fine-tuned on only 20% of the benchmark’s training data, it matched a Llama2-7b trained on all of it. These results tie the gap to domain and taxonomy fit.
How ToxicChat differs from RealToxicityPrompts
The two names sound alike, but the benchmarks test different things. RealToxicityPrompts is a set of 100K sentence-level prompts taken from English web text, each paired with a Perspective API toxicity score. A prompt counts as toxic at a score of 0.5 or above, and 22K prompts meet that bar. Its purpose is to measure toxic degeneration: whether a pretrained language model continues even harmless-looking prompts into toxic text. The ToxicChat authors point out that RealToxicityPrompts was not built for conversational AI and does not cover how people actually use chatbots.
| ToxicChat | RealToxicityPrompts | |
|---|---|---|
| Source text | User prompts from the Vicuna demo | Sentence-level prompts from English web text |
| Labels | Binary toxicity and jailbreaking, strict human majority vote with a Perspective pre-filter | Perspective API toxicity scores, toxic at 0.5 or above |
| Size | About 10K, split 50/50 train/test | 100K, 22K toxic |
| What it evaluates | A classifier’s ability to flag toxic and jailbreaking user inputs | A generator’s tendency to produce toxic continuations |
| Known label bias | Labels follow the Vicuna demo population, over 99% English | The Perspective measure leans on lexical cues and correlates with African American English |
In short, RealToxicityPrompts tests the model that generates text, and ToxicChat tests the guard in front of it. A customer-facing system needs evidence on both sides, and neither benchmark can stand in for the other.
What does a ToxicChat score tell you about a moderation model for your chatbot?
A good score tells you that a classifier flags toxic and jailbreaking English chatbot prompts as the dataset’s annotators labelled them. That is a real signal: classifiers trained on social-media data often do badly on exactly those prompts. Several limits apply before the number transfers to your product.
- It covers prompts only. ToxicChat classifies user prompts, not model outputs, and the Llama Guard authors treat prompt and response classification as two separate tasks. A strong score says nothing about whether your model’s replies are safe.
- Its population is narrow. The data comes from the Vicuna demo, not from higher-traffic services. The authors note that those services would better reflect real user demographics and languages. The set is over 99% English and may not evaluate non-English conversations well.
- Its labels are coarse. The labels are binary, with no severity grades. A 2026 benchmark paper names this lack of granularity as a limit on detailed safety analysis.
- Its taxonomy may not match your policy. Moderation tools are trained against different taxonomies, which makes comparisons across models and datasets hard. A classifier scored against ToxicChat’s notion of toxic is not being scored against your terms of use.
- It is a static benchmark. The GuardEval authors note that static benchmarks, unlike interactive red-teaming, may not capture evolving jailbreak attempts.
The threshold is where these limits turn into costs. Even during dataset construction, the authors found no score cut-off that cleanly separated toxic from non-toxic prompts. Raising sensitivity to catch more jailbreaks blocks more legitimate requests. In one 2026 evaluation, the authors’ own moderation model produced false positives that clustered on content involving particular identity groups and on non-profane but sensitive speech. OpenAI’s own moderation guide says to treat scores as signals for your application’s policy, not as automatic blocking decisions. It also notes that a refusal which discusses harmful content can still be flagged. A benchmark result helps you pick a threshold. It does not give you one.
Running ToxicChat as a regression check in production LLM evaluation
ToxicChat works best as a fixed regression check that runs every time the moderation layer changes. It does not tell you on its own whether a feature is safe to launch.
For teams running a chatbot or agent in a production SaaS product, TechnoLynx can build the loop below: the eval harness, regression suite and release gate around a moderation model.
A workable loop has a few steps that need to run in order:
- Pin the version and the subset. Use one configuration for every comparison, ideally the latest, and score the human-annotated rows of the test split, as the paper does.
- Record the full curve. Store AUPRC alongside precision, recall and jailbreaking recall at your production threshold, so that a threshold change and a model change show up separately.
- Gate releases on change, not on a public leaderboard. Compare each candidate classifier, prompt template or threshold against the last released one on the same pinned data, and block a release when jailbreaking recall or precision drops beyond a margin you have agreed in advance.
- Test responses separately. Tooling such as promptfoo’s ToxicChat plugin pulls the prompts labelled toxic or jailbreaking from Hugging Face, sends them to your model and checks whether the responses refuse or comply. That tests the generator end to end, not the classifier.
- Add your own traffic. Vendor observability tooling such as Microsoft Foundry describes continuous safety evaluation of sampled production traffic, scheduled evaluation against test datasets to detect drift, and scheduled red teaming. Your labelled production samples are the part ToxicChat cannot supply.
- Split errors by subgroup. Audit false positives and false negatives by content and demographic subgroup, and send high-uncertainty cases to human review.
Our piece on ToxicChat and detection reliability in practice looks at the alerting side of the same loop. The broader point about why GenAI fails on production data applies here too.
Before a ToxicChat number goes into a release decision, the question to settle is whose definition of toxic you are measuring against. If the answer is “the annotators of a Vicuna demo sample”, the score is a useful floor and no more. Your own labelled traffic, scored against your own policy, is still the evaluation that decides.