An NSFW detector returns a score, not a moderation decision. A classifier or hosted API returns a probability score between 0 and 1 for each image, usually with category labels such as nudity, suggestive content and violence. The platform then chooses thresholds: block above the upper one, pass below the lower one, and send the middle band to a human. The hard problems sit in that last step. Where do the thresholds go for each policy category, what goes wrong on content the model never saw in training, and how do you show afterwards that a decision was sound?
TechnoLynx’s computer vision R&D team engineers the calibration, routing and evaluation harness around any detector, hosted or not, and the self-hosted or fine-tuned path where a platform needs one: model selection, fine-tuning and GPU inference tuned for batch throughput.
What an image safety classifier scores, and what it leaves to you
An image safety classifier is a vision model trained to map an image to scores for one or more unsafe categories. Yahoo’s Open NSFW, for example, is a thin ResNet-50 (half the filters per layer) pretrained on ImageNet and fine-tuned on pornographic images, and it outputs a single score where values below 0.2 are likely safe and values above 0.8 are highly probable NSFW. Its scope is narrow by design: it covers pornographic images only, not sketches, cartoons or graphic violence.
Another is NSFW_Detector, which puts a multilayer perceptron head on the CLIP image encoder. The Stable Diffusion safety filter uses CLIP differently. It measures how close an image’s CLIP embedding sits to the text embeddings of 20 sensitive concepts and blocks the image if any distance crosses a preset threshold. Vision-language models (VLMs) take a third approach: they receive the image plus a prompt and identify whether it contains unsafe content.
For a scoring model, the platform has to choose the thresholds that fit its own policy. Open NSFW’s maintainers say so directly: when using the model as it is, build an evaluation set that matches your own definition of safe, fit a ROC curve, choose a threshold, and use human moderation for edge cases. For the request-and-response view of the hosted endpoints, our explainer on what an NSFW image detection API returns goes field by field.
How does NSFW detection work inside a moderation workflow, from upload to decision?
An upload is scored per category by the classifier or API. Each score is compared with the platform’s thresholds, and the image lands in one of three outcomes: block, human queue or pass. Hosted services expose the categories at different granularities. Amazon Rekognition uses a three-level hierarchical taxonomy. AWS recommends moderating on Level 1 or Level 2 labels and using Level 3 labels only to carve out specific concepts the platform does not want to moderate. Azure AI Content Safety classifies images into four harm categories (hate, sexual, violence and self-harm), each with a severity of Safe, Low, Medium or High. Its image model returns only severities 0, 2, 4 and 6 on the underlying 0-7 scale.
A platform maps its policy onto those outputs one category at a time. In Rekognition the confidence threshold is set with MinConfidence. When it is not set, labels are returned at 50% confidence or above. Values below 50% raise recall at the cost of more false positives, and values above 50% do the reverse. In Azure the platform picks a blocking level per category, and Microsoft notes that changing the blocking level changes which severities are blocked, not how the content is classified. Microsoft lists the factors to weigh when choosing the level. It asks about the harm of allowing content, the harm of blocking legitimate content, whether the workload legitimately discusses health, history or current events, and whether human review or other mitigations exist.
The same pixels can deserve different outcomes in different settings. In the VModA study, moderation of violent protest images depended on the severity of the violence. A school-bullying dataset flagged even mild physical conflict, because child-protection policies apply. NSFW judgements also vary across regions and communities because the law differs. A policy tier is therefore a separate threshold set per category, not one global “strictness” dial.
Three-outcome threshold bands in published configurations
| Source and setting | Pass | Human queue | Block | Status of the numbers |
|---|---|---|---|---|
| Open NSFW model documentation | score below 0.2 likely safe | middle range, binned by level | score above 0.8 highly probable NSFW | the maintainers’ guidance, to be re-fitted on your own evaluation set |
| AWS blog, Rekognition with Amazon A2I | below the review range | confidence 50-90 sent to human review | above the review range | an example configuration |
| Practitioner test of NSFW APIs (Aireiter) | below about 0.3 | middle band to an LLM-vision check, then a human | above about 0.85 | labelled by its author as an illustrative starting point only |
| Amazon Rekognition default | labels under 50% confidence not returned | set by the platform | set by the platform | API default, not a policy |
None of these rows is a recommendation for your traffic. The Open NSFW maintainers and the practitioner test both tell the reader to calibrate against representative content of their own. Ethical-Lens, an academic text-to-image alignment framework, gives an example. It set its per-category thresholds empirically, using a calibration subset of images whose toxicity was already known.
The pipeline also needs a rule for failures. If the classifier errors out, times out or returns output that does not parse, the image must go to review or a retry, never through as safe. In that practitioner test, one LLM-vision model returned “safe” for an image while admitting it could not read it. That is a silent fail-open, and nothing downstream would catch it. For the end-to-end architecture of the moderation pipeline around the detector, including the queueing infrastructure, see the companion piece.
How does inappropriate image detection sit inside the broader detection-plus-provenance stack?
Content detection answers one question: what is in this image. It does not answer where the image came from, and it is brittle in exactly the places a platform is most exposed. The UnsafeBench study evaluated conventional classifiers and VLM-based classifiers on 10,000 real-world and AI-generated images across 11 unsafe categories. It concluded that existing image safety classifiers are not comprehensive or effective enough to deal with unsafe images as a whole.
The failures fall into a few recognisable groups:
- Uneven category coverage. In UnsafeBench, sexual and shocking content averaged an F1-score close to 0.8. Hate, harassment and self-harm averaged below 0.6. The best hate-detection F1 among five capable classifiers was 0.533 (Q16). Even GPT-4V missed some hateful symbols, such as Neo-Nazi symbols in tattoos.
- AI-generated images outside the training distribution. The F1-score of the Stable Diffusion filter, for example, fell from 0.833 on real-world images to 0.727 on AI-generated ones. Artistic renderings and grid layouts of four or nine sub-images were frequently misclassified. AI-generated images also tended to receive lower confidence scores, which the authors link to higher robustness susceptibility.
- Evasion through composition. VModA documented NSFW content hidden in image-text combinations, multi-image collages, and small icons or peripheral regions. Humans can still read these images, but existing detectors miss them.
- Adversarial perturbation. Conventional classifiers kept a robust accuracy of only 0.299-0.507 on perturbed LAION-5B images. VLM-based classifiers reached 0.563-0.725. Models trained from scratch without pretraining, such as NudeNet, were the most vulnerable.
- Borderline-safe content. In the Aireiter test, NudeNet flagged a marble statue at 0.70 because it detects body parts, not meaning. Dedicated detectors also wrongly flag swimsuits, art and medical photos as explicit.
- VLM-specific behaviour. Safety alignment leads VLMs to refuse to analyse or to return sanitised output, especially on pornography. Unconstrained VLMs also make unfounded associations: Deepseek-VL classified 0.816 of near-zero-violence protest images as violent.
- Narrow built-in filters. Red-teaming showed that the Stable Diffusion safety filter targets sexual content and ignores violence and gore. DiffusionDB still contained NSFW images that the filter and the server moderators had both missed.
Child sexual abuse material is a separate case and does not belong in this stack. Amazon states that its moderation APIs do not detect illegal content such as CSAM. The established route is hash-matching against known databases, plus reporting under the provider’s legal duties, not a classifier you tune.
Two other layers make the posture defensible. The first is provenance: whether an image carries evidence of how it was made. That belongs with AI-generated content detection, from forensics to cryptographic provenance. The second applies to platforms that generate images themselves, which can check before the image is fully synthesized. Prompt filters are vulnerable to misspellings and vague wording. Post-generation image checks are more accurate but add latency. In-generation detection on the diffusion model’s predicted noise reached 93.77% accuracy on adversarial NSFW prompts in a 2025 study.
How should the contested confidence band be routed to human review?
Route by confidence range, per label, and keep a random sample flowing from every band, so the thresholds stay measured instead of assumed. Amazon A2I implements both halves. ModerationLabelConfidenceCheck opens a human loop when confidence for specified labels falls in a chosen range, and you can set different thresholds for each label. A separate Sampling condition sends a set percentage of all inferences, between 0.01 and 100, to humans to audit the model. AWS’s documented example always sends Graphic Male Nudity inferences below 60 confidence to review and samples high-confidence ones at a set rate.
Running the cheap decisions first keeps costs low while cutting false alarms. The tiered setup Aireiter recommends runs a fast detector on every upload. Only the middle band goes to an LLM-vision check that reads context before any human sees it, which keeps dedicated-API economics on the bulk of traffic. The second stage is not free: the LLMs took 4-14 seconds per image against NudeNet’s roughly 20 ms. The author calls it a small, informal three-image test, not a benchmark.
Once content reaches humans, the routing is a capacity problem. QUEST, a queue-simulation study of moderation at scale, assigns jobs to queues by content type, language, market and suspected violation type. It prioritises jobs by severity, predicted content views and review time, with the goal of getting the most review value from a fixed supply of reviewer hours. Its case study found that larger validation samples of classifier-flagged content lowered the misspecification rate. Its authors used simulation to choose thresholds, because the effective use of review capacity depends on them.
We would size the band from reviewer capacity and the cost of each error type, then simulate it on historical traffic before changing anything in production. Narrowing the band is the easy way to shrink the queue, and also the way harmful content gets through. The random samples from the auto-pass and auto-block regions are what show you which of the two is happening.
How do you measure detector quality without false-positive bias distorting the numbers?
Measure precision and recall per category against human labels, on a sample drawn across the whole score range, not just on what the classifier flagged. Accuracy misleads here. Violating content is a small minority, so a classifier that outputs the majority label still scores well: the 2023 analysis of “accuracy” under the EU Digital Services Act showed even a simple classifier reaching 95% accuracy because of label imbalance.
Precision is the fraction of flagged items that really violate policy, and recall is the fraction of violating items that were flagged. The same 2023 analysis ties each to a consequence. Low precision means over-moderation of legitimate expression. Low recall means harmful content stays visible. Moving the decision threshold trades one against the other.
The sources show bias entering in three ways:
- Scoring against the classifier’s own output. An unbiased estimate needs human ground-truth labels, because relying on classifiers alone builds their errors into the metric.
- Sampling only flagged content. Flagged content shows precision but not recall. The DSA analysis estimates recall efficiently by stratified sampling, binning content by classifier score, and found quantile binning outperforms equal-width binning.
- Noisy reference labels. On VModA’s pornography dataset, manual review of 82 images that disagreed with their “porn” label confirmed 62 were in fact normal or should not have been labelled pornographic. On the blood dataset, red pigment and horror visuals had been labelled as blood.
The evaluation set must also include content that should be allowed, such as educational, medical and historical images. Then both unwanted blocks and harmful content that passes are counted. Labelling that set takes expertise and care. UnsafeBench used three expert annotators instead of crowdsourcing, because identifying hate symbols needs domain knowledge and because the authors wanted to limit how many people were exposed to unsafe content.
Measurement is not a one-off. Microsoft advises repeating the evaluation whenever the model, the configuration, the workload or the input pipeline changes. Rekognition returns the moderation model version with its results, so each evaluation can be tied to a version. Our position is that this belongs to production reliability, a discipline separate from model accuracy: a task-specific evaluation harness and regression suite act as the release gate, catching silent quality drift after a model update before users do. For the wider causes of drift, see the data quality problems that degrade computer vision after deployment.
Should a platform use a hosted NSFW detection API or build its own?
For a narrow policy centred on sexual content and moderate volume, we would start with a hosted API plus your own calibration layer. Building or fine-tuning pays off when your categories, your content distribution (AI-generated images especially) or your volume move outside what the API was trained and priced for.
| Route | What the sources show | Where it fits |
|---|---|---|
| Hosted API (e.g. Rekognition, Azure AI Content Safety) | about $0.001 per image for Rekognition in one practitioner’s figures; vendors state the output is not an exhaustive or definitive filter | standard categories, fast start, vendor-maintained model |
| Self-hosted open model (e.g. NudeNet, Open NSFW) | free to self-host and about 18-26 ms per image in one test; Open NSFW covers pornography only; NudeNet was the most adversarially vulnerable type in UnsafeBench | first-pass filter on every upload, data that cannot leave your infrastructure |
| Zero-shot commercial VLM (e.g. GPT-4V) | top overall F1 in UnsafeBench but about $20 and 50 minutes per 1,000 images | second look at the contested band, off the hot path |
| Fine-tuned open VLM | LoRA-fine-tuned LLaVA reached an average F1 of 0.836 on external datasets; its robust accuracy rose to 0.952 on the test set and 0.912 on out-of-distribution AI-generated images | wide category coverage, heavy AI-generated content, teams able to curate training data |
Fine-tuning has its own requirements. UnsafeBench’s authors say that platforms must include AI-generated unsafe images in training data. Collecting diverse NSFW samples is constrained by law and by regional differences. Stitching open models together is another route: combining Q16 with NudeNet covered many categories, at an F1 of 0.665 across those categories.
At very large volumes, GPU choice and batch size drive the cost. The figures below come from document classification and embedding workloads, not image NSFW classifiers, so read them as direction, not as figures for your pipeline. An engineering post on running a billion classifications found batch size 64 optimal for classification on an NVIDIA L4 at $0.80 per hour. That came to $253.82 per billion inputs, and CPUs were not competitive at scale. The same post priced ColBERT-style vision embeddings for image retrieval at over $44,000 per billion, two orders of magnitude above the text tasks. That gap is why we would benchmark the actual image model at the actual resolution before committing to hardware.
Teams that only need a standard sexual-content filter at modest volume, with a policy that fits a vendor’s categories, need little beyond a hosted API and an in-house threshold review. We do not provide legal sign-off on whether a policy meets the law in a given jurisdiction. That stays with counsel and the platform’s trust-and-safety lead.
What audit trail does a defensible moderation decision need?
Each decision needs a record that rebuilds why it happened: the model and version, the per-category scores, the threshold configuration in force, the route taken, any human verdict, and timestamps. In the EU, each such decision also needs a statement of reasons. Article 17 of the Digital Services Act requires a clear and specific statement of reasons, and since February 2024 this applies to all online platforms regardless of size.
The DSA Transparency Database documentation sets out what a statement contains:
- the content type, which can take several values, including image and synthetic media;
- at least one restriction type: visibility, monetary, service or account. Visibility restrictions include removal, demotion, age restriction and labelling;
- whether automated means detected the content, and whether the decision was fully, partially or not automated;
- the legal or contractual ground, an explanation, and a category of illegality or terms-of-service incompatibility.
The automation fields are where a model-assisted pipeline has to be precise. Under the documentation, “fully automated” means the entire decision process ran without human intervention, and “partially automated” means both automated means and humans took part. A threshold auto-block and a human-confirmed block from the review band are therefore different records. Logging the route per decision is what allows a platform to report them correctly.
Reporting has gone wrong in public. An audit of the database’s first 100 days (25 September 2023 to 2 January 2024) found that Facebook and Instagram declared heavy use of fully automated moderation in their transparency reports, yet submitted no fully automated statements of reasons. The audit’s authors argued that the yes/no automated-detection attribute does not show how much automation was used. Another analysis found platforms leaning on generic categories and rarely filling optional fields such as the decision-ground reference URL or the illegal-content explanation. It proposed cross-checking transparency reports against the database. Separately, Article 15 asks for “indicators of the accuracy and the possible rate of error” of automated means. The term is open enough that providers could report whichever metric flatters them. The precision and recall estimates from the measurement section are the defensible answer to that requirement.
FAQ
What are false positives in AI detection?
A false positive is content the detector flags as violating when it does not, such as the marble statue a body-part detector scored as nudity. Precision measures how often a flag is correct. In moderation, low precision means legitimate expression is being restricted.
