A model that scores 92% on movie reviews will not read your support tickets at 92%. That single sentence is where most sentiment-analysis projects quietly go wrong: the headline benchmark gets booked against the whole workload, and the domain gap is discovered in production — after a triage dashboard is already routing tickets on numbers that don’t hold. Machine learning for sentiment analysis is genuinely feasible for some of what people ask of it, and speculative for the rest. The trouble is that the feasible slice and the speculative slice look identical in a proposal. Both get described as “read customer tone and classify it.” Only one of them survives contact with your actual data. How does machine learning for sentiment analysis work in practice? Strip away the branding and sentiment analysis is a text-classification problem. You have a piece of text and you want a label — positive, negative, neutral, or something more granular — and a model learns the mapping from labelled examples. The mechanics have gone through three broad eras, and all three are still in production somewhere. Classical machine-learning classifiers — logistic regression or a linear support vector machine over TF-IDF features, or bag-of-n-grams — remain surprisingly strong on clean, single-topic polarity tasks, and they train in seconds. Then came sequence models and, decisively, the transformer: fine-tuning a pretrained encoder like BERT or RoBERTa on a few thousand labelled examples became the default for most serious sentiment work, because the model arrives already knowing how English (or your target language) tends to hang together. Most recently, large language models can be prompted for sentiment zero-shot or few-shot, trading the cost of a labelled training set for the cost of tokens at inference time. What every one of these approaches shares is the thing people forget: the model learns the sentiment of the data it was trained or prompted on, not sentiment in the abstract. That distinction is the whole feasibility question. A sentiment classifier is a function fitted to a distribution — the vocabulary, the tone, the sarcasm norms, the way your customers say things — and it degrades exactly as far as your real text drifts from that distribution. Document, sentence, or aspect: which level is realistically automatable? “Sentiment analysis” names at least three different jobs, and their feasibility diverges sharply. Conflating them is the single most common scoping error we see. Document-level — one label for a whole review, email, or ticket. This is the easy end. On clean, single-topic text it is genuinely automatable. Sentence-level — a label per sentence. Harder, because context spills across sentence boundaries and a single sentence often carries less signal. Aspect-based (ABSA) — sentiment per topic within one text: “the food was great but the service was slow” is positive-food and negative-service in the same breath. This is where the interesting business value usually lives, and it is materially harder. The gap is not marginal. A document-level polarity classifier on in-domain text frequently reaches usable accuracy — often 85% or better on clean tasks (an observed pattern across sentiment engagements, not a single named benchmark). Aspect-based extraction on the same corpus can land 20–40 points below that, because it has to do two hard things at once: find the aspects and attribute sentiment to each. If a stakeholder asks for “sentiment on our product features,” they are asking for ABSA and quoting a document-level number. Those are different projects. This is the same reasoning we apply when we classify a use case as automatable, speculative, or research: the label on the workload matters less than which slice of it clears the accuracy bar you actually need. Why do models that ace public benchmarks fail on your domain? Here is the mechanism, because it is not mysterious. Public sentiment benchmarks — SST, IMDB reviews, Amazon product corpora — are curated, mostly single-topic, mostly written in complete sentences by people trying to be understood. Your support tickets, clinical notes, or trader chat are none of those things. They carry jargon, abbreviations, internal shorthand, negation stacked three deep, and a tone your customers use precisely because everyone in that context already understands it. A model fitted to one distribution and evaluated on another is measuring the wrong thing. The published 92% is real — for movie reviews. On your text, the vocabulary is out-of-distribution, the label definitions may differ (is a factual bug report “negative”? your annotators need to agree), and the failure modes shift. This is the general reason GenAI systems underperform on production data: the demo distribution and the deployment distribution are not the same distribution, and nobody measured the gap. The corollary is simple and non-negotiable: you cannot know a sentiment model’s feasibility for your use case without a labelled sample of your own data. Not a benchmark. Not a vendor’s demo. Yours. A feasibility rubric for a sentiment analysis use case Before committing to a build, classify each slice of the workload. This is a decision surface, not a formality — it is exactly the classification a GenAI feasibility assessment produces for any single use case. Signal Automatable Speculative Research Granularity Document-level polarity Sentence-level, some ABSA Fine-grained ABSA + emotion Text cleanliness Well-formed, single-topic Mixed quality, some jargon Heavy slang, sarcasm, code-switching Label availability 2k+ in-domain labelled examples A few hundred, noisy labels No agreed label definition yet Annotator agreement High (κ > 0.7) Moderate Low — humans disagree too Accuracy needed Triage / prioritisation Reporting with human review Automated action, no human In-domain measured accuracy Meets the bar on your sample Below bar, gap is closable Unknown or far below bar Read it column by column. If your slice lands mostly in the left column, build it — the ROI is real and bankable. If it drifts right, the honest move is a scoped pilot on your own labelled sample before any production commitment. The trap is booking the automatable ROI against the whole workload while the speculative slice quietly runs at chance. Note the annotator-agreement row. If your own reviewers cannot agree whether a sarcastic “great, another outage” is negative, no model will do better than they do — and you have discovered a labelling problem, not a modelling problem. That is often the most valuable output of the assessment. Where sarcasm, mixed emotion, and negation break the model These are not exotic edge cases. They are the everyday texture of real customer language, and they are precisely where the accuracy you booked evaporates. Sarcasm inverts surface sentiment: “brilliant, love waiting an hour on hold” is lexically positive and actually furious. Mixed emotion carries two valences in one utterance, which a single-label document classifier cannot represent at all — it has to pick, and it picks wrong. Negation and its scope (“not bad” vs “not good, bad”) trip up bag-of-words features entirely and still challenge transformers when the negation is distant from what it negates. The way to handle this is not to hope the model is smart enough. It is to build a targeted test set of exactly these cases — pull the sarcastic, mixed, and heavily-negated examples out of your own data, label them carefully, and measure separately. Your headline accuracy will hide a 30-point hole in this segment; a segmented eval surfaces it. In our experience, teams that skip the segmented test set are the ones surprised in production, because the aggregate number averaged the hole away. How do LLM-based approaches change the calculus? Large language models genuinely shift the feasibility line, but not uniformly. Prompted zero-shot or few-shot, an LLM can read nuance — sarcasm, mixed emotion, implicit tone — better than a small fine-tuned classifier, and it needs no labelled training set to start. For low-volume, high-nuance work, that can move a slice from speculative to automatable. The trade-offs are real and they run the other way on the axes that matter operationally. A fine-tuned RoBERTa classifier is small, cheap, and fast — milliseconds per inference on modest hardware, which matters when you are scoring millions of tickets. An LLM call is slower, costs tokens, and is harder to make deterministic. For high-volume triage, the classical or fine-tuned encoder often still wins on cost-per-decision even when the LLM edges it on accuracy. The right comparison depends on your volume, your latency budget, and your nuance requirement — the same context-dependent choice that runs through any natural language processing algorithm selection. There is no universal winner; there is a fit to your workload. What LLMs do not change: they still need a labelled evaluation set from your domain to prove they clear the bar, and they still fail in the segmented tests above unless you check. The zero-shot promise removes the training-data cost, not the measurement obligation. FAQ How does machine learning for sentiment analysis actually work? It is a text-classification problem: a model learns to map text to a sentiment label from labelled examples, using anything from a TF-IDF logistic regression to a fine-tuned transformer like BERT to a prompted LLM. In practice the key point is that the model learns the sentiment of its training or prompt distribution, not sentiment in the abstract — so it performs only as well as your real text resembles what it learned from. What is the difference between document-level, sentence-level, and aspect-level sentiment analysis, and which is realistically automatable? Document-level assigns one label to a whole text and is the most automatable on clean, single-topic data. Sentence-level labels each sentence and is harder because context crosses boundaries. Aspect-based (ABSA) attributes sentiment to each topic within a text — the highest business value and the hardest, often landing 20–40 points below document-level accuracy on the same corpus. Why do sentiment models that score well on public benchmarks degrade on my own domain data? Public benchmarks are curated, single-topic, well-formed text; your tickets or notes carry jargon, negation, and shorthand that are out-of-distribution for the trained model. A model fitted to one distribution and measured on another is measuring the wrong thing, so its published accuracy simply does not transfer. You cannot know feasibility without a labelled sample of your own data. How do I classify a sentiment analysis use case as automatable, speculative, or research? Score each slice of the workload against granularity, text cleanliness, label availability, annotator agreement, required accuracy, and — decisively — measured in-domain accuracy on your own labelled sample. Slices that clear your accuracy bar on your data are automatable; slices that fall short but look closable are speculative pilots; slices with no agreed label definition or far-below-bar accuracy are research. This is the same classification a GenAI feasibility assessment produces per use case. What data and labelling do I need before committing to a sentiment analysis build? At minimum a few thousand in-domain labelled examples with an agreed label definition and demonstrable annotator agreement (aim for κ above 0.7). Low agreement is a signal that the problem is under-specified, not that you need a bigger model. Without your own labelled data you have no basis to estimate feasibility, only a vendor’s benchmark on someone else’s text. Where do sarcasm, mixed emotion, and negation cause sentiment models to fail, and how do I test for it? Sarcasm inverts surface sentiment, mixed emotion carries two valences a single label cannot represent, and negation scope defeats bag-of-words features and challenges transformers. Test for it by building a segmented evaluation set of exactly these cases from your own data and measuring separately — the aggregate accuracy averages the hole away, so only a segmented eval surfaces the 20–40 point drop. How do modern LLM-based approaches change the feasibility calculus compared with classical ML classifiers? Prompted LLMs read nuance better and need no labelled training set, which can move low-volume, high-nuance slices from speculative to automatable. But they are slower and cost tokens per inference, so for high-volume triage a small fine-tuned encoder often wins on cost-per-decision. Either way you still need a labelled in-domain evaluation set to prove the bar is cleared. The question worth asking first The instinct is to ask “which model should we use for sentiment analysis?” The more useful question is narrower: which slice of this specific workload clears the accuracy bar I actually need, measured on my own labelled data? Answer that and the model choice usually settles itself. Skip it and you will ship a triage system that averages a hole in its own coverage — and whether a sentiment project is worth starting at all is a readiness question that sits upstream of the model, which is why we treat organisational AI readiness as the gate before per-use-case feasibility. The failure class here is the transferred benchmark; the artifact that catches it is a feasibility assessment run on your text, not a vendor’s.
