A sentiment model that tops a movie-review leaderboard is not a sentiment model that works on your support tickets. The two facts feel like they should be the same fact. They are not, and the gap between them is where most sentiment deployments quietly go wrong. Sentiment analysis with machine learning is the task of assigning a polarity label — usually positive, negative, or neutral — to a piece of text. In practice the “machine learning” part is a classifier: a model that maps text into one of those classes based on patterns it learned from labelled examples. Modern systems are usually a transformer encoder (a BERT-family model, or an LLM prompted to classify) with a small classification head or a parsing step on top. That much is well understood. The part that trips up procurement is what “evaluating” one of these models actually requires. How does sentiment analysis with machine learning work? Mechanically, the flow is short. Text goes in. It gets tokenized — split into subword units the model recognises; the way that split happens is itself a source of behaviour worth understanding, which we cover in BERT tokenization and what it means for evals. The model produces a distribution over the label set, and you take the argmax as the predicted class. If you fine-tuned a BERT-style encoder in PyTorch, that distribution comes from a softmax over three logits. If you prompted an LLM, it comes from the model committing to a word like “negative.” Either way you get a label per input. What “works in practice” means is narrower than it sounds. A sentiment model does not detect sentiment in some universal sense. It reproduces the labelling decisions embedded in the data it was trained or benchmarked on. SST-2, the Stanford Sentiment Treebank, is built from movie reviews. IMDb is movie reviews. A model that scores in the high 90s on those datasets has learned what positive and negative look like in film criticism written for a general audience. That is a real skill. It is also a specific one. The core claim here is simple and it is the one to hold onto: sentiment classification is not a solved capability you buy off a leaderboard — it is a task you have to define and measure against your own labelled data before the accuracy number means anything. Everything below is an unpacking of why. Why does a sentiment model that scores well on a public benchmark fail on our own text? Because the benchmark never tested your text, and “sentiment” is not one problem. It is a family of problems that happen to share a label set. Consider a support ticket that reads: “Great, another outage. Third one this week. Love it.” A movie-review-trained model sees “great” and “love” and leans positive. A human reads sarcasm and files it under negative. The words carry positive polarity; the meaning is the opposite. Sarcasm is not an edge case in customer feedback — in the support-and-social corpora we have looked at, it shows up often enough to move a class boundary (observed pattern across engagements, not a benchmarked rate). The public benchmark never had to handle it because polished movie reviews rarely rely on inversion. Then there is domain jargon. In a financial filing, “the company recognised an impairment” is a materially negative statement dressed in neutral, technical language. There is no sentiment-bearing adjective for the model to grab. In clinical notes, “patient denies chest pain” is reassuring, but the token “pain” pulls a naive model negative and “denies” is a negation the model may not weight correctly. These are not the model being bad. They are the model being asked a question it was never trained to answer. Mixed polarity is the third structural break. “The hardware is excellent but support is a nightmare” contains both classes in one sentence. Aspect-based sentiment — which entity is the positive one and which is the negative one — is a genuinely different task from document-level polarity, and a model tuned for the latter will flatten the former into a single confused label. If your real texts are mixed-polarity and your benchmark texts were single-polarity, the aggregate accuracy figure is measuring the wrong distribution. None of this is visible in a single headline accuracy number. That is the whole problem. A vendor’s “94% accuracy” is an average over their test distribution, and averages hide exactly the failures that will hurt you. How do we define sentiment labels for our domain so the eval measures the right thing? Before you measure anything, you have to decide what you are measuring. This is the step teams skip, and skipping it is why two reasonable people annotate the same 200 tickets and disagree on a third of them. Define the label set explicitly. Three classes (positive / negative / neutral) is the common default, but for financial or operational text you may need “negative” split from “risk-flagging,” or a “not applicable” class for text that carries no sentiment at all. Decide where neutral ends and negative begins — is a factual complaint neutral or negative? Write it down. Decide how mixed-polarity text is labelled: does it get the dominant polarity, both labels, or does it route to aspect-level annotation? These are policy decisions, not statistical ones, and the model can only be as consistent as the labels you hand it. Then assemble the sample from your actual data, not from a public set that looks similar. A few hundred to a few thousand examples drawn from the real distribution — real tickets, real filings, real notes — is worth more than a million examples of the wrong kind of text. Have at least two annotators label an overlapping subset and measure their agreement (Cohen’s kappa or similar). If the humans cannot agree, no model will hit a defensible number, and you have found a spec problem, not a model problem. What per-class metrics should a sentiment eval report beyond overall accuracy? Aggregate accuracy is the metric to distrust first, because it hides class imbalance and hides which class the model fails on. If 80% of your tickets are neutral, a model that predicts “neutral” for everything scores 80% and is useless. The metrics that actually decide the choice are per-class, and the grid they come from is the confusion matrix — how to read precision and recall off that grid is worth internalising before you compare candidates. Here is the minimum set a sentiment eval should report, and what each one protects you from: Metric What it answers Failure it exposes Evidence class Per-class precision When the model says “negative,” how often is it right? False alarms flooding a triage queue benchmark (on your labelled sample) Per-class recall Of the truly negative texts, how many did it catch? Missed complaints that should have escalated benchmark (on your labelled sample) Macro-F1 Balanced performance across all classes A model that only works on the majority class benchmark (on your labelled sample) Confusion matrix Which classes get mistaken for which Neutral bleeding into negative, or sarcasm inversions benchmark (on your labelled sample) Overall accuracy Aggregate correctness Little on its own; reported for context only benchmark (on your labelled sample) The reason to insist on per-class recall for the negative class specifically: in most business uses of sentiment, missing a negative signal costs more than misfiring on a positive one. A churned customer you never flagged is more expensive than a happy one you accidentally escalated. The metric hierarchy should follow the cost structure of your use case, not the other way round. For a broader treatment of which metrics decide a serving decision, see machine learning model metrics: which ones actually decide a serving config. How does a sentiment eval fit into the task-specific evaluation a procurement committee reviews? This is where the exercise pays for itself. A sentiment eval is one concrete instance of a task-specific evaluation — the discipline of measuring candidate models against your own labelled data under your own conditions, rather than trusting a vendor’s benchmark. When we build a task-specific eval for a buyer’s candidate models, sentiment classification is often the entry task precisely because it is easy to label and its failures are legible; the same [production AI monitoring harness](Production AI Monitoring Harness) that runs the sentiment comparison generalises to the rest of a buyer’s tasks. For the committee, the deliverable changes shape. Instead of “the vendor claims 94% accuracy,” they get a per-class table on the company’s own data, an inter-annotator agreement number that establishes the labels are trustworthy, and a documented definition of what each class means. That is decision-grade evidence: it is defensible, it is reproducible, and it converts a vague vendor claim into a class-level comparison the committee can actually reason about. In our experience this both shortens time-to-approval and reduces the post-deployment misclassification surprises that erode trust in the system’s first weeks. It also matters in regulated contexts, where a sentiment misclassification can have downstream consequences the committee is accountable for. That is the failure class a procurement-grade evaluation evidence pack exists to surface before approval, not after. TechnoLynx works with teams building these evals for exactly that reason — the point is not a better model, it is a defensible model choice, and that lives inside the broader AI infrastructure practice. FAQ What matters most about sentiment analysis with machine learning in practice? A classifier — usually a transformer encoder with a classification head, or a prompted LLM — maps tokenized text to a polarity label such as positive, negative, or neutral. In practice it does not detect sentiment universally; it reproduces the labelling decisions embedded in its training or benchmark data, which is why the specific data it learned from matters as much as the architecture. Why does a sentiment model that scores well on a public benchmark fail on our own text? Public benchmarks like SST-2 and IMDb are built from polished movie reviews, so a high score means the model handles film criticism, not your support tickets or filings. Your text often contains sarcasm, domain jargon, and mixed polarity — assumptions the benchmark never tested — and a single aggregate accuracy number hides exactly those failures. How do we define sentiment labels for our domain so the eval measures the right thing? Write down the label set explicitly, decide where neutral ends and negative begins, and set a policy for mixed-polarity text before annotating. Then draw a sample from your real data, have at least two annotators label an overlapping subset, and measure their agreement — if humans cannot agree, no model will hit a defensible number. What per-class metrics should a sentiment eval report beyond overall accuracy? Report per-class precision and recall, macro-F1, and the confusion matrix, with overall accuracy shown only for context. Aggregate accuracy hides class imbalance and which class the model fails on; per-class recall for the negative class usually matters most because a missed complaint costs more than a false alarm. How do sarcasm, mixed polarity, and domain jargon break naive sentiment models? Sarcasm inverts polarity (“Great, another outage” reads positive lexically but negative in meaning), mixed polarity puts two classes in one sentence and flattens into a confused label, and jargon carries sentiment with no sentiment-bearing adjective (“recognised an impairment” is negative but neutral-looking). Movie-review benchmarks rarely contain these, so models tuned on them never learned to handle them. How does a sentiment eval fit into the task-specific evaluation a procurement committee reviews? It is one concrete instance of measuring candidate models against your own labelled data. The committee receives a per-class table on the company’s own text, an inter-annotator agreement number, and documented label definitions — decision-grade evidence that replaces a single vendor accuracy figure, shortens time-to-approval, and surfaces regulated-context risks before deployment. What does the sentiment eval miss that operational monitoring later catches? The eval measures a fixed labelled snapshot, so it cannot see distribution drift — new product names, changing customer language, or seasonal complaint patterns that shift after deployment. Operational monitoring catches those, which is why the eval is the entry point to a monitoring harness, not a one-time gate. A sentiment eval tells you which model to pick today; it cannot tell you when the world your model classifies has changed underneath it. The label set you defined this quarter will drift, the jargon will evolve, and the confusion matrix that looked clean at approval will develop a new hot cell. That is the boundary between evaluation and model monitoring in production — and the reason the sentiment eval is where the conversation starts, not where it ends.
