CLS Pooling Explained: How It Works in Transformer Embeddings

CLS pooling uses the [CLS] token's final hidden state as the sentence embedding. It works when the model was trained for it, and fails on raw BERT.

CLS Pooling Explained: How It Works in Transformer Embeddings
Written by TechnoLynx Published on 11 Jul 2026

CLS pooling takes the final hidden state of one special token, [CLS], which is put at the front of every input, and uses that single vector as the embedding for the whole sequence. In code it is one line, model_output[0][:, 0]. Whether that vector is any good depends on training. CLS pooling is the right choice when the model was trained to put its summary into that position, as BGE and supervised SimCSE are. On an off-the-shelf BERT checkpoint it gives a weak one: Reimers and Gurevych (2019) measured an average STS correlation of 29.19 for the raw BERT CLS output, against 54.81 for averaging the token outputs.

What the [CLS] token is in BERT

BERT, introduced by Devlin et al. (2018, revised 2019), puts a special classification token in front of every input sequence. During pre-training, the representation at that position feeds a binary task called Next Sentence Prediction (NSP), which asks whether one text segment follows another in the training corpus. You can see the token in the input IDs. In the BGE documentation’s examples, each tokenised sentence starts with ID 101 ([CLS]) and ends with 102 ([SEP]). (How the rest of the sequence becomes subword IDs is covered in our note on BERT tokenization and WordPiece subwords.)

The [CLS] token is a special token, not a word from the text. It is a learned input vector that goes through every self-attention layer like any other token, so at the output it has attended to the whole sequence. Self-attention lets each position mix information from every other position, which is why one output position can stand for the sequence at all. Our explainer on how transformers work in deep learning covers that mechanism. Whether the position actually holds a useful summary depends on how the model was trained.

How CLS pooling turns a token into a sentence embedding

A transformer encoder outputs one vector per token, so a sentence of 12 tokens and a sentence of 40 tokens produce outputs of different lengths. Pooling turns those variable-length token embeddings into one fixed-size sentence embedding. The Sentence Transformers documentation names mean pooling and CLS pooling as the common strategies. bert-as-service calls the CLS variant FIRST_TOKEN, which takes the hidden state of the first token and nothing else.

That is all CLS pooling does: it selects one vector rather than computing anything. The other N−1 output embeddings are thrown away. That only works if training made that position a sentence summary.

Why does BERT use the CLS token for classification instead of averaging all token outputs?

BERT was built so that one additional output layer, fine-tuned on the task, would be enough for a wide range of tasks. The [CLS] position was already trained as a sequence-level slot through NSP. Devlin et al. designed BERT to be fine-tuned with one additional output layer, and the class-token output is the embedding taken for downstream tasks. The [CLS] route keeps the architecture simple and uses a position that pre-training has already made sequence-level.

There is a subtlety that explains a lot of confused forum threads. A classifier trained on top of the CLS vector can give some dimensions more weight than others. Reimers and Gurevych (2019) found that raw BERT CLS outputs did badly on STS tasks scored with cosine similarity, but reached decent results on SentEval, where a logistic regression classifier is fitted to the embeddings. Cosine similarity treats every dimension equally, and that is where the raw CLS vector did badly. They concluded that raw BERT CLS and averaged outputs are not suited to cosine, Manhattan or Euclidean distance. So “CLS works for classification” and “CLS is a bad sentence embedding” can both be true of the same checkpoint.

When CLS pooling gives a good sentence embedding, and when it fails

Whether the model was trained for it decides. NSP trains the [CLS] position on one binary question: whether one segment follows another. That does not make it place sentences with similar meanings close together in vector space, which is what semantic search and clustering need. The measurements agree.

Off-the-shelf encoders: CLS loses

  • Sentence-BERT (Reimers and Gurevych, 2019): raw BERT CLS output averaged 29.19 STS correlation, and mean pooling 54.81. Both scored below averaged GloVe embeddings.
  • DeCLUTR (Giorgi et al., 2021): on the two aggregate scores in its comparison table, BERT-base scored 63.53 and 69.57 with CLS against 71.98 and 73.37 with mean. ALBERT-base-v2, trained with Sentence-Order Prediction, scored 58.75 and 69.88 with CLS against 69.39 and 74.83 with mean. RoBERTa-base, trained with neither NSP nor SOP, scored 68.53 and 66.92 with CLS against 72.84 and 74.59 with mean.
  • BERT-flow (Li et al., 2020): the authors averaged the context embeddings of the last one or two layers because that was consistently better than the [CLS] vector. They also report that BERT without fine-tuning produces an anisotropic sentence space that hurts semantic similarity.

The bert-as-service FAQ gives the practical rule. A pre-trained model that has not been fine-tuned on a downstream task does not give a good sentence representation in its [CLS] hidden state, and after fine-tuning you may use [CLS]. The tool’s default strategy instead averages the second-to-last layer.

Models trained for CLS: CLS wins, and mean pooling breaks them

BGE uses a BERT-base architecture (12 layers, hidden size 768) and is trained to use the last hidden state of [CLS] as the sentence embedding. Its documentation warns that switching to mean pooling causes a significant drop in performance. SimCSE’s README tells supervised SimCSE users to use the cls pooler. The Sentence Transformers checkpoint nli-bert-base-cls-pooling has pooling_mode_cls_token: True and mean pooling switched off in its Pooling config.

Training also narrows the gap. In the SBERT pooling ablation, CLS scored 79.80 and 86.62 on the two reported measures, against 80.78 and 87.44 for mean pooling and 79.07 and 69.92 for max pooling. The 29.19 figure is for BERT’s output used directly, without fine-tuning.

Quick reference: which pooling the checkpoint expects

Checkpoint Trained for Use
Vanilla BERT, RoBERTa, ALBERT, no fine-tuning Masked LM plus NSP/SOP, or neither Not CLS. Raw CLS trails mean pooling on similarity tasks
BGE / BGE-v1.5 CLS last hidden state CLS. Mean pooling degrades it significantly
Supervised SimCSE SimCSE README: CLS with the linear+activation layer cls
Unsupervised SimCSE SimCSE README: CLS without the extra layer cls_before_pooler
SBERT / SRoBERTa checkpoints SimCSE README: average of the last layer Average pooling
nli-bert-base-cls-pooling CLS CLS, per its Pooling config
Pure BERT-like model loaded into Sentence Transformers Nothing specific Library default is mean pooling

The mismatch to guard against: a CLS-trained model loaded through a code path that defaults to mean pooling, or the reverse. The cheapest safeguard is to read the model card or the saved Pooling config before writing any embedding code. It also applies to any bi-encoder used for retrieval.

The question here is narrower: does this checkpoint’s training make its [CLS] position trustworthy?

The CLS hidden state is not the same vector as pooled_output

In Hugging Face Transformers, a BERT-family model returns two candidates that people often treat as the same thing. last_hidden_state[:, 0] is the raw final-layer vector at the [CLS] position. pooler_output is that same vector passed through a linear layer and a tanh activation. For BERT, the documentation states that the weights of that linear layer were trained on the next sentence prediction objective during pre-training. Researchers extracting BERT features describe it the same way: the pooled representation is a transformed version of the [CLS] embedding, passed through a hidden layer and then tanh.

SimCSE makes the distinction explicit with two options. cls keeps the linear+activation layer and is meant for supervised SimCSE. cls_before_pooler drops it and is meant for unsupervised SimCSE. Taking pooler_output from a model trained on the raw hidden state, or the other way round, is the same mismatch as the pooling-mode one, just one layer later.

What is the purpose of the CLS token in a Vision Transformer (ViT)?

Dosovitskiy et al. (2021) added an extra learnable classification token to the sequence of patch embeddings to stay as close as possible to the original text Transformer. A classification head is attached to that token’s final output, as an MLP with one hidden layer during pre-training and a single linear layer during fine-tuning. The Hugging Face ViT classifier is likewise a linear layer on the final hidden state of [CLS].

The ViT authors also tested the alternative. Global average pooling over the patch embeddings did very poorly at first. They then found the gap was explained entirely by the need for a different learning rate, not by the class token or by averaging itself. So in ViT, the class token is a design inherited from text Transformers, not a necessity. Standard ViT implementations use the class token, and a minority use average pooling.

Self-supervised training gives the token a different role. In DINO (Caron et al., 2021), the [CLS] token is not attached to any label or supervision. Its job is to aggregate information from the whole sequence for the projection head. Its last-layer attention maps still pick out object regions without any labels. For some downstream tasks the DINO authors concatenated the [CLS] output with pooled patch tokens. In one comparison of pooling methods for a ViT tested on CIFAR, the class token trailed both average and max pooling: 84.52% top-1 on CIFAR-10 against 87.15% for average pooling. That is a single small-scale setting, not a general ranking, but it is a reason to test the pooling choice on your own vision task rather than assume CLS is best.

How to extract the CLS token embedding from a BERT model in code

With the transformers package, you run the input through the model and take the last hidden state of the first token. The BGE model card writes this as model_output[0][:, 0], and last_hidden_state[:, 0] is the same tensor accessed by name.

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

name = "BAAI/bge-large-en-v1.5"   # a model trained for CLS pooling
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name).eval()

batch = tokenizer(
    ["How do I reset my password?", "Steps to recover account access"],
    padding=True, truncation=True, return_tensors="pt",
)

with torch.no_grad():
    out = model(**batch)

cls = out.last_hidden_state[:, 0]        # raw [CLS] hidden state, shape (batch, hidden)
# out.pooler_output would be the linear + tanh version: a different vector
cls = F.normalize(cls, p=2, dim=1)        # unit length for cosine similarity

similarity = cls @ cls.T

Sentence Transformers provides L2 normalisation as a module that runs on the pooled sentence embedding.

A pure BERT-like checkpoint loaded without a config gets mean pooling by default, so CLS has to be requested explicitly:

from sentence_transformers import SentenceTransformer, models

word = models.Transformer("bert-base-uncased", max_seq_length=128)
pool = models.Pooling(word.get_word_embedding_dimension(), pooling_mode="cls")
model = SentenceTransformer(modules=[word, pool])

Requesting CLS only makes sense if you are about to fine-tune this model, or if the checkpoint was already trained for CLS. Applied to plain bert-base-uncased as it stands, this code gives you the 29.19-correlation case. Keep the pooling, the normalisation and the checkpoint consistent between the embedding job that builds an index and the service that answers queries.

FAQ

What is the difference between average pooling and max pooling?

Average pooling takes the mean of the token hidden states across the sequence, and max pooling takes the maximum of each dimension across the sequence. A 2025 analysis of transformer pooling shows that average pooling is optimal only when the input contains no noise, or when signal and noise share a centroid, while max pooling is optimal only when a single vector holds the maximal value in every feature.

Is the CLS token output the same in every layer?

No. BERT produces a [CLS] representation at each of its 12 hidden layers, plus the pooled output, and researchers choose between them as different features. bert-as-service notes that the last layer is close to the training output and may be biased to the training targets, which can give a bad representation if you have not fine-tuned the model.

Back See Blogs
arrow icon