InfoNCE Loss

InfoNCE — Information Noise-Contrastive Estimation — is the loss function powering almost every modern embedding model. For a query , its positive document , and a set of negatives , the loss is the cross-entropy of a softmax over the similarity scores, where the positive is the correct class:

L = -log( exp(sim(q, d+) / τ) / Σ_i exp(sim(q, d_i) / τ) )

The denominator sums across the positive and all negatives. is typically cosine similarity on L2-normalized vectors; is the temperature.

That’s it. One line of math. Every modern open-weights embedder — E5, BGE, GTE, Snowflake Arctic, zembed-1’s bootstrap stage — minimizes some variant of it.

The three knobs that actually matter

Three things determine whether InfoNCE produces a good embedding model:

• Number of negatives . More negatives = tighter mutual-information bound = better embeddings. The free way to scale this is in-batch negatives — every other example in the batch is a negative for free. That’s why state-of-the-art embedders use batch sizes in the thousands.
• Quality of negatives. Random negatives saturate fast; the model gets them right and learns nothing. Hard-negative mining is what unlocks the second half of training.
• Temperature . Lower makes the softmax sharper. The model focuses gradient on the hardest negative — usually good, sometimes destabilizing.

The mutual-information story

Oord, Li & Vinyals (2018) showed that minimizing InfoNCE lower-bounds the mutual information between query and document representations:

I(q; d) ≥ log(N) - L_InfoNCE

So minimizing the loss maximizes a bound on — pushing the encoder toward representations that maximally preserve the dependence between matched pairs. The bound tightens with , which is the formal reason batch size matters.

Where InfoNCE breaks down

Variants you’ll see in papers

• Symmetric InfoNCE (CLIP-style): average the loss over both directions — and . Standard for multimodal embeddings .
• Multiple-negatives ranking loss (MNRL): functionally identical to InfoNCE with in-batch negatives; the name predates InfoNCE’s adoption.
• Decoupled contrastive loss: removes the positive from the denominator, claimed to be more stable. Marginal in practice.
• SupCon (supervised contrastive): generalizes to multiple positives per anchor, useful when labels are class-level.

The literature is full of “novel” contrastive losses that turn out to be InfoNCE in a wig. When evaluating a paper’s loss claim, check whether the math reduces to weighted log-sum-exp over similarities. Usually it does.