But as we explored in Beyond Binary, MTEB has a discrimination problem: given its (overwhelmingly) binary annotations, it can’t tell the difference between a document which perfectly answers a query and one which may only tangentially address it. So we ran all three Harrier models through the same graded evaluation pipeline we use for our evals dashboard — 28 diverse datasets, three independent LLM judges , continuous relevance scores from 0 to 10.

It is not a question of harrier-27b is a good model. As a matter of fact, it is. (And at 27 billion parameters and a whopping 5,376 output vector dimensionality, we would certainly hope it would be). But is it the best?

The Three Model Problem: MTEB Evals

On the global average across all 28 evaluation datasets, there are three embedding models which markedly outperform the rest:

Below those, qwen3-4b, jina-v5-text-small, cohere-embed-v4, and openai-v3-large (in that order) form a cluster of second tier performance. But if you need top-tier accuracy, your choice lies in that trio.

So what separates them? On NDCG@10, not much — roughly one point across the trio (though it’s worth noting that harrier still comes out worst). But NDCG@10 is not the whole story.

That is where the separation becomes real. A reranker or other downstream system can reorder or rework whatever the embedding model surfaces, but it cannot conjure up a document the embedder failed to retrieve. zembed-1’s recall advantage compounds downstream: fewer relevant documents lost at the first stage means a strictly better candidate set for everything that follows.

Head-to-Head: zembed-1 vs harrier-27b

Averages, of course, can obscure as much as they reveal. So let us go dataset by dataset. Our evals dashboard covers 28 datasets drawn from three MTEB task categories — retrieval, reranking, and instruction retrieval — spanning legal (AILAStatutes, LegalBench), medical (CovidRetrieval, TRECCOVID), financial (FiQA, FinQA), multilingual (MIRACL, MLQA, Belebele, WikipediaRetrieval), and technical domains (StackOverflowQA, SCIDOCS), among others.

The pattern of where each model wins is telling. zembed-1 dominates on instruction retrieval (Core17, News21, Robust04 — tasks which require parsing nuanced query intent, not merely matching keywords), medical and legal domains (CovidRetrieval, LegalBench, TRECCOVID), finance (FiQA2018, FinQARetrieval), and technology (StackOverflowQA). harrier-27b, for its part, shows strength on multilingual reranking and a handful of niche benchmarks (RuBQReranking, Russian paragraph reranking; and TwitterHjerne, Danish Twitter retrieval).

The Race for Second Place

As we established in our previous head-to-head, voyage-4 has been the reigning #2 embedding model. With harrier-27b now in the picture, that position is genuinely contested:

It is… remarkably close. The two models split dataset wins exactly (14–14), and depending on your vertical, either could be the better runner-up. If forced to pick, voyage-4 still holds thin but consistent edges on the overall averages — enough to keep the #2 title, but barely. (Neither, however, threatens first place.)

Harrier’s Scaling Story

One genuinely interesting aspect of the Harrier family is its range of sizes. The scaling is clean — and instructive:

A +3.3 point NDCG jump from 270M to 0.6B, then +4.9 points from 0.6B to 27B. Returns to scale are not completely diminishing — the largest absolute improvement comes from the largest model. Credit where credit is due: this is a decently-executed scaling curve, particularly for the 0.6b model which demonstrates equal or better performance to Cohere’s flagship embed-v4.

The contrast in size between harrier-27b and all other models bears emphasis: 27 billion parameters is absolutely massive for an embedding model, and that’s not a compliment.

zembed-1 achieves its #1 ranking with 4 billion parameters and a 2,560-dimensional output. harrier-27b needs nearly 7x the parameter count and 2x the vector dimensionality to land 0.9 points behind on NDCG@10. In a production setting — where embedding compute, storage costs, and index size are real constraints — the efficiency gap is hardly academic. Would you pay for a model with 7x higher inference cost, probably 7x as much latency, which outputs an embedding that’s twice as costly to store, just to get worse results?

We wouldn’t.

What This Means

harrier-27b is a legitimate top-three embedding model — quite possibly the strongest new entrant we have seen since voyage-4. It is genuinely competitive, especially on multilingual reranking tasks, and we expect Microsoft will continue to iterate on the family.

But the leaderboard has not changed:

For the full interactive breakdown across all models, datasets, metrics, and reranker combinations, explore the evaluation dashboard.

from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
response = zclient.models.embed(
model="zembed-1",
input_type="query", # "query" or "document"
input="What is retrieval augmented generation?", # string or list[str]
dimensions=2560, # optional: must be one of [2560, 1280, 640, 320, 160, 80, 40]
encoding_format="float", # "float" or "base64"
latency="fast", # "fast" or "slow"; omit for auto
)