Quick Verdict
MTEB Leaderboard is a hosted Hugging Face Space for exploring submitted embedding and retrieval results. It should be distinguished from MTEB itself: the leaderboard is a web interface over public results, while the Apache-2.0 mteb Python project is an installable evaluator that runs models and tasks on infrastructure you control. The distinction matters. Reading a score costs little; reproducing it may require model downloads, dataset licenses, custom model code, substantial compute, and precise version tracking. As of July 21, 2026, the repository is highly active, with release 2.18.5 published on July 19. MTEB is one of the strongest starting points for reducing a large embedding market to a defensible shortlist. It is not a procurement oracle. A final decision should filter by language, domain, task, latency, vector dimensions, and licenses, then retest finalists on private or recently collected data.
Best For
- Engineers selecting embedding models for RAG, semantic search, clustering, classification, or multilingual retrieval.
- Researchers who need an executable evaluation protocol rather than vendor-reported state-of-the-art claims.
- Developers using Hugging Face to inspect model cards, revisions, and public result artifacts.
- Teams building retrieval pipelines with LangChain or similar frameworks and planning a domain-specific second stage.
It is less useful to buyers who want a single permanent winner without running tests. Model quality is conditional, and operational cost can reverse a leaderboard ordering.
Key Features
The hosted board supports comparison across benchmarks, tasks, languages, and model metadata. The evaluator installs with pip install mteb or uv add mteb, exposes Python and CLI workflows, and covers classification, clustering, retrieval, reranking, semantic textual similarity, bitext mining, and expanding multimodal evaluation. Contributors can define models, add datasets or benchmarks, run selected splits, load result files, and submit compatible results.
MTEB provides structure, not universal assurance. A public result does not prove that benchmark examples were absent from model training. The Apache license on MTEB’s code does not relicense every dataset or model. Likewise, loading a repository that needs trust_remote_code=True can execute author-supplied Python; a high score does not make that code safe.
Use Cases
A practical selection process starts on the leaderboard by filtering for the target language and task, such as retrieval. Build a small candidate set, then compare parameter count, embedding dimensions, context length, model license, and inference requirements. Install the evaluator in an isolated environment, run a few relevant public tasks, and add a business-specific set containing real queries, documents, hard negatives, and failure cases.
For Chinese, legal, healthcare, or code retrieval, do not let a broad English average dominate the decision. When publishing or submitting results, record the MTEB version, model revision, dataset revision, dependencies, hardware, prompts or instructions, and output files. Otherwise a later task update can move a score without leaving enough evidence to explain why.
Pricing
Browsing the leaderboard is free, and MTEB’s evaluator code uses Apache-2.0. Users still pay for GPU or CPU time, storage, network transfer, and engineering work. Full evaluation matrices can be expensive, especially for large models and retrieval corpora.
Licensing must be handled per artifact. MTEB’s software license, benchmark dataset terms, model-weight license, and any training-data restrictions are separate. Pin the mteb release, model commit, dataset revision, and evaluation configuration. Recording only a leaderboard rank is not enough for auditability or reproduction.
Pros
- Combines a convenient discovery interface with an evaluator teams can run locally.
- Active, broad, and permissively licensed core project.
- Task- and language-level views are more informative than one vendor score.
- Python and CLI interfaces fit reproducible experiment pipelines.
- Open task definitions and community submissions make disagreements inspectable.
- Supports a growing range of embedding and retrieval system types.
Cons
- Overall averages can hide domain, language, latency, and memory constraints.
- Public benchmarks may be present in training corpora, creating contamination or overfitting.
- Dataset and model licenses are not inherited from the evaluator’s Apache license.
- Some model loaders execute remote custom code and create a software-supply-chain risk.
- Comprehensive runs consume meaningful compute, bandwidth, and storage.
- Results can shift with evaluator, model, dependency, dataset, or hardware revisions.
Alternatives
| Alternative | Best when | Difference from MTEB |
|---|---|---|
| BEIR | Focusing on zero-shot information retrieval | Narrower, classic retrieval-centered benchmark suite |
| C-MTEB | Investigating Chinese embeddings | More Chinese-focused; exact task and dataset terms still matter |
| MIRACL | Comparing multilingual retrievers | Concentrates on multilingual retrieval rather than a general task matrix |
| A private gold set | Making a production decision | Closest to the business, but expensive to label and hard to compare publicly |
| Jina AI | Buying hosted embedding or retrieval capabilities | A product service, not a neutral benchmark protocol |
| Hugging Face Model Hub | Discovering models and model cards | Artifact hosting does not itself create uniform evaluation |
These methods are complementary. MTEB is effective for broad screening; a focused public benchmark and a private test set should decide the final deployment.
FAQ
Are MTEB Leaderboard and the MTEB Python package the same product?
No. The leaderboard is a hosted results browser. The package is the installable evaluator used to run and reproduce tasks.
Is the highest-ranked model always best for RAG?
No. Target language, corpus domain, query distribution, dimensions, latency, memory, and licensing can all matter more than the overall score.
Is running MTEB completely free?
The code is free, but compute, storage, downloads, and engineering time are not.
Does Apache-2.0 mean every benchmark dataset is commercially usable?
No. Each dataset and model has its own terms and must be reviewed separately.
How can a team reduce benchmark-contamination risk?
Review training disclosures, use recent or private examples, maintain hidden test sets, and report public and private results separately.
Is it safe to enable trust_remote_code?
Not by default. Pin the repository revision, inspect the code, and run it in a restricted environment without credentials or unnecessary network access.
Bottom Line
MTEB is valuable because it offers a shared, executable comparison framework, not because it names an eternal champion. Use the hosted board to find candidates, then pin and run the evaluator with controlled model and dataset versions. A decision that also documents contamination, licensing, remote-code, and compute concerns is much more durable than a screenshot of the current top row.