In a significant breakthrough, Answer.AI and LightOn have released ModernBERT, a new family of state-of-the-art encoder-only models that represent improvements over older generation encoders across the board. With a 8192 sequence length, better downstream performance, and much faster processing, ModernBERT is poised to become the new standard in numerous applications where encoder-only models are deployed.

**Background and Context**

BERT was released in 2018 and has remained widely used today due to its encoder-only architecture, which makes it ideal for real-world problems such as retrieval, classification, and entity extraction. However, after six years of advances in large language models (LLMs), a replacement was needed. ModernBERT takes dozens of recent advances from LLMs and applies them to a BERT-style model, including updates to the architecture and training process.

**Why it Matters**

ModernBERT's improvements over BERT and its younger siblings are significant. It is faster and more accurate, with an increased context length of 8k tokens compared to just 512 for most models. Additionally, ModernBERT includes a large amount of code in its training data, opening up new application areas that were previously inaccessible through open models.

**Key Facts**

  • ModernBERT is available as a slot-in replacement for any BERT-like models.
  • It has both a base (149M params) and large (395M params) model size.
  • ModernBERT was trained on 2 trillion tokens, resulting in 140 million and 395 million parameter models.
  • The parameter count hasn't increased much because it's designed to run on lower-end hardware.
  • It has a 8192 token context length, a significant improvement on BERT's 512.

**What Comes Next**

With ModernBERT's release, we can expect to see it become the new standard in numerous applications where encoder-only models are deployed. Its increased context length and inclusion of code in its training data open up new application areas that were previously inaccessible through open models. As the industry continues to evolve, it will be exciting to see how ModernBERT is used and adapted in various applications.

**Conclusion**

In conclusion, ModernBERT represents a significant breakthrough in the field of large language models. Its improvements over BERT and its younger siblings make it an attractive option for numerous applications where encoder-only models are deployed. As the industry continues to evolve, we can expect to see ModernBERT become the new standard in many areas.

**Links**

  • 🤗ModernBERT-Base
  • 🤗ModernBERT-Large
  • 📝arXiv
  • 🤗ModernBERT documentation page

**Fine-tuning ModernBERT with Sentence Transformers**

ModernBERT pairs well with Sentence Transformers for retrieval and similarity tasks. The following blogposts walk through fine-tuning ModernBERT (or any encoder) into embedding, reranker, or sparse models:

  • Training and Finetuning Embedding Models with Sentence Transformers: fine-tune ModernBERT as a dense embedding model.
  • Training and Finetuning Reranker Models with Sentence Transformers: fine-tune ModernBERT as a Cross Encoder (reranker); the post shows a ModernBERT-base reranker that beats much larger baselines.
  • Training and Finetuning Sparse Embedding Models with Sentence Transformers: fine-tune ModernBERT as a SPLADE sparse encoder.

**Community**

The community has been excited about ModernBERT, with many users sharing their experiences and applications of the model. Some notable comments include:

  • "Loved it, thanks for having done the work, explained it and provided the new BERT model." - cbonnett
  • "That would be a heck of a distilling! @cbonnett" - JohnGenetica
  • "This looks amazing, the optimizations are very welcome to anyone using fine-tuned encoders. However, it's disappointing to see it's been only trained with English text with no mention to multilingual benchmarks or a multilingual version in the plans." - erickrf

**Sources**

  • [1] Finally, a Replacement for BERT: Introducing ModernBERT – Answer.AI
  • [2] Finally, a replacement for BERT: Introducing ModernBERT
  • [3] LightOn - Sovereign AI Platform for Enterprise GenAI