Ettin Suite Outperforms Decoders: Revolutionary Language Models for Classification and Retrieval

**Ettin Suite Revolutionizes Language Models: Encoders Outperform Decoders on Retrieval and Classification Tasks**

The language model community has been abuzz with excitement over the recent release of the Ettin suite, a collection of paired encoder-only and decoder-only models ranging from 17 million to 1 billion parameters. Trained on identical data and using the same recipe for both encoder-only and decoder-only models, the Ettin suite has produced state-of-the-art (SOTA) recipes in both categories for their respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as decoders.

**What Happened**

The Ettin suite was developed by a team of researchers from the Johns Hopkins University's Center for Language and Speech Processing (CLSP) and LightOn, a company specializing in large-scale language model training. The team aimed to create a comprehensive comparison between encoder-only and decoder-only models, which has been a long-standing debate in the industry. By training both types of models on identical data and using the same recipe, the researchers were able to isolate the effects of architecture on performance.

**Background and Context**

The large language model (LLM) community has largely converged on decoder-only models like GPT, Llama, and Qwen for their generative capabilities. However, encoder-only models like BERT have remained a staple in production systems for tasks such as classification, retrieval, and embedding. The key difference between these two architectures lies in their attention patterns: encoder models use bidirectional attention, allowing each token to "see" all other tokens in the sequence, while decoder models use causal attention, where tokens can only "see" previous tokens.

**Why it Matters**

The Ettin suite's findings have significant implications for the industry. The team discovered that encoder-only models excel at classification and retrieval tasks, while decoders excel at generative tasks. However, they also found that adapting a decoder model to an encoder task (and vice versa) through continued training is subpar compared to using only the reverse objective. This suggests that pretraining an encoder beats converting a decoder to an encoder plus further finetuning.

**What Comes Next**

The Ettin suite's release has sparked interest in exploring the possibilities of paired encoders and decoders. The team plans to open-source all artifacts of this study, including training data, training order segmented by checkpoint, and 200+ checkpoints, allowing future work to analyze or extend all aspects of training.

**Key Facts**

• Ettin suite includes paired encoder-only and decoder-only models ranging from 17 million to 1 billion parameters.
• Models were trained on identical data using the same recipe for both encoder-only and decoder-only models.
• Encoder-only models excel at classification and retrieval tasks, while decoders excel at generative tasks.
• Adapting a decoder model to an encoder task (and vice versa) through continued training is subpar compared to using only the reverse objective.
• Ettin suite's findings have significant implications for the industry, suggesting that pretraining an encoder beats converting a decoder to an encoder plus further finetuning.

The Ettin suite has opened up new avenues of research in language models and their applications. As the industry continues to evolve, it will be exciting to see how these findings shape the development of future language models.