As artificial intelligence applications continue to expand beyond chatbots and generative AI systems, the importance of efficient encoder models has become increasingly evident. While large language models often receive most of the attention, encoders remain the backbone of countless enterprise AI workflows, powering everything from text classification and intent routing to safety moderation and personally identifiable information (PII) detection.
Recognizing the growing demand for long-context, CPU-friendly encoder models, Liquid AI has announced the release of LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, two open-weight bidirectional encoders built on the company’s LFM2 hybrid architecture.
The new models introduce support for an 8,192-token context window, multilingual capabilities across 15 languages, and performance characteristics aimed at organizations running AI workloads on CPUs rather than GPUs. According to Liquid AI, the architecture is specifically designed to maintain efficiency as context lengths increase, making it suitable for long-document processing in real-world environments.
This release represents another step in the evolution of encoder architectures that began with BERT and later advanced through models such as ModernBERT. Liquid AI believes the LFM2 architecture pushes this progression further by delivering improved scalability and lower computational costs on longer inputs.
Why Encoder Models Still Matter in Modern AI
The rapid growth of generative AI has created the impression that decoder-based large language models dominate every AI application. However, encoders continue to play a critical role in production systems.
Organizations rely on encoder models for tasks such as:
- Document classification
- Intent recognition
- Content moderation
- Safety filtering
- Semantic understanding
- Search ranking
- PII detection
- Routing user requests
- Legal document analysis
- Healthcare record processing
Unlike conversational AI systems, these workloads often run continuously in the background and frequently operate on CPU infrastructure rather than expensive GPU clusters.
Many enterprise deployments require processing thousands or even millions of documents daily, making efficiency a key requirement.
As document sizes grow, traditional architectures can become increasingly costly to operate. Liquid AI argues that this is precisely where the LFM2 framework offers advantages, thanks to a design whose computational costs grow more gradually as context lengths increase.
Introducing LFM2.5-Encoder-230M and LFM2.5-Encoder-350M
The newly released models are:
- LFM2.5-Encoder-230M
- LFM2.5-Encoder-350M
Both models are open-weight bidirectional encoders developed from Liquid AI’s LFM2.5 decoder family.
Key specifications include:
| Feature | LFM2.5-Encoder-230M | LFM2.5-Encoder-350M |
|---|---|---|
| Architecture | LFM2 Hybrid | LFM2 Hybrid |
| Context Window | 8,192 Tokens | 8,192 Tokens |
| Hidden Size | 1024 | 1024 |
| Vocabulary Size | 65,536 Tokens | 65,536 Tokens |
| Languages Supported | 15 | 15 |
| License | LFM Open License v1.0 | LFM Open License v1.0 |
Unlike traditional encoder releases that are trained entirely from scratch, Liquid AI took a different approach by converting existing decoder models into bidirectional encoders.
This strategy allows the company to leverage knowledge already embedded in the decoder backbone while adapting the architecture for understanding and classification tasks.
How Liquid AI Turned a Decoder into an Encoder
One of the most interesting aspects of this release is the conversion process itself.
Rather than building new encoder architectures from the ground up, Liquid AI initialized the models from the existing:
- LFM2.5-230M decoder
- LFM2.5-350M decoder
The research team then modified the architecture using three major changes.
1. Replacing Causal Attention with Bidirectional Attention
Decoder models typically use causal attention, meaning each token can only attend to tokens that come before it.
This design is ideal for text generation but less effective for understanding tasks.
To create an encoder, Liquid AI replaced the causal attention mask with a bidirectional mask.
As a result, every token can attend to both previous and future tokens within the sequence.
This enables richer contextual understanding and aligns the architecture with traditional encoder behavior.
2. Converting Short Convolutions to Non-Causal Operations
The LFM2 architecture incorporates short convolution layers.
In the original decoder architecture, these convolutions operate causally.
For the encoder version, Liquid AI transformed them into non-causal convolutions using symmetric center padding.
This adjustment allows each token to incorporate information from neighboring tokens on both sides rather than only from preceding text.
The result is a more balanced contextual representation suitable for classification and retrieval-related tasks.
3. Training with a 30% Masked Language Modeling Rate
The third major modification involved training the model using a masked language modeling (MLM) objective.
Traditional BERT models use a 15% masking rate.
Liquid AI increased this figure to 30%, citing evidence that higher masking rates improve performance at this model scale.
By forcing the network to predict a larger percentage of hidden tokens, the model learns stronger contextual representations.
The Two-Stage Training Process
After converting the architecture, Liquid AI trained the models using a two-stage process.
Stage One: General Language Understanding
The first stage focused on establishing broad language competence.
Key characteristics included:
- Large-scale web corpus
- MLM objective
- 1,024-token context length
This stage teaches the model foundational linguistic knowledge and general semantic understanding.
Stage Two: Long-Context Specialization
The second phase expanded the context window to the full 8,192 tokens.
Training utilized the complete data mixture while emphasizing:
- Factual knowledge
- Legal content
- Multilingual understanding
- Long-document reasoning
This stage was specifically designed to strengthen performance on real-world enterprise workloads that involve lengthy documents.
Inside the LFM2 Hybrid Architecture
Both encoder models are built upon the same hybrid backbone described in the LFM2 technical report.
The architecture combines:
- Gated short-convolution blocks
- Grouped-query attention mechanisms
This hybrid design aims to balance computational efficiency with contextual understanding.
According to Liquid AI, the approach allows performance to scale more effectively as sequence lengths grow.
The result is an architecture intended to remain practical even when processing full contracts, healthcare records, and extensive documentation.
Benchmark Evaluation: Measuring Real Encoder Performance
To evaluate the effectiveness of the new encoders, Liquid AI conducted a comprehensive benchmark study involving:
- 14 encoder models
- 17 downstream tasks
- GLUE benchmarks
- SuperGLUE benchmarks
- Multilingual classification tasks
Every model in the study underwent full fine-tuning for each task.
The reported scores represent performance after task-specific training rather than zero-shot evaluation.
This methodology provides a clearer picture of real deployment scenarios.
LFM2.5-Encoder-350M Performance Results
Among the evaluated models, LFM2.5-Encoder-350M achieved impressive results.
Overall Score
81.02 (±1.00)
This performance placed the model:
4th out of 14 total models
Notably, the three models ranking above it were all substantially larger:
| Model | Parameters | Average Score |
|---|---|---|
| XLM-R XL | 3.5B | 83.06 |
| ModernBERT-large | 395M | 81.68 |
| XLM-R large | 560M | 81.34 |
| LFM2.5-Encoder-350M | 350M | 81.02 |
The highest-ranking model contains approximately ten times more parameters than LFM2.5-Encoder-350M.
This highlights the efficiency of Liquid AI’s architecture relative to significantly larger competitors.
LFM2.5-Encoder-230M Performance Results
The smaller model also delivered strong results.
Overall Score
79.29 (±1.02)
This ranking placed it:
6th among 14 evaluated models
The model outperformed several established alternatives, including:
- ModernBERT-base (78.19)
- EuroBERT-610M (75.87)
- EuroBERT-2.1B (72.19)
Achieving these results with only 230 million parameters demonstrates the effectiveness of the LFM2 conversion strategy.
Why Liquid AI Built Dedicated Encoders
Before releasing these models, Liquid AI already offered retrieval-focused systems such as:
- LFM2.5-ColBERT-350M
- LFM2.5-Embedding-350M
However, benchmark comparisons revealed a significant performance gap.
Scores reported by Liquid AI include:
| Model | Score |
|---|---|
| LFM2.5-Encoder-350M | 81.02 |
| LFM2.5-Encoder-230M | 79.29 |
| LFM2.5-ColBERT-350M | 76.18 |
| LFM2.5-Embedding-350M | 75.68 |
The difference illustrates why specialized encoders remain valuable even when embedding and retrieval models are available.
General-purpose encoders are optimized for broader classification and understanding tasks rather than retrieval-specific objectives.
Transparent and Reproducible Benchmarking
A notable aspect of Liquid AI’s release is its commitment to reproducibility.
The complete benchmarking framework has been open-sourced under the Apache 2.0 license.
The evaluation methodology includes:
- FP32 master weights
- BF16 autocasting
- Shared AdamW optimization recipe
- Learning-rate sweeps across 10 values
- Three selection seeds
- Five fresh evaluation seeds
- Transformers version pinned to 4.56.2
By controlling variables such as software dependencies and optimization strategies, Liquid AI sought to ensure fair model comparisons.
This level of transparency enables researchers and organizations to reproduce results independently.
Real-World Deployment Scenarios
Liquid AI identifies three primary deployment categories for the new encoders.
Edge and Embedded Systems
Many devices operate without dedicated GPUs.
Examples include:
- Automotive systems
- Industrial controllers
- Smart devices
- Embedded computing platforms
For these environments, CPU efficiency is critical.
Regulated Industries
Organizations operating in highly regulated sectors often cannot send sensitive information to external cloud services.
Examples include:
- Healthcare
- Finance
- Legal services
- Government agencies
The ability to process long documents entirely on local infrastructure provides significant advantages.
Cost-Sensitive AI Pipelines
Many enterprises use smaller AI models as an initial filtering layer before invoking larger and more expensive systems.
This architecture can dramatically reduce operational costs.
The LFM2.5 encoders are designed to serve as efficient first-pass models in these workflows.
Understanding the 8K Context Window
The models support an 8,192-token context length.
According to Liquid AI, this corresponds to approximately:
13–15 pages of text
This means a single forward pass can process:
- Full legal contracts
- Comprehensive patient records
- Detailed financial reports
- Long compliance documents
- Extended technical manuals
For enterprise applications, reducing document chunking requirements can simplify workflows and improve contextual consistency.
CPU-Only Demonstrations
To showcase practical applications, Liquid AI released five demonstration systems running entirely on CPU-based Hugging Face Spaces.
The demos include:
Zero-Shot Prompt Routing
Automatically directing requests to the most appropriate workflow.
Zero-Shot Policy Linting
Identifying potential policy violations without task-specific retraining.
Spell Checking
Performing text correction using encoder-based representations.
Multilingual PII Detection
- 40 information categories
- 16 languages
Masked-Diffusion Chatbot
A unique demonstration where the encoder generates responses through iterative token unmasking.
This experimental approach illustrates alternative generation methods beyond standard autoregressive decoding.
Getting Started with the New Encoders
The models integrate directly with the Hugging Face Transformers ecosystem.
Liquid AI exposes:
Lfm2BidirectionalModelLfm2BidirectionalForMaskedLM
The implementation uses auto_map, requiring:
trust_remote_code=Trueduring model loading.
Fine-Tuning Is Required
An important consideration is that these checkpoints serve as foundational encoders rather than task-specific models.
As a result, fine-tuning is necessary before deployment.
Liquid AI provides a dedicated tutorial demonstrating how to fine-tune the models for long-document legal workloads using the full 8K context window.
Choosing Between the 230M and 350M Models
Liquid AI offers straightforward guidance regarding model selection.
Choose LFM2.5-Encoder-350M When:
- Accuracy is the top priority
- Hardware resources are available
- Performance matters more than throughput
Choose LFM2.5-Encoder-230M When:
- CPU resources are limited
- Throughput requirements are high
- Lower latency is essential
- Infrastructure budgets are constrained
One of the most notable performance figures released by the company is CPU inference speed.
At an 8K context length, the 230M model reportedly completes a forward pass in approximately 28 seconds, compared with more than 90 seconds for ModernBERT-base.
Final Thoughts
The release of LFM2.5-Encoder-230M and LFM2.5-Encoder-350M highlights Liquid AI’s continued focus on efficient, practical AI systems designed for real-world deployment.
By transforming decoder backbones into bidirectional encoders through architectural modifications and long-context training, the company has created models that combine strong benchmark performance with CPU-oriented efficiency.
The benchmark results place the models among the strongest performers in their class, while support for 8,192-token contexts, multilingual capabilities, and open-weight availability make them attractive options for enterprises handling long documents and cost-sensitive workloads.
As organizations increasingly deploy AI beyond GPUs and cloud environments, solutions like the LFM2.5 encoder family demonstrate that efficient, long-context encoder architectures remain a critical part of the modern AI ecosystem.
Discover more from AiTechtonic - AI & Informative News
Subscribe to get the latest posts sent to your email.