Meet Needle 2: The Tiny 45M-Parameter AI Model That Runs Tool Calling in Just 28MB of RAM

Artificial intelligence models have become increasingly powerful, but they have also grown larger, more expensive and more dependent on cloud infrastructure. Cactus Compute is taking a different approach with the launch of Needle 2, an open-source AI model designed specifically for tool calling, device control and structured data extraction while running entirely on low-power hardware.

With only 45 million parameters, Needle 2 is remarkably small compared with modern large language models. The entire system is packaged as a 14MB binary, requiring only about 28MB of RAM to run a complete session. Unlike many AI solutions that depend on GPUs, cloud servers or dedicated AI chips, Needle 2 is designed to operate efficiently on devices with limited computing resources, including smartphones, wearables, embedded systems and Internet of Things (IoT) hardware.

The model introduces a different philosophy for edge AI. Instead of trying to answer every question or generate long-form text, Needle 2 focuses on a specific task: translating user requests into structured actions and function calls. According to Cactus Compute, mapping a spoken or written instruction to a predefined function does not require massive world knowledge, making a lightweight model sufficient for many real-world applications.

What Is Needle 2?

Needle 2 is an open AI model developed by Cactus Compute for offline tool calling, local device control and structured extraction tasks.

The model uses CQ2-bit quantization, a technique that compresses weights during training rather than after the model is built. This means the deployed version is identical to the trained version, helping maintain efficiency without requiring additional conversion steps.

Needle 2 is packaged inside a custom C++ inference engine, eliminating the need for external runtimes or model downloads during inference. Users simply deploy the binary and run the model locally.

This design allows Needle 2 to operate on a wide variety of hardware platforms, including:

• macOS
• Windows
• Linux
• Android
• iOS
• watchOS
• tvOS
• WebAssembly
• ARM-based devices
• RISC-V processors
• Embedded systems

The company notes that the Pebble Index 01 app already uses Needle locally for offline voice commands.

High Performance on Low-Cost Hardware

One of Needle 2’s biggest strengths is its ability to deliver fast AI processing without specialized hardware.

According to Cactus Compute, the model can generate:

• Around 500 tokens per second on a Raspberry Pi 5
• Between 400 and 1,500 tokens per second on Meta Quest 3S and Apple Vision Pro
• Approximately 300 to 700 tokens per second on smartphones priced below $200

These speeds are notable because the model does not rely on GPUs or neural processing units (NPUs). Instead, it runs directly on standard CPUs.

For edge devices, lower memory requirements translate into lower power consumption, reduced costs and better offline functionality.

The Simple Attention Network Architecture

Needle 2 is built using a custom architecture called the Simple Attention Network.

The model contains 27 layers with a width of 512 units, using a combination of several techniques to improve efficiency:

• Grouped Query Attention (GQA)
• Hadamard MLPs replacing traditional feed-forward networks
• Engram key-value memory based on hashed n-gram tables
• Multi-lane hyper-connections

The underlying research has been published in an academic paper titled A Controlled Study of Attention-Only Transformers.

Needle 2 was pretrained using a proprietary dataset of 115 billion tokens, followed by 38 billion tokens of post-training.

Despite its small size, the model uses only 70 million floating-point operations per token, significantly lower than many competing models.

For comparison:

• LFM2.5 230M uses 460 MFLOPs
• FunctionGemma 270M uses 540 MFLOPs
• Apple FM approaches 6,000 MFLOPs

This efficiency makes Needle 2 particularly suitable for hardware with limited processing capabilities.

Optimized for Offline AI and Structured Output

Needle 2 has been designed with practical deployment in mind.

The model’s compressed weights never fully expand into memory. Instead, 2-bit codes are processed directly within vector registers, allowing computations to remain in an efficient integer format.

At startup, the software automatically detects the device’s processor and selects the best optimization path, including:

• NEON
• AVX2
• SDOT
• RISC-V vectors
• WebAssembly SIMD
• Standard CPU processing

This enables strong performance across a broad range of hardware platforms.

Another key feature is the model’s grammar-based output system.

Developers can define JSON schemas, and Needle 2 constrains every generated token to match the required format. Because the engine knows which tokens are valid in advance, it can skip unnecessary calculations, reducing processing requirements.

This approach is especially useful for:

• API calls
• Device commands
• Data extraction
• Automation workflows
• Structured responses

Memory Management and Tool Retrieval

Memory efficiency is another major advantage of Needle 2.

The model uses a 256-token sliding attention window, while important system instructions and tool definitions remain fixed in memory.

As a result, RAM usage stays close to 28MB, regardless of conversation length.

Tool selection is also optimized.

If an application contains five or fewer tools, all remain directly accessible. For larger tool libraries, Needle 2 uses a retrieval system that selects the most relevant functions for each request.

Only the top five tools are made available to the model, reducing complexity and improving accuracy.

This is particularly valuable for embedded systems and applications with many possible commands.

Built-In Confidence Scoring

Needle 2 includes a confidence mechanism that helps determine whether the model is sufficiently certain about an action.

Each response includes a confidence score derived from:

• A calibration model
• Decoding probabilities

If the model cannot confidently determine the correct action, it returns an empty call represented by:

[]

This design allows developers to create systems that:

• Execute actions automatically when confidence is high
• Ask users for clarification when confidence is lower
• Escalate requests to cloud AI services when necessary

Such hybrid workflows can reduce cloud costs while maintaining reliability.

Benchmark Performance

Cactus Compute evaluated Needle 2 using several public tool-calling benchmarks.

Results show that Needle 2 performs strongly in domains related to consumer devices and structured actions.

The model achieved:

• 63.7 on Mobile Actions
• 17.0 on DroidCall
• 32.6 on Seal-Tools In-Domain
• 28.7 on Seal-Tools Out-of-Domain
• 42.6 on BFCL v4 Single-Turn

Needle 2 led both Seal-Tools evaluations and achieved 98.3% function-name accuracy on Mobile Actions.

However, larger models performed better on broader benchmarks such as BFCL v4.

Cactus Compute attributes this difference to specialization. Needle 2 was designed primarily for consumer-device actions rather than general-purpose enterprise APIs.

The model also achieved a 93.4% well-formed output rate, demonstrating its ability to generate reliable structured responses.

Where Needle 2 Could Be Used

Needle 2 opens up new possibilities for offline AI applications.

Potential use cases include:

• Smart home devices
• Wearables
• Automotive voice systems
• Service robots
• Retail kiosks
• Point-of-sale systems
• IP cameras
• Routers
• Medical devices
• Industrial automation

It is especially valuable in privacy-sensitive environments where user data should remain on the device rather than being sent to the cloud.

The Future of Edge AI

Needle 2 highlights an important trend in artificial intelligence: not every task requires a massive language model.

For many applications, particularly those involving function calling, device control and structured data extraction, smaller specialized models can deliver excellent results while operating entirely offline.

By combining 45 million parameters, a 14MB footprint, 28MB RAM usage and strong tool-calling capabilities, Needle 2 demonstrates how efficient AI can extend beyond cloud servers and into everyday devices.

As edge computing continues to grow, lightweight models such as Needle 2 could play a major role in making AI faster, cheaper, more private and more accessible across billions of connected devices.