The race to build better AI coding agents has accelerated dramatically over the last year. While benchmarks for general reasoning, coding, and agentic workflows are plentiful, many fail to reflect the realities developers encounter when building production applications. To address this gap, Supabase has introduced Supabase Evals, an open-source benchmark and evaluation framework specifically designed to measure how effectively AI agents work with the Supabase ecosystem.
Unlike synthetic coding tests or isolated programming challenges, Supabase Evals evaluates agents on real-world development tasks. These include creating database schemas, troubleshooting failed Edge Functions, correcting broken Row Level Security (RLS) policies, and resolving infrastructure issues that developers commonly face when building applications with Supabase.
The framework powers both Supabase’s public leaderboard and its internal regression-testing system, enabling continuous measurement of AI agent performance across realistic scenarios. By open sourcing the project, Supabase is giving developers, researchers, and AI companies a transparent way to compare coding agents in environments that closely resemble production workflows.
What Is Supabase Evals?
Supabase Evals is an open-source benchmarking framework that measures how well AI coding agents perform practical Supabase development tasks.
The benchmark evaluates popular coding agents including:
- Claude Code
- Codex
- OpenCode
Rather than focusing solely on code generation, the framework examines whether an AI agent can successfully navigate real development workflows from start to finish.
For example, an agent may be asked to:
- Build a database schema
- Deploy infrastructure components
- Investigate system failures
- Debug Edge Functions
- Repair security policies
- Resolve migration issues
- Configure authentication workflows
Each task is scored based on the quality and correctness of the completed solution.
The benchmark serves as the foundation for the public leaderboard available through Supabase’s evaluation platform and is also used internally by Supabase teams to monitor performance regressions on a daily basis.
Why Supabase Created Evals
The growing popularity of AI coding assistants has created a new challenge for software teams: determining whether an agent can reliably handle production-grade development tasks.
Many existing benchmarks focus on:
- Algorithmic coding problems
- Academic programming challenges
- Isolated software engineering tests
- Synthetic datasets
However, modern application development involves significantly more complexity.
Developers frequently work across:
- Databases
- Authentication systems
- Storage services
- APIs
- Infrastructure layers
- Security policies
- Deployment environments
An AI model that performs well on generic coding benchmarks may still struggle when interacting with a real application stack.
Supabase Evals aims to bridge this gap by testing coding agents against scenarios inspired by:
- Support tickets
- Customer-reported issues
- Bug reports
- GitHub issues
- Real developer workflows
This approach makes the benchmark far more representative of practical software development environments.
Is Supabase Evals Deployable Today?
Yes.
Supabase has released the project publicly under the Apache-2.0 license, making it available for immediate use by developers, enterprises, and AI researchers.
The repository can be executed locally using pnpm, allowing teams to run evaluations against their own models and agent frameworks.
Because the project is fully open source, organizations can:
- Modify benchmark scenarios
- Create custom evaluation suites
- Integrate evaluations into CI/CD pipelines
- Compare multiple coding agents
- Monitor model regressions over time
This flexibility makes Supabase Evals useful beyond simple benchmarking.
Many teams can leverage it as a quality assurance framework for AI-powered development workflows.
Industries That Can Benefit from Supabase Evals
Although the benchmark focuses on the Supabase ecosystem, its implications extend to a broad range of industries.
Developer Tooling
Companies building coding assistants, agent frameworks, and developer productivity tools can use Supabase Evals to validate performance improvements and compare competing approaches.
Cloud Infrastructure
Organizations operating cloud services can evaluate how effectively AI agents interact with infrastructure components, deployment systems, and operational workflows.
Data Platforms
Businesses that depend heavily on databases, analytics systems, and backend services can measure how reliably agents manage critical data operations.
Regulated Industries
Sectors such as fintech and healthcare have especially strong incentives to validate AI-generated code.
In these environments, a single incorrectly configured security policy can create significant compliance and security risks.
For example, an AI agent that writes an incorrect Row Level Security (RLS) policy could unintentionally expose sensitive customer data, turning a coding error into a serious security incident.
Practical Applications of Supabase Evals
Supabase Evals is designed for much more than leaderboard rankings.
Organizations can use the framework for several operational purposes.
Regression Testing
Documentation updates, prompt changes, and skill modifications can affect agent behavior.
Regression evaluations help teams identify performance degradation before it reaches production.
SDK Release Validation
Software development kits often introduce new functionality that AI agents must understand.
Benchmarking can act as a release gate to ensure coding agents continue performing correctly after updates.
Agent Comparison
Teams evaluating multiple AI systems can run head-to-head comparisons between different models, frameworks, and harnesses.
This provides objective evidence for selecting the most capable coding assistant.
Internal Monitoring
Organizations deploying AI-powered development workflows can continuously measure performance to detect regressions early.
Deployment Requirements and Constraints
Although Supabase Evals is easy to access, running the framework requires several components.
Users need:
- A Docker daemon
- Provider API keys
- Available ports 54321–54329
The benchmark relies heavily on containerized environments to reproduce realistic development scenarios.
Without these resources, the evaluation system cannot execute tasks accurately.
For organizations already working with containerized infrastructure, these requirements are generally straightforward to satisfy.
How Supabase Designed the Evaluation Framework
One of the most interesting aspects of Supabase Evals is the structured methodology used to create benchmark scenarios.
Rather than assembling random tasks, Supabase organized evaluations across three distinct dimensions:
Products
The benchmark covers a wide range of Supabase services:
- Database
- Auth
- Storage
- Edge Functions
- Realtime
- Cron
- Queues
- Vectors
- Data API
This ensures that evaluations represent the broader Supabase ecosystem instead of focusing exclusively on one component.
Topics
Each scenario incorporates critical development concepts including:
- RLS
- Security
- Migrations
- SQL
- SDK
- Observability
- Self-hosting
- Tests
- Declarative schema
These categories reflect many of the challenges developers encounter when building and maintaining modern backend systems.
Stages
Supabase also evaluates agents across multiple workflow stages:
- Build
- Deploy
- Investigate
- Resolve
This structure allows the benchmark to measure not only coding ability but also debugging and operational reasoning.
Building the Scenario Set
After defining products, topics, and stages, Supabase selected the smallest possible set of scenarios that touched every dimension at least once.
The scenarios were grounded in real-world sources, including:
- Support tickets
- Bug reports
- GitHub issues
This design philosophy helps ensure the benchmark remains representative of actual developer experiences rather than hypothetical challenges.
As a result, agents are tested on problems that engineers genuinely encounter while building and operating applications.
Benchmark Scenarios vs Regression Scenarios
Supabase Evals separates tasks into two different evaluation suites.
Benchmark Scenarios
Benchmark scenarios are public and focus on breadth.
These tasks appear in published rankings and leaderboard results.
Their purpose is to provide a standardized measure of overall agent performance.
Regression Scenarios
Regression scenarios focus on previously identified failure modes.
These evaluations are refreshed daily and help Supabase detect emerging issues.
Importantly, regression results do not affect public benchmark scores.
This separation allows Supabase to maintain leaderboard consistency while continuously testing new edge cases internally.
Running Against Real Environments Instead of Mocks
A major differentiator of Supabase Evals is its reliance on real infrastructure.
Many AI benchmarks use mocked environments or simplified simulations.
Supabase chose a different approach.
Every evaluation runs against an actual environment.
The framework launches:
- A hosted-like stack
- A local CLI project
- Containerized infrastructure
This allows agents to interact with:
- The real MCP server
- The actual Supabase CLI
Because the environment behaves like a production deployment, benchmark results are far more reflective of real-world performance.
Platform-Lite Runtime Architecture
The framework includes a platform-lite runtime that exposes a Management API-compatible interface.
This layer is powered by:
The runtime enables realistic management operations while keeping evaluation environments lightweight and reproducible.
As a result, agents can execute tasks in conditions that closely mirror production systems without requiring full-scale infrastructure deployments.
How Supabase Scores AI Agents
Benchmark scoring combines multiple evaluation techniques.
Deterministic Validation
Objective checks verify whether the expected outcome was achieved.
Examples include:
- Correct schema creation
- Successful deployment
- Fixed configuration issues
- Working security policies
These checks eliminate ambiguity and provide reproducible results.
LLM-as-a-Judge
Supabase supplements deterministic scoring with an LLM-based evaluation process.
This approach helps assess outcomes that cannot easily be validated using binary tests alone.
The combination of deterministic checks and LLM evaluation provides a more complete assessment of agent performance.
Retry Policy
Each agent receives:
- One retry before grading
This mirrors real-world development behavior, where developers often get a second attempt to correct mistakes before final evaluation.
Anatomy of an Evaluation Scenario
Every evaluation directory follows a standardized structure.
The framework includes:
PROMPT.md
Contains:
- Task instructions
- Frontmatter metadata
- Evaluation context
EVAL.ts
Contains:
- Scoring logic
- Validation procedures
- Success criteria
Optional Remote and Local States
Scenarios may also include:
- remote/
- local/
These directories define the starting environment for each task.
When a scenario includes a local workspace or specifies:
- interface: cli
the framework automatically launches a Docker sandbox containing the real Supabase CLI.
This ensures agents operate within authentic development environments.
Key Findings from Early Benchmark Results
One of the most surprising discoveries from Supabase’s evaluations is how capable modern coding agents have become even without specialized skills.
According to the benchmark:
Build Stage Performance
Both:
- Opus 5
- Kimi K3
achieved:
- 100% scores without assistance
This indicates that top-tier models can already complete many Supabase development tasks successfully without additional guidance.
The Impact of Skills
While leading models performed strongly on their own, skills still improved outcomes for several systems.
Reported improvements include:
| Model | Without Skills | With Skills |
|---|---|---|
| Sonnet 5 | 78% | 100% |
| GPT-5.6 Sol | 89% | 100% |
| GPT-5.4 mini | 78% | 89% |
These results suggest that skills provide the greatest benefit for smaller or less capable models.
Top-performing systems appear less dependent on supplementary instructions.
Three Major Weaknesses Identified
The benchmark also revealed recurring agent weaknesses.
1. Migration Strategy Problems
Many agents manually write migrations instead of using declarative schemas.
This behavior prompted Supabase to update its skill guidance to encourage best practices.
2. Authentication Verification Issues
Agents frequently verify authentication manually rather than using:
To address this, Supabase introduced additional package selection guidance.
3. Inconsistent Documentation Usage
Documentation behavior varied dramatically across models.
Supabase observed that:
Codex / GPT-5.6
- Read approximately 8 documentation pages per scenario
Claude Code
- Read roughly 2 documentation pages per scenario
- Consulted documentation in fewer than 40% of scenarios, even when skills were loaded
These findings highlight meaningful differences in how coding agents gather and utilize external knowledge.
Why Supabase Evals Matters
As AI coding assistants increasingly participate in production software development, organizations need reliable ways to measure performance.
Supabase Evals addresses several critical industry challenges:
- Real-world benchmarking
- Security validation
- Regression monitoring
- Agent comparison
- Infrastructure-aware testing
Unlike synthetic coding evaluations, it measures whether agents can successfully operate within realistic backend environments.
For companies building AI-powered development tools, this provides a more accurate picture of practical capabilities and limitations.
Key Takeaways
Supabase Evals represents a significant step forward in AI coding evaluation by focusing on authentic developer workflows rather than isolated programming challenges.
Key highlights include:
- Supabase open sourced supabase/evals under Apache-2.0.
- Scenarios run against real containerized Supabase stacks, not mocks.
- Scoring mixes deterministic checks with LLM-as-a-judge; one retry allowed.
- Skills mattered least for top models, most for smaller ones.
As AI agents continue moving from experimentation to production software development, benchmarks like Supabase Evals will play an increasingly important role in determining which systems can be trusted to build, debug, secure, and maintain real-world applications.
Discover more from AiTechtonic - AI & Informative News
Subscribe to get the latest posts sent to your email.