The rapid evolution of generative artificial intelligence is pushing enterprises beyond experimentation into full-scale deployment. While early AI agent prototypes demonstrated impressive capabilities, scaling them into production systems has exposed deep engineering challenges—particularly around reliability, cost, and maintainability.
One architectural shift gaining attention among researchers and enterprise developers is the separation of workflow logic from inference-time search strategies. By decoupling what an AI agent should do from how it navigates uncertainty, organisations can build systems that scale more efficiently, remain easier to maintain, and deliver more predictable outcomes.
A research collaboration involving experts from Asari AI, MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), and Caltech proposes that this separation may be essential for the next generation of enterprise-grade AI agents. Their framework introduces a new programming model and runtime system designed to address the scalability bottlenecks facing agentic AI deployments.
From AI Prototypes to Production Systems
Generative AI agents have moved quickly from proof-of-concept experiments into real operational workflows. Businesses now deploy agents to:
- Automate customer support
- Translate and migrate legacy code
- Generate compliance documentation
- Conduct research synthesis
- Manage internal knowledge systems
However, as adoption grows, so does the need for reliability. Large language models (LLMs) are inherently probabilistic. The same prompt can produce different outputs across runs, creating inconsistency in automated workflows.
In prototype environments, variability is manageable. In enterprise systems handling financial, legal, or operational tasks, unpredictability becomes a critical risk.
To compensate, development teams typically embed safeguards directly into application code—adding retries, validation loops, fallback prompts, and decision branches.
While effective in the short term, this approach introduces long-term engineering complexity.
The Reliability–Complexity Tradeoff
When reliability mechanisms are hard-coded into agent workflows, systems become difficult to maintain.
Developers must intertwine two fundamentally different concerns:
- Workflow logic – the sequence of business steps an agent must execute
- Inference strategy – the method used to handle uncertainty in LLM outputs
For example, an agent generating legal summaries might include:
- Multiple draft generations
- Rubric-based scoring
- Self-critique loops
- Human approval checkpoints
Embedding these processes directly into workflow code results in sprawling control structures filled with retries and exception handling.
Over time, the codebase becomes brittle. Even minor changes to inference strategies—such as increasing sampling depth or adding validation layers—require structural rewrites.
This entanglement creates technical debt that slows innovation and limits experimentation.
The Entanglement Problem in Agent Design
Researchers describe this issue as an “entanglement problem.” When logic and search are fused, developers lose flexibility in optimising either layer independently.
Consider a simple “best-of-N” generation strategy, where an LLM produces multiple outputs and selects the highest-scoring result. Implementing this requires wrapping the entire agent workflow in iterative loops.
If a team later wants to adopt a more advanced method—such as beam search or tree exploration—they often must redesign the application’s control flow from scratch.
As inference strategies grow more sophisticated, engineering overhead increases exponentially.
This cost discourages experimentation. Teams may stick with suboptimal reliability techniques simply because upgrading them would require too much redevelopment.
Introducing Probabilistic Angelic Nondeterminism (PAN)
To address this architectural bottleneck, researchers introduced a programming paradigm called Probabilistic Angelic Nondeterminism (PAN).
PAN reframes how developers design AI agents. Instead of embedding uncertainty handling directly into workflows, it allows engineers to write deterministic “happy path” logic while delegating probabilistic exploration to a runtime engine.
In simpler terms:
- Developers define what success looks like.
- The runtime system determines how to achieve it under uncertainty.
This model aligns with classical computer science principles, where nondeterministic programs describe possible execution paths while external systems manage optimisation.
ENCOMPASS: A Python Implementation
To operationalise PAN, the research team built a Python framework called ENCOMPASS.
ENCOMPASS introduces a primitive function known as branchpoint(). Developers insert this marker at points in code where LLM inference occurs or where execution may diverge.
These markers signal to the runtime engine that multiple outcomes are possible.
Instead of executing linearly, the system constructs a search tree of execution paths at runtime, exploring alternatives to identify the most successful trajectory.
Crucially, developers do not need to rewrite business logic to test different inference strategies.
“Program-in-Control” vs “LLM-in-Control” Agents
This architecture supports what researchers call program-in-control agents.
In this model:
- The workflow is governed by deterministic code.
- LLMs handle specific subtasks within defined boundaries.
This contrasts with LLM-in-control systems, where the model determines the entire action sequence autonomously.
Enterprises often prefer program-in-control designs because they offer:
- Higher predictability
- Easier auditing
- Regulatory compliance support
- Deterministic failover paths
By separating logic from search, organisations retain governance over agent behaviour while still benefiting from generative flexibility.
Search Algorithms Without Code Rewrites
One of the most powerful aspects of this framework is inference strategy modularity.
Because search operates at the runtime layer, teams can experiment with multiple algorithms without altering application code, including:
- Depth-first search
- Beam search
- Best-first search
- Monte Carlo tree search
Each method balances cost, speed, and accuracy differently.
Developers can optimise performance simply by adjusting runtime parameters rather than rewriting workflows.
Real-World Application: Legacy Code Migration
To demonstrate practical value, researchers applied the framework to a complex enterprise scenario: Java-to-Python legacy code translation.
The agent workflow included:
- Translating source files
- Generating execution inputs
- Running compiled code
- Validating outputs via tests
In traditional implementations, adding search logic required building a full state machine—fragmenting business logic into discrete steps and manually tracking execution states.
This made the code difficult to read, maintain, and debug.
Simplifying Complex Workflows
Using ENCOMPASS, engineers instead inserted branchpoint() markers before each LLM translation step.
The workflow remained linear and human-readable. The runtime engine handled exploration of alternative translations automatically.
This separation allowed teams to deploy advanced search strategies—such as fine-grained beam search—without restructuring the agent.
The result was both improved translation accuracy and reduced engineering complexity.
Performance Scaling Insights
The study revealed notable performance trends.
Applying search strategies at multiple granularities—file level and method level—outperformed simple sampling approaches.
Researchers observed that performance scaled linearly with the logarithm of inference cost. In practical terms, smarter search produced better results without proportionally increasing compute spending.
Fine-grained beam search delivered the strongest outcomes, despite being one of the most complex strategies to implement in traditional architectures.
Cost Efficiency in Enterprise AI
Inference cost remains a central concern for organisations deploying AI agents at scale.
Every additional generation, critique loop, or retry consumes compute resources.
The research compared two optimisation approaches using a “Reflexion” agent pattern, where an LLM evaluates and refines its own outputs:
- Scaling refinement loops
- Applying best-first search
The search-based method achieved similar accuracy at lower cost per task.
This suggests that inference strategy design—not just model size or prompt quality—is a critical lever for financial optimisation.
Flexible Cost–Accuracy Tradeoffs
By externalising search, enterprises gain flexibility in allocating compute budgets.
For example:
- Internal knowledge tools may use low-cost greedy search.
- Customer-facing compliance systems may use exhaustive exploration.
Both can operate on the same underlying workflow code.
This decoupling allows finance and engineering teams to tune performance dynamically without redeveloping applications.
Integration With Existing AI Frameworks
The proposed architecture is not designed to replace popular agent frameworks like LangChain.
Instead, it operates at a different layer—managing execution control flow rather than prompts or tool integrations.
This means organisations can integrate PAN/ENCOMPASS principles into existing stacks without abandoning prior investments.
It acts as an orchestration layer for uncertainty handling.
Engineering Challenges and Limitations
Despite its advantages, the approach introduces new design considerations.
Developers must still:
- Identify correct branch points
- Define success metrics
- Establish evaluation criteria
Search effectiveness depends heavily on scoring mechanisms.
In code translation, correctness can be verified through unit tests. In subjective domains—such as summarisation or creative writing—defining objective scoring functions is far more difficult.
This remains a bottleneck for scalable deployment in content-driven workflows.
Managing State and Side Effects
Another technical challenge involves state management.
At each branch point, the system must replicate program state to explore alternative paths.
While ENCOMPASS manages variable scope and memory internally, external side effects require careful handling, including:
- Database writes
- API transactions
- File system changes
Without safeguards, search exploration could trigger duplicate actions.
Developers must design idempotent operations or sandboxed execution layers to prevent unintended consequences.
Governance and Auditability Benefits
Separating logic from search also enhances governance.
If a particular inference strategy produces hallucinations or compliance risks, teams can modify or replace it globally without altering individual agent workflows.
This supports:
- Version control of AI behaviour
- Regulatory audits
- Risk mitigation reviews
- Enterprise policy enforcement
In regulated sectors such as finance or healthcare, understanding not just outcomes but decision pathways is essential.
Reducing Technical Debt in Agent Systems
Hard-coding probabilistic reasoning into business logic accumulates technical debt over time.
It complicates:
- Testing frameworks
- Monitoring systems
- Upgrade cycles
- Cross-team collaboration
By modularising inference strategies, organisations can maintain agents with the same software engineering discipline applied to traditional systems.
This improves long-term scalability and operational resilience.
Aligning With Modular Software Principles
The PAN model reflects longstanding best practices in software design:
- Separation of concerns
- Modularity
- Abstraction layers
- Reusable components
As AI agents become embedded in mission-critical processes, applying these principles becomes non-negotiable.
Agent engineering is evolving from prompt experimentation into full lifecycle software architecture.
Future Implications for AI Agent Scalability
As inference-time compute grows and models become more capable, execution complexity will increase.
Agents will handle:
- Multi-step reasoning
- Tool orchestration
- Cross-system automation
- Autonomous decision loops
Architectures that isolate uncertainty handling from workflow logic will be better positioned to scale.
They allow innovation at the inference layer without destabilising operational systems.
Toward Enterprise-Grade Agent Infrastructure
The research from Asari AI, MIT CSAIL, and Caltech signals a maturation point in agent engineering.
Key takeaways include:
- Reliability requires architectural change, not just better prompts.
- Search strategy design is central to cost and performance.
- Decoupling logic improves maintainability and experimentation.
- Governance benefits from modular inference control.
As enterprises deepen AI adoption, these architectural principles may define the difference between experimental deployments and durable production infrastructure.
Conclusion
Separating logic from search represents a foundational shift in how AI agents are built and scaled.
By decoupling deterministic workflows from probabilistic inference strategies, frameworks like PAN and ENCOMPASS offer a path to more reliable, cost-efficient, and governable AI systems.
For organisations investing in agentic automation, the implications are significant. Scalability will depend not only on model capability but on the software architectures that orchestrate them.
As AI transitions from novelty to necessity, modular agent design may become the backbone of enterprise intelligent systems—enabling businesses to innovate rapidly without sacrificing control, transparency, or operational stability.