https://arxiv.org/abs/2509.09498

SEDM gives AIs a living memory that automatically filters for useful experiences, preventing data overload and helping them learn and adapt more efficiently across different tasks.

The Memory Crisis: A Bottleneck at the Foundation

Long-running multi-agent systems face a critical constraint: the unbounded accumulation of interaction history. As agents engage with environments and collaborate over time, they generate vast trajectories of experience—creating an inevitable collision between scale and performance.

Three fundamental challenges emerge:

Signal Degradation: Memory repositories fill with redundant or obsolete information, drowning critical insights in noise and degrading decision quality at crucial moments.
Computational Collapse: Unmanaged growth triggers exponential increases in retrieval costs and system latency, making sustained operation economically and technically infeasible.
Transfer Failure: Hard-won knowledge from one domain remains locked in context, unable to generalize—limiting agents to narrow, brittle expertise.

Vector retrieval and hierarchical storage—the incumbent approaches—fail to address the root problem: they manage quantity, not quality.

A Living Memory System: The SEDM Vision

Memory should not be a passive archive. It should be a living substrate—one that evaluates, optimizes, and evolves alongside the agent itself.

The memory framework we are building:

Guards Its Boundaries: Verifies the value of candidate memories before admission, ensuring only high-utility experiences enter the repository.
Evolves Continuously: Consolidates redundant knowledge, amplifies proven insights, and prunes outdated or harmful entries—maintaining a lean, high-signal substrate.
Transfers Fluidly: Abstracts task-specific patterns into generalizable principles, then diffuses them across domains—accelerating learning and adaptation.

This is SEDM (Scalable Self-Evolving Distributed Memory): memory as the engine of continuous intelligence.

The SEDM Architecture: Admission, Scheduling, Diffusion

SEDM establishes a complete lifecycle for memory—from candidate evaluation to cross-domain transfer—built on three core innovations:

Verifiable Admission: Only Proven Value Enters

Every candidate memory must earn its place through empirical validation.

Self-Contained Execution Contexts (SCEC): Each agent decision—inputs, reasoning, actions, outcomes—is encapsulated as a reproducible, standalone unit. This creates a universal substrate for verification.
Distributed A/B Validation: SCECs enable massive parallelization. Thousands of execution contexts are replayed across compute clusters, comparing outcomes with and without the candidate memory—simulating counterfactual performance at scale.
Utility-Gated Storage: Only memories that demonstrably improve task success—higher accuracy, lower cost—are admitted. Each receives a quantified utility weight upon entry.

This mechanism guarantees that every item in long-term memory has a verified value. Quality is enforced at the foundation.

Adaptive Scheduling: Dynamic Evolution in Production

An intelligent Memory Controller orchestrates the entire lifecycle post-admission.

Utility-Driven Retrieval: Recall prioritizes not just relevance, but proven impact. High-utility memories surface first, ensuring agents leverage their most powerful experiences.
Continuous Optimization: The controller actively maintains repository health—consolidating redundant entries, amplifying frequently validated insights, and pruning obsolete or counterproductive memories.

The result: a memory substrate that remains efficient, high-signal, and performant—even as the system scales.

Cross-Domain Diffusion: Scaling Knowledge, Not Just Data

SEDM enables systematic knowledge transfer across task boundaries.

Abstraction: Specific memories ("Strategy X succeeded in task A") are distilled into generalizable patterns—extracted principles that transcend narrow contexts.
Transfer and Re-Validation: Abstracted knowledge is injected as a candidate into new domains, where it undergoes rapid SCEC-based verification and weight recalibration.

Experience from one frontier—fact verification—becomes leverage in another—multi-hop reasoning. This is not transfer learning. This is knowledge diffusion.

Breakthrough Results: Beyond the Ceiling

Rigorous evaluation across industry benchmarks reveals a clear frontier shift:

Accuracy Gains: On complex reasoning tasks (FEVER, HotpotQA), SEDM surpassed state-of-the-art baselines. On FEVER, SEDM achieved 66—decisively outperforming G-Memory (62) and no-memory baselines (57).
Efficiency Breakthrough: Not only did SEDM achieve higher accuracy—it did so while consuming fewer tokens than competing systems. A double win that previous architectures could not deliver.
Verified Transfer: Knowledge diffusion proved effective. Leveraging cross-domain memory from fact verification, SEDM's multi-hop reasoning performance exceeded in-domain-only training (39 → 41)—for the first time demonstrating systematic, verifiable knowledge transfer.

The Next Era: Sustainable, Generalizable AI

SEDM is not an optimization. It is a foundation for the next paradigm.

Sustainability: By solving the memory bottleneck, SEDM makes long-term autonomous systems economically and technically viable—agents that operate reliably over months, not minutes.
Efficiency at Scale: Dramatic improvements in compute efficiency unlock advanced AI capabilities for cost-sensitive, production-scale deployments.
Generalization: Systematic knowledge transfer moves agents beyond narrow expertise—toward systems that learn once and apply broadly.

We are shifting memory from a static substrate to a co-evolving intelligence layer—one that grows, adapts, and optimizes in tandem with the agents it serves.

The Path Forward

SEDM proves that self-optimizing memory is not only feasible—it is the only path to scalable, generalizable AI systems.

The next frontier:

Scaling to massively distributed multi-agent networks with shared memory substrates.
Advancing abstraction mechanisms—toward symbolic reasoning and meta-learning.
Building open, collective memory networks—where any agent can contribute to and benefit from a shared web of validated knowledge.

The age of scaling parameters is giving way to the age of scaling minds. Through its admission-scheduling-diffusion architecture, SEDM provides the first complete solution to the memory frontier—a foundation for AI systems that learn continuously, adapt fluidly, and operate sustainably.

The strongest intelligence may be distributed. The longest-lasting intelligence must evolve.

Technical Terms Glossary

Technical Term	Wikipedia Link
multi-agent system	https://en.wikipedia.org/wiki/Multi-agent_system
RAG	https://en.wikipedia.org/wiki/Retrieval-augmented_generation
ABTest	https://en.wikipedia.org/wiki/A/B_testing
multi-hop reasoning	https://www.moveworks.com/us/en/resources/ai-terms-glossary/multi-hop-reasoning
G-Memory	https://arxiv.org/abs/2506.07398
HotpotQA Dataset	https://hotpotqa.github.io/
Fever Dataset	https://fever.ai/dataset/fever.html
LoCoMo Dataset	LoCoMo: Data and Code for the ACL 2024 Paper "Evaluating Very Long-Term Conversational Memory of LLM Agents"