Latent Compaction and Cortex Context Preservation

Transcript

Introduction Welcome to arXiv in 5 — LLM memory. Today, we examine two October fourteenth preprints exploring how to preserve useful context in language models through Latent Compaction and Cortex. Latent Compaction with Test-Time Training When language models process long texts, they store large key-value caches that dramatically increase memory use and attention costs during generation. Compressing these caches is essential, but it must be done in a way that preserves crucial information for future, unseen questions. To address this challenge, researchers have introduced a technique called latent compaction, which learns a compact, document-specific cache through test-time training . To set up this process, the authors initialize cache entries using a combination of reconstruction importance, teacher attention, and key-space diversity. The training procedure itself happens in two distinct phases. First, it fits the attention outputs and log-normalizers to match those of the full, uncompressed cache, while keeping the reconstruction-selected entries completely frozen. Next, it jointly optimizes the entire cache using end-to-end distillation from the full-cache model. It is important to note that the model weights themselves remain entirely frozen throughout this process. Furthermore, the training relies exclusively on generic prompts about the source document, meaning it operates without any prior access to downstream questions or target answers . When evaluating their approach, the authors tested the method on benchmarks including RULER, LongBench, and Phonebook Lookup. Their experiments demonstrated substantial performance gains over standard budget-matched compression baselines. For instance, on the RULER benchmark, the method achieved ninety-six point seven percent accuracy under a thirty-two times compression ratio, compared to just forty percent for the KVZip+ baseline. The authors also report that the method successfully transfers across different model scales and families, showing clear retrieval gains on both Qwen3 and MiniCPM5 architectures. Ultimately, these findings suggest that treating the key-value cache as a trainable neural network and optimizing it via test-time training can effectively maintain high retrieval accuracy even under aggressive memory compression. Cortex: Consolidating Context into Parametric Memory Let us turn to the second preprint submitted on October fourteenth, titled Cortex: Consolidating Context into Parametric Memory. The authors address a fundamental limitation in how large language models handle context. While context can significantly improve performance, those gains are often completely lost once the context is removed. Inspired by how biological systems handle memory consolidation, the authors formulate Memory Consolidation as a way to internalize these performance gains without losing non-target capabilities. This approach connects context-specified editing, skill learning, and behavioral steering under a single common objective. To rigorously test this, the authors introduce a five-task-family testbed that measures both consolidation and retention across single, batch, and sequential updates. To solve this challenge, they propose Cortex. The system works by generating prospective queries that anticipate how context might be used later. It then distills the context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. The authors evaluate Cortex across model scales ranging from four to one hundred and twenty billion parameters, covering dense and mixture-of-experts backbones, as well as text-only and multimodal models. According to the evaluation, Cortex sits on the empirical consolidation-retention Pareto frontier in most of the tested settings. For instance, the authors report that after performing fifty sequential factual edits on a fourteen billion parameter model, Cortex-Structural successfully recovered eighty-eight point nine percent of the context-induced performance gain while maintaining ninety-nine point eight percent of its original non-target performance. Keep in mind that this overview is based strictly on the authors' abstract, and full-paper review is required for complete implementation details. Conclusion Both Latent Compaction and Cortex offer distinct ways to preserve valuable information in language models. You can find the exact source links in the episode notes. Thank you for listening to arXiv in 5 — LLM memory.