Nvidia shrinks LLM memory 20x without changing model weights

Nvidia's KV Cache Transform Coding (KVTC) compresses LLM key-value cache by 20x without model changes, cutting GPU memory costs and time-to-first-token by up to 8x for multi-turn AI applications.
TMCnet

Breaking the 100M Token Limit: EverMind's MSA Architecture Achieves Efficient End-to-End Long-Term Memory for LLMs

This approach can be viewed as a memory plug-in for large models, providing a fresh perspective and direction for solving the ...

New KV cache compaction technique cuts LLM memory 50x without accuracy loss

MIT researchers developed Attention Matching, a KV cache compaction technique that compresses LLM memory by 50x in seconds — ...
WFXG

Nota AI Reduces Memory Usage of Upstage's Solar LLM by 72%, Demonstrating Proprietary Quantization Technology

Nota AI, an AI optimization technology company behind the Nota AI brand, announced that it has developed a next-generation quantization technology that significantly compresses the size of Solar, a ...

Marvell Launches Next-generation CXL Switch, Enabling Memory Pooling to Break Through the AI "Memory Wall"

Marvell Technology, Inc. (NASDAQ: MRVL), a leader in data infrastructure semiconductor solutions, today announced Marvell® ...
InfoWorld

Unlocking LLM superpowers: How PagedAttention helps the memory maze

Large language models (LLMs) like GPT and PaLM are transforming how we work and interact, powering everything from programming assistants to universal chatbots. But here’s the catch: running these ...
Semiconductor Engineering

Pooling CPU Memory for LLM Inference With Lower Latency and Higher Throughput (UC Berkeley)

“The rapid growth of LLMs has revolutionized natural language processing and AI analysis, but their increasing size and memory demands present significant challenges. A common solution is to spill ...
EurekAlert!

SNU researchers develop AI technology that compresses LLM chatbot ‘conversation memory’ by 3–4 times

In long conversations, chatbots generate large “conversation memories” (KV). KVzip selectively retains only the information useful for any future question, autonomously verifying and compressing its ...
Semiconductor Engineering

Optimizing LLM Training Under GPU Memory Constraints (Argonne, RIT)

A new technical paper titled “MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall” was published by researchers at Argonne National Laboratory and ...
InfoWorld

Maximizing speed: How continuous batching unlocks unprecedented LLM throughput

Think of continuous batching as the LLM world’s turbocharger — keeping GPUs busy nonstop and cranking out results up to 20x faster. I discussed how PagedAttention cracked the code on LLM memory chaos ...
Morningstar

EverMemOS Redefines Efficiency in AI Memory, Surpassing LLM Full-Context Perfomances with Far Fewer Tokens in Open Evaluation

The evaluation framework was developed to address a critical bottleneck in the AI industry: the absence of consistent, transparent methods to measure memory quality. Today's agents rely on a ...