Researchers at Pathway, Bielik AI, and NYU have just published results for a new model called BDH-CQ, which breaks the previously reported cost-accuracy Pareto frontier in ARC-AGI-1 and sets a new state-of-the-art for benchmark cost efficiency.

• BDH-CQ is a 150M-parameter reasoning model.

• It learns each new task from the examples it’s shown at inference time (in-context learning), which progressively updates a recurrent memory rather than filling a growing context window. It then solves a given query by iteratively reasoning in a structured, continuous latent state rather than using a verbalized chain of thought.

• It achieves 29.5% pass@2 on ARC-AGI-1 at an inference cost of $0.0007 per task. This is less than 1/10^th of a cent per task!

• Although this is not the highest-accuracy result, its significance is clear when viewed alongside other LLMs’ results.

  • BDH-CQ is ~57x cheaper than GPT 5.6 Luna (Low), which scores 34.2% (only 4.7% higher) at $0.040.

  • Even after OpenAI announced a recent 80% API price reduction, BDH-CQ is still ~11x cheaper than GPT 5.6 Luna (Low).

  • GLM 5 scores 44.7% (15.2% higher), but costs 243x more per task than BDH-CQ.

• BDH-CQ is based on a post-Transformer sequence-model architecture called ‘Dragon Hatchling’ (BDH).

• Early pretraining experiments show that it has Transformer-like scaling behavior across scales from 1B to 600B parameters while preserving its latent reasoning capabilities.

Read more about this research using these links: Hugging Face | ArXiv | Blog

1. Qwen3.8-Max

• Alibaba released its most capable model, Qwen3.8-Max, which is heavily focused on coding, autonomous agents, and long-horizon tasks.

• It outperforms previous Qwen models in agentic coding, computer use, research tasks, and co-working workflows.

• The model has 2.4 trillion total parameters (95B active per token) and uses a Mixture-of-Experts architecture.

• With this model, the team is, for the first time, open-sourcing the weights of a Qwen-Max-class model.

Read more about this release using this link.

Subscribe now

2. On-Policy Self-Distillation without Any Supervision

• U-OPSD (Unsupervised On-Policy Self-Distillation) is an algorithm that helps an LLM improve its reasoning without ground-truth answers, rewards, or a stronger teacher model. With U-OPSD, an LLM learns and improves using only its own generations, guided by internal consistency.

• The process starts with the model generating multiple solutions for each problem and using a majority vote to form its own pseudo-answer, provided it meets a self-consistency threshold.

• It then conditions the model’s distribution on the pseudo-solution and distills itself on the disagreeing completions, letting the model correct itself precisely where it is confidently wrong.

• U-OPSD consistently outperforms the base models and matches or surpasses supervised methods with ground truth (GT), such as OPSD and GRPO.

Read more about this research using this link.

3. Leanstral

• Leanstral is Mistral AI’s series of open-source generalist code-agent models for Lean 4.

• The model has a Mixture-of-Experts architecture with 119B total and 6B active parameters.

• It runs within the open-source Mistral Vibe coding-agent harness rather than a specialized theorem-proving scaffold, and uses no test-time scaling method beyond context compaction.

• Leanstral 1.5's performance is comparable to far larger, proprietary systems, as it saturates miniF2F, solves 587/672 problems on PutnamBench, and reaches a new state-of-the-art of 34% on FATE-X and 43.2% pass@8 on FLTEval.

• Beyond competition mathematics, Leanstral can formally verify code and resolve bugs and issues in real-world repositories across graduate-level mathematics, mathematical finance, and code verification.

Read more about this research using this link.

4. The Beginning of ChatGPT Ads

• This is the first empirical study of advertising content shown within ChatGPT during the early 2026 rollout.

• Researchers created 91 simulated U.S. accounts and ran hundreds of prompts to collect ads from 191 unique advertisers across 127,801 conversations.

• The results show that lower-income simulated accounts were significantly more likely to receive ads irrespective of their race.

• Product/recommendation-style prompts were more likely to trigger advertising.

• The ads were heavily skewed towards consumer goods and directed users to a specific advertiser rather than a particular product.

• Ads were clearly marked “Sponsored” and separated from the LLM’s response text rather than embedded within it.

• There were near-zero ad rates for medical conditions, mental health, and political prompts.

Read more about this research using this link.

5. Ten Advances in Mathematics and Theoretical Computer Science

• An internal OpenAI model, Astra, produced 10 research advances in mathematics and theoretical computer science.

• These results go beyond reproducing known proofs to include new resolutions, counterexamples, and improved bounds for many open problems in these subjects.

Read more about this research using this link.

6. Maple-Preview

• Maple-Preview is DeepGrove’s open-source ternary-weight reasoning model, designed to run efficiently on consumer hardware.

• The model uses a Mixture-of-Experts architecture with 256 experts (8 active) and has 20.2B total parameters (1.49B active).

• The model has a 131k-token context window and is competitive with larger models at strong mathematical/general reasoning, including IMO-level problems.

• It runs at 218 tokens/sec on an M4 Mac mini, which is 5-16× faster than efficient models like Gemma 4, Qwen3.5, and gpt-oss.

Read more about this release using this link.

Subscribe now

7. Scaling Automated Post-Training with Locus

• Locus is Intology’s automated research agent, extended to autonomously run the experiments needed to post-train other AI models.

• PostTrainBench is a benchmark where an agent is given a base model, a target benchmark, a single-H100 compute node, and internet access, and its task is to produce a post-trained version of the model optimized for the target benchmark.

• Locus with Opus 5 has state-of-the-art post-training capabilities on PostTrainBench, where it outperforms every frontier-agent baseline.

• On a larger compute variant of PostTrainBench, called PostTrainBench+, when given thousands of H100-hours, Locus continues improving while general-purpose coding agents plateau, and Locus-trained Qwen3-1.7B models collectively surpass Qwen’s official human-tuned checkpoint.

• Beyond post-training, across live prize-money Kaggle competitions, Locus achieved an average rank beating 89.5% of human competitors.

• Working with Bubble, a no-code app development platform, Locus discovered and executed a post-training recipe end-to-end, and the resulting model now serves millions of users at ~2.8× lower error, ~5.4× lower latency, and ~100× lower cost than the frontier API it replaced.

Read more about this release using this link.

8. LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

• Instead of sending every user query to a single frontier model, model routing dynamically routes each query to the model best suited to it.

• LLMRouter is an open-source framework and benchmark for standardized and modular implementation of LLM routers.

• It defines routing as a sequential decision process using five components:

  • Context encoder

  • Model encoder

  • Scoring function

  • Decision rule

  • Learning signals

• Existing methods can then be organized into three families: single-turn, multi-turn, and personalized routing.

• The authors also introduce xRouteBench, a benchmark for evaluating routers across general LLM tasks, conversational memory, vision/video, time-series, and personalization using a pool of 18 models with different costs and capabilities.

• Using these, an empirical study of LLM routing shows that learned routers achieve a 14.6% relative improvement over the strongest fixed-model baseline.

• The results also show that multi-turn routing is not consistently better than simple single-turn routing because decomposition and aggregation can introduce extra cost and redundant information.

Read more about this research using this link.

Subscribe now

9. OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents

• OneDayAgent is a harness for autonomous agents that helps them complete long-horizon, multi-step everyday tasks without losing sight of their goals, state, or context.

• It turns an open-ended request into a managed execution process that decomposes tasks into bounded subtasks, maintains execution memory under context pressure, and verifies and repairs the final output.

• OneDayAgent with GLM-5.2 sets a new state-of-the-art with an overall score of 0.821 on AgentIF-OneDay, which contains 104 realistic work, study, and life tasks.

• The same harness runs across five backend LLMs from three model families, showing that it generalizes well without any tuning.

Read more about this research using this link.

10. ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

• ResidencyRL is an RL method from Google DeepMind for training clinical AI agents through simulated multi-turn clinical encounters rather than only on static medical Q&A.

• An agent, initialized from Gemini 3.5 Flash, is trained on ~57,000 simulated clinical cases, with each encounter running for up to 60 dialogue turns and including 8 tool calls. The training uses GRPO with rewards covering diagnostic accuracy, management, history-taking, communication, documentation, and safety.

• On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7% under adversarial conditions and reduces missed red flag rates by 31%.

• Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of cases.

• The gains also transferred to unseen benchmarks (AMIE, AgentClinic, and CRAFT-MD) and specialties(Oncology).

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️

Share

The process of Knowledge distillation (KD) was described in a 2015 paper by Hinton et al. It involves training a weaker/smaller model called the “student model” to imitate the outputs of a stronger/ larger model called the “teacher model”.

The student learns from teacher-generated output trajectories, rather than from trajectories generated by itself. This is why conventional KD is an Off-policy post-training method (as opposed to On-policy distillation).

The process of KD goes like this:

1. Take a larger/stronger teacher model and a smaller/weaker student.

2. Create a dataset of prompts relevant to the training task.

3. Pass a prompt to the teacher model and collect the teacher's next-token probability distribution over the vocabulary. These probabilities are called “soft targets” (as opposed to “hard targets” or training labels).

4. Pass the prompt and the teacher’s response to the student model and collect the student’s next-token probability distribution conditioned on the same prefix as the teacher.

5. Use a loss function to measure how much the teacher’s and student’s probability distributions differ. A commonly used loss is the KL Divergence.
   Optionally, a hyperparameter called Temperature is applied to the logits of both the teacher and the student models when calculating the distillation loss. This helps flatten the teacher’s probability distribution of predictions, which better reveals the relationships between different tokens.

6. In many cases, alongside KL Divergence, a Cross-entropy loss between the student’s final prediction and the ground-truth label, or “hard target”, is also calculated and used to formulate the combined distillation loss. This combined loss helps the student model to learn from both the teacher model and from ground-truth data.

7. The teacher model is kept frozen, but the student model’s parameters are updated using backpropagation.

8. This process is repeated over all prompts in the dataset as the loss decreases and the student model learns to approximate the teacher model's outputs.

Distillation with real numbers

The process of knowledge distillation is shown in detail in the illustration below.

Read more

1. DeepSeek-V4-Flash-0731

DeepSeek-V4-Flash-0731 is the official release of DeepSeek-V4-Flash that replaces its preview version.

The model has a 304B-parameter architecture, uses DSpark speculative decoding for faster inference, and supports low, high, and max reasoning-effort levels.

It outperforms the preview version on multiple benchmarks while activating fewer parameters, and has competitive performance with the strongest proprietary models available today.

It scores 50 points on the Artificial Analysis Intelligence Index, similar to GPT-5.6 Luna (51 points) and Google Gemini 3.6 Flash (50 points), while costing ~60% less per task than Luna and 10x less than Gemini, with a 98% discount on cache hits.

Read more about this release using this link.

2. Pangram 4

Pangram 4 is the latest deep-learning-based AI-text classification model from Pangram Labs with state-of-the-art performance in AI text detection and is being used by to detect ever-increasing AI slop.

Pangram 4 is a LoRA fine-tuned MoE model trained on human text paired with AI-rewritten and AI-edited versions of itself, labeled clause-by-clause by matching against the human source. At inference, it scores overlapping 512-token windows, averages the logits, and uses a CRF (Conditional Random Field) to decode the authorship spans.

On AI-text classification, it achieves an AUROC of 0.9916, a false positive rate of 0.0041% (roughly 1 in 24,000), and a false negative rate of 0.3396%.

Read more about this research using this link.

Join the paid tier today to get access to all articles in this newsletter and level up as an AI engineer.

Become a paid subscriber today

3. Frontis-MA1

This research introduces OpenMLE, an open full-stack system for Recursive self-improvement (RSI) research covering:

• Verifiable task environments with execution feedback (OpenMLE-Gym)

• Operator learning (OpenMLE-ERL)

• Long-horizon search (OpenMLE-Evo)

Using it, the authors train Frontis-MA1-35B, a meta-evolution agent for ML engineering, that learns four operations (Draft, Improve, Debug, and Crossover), then applies them over long experiment chains.

On MLE-Bench Lite under a 12-hour per-task budget on one RTX 4090 with 12 GB VRAM, Frontis-MA1-35B improves medal average from 39.39% to 60.61% over its base model with OpenMLE-Evo, and reaches 71.21% with OpenMLE-Evo-Max, exceeding GPT-5.5 + Codex and approaching GPT-5.6 Sol and Kimi K3.

Read more about this research using this link.

4. Metis: Memory Foundation Model

This research introduces Metis, a memory foundation model, an LLM with persistent memory built into its architecture rather than implemented through external modules.

Metis equips a foundation model with a native memory state, allowing past information to be compressed into an internal memory state and accessed through memory attention. At inference time, all learned model weights remain frozen, while the native memory states are autonomously updated with forward computation.

Metis is trained on large-scale memory-specific datasets, and multiple mid-training optimization objectives are used to acquire these native memory procedures.

Experiments show that Metis-27B substantially outperforms other no-context memory methods on LoCoMo and NextMem benchmarks, but still struggles with very long histories.

Read more about this research using this link.

5. Memory for Large Language Models

This survey systematically organizes the fragmented field of memory in LLMs. It classifies memory across three axes:

1. Representation (implicit versus explicit)

2. Update dynamics (offline versus online)

3. Persistence (short-term versus long-term)

It distinguishes model-level memory from external agent memory or prompt-based RAG pipelines, highlights key trade-offs, and calls for improved memory-management methods and unified benchmarks.

Read more about this research using this link.

Subscribe now

6. PhiZero: A World Model Built Around Physical Language

This research presents PhiZero, a physical world model built around “physical language” or compact discrete representation of world-state transitions.

The model is trained to learn the physical language from in-the-wild videos with self-supervision and uses it to explicitly reason about how the physical world evolves.

Instead of taking the usual approach of predicting future videos directly in pixel space, PhiZero uses a reason-then-render approach. This means that it first infers future world evolution as a physical-language sequence and then renders the inferred transitions into videos.

Experiments show that it has leading results on several physical-video benchmarks and shows potential for realistic and interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.

Read more about this research using this link.

7. Weak-to-Strong On-Policy Distillation

On-policy distillation (OPD) has become a popular post-training method for LLMs that aligns a student with the teacher’s token-level distribution on the student’s own rollouts.

This method either distills a larger model into a smaller one or trains multiple domain experts from a shared base and distills them into one student (MOPD). These approaches fail at the frontier when no larger teacher exists or turn out to be costly for multiple teachers.

This research works on these issues and introduces Weak-to-Strong On-Policy Distillation (W2S-OPD), an algorithm that improves the strong student by distilling from multiple weak models.

W2S-OPD creates a proxy teacher by comparing a positive and a negative model, both smaller than the student. Their logit difference captures a specific capability direction, which is added to the student’s base-model logits. Because the resulting proxy remains close to the student’s distribution, the student can learn from it by minimizing per-token reverse KL divergence on its own generated outputs.

W2S-OPD consistently outperforms OPD on multiple math and coding benchmarks and even makes the student surpass the domain teacher and continue to improve when every supervision source is weaker.

Read more about this research using this link.

8. Gemini Robotics ER 2

Google released Gemini Robotics ER 2, its most capable embodied reasoning model for robotics. (ER stands for Embodied Reasoning.)

The model can handle conversation, understand the physical world, plan multi-step tasks, reason and call tools, and can hand off motor execution to any given lower-level VLA model. It can also track task progress, detect failures, adjust actions in real time, and allow multi-robot collaboration that lets multiple robots work together in shared spaces and complete complex workflows too tough for a single robot.

The model has 91.3% accuracy in critical moment-finding tasks in videos, comes with sub-second latency, consistently achieves the highest accuracy across all core ER capabilities, and is Google’s safest model that stops a robot when a person is nearby and resumes only when the area is clear.

Read more about this release using this link.

9. PatientAgentBench

This research introduces PatientAgentBench, a patient-facing agentic healthcare benchmark. It evaluates an agentic LLM using realistic, multi-turn conversations involving simulated patients, medical records, and healthcare tools.

Each conversation is scored by an LLM-as-a-Jury across 6 dimensions using 100+ conversation-agnostic, clinician-grounded criteria, with its automated judging showing 79-93% agreement with licensed clinicians.

Experiments over 10 models across four families on 1200 scenarios show major gaps in triage quality and safety.

• Pass rates for triage quality range from 32% for the weakest models to 88% for the strongest, with agents frequently acting on administrative requests without clinical screening.

• For clinical safety and workflow accuracy, the weakest models frequently fail by fabricating unexecuted actions, while frontier models fail on only 1-3% of cases, from unverified tool outputs and omitted crisis resources in an emergency.

• More capable models can narrow these gaps but do not close them, and the strongest model, Claude Opus 4.8, scores only 4.25 of 5 overall.

The benchmark also confirms that static benchmarks are insufficient to evaluate healthcare agentic systems, and sustained, tool-using conversations against realistic patient records are the only way to surface their failures.

Read more about this research using this link.

Subscribe now

10. HiSkill

This research introduces HiSkill, which organizes past interaction experiences into a hierarchical skill graph, rather than a flat list of textual skills.

The hierarchical skill graph is a directed graph with skill nodes, AtomicOp (Atomic Operation) nodes, and typed edges that connect reusable high-level skills with executable action templates, while also capturing decomposition, temporal transition, compatibility, support, and recovery relations among them.

At inference time, for each task, HiSkill retrieves a small task-relevant subgraph, tracks its state, selects or switches skills, and iteratively converts AtomicOps into executable actions.

Experiments on three interactive environments show that HiSkill outperforms state-of-the-art baselines by achieving average relative improvements of 17.33% in success rate, while reducing inference token consumption by 78.75% over the strongest baseline.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

LLM inference optimization is one of the hottest topics today, and everyone wants text generation that's faster, more accurate, and cheaper. Here are 10 techniques that can help you get there.

1. KV Caching

During a step of autoregressive text generation, an LLM compares the last token’s Query vector to the Key vectors of all previous tokens, including itself. This creates attention weights, which are used to compute a weighted sum over the corresponding Value vectors. This results in the attention output for that token.

For example, considering a word as a token, when generating the token after "The capital of the U.K. is", the LLM:

• Converts the current token ("is") into a Query vector

• Scores it against the Key of each preceding token to find that "capital" and "U.K." are most related to it

• Combines those tokens' Value vectors to decide that the next token is most likely to be "London"

This process is repeated at each step of text generation.

Calculating Key and Value vectors for all previous tokens at each step is expensive, especially as the sequence length grows. Using a KV cache helps solve this issue of repeated computations.

The KV cache stores each token’s Key and Value vectors once, rather than recomputing them for the entire sequence at each step. This means the LLM now has to calculate (and append to the KV cache) only the new token’s Key and Value, reusing the rest from the cache at each step.

This massively reduces the amount of computation during inference but comes at the cost of using up HBM memory. (There’s no free lunch!)

Using KV caching is one of the simplest techniques that you can take to improve inference.

Before we move forward, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

2. Efficient Attention mechanisms

As we discussed earlier, the KV cache takes space in a GPU’s HBM, and managing its size helps make inference fast.

The traditional Multi-head self-attention (MHA) uses multiple heads, each with its own Key and Value vectors.

There are a few attention variants that reduce the number of Key and Value vectors used (and therefore the KV cache size) as follows:

• In Multi-Query Attention (MQA), all attention heads share the same Key (K) and Value (V) vectors, while each head still has its own Query (Q) vector.

• In Grouped Query Attention (GQA), Query (Q) heads are grouped, and each group shares one Key (K) and Value (V).

There is also Multi-head Latent Attention (MLA), which compresses the Key and Value vectors into a low-dimensional latent vector and caches it. This uses much less memory than the full-sized KV cache. The full Key and Value vectors are reconstructed from this latent vector when needed.

3. Continuous Batching

A naive way to process LLM requests is to handle them sequentially. This is very inefficient, as different requests can arrive at different times and can have variable input and output lengths (and therefore processing times).

Requests are therefore batched and processed together. This improves the system's overall throughput and reduces underutilized GPU compute.

Batching can be done at the request and token levels. When done at the request level, the system has to wait for a certain predetermined number of requests to arrive and be batched before they can be processed. This is called Static Batching.

A system can also be configured to wait only until a pre-decided maximum time rather than indefinitely to fill the full batch before the batch can start processing. This is called Dynamic Batching.

In either case, the entire batch must finish processing before any new request can join.

In contrast, when done at the token level, as in Continuous (in-flight) batching, requests are processed token by token. Instead of waiting for the entire batch to finish, the system evicts a completed request and immediately fills the vacated slot with a new request. This helps utilize GPU resources more effectively than the previous two approaches.

Subscribe now

4. Optimized Kernels

Inference Decode is memory-bound, which means that the GPU spends more time moving data from memory (HBM) to the compute cores than performing computations.

A kernel is a function that runs on a GPU and performs multiple parallel operations. A kernel can be optimized to keep data in on-chip memory (SRAM/registers), avoid re-reading it from the slow HBM, and feed it to the compute cores to better use them.

This can be done using Kernel fusion, a technique in which multiple individual operations in an algorithm are merged (“fused”) into a single kernel to avoid data movement across memory.

FlashAttention is a well-known example of this, in which operations from the attention computation are merged into a single kernel.

This is how standard attention is computed:

• Query and Key matrices are read from HBM and multiplied to produce an attention score matrix in the compute cores

• This matrix is written back to the HBM

• The matrix is re-read to apply Softmax

• The result is written back to the HBM

• The result is read back to multiply it by the Values matrix to create the final output of attention computation, finally writing it back to the HBM

FlashAttention avoids all this data movement by never creating the full attention matrix in HBM. It splits the Query, Key, and Value matrices into smaller blocks called Tiles that fit in on-chip SRAM and performs attention calculations one tile at a time.

The attention scores and softmax are computed on-chip, and only the final output is written back to the HBM. This reduces data movement between HBM and the compute cores, resulting in faster attention calculations using far less memory.

5. Quantization

Quantization is the process of storing and computing LLM parameters, activations, and KV cache in lower-precision formats such as INT8 (8 bits) and INT4 (4 bits) instead of full-precision 32-bit floating-point (FP32) or 16-bit floating-point (FP16 or BF16) formats.

This reduces the model's memory requirements and increases inference throughput by reducing data transferred from HBM to the compute cores, at the cost of a slight drop in accuracy.

Take the example of Llama 3 70B. In full precision (FP32), each parameter occupies 4 bytes of memory. The total memory requirement for this model is 280 GB (70B parameters X 4 bytes/parameter). This means it will not fit on a single H100 GPU with only 80 GB of memory.

Reducing the precision to INT4, each parameter occupies 0.5 bytes of memory, and the total memory requirement for this model reduces to 35 GB (70B parameters × 0.5 bytes per parameter), making it possible to fit it on a single GPU.

A commonly used convention for quantized models is W(x)A(y) which tells how much a model’s weights (W) and activations (A) have been quantized. For example:

• W4A16 with 4-bit weights and 16-bit activations

• W8A8 with 8-bit weights and 8-bit activations

• W4A4 with 4-bit weights and 4-bit activations

You will also find quantized models suffixed with “-GPTQ” and “-AWQ”. These are the commonly used techniques for model quantization:

• GPTQ (Generative Pre-trained Transformer Quantization)

• AWQ (Activation-Aware Weight Quantization)

6. Prefix / Prompt caching & RadixAttention

LLM inference has two phases:

• Prefill, where all tokens in the input prompt are processed together, making it compute-bound. Prefill determines the Time-to-first-token (TTFT), and it is when the KV cache is built.

• Decode, where subsequent tokens are generated one at a time, making it memory-bound since very little computation is done at each step. Decode determines the Time per Output Token (TPOT).

Prefix (or prompt) caching is an optimization technique that reuses the computed KV cache for a shared portion/ prefix of the input prompt across multiple user requests. This reduces redundant computation during Prefill and leads to a shorter Time-to-first-token (TTFT).

For example, consider a long system prompt “System: You are a helpful assistant…”.

• Without prefix caching, each user request computes Prefill for all tokens in the system prompt and the user's question.

• With prefix caching, the KV cache for the system prompt is computed once and stored. Every user request then reuses it and computes Prefill only for the user's actual question.

RadixAttention is a technique introduced with the SGLang serving framework that efficiently implements Prefix caching using a data structure called a radix tree.

In a radix tree, each path from the root is a token sequence, and shared prefixes (prompt portions common to user requests) are shared branches.

For a new user request, the algorithm walks the radix tree, matching tokens as far as possible and reusing the KV cache along the matched path. Prefill is computed only for the unique tokens in the user request.

7. Pruning

Pruning is the technique of removing parts of a model that do not contribute much to its accuracy, making it smaller and therefore cheaper to perform inference on.

Pruning can be done in three ways:

1. Unstructured pruning: Involves setting low-magnitude individual parameters to zero

2. Structured pruning: Involves removing complete sections (neurons, attention heads, fully-connected layers) of the model

3. Semi-structured (N: M) pruning/ sparsity: Involves setting N of every M consecutive weights to zero. Newer NVIDIA GPUs (Ampere onward) have hardware support for 2:4 sparsity, which leads to ~2x increase in matrix multiplication throughput.

Subscribe now

8. PagedAttention

PagedAttention is a technique introduced in the vLLM serving framework to efficiently manage the KV cache's memory requirements. The technique is inspired by how operating systems manage memory using Paging.

Instead of storing the KV cache in a single large contiguous chunk of GPU memory upfront, it is dynamically stored in multiple fixed-size, small, non-contiguous blocks called pages.

A per-sequence block table maps logical token positions to physical memory blocks, and the attention kernel uses it to locate the correct KV pairs when needed.

9. Speculative Decoding

Speculative Decoding is a technique used to accelerate inference with large LLMs.

To do so, it uses a smaller LLM from the same family as the larger one, with both sharing the tokenizer and vocabulary. The smaller LLM in this case is called the ‘Draft model’, and the larger LLM is called the ‘Target model‘. This smaller LLM naturally runs inference faster because it has fewer parameters than the larger one.

This is how it works:

• For a given prompt, the smaller/ Draft model first generates K tokens (a draft). This is the Speculation/ Drafting step.

• These K tokens, along with the original prompt, are given to the larger/ Target model.
  In a single forward pass, the larger model outputs its probability distribution for the next token at every position in the draft, plus one for an additional position beyond it. So the total number of target predictions is K + 1 = 6. Because Decode is memory-bound, this single forward pass takes almost as long as generating just one token from the model.

• Next comes the Verification step. Each drafted token is checked against the larger/ target model’s prediction at that position. If they agree (when using greedy decoding) or the target finds it likely enough (when using probabilistic decoding via rejection sampling), the token is accepted.

  
  At the first token where this check fails, that token and all subsequent drafted tokens are discarded, and the larger model provides a replacement token for that position. All subsequent tokens from the target LLM are also discarded because they were conditioned on the rejected drafted token.
  
  Drafting then resumes from this position.

The benefits from this are quite evident.

• Consider a case where all K draft tokens are accepted, we get K+1 tokens in the final output. This is the best-case scenario.

• More commonly, if the first i draft tokens are accepted, we get i+1 tokens, i.e. i draft tokens + 1 resampled replacement token from the target model.

• In the worst case, if the first draft token is rejected, we get just 1 token, which is the resampled replacement token from the target model. This is the same as normal decoding from the target model.

If you want to read more about Speculative Decoding, please check out the following lesson.

10. Prefill-Decode Disaggregation

We have previously discussed that:

• Prefill is compute-bound and determines the Time to first token (TTFT)

• Decode is memory-bound and determines the Time Per Output Token (TPOT), also called Inter-token latency (ITL)

These are two fundamentally different problems, and optimizing each on a single GPU can be tough, especially when the prompts are long relative to the outputs (prefill-heavy requests), and one wants both low TTFT and TPOT at the same time.

Prefill-decode disaggregation is a serving architecture that helps in this case by moving the two stages of inference to separate dedicated pools of GPUs.

In this case, the dedicated pool of GPUs for Prefill would be optimized for higher compute, while the dedicated pool of GPUs for Decode would be optimized for higher memory bandwidth and capacity.

For example, the H200 GPU, compared to the H100, has much faster and larger memory but the same core compute. Therefore, H200s are best used in the Decode pool, and H100s in the Prefill pool.

One of the major overheads in this serving technique is transferring the KV cache between the Prefill-dedicated GPU pool and the Decode-dedicated GPU pool, which can be optimized with faster interconnects such as NVLink or InfiniBand.

Thanks for reading this article. If you found it valuable, show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Arithmetic Intensity, Simply Explained

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Become a paid subscriber today

1. Claude Opus 5

Anthropic released Claude Opus 5, a model that comes close to the frontier intelligence of Claude Fable 5 at half the price.

Opus 5 is the new SOTA for coding and knowledge work evals and is designed to be used efficiently for everyday tasks, but it remains behind Mythos 5 on cybersecurity tasks. It also improves on Opus 4.8 across multiple benchmarks evaluating automation, computer use, scientific research, and visual output.

Opus 5 is described as Anthropic’s most aligned model yet and costs $5 per million input tokens and $25 per million output tokens (the same as Opus 4.8).

Read more about this release using this link.

2. Hilbert Operator for Progressive Encoding (HOPE)

This research paper introduces Hilbert Operator for Progressive Encoding (HOPE), a data-free and hyperparameter-free mathematical framework that helps deconstruct the internal knowledge stored inside deep neural networks.

While network compression is usually used to make models smaller, it can also help analyze what a model has learned. This is because removing a neuron tests whether it contributed to important knowledge, and merging neurons tells whether they perform functionally similar roles.

HOPE, or Hilbert Operator for Progressive Encoding, models each neuron as a rank-1 Hilbert-Schmidt operator. This allows evaluation of neurons in a Hilbert space of continuous functions and simplifies a neural network based on what each component actually does (its function) rather than relying only on the size of its weights.

With HOPE, different compression actions (pruning neurons, merging neurons, and removing residual blocks) can be treated as forms of the same low-rank projection problem and compared using a single scale-invariant metric.

Read more about this research using this link.

Before we move forward, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

3. LLMs Get Lost in Evolving User Intent

This research paper evaluates how well LLMs track and act on user intent as it evolves over the course of a conversation.

This is done using a framework that changes static, single-turn tasks into dynamic multi-turn conversations in which the user’s intent evolves across turns while preserving each task’s original evaluation method, allowing existing benchmarks to be reused as controlled test environments without requiring new annotations.

The results show that LLMs do not faithfully track and act on the user’s evolving intent, a capability that is critical for collaborative agents. For example, the performance of GPT-5.5 goes down from 99% to 80.5% on GSM8K after 6 intent changes.

Task switching is the most difficult transition for LLMs, and simple memory prompts or even restating the user’s correct current intent improve results but do not restore single-turn performance.

Read more about this research using this link.

4. Laguna S 2.1

Poolside released Laguna S 2.1, an open-weight, agentic coding model. It comes with an 118B total-parameter Mixture-of-Experts (MoE) architecture with 8B activated parameters per token and supports a context window of 1M tokens.

The model’s performance is a result of strong post-training that rewards verification, backtracking, and persistence at tasks, using more generous rollout budgets, better RL sandbox infrastructure, and multi-harness rollouts (same prompts rolled out in several agent harnesses).

Laguna S 2.1 performs competitively with much larger LLMs, scoring 70.2% on Terminal-Bench 2.1 and 78.5% on SWE-Bench Multilingual. Some impressive tasks that it achieves are:

• Building a basic browser engine from scratch

• Improving its software harness used for training/evaluation and user interactions, making it 5.2% faster with ~70% lower memory allocation

• Independently re-discovering a proof to Erdős problem #397 (the first proof to the conjecture was found earlier in January 2026 by GPT-5.2 Pro)

Read more about this release using this link.

5. Music-JEPA

This research paper introduces Music-JEPA, a world model that learns a world model of piano sound using JEPA by framing music as an action-conditioned system where the audio is treated as the state, and the pianoroll as the instrument action.

Given a current audio state and an action, the model predicts the resulting future audio state directly in a latent space rather than reconstructing the raw sound. The model is trained in a fully offline setting on roughly 200 hours of paired piano audio and MIDI, without environment interaction.

Experiments show that the learned model captures relationships between musical actions and their resulting sound, including pitch, timing, velocity, and pedal control, and outperforms JEPA models trained on passive audio alone.

The resulting representations can be used for downstream tasks such as beat tracking, composer identification, and key estimation, and enable piano transcription by planning, i.e., working backward to identify the actions most likely to have produced a given sound.

Although impressive, Music-JEPA focuses on classical piano with aligned MIDI, and it remains unclear how well the model scales or generalizes to broader musical settings where action annotations are unavailable.

Read more about this research using this link.

6. LLM-as-a-Coach

This research paper introduces Experiential Learning (EL), a method for improving LLMs on open-ended tasks (such as creative writing and summarization), where there is no single correct answer.

In standard RL, an LLM is used as a judge, and its detailed evaluation is reduced to a scalar reward. In EL, an LLM is used as a coach where it evaluates the model’s responses and produces detailed guidance, which is distilled into the model’s weights through a teacher model using On-policy context distillation.

LLM-as-a-Coach creates a dense supervision signal with 17,600 bits of information per sample, which is more than 5,000× the bandwidth of a scalar reward.

This approach consistently outperforms rubric-based RL on held-out and unseen open-ended tasks, generalizes better beyond the training distribution, and reduces reward hacking.

Read more about this research using this link.

7. Nanbeige 4.2

This research paper introduces Nanbeige 4.2-3B, an open-weight agentic model with strong performance in coding office tasks and complex tool-use tasks, and highly competitive reasoning capabilities in mathematics, coding, and science.

The model has only 3B non-embedding parameters and uses a Looped Transformer that sends information through the same layer stack twice to increase effective depth without adding parameters.

Nanbeige 4.2-3B was pretrained from scratch, then post-trained on execution-grounded agent trajectories and improves through multi-stage RL focused on response quality, reasoning efficiency, and stable agentic learning.

Evaluations show that it outperforms larger models such as Qwen3.5-9B and Gemma4-12B across diverse agentic benchmarks while remaining competitive on reasoning and alignment tasks.

Read more about this research using this link.

8. Loop the Loopies!

This research paper presents the Loopie series of Mixture-of-Experts (MoE) models:

• Loopie-20B-A2B: 20B-parameter model with 2B active parameters

• Loopie-6B-A0.6B: 6B-parameter model with 0.6B active parameters

These models use Looped transformers, where each Transformer layer is reused twice before moving to the next layer.

Looped transformers previously faced the challenge that, on an N-times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times.

This is fixed in the “Loopie Recipe,” which reuses each Transformer layer twice before moving to the next, halving the number of stored layers while preserving effective depth. The saved memory is then used to increase training throughput (batch size and model width) under the same wall-clock budget.

Comparisons with a vanilla 30B-A3B model show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget and, with a new post-training method, reaches frontier-level reasoning performance.

Read more about this research using this link.

9. Fugu-Cyber

Sakana AI released Fugu-Cyber, a cybersecurity-focused multi-agent orchestration system that behaves like a single model available as one API endpoint while dynamically assigning tasks to specialized agentic models.

This system is based on the Fugu orchestration model, which dynamically orchestrates the world's best models to solve complex, multi-step problems and reach frontier-level performance without single-vendor dependency.

Fugu-Cyber achieves SOTA performance on the industry’s most challenging security benchmarks, reaching a success rate of 86.9% on CyberGym and 72.1% on CTI-REALM.

These results are comparable to leading cybersecurity-focused frontier models such as GPT-5.5-Cyber and Mythos-Preview.

Read more about this release using this link.

10. SANA-Video 2.0

This research paper introduces SANA-Video 2.0, a hybrid video-generation model available at 5B and 14B scales that can generate high-quality video up to 720p on a single GPU.

The model uses hybrid Linear–Softmax Attention, which combines gated linear attention with periodic gated-softmax attention layers in a 3:1 ratio.

The softmax layers restore detailed token interactions that pure linear attention lacks, and these are propagated across depth (later layers) using Block Attention Residuals (AttnRes).

Trained from scratch, the 5B model achieves a VBench score of 84.30 while generating 480p video in 13.2 seconds on a single H100, remaining competitive with far larger softmax video DiTs at a fraction of the latency.

Its compiled DiT forward pass is 3.2× faster than a matched full-softmax baseline at 720p/60s, a gap that increases with video duration.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Hugging Face recently disclosed a security incident in its production infrastructure that is unlike anything it has handled before. This cyberattack was completely driven by an autonomous AI agent system comprising GPT‑5.6 Sol and a more capable unnamed pre-release model from OpenAI.

What went wrong?

The attack occurred when OpenAI ran an internal evaluation on a benchmark called ExploitGym. This benchmark tests how capable an LLM is at performing cyberattacks. Model versions with reduced cyber refusals were used in this test, and protective classifiers that prevent models from engaging in high-risk cyber activity were deliberately turned off.

The benchmark was supposed to run in a highly isolated environment. The only network access given was for installing packages using an internally hosted third-party software that acted as a proxy and cache for package registries. And this is where things all went wrong.

During evaluation, the models used a good amount of inference compute to find and exploit a zero-day vulnerability in this package registry cache proxy. They then performed a series of privilege escalation and lateral movement actions until they reached a node that had access to the Internet.

After gaining Internet access, the models worked out that Hugging Face potentially hosted models, datasets, and solutions for the ExploitGym benchmark, which they could use to cheat on the benchmark.

Using this information and over the course of the weekend, the model combined multiple attack vectors, stealing cloud and cluster credentials and exploiting zero-day vulnerabilities to identify a remote code execution path on the Hugging Face servers. This is when Hugging Face got alerted about the attack.

Subscribe now

How was the attack stopped?

The cyberattack was flagged by Hugging Face’s internal anomaly-detection AI pipeline, which uses LLM-based triage on its security telemetry. Based on suspicious signals, the team decided to analyze the full attacker action log, which was made up of more than 17,000 recorded events.

They first turned to frontier commercial model APIs to analyze the logs, but these models' safety guardrails refused to process these requests.

This is such a big power asymmetry, since the safety guardrails held back the compromised Hugging Face team rather than stopping the attackers!

The team then moved towards using the open-weight GLM 5.2 on their own infrastructure. This worked well and ensured that attacker data and credentials never left their environment during the analysis. Using this LLM-first approach, the team was able to do in hours what would usually take days, and match the attacker’s speed.

When the attack occurred, Hugging Face was still largely unaware of which model (whether a jailbroken hosted model or an unrestricted open-weight model) was responsible for it. OpenAI’s security team, however, discovered this activity internally and in due course reported it to Hugging Face while the attack was being contained.

Hugging Face said that they found no evidence that the attack involved public, user-facing models, datasets, or Spaces. Their software supply chain (container images and published packages) was also verified to be clean.

The benchmarks saw this coming

ExploitGym, the benchmark that was being run during this exploit, consists of 898 instances drawn from real-world vulnerabilities across three domains: userspace programs, Google’s V8 JavaScript engine, and the Linux kernel.

Initial tests on this benchmark showed that frontier models could successfully exploit a big chunk of these vulnerabilities. Claude Mythos Preview and OpenAI’s GPT-5.5 could generate working exploits for 157 and 120 instances on this benchmark, respectively.

These offensive capabilities even remained when the widely used defenses were enabled.

The UK AI Security Institute’s evaluation showed similar results, indicating that frontier models such as GPT‑5.5 and Mythos/Claude could autonomously execute long-time-horizon, complex multi-step cyberattacks.

Autonomous AI-driven cyberattacks are no longer a thing of the future. They are here. And given what the benchmarks and red-team evals have been signaling for months, it's no surprise this was coming.

Companies and organizations that still haven't prepared need to start now. This means treating the data and model surface as primary targets for attackers and putting AI to work in defense to guard against such attacks.

Incidents like this will happen faster and hit harder from here. This is just the beginning.

Further Reading

• UK AI Security Institute’s report titled ‘Our evaluation of OpenAI’s GPT-5.5 cyber capabilities’

• ArXiv paper titled ‘ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?’

• Hugging Face’s security incident disclosure

• OpenAI’s security incident disclosure

1. Kimi K3

Moonshot AI released Kimi K3, the world's first open 3T-parameter class model, designed for frontier intelligence across long-horizon coding, knowledge work with native vision, and reasoning.

The model has 2.8T parameters and comes with a 1M-token context window. It uses Kimi Delta Attention, Attention Residuals, and a highly sparse MoE architecture (Stable LatentMoE) that activates 16 of 896 experts, resulting in 2.5× better scaling efficiency than Kimi K2.

While its overall performance still remains behind the most powerful proprietary models, Claude Fable 5 and GPT 5.6 Sol, Kimi K3 shows frontier-level performance across diverse benchmarks, consistently outperforming other tested models.

Some limitations of the model are that it performs less reliably when earlier reasoning steps or important context are missing. It also tends to act too proactively when user instructions are unclear, making unexpected decisions on the user's behalf.

Read the technical blog using this link.

Before we move forward, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

2.xHC: Expanded Hyper-Connections

Hyper-Connections (HC) expand the residual stream of Transformers into N multiple parallel streams, which enables memory scaling beyond model width and depth, while Manifold-Constrained HC (mHC) stabilizes this approach at scale.

However, scaling mHC beyond N=4 leads to diminishing performance gains and rapidly increasing training cost because very little information is written back across the growing number of streams, while residual-mixing generation scales cubically with N.

This research paper addresses this by introducing xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond N=4. xHC does this by maintaining 16 readable streams, providing dense access to the full residual state, while updating only 4 streams at each layer.

xHC improves performance across knowledge, reasoning, coding, and Chinese-language benchmarks across MoE models. To reach the same training loss, standard Transformers and mHC require 1.50× and 1.19× as much compute as xHC, respectively.

Read more about this research paper using this link.

Subscribe now

3. VideoChat3

VideoChat3 is a fully open (model weights, training code, training strategy, and complete training datasets), 4B-parameter multimodal model with generalist video-centric capabilities.

VideoChat3 uses the Inflated 3D Vision Transformer (I3D-ViT) and adaptive frame resolution to efficiently compress video while preserving spatiotemporal information, and reduces the cost of processing video inputs during training and inference.

Trained on roughly 3 million instruction samples, VideoChat3 achieves a rare balance of broad generalization and computational efficiency, outperforming prior open-source models with equal or larger parameter counts while using only 4B parameters.

Read more about this research using this link.

4. Inkling

Thinking Machines Lab released Inkling, an open-weight model for reasoning, coding, tool use, and multimodal tasks. It uses the Mixture-of-Experts architecture with 975B total parameters and 41B active parameters, and supports a context window of up to 1M tokens.

Alongside this, the team released a preview of Inkling-Small, a lighter-weight model with 12B active parameters, trained with a similar recipe, that achieves strong performance with even lower cost and latency.

Inkling comes with adjustable reasoning effort and can be fine-tuned through the Tinker platform.

While it is not the strongest overall model available today (open or closed), its diverse capabilities make it a good open-weights base for customization.

Read the technical blog using this link.

5. AIDE²: The First Evidence of Recursive Self-Improvement

AIDE² is a recursive self-improvement (RSI) system that uses two autoresearch loops:

• An inner loop that functions as a normal autoresearch agent, optimizing code against an eval.

• An outer loop that optimizes the inner-loop agent’s harness code.

Over 100 autonomous outer-loop iterations running for eight days, AIDE²:

• Discovered 7 successive improved versions of AIDE

• Developed a new search algorithm

• Reduced the prompt size by 16×

• Outperformed a manually tuned agent that had been iterated on for two years, on several unseen tasks under a fixed compute budget

• Reduced reward hacking on a held-out GPU benchmark from 63% to 34%

The team describes this as “Level 1” recursive self-improvement because the system improved itself more efficiently than human researchers. However, it did not reach “Level 2” or “Ignition,” because the improved agent was not conclusively better at running the outer self-improvement loop, and its evolved code became complex and difficult to maintain.

Read the technical blog using this link.

Subscribe now

6. WanSong v1.0

WanSong is Alibaba’s pure diffusion-based model for generating commercial-grade, multilingual songs up to 5 minutes long. Trained on more than six million hours of audio, the model produces complete audio in a single run and separately outputs vocals and background music.

On a multilingual and multi-genre benchmark, WanSong achieved the lowest reported pronunciation error and the highest musicality score compared with LeVo, Mureka V7.6, and Suno V5.

Read more about this research using this link.

Subscribe now

7. Bonsai 27B

Bonsai 27B is PrismML's new multimodal model based on Qwen3.6 27B, capable of multi-step reasoning, structured tool calls, vision tasks, and agentic loops for computer use.

Models of this size typically require 54GB of memory at 16-bit precision (27B × 2 bytes per parameter). This far exceeds the memory of any typical phone.

Bonsai 27B changes this with its two variants:

1. Ternary Bonsai 27B: Uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, leading to 1.71 effective bits per weight. This totals 5.9 GB of memory, which makes it suitable for running on an everyday laptop.

2. 1-bit Bonsai 27B uses binary {−1, +1} weights with FP16 group-wise scaling, leading to 1.125 effective bits per weight. This totals 3.9 GB, which means it fits and runs comfortably on an iPhone 17 Pro Max.

Ternary Bonsai 27B achieves 95% of the full-precision baseline, and 1-bit Bonsai 27B achieves 90% across 15 benchmarks in knowledge, reasoning, math, coding, instruction following, tool calling, and vision.

Read the technical blog using this link.

8. Mach-Mind-4-Flash

This research presents Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts (MoE) agentic model (3B activated) that, through post-training optimization alone, achieves performance on par with or surpassing that of 100B-parameter-class models.

Its three-stage training pipeline combines:

1. A unified RL and on-policy distillation infrastructure, with dynamic multi-teacher scheduling and operator-level acceleration, which gives it a 17% end-to-end training speedup

2. Multiple domain-specific RL experts trained in parallel across Reasoning, General, and Agent tracks, then combined into a single generalist using Multi-Teacher On-Policy Distillation (MOPD)

3. Hybrid Median-length Policy Optimization (HMPO), which compresses reasoning chains by 19–46% with not more than a 0.7-point loss in accuracy.

Mach-Mind-4-Flash scores competitively matching or outperforming models with 10-30× its activated size while using substantially less inference compute.

Read more about the research using this link.

9. Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values

This research identifies a type of misalignment in LLMs called Covert value leakage. This is when the information provided by an LLM is influenced by its own values, and this influence is never disclosed to the user.

For example, when a user asks whether they are considering investing in an AI company and wants to know how likely the AI bubble is to pop, Claude Opus 4.8 gives a lower probability when the company under consideration is Anthropic rather than OpenAI, without disclosing this influence to the user.

The research shows that models are influenced by different types of values, including:

• Preferences for morally good outcomes

• Preferences for the company that developed them

• Preferences for some human leisure activities over others

Models differed in how openly they revealed these biases. For example, Claude models falsely claim to provide unbiased answers during chain-of-thought reasoning, whereas Qwen models explain how their values bias their answers.

Read more about this research using this link.

10. On-Policy Delta Distillation

On-policy distillation (OPD) is an RL post-training method that trains a student model based on its own responses using token-level feedback from a stronger teacher model. This encourages the student model to generate tokens that the teacher prefers.

This research paper improves OPD by introducing a new distillation reward, called the ‘delta signal’, instead of the teacher’s output distribution.

The delta signal is the difference between the teacher model and its base model prior to post-training for reasoning. This helps capture and transfer the reasoning skills gained during the teacher model’s post-training rather than copying all of its knowledge and style. This new approach is termed On-Policy Delta Distillation (OPD²).

Experiments across mathematics, science, and code-reasoning benchmarks show that OPD² consistently outperforms conventional on-policy distillation. However, it requires access to both the teacher and its base model and increases training time compared with standard OPD.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Building Your First AI Agent

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Become a paid subscriber today

AI/ML research is hard to keep up with. I have found three main reasons for this:

1. Many engineers are not used to reading math-heavy research papers

2. The language of research papers is dense and requires strong foundations to understand

3. There is just too much research being published every day, and it’s tough to read it all

Let’s address each of these issues one by one to make AI/ML research easy to follow.

Before we start, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

Math-heavy papers & what to do about them?

Check out a paragraph from the research paper titled ‘Improving Language Understanding by Generative Pre-Training’, which introduces the GPT architecture.

The paragraph describes the pretraining objective that is maximized during LLM training.

It looks intimidating, but it actually isn't. It is written this way because math is a great way to condense ideas and concepts and avoid verbosity.

To make it easy to understand, simply pass it to an LLM with a simple prompt as follows.

And this is what you get as a response.

You could also prompt an LLM further to explain a concept in more detail or, with step-by-step examples, if you’re still struggling to understand something.

All thanks to LLMs, this is one of the best (and judgment-free) ways to understand math-heavy research papers.

Subscribe now

“Should I read a math book before reading a research paper?”

There are two ways to approach the mathematics in AI/ML research:

1. Bottom-up: Read all the foundational math first (Linear algebra, calculus, control theory, mathematical optimization, and so on), and then start reading AI/ML research.

2. Top-down: Read an AI/ML research paper first and then learn the relevant math.

I’ve followed both approaches in the past and found that the ‘Bottom-up’ one simply took too much time and effort, making it impractical. (I was stuck trying to learn all the different ways to integrate equations and multiply matrices, which later proved to be of not much use.)

I have long since dropped this approach and now read research papers first, then go ‘Top-down’ to read only the math essential to understanding the paper. This is what I find most efficient and recommend to you, unless you’re absolutely new to math (which is usually not the case for engineers).

Dense language & what to do about it?

Research papers are deliberately written concisely, avoiding verbosity. Otherwise, a single one would have pages enough to fill an entire textbook.

Here’s an excerpt from ‘Attention is all you need’, the famous research paper that introduces the Transformer architecture.

This short text introduces you to roughly a dozen concepts, including:

• Recurrent neural networks

• LSTM

• GRU

• Sequence modeling/ Language modeling

• Machine translation

• Encoder-Decoder architectures

• Sequential computation precluding parallelization

• Memory constraints limiting batching across examples

• Factorization tricks

• Conditional computation, and so on.

To a person just getting introduced to AI/ML research, this is a nightmare.

Here are three ways to read and understand such dense language:

1. Scan and read in passes

   Give yourself time and read the research paper in multiple passes rather than in one go. First, scan the research paper to get its basic layout and reassure yourself that the paper does not go on and on forever. (Sounds silly, but it works.) Then, patiently read each section one at a time.

   If you’re completely new to AI/ML research, there’s a high chance that none of it will make any sense in the first go. This is completely normal.

2. Let an LLM expand the dense sections

   Pass each dense paragraph one at a time to an LLM and ask it to “Explain this paragraph line by line in simple words.” Take your time to understand the concepts, and keep returning to the earlier sections until everything fits together in a coherent sequence.

3. Don’t let the jargon overwhelm you

   Look up unfamiliar terms with a simple web search and follow references.

If you follow these steps patiently, I promise everything will eventually start making sense.

Too long, too many, no time

Now, if you’re completely comfortable with reading AI/ML research, there’s a chance that you might feel overwhelmed by the number of research papers being published. Each paper is 15-20+ pages long, and hundreds to thousands land on arXiv every day.

Finding the relevant ones, let alone reading everything published each day, seems impossible.

I solve this by simply curating my social feeds (I mostly use LinkedIn and Substack) towards AI/ML research, releases, and developments.

Here are some accounts that you could follow on LinkedIn that talk about AI and ML:

• Yann LeCun

• Sebastian Raschka

• Cameron R. Wolfe

• Ignacio de Gregorio Noblejas

• Deedy Das

• Karun Thankachan

• Dr. Ashish Bamania

Here are some of my recommendations from Substack in the AI/ ML engineering, trends, and macroeconomics space:

I also have an AI agent that brings me some interesting research papers every week, and you could build one for yourself following this tutorial.

Alongside this passive approach of simply letting my feed decide what I am updated on, I look out for recently published research papers on:

• Hugging Face Trending papers

• AlphaXiv

I write about what I found important each week in a newsletter edition titled “This Week in AI Research” and send it out every week. This could be a source you can use to stay up to date on AI/ML research.

How to quickly filter through AI/ML research?

Now that you have a curated feed and other trusted sources of AI/ML research, there’s still the problem of going through them all.

I use a simple system to find important research papers that you can steal.

• Quickly read through the abstract, check the figures, and read the conclusion.

• If a paper seems promising or directly relevant to your work or curiousity, then go deep into the methodology and evaluation sections. This is where the most important details of a paper lie: whether baselines are fair, whether ablations are present, and whether benchmarks were cherry-picked.

How to actually learn from a research paper?

Simply reading a research paper end-to-end is less useful than applying what it teaches. Some of the ways to make the learnings stick are:

• Implement them in code: Manually code up the core idea from a research paper. Use an LLM to review your code, but don’t rely on vibe-coding. I frequently use this approach to learn better. (Example)

• Use them at work or in your personal projects: Always ask how you can apply your learnings to the product or pipeline you’re working on. This will give you compounding returns in your career.

• Start writing: Writing personal notes or an explainer/tutorial for others is another great way to learn better. Writing forces you to really get good at something before you can simplify it and explain it to others.

“Which papers to read if I am starting out?”

Pick 2-3 areas that deeply interest you or are relevant to your current work or career ambitions. Prioritize the early work and foundational papers in these areas and read them first.

A great way to build a strong foundation is to keep following and reading the research papers in the references. This will help you master a field’s foundational concepts faster than you might expect.

For example, before reading about Heavily Compressed Attention (HCA), I would follow references and read about:

• DeepSeek Sparse Attention

• Multi-head Latent Attention

• Self-Attention

• Bahdanau attention

How to build a research reading habit?

• Curiosity and necessity are two great forces that might motivate you to read AI/ML research. For me, curiosity has been a big one.

• Curate your feed and connect with others who love reading and keeping up with AI/ML research.

• If you’re a beginner, set aside 1 to 2 30- to 60-minute sessions dedicated to reading AI/ML research every week. Put them in your calendar as protected time and do not skip them.

Reading for just an hour a week beats a heroic weekend binge every few months, and this is how you build a habit and win with slow compounding without burning out.

No one keeps up with all AI/ML research

My final advice to you before we conclude is to drop the unrealistic goal of keeping up with everything happening in AI/ML. No one can do this, including full-time researchers. Let go of this goal, be easy on yourself, and take your time.

👋🏻 Feel free to leave a comment or DM me if you have any questions, and share this with others if you found it valuable!

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Building Your First AI Agent

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Become a paid subscriber

1. Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Existing RL training approaches update all model parameters uniformly, assuming that every layer contributes equally to the gains obtained.

This research paper challenges this idea and finds that training a single transformer layer can recover most of the gains achieved by full-parameter RL training and, in some cases, even surpass them.

Studying seven models across two Qwen families (Qwen2.5 and Qwen3), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains, it is found that RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layer.

These high-contribution layers are usually in the middle of the transformer stack, while layers near the input and output ends contribute substantially less.

Based on these findings, the researchers develop simple layer-aware training strategies that consistently outperform standard full-parameter RL training.

Read more about the research paper using this link.

Before we move forward, I want to introduce you to the Visual Tech Bundle.

It is a collection of visual guides that explain core AI, LLM, Systems design, and Computer science concepts via image-first lessons.

Others are already loving these books. Why not give them a try?

Link to the Visual Tech Bundle

2. LLM-as-a-Verifier

This research paper introduces LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training.

Unlike standard LM judges, which are LLMs prompted to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to produce continuous scores.

This probabilistic approach reduces tie rates when comparing complex solutions and enables verification across multiple dimensions:

• Granularity of score tokens

• Number of repeated evaluations

• Decomposition of evaluation criteria

To make verification scaling practical, the researchers further introduce a cost-efficient ranking algorithm to select the best solution among candidates, using preference probabilities derived from the verifier’s continuous scores.

LLM-as-a-Verifier achieves SOTA performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%).

Beyond verification, it can also act as a proxy for estimating task progress and as a dense reward signal for RL, making training more sample-efficient.

Read more about the research paper using this link.

3. Mixture-of-Parallelisms

This research paper proposes a memory-efficient training stack called Mixture-of-Parallelisms (MoP) for Mixture-of-Experts (MoE) models.

It uses multiple parallelism techniques across the training pipeline to maximize efficiency under the cluster constraints, including a new strategy for the optimizer step.

MoP achieves high throughput and memory efficiency, enabling lossless pre-training/fine-tuning of trillion-parameter-scale models at a million-context length using just under 12 8x H200 GPU nodes.

It delivers 4.7-8.2× higher per-GPU throughput than a strongly-tuned FSDP2 baseline and sustains training at context lengths up to 1M tokens, whereas the baseline runs out of memory beyond 64–128K context length.

Read more about this research paper using this link.

4. Measuring the Gap Between Human and LLM Research Ideas

This research paper examines how LLM-generated research ideas differ from those of human researchers.

Studying thousands of published papers and prompting models from families including GPT, Claude, Gemini, DeepSeek, and Qwen to generate ideas from the same prior-work context, the authors find that:

• LLM ideas are disproportionately concentrated around connecting existing fields or synthesizing existing methods

• Human ideas cover a wider range of motivations and contributions, which include identifying failures, explaining contradictions, relaxing assumptions, and developing new systems

This suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.

Read more about this research using this link.

5. Infinite Worlds with Versatile Interactions

This research paper presents LingBot-World 2.0 (LingBot-World-Infinity), a model that generates responsive, open-ended, infinite virtual environments for exploration.

Compared to the earlier version, this model maintains consistent output quality, supports up to 720p at 60 fps, and generates highly diverse interactive elements that enable broader actions (e.g., attacking, archery, spell-casting, and shooting) and a variety of text-driven events.

Additionally, the model is integrated with an agentic harness with two autonomous agents:

• a pilot agent that plans and runs character behavior, and

• a director agent that creates new environmental elements as the scene progresses.

The model is available in 14B and 1.3B versions, with the 1.3 B version capable of running on a single GPU.

Read more about this research using this link.

6. AutoMem: Automated Learning of Memory as a Cognitive Skill

This research paper introduces AutoMem, a framework that reframes LLM memory management as a trainable skill rather than a fixed system.

An LLM agent using it gets file-system operations (read, write, search, append) as first-class actions, allowing it to decide what to remember and retrieve.

A strong LLM then improves this automatically by reviewing long episode traces to revise the agent's memory scaffold, and fine-tuning a dedicated memory model on the agent's own good decisions.

Across three procedurally generated long-horizon games (Crafter, MiniHack, and NetHack), optimizing memory alone (without modifying the model’s task-action behavior) improves the base agent’s performance by 2-4×, bringing a 32B open-weight model to a level competitive with frontier systems such as Claude Opus 4.5 and Gemini 3.1 Pro Thinking.

Read more about this research using this link.

7. GPT‑Live

OpenAI launched its new generation of voice models called GPT-Live for natural real-time human-AI interaction, with its two versions being GPT‑Live‑1 and GPT‑Live‑1 mini.

GPT‑Live is built on a full-duplex architecture, which means that meaning it can listen and speak at the same time, acknowledge you while you talk, pause when you need time, handle interruptions more naturally, and delegate harder tasks like search or reasoning to frontier models such as GPT-5.5 in the background.

Read more about this release using this link.

8. Grok 4.5

SpaceXAI released its new flagship model called Grok 4.5. This model is built for coding, agentic tasks, and knowledge work, with a big focus on software engineering and efficient reasoning.

Grok 4.5 was trained on tens of thousands of NVIDIA GB300 GPUs using large-scale reinforcement learning, with a focus on per-token intelligence.

It reaches 83.3% on Terminal Bench 2.1, resolves SWE-Bench Pro tasks with ~4.2× fewer output tokens than Opus 4.8 (max).

The model is served at about 80 tokens per second generation speed and is priced at $2 per million input tokens and $6 per million output tokens.

Read more about this release using this link.

9. SWE-1.7

SWE-1.7 is Cognition’s most capable software-engineering model to date. It is designed for long-horizon, asynchronous coding tasks and is built by RL post-training a Kimi K2.7 base model.

It scores 42.3% on FrontierCode 1.1, 81.5% on Terminal-Bench 2.1, and 77.8% on SWE-Bench Multilingual, bringing it close to frontier models like GPT-5.5 and Claude Opus, while achieving better cost efficiency.

Read more about this release using this link.

10. Muse Spark 1.1

Muse Spark 1.1 is a multimodal reasoning model from Meta Superintelligence Labs built for exceptional performance in agentic workflows, tool and computer use, coding, and multimodal understanding.

The model can orchestrate parallel subagents, has a 1-million-token context window, and outperforms Gemini 3.1 Pro (high), Opus 4.8 (max), and GPT 5.5(xhigh) on multiple difficult benchmarks.

Read more about this release using this link.

Join the paid tier today to get support my writing and get access to all posts in this newsletter.

Join 'Into AI' premium today

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Keeping up with ML research is hard these days, given the mammoth number of papers published every single day.

This Saturday (July 11, 2026), I’m going live with Karun Thankachan, a brilliant Senior Data Scientist at Walmart, to share the exact framework we use to stay on top of it all.

In one session, you’ll learn:

• How to spot quality research among the endless stream of new papers

• How to read papers efficiently and skip what doesn’t matter

• How to pull out takeaways you can use in your day-to-day work

Everyone who registers also gets a handout with all the resources discussed in the session.

Save the date: 📅 Saturday, July 11 | 🕑 12:00 PM EST

It’s free to join, and you can register using the link below. :)

Join the webinar →

Seats are limited and filling fast!

See you inside,
Ashish

I’m offering a handful of consulting slots for this quarter to help solve your problems.

This is for you if:

• You want to launch an AI product or feature at your company, but you don’t know where to start or how to make it happen

• You have a business idea but are unsure how to begin with the tech

• Your team is overwhelmed, and you want someone to review your workflows and create systems that save you time and increase your revenue

• You have a half-finished or stalled AI project that needs to get moving

• You’re pouring effort into your social media, and it’s just not growing, and you’d love to turn it into something you could eventually do full-time

• You’re looking to market your product and want it to reach the right audience

The first step is a free 30-minute call to see how I can help.

🤝 If you’re interested, just reply to this email in a few lines about what you’d like help with, and I’ll send you a link to schedule a time to connect.

P.S. This is not a sales-maxing pitch, and I’m only taking on a few clients that I can truly help, so the spots are limited. If this sounds like you, don’t wait up.

Talk soon,
Ashish

What is Arithmetic Intensity?

Arithmetic Intensity (AI) is the ratio of arithmetic operations/computations performed to the data moved to and from memory.

It indicates how much computation is performed per unit of data transferred from memory. If a workload/kernel performs very few arithmetic operations but requires massive data reading and writing from memory, it has low arithmetic intensity, and vice versa.

Note that:

• FLOP is a floating-point operation (a single arithmetic operation on floating-point numbers).

• FLOPs is the plural form of FLOP or the total number of floating-point operations.

• FLOPS is floating-point operations per second.

How is it used in the Roofline model?

In a GPU, HBM is the slowest but the largest, while registers are extremely fast but the smallest units of memory.

Read more

👋🏻 Hey friend!

Nearly every company sets aside a learning & development budget for books, courses, conferences, and technical subscriptions, and every year a large chunk of it goes unspent.

Into AI fits perfectly into this category.

Reading lessons in AI foundations, staying up to date with the latest research, and knowing how to implement it aren't side hobbies but make you and your team better at your job.

If you've been paying for Into AI yourself, or if you've been meaning to upgrade to a paid plan, there's a good chance you don't have to spend your own money at all.

You could follow either of these two approaches when you become a paid member:

1. You’ll get a payment receipt which you can forward straight to your manager for reimbursement.

2. Use the email template below, edit the details, and send it to your manager for approval.

Email template

I also offer a 30% discount on group subscriptions for your team which you can grab below.

30% group subscription discount →

Reply to this email directly if you have any questions, and I will be happy to answer them for you! 😊

This book guides you through 30 real-world agent architectures, covering the core building blocks of perception, memory, reasoning, and planning, and teaches you LangChain and LangGraph to create agents across finance, legal, healthcare, and more.

It also helps you learn how to deploy, evaluate, and guard your agents to ensure that they perform well in production. Grab your copy today using the link below.

Build production ready AI agents →

10. Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings

This research introduces Brain2Qwerty v2, an AI model that can decode natural sentences a person is typing from their magnetoencephalography (MEG) recordings in real time.

The model has an average word error rate of 39%, and for the best participant, it can accurately decode half of the sentences with one word error or less.

The model’s accuracy also improves log-linearly with more data. This means that increasing the number of recordings could close the gap with surgically implanted brain-computer interfaces, reaching accuracy levels that were once believed to be possible only with implants.

Read more about this research using this link.

9. GPT‑5.6 Sol

OpenAI released its GPT‑5.6 series with three models:

• Sol (flagship model)

• Terra (balanced model for everyday work)

• Luna (a fast and affordable model)

Among these, GPT‑5.6 Sol is the strongest model for tough agentic work such as coding, scientific analysis, biological workflows, cybersecurity, and long-horizon tool use.

For coding workflows, GPT‑5.6 Sol sets a new state of the art on Terminal‑Bench 2.1.

It also performs competitively with Mythos Preview using only a third of the output tokens on ExploitBench.

Read more about this release using this link.

8. Sakana Fugu

This research paper introduces Sakana Fugu, a family of orchestrator LLMs trained to understand user queries and dynamically create agentic scaffolds to solve them.

The two models in the Sakana Fugu family are:

• Fugu (balances performance with latency for everyday use)

• Fugu-Ultra (prioritizes answer quality on the most difficult problems)

Using adaptive scaffolds, these models achieve performance beyond that of any individual LLM agent, and reach SOTA results compared to other publicly accessible models across a wide range of challenging benchmarks (SWE-Bench Pro, Terminal Bench, LiveCodeBench, GPQA-Diamond, Humanity’s Last Exam, and CharXiv Reasoning).

Read more about this research using this link.

7. The Red Queen Gödel Machine

This research paper presents the Red Queen Gödel Machine (RQGM), which is based on the evolutionary insight that species do not optimize against a static environment but adapt as their environments change with them.

Red Queen Gödel Machine (RQGM) is a framework for recursively self-improving agents in which both the agent and its evaluator evolve together rather than relying on a fixed benchmark.

It uses “Controlled utility evolution”, a technique that keeps evaluation constant within each epoch but allows it to change between epochs.

RQGM improves test pass rates compared to previous SOTA by adding an agent-as-a-judge code-review signal while using 1.35×-1.72× fewer tokens because the reviewer is queried only once.

With RQGM, co-evolved scientific paper writing and reviewing agents reach 1.78×–1.86× higher acceptance rates under a diverse agent-as-a-judge panel, while co-evolved Olympiad-level proof writing and grading agents reach 9% higher ground-truth accuracy.

Read more about this research using this link.

6. OCR 4

Mistral released OCR 4, which they claim to be the best OCR model to date.

Instead of just turning a document into text, OCR 4:

• Returns bounding boxes around text items

• Classifies each box into titles, tables, equations, signatures, and more

• Tells how sure it is about each extraction (inline confidence scores)

The model also:

• Supports 170 languages across 10 language groups

• Is small enough to run in a single container for fully self-hosted deployments

• Costs $2-5 per 1,000 processed pages

OCR 4 has a 72% average win rate across all leading OCR systems in blind human evaluations and achieves top scores on OlmOCRBench (85.20) and OmniDocBench (93.07).

Read more about this release using this link.

Subscribe now

5. MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

This research presents Multi-teacher On-Policy Distillation (MOPD), a post-training method that combines the capabilities of multiple domain RL-trained teacher models into one student model.

The method first RL trains separate domain-expert/teacher models. Then, it distills them into the student using the student’s own on-policy rollouts. This eliminates exposure bias and gives a denser optimization signal than off-policy finetuning.

On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, while inheriting nearly all of each teacher’s specialized capability.

It also enables parallel training of domain teachers, followed by merging their strengths into a single deployable model.

MOPD has been used in the post-training of MiMo-V2-Flash, a powerful, efficient, and ultra-fast foundation language model that particularly excels in reasoning, coding, and agentic scenarios.

Read more about this research using this link.

4. Ask, Don’t Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

This research paper presents BINEVAL, an alternative to “LLM-as-a-judge” scoring, which often produces opaque scores that are hard to debug.

Instead of asking a model to provide one overall score, BINEVAL divides each evaluation criterion into multiple simple yes/no (binary) questions.

Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output. It then scores each of these answers independently and combines them into interpretable, multi-dimensional scores.

This makes it easier to identify which checks failed, such as factual consistency, missing information, relevance, or redundancy.

This question-level feedback can also be used to iteratively improve evaluator prompts for summarization and generation.

Across multiple benchmarks such as SummEval, Topical-Chat, and QAGS, BINEVAL performs as well as or better than strong baselines such as UniEval and G-Eval, with particularly strong results on factual consistency.

Read more about this research using this link.

3. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Speculative Decoding speeds up LLM inference by using a small draft model that suggests tokens while the larger LLM checks them.

While the draft model is fast, it often loses token consistency due to a lack of inter-token dependencies. Also, indiscriminately verifying long drafts wastes compute, severely reducing throughput in high-concurrency serving systems.

This research paper proposes DSpark, a speculative decoding framework that combines high-throughput parallel generation with adaptive, load-aware verification.

DSpark uses:

• A semi-autoregressive architecture that adds a lightweight sequential dependency module to introduce intra-block dependency modeling and reduce suffix decay.

• Confidence-scheduled verification to dynamically decide how many draft tokens to verify for each request based on the chance that the prefix will survive and the current throughput profile of the serving engine

DSpark substantially improves the accepted length compared to SOTA autoregressive and parallel drafters across multiple offline benchmarks.

When deployed within the DeepSeek-V4 serving system under live user traffic, DSpark successfully reduces verification waste. It improves per-user generation speeds by 60-85% at matched throughput levels compared to the established production baseline (MTP-1).

By preventing severe throughput degradation under strict interactivity constraints, it also enables performance tiers that were previously unattainable, shifting the Pareto frontier of the DeepSeek-V4 serving system.

Read more about this research using this link.

Subscribe now

2. Claude Sonnet 5

Anthropic released Claude Sonnet 5, their most agentic Sonnet model yet. The model is well suited for coding, tool use, browser and terminal workflows, long-running agentic workflows, and professional knowledge work.

The model substantially outperforms Sonnet 4.6 across multiple domains, and its performance is close to that of Claude Opus 4.8 on many benchmarks, but at a lower price.

It also has lower rates of hallucination, sycophancy, and unwanted behaviors compared to Sonnet 4.6, has better resistance to prompt injection, and comes with cyber safeguards that detect and block dangerous cyber activity in real time.

Read more about this release using this link.

1. Reaching Trillion-Parameter Performance with a 35B Agent

This research introduces Agents-A1, a 35B Mixture-of-Experts agentic model that achieves performance comparable to trillion-parameter models by expanding the agent horizon rather than model size/ parameters.

The authors build a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes to produce agentic trajectories averaging 45K tokens.

Based on these, Agents-A1 is trained in three stages:

• Full-domain supervised fine-tuning to align the base model with broad agentic behaviors

• Domain-level teacher models to capture specialized skills in each domain

• Multi-teacher domain-routed on-policy distillation to improve knowledge transfer efficiency across different domains and combine six different domains into a single deployable student model

The resulting model achieves strong performance on long-horizon agent benchmarks compared with 1T-parameter models such as Kimi-K2.6 and DeepSeek-V4-Pro.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Are you struggling to keep up with LLMs and developments around them? I wrote this book to exactly solve this for you!

“LLMs in 100 Images” turns the difficult parts of modern LLM systems (Attention variants, prompting techniques, sampling techniques, post-training algorithms, and more) into visuals, making them simple to understand.

I’m running a flash sale on the book for a very limited time, and you can grab your copy today at a 30% discount!

Grab your 30% discount →

It’s now time for the lesson.

GPUs are driving the current AI revolution, and understanding them well will make you invaluable in the AI ecosystem. In this lesson, we will learn about NVIDIA GPUs and the interconnects that scale them to massive data centers used for training and serving LLMs.

Let’s begin!

But first, what is a GPU and why is it needed?

A GPU, or Graphics Processing Unit, is a specialized chip originally designed to render 3D graphics rapidly and efficiently.

But it soon became clear that the same mathematical operations (matrix addition and multiplication) used in graphics could also be used to train and serve AI. This made NVIDIA, a company that started as a chipmaker for video games, the leading company building GPUs for AI today.

GPUs are quite different from CPUs, which are the chips used for general processing in the computer. While CPUs contain a few powerful cores that are perfect for executing tasks with sequential and branching logic at very low latency, GPUs are the masters of parallel computing.

They have thousands of cores, each slower than a CPU core, but together they produce massive throughput for parallel computations, especially matrix or tensor operations. (A matrix is a 2D tensor).

A GPU has units called Streaming Multiprocessors (SMs), where calculations actually take place. Each SM contains smaller specialized components called:

• CUDA cores: that perform fast general mathematical operations

• Tensor cores: that perform fast matrix operations

Alongside this, a GPU has two types of memory:

1. On-chip memory (L2 cache, L1 cache, and registers) that is physically etched on the GPU die

2. High Bandwidth Memory (HBM), generally called global memory or VRAM, that is mounted alongside the die

HBM is a specialized type of Dynamic random-access memory (DRAM) designed for massive parallel data throughput.

On-chip memory components are Static random-access memory (SRAM), which is extremely fast but much smaller and more expensive than the GPU HBM.

If you’re interested in reading about how data flows through CPU and GPU during LLM inference, here is a lesson that will help:

The NVIDIA ecosystem of GPUs

NVIDIA GPUs come in different architectures, each named after a famous scientist or mathematician.

The company produces consumer-grade GPUs for graphics and gaming in its GeForce GTX (older) and RTX series. It also produces the RTX PRO (previously known as Quadro) series for commercial/ scientific/ creative workloads on workstations.

We won’t be discussing these GPU architectures in this lesson, but we'll focus more on data center-grade GPUs designed for deep learning/AI.

In 2017, NVIDIA introduced Tensor Cores and FP16 mixed-precision training for deep learning in its Volta series of GPUs (Tesla V100). OpenAI used a cluster of these GPUs to train GPT-3.

Since then, NVIDIA has made its GPUs more performant, and some popular architectures with their flagship GPU models released are:

• Turing series in 2018 (Tesla T4): Optimized for lower-precision integer (INT8/INT4) inference pipelines

• Ampere series in 2020 (A100): Used to train early LLMs and serve many GPT-3-scale models.

• Hopper series in 2022 (H100, H200): Introduced the Transformer Engine and FP8 precision, which were used in the training of Llama 3 and similar models

• Ada Lovelace series in 2022 (L40, L40S): Designed for high-performance AI inference and graphics rendering tasks

• Blackwell series in 2024 (B100, B200): Built for training and serving trillion-parameter reasoning models with support for FP4 (4-bit floating-point) precision

• Blackwell Ultra series in 2025 (B300): Improved Blackwell generation with more memory and higher performance (~50% higher FP4 compute)

• Rubin series in 2026 (Rubin GPU): Designed to provide roughly double the FP4 compute (50 vs 20 PFLOPS) and GPU-to-GPU bandwidth (3.6 TB/s vs 1.8 TB/s) compared to the Blackwell series GPUs.

• Rubin Ultra (announced to be released in 2027)

• Feynman (announced to be released in 2028)

Subscribe now

Understanding inter-GPU connections

A popular GPU like H100 from the Hopper series has 80GB of HBM. A single such GPU is too small to fit the weights of today's trillion-parameter large reasoning models, which have hundreds of GBs of parameters. This is why multiple GPUs are connected to form a large server (also called a Node). This approach is called Vertical scaling or the “scaling-up” approach.

There are different ways GPUs can be connected in a server:

• PCIe (Peripheral Component Interconnect Express): This is the general-purpose connection used inside servers to connect GPUs, network cards, SSDs, and other devices to the CPU. The 6th generation PCIe offers 256 GB/s of GPU-to-GPU bandwidth. PCIe makes GPU-to-GPU data take a slower path through the CPU's root complex, which increases latency and reduces bandwidth.

• NVLink: This is NVIDIA’s proprietary communication channel that gives a dedicated high-speed data path between GPUs. The bidirectional bandwidth between two GPUs provided by different generations of NVLinks is as follows:

  • 900 GB/s on H100 (NVLink 4)

  • 1.8 TB/s on B200 (NVLink 5)

  • 3.6 TB/s on Rubin (NVLink 6)

This is massive compared to the 256 GB/s offered by the 6th-generation PCIe (a 14× increase with NVLink 6)!

NVLink has an issue, though, in that it splits each GPU’s total bandwidth among the GPUs it connects. Check out the following example, where each Rubin GPU splits its 3.6 TB/s bandwidth among the other three, resulting in 1.2 TB/s per connection.

The general formula for the effective inter-GPU bandwidth with NVLink is B/N, where

• B is a GPU’s total NVLink bandwidth

• N is the number of GPUs it is connected to

The solution in this case is an NVSwitch.

• NVSwitch: This is a high-bandwidth, low-latency fabric that connects multiple GPUs within a system, ensuring GPUs can communicate with one another at full bandwidth simultaneously, regardless of how many are connected. This lets all the GPUs work together and act like a single massive GPU in a server.
  
  It must be noted that NVSwitches are more expensive than other connections and might be unnecessary if you’re working with smaller (a few billion parameters) LLMs.

GPUs aren’t all that we need

While GPUs are used for fast parallel operations during LLM training and inference, CPUs are needed to:

• preprocess and load/ unload data into GPUs

• orchestrate jobs between GPUs

• manage storage and networking

• run other serial/ non-parallel operations

If the CPUs used in the process are too slow, the expensive GPUs would sit idle most of the time. This is why NVIDIA has launched its own Arm-based CPU models, namely:

• Grace

• Vera

These are used alongside GPUs and connected to them using NVLink-C2C (C2C stands for chip-to-chip). This is a superfast CPU-to-GPU connection that replaces the slower PCIe connection and provides both the GPU and CPU with a unified memory space, allowing them to access each other’s memory directly without manual copying.

NVIDIA combines its GPUs and CPUs together in a server rack, with three popular ones being:

• GB200 NVL72: Combines 72 Blackwell GPUs and 36 Grace CPUs

• GB300 NVL72: Combines 72 Blackwell Ultra GPUs and 36 Grace CPUs

• Vera Rubin NVL72: Combines 72 Rubin GPUs and 36 Vera CPUs

(GB stands for Grace-Blackwell, and NVL tells how many GPUs are connected using NVLink/NVSwitch.)

But these racks aren’t the only way NVIDIA packages these components. NVIDIA also makes multi-GPU (no CPU included) baseboards in its HGX series, with the popular ones being:

• HGX A100: Combines 4, 8, or 16 A100 GPUs

• HGX H100: Combines 4 or 8 H100 GPUs

• HGX Rubin NVL8: Combines 8 Rubin GPUs

In the DGX series, NVIDIA combines GPUs, CPUs, memory, networking, storage, cooling, and software into a complete AI system (for example, the DGX H100 server).

Its MGX series is a modular server architecture in which the above-described components are swappable, letting server makers to combine different options to build custom server configurations.

Connecting servers & scaling horizontally

There’s a limit to how much we can vertically scale GPUs. This is due to the chip's physical constraints, as well as power and cooling requirements. In such a case, multiple servers are connected in an approach called Horizontal scaling or the “scaling-out” approach.

Although data transfer speed is lower than with vertically scaled GPUs, the benefit of horizontal scaling is that one can theoretically connect any number of GPUs together.

NVIDIA offers two high-throughput connections for horizontal scaling, both of which use GPUDirect RDMA (Remote Direct Memory Access), which lets GPUs exchange data directly from memory while bypassing the CPU and OS.

1. Spectrum-X Ethernet: This is the standard AI-tuned Ethernet connection in a data center that offers high performance in AI workflows. It uses RoCE (RDMA over Converged Ethernet), a protocol that helps run RDMA over an Ethernet network.

2. Quantum InfiniBand: This is a specialized connection that delivers ultra-low latency for high-end LLM training data centers. It uses RDMA natively without using RoCE.

Both have a throughput of about 100 GB/s per connection, which is far lower than that of NVLink with NVSwitch (3.6 TB/s per connection). This means that workloads that require super-fast data transfer must be kept between vertically scaled GPUs, while the others could be directed to GPUs shared across servers.

For example, during training, Tensor parallelism (TP) is implemented within GPUs on a single server. On the other hand, Data parallelism (DP) and Pipeline parallelism (PP) are implemented across servers as they exchange data less often.

NVIDIA Collective Communications Library (NCCL) handles and coordinates data exchange between GPUs, automatically routing each workload to the fastest available link.

If you’re new to the terms TP, DP, and PP, the following lesson would help:

The hierarchy of GPU connections

Revisiting what we discussed before, GPUs are arranged hierarchically in large data centers as follows:

• Server or Node (4 to 8 GPUs)

• Server rack (10s to 100s GPUs)

• Cluster or Pod (100s to 1000s GPUs)

• Data center (10,000 to 100,000 GPUs)

An example of a massive data center is Colossus, developed by xAI primarily for training Grok. Today, this data center also powers LLMs and research labs at Anthropic, Google, and Reflection AI. Colossus consists of 200,000 H100 GPUs and is intended to be scaled to 1 million GPUs (including H100, H200, and GB200).

TL;DR

To summarise:

• GPUs, initially designed for graphics rendering, are now the backbone of the current AI ecosystem, thanks to their ability to perform large numbers of matrix operations in parallel.

• While CPUs have few cores that can perform sequential work faster than GPUs, GPUs have thousands of slower cores that, overall, deliver massive parallel throughput.

• A GPU is made up of Streaming Multiprocessors (SMs) containing CUDA cores (for general math operations) and Tensor cores (for matrix math operations).

• A GPU has smaller but faster on-chip SRAM (L1 cache/L2 cache/registers) and large off-chip HBM/VRAM, which is a type of DRAM.

• The most commonly used NVIDIA GPU series today are Ampere, Hopper, Blackwell, and Rubin.

• GPUs are connected in a single server using PCIe, NVLink, or NVSwitch. This approach is called Vertical scaling.

• NVIDIA pairs its GPUs with its own Arm-based CPUs (Grace and Vera) using NVLink-C2C, a high-speed connection that provides both chips with a unified memory space.

• Multiple servers are connected together using Spectrum-X Ethernet or Quantum InfiniBand to build massive data centers. This approach is called Horizontal scaling.

This article is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Also, don’t forget to grab your copy of “LLMs In 100 Images” at a 30% discount! ✨

Grab your 30% discount →

While most developers can spin up a RAG pipeline in an afternoon using LangChain or LlamaIndex, very few understand its internals well and know how to fix it when things go wrong.

‘RAG from First Principles’ takes the RAG stack apart layer by layer and teaches you about ingestion, chunking, embeddings, vector indexes, hybrid search, reranking, and evaluation.

Each chapter answers the questions practitioners actually hit in production, building from data import all the way to GraphRAG, Agentic RAG, and Modular RAG.

By the end, you’ll be able to optimize, debug, and extend your RAG systems with confidence and not guesswork.

Master RAG today →

10. ENPIRE: Agentic Robot Policy Self-Improvement in the Real World

This research from NVIDIA introduces ENPIRE, a framework for coding agents that enables them to autonomously improve robot policies in real-world settings.

ENPIRE has four core modules:

1. Environment module (EN) that automatically resets the scene and checks whether a task succeeded

2. Policy Improvement module (PI) that launches policy refinement

3. Rollout module (R) that evaluates policies with single or multiple physical robots operating in parallel

4. Evolution module (E) that lets coding agents analyze logs, consult literature, improve training infrastructure, and algorithm code to fix failures

Using ENPIRE, frontier coding agents can autonomously build a policy that achieves a 99% success rate on challenging dexterous manipulation tasks such as PushT, organizing pins into a pin box, and using a cutter to cut a zip tie.

Read more about this research using this link.

Subscribe now

9. Looped World Models

World models today have significant computational requirements to run faithful long-horizon simulations. This makes them expensive to deploy and prone to compounding errors.

This research addresses this by introducing Looped World Models (LoopWM), which use looped, parameter-shared transformer blocks to refine latent environment states through repeated internal iterations, rather than adding multiple separate layers.

This leads to ~100× parameter efficiency over conventional approaches with adaptive computation that automatically scales depth to match the complexity of each prediction step.

LoopWM introduces iterative latent depth as a new scaling dimension for world simulation, rather than increasing model size or training data.

Read more about this research using this link.

8. Surpassing Frontier Performance with Fusion

This research from OpenRouter introduces Fusion, an approach that allows selecting a panel of participant models alongside a judge model. It then sends a prompt to multiple participant models in parallel and uses the judge model to combine their answers into a single stronger response.

The results show that:

1. Panels of models consistently outperform individual models

2. Frontier panels can achieve beyond-frontier performance

3. Panels of budget models can beat frontier models and get close to frontier panel performance

Two notable examples from the results are:

• Fable 5 + GPT-5.5 scores 69% on OpenRouter’s DRACO deep-research benchmark, while Fable 5 alone scores 65.3% on this benchmark.

• A budget panel of Gemini 3 Flash, Kimi K2.6, and DeepSeek V4 Pro outperforms GPT-5.5 and Opus 4.8. It also scores within 1% of Fable 5’s score while costing half as much.

Although highly performant, it must be noted that this method is slower, costlier, and not a drop-in replacement for coding or long-horizon agents.

Read more about this method using this link.

7. Next-Latent Prediction Transformers Learn Compact World Models

This research paper introduces NextLatent Prediction (NextLat), which adds a self-supervised next-latent prediction loss to transformers, helping them learn latent representations that predict the next latent state given the next token.

These latest representations form “belief states”, which are compressed information about the history necessary to predict the future (compact internal world models).

Across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat leads to significant gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning.

It also enables variable-length self-speculative decoding, improving inference by up to 3.3× in language modeling.

Read more about this research paper using this link.

6. HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining

This research paper presents a systematic study that compares egocentric human video with teleoperated real-robot trajectories as pretraining data for embodied robot foundation models.

Human egocentric data is not only scalable, substantially lower-cost, and more diverse than teleoperated real-robot data, but under fixed pretraining, post-training, and validation protocols, it also leads to superior performance.

With the same amount of pretraining data, models pretrained on egocentric data achieve:

• 24% lower validation loss on real-robot action prediction,

• 52.5% higher success rates on in-distribution real-robot task execution

• 90% higher success rates on out-of-distribution real-robot task execution

This suggests that the best approach to training an embodied foundation model is to pretrain on egocentric human video to learn diverse world representations, then adapt using a small amount of labeled real-robot data for action-space alignment.

Read more about this research using this link.

5. Variable-Width Transformers

Standard Transformers use the same width for all layers. This means that each layer has the same number of parameters and compute budget, even though they might have different functions and computational needs in language modeling.

This research paper from MIT changes this by introducing an X-shaped Transformer called the “>< former”, in which the early and late layers remain wide, while the middle layers are narrower.

This approach works surprisingly well and outperforms parameter-matched standard Transformers (ranging from 200M to 3B parameters) in terms of language modeling loss and on most downstream tasks.

This architecture also requires fewer overall FLOPs (a 22% reduction) and smaller KV cache memory and I/O costs (a 15% reduction).

Read more about this research using this link.

4. Kimi K2.7 Code

Moonshot AI released Kimi K2.7 Code, its new open-source agentic model that is optimized for long-horizon software engineering tasks.

It is a 1T-parameter MoE model with 32B active parameters and uses MLA attention and the MoonViT vision encoder.

It improves over K2.6 on multiple coding and agent benchmarks, has better instruction-following capabilities in long contexts, uses about 30% fewer thinking tokens, and supports a 256K context window.

Alongside this, its performance is close to that of GPT-5.5 and Claude Opus 4.8 on many benchmarks while it is roughly 5-7x cheaper to run.

Read more about this release using this link.

3. VibeThinker-3B

This research paper from Weibo introduces VibeThinker-3B, a 3B-parameter dense reasoning model that achieves performance comparable to that of large frontier models on math and coding tasks.

The model is post-trained using Qwen2.5-Coder-3B as the base model, using several techniques such as:

• Curriculum-based two-stage SFT: Training first on broad reasoning/dialogue data, then progressively harder long-horizon math/code/STEM examples

• Multi-domain RLVR using MGPO: Training across math, coding, and STEM examples using the MGPO algorithm. MaxEnt-Guided Policy Optimization (MGPO) helps the model explore different reasoning paths near its current capability, then amplifies the paths that produce verifiably correct answers.

• Offline Self-Distillation: Collecting the best reasoning traces from the model’s own RL-specialized checkpoints and distilling them back into a single 3B model

• Instruct RL: Training the model to follow user instructions and stick to output formats and user constraints reliably without losing reasoning gains

The model scores 94.3% on AIME26 (97.1% with Claim-level Reliability), 80.2% Pass@1 on LiveCodeBench v6, and achieves a 96.1% acceptance rate on recent unseen LeetCode contests.

Read more about this research using this link.

2. GLM-5.2

Z.ai introduced GLM-5.2, its open MIT-licensed flagship model for long-horizon coding and agent tasks.

The model:

• Has a 1M-token context window

• Has stronger coding capabilities with multiple thinking effort levels to balance performance and latency

• Uses IndexShare, which reuses the same indexer across every four sparse attention layers, reducing per-token FLOPs by 2.9× at a 1M context length.

• MTP layer is also better suited for speculative decoding, increasing the acceptance length by up to 20%.

GLM-5.2 is the strongest open model on several coding and agentic benchmarks, with performance close to frontier models on long-horizon tasks.

Read more about this release using this link.

1. Towards autonomous medical artificial intelligence agents

This research paper introduces MIRA (Medical Intelligence for Reasoning and Action), an autonomous AI agent capable of operating within a sandboxed EHR (Electronic Health Record) environment.

It can:

• Take histories

• Order and interpret laboratory, imaging, and microbiology tests

• Generate differential diagnoses

• Formulate treatment plans, including prescribing medications, scheduling surgical procedures, and planning admissions.

On 574 real MIMIC-IV cases across 8 diseases, MIRA achieved 88.9% diagnostic accuracy and, in head-to-head testing, outperformed board-certified physicians (87.8% vs 78.1% accuracy), while showing strong medication safety and alignment with medical guidelines.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Building Your First AI Agent

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Join 'Into AI' premium today

Your agent just ran for an hour. It changed multiple files, called multiple tools, followed leads, hit dead ends, made decisions, and maybe touched something you wish it hadn’t. Most of this is invisible or too big for you to go through.

Spotlight by Backplanes turns your Claude Code and Codex sessions into valuable reports, so you can understand the agent run without digging through logs.

Spotlight is free for individual developers and the teams they work with (no credit card required). They also remove sensitive info, encrypt your data, use providers that do not store it, never sell it, and delete it when you delete your sessions, projects, or account.

Btw, I personally used it and, embarrassingly, found out that Claude Code was reading my API keys during my coding sessions.

Try Spotlight today →

LLM Inference is the process of running a forward pass through a trained model to produce text. Understanding what happens at inference time at the CPU/GPU level will help you optimize this process more effectively. Here is a lesson where we discuss exactly this.

The process starts with loading parameters into the CPU memory

To begin with, a trained model’s parameters are stored in the hard drive (preferably an NVMe SSD) and can have different formats such as:

• Safetensors

• bin or pt when working with PyTorch

• GGUF

These come alongside a config.json file that tells about the model architecture, hyperparameters, and data type.

(Check out the config.json and model parameter files for the DeepSeek-V4-Flash model to understand this better.)

A model loader uses this file to build the model skeleton and then loads the parameters into the CPU memory (also called System memory). This memory is typically Dynamic random-access memory (DRAM), which offers larger capacity and is cheaper than the GPU memory.

The model parameters are next transferred to the GPU's HBM (High Bandwidth Memory), also known as global memory (or, generally, VRAM). This transfer takes place over PCIe, a high-speed connection between the CPU and GPU.

Seen tensor.to('cuda') method while training models? This is what happens under the hood when you call this method.

HBM is a specialized type of DRAM designed for massive parallel data throughput. While CPU memory has a throughput of 50 to a few hundred GB/s, HBM can deliver a throughput of a few TB/s. Although fast, HBM is smaller and much more expensive than CPU memory.

What if the LLM is too big for the GPU memory?

Let’s go back to the config.json file for DeepSeek-V4-Flash, which is a Mixture-of-Experts model with 284 billion total parameters.

If each parameter of this model is loaded in FP8 (8-bit floating-point) precision, which means that 1 parameter is represented in 8 bits or 1 byte, the total memory requirement of this model for parameter storage will be:

284B parameters × 1 byte/ parameter = 284 GB

But this is not the only thing that the GPU HBM needs to store. We also need space for KV cache, activations, and other overheads.

The popularly used NVIDIA H100 GPU comes with 80 GB of HBM. This is nowhere near enough to store 284 GB of parameters, let alone the KV cache and others. This means that the parameters must be distributed across multiple GPUs.

A standard architecture for hosting a model is an 8-GPU NVIDIA HGX H100 server. This server has 8 H100 GPUs, each with 80 GB of HBM, which sums to a total of 640 GB of memory.

The GPUs in the server are linked using an NVSwitch, which provides each GPU with a full-bandwidth path to every other GPU, enabling them to transfer data fast enough (900GB/s) to behave as a single large accelerator.

The term “full-bandwidth” is important here because if no NVSwitch is used with NVLinks, the bandwidth is split between GPU pairs. In our 8-GPU setup, any single GPU-to-GPU pair would get only about 128 GB/s of bandwidth, compared to 900 GB/s (full bandwidth) with NVSwitch.

There are many techniques that are used to distribute the inference load across these GPUs. Some techniques split the model itself, while others split the input workload.

These are described as follows:

• Tensor parallelism (TP): Splitting parameter/ weight matrices across GPUs, with each GPU performing part of the matrix multiplication and then syncing the results.

• Pipeline parallelism (PP): Dividing the model’s layers across GPUs, with each GPU handling a consecutive block of layers and passing the activations to the next.

• Context parallelism (CP): Splitting a single sequence and its KV cache across GPUs, with each GPU managing a part of the context to support very long sequence lengths.

• Expert parallelism (EP): Distributing the experts of a Mixture-of-Experts (MoE) model across GPUs and routing the tokens to the GPU that holds their chosen experts.

• Data parallelism (DP): Replicating the entire model on each GPU when it is small enough to fit, or replicating a fully sharded model across multiple GPU groups when the model is too large for a single GPU. Each replica then handles different user requests.

• Hybrid parallelism: Combining several of these techniques discussed above. For example, TP, PP, CP, and EP are combined to create a complete sharded model replica. DP is then added by creating multiple such replicas, each serving different user requests in parallel.

If you’re completely new to these techniques, the following lesson will help.

What's next after loading parameters into GPU memory?

Once the parameters are loaded into GPU HBM, they are ready to process incoming user requests.

Let’s talk a bit about the main components of a GPU like the H100. These are:

• Streaming Multiprocessors (SMs)

• High-bandwidth memory (HBM)

• On-chip memory

Streaming Multiprocessors, or SMs, are the processing units where calculations happen in a GPU. They contain hundreds of smaller components called:

• CUDA cores: that perform fast general mathematical operations

• Tensor cores: that perform fast matrix operations (matrix multiplications)

The on-chip memory is etched directly on the GPU die, unlike HBM, which is mounted alongside the die. This memory further consists of (arranged in ascending order of speed and descending order of capacity):

• L2 memory/cache

• L1 or shared memory/cache (shared across components of an SM)

• Registers

On-chip memory components are Static random-access memory (SRAM), which is extremely fast but much smaller than other types of DRAM (CPU memory and GPU HBM) we discussed earlier.

For every user request/prompt, the model performs forward passes through its parameters in two phases: Prefill and Decode.

In each forward pass, every layer’s parameters are streamed from HBM to on-chip memory and then to the CUDA/Tensor cores, where the calculations actually occur. The weights are discarded once a token is produced, and this process repeats until the full sequence is generated or the maximum token-generation limit is reached.

If you want to better understand what calculations occur at the Transformer level, we have covered this in the previous lessons that you can find here:

What is Prefill and Decode?

Prefill is the first phase where all tokens in a user’s prompt are processed together in a single forward pass through the LLM.

During this phase, each token computes a key (K) and value (V) vector, which are cached to build a KV cache for the entire initial user prompt. This KV cache is stored alongside the model parameters and intermediate activations in the HBM.

Prefill relies heavily on the GPU's tensor cores and is compute-bound. This is because all tokens of the initial user prompt are being processed in parallel.

A faster prefill means a shorter Time to First Token (TTFT), which is the delay before the model begins responding to the user's prompt.

Next comes Decode, the second phase, where the LLM generates one token at a time, with a single forward pass through the model for each token.

Instead of recomputing the key (K) and value (V) vectors for every previous token at each step, the model reuses these from the KV cache and only computes them for the new token. The newly computed K and V values are added to the KV cache for use in the next step of Decode.

Decode is memory-bound and does not utilize the full capacity of the GPU’s Tensor cores. This is because, for every generated token, all model parameters must be streamed from HBM to the Tensor cores for a small amount of computation, discarded when the token is generated, and then re-read in full for the next token.

In this process, the calculations performed per token relative to the volume of weights moved are so small that the cores spend most of their time waiting on memory rather than computing.

Why is Decode memory-bound?

Let’s understand this better using an example.

Let’s say that we are using a 30B Dense model in 16-bit precision on a single H100 (with 80 GB of HBM). This is 60 GB of parameters (30B × 2 bytes).

To produce one token, all 60 GB are streamed from HBM to the Tensor cores. At the higher end of the H100’s memory bandwidth of 3.35 TB/s, it takes about 18 ms to generate one token. This gives a throughput of around 56 tokens per second.

Now let’s compare this with the computation involved.

A forward pass costs roughly 2 FLOPs per parameter per token (one multiply and one add operation), so the computational cost of processing a token through a 30B dense model is 60 GFLOPs (2 FLOPs × 30B).

The H100's Tensor cores can perform 990 TFLOP/s of FP16 operations. This means that 60 GFLOPs of operations will take around 0.06 ms (60 / 990,000).

Taken together, the computation takes roughly 0.06 ms, while streaming the 60 GB of weights takes 18 ms.

The cores only compute for 0.3% of the time required for the memory transfer. For the rest, they sit idle, waiting for the next set of parameters to arrive, making Decode memory-bound.

Over the past 20 years, peak server hardware FLOPS has scaled by a factor of 3 every 2 years, outpacing the growth of DRAM and interconnect bandwidth, which have scaled by factors of 1.6× and 1.4× every 2 years, respectively. This has made memory (and not compute) the primary bottleneck in LLM inference. (Source)

Why is the KV Cache so important for Decode?

The KV cache ensures that the keys (K) and values (V) for past tokens are computed once and stored, so each Decode step computes only one new token's K and V and reads the rest from cache.

This slows the increase in the cost of generating a single token as the sequence length grows, making Decode faster and more efficient.

You must remember that using a KV cache is not always a “free lunch” since it also takes space in the HBM. As its size grows (with increasing context length and the number of batched requests), it could consume as much memory as the parameters themselves. Managing its size is something that one needs to keep in mind when serving LLMs.

TL;DR

To summarise:

• During inference, model parameters are moved from SSD to CPU memory, GPU HBM, on-chip memory, and finally to the computation cores.

• Models that are too large for a single GPU are split across multiple GPUs using multiple parallelism techniques.

• The multi-GPU server uses NVSwitch to connect the GPUs, resulting in full communication bandwidth between them.

• Each inference request is processed in two phases: Prefill and Decode.

• Prefill processes all prompt tokens at once. It is a compute-bound process, and the Time to First Token (TTFT) depends on it.

• Decode generates one token at a time. It is memory-bound and acts as the bottleneck of inference. This is because each token generation involves moving a large number of parameters while performing very little computation, leaving the GPU's computation cores mostly idle while waiting on memory.

• The KV cache helps keep token generation costs from growing rapidly as sequence length increases.

• KV cache also consumes HBM alongside model parameters and must be managed for efficient LLM serving.

✨ Courtesy of Backplanes, this newsletter edition is completely free to read. ✨

Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

10. DiffusionGemma

Google released DiffusionGemma, an experimental open model that uses Diffusion to generate entire blocks of text simultaneously, leading to 4x faster text generation on dedicated GPUs.

DiffusionGemma is a 26B Mixture-of-Experts (MoE) model that activates only 3.8 billion parameters during inference. It uses bidirectional attention to generate 256 tokens in parallel, with each forward pass allowing every token to attend to all others.

The model comes with native support for NVIDIA’s new NVFP4 (4-bit floating-point) format on Blackwell GPUs, which dramatically increases compute throughput, enabling it to run at faster speeds with near-lossless accuracy.

The model’s impressive capabilities make it particularly helpful for speed-critical local workflows such as inline editing, rapid iteration, code infilling, non-linear text structures, amino acid sequences, mathematical graphs, and tasks like Sudoku.

However, it must also be noted that the standard Gemma 4 still produces higher-quality outputs than DiffusionGemma.

Read more about this release using this link.

If you’re new to Diffusion LLMs, you can read more about them using the following lessons.

9. Frontier Code

Cognition introduced FrontierCode, a benchmark to test whether AI coding agents can produce production-quality, mergeable code rather than just passing tests.

Built with over 20 open-source maintainers across 36 major repositories, it includes 150 tasks divided into three tiers: Extended, Main, and the hardest Diamond tier.

These tasks are graded on correctness, regression safety, test quality, scope control, style, and codebase conventions. Its evaluation combines unit tests, rubrics, command checks, LLM review, and Cognition’s “mutagent” (an LLM-based tool to surgically patch the test environment/ application code and align with the agent’s implementation details), all with strong quality control.

Cognition claims that FrontierCode has an 81% lower false-positive rate than SWE-Bench Pro and that it is the first-ever benchmark measuring code quality and subtle human preferences.

Claude Opus 4.8 leads the benchmark with a score of 13.4%, GPT-5.5 scores 6.3% while using fewer tokens, and Kimi K2.6 is the top open-source model with a 3.8% score, all on the Diamond tier.

Read more about this release using this link.

Before we move forward, I want to introduce you to my book called ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master LLMs today.

Grab your copy today at a special discount using this link.

8. Claude Fable 5 & Mythos 5

Anthropic released Claude Fable 5, a public Mythos-class model, and Claude Mythos 5, the same underlying model with safeguards lifted for trusted cyberdefenders and infrastructure providers.

This model outperforms all of Anthropic’s previous models on nearly all tested benchmarks and is especially strong on long and complex tasks.

It also has major cybersecurity and scientific research capabilities, including faster drug-design workflows and the generation of novel biological hypotheses.

The publicly released Fable 5 includes safeguards for cybersecurity, biology/chemistry, and distillation-related requests and routes these to the previous model, Claude Opus 4.8.

As of 12th June 2026, access to Claude Fable 5 and Claude Mythos 5 has been suspended due to national security concerns raised by the US government.

Read more about this release using this link.

7. FlashMemory-DeepSeek-V4

This research paper introduces Lookahead Sparse Attention (LSA), which makes ultra-long-context LLM inference more memory-efficient.

Instead of passively attending to all previous tokens, LSA proactively predicts which parts of the context it'll need in the future and keeps only those KV chunks in GPU memory.

It uses a small "Neural Memory Indexer" built on the DeepSeek-V4 architecture to decide which parts of the context are worth keeping.

Across long-context benchmarks (LongBench-v2, LongMemEval, and RULER), this uses just 13.5% of the memory on average while maintaining the same accuracy.

And at 500K-token lengths, it reduces memory overhead by more than 90% without compromising the model's reasoning ability.

Read more about this research using this link.

Subscribe now

6. Latent Spatial Memory for Video World Models

This research paper introduces Mirage, a latent-space spatial memory framework for making generated videos more 3D-consistent over long camera trajectories.

Rather than storing scene memory as an RGB point cloud that requires repeated rendering and re-encoding, Mirage stores static scene information as 3D latent tokens within the diffusion model’s latent space.

It builds this memory by lifting latent tokens into 3D with depth-guided back-projection, reads it through direct latent-space warping, and updates it chunk by chunk while filtering dynamic objects.

Experiments show that Mirage achieves up to 10.57× faster end-to-end video generation and a 55× reduction in memory footprint relative to explicit 3D baselines. It also attains SOTA performance on WorldScore and strong reconstruction quality on RealEstate10K.

Read more about this research using this link.

5. MiniMax Sparse Attention (MSA)

This research paper introduces MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA).

Traditional attention has a cost that grows quadratically with the number of tokens it attends to. This becomes far too expensive for LLMs with ultra-long contexts.

MSA uses a lightweight “Index Branch” to score KV blocks and select a Top-k subset for each query group, while a “Main Branch” performs exact sparse attention only over those selected blocks.

On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4× at 1M context. With an efficient kernel, it also achieves 14.2× prefill and 7.6× decoding speedups on the H800 GPU.

Read more about this research using this link.

4. First Steps Toward Automated AI Research

This blog post from Recursive introduces their automated AI research system, which achieves SOTA results across three benchmarks (fixed-budget language model training, small-model training speed, and GPU kernel optimization).

The system runs as an end-to-end research loop that includes proposing ideas, implementing them, running experiments, validating results, and using what it learns to choose future experiments.

On NanoChat Autoresearch, it improved fixed-budget language model training from 0.9372 to 0.9109 validation BPB.

On NanoGPT Speedrun, it reduced training time from 79.7s to 77.5s.

And on SOL-ExecBench, it raised the mean GPU-kernel optimization score from 0.699 to 0.754, which is an 18% reduction in the gap to the estimated hardware optimum.

Read more about this research using this link.

3. End-to-End Context Compression at Scale

This research paper introduces Latent Context Language Models (LCLMs), a family of encoder-decoder compressors that improve the efficiency of long-context LLM inference compared to using full KV caches.

LCLMs compress long token sequences into shorter latent embeddings, which the decoder can use directly. This improves the trade-off among task performance, compression speed, and peak memory usage while avoiding common limitations of KV compression (quality loss, high compression costs, constraints on target context windows, and poor compatibility with production inference systems).

Read more about this research paper using this link.

Subscribe now

2. General-purpose large language models outperform specialized clinical AI tools on medical benchmarks

This research paper evaluates two clinical AI tools, OpenEvidence and UpToDate Expert AI, against three frontier LLMs (GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6) across:

1. 500 MedQA questions testing medical knowledge

2. 500 HealthBench items measuring alignment with clinicians

3. Real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment

The results show that frontier LLMs outperform clinical AI tools in all three evaluations.

On the RCQ benchmark, Clinical AI tools perform comparably to the auto-enabled Google Search AI Overview.

Read more about this research paper using this link.

1. Self-Harness: Harnesses That Improve Themselves

This research paper introduces Self-Harness, which lets LLM-based agents improve their own operating harness without relying on human engineers or stronger external agents.

Self-Harness works iteratively in three stages:

1. It first checks the execution traces and finds which model-specific patterns led to failure (Weakness Mining)

2. It then generates harness modifications tied to these failures (Harness Proposal)

3. Finally, it accepts the modifications after successful regression testing (Proposal Validation)

When tested on Terminal-Bench-2.0 using three models (MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5) initialized with a minimal harness, Self-Harness consistently improves held-out pass rates by 21.4%, 14.3%, and 14.2%, respectively.

Read more about this research paper using this link.

This newsletter edition is completely free to read.

If you found it valuable, click the like button ❤️ and consider subscribing for more such content every week.

If you have any questions or suggestions, feel free to leave a comment below.

Into AI is a reader-supported newsletter. Gain access to deeper, members-only content by becoming a paid subscriber today.

Join 'Into AI' premium today

We then deepened our understanding by building and training a Mixture-of-Experts (MoE) LLM from scratch.

Both these models were trained using the next-token prediction objective and generate tokens one at a time, left to right, autoregressively.

But this is not the only way that a model can be used to generate text.

We have Diffusion LLMs that can generate tokens in parallel using a process called Diffusion. One of the most successful examples of this type of LLM is LLaDA (Large Language Diffusion with mAsking), which we discussed in depth in the following lesson.

Google also released its experimental open-source model, DiffusionGemma, this week, which operates on the same principle.

In this lesson, we will take our understanding to the next level and learn to:

• Implement a 13-million-parameter Diffusion LLM from scratch

• Train it on a publicly available pre-training dataset using a free GPU

• Generate text using it

Let’s begin!

Setting up the environment

We will code in PyTorch, use the Hugging Face datasets library for the training dataset, and transformers library to obtain the tokenizer.

The code is meant to run on Google Colaboratory and uses the free NVIDIA T4 GPU to train our model.

# Install packages
!uv pip install torch datasets transformers tqdm

# PyTorch core imports
import torch
import torch.nn as nn
import torch.nn.functional as F

# For numerical operations
import math

# For data processing
from torch.utils.data import DataLoader, Dataset

# Tokenizer
from transformers import AutoTokenizer

# Optimizer
import torch.optim as optim

# For mixed-precision training 
from torch import amp
from torch.nn.utils import clip_grad_norm_

# To visualise progress bar
from tqdm import tqdm

# Hide deprecation warnings
import warnings
warnings.filterwarnings('ignore')

# Set the device (GPU or CPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

Setting up the tokenizer

Instead of building the tokenizer from scratch, we will use the BPE tokenizer for GPT-2. This gives us a 50,257-token vocabulary made up of subwords.

tokenizer = AutoTokenizer.from_pretrained("gpt2")

print(f"Original vocabulary size: {tokenizer.vocab_size}")
# Original vocabulary size: 50257

The end-of-sequence (EOS) token is at the last index in the vocabulary.

print(tokenizer.eos_token)
# <|endoftext|>

EOS_ID = tokenizer.eos_token_id 

print(EOS_ID)
# 50256

To train a diffusion model, we need a special <MASK> token. This token acts as a placeholder for the model to identify the input positions it should fill. We simply append it as a new ID at the end of the vocabulary.

MASK_ID = tokenizer.vocab_size

print(MASK_ID)
# 50257

# Increase the vocabulary size by 1 for the newly added <MASK> token
VOCAB_SIZE = tokenizer.vocab_size + 1 

print(VOCAB_SIZE)
# 50258

Next, we create two helper functions to encode and decode text using the tokenizer.

# Convert input text into a list of token IDs using the tokenizer
def encode(text): 
    return tokenizer.encode(text, add_special_tokens=False)

# Remove any MASK_ID tokens from the sequence and convert the list of token IDs back into readable text
def decode(ids):
    ids = [i for i in ids if i != MASK_ID]
    return tokenizer.decode(ids, skip_special_tokens=True)

Getting our data ready

We will train our diffusion LLM on the TinyStories dataset. It is a synthetic dataset of short stories that contains the vocabulary used by a 3-year-old, generated by GPT-3.5 and GPT-4.

We will use a subset of this dataset that is small enough to train our model on a free-tier GPU, yet rich enough for it to learn semantic details. It is downloaded as follows.

from datasets import load_dataset

# Download the 5000 stories from the 'train' split of TinyStories
dataset = load_dataset("roneneldan/TinyStories", split="train[:5000]")

Each row in the dataset is one complete story. Check out an example of one of them.

print(f"Example Story: \n\n{dataset[0]["text"]}")

"""
Example Story: 

One day, a little girl named Lily found a needle in her room. She knew it was difficult to play with it because it was sharp. Lily wanted to share the needle with her mom, so she could sew a button on her shirt.
Lily went to her mom and said, "Mom, I found this needle. Can you share it with me and sew my shirt?" Her mom smiled and said, "Yes, Lily, we can share the needle and fix your shirt."
Together, they shared the needle and sewed the button on Lily's shirt. It was not difficult for them because they were sharing and helping each other. After they finished, Lily thanked her mom for sharing the needle and fixing her shirt. They both felt happy because they had shared and worked together.
"""

We pre-process this dataset of stories by:

• Tokenizing each story (converting sub-words into token IDs)

• Joining them into a single list of token IDs

• Inserting an EOS token between stories to help the model learn the boundaries between different stories

def clean_and_tokenize(dataset):
    token_ids = []

    for text in dataset["text"]:
        # Remove leading/trailing whitespace from each story
        story = text.strip()

        # Skip empty entries
        if not story:
            continue

        # Encode the story into token IDs and append to the sequence
        token_ids.extend(encode(story))

        # Add an EOS token to separate stories
        token_ids.append(EOS_ID)

    return token_ids

print("Preprocessing dataset...")

token_ids = clean_and_tokenize(dataset)

print(f"Total training tokens: {len(token_ids):,}")
# Total training tokens: 1,033,087

If you’ve previously trained an autoregressive LLM from scratch, you must be familiar with the standard approach of shifting the input sequence by one token and using it as the target during training.

Diffusion LLMs aren’t trained this way.

Instead of next-token prediction, they use fixed-length, probabilistically masked sequences, and the model is trained to predict the original clean input from these masked versions.

Read more

It was adapted in NVIDIA’s Nemotron-3 Super and Ultra models, and now Microsoft has built its first in-house reasoning model, called MAI-Thinking-1, using it. All of these models have reported meaningful gains in accuracy without sacrificing inference throughput or latency using this architecture.

Latent Mixture-of-Experts is an improvement over the popular Mixture-of-Experts (MoE) architecture. Let’s build our foundations by understanding the MoE architecture in depth before we learn about Latent MoE.

What is the MoE architecture?

LLMs based on the Mixture-of-Experts (MoE) architecture contain multiple small feed-forward networks (called Experts) instead of a conventional large feed-forward network in their Transformers. These experts handle different tokens by using another network, called a Router, that selects which expert to use for each token.

MoE enables LLMs to scale their parameter count while keeping the compute cost, or the number of Floating-point Operations (FLOPs) per token, fixed. This is because each token is not processed by all experts in the model as in conventional dense LLMs.

In an MoE LLM with ‘N’ experts, the router directs each token towards only the top-K selected experts, so the active parameter count (which determines FLOPs) stays fixed while the total parameter count of the model (which stores knowledge) can grow enormously.

A great example of an MoE LLM is DeepSeek-V4-Pro, which has 1.6 trillion parameters, but only 49 billion are activated per token during inference.

What works best for Mixture-of-Experts models

1. Memory bandwidth is the real bottleneck for small batches

Consider the following Roofline plot, which shows the maximum performance a workload can achieve as a function of its arithmetic intensity (how much computation is performed per byte of data moved in memory). This plot tells where the workload is limited by memory bandwidth or by compute.

It is built using Qwen3-235B-A22B served on NVIDIA GB200 GPUs connected over NVLink.

When served in a low-latency setting, such as responding to a small number of requests, often from just one user at a time, the real bottleneck for MoE LLM inference is memory bandwidth rather than compute.

This means that when only a few tokens are being processed by the MoE model, the GPU spends almost all its time loading model parameters from memory rather than performing computations. The GPU compute units sit idle most of the time, waiting for parameters to be loaded.

Read more