👋🏻 Hey friend!

Nearly every company sets aside a learning & development budget for books, courses, conferences, and technical subscriptions, and every year a large chunk of it goes unspent.

Into AI fits perfectly into this category.

Reading lessons in AI foundations, staying up to date with the latest research, and knowing how to implement it aren't side hobbies but make you and your team better at your job.

If you've been paying for Into AI yourself, or if you've been meaning to upgrade to a paid plan, there's a good chance you don't have to spend your own money at all.

You could follow either of these two approaches when you become a paid member:

1. You’ll get a payment receipt which you can forward straight to your manager for reimbursement.

2. Use the email template below, edit the details, and send it to your manager for approval.

Email template

I also offer a 30% discount on group subscriptions for your team which you can grab below.

30% group subscription discount →

Reply to this email directly if you have any questions, and I will be happy to answer them for you! 😊

This book guides you through 30 real-world agent architectures, covering the core building blocks of perception, memory, reasoning, and planning, and teaches you LangChain and LangGraph to create agents across finance, legal, healthcare, and more.

It also helps you learn how to deploy, evaluate, and guard your agents to ensure that they perform well in production. Grab your copy today using the link below.

Build production ready AI agents →

10. Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings

This research introduces Brain2Qwerty v2, an AI model that can decode natural sentences a person is typing from their magnetoencephalography (MEG) recordings in real time.

The model has an average word error rate of 39%, and for the best participant, it can accurately decode half of the sentences with one word error or less.

The model’s accuracy also improves log-linearly with more data. This means that increasing the number of recordings could close the gap with surgically implanted brain-computer interfaces, reaching accuracy levels that were once believed to be possible only with implants.

Read more about this research using this link.

9. GPT‑5.6 Sol

OpenAI released its GPT‑5.6 series with three models:

• Sol (flagship model)

• Terra (balanced model for everyday work)

• Luna (a fast and affordable model)

Among these, GPT‑5.6 Sol is the strongest model for tough agentic work such as coding, scientific analysis, biological workflows, cybersecurity, and long-horizon tool use.

For coding workflows, GPT‑5.6 Sol sets a new state of the art on Terminal‑Bench 2.1.

It also performs competitively with Mythos Preview using only a third of the output tokens on ExploitBench.

Read more about this release using this link.

8. Sakana Fugu

This research paper introduces Sakana Fugu, a family of orchestrator LLMs trained to understand user queries and dynamically create agentic scaffolds to solve them.

The two models in the Sakana Fugu family are:

• Fugu (balances performance with latency for everyday use)

• Fugu-Ultra (prioritizes answer quality on the most difficult problems)

Using adaptive scaffolds, these models achieve performance beyond that of any individual LLM agent, and reach SOTA results compared to other publicly accessible models across a wide range of challenging benchmarks (SWE-Bench Pro, Terminal Bench, LiveCodeBench, GPQA-Diamond, Humanity’s Last Exam, and CharXiv Reasoning).

Read more about this research using this link.

7. The Red Queen Gödel Machine

This research paper presents the Red Queen Gödel Machine (RQGM), which is based on the evolutionary insight that species do not optimize against a static environment but adapt as their environments change with them.

Red Queen Gödel Machine (RQGM) is a framework for recursively self-improving agents in which both the agent and its evaluator evolve together rather than relying on a fixed benchmark.

It uses “Controlled utility evolution”, a technique that keeps evaluation constant within each epoch but allows it to change between epochs.

RQGM improves test pass rates compared to previous SOTA by adding an agent-as-a-judge code-review signal while using 1.35×-1.72× fewer tokens because the reviewer is queried only once.

With RQGM, co-evolved scientific paper writing and reviewing agents reach 1.78×–1.86× higher acceptance rates under a diverse agent-as-a-judge panel, while co-evolved Olympiad-level proof writing and grading agents reach 9% higher ground-truth accuracy.

Read more about this research using this link.

6. OCR 4

Mistral released OCR 4, which they claim to be the best OCR model to date.

Instead of just turning a document into text, OCR 4:

• Returns bounding boxes around text items

• Classifies each box into titles, tables, equations, signatures, and more

• Tells how sure it is about each extraction (inline confidence scores)

The model also:

• Supports 170 languages across 10 language groups

• Is small enough to run in a single container for fully self-hosted deployments

• Costs $2-5 per 1,000 processed pages

OCR 4 has a 72% average win rate across all leading OCR systems in blind human evaluations and achieves top scores on OlmOCRBench (85.20) and OmniDocBench (93.07).

Read more about this release using this link.

Subscribe now

5. MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

This research presents Multi-teacher On-Policy Distillation (MOPD), a post-training method that combines the capabilities of multiple domain RL-trained teacher models into one student model.

The method first RL trains separate domain-expert/teacher models. Then, it distills them into the student using the student’s own on-policy rollouts. This eliminates exposure bias and gives a denser optimization signal than off-policy finetuning.

On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, while inheriting nearly all of each teacher’s specialized capability.

It also enables parallel training of domain teachers, followed by merging their strengths into a single deployable model.

MOPD has been used in the post-training of MiMo-V2-Flash, a powerful, efficient, and ultra-fast foundation language model that particularly excels in reasoning, coding, and agentic scenarios.

Read more about this research using this link.

4. Ask, Don’t Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

This research paper presents BINEVAL, an alternative to “LLM-as-a-judge” scoring, which often produces opaque scores that are hard to debug.

Instead of asking a model to provide one overall score, BINEVAL divides each evaluation criterion into multiple simple yes/no (binary) questions.

Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output. It then scores each of these answers independently and combines them into interpretable, multi-dimensional scores.

This makes it easier to identify which checks failed, such as factual consistency, missing information, relevance, or redundancy.

This question-level feedback can also be used to iteratively improve evaluator prompts for summarization and generation.

Across multiple benchmarks such as SummEval, Topical-Chat, and QAGS, BINEVAL performs as well as or better than strong baselines such as UniEval and G-Eval, with particularly strong results on factual consistency.

Read more about this research using this link.

3. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Speculative Decoding speeds up LLM inference by using a small draft model that suggests tokens while the larger LLM checks them.

While the draft model is fast, it often loses token consistency due to a lack of inter-token dependencies. Also, indiscriminately verifying long drafts wastes compute, severely reducing throughput in high-concurrency serving systems.

This research paper proposes DSpark, a speculative decoding framework that combines high-throughput parallel generation with adaptive, load-aware verification.

DSpark uses:

• A semi-autoregressive architecture that adds a lightweight sequential dependency module to introduce intra-block dependency modeling and reduce suffix decay.

• Confidence-scheduled verification to dynamically decide how many draft tokens to verify for each request based on the chance that the prefix will survive and the current throughput profile of the serving engine

DSpark substantially improves the accepted length compared to SOTA autoregressive and parallel drafters across multiple offline benchmarks.

When deployed within the DeepSeek-V4 serving system under live user traffic, DSpark successfully reduces verification waste. It improves per-user generation speeds by 60-85% at matched throughput levels compared to the established production baseline (MTP-1).

By preventing severe throughput degradation under strict interactivity constraints, it also enables performance tiers that were previously unattainable, shifting the Pareto frontier of the DeepSeek-V4 serving system.

Read more about this research using this link.

Subscribe now

2. Claude Sonnet 5

Anthropic released Claude Sonnet 5, their most agentic Sonnet model yet. The model is well suited for coding, tool use, browser and terminal workflows, long-running agentic workflows, and professional knowledge work.

The model substantially outperforms Sonnet 4.6 across multiple domains, and its performance is close to that of Claude Opus 4.8 on many benchmarks, but at a lower price.

It also has lower rates of hallucination, sycophancy, and unwanted behaviors compared to Sonnet 4.6, has better resistance to prompt injection, and comes with cyber safeguards that detect and block dangerous cyber activity in real time.

Read more about this release using this link.

1. Reaching Trillion-Parameter Performance with a 35B Agent

This research introduces Agents-A1, a 35B Mixture-of-Experts agentic model that achieves performance comparable to trillion-parameter models by expanding the agent horizon rather than model size/ parameters.

The authors build a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes to produce agentic trajectories averaging 45K tokens.

Based on these, Agents-A1 is trained in three stages:

• Full-domain supervised fine-tuning to align the base model with broad agentic behaviors

• Domain-level teacher models to capture specialized skills in each domain

• Multi-teacher domain-routed on-policy distillation to improve knowledge transfer efficiency across different domains and combine six different domains into a single deployable student model

The resulting model achieves strong performance on long-horizon agent benchmarks compared with 1T-parameter models such as Kimi-K2.6 and DeepSeek-V4-Pro.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Are you struggling to keep up with LLMs and developments around them? I wrote this book to exactly solve this for you!

“LLMs in 100 Images” turns the difficult parts of modern LLM systems (Attention variants, prompting techniques, sampling techniques, post-training algorithms, and more) into visuals, making them simple to understand.

I’m running a flash sale on the book for a very limited time, and you can grab your copy today at a 30% discount!

Grab your 30% discount →

It’s now time for the lesson.

GPUs are driving the current AI revolution, and understanding them well will make you invaluable in the AI ecosystem. In this lesson, we will learn about NVIDIA GPUs and the interconnects that scale them to massive data centers used for training and serving LLMs.

Let’s begin!

But first, what is a GPU and why is it needed?

A GPU, or Graphics Processing Unit, is a specialized chip originally designed to render 3D graphics rapidly and efficiently.

But it soon became clear that the same mathematical operations (matrix addition and multiplication) used in graphics could also be used to train and serve AI. This made NVIDIA, a company that started as a chipmaker for video games, the leading company building GPUs for AI today.

GPUs are quite different from CPUs, which are the chips used for general processing in the computer. While CPUs contain a few powerful cores that are perfect for executing tasks with sequential and branching logic at very low latency, GPUs are the masters of parallel computing.

They have thousands of cores, each slower than a CPU core, but together they produce massive throughput for parallel computations, especially matrix or tensor operations. (A matrix is a 2D tensor).

A GPU has units called Streaming Multiprocessors (SMs), where calculations actually take place. Each SM contains smaller specialized components called:

• CUDA cores: that perform fast general mathematical operations

• Tensor cores: that perform fast matrix operations

Alongside this, a GPU has two types of memory:

1. On-chip memory (L2 cache, L1 cache, and registers) that is physically etched on the GPU die

2. High Bandwidth Memory (HBM), generally called global memory or VRAM, that is mounted alongside the die

HBM is a specialized type of Dynamic random-access memory (DRAM) designed for massive parallel data throughput.

On-chip memory components are Static random-access memory (SRAM), which is extremely fast but much smaller and more expensive than the GPU HBM.

If you’re interested in reading about how data flows through CPU and GPU during LLM inference, here is a lesson that will help:

The NVIDIA ecosystem of GPUs

NVIDIA GPUs come in different architectures, each named after a famous scientist or mathematician.

The company produces consumer-grade GPUs for graphics and gaming in its GeForce GTX (older) and RTX series. It also produces the RTX PRO (previously known as Quadro) series for commercial/ scientific/ creative workloads on workstations.

We won’t be discussing these GPU architectures in this lesson, but we'll focus more on data center-grade GPUs designed for deep learning/AI.

In 2017, NVIDIA introduced Tensor Cores and FP16 mixed-precision training for deep learning in its Volta series of GPUs (Tesla V100). OpenAI used a cluster of these GPUs to train GPT-3.

Since then, NVIDIA has made its GPUs more performant, and some popular architectures with their flagship GPU models released are:

• Turing series in 2018 (Tesla T4): Optimized for lower-precision integer (INT8/INT4) inference pipelines

• Ampere series in 2020 (A100): Used to train early LLMs and serve many GPT-3-scale models.

• Hopper series in 2022 (H100, H200): Introduced the Transformer Engine and FP8 precision, which were used in the training of Llama 3 and similar models

• Ada Lovelace series in 2022 (L40, L40S): Designed for high-performance AI inference and graphics rendering tasks

• Blackwell series in 2024 (B100, B200): Built for training and serving trillion-parameter reasoning models with support for FP4 (4-bit floating-point) precision

• Blackwell Ultra series in 2025 (B300): Improved Blackwell generation with more memory and higher performance (~50% higher FP4 compute)

• Rubin series in 2026 (Rubin GPU): Designed to provide roughly double the FP4 compute (50 vs 20 PFLOPS) and GPU-to-GPU bandwidth (3.6 TB/s vs 1.8 TB/s) compared to the Blackwell series GPUs.

• Rubin Ultra (announced to be released in 2027)

• Feynman (announced to be released in 2028)

Subscribe now

Understanding inter-GPU connections

A popular GPU like H100 from the Hopper series has 80GB of HBM. A single such GPU is too small to fit the weights of today's trillion-parameter large reasoning models, which have hundreds of GBs of parameters. This is why multiple GPUs are connected to form a large server (also called a Node). This approach is called Vertical scaling or the “scaling-up” approach.

There are different ways GPUs can be connected in a server:

• PCIe (Peripheral Component Interconnect Express): This is the general-purpose connection used inside servers to connect GPUs, network cards, SSDs, and other devices to the CPU. The 6th generation PCIe offers 256 GB/s of GPU-to-GPU bandwidth. PCIe makes GPU-to-GPU data take a slower path through the CPU's root complex, which increases latency and reduces bandwidth.

• NVLink: This is NVIDIA’s proprietary communication channel that gives a dedicated high-speed data path between GPUs. The bidirectional bandwidth between two GPUs provided by different generations of NVLinks is as follows:

  • 900 GB/s on H100 (NVLink 4)

  • 1.8 TB/s on B200 (NVLink 5)

  • 3.6 TB/s on Rubin (NVLink 6)

This is massive compared to the 256 GB/s offered by the 6th-generation PCIe (a 14× increase with NVLink 6)!

NVLink has an issue, though, in that it splits each GPU’s total bandwidth among the GPUs it connects. Check out the following example, where each Rubin GPU splits its 3.6 TB/s bandwidth among the other three, resulting in 1.2 TB/s per connection.

The general formula for the effective inter-GPU bandwidth with NVLink is B/N, where

• B is a GPU’s total NVLink bandwidth

• N is the number of GPUs it is connected to

The solution in this case is an NVSwitch.

• NVSwitch: This is a high-bandwidth, low-latency fabric that connects multiple GPUs within a system, ensuring GPUs can communicate with one another at full bandwidth simultaneously, regardless of how many are connected. This lets all the GPUs work together and act like a single massive GPU in a server.
  
  It must be noted that NVSwitches are more expensive than other connections and might be unnecessary if you’re working with smaller (a few billion parameters) LLMs.

GPUs aren’t all that we need

While GPUs are used for fast parallel operations during LLM training and inference, CPUs are needed to:

• preprocess and load/ unload data into GPUs

• orchestrate jobs between GPUs

• manage storage and networking

• run other serial/ non-parallel operations

If the CPUs used in the process are too slow, the expensive GPUs would sit idle most of the time. This is why NVIDIA has launched its own Arm-based CPU models, namely:

• Grace

• Vera

These are used alongside GPUs and connected to them using NVLink-C2C (C2C stands for chip-to-chip). This is a superfast CPU-to-GPU connection that replaces the slower PCIe connection and provides both the GPU and CPU with a unified memory space, allowing them to access each other’s memory directly without manual copying.

NVIDIA combines its GPUs and CPUs together in a server rack, with three popular ones being:

• GB200 NVL72: Combines 72 Blackwell GPUs and 36 Grace CPUs

• GB300 NVL72: Combines 72 Blackwell Ultra GPUs and 36 Grace CPUs

• Vera Rubin NVL72: Combines 72 Rubin GPUs and 36 Vera CPUs

(GB stands for Grace-Blackwell, and NVL tells how many GPUs are connected using NVLink/NVSwitch.)

But these racks aren’t the only way NVIDIA packages these components. NVIDIA also makes multi-GPU (no CPU included) baseboards in its HGX series, with the popular ones being:

• HGX A100: Combines 4, 8, or 16 A100 GPUs

• HGX H100: Combines 4 or 8 H100 GPUs

• HGX Rubin NVL8: Combines 8 Rubin GPUs

In the DGX series, NVIDIA combines GPUs, CPUs, memory, networking, storage, cooling, and software into a complete AI system (for example, the DGX H100 server).

Its MGX series is a modular server architecture in which the above-described components are swappable, letting server makers to combine different options to build custom server configurations.

Connecting servers & scaling horizontally

There’s a limit to how much we can vertically scale GPUs. This is due to the chip's physical constraints, as well as power and cooling requirements. In such a case, multiple servers are connected in an approach called Horizontal scaling or the “scaling-out” approach.

Although data transfer speed is lower than with vertically scaled GPUs, the benefit of horizontal scaling is that one can theoretically connect any number of GPUs together.

NVIDIA offers two high-throughput connections for horizontal scaling, both of which use GPUDirect RDMA (Remote Direct Memory Access), which lets GPUs exchange data directly from memory while bypassing the CPU and OS.

1. Spectrum-X Ethernet: This is the standard AI-tuned Ethernet connection in a data center that offers high performance in AI workflows. It uses RoCE (RDMA over Converged Ethernet), a protocol that helps run RDMA over an Ethernet network.

2. Quantum InfiniBand: This is a specialized connection that delivers ultra-low latency for high-end LLM training data centers. It uses RDMA natively without using RoCE.

Both have a throughput of about 100 GB/s per connection, which is far lower than that of NVLink with NVSwitch (3.6 TB/s per connection). This means that workloads that require super-fast data transfer must be kept between vertically scaled GPUs, while the others could be directed to GPUs shared across servers.

For example, during training, Tensor parallelism (TP) is implemented within GPUs on a single server. On the other hand, Data parallelism (DP) and Pipeline parallelism (PP) are implemented across servers as they exchange data less often.

NVIDIA Collective Communications Library (NCCL) handles and coordinates data exchange between GPUs, automatically routing each workload to the fastest available link.

If you’re new to the terms TP, DP, and PP, the following lesson would help:

The hierarchy of GPU connections

Revisiting what we discussed before, GPUs are arranged hierarchically in large data centers as follows:

• Server or Node (4 to 8 GPUs)

• Server rack (10s to 100s GPUs)

• Cluster or Pod (100s to 1000s GPUs)

• Data center (10,000 to 100,000 GPUs)

An example of a massive data center is Colossus, developed by xAI primarily for training Grok. Today, this data center also powers LLMs and research labs at Anthropic, Google, and Reflection AI. Colossus consists of 200,000 H100 GPUs and is intended to be scaled to 1 million GPUs (including H100, H200, and GB200).

TL;DR

To summarise:

• GPUs, initially designed for graphics rendering, are now the backbone of the current AI ecosystem, thanks to their ability to perform large numbers of matrix operations in parallel.

• While CPUs have few cores that can perform sequential work faster than GPUs, GPUs have thousands of slower cores that, overall, deliver massive parallel throughput.

• A GPU is made up of Streaming Multiprocessors (SMs) containing CUDA cores (for general math operations) and Tensor cores (for matrix math operations).

• A GPU has smaller but faster on-chip SRAM (L1 cache/L2 cache/registers) and large off-chip HBM/VRAM, which is a type of DRAM.

• The most commonly used NVIDIA GPU series today are Ampere, Hopper, Blackwell, and Rubin.

• GPUs are connected in a single server using PCIe, NVLink, or NVSwitch. This approach is called Vertical scaling.

• NVIDIA pairs its GPUs with its own Arm-based CPUs (Grace and Vera) using NVLink-C2C, a high-speed connection that provides both chips with a unified memory space.

• Multiple servers are connected together using Spectrum-X Ethernet or Quantum InfiniBand to build massive data centers. This approach is called Horizontal scaling.

This article is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Also, don’t forget to grab your copy of “LLMs In 100 Images” at a 30% discount! ✨

Grab your 30% discount →

While most developers can spin up a RAG pipeline in an afternoon using LangChain or LlamaIndex, very few understand its internals well and know how to fix it when things go wrong.

‘RAG from First Principles’ takes the RAG stack apart layer by layer and teaches you about ingestion, chunking, embeddings, vector indexes, hybrid search, reranking, and evaluation.

Each chapter answers the questions practitioners actually hit in production, building from data import all the way to GraphRAG, Agentic RAG, and Modular RAG.

By the end, you’ll be able to optimize, debug, and extend your RAG systems with confidence and not guesswork.

Master RAG today →

10. ENPIRE: Agentic Robot Policy Self-Improvement in the Real World

This research from NVIDIA introduces ENPIRE, a framework for coding agents that enables them to autonomously improve robot policies in real-world settings.

ENPIRE has four core modules:

1. Environment module (EN) that automatically resets the scene and checks whether a task succeeded

2. Policy Improvement module (PI) that launches policy refinement

3. Rollout module (R) that evaluates policies with single or multiple physical robots operating in parallel

4. Evolution module (E) that lets coding agents analyze logs, consult literature, improve training infrastructure, and algorithm code to fix failures

Using ENPIRE, frontier coding agents can autonomously build a policy that achieves a 99% success rate on challenging dexterous manipulation tasks such as PushT, organizing pins into a pin box, and using a cutter to cut a zip tie.

Read more about this research using this link.

Subscribe now

9. Looped World Models

World models today have significant computational requirements to run faithful long-horizon simulations. This makes them expensive to deploy and prone to compounding errors.

This research addresses this by introducing Looped World Models (LoopWM), which use looped, parameter-shared transformer blocks to refine latent environment states through repeated internal iterations, rather than adding multiple separate layers.

This leads to ~100× parameter efficiency over conventional approaches with adaptive computation that automatically scales depth to match the complexity of each prediction step.

LoopWM introduces iterative latent depth as a new scaling dimension for world simulation, rather than increasing model size or training data.

Read more about this research using this link.

8. Surpassing Frontier Performance with Fusion

This research from OpenRouter introduces Fusion, an approach that allows selecting a panel of participant models alongside a judge model. It then sends a prompt to multiple participant models in parallel and uses the judge model to combine their answers into a single stronger response.

The results show that:

1. Panels of models consistently outperform individual models

2. Frontier panels can achieve beyond-frontier performance

3. Panels of budget models can beat frontier models and get close to frontier panel performance

Two notable examples from the results are:

• Fable 5 + GPT-5.5 scores 69% on OpenRouter’s DRACO deep-research benchmark, while Fable 5 alone scores 65.3% on this benchmark.

• A budget panel of Gemini 3 Flash, Kimi K2.6, and DeepSeek V4 Pro outperforms GPT-5.5 and Opus 4.8. It also scores within 1% of Fable 5’s score while costing half as much.

Although highly performant, it must be noted that this method is slower, costlier, and not a drop-in replacement for coding or long-horizon agents.

Read more about this method using this link.

7. Next-Latent Prediction Transformers Learn Compact World Models

This research paper introduces NextLatent Prediction (NextLat), which adds a self-supervised next-latent prediction loss to transformers, helping them learn latent representations that predict the next latent state given the next token.

These latest representations form “belief states”, which are compressed information about the history necessary to predict the future (compact internal world models).

Across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat leads to significant gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning.

It also enables variable-length self-speculative decoding, improving inference by up to 3.3× in language modeling.

Read more about this research paper using this link.

6. HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining

This research paper presents a systematic study that compares egocentric human video with teleoperated real-robot trajectories as pretraining data for embodied robot foundation models.

Human egocentric data is not only scalable, substantially lower-cost, and more diverse than teleoperated real-robot data, but under fixed pretraining, post-training, and validation protocols, it also leads to superior performance.

With the same amount of pretraining data, models pretrained on egocentric data achieve:

• 24% lower validation loss on real-robot action prediction,

• 52.5% higher success rates on in-distribution real-robot task execution

• 90% higher success rates on out-of-distribution real-robot task execution

This suggests that the best approach to training an embodied foundation model is to pretrain on egocentric human video to learn diverse world representations, then adapt using a small amount of labeled real-robot data for action-space alignment.

Read more about this research using this link.

5. Variable-Width Transformers

Standard Transformers use the same width for all layers. This means that each layer has the same number of parameters and compute budget, even though they might have different functions and computational needs in language modeling.

This research paper from MIT changes this by introducing an X-shaped Transformer called the “>< former”, in which the early and late layers remain wide, while the middle layers are narrower.

This approach works surprisingly well and outperforms parameter-matched standard Transformers (ranging from 200M to 3B parameters) in terms of language modeling loss and on most downstream tasks.

This architecture also requires fewer overall FLOPs (a 22% reduction) and smaller KV cache memory and I/O costs (a 15% reduction).

Read more about this research using this link.

4. Kimi K2.7 Code

Moonshot AI released Kimi K2.7 Code, its new open-source agentic model that is optimized for long-horizon software engineering tasks.

It is a 1T-parameter MoE model with 32B active parameters and uses MLA attention and the MoonViT vision encoder.

It improves over K2.6 on multiple coding and agent benchmarks, has better instruction-following capabilities in long contexts, uses about 30% fewer thinking tokens, and supports a 256K context window.

Alongside this, its performance is close to that of GPT-5.5 and Claude Opus 4.8 on many benchmarks while it is roughly 5-7x cheaper to run.

Read more about this release using this link.

3. VibeThinker-3B

This research paper from Weibo introduces VibeThinker-3B, a 3B-parameter dense reasoning model that achieves performance comparable to that of large frontier models on math and coding tasks.

The model is post-trained using Qwen2.5-Coder-3B as the base model, using several techniques such as:

• Curriculum-based two-stage SFT: Training first on broad reasoning/dialogue data, then progressively harder long-horizon math/code/STEM examples

• Multi-domain RLVR using MGPO: Training across math, coding, and STEM examples using the MGPO algorithm. MaxEnt-Guided Policy Optimization (MGPO) helps the model explore different reasoning paths near its current capability, then amplifies the paths that produce verifiably correct answers.

• Offline Self-Distillation: Collecting the best reasoning traces from the model’s own RL-specialized checkpoints and distilling them back into a single 3B model

• Instruct RL: Training the model to follow user instructions and stick to output formats and user constraints reliably without losing reasoning gains

The model scores 94.3% on AIME26 (97.1% with Claim-level Reliability), 80.2% Pass@1 on LiveCodeBench v6, and achieves a 96.1% acceptance rate on recent unseen LeetCode contests.

Read more about this research using this link.

2. GLM-5.2

Z.ai introduced GLM-5.2, its open MIT-licensed flagship model for long-horizon coding and agent tasks.

The model:

• Has a 1M-token context window

• Has stronger coding capabilities with multiple thinking effort levels to balance performance and latency

• Uses IndexShare, which reuses the same indexer across every four sparse attention layers, reducing per-token FLOPs by 2.9× at a 1M context length.

• MTP layer is also better suited for speculative decoding, increasing the acceptance length by up to 20%.

GLM-5.2 is the strongest open model on several coding and agentic benchmarks, with performance close to frontier models on long-horizon tasks.

Read more about this release using this link.

1. Towards autonomous medical artificial intelligence agents

This research paper introduces MIRA (Medical Intelligence for Reasoning and Action), an autonomous AI agent capable of operating within a sandboxed EHR (Electronic Health Record) environment.

It can:

• Take histories

• Order and interpret laboratory, imaging, and microbiology tests

• Generate differential diagnoses

• Formulate treatment plans, including prescribing medications, scheduling surgical procedures, and planning admissions.

On 574 real MIMIC-IV cases across 8 diseases, MIRA achieved 88.9% diagnostic accuracy and, in head-to-head testing, outperformed board-certified physicians (87.8% vs 78.1% accuracy), while showing strong medication safety and alignment with medical guidelines.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Building Your First AI Agent

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Join 'Into AI' premium today

Your agent just ran for an hour. It changed multiple files, called multiple tools, followed leads, hit dead ends, made decisions, and maybe touched something you wish it hadn’t. Most of this is invisible or too big for you to go through.

Spotlight by Backplanes turns your Claude Code and Codex sessions into valuable reports, so you can understand the agent run without digging through logs.

Spotlight is free for individual developers and the teams they work with (no credit card required). They also remove sensitive info, encrypt your data, use providers that do not store it, never sell it, and delete it when you delete your sessions, projects, or account.

Btw, I personally used it and, embarrassingly, found out that Claude Code was reading my API keys during my coding sessions.

Try Spotlight today →

LLM Inference is the process of running a forward pass through a trained model to produce text. Understanding what happens at inference time at the CPU/GPU level will help you optimize this process more effectively. Here is a lesson where we discuss exactly this.

The process starts with loading parameters into the CPU memory

To begin with, a trained model’s parameters are stored in the hard drive (preferably an NVMe SSD) and can have different formats such as:

• Safetensors

• bin or pt when working with PyTorch

• GGUF

These come alongside a config.json file that tells about the model architecture, hyperparameters, and data type.

(Check out the config.json and model parameter files for the DeepSeek-V4-Flash model to understand this better.)

A model loader uses this file to build the model skeleton and then loads the parameters into the CPU memory (also called System memory). This memory is typically Dynamic random-access memory (DRAM), which offers larger capacity and is cheaper than the GPU memory.

The model parameters are next transferred to the GPU's HBM (High Bandwidth Memory), also known as global memory (or, generally, VRAM). This transfer takes place over PCIe, a high-speed connection between the CPU and GPU.

Seen tensor.to('cuda') method while training models? This is what happens under the hood when you call this method.

HBM is a specialized type of DRAM designed for massive parallel data throughput. While CPU memory has a throughput of 50 to a few hundred GB/s, HBM can deliver a throughput of a few TB/s. Although fast, HBM is smaller and much more expensive than CPU memory.

What if the LLM is too big for the GPU memory?

Let’s go back to the config.json file for DeepSeek-V4-Flash, which is a Mixture-of-Experts model with 284 billion total parameters.

If each parameter of this model is loaded in FP8 (8-bit floating-point) precision, which means that 1 parameter is represented in 8 bits or 1 byte, the total memory requirement of this model for parameter storage will be:

284B parameters × 1 byte/ parameter = 284 GB

But this is not the only thing that the GPU HBM needs to store. We also need space for KV cache, activations, and other overheads.

The popularly used NVIDIA H100 GPU comes with 80 GB of HBM. This is nowhere near enough to store 284 GB of parameters, let alone the KV cache and others. This means that the parameters must be distributed across multiple GPUs.

A standard architecture for hosting a model is an 8-GPU NVIDIA HGX H100 server. This server has 8 H100 GPUs, each with 80 GB of HBM, which sums to a total of 640 GB of memory.

The GPUs in the server are linked using an NVSwitch, which provides each GPU with a full-bandwidth path to every other GPU, enabling them to transfer data fast enough (900GB/s) to behave as a single large accelerator.

The term “full-bandwidth” is important here because if no NVSwitch is used with NVLinks, the bandwidth is split between GPU pairs. In our 8-GPU setup, any single GPU-to-GPU pair would get only about 128 GB/s of bandwidth, compared to 900 GB/s (full bandwidth) with NVSwitch.

There are many techniques that are used to distribute the inference load across these GPUs. Some techniques split the model itself, while others split the input workload.

These are described as follows:

• Tensor parallelism (TP): Splitting parameter/ weight matrices across GPUs, with each GPU performing part of the matrix multiplication and then syncing the results.

• Pipeline parallelism (PP): Dividing the model’s layers across GPUs, with each GPU handling a consecutive block of layers and passing the activations to the next.

• Context parallelism (CP): Splitting a single sequence and its KV cache across GPUs, with each GPU managing a part of the context to support very long sequence lengths.

• Expert parallelism (EP): Distributing the experts of a Mixture-of-Experts (MoE) model across GPUs and routing the tokens to the GPU that holds their chosen experts.

• Data parallelism (DP): Replicating the entire model on each GPU when it is small enough to fit, or replicating a fully sharded model across multiple GPU groups when the model is too large for a single GPU. Each replica then handles different user requests.

• Hybrid parallelism: Combining several of these techniques discussed above. For example, TP, PP, CP, and EP are combined to create a complete sharded model replica. DP is then added by creating multiple such replicas, each serving different user requests in parallel.

If you’re completely new to these techniques, the following lesson will help.

What's next after loading parameters into GPU memory?

Once the parameters are loaded into GPU HBM, they are ready to process incoming user requests.

Let’s talk a bit about the main components of a GPU like the H100. These are:

• Streaming Multiprocessors (SMs)

• High-bandwidth memory (HBM)

• On-chip memory

Streaming Multiprocessors, or SMs, are the processing units where calculations happen in a GPU. They contain hundreds of smaller components called:

• CUDA cores: that perform fast general mathematical operations

• Tensor cores: that perform fast matrix operations (matrix multiplications)

The on-chip memory is etched directly on the GPU die, unlike HBM, which is mounted alongside the die. This memory further consists of (arranged in ascending order of speed and descending order of capacity):

• L2 memory/cache

• L1 or shared memory/cache (shared across components of an SM)

• Registers

On-chip memory components are Static random-access memory (SRAM), which is extremely fast but much smaller than other types of DRAM (CPU memory and GPU HBM) we discussed earlier.

For every user request/prompt, the model performs forward passes through its parameters in two phases: Prefill and Decode.

In each forward pass, every layer’s parameters are streamed from HBM to on-chip memory and then to the CUDA/Tensor cores, where the calculations actually occur. The weights are discarded once a token is produced, and this process repeats until the full sequence is generated or the maximum token-generation limit is reached.

If you want to better understand what calculations occur at the Transformer level, we have covered this in the previous lessons that you can find here:

What is Prefill and Decode?

Prefill is the first phase where all tokens in a user’s prompt are processed together in a single forward pass through the LLM.

During this phase, each token computes a key (K) and value (V) vector, which are cached to build a KV cache for the entire initial user prompt. This KV cache is stored alongside the model parameters and intermediate activations in the HBM.

Prefill relies heavily on the GPU's tensor cores and is compute-bound. This is because all tokens of the initial user prompt are being processed in parallel.

A faster prefill means a shorter Time to First Token (TTFT), which is the delay before the model begins responding to the user's prompt.

Next comes Decode, the second phase, where the LLM generates one token at a time, with a single forward pass through the model for each token.

Instead of recomputing the key (K) and value (V) vectors for every previous token at each step, the model reuses these from the KV cache and only computes them for the new token. The newly computed K and V values are added to the KV cache for use in the next step of Decode.

Decode is memory-bound and does not utilize the full capacity of the GPU’s Tensor cores. This is because, for every generated token, all model parameters must be streamed from HBM to the Tensor cores for a small amount of computation, discarded when the token is generated, and then re-read in full for the next token.

In this process, the calculations performed per token relative to the volume of weights moved are so small that the cores spend most of their time waiting on memory rather than computing.

Why is Decode memory-bound?

Let’s understand this better using an example.

Let’s say that we are using a 30B Dense model in 16-bit precision on a single H100 (with 80 GB of HBM). This is 60 GB of parameters (30B × 2 bytes).

To produce one token, all 60 GB are streamed from HBM to the Tensor cores. At the higher end of the H100’s memory bandwidth of 3.35 TB/s, it takes about 18 ms to generate one token. This gives a throughput of around 56 tokens per second.

Now let’s compare this with the computation involved.

A forward pass costs roughly 2 FLOPs per parameter per token (one multiply and one add operation), so the computational cost of processing a token through a 30B dense model is 60 GFLOPs (2 FLOPs × 30B).

The H100's Tensor cores can perform 990 TFLOP/s of FP16 operations. This means that 60 GFLOPs of operations will take around 0.06 ms (60 / 990,000).

Taken together, the computation takes roughly 0.06 ms, while streaming the 60 GB of weights takes 18 ms.

The cores only compute for 0.3% of the time required for the memory transfer. For the rest, they sit idle, waiting for the next set of parameters to arrive, making Decode memory-bound.

Over the past 20 years, peak server hardware FLOPS has scaled by a factor of 3 every 2 years, outpacing the growth of DRAM and interconnect bandwidth, which have scaled by factors of 1.6× and 1.4× every 2 years, respectively. This has made memory (and not compute) the primary bottleneck in LLM inference. (Source)

Why is the KV Cache so important for Decode?

The KV cache ensures that the keys (K) and values (V) for past tokens are computed once and stored, so each Decode step computes only one new token's K and V and reads the rest from cache.

This slows the increase in the cost of generating a single token as the sequence length grows, making Decode faster and more efficient.

You must remember that using a KV cache is not always a “free lunch” since it also takes space in the HBM. As its size grows (with increasing context length and the number of batched requests), it could consume as much memory as the parameters themselves. Managing its size is something that one needs to keep in mind when serving LLMs.

TL;DR

To summarise:

• During inference, model parameters are moved from SSD to CPU memory, GPU HBM, on-chip memory, and finally to the computation cores.

• Models that are too large for a single GPU are split across multiple GPUs using multiple parallelism techniques.

• The multi-GPU server uses NVSwitch to connect the GPUs, resulting in full communication bandwidth between them.

• Each inference request is processed in two phases: Prefill and Decode.

• Prefill processes all prompt tokens at once. It is a compute-bound process, and the Time to First Token (TTFT) depends on it.

• Decode generates one token at a time. It is memory-bound and acts as the bottleneck of inference. This is because each token generation involves moving a large number of parameters while performing very little computation, leaving the GPU's computation cores mostly idle while waiting on memory.

• The KV cache helps keep token generation costs from growing rapidly as sequence length increases.

• KV cache also consumes HBM alongside model parameters and must be managed for efficient LLM serving.

✨ Courtesy of Backplanes, this newsletter edition is completely free to read. ✨

Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

10. DiffusionGemma

Google released DiffusionGemma, an experimental open model that uses Diffusion to generate entire blocks of text simultaneously, leading to 4x faster text generation on dedicated GPUs.

DiffusionGemma is a 26B Mixture-of-Experts (MoE) model that activates only 3.8 billion parameters during inference. It uses bidirectional attention to generate 256 tokens in parallel, with each forward pass allowing every token to attend to all others.

The model comes with native support for NVIDIA’s new NVFP4 (4-bit floating-point) format on Blackwell GPUs, which dramatically increases compute throughput, enabling it to run at faster speeds with near-lossless accuracy.

The model’s impressive capabilities make it particularly helpful for speed-critical local workflows such as inline editing, rapid iteration, code infilling, non-linear text structures, amino acid sequences, mathematical graphs, and tasks like Sudoku.

However, it must also be noted that the standard Gemma 4 still produces higher-quality outputs than DiffusionGemma.

Read more about this release using this link.

If you’re new to Diffusion LLMs, you can read more about them using the following lessons.

9. Frontier Code

Cognition introduced FrontierCode, a benchmark to test whether AI coding agents can produce production-quality, mergeable code rather than just passing tests.

Built with over 20 open-source maintainers across 36 major repositories, it includes 150 tasks divided into three tiers: Extended, Main, and the hardest Diamond tier.

These tasks are graded on correctness, regression safety, test quality, scope control, style, and codebase conventions. Its evaluation combines unit tests, rubrics, command checks, LLM review, and Cognition’s “mutagent” (an LLM-based tool to surgically patch the test environment/ application code and align with the agent’s implementation details), all with strong quality control.

Cognition claims that FrontierCode has an 81% lower false-positive rate than SWE-Bench Pro and that it is the first-ever benchmark measuring code quality and subtle human preferences.

Claude Opus 4.8 leads the benchmark with a score of 13.4%, GPT-5.5 scores 6.3% while using fewer tokens, and Kimi K2.6 is the top open-source model with a 3.8% score, all on the Diamond tier.

Read more about this release using this link.

Before we move forward, I want to introduce you to my book called ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master LLMs today.

Grab your copy today at a special discount using this link.

8. Claude Fable 5 & Mythos 5

Anthropic released Claude Fable 5, a public Mythos-class model, and Claude Mythos 5, the same underlying model with safeguards lifted for trusted cyberdefenders and infrastructure providers.

This model outperforms all of Anthropic’s previous models on nearly all tested benchmarks and is especially strong on long and complex tasks.

It also has major cybersecurity and scientific research capabilities, including faster drug-design workflows and the generation of novel biological hypotheses.

The publicly released Fable 5 includes safeguards for cybersecurity, biology/chemistry, and distillation-related requests and routes these to the previous model, Claude Opus 4.8.

As of 12th June 2026, access to Claude Fable 5 and Claude Mythos 5 has been suspended due to national security concerns raised by the US government.

Read more about this release using this link.

7. FlashMemory-DeepSeek-V4

This research paper introduces Lookahead Sparse Attention (LSA), which makes ultra-long-context LLM inference more memory-efficient.

Instead of passively attending to all previous tokens, LSA proactively predicts which parts of the context it'll need in the future and keeps only those KV chunks in GPU memory.

It uses a small "Neural Memory Indexer" built on the DeepSeek-V4 architecture to decide which parts of the context are worth keeping.

Across long-context benchmarks (LongBench-v2, LongMemEval, and RULER), this uses just 13.5% of the memory on average while maintaining the same accuracy.

And at 500K-token lengths, it reduces memory overhead by more than 90% without compromising the model's reasoning ability.

Read more about this research using this link.

Subscribe now

6. Latent Spatial Memory for Video World Models

This research paper introduces Mirage, a latent-space spatial memory framework for making generated videos more 3D-consistent over long camera trajectories.

Rather than storing scene memory as an RGB point cloud that requires repeated rendering and re-encoding, Mirage stores static scene information as 3D latent tokens within the diffusion model’s latent space.

It builds this memory by lifting latent tokens into 3D with depth-guided back-projection, reads it through direct latent-space warping, and updates it chunk by chunk while filtering dynamic objects.

Experiments show that Mirage achieves up to 10.57× faster end-to-end video generation and a 55× reduction in memory footprint relative to explicit 3D baselines. It also attains SOTA performance on WorldScore and strong reconstruction quality on RealEstate10K.

Read more about this research using this link.

5. MiniMax Sparse Attention (MSA)

This research paper introduces MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA).

Traditional attention has a cost that grows quadratically with the number of tokens it attends to. This becomes far too expensive for LLMs with ultra-long contexts.

MSA uses a lightweight “Index Branch” to score KV blocks and select a Top-k subset for each query group, while a “Main Branch” performs exact sparse attention only over those selected blocks.

On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4× at 1M context. With an efficient kernel, it also achieves 14.2× prefill and 7.6× decoding speedups on the H800 GPU.

Read more about this research using this link.

4. First Steps Toward Automated AI Research

This blog post from Recursive introduces their automated AI research system, which achieves SOTA results across three benchmarks (fixed-budget language model training, small-model training speed, and GPU kernel optimization).

The system runs as an end-to-end research loop that includes proposing ideas, implementing them, running experiments, validating results, and using what it learns to choose future experiments.

On NanoChat Autoresearch, it improved fixed-budget language model training from 0.9372 to 0.9109 validation BPB.

On NanoGPT Speedrun, it reduced training time from 79.7s to 77.5s.

And on SOL-ExecBench, it raised the mean GPU-kernel optimization score from 0.699 to 0.754, which is an 18% reduction in the gap to the estimated hardware optimum.

Read more about this research using this link.

3. End-to-End Context Compression at Scale

This research paper introduces Latent Context Language Models (LCLMs), a family of encoder-decoder compressors that improve the efficiency of long-context LLM inference compared to using full KV caches.

LCLMs compress long token sequences into shorter latent embeddings, which the decoder can use directly. This improves the trade-off among task performance, compression speed, and peak memory usage while avoiding common limitations of KV compression (quality loss, high compression costs, constraints on target context windows, and poor compatibility with production inference systems).

Read more about this research paper using this link.

Subscribe now

2. General-purpose large language models outperform specialized clinical AI tools on medical benchmarks

This research paper evaluates two clinical AI tools, OpenEvidence and UpToDate Expert AI, against three frontier LLMs (GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6) across:

1. 500 MedQA questions testing medical knowledge

2. 500 HealthBench items measuring alignment with clinicians

3. Real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment

The results show that frontier LLMs outperform clinical AI tools in all three evaluations.

On the RCQ benchmark, Clinical AI tools perform comparably to the auto-enabled Google Search AI Overview.

Read more about this research paper using this link.

1. Self-Harness: Harnesses That Improve Themselves

This research paper introduces Self-Harness, which lets LLM-based agents improve their own operating harness without relying on human engineers or stronger external agents.

Self-Harness works iteratively in three stages:

1. It first checks the execution traces and finds which model-specific patterns led to failure (Weakness Mining)

2. It then generates harness modifications tied to these failures (Harness Proposal)

3. Finally, it accepts the modifications after successful regression testing (Proposal Validation)

When tested on Terminal-Bench-2.0 using three models (MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5) initialized with a minimal harness, Self-Harness consistently improves held-out pass rates by 21.4%, 14.3%, and 14.2%, respectively.

Read more about this research paper using this link.

This newsletter edition is completely free to read.

If you found it valuable, click the like button ❤️ and consider subscribing for more such content every week.

If you have any questions or suggestions, feel free to leave a comment below.

Into AI is a reader-supported newsletter. Gain access to deeper, members-only content by becoming a paid subscriber today.

Join 'Into AI' premium today

We then deepened our understanding by building and training a Mixture-of-Experts (MoE) LLM from scratch.

Both these models were trained using the next-token prediction objective and generate tokens one at a time, left to right, autoregressively.

But this is not the only way that a model can be used to generate text.

We have Diffusion LLMs that can generate tokens in parallel using a process called Diffusion. One of the most successful examples of this type of LLM is LLaDA (Large Language Diffusion with mAsking), which we discussed in depth in the following lesson.

Google also released its experimental open-source model, DiffusionGemma, this week, which operates on the same principle.

In this lesson, we will take our understanding to the next level and learn to:

• Implement a 13-million-parameter Diffusion LLM from scratch

• Train it on a publicly available pre-training dataset using a free GPU

• Generate text using it

Let’s begin!

Setting up the environment

We will code in PyTorch, use the Hugging Face datasets library for the training dataset, and transformers library to obtain the tokenizer.

The code is meant to run on Google Colaboratory and uses the free NVIDIA T4 GPU to train our model.

# Install packages
!uv pip install torch datasets transformers tqdm

# PyTorch core imports
import torch
import torch.nn as nn
import torch.nn.functional as F

# For numerical operations
import math

# For data processing
from torch.utils.data import DataLoader, Dataset

# Tokenizer
from transformers import AutoTokenizer

# Optimizer
import torch.optim as optim

# For mixed-precision training 
from torch import amp
from torch.nn.utils import clip_grad_norm_

# To visualise progress bar
from tqdm import tqdm

# Hide deprecation warnings
import warnings
warnings.filterwarnings('ignore')

# Set the device (GPU or CPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

Setting up the tokenizer

Instead of building the tokenizer from scratch, we will use the BPE tokenizer for GPT-2. This gives us a 50,257-token vocabulary made up of subwords.

tokenizer = AutoTokenizer.from_pretrained("gpt2")

print(f"Original vocabulary size: {tokenizer.vocab_size}")
# Original vocabulary size: 50257

The end-of-sequence (EOS) token is at the last index in the vocabulary.

print(tokenizer.eos_token)
# <|endoftext|>

EOS_ID = tokenizer.eos_token_id 

print(EOS_ID)
# 50256

To train a diffusion model, we need a special <MASK> token. This token acts as a placeholder for the model to identify the input positions it should fill. We simply append it as a new ID at the end of the vocabulary.

MASK_ID = tokenizer.vocab_size

print(MASK_ID)
# 50257

# Increase the vocabulary size by 1 for the newly added <MASK> token
VOCAB_SIZE = tokenizer.vocab_size + 1 

print(VOCAB_SIZE)
# 50258

Next, we create two helper functions to encode and decode text using the tokenizer.

# Convert input text into a list of token IDs using the tokenizer
def encode(text): 
    return tokenizer.encode(text, add_special_tokens=False)

# Remove any MASK_ID tokens from the sequence and convert the list of token IDs back into readable text
def decode(ids):
    ids = [i for i in ids if i != MASK_ID]
    return tokenizer.decode(ids, skip_special_tokens=True)

Getting our data ready

We will train our diffusion LLM on the TinyStories dataset. It is a synthetic dataset of short stories that contains the vocabulary used by a 3-year-old, generated by GPT-3.5 and GPT-4.

We will use a subset of this dataset that is small enough to train our model on a free-tier GPU, yet rich enough for it to learn semantic details. It is downloaded as follows.

from datasets import load_dataset

# Download the 5000 stories from the 'train' split of TinyStories
dataset = load_dataset("roneneldan/TinyStories", split="train[:5000]")

Each row in the dataset is one complete story. Check out an example of one of them.

print(f"Example Story: \n\n{dataset[0]["text"]}")

"""
Example Story: 

One day, a little girl named Lily found a needle in her room. She knew it was difficult to play with it because it was sharp. Lily wanted to share the needle with her mom, so she could sew a button on her shirt.
Lily went to her mom and said, "Mom, I found this needle. Can you share it with me and sew my shirt?" Her mom smiled and said, "Yes, Lily, we can share the needle and fix your shirt."
Together, they shared the needle and sewed the button on Lily's shirt. It was not difficult for them because they were sharing and helping each other. After they finished, Lily thanked her mom for sharing the needle and fixing her shirt. They both felt happy because they had shared and worked together.
"""

We pre-process this dataset of stories by:

• Tokenizing each story (converting sub-words into token IDs)

• Joining them into a single list of token IDs

• Inserting an EOS token between stories to help the model learn the boundaries between different stories

def clean_and_tokenize(dataset):
    token_ids = []

    for text in dataset["text"]:
        # Remove leading/trailing whitespace from each story
        story = text.strip()

        # Skip empty entries
        if not story:
            continue

        # Encode the story into token IDs and append to the sequence
        token_ids.extend(encode(story))

        # Add an EOS token to separate stories
        token_ids.append(EOS_ID)

    return token_ids

print("Preprocessing dataset...")

token_ids = clean_and_tokenize(dataset)

print(f"Total training tokens: {len(token_ids):,}")
# Total training tokens: 1,033,087

If you’ve previously trained an autoregressive LLM from scratch, you must be familiar with the standard approach of shifting the input sequence by one token and using it as the target during training.

Diffusion LLMs aren’t trained this way.

Instead of next-token prediction, they use fixed-length, probabilistically masked sequences, and the model is trained to predict the original clean input from these masked versions.

Read more

It was adapted in NVIDIA’s Nemotron-3 Super and Ultra models, and now Microsoft has built its first in-house reasoning model, called MAI-Thinking-1, using it. All of these models have reported meaningful gains in accuracy without sacrificing inference throughput or latency using this architecture.

Latent Mixture-of-Experts is an improvement over the popular Mixture-of-Experts (MoE) architecture. Let’s build our foundations by understanding the MoE architecture in depth before we learn about Latent MoE.

What is the MoE architecture?

LLMs based on the Mixture-of-Experts (MoE) architecture contain multiple small feed-forward networks (called Experts) instead of a conventional large feed-forward network in their Transformers. These experts handle different tokens by using another network, called a Router, that selects which expert to use for each token.

MoE enables LLMs to scale their parameter count while keeping the compute cost, or the number of Floating-point Operations (FLOPs) per token, fixed. This is because each token is not processed by all experts in the model as in conventional dense LLMs.

In an MoE LLM with ‘N’ experts, the router directs each token towards only the top-K selected experts, so the active parameter count (which determines FLOPs) stays fixed while the total parameter count of the model (which stores knowledge) can grow enormously.

A great example of an MoE LLM is DeepSeek-V4-Pro, which has 1.6 trillion parameters, but only 49 billion are activated per token during inference.

What works best for Mixture-of-Experts models

1. Memory bandwidth is the real bottleneck for small batches

Consider the following Roofline plot, which shows the maximum performance a workload can achieve as a function of its arithmetic intensity (how much computation is performed per byte of data moved in memory). This plot tells where the workload is limited by memory bandwidth or by compute.

It is built using Qwen3-235B-A22B served on NVIDIA GB200 GPUs connected over NVLink.

When served in a low-latency setting, such as responding to a small number of requests, often from just one user at a time, the real bottleneck for MoE LLM inference is memory bandwidth rather than compute.

This means that when only a few tokens are being processed by the MoE model, the GPU spends almost all its time loading model parameters from memory rather than performing computations. The GPU compute units sit idle most of the time, waiting for parameters to be loaded.

Read more

Anthropic released Fable 5 a few days ago, and everyone is going crazy over it.

Except me.

I have an unpopular opinion about this release.

Reading its system card taught me nothing about AI engineering or development.

There are some interesting observational findings that might scare or excite you depending on your perspective (I feel neutral about them), but they make no real difference to your knowledge or skills.

Here is what the “Training data and process section” says:

Mythos 5 and Fable 5 were trained on a proprietary mix of publicly available information from the internet, public and private datasets, and synthetic data generated by other models.

Throughout the training process, we used several data cleaning and filtering methods, including deduplication and classification.

…

After the pretraining process, the model underwent substantial post-training and fine-tuning, with the goal of making it an assistant whose behavior aligns with the values described in Claude’s constitution.

Claude is multilingual and will typically respond in the same language as the user’s input. Output quality varies by language. The model outputs text only.

This is the most generic text I’ve ever seen around model training. The team basically reveals nothing about how the model was developed.

You could read the following instead to learn about LLMs are trained.

AlgoMaster Newsletter

How LLMs are Actually Trained

In the last lesson, we learned how the Transformer architecture powers an LLM. In this one, we will discuss how LLMs are trained from scratch to become helpful assistants…

Read more

24 days ago · 34 likes · 1 comment · Ashish Pratap Singh and Dr. Ashish Bamania

In another blog post, they vaguely talk about self-improving systems but never share any meaningful algorithmic or systems-level details, unlike other recent open-source research papers.

Instead, they seem more interested in slowing AI progress (except for themselves) and limiting open-source growth.

So I am deciding not to write about this model and its release, even when I might lose the chance to be viral and gain a few thousand subscribers.

I would rather write and tell you about something unexciting yet deeply useful than give in to the hype.

I really appreciate you being here and value your time.

Cheers!

10. Dreaming: Better memory for a more helpful ChatGPT

OpenAI released its more capable and scalable system for synthesizing memory in ChatGPT, called Dreaming.

Dreaming is a method used by ChatGPT to automatically curate memories in the background by referencing chat history.

The newly released Dreaming system:

• Retains useful user context more effectively

• Helps ChatGPT better follow user preferences and constraints

• Stays up-to-date as situations change

Users can also review their memories through a summary page, where they can see, update, and correct what ChatGPT remembers about them.

Read more about this release using this link.

Into AI is a reader-supported newsletter. Gain access to deeper, members-only content by becoming a paid subscriber today.

Join 'Into AI' premium today

9. MAI-Thinking-1: Building a Hill-Climbing Machine

Microsoft introduced MAI-Thinking-1, a reasoning model built from scratch based on the Latent Mixture-of-Experts architecture with 1T parameters (35B active).

It is the first model developed using the Hill-climbing machine. This is an iterative system for building and improving data pipelines, training infrastructure, RL environments and rewards, evaluation suites, and safety tests, which allows advancing AI while grounding progress around human needs from the ground up.

MAI-Thinking-1 performs strongly among models of similar size on STEM reasoning and coding tasks (52.8% on SWE-Bench Pro, 97.0% on AIME 2025, and 87.7% on LiveCodeBench v6).

Read more about this research paper using this link.

8. Benchmark Everything Everywhere All at Once

This paper introduces the Benchmark Agent, a fully autonomous agentic system for building benchmarks. This system can orchestrate a complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control.

Evaluations with multiple experiments (human evaluation, LLM-as-a-judge assessment, and consistency checks) show that Benchmark Agent can generate high-quality benchmarks across text, multimodal, audio, image, and domain-specific reasoning tasks with minimal human involvement.

Read more about this research paper using this link.

Subscribe now

7. Audio Interaction Model

This research paper introduces Audio-Interaction, a single model that unifies offline audio tasks (such as speech recognition, translation, and audio Q&A) with real-time streaming tasks (such as live voice chat and proactive response).

The model operates using an always-on ‘perceive-decide-respond’ loop, continuously listening to sound, the environment, and instructions, and deciding when to respond based on streaming semantics. The model can manage real-time ASR, streaming audio instruction following, voice dialogue, and proactive help.

To make this Audio Interaction model possible, researchers use Soundflow, an end-to-end framework for streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference.

Across 8 benchmarks, Audio-Interaction has competitive performance on mainstream audio tasks while offering capabilities previously inaccessible to Large Audio Language Models (LALMs).

Read more about this research paper using this link.

6. Reasoning Structure of Large Language Models

LLMs are usually evaluated using metrics such as final-answer accuracy or token count, but even with identical scores, these models can have fundamentally different reasoning structures.

This research paper addresses this problem by introducing a scalable logic-puzzle benchmark and a pipeline that converts unstructured reasoning traces into verifiable reasoning graphs of claims and dependencies, whose topology can be quantitatively analyzed.

Building on this, a reasoning-efficiency metric is defined to quantify the concentration of the model’s logical flow. This acts as a practical tool for diagnosing failure modes and comparing how reasoning scales with puzzle difficulty.

Read more about this research paper using this link.

5. Do Transformers Need Three Projections?

This research paper examines whether the standard Transformer attention really needs separate query, key, and value projections.

This is done by comparing three types of shared projections:

• Q-K=V (Shared key-value)

• Q=K-V (Shared query-key)

• Q=K=V (Fully shared single projection)

Experiments across synthetic tasks, vision, and language modeling show that Transformers with shared projections perform on par with, or occasionally better than, the standard Transformer.

In language modeling, shared key-value projection (Q-K=V) achieves a 50% KV cache reduction with only a 3.1% increase in perplexity.

These transformers also preserve attention directionality because keys and values can share similar representation spaces and attention operates in a low-rank regime.

Since projection sharing is complementary to head-sharing (GQA/MQA), combining Q-K=V with GQA-4 and MQA leads to cache reductions of 87.5% and 96.9%, respectively. This has direct, quantifiable benefits for inference memory, which are especially valuable for edge deployment of LLMs.

Read more about this research paper using this link.

Subscribe now

4. Nemotron 3 Ultra

NVIDIA introduced its most capable model yet, called Nemotron 3 Ultra. This is a 550B-parameter (55 billion active) Mixture-of-Experts Hybrid Mamba-Attention language model with a 1M-token context window.

Nemotron 3 Ultra combines multiple key technologies such as:

• Latent-MoE architecture

• Multi Token Prediction (MTP)

• NVFP4 pre-training

• Multi-environment RLVR

• Multi-teacher On-Policy Distillation (MOPD) for post-training

• Reasoning budget control

Nemotron 3 Ultra delivers up to 6× higher inference throughput than SOTA publicly available LLMs while achieving similar accuracy, which makes it ideal for long-running autonomous agentic tasks.

Read more about this research paper using this link.

3. Cosmos 3

This research paper introduces Cosmos 3, a family of omnimodal world models by NVIDIA. These models are designed to jointly process and generate language, image, video, audio, and action sequences using a single Mixture-of-Transformers architecture.

The goal of the model is to serve as a general-purpose backbone for physical AI, combining capabilities that are usually split across vision-language models, video generators, world simulators, and robot policy/action models.

Cosmos 3 achieves SOTA results across multiple understanding and generation tasks, and the post-trained Cosmos 3 models are ranked as the best open-source Text-to-Image and Image-to-Video models by Artificial Analysis and the best policy model by RoboArena.

Read more about this research paper using this link.

2. Agents’ Last Exam

This research paper introduces Agents’ Last Exam (ALE), a benchmark designed to evaluate AI agents on long-horizon, economically valuable, real-world tasks with verifiable outcomes.

Built by 250+ industry experts, it spans 55 targeted sub-industries and includes 1,500+ tasks covering most major fields of professional work performed on a computer.

Results show that today’s mainstream agentic models are still far from solving the toughest tasks on this benchmark, with an average full pass rate of only 2.6%.

Rather than serving as a static benchmark, ALE is intended to grow continuously as new workflows and industries are onboarded, making it a tool for measuring real-world agent capability relevant to GDP impact.

Read more about this research paper using this link.

1. MiniMax M3

This blog post introduces MiniMax M3, currently the first and only open-weight model that brings the following three capabilities together:

1. Frontier-level performance on specialized tasks such as coding and agentic work

2. Ultra-long context windows of up to 1M tokens

3. A natively multimodal model that supports image and video input and can operate on a desktop computer

The model uses MiniMax Sparse Attention (MSA), a sparse attention mechanism that enables long-context processing at a much lower per-token compute and large speedups in prefill and decoding.

• MiniMax M3 surpasses GPT-5.5 and Gemini 3.1 Pro and approaches Opus 4.7 on SWE-Bench Pro.

• On SVG-Bench, an internal benchmark that comprehensively evaluates SVG generation performance, MiniMax M3 surpasses Opus 4.7.

• M3 also achieves top scores on Claw-Eval, an end-to-end evaluation framework for autonomous agents, and on OmniDocBench, a multimodal benchmark.

Read more about this release using this link.

This newsletter edition is completely free to read.

If you found it valuable, click the like button ❤️ and consider subscribing for more such content every week.

If you have any questions or suggestions, feel free to leave a comment below.

Into AI is a reader-supported newsletter. Gain access to deeper, members-only content by becoming a paid subscriber today.

Join 'Into AI' premium today

Understanding distributed setups for LLM training and inference is one of the biggest advantages that you can have as an engineer today. This is what we will work towards in this lesson by studying how Meta’s Llama 3 models were trained in a distributed setting.

Let’s begin!

How big are modern-day GPU clusters?

Building an LLM requires thousands of GPUs working together in sync and months of training time. This is because no single GPU has enough memory to store the model and optimizer parameters, nor does it have the compute to train a billion-parameter model on trillions of tokens in any reasonable amount of time.

The Llama 3-405B model was trained on a cluster of 16,000 GPUs (each with 80GB of HBM3) across multiple servers. Each server contained 8 GPUs and 2 CPUs, with the GPUs connected via NVLink. It took 54 days and 30.84M GPU hours to train this model.

How does LLM training occur on GPU clusters?

Four different types of parallelism were used to train the Llama 3-405B model. These are combined together, resulting in a technique called 4D parallelism.

This technique efficiently distributes computation across multiple GPUs and ensures that the following fit within each GPU’s HBM (memory):

• Model parameters

• Optimizer states

• Gradients

• Activations

The four parallelism techniques used in 4D parallelism are:

1. Data parallelism (DP)

2. Context parallelism (CP)

3. Tensor parallelism (TP)

4. Pipeline parallelism (PP)

These last two (TP and PP) are also known as Model parallelism techniques as they split the model across GPUs.

Let’s discuss them one by one.

Subscribe now

1. Data Parallelism (DP)

The idea behind Data parallelism (DP) is to replicate the model across multiple GPUs and run the forward and backward passes in parallel on small and different batches of training data on each GPU. This is then followed by synchronizing the gradients between all GPUs.

If this were a small model being trained, DP would have worked well. But a 405B model can’t be replicated on each GPU with a mere 80 GB of memory.

This is why Llama’s 4D parallelism uses a type of Data parallelism called FSDP (Fully Sharded Data Parallelism), where, instead of replicating the model across GPUs (like in DP), FSDP shards the model parameters, optimizer states, and gradients across GPUs and temporarily gathers them on demand when they're needed for computation.

2. Context Parallelism (CP)

Training LLMs on long sequences (128K+ tokens per sequence) can exceed the memory that is available per GPU. This is because of the attention mechanism in LLMs, which scales quadratically with sequence length, as every token attends to every other token in the sequence.

This bottleneck is solved by Context parallelism (CP), which divides the input sequence into smaller segments and splits them across multiple GPUs.

You can read more about CP

3. Tensor Parallelism (TP)

Tensor parallelism (TP) splits individual weight tensors along the model’s hidden dimension into multiple chunks and distributes them across GPUs.

This means that each GPU stores and computes only a fraction of matrix multiplication operations. This matrix splitting can occur column-wise or row-wise depending on which layer they are a part of in the transformer block. Partial calculations are followed by synchronization to combine the results across all GPUs.

4. Pipeline Parallelism (PP)

Pipeline parallelism (PP) splits the layers of the LLM into partitions called stages (groups of model layers) across multiple GPUs and trains them with small batches of data (micro-batches) as in an assembly line/ pipeline. In this way, each GPU only needs to store and process a portion of the model, which significantly reduces the memory requirements per GPU.

How do these come together in 4D parallelism?

For training Llama 3-405B, GPUs in a cluster are divided into groups and labeled using a vector as [TP, CP, PP, DP], where DP is actually FSDP.

The following example shows a cluster with 16 GPUs, each of which is assigned a unique label.

The vector label intentionally arranges the four dimensions of parallelism from the highest to the lowest communication bandwidth each requires (TP > CP > PP > DP/FSDP).

TP requires a high-bandwidth / low latency connection between GPUs to function well. This is why it is implemented within a single server where NVLink connects the 8 GPUs together.

On the other hand, PP and FSDP have lower bandwidth requirements and tolerate communication over a multi-hop network with higher latency than NVLink.

The team used Meta’s NCCLX collective communication framework to optimize this 4D parallelized large-scale LLM training.

Are there more dimensions to parallelism?

Yes, there are!

Llama 3 models are dense models. This means it contains fully connected (dense) feed-forward layers in its Transformer architecture.

But if they were Mixture-of-Experts (MoE) models, there’s another parallelism technique that could be applied. It’s called Expert Parallelism (EP).

A MoE model contains multiple small feed-forward networks, called Experts, that handle different tokens, using another network, called a Router, that selects which Expert to use for each token.

Because an MoE model can have hundreds of Experts, all of them won't fit on a single GPU. This is the bottleneck that Expert Parallelism (EP) works around. EP spreads the experts across GPUs, so each GPU holds the weights of a subset of experts at a time.

When a token is processed by the MoE layer and a router selects the expert(s) best suited to it, the token is routed to the GPU containing those experts. These experts perform the computation and send the result back to the token's original GPU.

Expert Parallelism (EP), when combined with other parallelism techniques, is referred to as 5D parallelism.

TL;DR

To summarise:

• Meta used 4D parallelism to train Llama 3 models

• 4D parallelism used in this case consists of:

  • Fully Sharded Data Parallelism (FSDP)

  • Context Parallelism (CP)

  • Two Model Parallelism techniques called Tensor Parallelism (TP) and Pipeline Parallelism (PP)

• Fully Sharded Data Parallelism (FSDP) shards model parameters, gradients, and optimizer states across GPUs.

• Context Parallelism (CP) divides the input data along the sequence length dimension to handle very long text inputs.

• Tensor Parallelism (TP) splits individual weight tensors and activations within a single layer along the model’s hidden dimension.

• Pipeline Parallelism (PP) splits the model's layers sequentially into groups called stages.

• Expert Parallelism (EP) is used to train Mixture-of-Experts (MoE) models. EP splits different experts across GPUs.

• EP combined with 4D parallelism is referred to as 5D parallelism.

Further Reading

• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

• Ring Attention with Blockwise Transformers for Near-Infinite Context

• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

• MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core

• The Llama 3 Herd of Models

• More articles on distributed training in the newsletter:

Support my writing and join the paid tier today to get access to all posts in this newsletter:

• 👾 What makes DeekSeek-V4 so good?

• 👨‍🔬 Build and Train a Mixture-of-Experts (MoE) LLM from scratch

• 🌈 Diffusion LLMs, Explained Simply

• 📚 20 PyTorch Concepts, Explained Simply

• 🚀 Train a Diffusion LLM from scratch

and so many more!

Join 'Into AI' premium today

10. Language Models Need Sleep

The attention mechanism in Transformer-based LLMs scales poorly with increasing context length due to its quadratic complexity.

This research paper proposes a sleep-like mechanism to address this problem, in which an LLM periodically converts recent context into persistent fast weights before clearing its KV cache.

These fast weights are implemented as State-space model (SSM) blocks, and during sleep, N offline recurrent passes over the accumulated context update these weights.

During inference, this shifts extra computation to sleep while preserving the latency of wake-time prediction.

When tested on synthetic tasks such as cellular automata, multi-hop graph retrieval, and a realistic math reasoning task, both regular transformers and vanilla SSM-attention hybrid models fail.

In contrast, this is not true for the sleep-augmented SSM-attention hybrid model, where increasing sleep duration (number of recurrent passes N) improves performance, with the largest gains on tasks that require deeper reasoning.

Read more about this research paper using this link.

Join the paid tier today to get access to all posts in this newsletter:

• 👾 What makes DeekSeek-V4 so good?

• 🌈 Diffusion LLMs, Explained Simply

• 📚 20 PyTorch Concepts, Explained Simply

• 👨‍🔬 Build and Train a Mixture-of-Experts (MoE) LLM from scratch

• ⌨️ Learn to train deep learning models on multiple GPUs

• 🚀 Train a Diffusion LLM from scratch (out this week)

and so many more!

Join 'Into AI' premium today

9. LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

This research paper presents LocateAnything, a vision-language grounding and detection framework that replaces slow token-by-token coordinate generation with fast Parallel Box Decoding.

This method treats boxes or points as atomic geometric units, decoding them in one step, and improves both decoding throughput and localization consistency.

Researchers also developed a scalable data engine and curated LocateAnything-Data, a large-scale dataset with more than 138M training samples, which substantially increases data diversity for high-precision localization.

Evals show that LocateAnything advances the speed-accuracy frontier, achieving significantly higher decoding throughput while improving high-IoU (Intersection over Union) localization quality across different benchmarks.

Read more about this research paper using this link.

8. The MiniMax-M2 Series

This research paper presents the MiniMax-M2 series of highly sparse Mixture-of-Experts LLM models designed for efficient agentic work.

The flagship M2.7 model has:

• 229.9B total parameters

• Activates only 9.8B per token

• A 192K-token context and multi-token prediction for faster inference

This series uses three important components:

• An agent-driven data pipeline that produces large-scale, verifiable trajectories across agentic coding and agentic co-work

• Forge, a scalable agent-native RL system for long-horizon agentic tasks

• A self-evolution mechanism in the M2.7 model, where the model autonomously debugs training runs and modifies its own scaffold.

Evals show that this series of models produces frontier-tier performance on agentic coding, deep search, office-task, and reasoning benchmarks, despite activating so few parameters per token.

The MiniMax M3 model is also out today, and we will discuss it in the next edition of the newsletter.

Read more about this research paper using this link.

7. InstructSAM: Segment Any Instance with Any Instructions

This research paper presents InstructSAM, a 2B vision-language segmentation model that enables users to segment objects using free-form instructions that include attributes, relations, counting, exclusion, or implicit intent, rather than only simple object names.

It connects a VLM to SAM3 using parallel learnable instance queries, enabling it to reason about the instruction and produce multiple instance masks in a single forward pass.

Researchers also introduce Inst2Seg, a large instruction-based segmentation dataset and benchmark consisting of free-form instructions with instance-level masks.

On this benchmark, InstructSAM outperforms previous end-to-end methods and SAM3’s agentic pipeline while being much faster (1.1 seconds inference time compared to 29.6 seconds for SAM3-Agent-Qwen3-VL-2B).

Read more about this research paper using this link.

6. PowLU: An Activation Function for Stable Pre-Training of LLM

SwiGLU is a widely used activation function in LLMs today. For large positive inputs, SwiGLU behaves roughly like the quadratic function x², which gives it strong nonlinearity and expressive capacity.

However, this property also leads to numerical instability as the input or model scale increases, especially in low-precision LLM training, because it widens the output range and worsens outliers.

To address this, this research paper proposes a new activation function called the Power Linear Unit (PowLU) for large-scale LLM pre-training.

PowLU uses a rational power function to achieve adaptive nonlinearity while reducing extreme amplification, making training more stable.

Scaling-law experiments and experiments on the Ling 7.9B and 124B models show that PowLU achieves results competitive with SwiGLU and SwiGLU-Clip, while improving the stability and scalability of large-scale LLM training.

Read more about this research paper using this link.

Subscribe now

5. When Does LeJEPA Learn a World Model?

LeJEPA is a self-supervised learning method that combines a predictive alignment loss with Gaussian regularization (SIGReg).

This research paper presents a theory of when LeJEPA captures a representation that truly becomes a world model or a faithful map of the world’s latent structure.

The paper proves that, under Gaussian latent variables, stationary additive-noise transitions, alignment loss, and Gaussian regularization, LeJEPA can identify the true latent variables up to a linear rotation, making the representation useful for planning.

Experiments in toy settings, high-dimensional latent spaces, and pixel-based robotic control support the theory.

Read more about this research paper using this link.

4. AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation

This research paper presents AutoScientists, a decentralized multi-agent system designed for long-running scientific experimentation.

In this system, AI agents share the experimental state, self-organize to form teams around promising hypotheses, critique proposals before using experimental compute, share successes and track failures to prevent repetition, and reorganize as the evidence changes.

This contrasts with existing agentic approaches that either follow a single research trajectory or coordinate through a central planner with fixed objectives.

Evals show AutoScientists achieves a mean leaderboard percentile of 74.4% on BioML-Bench, improving over the strongest AI agent by +8.33%.

For GPT training optimization, it reaches a target validation bits-per-byte 1.9x faster than Autoresearch.

It also discovers a method for ACE2-Spike binding that improves over the current SOTA model by +12.5% in Spearman correlation.

Read more about this research paper using this link.

3. SIA: Self-Improving AI with Harness & Weight Updates

This research paper introduces SIA, a self-improving AI framework in which a language-model agent (the Feedback-Agent) improves both the agent harness and the model’s weights using LoRA/RL updates.

In Chinese legal charge classification, low-level GPU kernel optimization, and single-cell RNA denoising tasks, SIA-W+H (combined weight and harness updates) works better than the approach that only improves the harness (SIA-H).

SIA-W+H achieves 25.1% over prior SOTA on LawBench, 12.4% faster GPU kernels than prior SOTA, and 20.4% over prior SOTA on denoising.

Read more about this research paper using this link.

Join 'Into AI' premium today

2. Self-Improving Language Models with Bidirectional Evolutionary Search

This research paper introduces Bidirectional Evolutionary Search (BES), a search framework for self-improving LLMs and agentic systems for post-training sample generation and inference.

Commonly used methods such as best-of-N sampling and tree search have limitations because they rely on sparse verification signals and primarily explore high-probability candidates constructed through autoregressive expansion.

BES addresses this issue using:

• Forward evolutionary search, which augments standard expansion with evolution operators that recombine partial trajectories to generate candidates that are difficult to obtain from a single model rollout

• Backward goal decomposition, which divides tasks into checkable subgoals, producing dense intermediate feedback that guides forward search

Theoretically, this approach increases exploration and can exponentially reduce the number of required samples to find a correct answer, leading to consistent improvements on tough post-training tasks and open problem-solving benchmarks.

Read more about this research paper using this link.

1. Introducing Claude Opus 4.8

Anthropic released Claude Opus 4.8, their new flagship model, with improvements across benchmarks compared to their previous models and competitors.

One of the most prominent improvements in Opus 4.8 is its honesty. Opus 4.8 is less likely to make unsupported claims and is about 4X less likely than its predecessor to let flaws in code it has written go unremarked.

Opus 4.8’s rates of misaligned behavior (such as deception or cooperation with misuse) are substantially lower than Opus 4.7's and are similar to those of Anthropic’s best-aligned model, Claude Mythos Preview.

It also comes with a fast mode (where the model can work at 2.5× the speed), which is 3X cheaper than it was for previous models.

Read more about this release using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

👋🏻 Hi everyone!

I’ve been quietly building something on the side for the last few months, and it’s finally ready for you to try out.

I really appreciate your subscribing to the publication, and I am personally inviting you to try it 🤗

Tensorwise is a crossword puzzle game for AI enthusiasts just like you!

It keeps you up to date with the latest in AI tech and research without overwhelming you with new terminology.

One of the coolest features is that you can learn more about each term and better understand them with the attached resource.

Tensorwise also lets you compete with other players for a spot on the leaderboard!
(And, I might bring in prizes for you all in the future for scoring the best.)

A quick note: All crosswords are handmade by me and not AI-generated. I can’t provide enough evidence for this because current frontier LLMs are terrible at grid-like environments and cannot generate reliable technical crosswords. So if you doubt me, try getting an LLM to make a decent puzzle yourself.

Excited to give it a try and become more Tensor-wiser?

Try Tensorwise

🍀 I’d love to hear how well you did, what you enjoyed, and which clue made you swear at your screen. (Also, if something broke.)

Cheers!

10. On-policy vs. Off-policy learning

A Policy is the strategy by which an AI agent chooses actions in a given state. For an LLM, the model is the policy itself.

With on-policy learning for an LLM, the model learns from its own responses/ outputs.

A commonly used approach to train LLMs to improve them in math/ coding tasks goes like this:

• Given a query or prompt, a model produces a group of responses

• The responses are scored using a verifier or a reward model

• An algorithm like GRPO (Group Relative Policy Optimization) is used to train the model to produce responses that score above average while pushing down below-average responses.

In contrast to the above, with off-policy learning, an LLM learns from the responses/outputs that are not generated by itself. These could come from a stronger model (the teacher model), a different version of the same model, or a dataset.

A commonly used approach for training LLMs to improve their performance in specific domains is called Distillation. In this approach:

• Responses or reasoning traces are generated using a strong (teacher) model

• A weaker (student) model is trained on those to imitate the teacher's responses

There’s also a recently introduced learning approach that's gaining popularity and combines on-policy and off-policy learning. It is called On-policy distillation. Curious readers are encouraged to learn more about it.

The images used in this lesson come from my book LLMs In 100 Images, which is a collection of 100 easy-to-follow visuals that explain the most important concepts you need to master to understand LLMs today.

Grab your copy today at a 30% discount using this link.

9. Pretraining vs. Mid-training vs. Post-training

All three are phases of training LLMs before they are released to end users.

Pretraining is the initial training phase that teaches a model the structure of a language and gives it basic factual knowledge of the world.

Pretraining occurs on massive datasets such as Common Crawl/ FineWeb, which consists of trillions of tokens from the web. This process is usually highly compute-intensive, and a pre-trained model is called the “Base model”.

Mid-training is the next phase, which involves further training on the base model using higher-quality, domain-specific data to improve its capabilities. This data might focus on domains like health, law, math, or code, or on improving reasoning, extending context length, or adding a new language.

Post-training is the final phase that teaches a model to become a useful and human-value-aligned assistant, rather than one that just produces the next token. Some common post-training techniques include:

• Supervised fine-tuning (SFT) on datasets consisting of instruction-response pairs

• RL training methods like RLHF to teach a model to produce human-value aligned responses, or RLVR to teach a model to reason well through math/ code related problems to correctly solve them

8. Zero-shot vs. One-shot vs. Few-shot prompting

All three are approaches for prompting an LLM to complete a task well.

Zero-shot prompting is when a user describes a task in the prompt and the model uses what it has already learned during its training to complete it.

One-shot prompting is when a user provides a single example of how to complete a task in the prompt.

Few-shot prompting is when a user describes how to complete a task with several examples in the prompt.

7. CPU vs. GPU vs. TPU in LLM workflows

The role of different semiconductor chips in LLM training and inference is frequently confused.

CPU (Central Processing Unit) is the standard chip that acts as the brain of every modern computer. It contains a few powerful cores that are perfect for executing tasks with sequential and branching logic at very low latency.

However, a CPU is not optimized for the parallel matrix multiplication operations that take place during LLM training or inference. Its role is in orchestration, which involves data loading and preprocessing/postprocessing, tokenization/detokenization, scheduling work onto GPUs/TPUs, running the training loop's control logic, and serving infrastructure.

CPUs have recently found a great use case in agentic workflows. This is because these workflows are dominated by tasks such as tool calls, API requests, file or database reads, and parsing outputs rather than heavy parallel computations. This is exactly where CPUs shine.

GPUs (Graphics Processing Units) are the masters of parallel computing. They have thousands of cores that are individually slower than CPU cores, but together produce massive throughput on parallel computations, especially matrix or tensor operations, which are core to LLM training and inference.

Modern GPUs also come with dedicated cores and memory for matrix multiplications. NVIDIA GPUs can be programmed with CUDA to implement highly efficient kernels (programs written for the GPU) for deep learning algorithms (for example, FlashAttention).

Tensor Processing Units (TPUs) are specialized chips designed by Google, optimized for fast and efficient tensor/matrix multiplication operations. These chips are particularly well-suited to Google's JAX ecosystem, giving it an advantage in large-scale training and inference.

A great book to learn more about TPUs can be found here.

5 resources to get started with programming LLMs at scale using GPUs/TPUs are listed below.

6. Words vs. Tokens

Words are units of language that humans understand. Tokens, on the other hand, are units of language that LLMs work with.

Tokens are created by a Tokenizer and can be whole words, subwords, or single characters. The complete set of all tokens that an LLM is familiar with is called its Vocabulary.

OpenAI models use tokenizers based on the Byte Pair Encoding (BPE) algorithm that convert human language into subwords for an LLM to work with.

5. Logits vs. Probabilities

LLMs produce raw scores called logits for each token in their vocabulary at each step of text generation. These scores are real numbers that can be negative, zero, or positive.

Logits are converted to probabilities using the softmax function, which exponentiates each logit and then divides by the sum of those exponentials.

The resulting probabilities are normalized, which means that they range between 0 and 1 and sum to 1. Each probability is the model’s estimated chance of being the next token in the sequence.

4. Context window vs. Memory

Memory is a broad term, while the context window is a specific type of memory that LLMs use by default.

The context window is the maximum number of tokens (both input and generated) an LLM can process at once. It is the working memory of an LLM that the model attends to during use. Once a chat session ends, this information stored in the context window is discarded.

Memory, on the other hand, is a broad term that includes:

1. Working memory, or Short-term memory, that stores information about what the LLM is thinking about during ongoing problem-solving. This is the LLM’s context window.

2. Long-term memory, which is further divided into three:

   • Procedural memory: This stores rules or skills that can be applied to working memory to determine the LLM’s behavior.

   • Semantic memory: This stores general facts and knowledge about the world.

   • Episodic memory: This stores sequences of the LLM’s past behaviors and interactions.

A few examples of long-term memory systems are Zep, Mem0, and LangMem.

You can read more about LLM memory using the following links.

3. LLM vs. AI agents

An LLM is the model that generates output/ response for a given input/ prompt. It does not remember its past actions beyond its context window and cannot fetch external data.

An AI agent, on the other hand, has agency. It can take actions to work towards a goal by using an LLM (the “brain”) in a loop.

An AI agent:

• Can call tools (web search, code execution, etc.) to access new data

• Has memory to remember its past actions

• Uses a control loop to plan (using an LLM), act, observe, and decide the next action until its goal is achieved

• Can work with other AI agents to achieve a goal

2. Inference vs. Decoding

Inference is the process of running an LLM to produce outputs. It involves processing the given prompt, running a forward pass, and generating the output tokens. Modern LLM inference also involves batching multiple user requests, processing them efficiently on available hardware to achieve higher throughput, and streaming responses back token by token.

LLM inference takes place in two stages:

• Prefill: Processing the given prompt to build the KV cache for the LLM to speed up text generation in the subsequent steps

• Decode/ Decoding: Text generation by producing tokens one at a time

Decoding is a part of Inference, which involves choosing how each output token is generated.

At every step of text generation, an LLM outputs a probability distribution over all the tokens in its vocabulary, and the decoding strategy tells which token to pick from it as the next token.

Some commonly used decoding strategies are:

• Greedy

• Beam search

• Top-P sampling

• Top-K sampling

A hyperparameter called Temperature can be used with any sampling-based decoding strategy to control the randomness or creativity of the generation process.

You can read about these in detail using the link below.

1. Pruning vs. Quantization

Pruning and Quantization are both model compression techniques that reduce an LLM's memory and compute requirements while preserving accuracy.

Pruning means reducing the number of parameters in a model by setting their values to zero or by removing entire neurons, attention heads, or full layers. The removed ones are usually parameters that contribute little to the model's output and are not very important for accuracy.

Quantization, on the other hand, keeps all the parameters but reduces their numerical precision. For example, instead of representing weights as 32 or 16-bit floating-point numbers, they are converted to lower-precision numbers, such as 8 or 4-bit integers. In this way, each parameter is represented using fewer bits, requiring less memory and compute to run.

Join the paid tier today to get access to all posts in this newsletter:

• 👾 What makes DeekSeek-V4 so good?

• 🌈 Diffusion LLMs, Explained Simply

• 📚 20 PyTorch Concepts, Explained Simply

• 👨‍🔬 Build and Train a Mixture-of-Experts (MoE) LLM from scratch

• ⌨️ Learn to train deep learning models on multiple GPUs

• 🚀 Train a Diffusion LLM from scratch (out next week)

and so many more!

Join 'Into AI' premium today

10. Vector Policy Optimization

This research paper introduces Vector Policy Optimization (VPO), an RL post-training method for LLMs. VPO improves test-time search by preserving diversity rather than narrowing to a single high-reward answer.

Unlike GRPO, which focuses on optimizing a single scalar reward, VPO employs vector-valued rewards (such as per-test-case correctness, multiple user preferences, or sub-task scores) and trains the model to produce sets of solutions that cater to different reward trade-offs across the Pareto frontier.

VPO matches or exceeds the performance of strong scalar RL baselines on test-time search (best@k and pass@k metrics) across multi-hop question answering, logic reasoning, navigation, tool use, and coding. These improvements become more significant as the search budget increases.

For evolutionary search, VPO models unlock problems that GRPO models cannot solve at all.

According to the authors, as test-time search becomes more standardized, optimizing for diversity may need to become the default post-training objective.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter:

• 👾 What makes DeekSeek-V4 so good?

• 🌈 Diffusion LLMs, Explained Simply

• 📚 20 PyTorch Concepts, Explained Simply

• 👨‍🔬 Build and Train a Mixture-of-Experts (MoE) LLM from scratch

• ⌨️ Learn to train deep learning models on multiple GPUs

• 🚀 Train a Diffusion LLM from scratch (out next week)

and so many more!

Join 'Into AI' premium today

9. Code as Agent Harness

This survey argues that in modern agentic AI systems, code is no longer just an output produced by LLMs. Instead, it is the executable, inspectable, and stateful substrate that allows agents to reason, act, model environments, receive feedback, and verify their progress.

To systematically study this perspective, the authors present a three-layer taxonomy that includes:

• Harness interfaces, where code connects agents to reasoning, action, and environment modeling

• Harness mechanisms, which include planning, memory, and tool use for long-horizon execution, together with feedback-driven control and optimization that make harness reliable and adaptive

• Scaling the harness, from single-agent systems to multi-agent settings, where shared code artifacts support multi-agent coordination, review, and verification

Across these layers, the authors summarize representative methods and practical applications of Code as agent harness, and outline open challenges for harness engineering.

Read more about this research using this link.

8. Tokenization via Convex Relaxations

This research paper introduces a new tokenization algorithm called ConvexTok.

Current tokenization algorithms, such as BPE and Unigram, are greedy algorithms that make locally optimal decisions without considering the resulting vocabulary as a whole.

ConvexTok, instead, formulates the construction of a tokenizer as a linear program and solves it using a convex optimization tool.

It improves intrinsic tokenization metrics, bits-per-byte (BpB), and downstream task performance of LLMs. It also allows users to certify how close their tokenizer is to the global optimum. Empirically, ConvexTok tokenizers are within 1% of optimal for common vocabulary sizes.

Read more about this research using this link.

Subscribe now

7. Probabilistic Tiny Recursive Model

This research paper introduces Probabilistic TRM (PTRM), which improves Tiny Recursive Models by adding Gaussian noise during inference, to create multiple stochastic latent rollouts rather than a single deterministic trajectory.

The model then uses its existing Q head to select the most promising answer, avoiding bad solutions without retraining or task-specific augmentations.

PTRM improves accuracy across multiple benchmarks, including Sudoku-Extreme (87.4% to 98.75%) and Pencil Puzzle Bench (62.6% to 91.2%).

On Pencil Puzzle Bench, PTRM achieves nearly double the accuracy of frontier LLMs (91.2% vs. 55.1%) at less than 0.0001x the cost, using only 7M parameters!

Read more about this research using this link.

6. A Bitter Lesson for Data Filtering

This research paper studies data filtering for LLM pretraining in high-compute, data-scarce settings and challenges the common belief that filtering data to include only high-quality information is essential.

Experiments show that with enough compute, the best data filter is no data filter. Sufficiently trained large parameter LLMs not only tolerate low-quality data but also benefit from it.

Read more about this research using this link.

5. Strong Teacher Not Needed? On Distillation in LLM Pretraining

In this research paper, the authors challenge a general assumption in Knowledge distillation, that stronger teachers lead to better students.

Their findings show that with proper mixing of the language modeling and knowledge distillation losses, even small and undertrained teachers improve larger students.

Pushing a strong teacher further, through more parameters or more training tokens, can saturate or even reverse the distillation gains.

It is further observed that distillation helps the student model generalize more (perform better on out-of-distribution and downstream tasks) than it helps improve on the training data (in-domain tasks).

Read more about this research using this link.

4. HRM-Text: Efficient Pretraining Beyond Scaling

This research paper introduces HRM-Text, a language model that replaces standard Transformers with a Hierarchical Recurrent Model (HRM), an architecture inspired by the human brain that uses slow (strategic) and fast (execution) layers.

To stabilize training for language modeling, the authors introduce MagicNorm and warmup deep credit assignment.

Alongside this, rather than standard raw-text pretraining, the model is trained exclusively on instruction-response pairs, using a task-completion objective and PrefixLM masking.

A 1B parameter HRM-Text model trained from scratch with a $1,500 budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH.

Despite using ~100-900x fewer training tokens and ~96-432x less compute than standard baselines, HRM-Text performs competitively with 2–7B parameter open models.

Read more about this research using this link.

Subscribe now

3. Gated DeltaNet-2

This research paper introduces Gated DeltaNet-2, which improves linear-attention models by separating two memory operations, erasing old information and writing new information, that previously introduced Gated Delta Networks and Kimi Delta Attention (KDA) tied together.

It uses separate channel-wise erase and write gates while supporting efficient recurrent and parallel training. This allows the model to manage compressed long-context memory better without the high cost of standard attention.

With 1.3B parameters, Gated DeltaNet-2 outperforms Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants in language modeling, commonsense reasoning, and retrieval, particularly in long-context needle-in-a-haystack tasks.

Read more about this research using this link.

2. Advancing Mathematics Research with AI-Driven Formal Proof Search

This research paper presents AlphaProof Nexus, a framework that uses LLM-based agents to generate and iteratively improve Lean formal proofs using compiler feedback, optional AlphaProof calls, and evolutionary search.

In a large-scale test on open research problems, the strongest agent solved 9 of 353 Erdős problems, proved 44 of 492 OEIS conjectures, and contributed to multiple other open mathematical problems.

Read more about this research using this link.

1. An OpenAI model has disproved a central conjecture in discrete geometry

This blog post reports that an internal reasoning model at OpenAI has disproved Erdős’s long-standing unit distance conjecture in discrete geometry, that the maximum number of unit-distance pairs among n points in the plane were bounded by n¹⁺ᵒ⁽¹⁾.

The model produced an infinite set of point configurations with at least n¹⁺δ unit distances for some fixed δ greater than 0, using unexpected tools from algebraic number theory, such as class field towers and Golod–Shafarevich theory.

The proof has been verified by external mathematicians, and this is a major milestone where an AI system has autonomously resolved an important open mathematical problem, revealing a surprising connection between algebraic number theory and discrete geometry.

Read more about this using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

👋🏻 Hi everyone!

I've been quietly building something on the side for the last few months, and it's finally ready for you to try out.

Read more

LLMs are trained to predict next tokens well.

Given all the classes/ tokens in the vocabulary, an LLM is trained to pick the right one at each training step. This is essentially a Multi-class classification problem in machine learning.

The Categorical Cross-entropy loss (or simply the Cross-entropy loss) is used to train an LLM to solve this multi-class classification problem.

Read more

10. Achieving Gold-Medal-Level Olympiad Reasoning via Simple and Unified Scaling

This research paper introduces a simple method for converting a post-trained reasoning LLM into a rigorous Olympiad-level solver.

The method first uses a reverse-perplexity curriculum for SFT to instill rigorous proof-search and self-checking behaviors, then scales these behaviors through a two-stage RL pipeline that progresses from RL with verifiable rewards to more delicate proof-level RL, and finally boosts solving performance with test-time scaling.

The authors train a 30B model, SU-01, using this method, which reasons on difficult problems with trajectories exceeding 100K tokens, achieving gold-medal-level scores on IMO 2025 and USAMO 2026.

This model also shows strong generalization of scientific reasoning to domains beyond mathematics and physics.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter:

• 🍔 What makes DeekSeek-V4 so good?

• 🍕 Diffusion LLMs, Explained Simply

• 🍟 20 PyTorch Concepts, Explained Simply

• 🥯 Build and Train a Mixture-of-Experts (MoE) LLM from scratch

• 🍩 Building Your First AI Agent

• 🍓 Learn to train deep learning models on multiple GPUs

and so many more!

Join 'Into AI' premium today

9. Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

This research paper examines how different search tools impact LLM agents when handling retrieval-heavy tasks.

The authors conducted two experiments:

• Compare grep and vector retrieval on a 116-question sample from LongMemEval, using a custom agent harness (Chronos) and provider-native CLI harnesses (Claude Code, Codex, and Gemini CLI)

• Compare grep-only and vector-only retrieval while progressively mixing in additional unrelated conversation history, so that each query is embedded in more distracting material alongside the passages that matter.

Across Chronos and the provider CLIs, grep generally leads to higher accuracy than vector retrieval in the first experiment.

At the same time, overall scores still depend heavily on the agent harness and how the tool results are presented to the model, even when the underlying conversation data are the same.

Read more about this research using this link.

8. δ-mem: Efficient Online Memory for Large Language Models

δ-mem is a lightweight online memory system for LLMs that lets them reuse past information without expanding the context window or retraining the entire model.

δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the LLM’s attention computation during generation.

With only an 8 × 8 online memory state, δ-mem improves the average score to 1.10× that of the LLM backbone and 1.15× that of the strongest non-δ-mem memory baseline.

It achieves larger gains on memory-heavy benchmarks, reaching 1.31× on MemoryAgentBench and 1.20× on LoCoMo, while largely preserving general capabilities.

Read more about this research using this link.

Subscribe now

7. FutureSim: Replaying World Events to Evaluate Adaptive Agents

This research paper introduces FutureSim, a benchmark for testing whether AI agents can learn over time by replaying real-world news events in order and predicting future outcomes beyond their knowledge limits.

Agents can only use the news available up to each simulated date, update predictions as new information comes in, and are scored based on accuracy and Brier skill score.

The results show that the best agent’s accuracy is 25%, and many have worse Brier skill scores than making no prediction at all.

Read more about this research using this link.

6. World Action Models

World Action Models (WAMs) are the next step beyond Vision-Language-Action robot policies that, instead of directly mapping observations and language to actions, model future world states and actions together, giving embodied agents a form of predictive physical foresight.

This survey is the first systematic account of the WAMs landscape, clarifying key architectural paradigms and their trade-offs, and identifying open challenges and future opportunities for this rapidly evolving field.

Read more about this research using this link.

5. The Truth Lies Somewhere in the Middle (of the Generated Tokens)

The research paper suggests that, for autoregressive LLMs, the best semantic representation is often not the final generated token or a prompt token, but the mean-pooled hidden states across the generated tokens.

This finding is quantified through kernel alignment to reference spaces in language, vision, and protein domains, where results indicate that improvements from mean pooling are consistent with information being distributed across generated tokens rather than localized to a single position.

Furthermore, representations coming from generated tokens outperform those from prompt tokens.

Read more about this research using this link.

Subscribe now

4. MetaBackdoor: Exploiting Positional Encoding as a Backdoor Attack Surface in LLMs

The research paper presents MetaBackdoor, a new type of LLM backdoor attack that relies on positional information, particularly input or conversation length, as the trigger instead of suspicious words or hidden characters.

This means a model might act normally with ordinary text but switch to a harmful mode once the prompt or chat history reaches a particular length. This switch could lead to leaking system prompts, private context, or making harmful tool calls.

While current backdoor defenses typically search for odd input content, this attack shows that even clean, semantically normal inputs can trigger hidden behavior via the model’s positional encoding.

Read more about this research using this link.

3.LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

This research paper presents AutoTTS, a framework in which LLM agents autonomously discover better test-time scaling strategies, rather than relying on humans.

AutoTTS views test-time scaling as a controller search problem based on pre-collected reasoning traces. In this way, strategies can be tested cheaply without needing to call the LLM repeatedly.

Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy–cost tradeoff over strong manually designed baselines.

The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes.

Read more about this research using this link.

2. VGGT-Ω

The research paper presents VGGT-Ω, a larger and more efficient 3D reconstruction model for static and dynamic scenes.

It builds on the Visual Geometry Grounded Transformer (VGGT) by simplifying the design, using compact registers to share scene information across frames, and reducing training memory to about 30% of the previous model.

This allows it to train on significantly more data and achieve better reconstruction and camera estimation, including a reported 77% improvement in camera accuracy on Sintel.

Read more about this research using this link.

1. ELF: Embedded Language Flows

The paper introduces ELF (Embedded Language Flows), a new class of diffusion-based language model that generates text primarily in continuous embedding space instead of directly using discrete tokens.

It employs Flow Matching to clean up embeddings, removing noise and turning them into clear language representations, only converting them back into tokens at the final step.

ELF surpasses previous discrete and continuous diffusion language models, achieving better generation quality with fewer sampling steps and using 10X fewer training tokens.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter:

• 🍔 What makes DeekSeek-V4 so good?

• 🍕 Diffusion LLMs, Explained Simply

• 🍟 20 PyTorch Concepts, Explained Simply

• 🥯 Build and Train a Mixture-of-Experts (MoE) LLM from scratch

• 🍩 Building Your First AI Agent

• 🍓 Learn to train deep learning models on multiple GPUs

and so many more!

Join 'Into AI' premium today

DeepSeek recently released the preview of two models in its V4 series:

• DeepSeek-V4-Pro with 1.6T parameters (49B activated during inference)

• DeepSeek-V4-Flash with 284B parameters (13B activated during inference)

Both models are Mixtures-of-Experts (MoE) models and support a context length of 1 million tokens.

These models use multiple architectural and optimization updates, with the four main ones being:

1. A new hybrid Attention architecture that uses Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to reduce the memory bottleneck

2. Manifold-Constrained Hyper-Connections (mHC) to strengthen conventional residual connections, which improve the stability of signal propagation across the layers while preserving model expressivity

3. Muon Optimizer for faster convergence and greater training stability

These major updates (alongside many others) make these models highly efficient. DeepSeek-V4-Pro requires only 27% of the single-token inference FLOPs and 10% of the KV cache compared with DeepSeek-V3.2.

And according to the following post on X, if Uber had used DeepSeek instead of Claude, their 2026 AI budget would have lasted 7 years rather than only 4 months. That’s incredibly efficient!

The magic of Mixture-of-Experts

In an LLM, the feed-forward network (FFN) in the Transformer block is often the most computationally demanding and holds the largest number of parameters.

This is why a dense feed-forward network is frequently replaced by a Mixture-of-Experts (MoE) feed-forward network. Mixture-of-Experts network improves computational efficiency by activating only a subset of expert modules during inference.

DeepSeek models from V2 onwards (including V4) use DeepSeekMoE, which splits the FFN into many smaller experts and routes each token to only a few of them (Routed Experts), while keeping some experts active for all tokens (Shared Experts). This significantly reduces its computational cost.

But given that V4 supports a 1M-token context window (compared to V3.2’s 128K), the attention mechanism becomes a significant bottleneck at inference.

This is where the new attention architecture innovated by DeepSeek-V4 shines! V4 models use a hybrid attention mechanism combining:

• Compressed Sparse Attention (CSA)

• Heavily Compressed Attention (HCA)

To understand how these attention mechanisms work and improve V4's efficiency so well, we will have to understand them from the ground up. Let’s start with the very basics and build from there, step by step.

What is Attention?

The (Self) Attention mechanism in a Transformer-based LLM helps the model understand how each word/ token relates to every other word/ token in a text sequence.

Given the embedding of each token:

• It is first projected into Query (Q), Key (K), and Value (V) vectors.

• The query vector is compared (technically via the dot product) with keys generated from itself and from previous tokens. This comparison produces attention scores that measure how closely the tokens are related to this one.

• The scores are scaled, masked to block a token from attending to future tokens, passed through softmax, and then used to compute a weighted sum of the value vectors.

• This weighted sum is the token’s context-aware representation, called Masked and Scaled Dot-Product Self-Attention.

Since every token is attending to itself and all the previous tokens, for an input sequence length of N, the attention mechanism processing it has an O(N²) computational complexity. This quadratic complexity isn’t ideal when processing long text sequences.

What is Multi-head Attention?

To improve performance, instead of performing attention operations once, these are done several times in parallel using different learned projections of the same token embeddings.

Each unit of attention calculation in this case is called a Head, and the mechanism is called Multi-head Attention (MHA).

Each attention head has its own Query, Key, and Value projections, and each head computes masked self-attention independently. The outputs of all heads are then concatenated into a single representation.

The use of multiple heads helps capture different kinds of semantic relationships between the tokens.

There are two efficient types of MHA that reduce the number of Key/Value projections stored and used during inference. These are:

• Multi-Query Attention (MQA), where each attention head has its own Query, but all Query heads share the same Key and Value.

• Grouped-Query Attention (GQA), where Queries are split into groups, and each group of Query heads shares one Key and Value head.

Both MQA and GQA make inference faster and cheaper, but at the expense of generation quality.

Attention mechanisms are made much more efficient using a KV cache.

KV cache to the rescue

Note that at each step of text generation/ inference, we need to calculate the K and V vectors for all previous tokens.

How about we store the Key and Value vectors from previous tokens, and reuse them later? That’s exactly what a KV cache does.

When using a KV cache:

• For each generated token, we store its K and V vectors in the cache.

• Then, in the next token generation step, we compute only the new token’s Q, K, and V.

• Next, we reuse the cached KV of previous tokens, with the new token’s Q attending to both the cached KV and its newly calculated KV. This saves on lots of compute during inference.

Towards Multi-head Latent Attention

Released in 2024, DeepSeek-V2 introduced an efficient version of attention called Multi-head Latent Attention (MLA) that improves KV cache by not caching full Keys and Values. Instead, it stores a much smaller latent representation (one latent KV entry per token) and reconstructs the full KV information from it when needed.

This reduces the memory required to store the full KV cache, making the inference much faster. MLA is therefore also used in the later model, DeepSeek-V3.

Check out the following illustration to see how MLA compares to the other attention variants.

The complete architecture of MLA with the latent KV representation is shown below. I have also written a deep dive post on how it works, which you can read here.

Sliding Window to improve Attention further

Till now, we have learned how:

• KV caching helps reduce the compute and memory requirements for attention at inference

• Building on this and using a latent representation of the KV cache in MLA helps further reduce the memory requirements

There’s another way to reduce the compute and memory requirements of attention. This is by reducing the number of previous tokens that a token needs to attend to, which brings us to the Sliding window attention (SWA).

It was introduced in the ‘Longformer’ paper in 2020. In SWA, each token attends only to a fixed-size window of previous tokens, rather than all previous ones.

We previously discussed how the full Attention has an O(N²) computational complexity, as every token attends to itself and all the previous tokens, for an input sequence length of N.

With SWA, for an input sequence of N and window size (fixed-sized window of previous tokens that a token attends to) of w, the computational complexity goes down to O(Nw).

Although it is efficient, since each token attends to a window of nearby tokens, some long-range information in language modeling is lost. Hence, it is often used in combination with:

• Full attention layers

• Global window (where some tokens are allowed to attend to the entire sequence of previous tokens)

SWA can also use a dilated window pattern, where each token attends to nearby previous tokens but skips some positions in a regular pattern. This helps a token attend to more distant previous tokens at roughly the same compute cost.

A popular example is the GPT-oss family of models, which uses SWA with a window size of 128 tokens in alternating layers with full attention (GQA with a configuration of 64 query heads and 8 KV heads).

Moving to DeepSeek Sparse Attention

Read more

1. ProgramBench: Can Language Models Rebuild Programs From Scratch?

ProgramBench is a benchmark by Meta researchers that measures the ability of software engineering agents to develop software holistically.

Given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable’s behavior.

Evaluation of 9 frontier LLMs across 200 tasks, ranging from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter, shows that none fully resolve any task, and the best model passes 95% of tests on only 3% of tasks.

Models favor monolithic, single-file implementations, which are very different from how humans write code.

Read more about this research using this link.

Subscribe now

2. AI Co-Mathematician

This research from Google DeepMind introduces the AI co-mathematician, a workbench that helps mathematicians use AI agents for open-ended research.

The AI co-mathematician is optimized for mathematical workflows, ranging from idea generation and literature searches to computational exploration, theorem proving, and theory building.

Early tests show the AI co-mathematician helped researchers solve open mathematical problems, find new research directions, and discover overlooked literature references.

Alongside this, the AI co-mathematician scored 48% on FrontierMath Tier 4, a hard problem-solving benchmark. This is the highest score of all AI systems evaluated.

Read more about this research using this link.

3. Continuous Latent Diffusion Language Model

This research paper presents Cola DLM, a continuous latent diffusion language model that generates text by compressing global semantic structure into a continuous latent space (using a Text VAE) and decoding it into text (using a block-causal DiT).

Cola DLM proves effective and shows strong scaling behavior for text generation across 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs.

Read more about this research using this link.

Subscribe now

4. Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

This research paper finds that the geometry of a model’s internal activations causally shapes its outputs rather than merely correlating with them.

The authors introduce Manifold steering, where interventions move along the curved manifold of natural activations rather than in a straight Euclidean direction.

Across language-model reasoning tasks, in-context learning tasks, and a video world model, steering along the activation manifold produces output behaviors that stay on the model’s natural “behavior manifold.” In contrast, ordinary linear steering often moves through unnatural off-manifold areas, resulting in distorted outputs.

Researchers also show that this relationship works in reverse. Optimizing for desired behavioral trajectories recovers curved activation paths that align with the internal manifold.

This means that reliable model control should focus less on finding a single steering direction and more on understanding and following the model’s underlying representational geometry.

Read more about this research using this link.

5.MolmoAct2

This research paper presents MolmoAct2, a fully open VLA robot control model designed for real-world deployment and use.

It builds on MolmoAct with a new embodied-reasoning backbone, Molmo2-ER, which is trained on 3.3 million samples with a specialize-then-rehearse recipe.

The release also includes:

• Three new robotics datasets

• MolmoAct2-FAST Tokenizer, an open-weight, open-data action tokenizer trained on millions of trajectories across five embodiments

• A new KV-conditioned VLA architecture

• MolmoAct2-Think, an adaptive-depth reasoning variant that maintains spatial reasoning while massively reducing latency

MolmoAct2 outperforms strong baselines, including π0.5, while Molmo2-ER surpasses GPT-5 and Gemini Robotics ER-1.5 across 13 embodied-reasoning benchmarks.

Read more about this research using this link.

6. Model Spec Midtraining

The research paper presents Model Spec Midtraining (MSM), a method that improves how alignment training generalizes by teaching a model its intended behavior rules before fine-tuning.

Instead of relying only on examples of aligned behavior, MSM trains models on synthetic documents that explain the Model Spec or constitution. This approach allows later demonstrations to be interpreted through that framework.

The authors demonstrate that the same fine-tuning data can generalize in very different ways based on the specific spec taught during mid-training. They show that MSM can reduce safety-related failures. For instance, agentic misalignment in Qwen3-32B decreases from 54% to 7%, which surpasses a deliberative alignment baseline of 14%.

The authors also show that specs perform better when they clarify the values behind the rules and provide specific guidance rather than vague suggestions.

Read more about this research using this link.

Subscribe now

7. Mamoda2.5

The research paper presents Mamoda2.5, a unified AR–Diffusion framework that integrates multimodal understanding and generation within a single architecture.

The model’s Diffusion Transformer backbone uses a fine-grained Mixture-of-Experts (MoE) architecture (128 experts, Top-8 routing), resulting in a 25B-parameter model that activates only 3B parameters, significantly reducing training costs while scaling up model capacity.

Mamoda2.5 achieves top-tier generation performance on VBench 2.0 and sets a new record in video editing quality, beating evaluated open-source models and matching the performance of current top-tier proprietary models, including the Kling O1 on OpenVE-Bench.

Compared to open-source baselines, Mamoda2.5 achieves up to 95.9× faster video editing inference. This is due to a joint few-step distillation and reinforcement learning framework that compresses the 30-step editing model into a 4-step model, greatly accelerating model inference.

In real-world applications, Mamoda2.5 has been successfully deployed for content moderation and creative restoration tasks in advertising scenarios, achieving a 98% success rate in an internal advertising video editing scenario.

Read more about this research using this link.

8. Superintelligent Retrieval Agent

This research paper introduces SuperIntelligent Retrieval Agent (SIRA), a retrieval agent that replaces slow multi-round searches with one effective, corpus-aware lexical query.

Instead of simply adding related words to a user query, SIRA combines an LLM with document-frequency statistics to select terms that help differentiate likely evidence from misleading corpus matches.

It improves documents offline by adding missing vocabulary, predicts terms that could serve as evidence but are not included in the query, filters out weak or overly common terms, and then performs a single weighted BM25 retrieval call combining the original query with the validated expansion.

In tests across 10 BEIR and QA benchmarks, SIRA outperformed dense retrievers and multi-round retrieval systems while remaining training-free, efficient, and interpretable.

Read more about this research using this link.

9. Nonsense Helps

This research paper presents Lorem Perturbation for Exploration (LoPE), a simple RL training technique for reasoning models that adds random Lorem Ipsum-style “nonsense” text to the prompt and resamples when GRPO gets stuck because all the sampled answers to a difficult question fail.

These irrelevant changes to the prompt can lead the model down different reasoning paths and improve exploration more than just resampling the original prompt.

Experiments across 1.7B, 4B, and 7B models show that LoPE significantly outperforms resampling with the original prompts. Further analysis reveals that other Latin-based random sequences with low perplexity are also effective perturbations.

Read more about this research using this link.

10. Retrieval from Within

This research presents INTRA (INTrinsic Retrieval via Attention), a RAG-style approach in which an attention-based encoder-decoder retrieves evidence from its precomputed internal representations rather than relying on a separate external retriever.

The decoder attention queries score the encoded evidence chunks, which are then reused directly as context for generation.

INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality on question-answering benchmarks.

This shows that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter and 100x your AI engineering skills.

Join 'Into AI' premium today

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.