Into AI

9. Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

This research paper examines how different search tools impact LLM agents when handling retrieval-heavy tasks.

The authors conducted two experiments:

Compare grep and vector retrieval on a 116-question sample from LongMemEval, using a custom agent harness (Chronos) and provider-native CLI harnesses (Claude Code, Codex, and Gemini CLI)
Compare grep-only and vector-only retrieval while progressively mixing in additional unrelated conversation history, so that each query is embedded in more distracting material alongside the passages that matter.

Across Chronos and the provider CLIs, grep generally leads to higher accuracy than vector retrieval in the first experiment.

At the same time, overall scores still depend heavily on the agent harness and how the tool results are presented to the model, even when the underlying conversation data are the same.

Read more about this research using this link.

8. δ-mem: Efficient Online Memory for Large Language Models

δ-mem is a lightweight online memory system for LLMs that lets them reuse past information without expanding the context window or retraining the entire model.

δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the LLM’s attention computation during generation.

With only an 8 × 8 online memory state, δ-mem improves the average score to 1.10× that of the LLM backbone and 1.15× that of the strongest non-δ-mem memory baseline.

It achieves larger gains on memory-heavy benchmarks, reaching 1.31× on MemoryAgentBench and 1.20× on LoCoMo, while largely preserving general capabilities.

Read more about this research using this link.

7. FutureSim: Replaying World Events to Evaluate Adaptive Agents

This research paper introduces FutureSim, a benchmark for testing whether AI agents can learn over time by replaying real-world news events in order and predicting future outcomes beyond their knowledge limits.

Agents can only use the news available up to each simulated date, update predictions as new information comes in, and are scored based on accuracy and Brier skill score.

The results show that the best agent’s accuracy is 25%, and many have worse Brier skill scores than making no prediction at all.

Read more about this research using this link.

6. World Action Models

World Action Models (WAMs) are the next step beyond Vision-Language-Action robot policies that, instead of directly mapping observations and language to actions, model future world states and actions together, giving embodied agents a form of predictive physical foresight.

This survey is the first systematic account of the WAMs landscape, clarifying key architectural paradigms and their trade-offs, and identifying open challenges and future opportunities for this rapidly evolving field.

Read more about this research using this link.

5. The Truth Lies Somewhere in the Middle (of the Generated Tokens)

The research paper suggests that, for autoregressive LLMs, the best semantic representation is often not the final generated token or a prompt token, but the mean-pooled hidden states across the generated tokens.

This finding is quantified through kernel alignment to reference spaces in language, vision, and protein domains, where results indicate that improvements from mean pooling are consistent with information being distributed across generated tokens rather than localized to a single position.

Furthermore, representations coming from generated tokens outperform those from prompt tokens.

Read more about this research using this link.

4. MetaBackdoor: Exploiting Positional Encoding as a Backdoor Attack Surface in LLMs

The research paper presents MetaBackdoor, a new type of LLM backdoor attack that relies on positional information, particularly input or conversation length, as the trigger instead of suspicious words or hidden characters.

This means a model might act normally with ordinary text but switch to a harmful mode once the prompt or chat history reaches a particular length. This switch could lead to leaking system prompts, private context, or making harmful tool calls.

While current backdoor defenses typically search for odd input content, this attack shows that even clean, semantically normal inputs can trigger hidden behavior via the model’s positional encoding.

Read more about this research using this link.

3.LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

This research paper presents AutoTTS, a framework in which LLM agents autonomously discover better test-time scaling strategies, rather than relying on humans.

AutoTTS views test-time scaling as a controller search problem based on pre-collected reasoning traces. In this way, strategies can be tested cheaply without needing to call the LLM repeatedly.

Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy–cost tradeoff over strong manually designed baselines.

The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes.

Read more about this research using this link.

2. VGGT-Ω

The research paper presents VGGT-Ω, a larger and more efficient 3D reconstruction model for static and dynamic scenes.

It builds on the Visual Geometry Grounded Transformer (VGGT) by simplifying the design, using compact registers to share scene information across frames, and reducing training memory to about 30% of the previous model.

This allows it to train on significantly more data and achieve better reconstruction and camera estimation, including a reported 77% improvement in camera accuracy on Sintel.

Read more about this research using this link.

1. ELF: Embedded Language Flows

The paper introduces ELF (Embedded Language Flows), a new class of diffusion-based language model that generates text primarily in continuous embedding space instead of directly using discrete tokens.

It employs Flow Matching to clean up embeddings, removing noise and turning them into clear language representations, only converting them back into tokens at the final step.

ELF surpasses previous discrete and continuous diffusion language models, achieving better generation quality with fewer sampling steps and using 10X fewer training tokens.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

What makes DeekSeek-V4 so good?

Dr. Ashish Bamania — Fri, 15 May 2026 17:02:06 GMT

DeepSeek recently released the preview of two models in its V4 series:

DeepSeek-V4-Pro with 1.6T parameters (49B activated during inference)
DeepSeek-V4-Flash with 284B parameters (13B activated during inference)

Both models are Mixtures-of-Experts (MoE) models and support a context length of 1 million tokens.

These models use multiple architectural and optimization updates, with the four main ones being:

A new hybrid Attention architecture that uses Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to reduce the memory bottleneck
Manifold-Constrained Hyper-Connections (mHC) to strengthen conventional residual connections, which improve the stability of signal propagation across the layers while preserving model expressivity
Muon Optimizer for faster convergence and greater training stability

These major updates (alongside many others) make these models highly efficient. DeepSeek-V4-Pro requires only 27% of the single-token inference FLOPs and 10% of the KV cache compared with DeepSeek-V3.2.

Benchmark performance, inference FLOPs, and KV cache size of DeepSeek-V4 and baselines (Source)

And according to the following post on X, if Uber had used DeepSeek instead of Claude, their 2026 AI budget would have lasted 7 years rather than only 4 months. That’s incredibly efficient!

The magic of Mixture-of-Experts

In an LLM, the feed-forward network (FFN) in the Transformer block is often the most computationally demanding and holds the largest number of parameters.

This is why a dense feed-forward network is frequently replaced by a Mixture-of-Experts (MoE) feed-forward network. Mixture-of-Experts network improves computational efficiency by activating only a subset of expert modules during inference.

DeepSeek models from V2 onwards (including V4) use DeepSeekMoE, which splits the FFN into many smaller experts and routes each token to only a few of them (Routed Experts), while keeping some experts active for all tokens (Shared Experts). This significantly reduces its computational cost.

Architecture of DeepSeekMoE (Source)

But given that V4 supports a 1M-token context window (compared to V3.2’s 128K), the attention mechanism becomes a significant bottleneck at inference.

This is where the new attention architecture innovated by DeepSeek-V4 shines! V4 models use a hybrid attention mechanism combining:

Compressed Sparse Attention (CSA)
Heavily Compressed Attention (HCA)

To understand how these attention mechanisms work and improve V4's efficiency so well, we will have to understand them from the ground up. Let’s start with the very basics and build from there, step by step.

What is Attention?

The (Self) Attention mechanism in a Transformer-based LLM helps the model understand how each word/ token relates to every other word/ token in a text sequence.

Given the embedding of each token:

It is first projected into Query (Q), Key (K), and Value (V) vectors.
The query vector is compared (technically via the dot product) with keys generated from itself and from previous tokens. This comparison produces attention scores that measure how closely the tokens are related to this one.
The scores are scaled, masked to block a token from attending to future tokens, passed through softmax, and then used to compute a weighted sum of the value vectors.
This weighted sum is the token’s context-aware representation, called Masked and Scaled Dot-Product Self-Attention.

Since every token is attending to itself and all the previous tokens, for an input sequence length of N, the attention mechanism processing it has an O(N²) computational complexity. This quadratic complexity isn’t ideal when processing long text sequences.

Calculations taking place in Self-attention

What is Multi-head Attention?

To improve performance, instead of performing attention operations once, these are done several times in parallel using different learned projections of the same token embeddings.

Each unit of attention calculation in this case is called a Head, and the mechanism is called Multi-head Attention (MHA).

Each attention head has its own Query, Key, and Value projections, and each head computes masked self-attention independently. The outputs of all heads are then concatenated into a single representation.

The use of multiple heads helps capture different kinds of semantic relationships between the tokens.

There are two efficient types of MHA that reduce the number of Key/Value projections stored and used during inference. These are:

Multi-Query Attention (MQA), where each attention head has its own Query, but all Query heads share the same Key and Value.
Grouped-Query Attention (GQA), where Queries are split into groups, and each group of Query heads shares one Key and Value head.

Both MQA and GQA make inference faster and cheaper, but at the expense of generation quality.

MHA vs. GQA vs MQA (Source)

Attention mechanisms are made much more efficient using a KV cache.

KV cache to the rescue

Note that at each step of text generation/ inference, we need to calculate the K and V vectors for all previous tokens.

How about we store the Key and Value vectors from previous tokens, and reuse them later? That’s exactly what a KV cache does.

When using a KV cache:

For each generated token, we store its K and V vectors in the cache.
Then, in the next token generation step, we compute only the new token’s Q, K, and V.
Next, we reuse the cached KV of previous tokens, with the new token’s Q attending to both the cached KV and its newly calculated KV. This saves on lots of compute during inference.

Towards Multi-head Latent Attention

Released in 2024, DeepSeek-V2 introduced an efficient version of attention called Multi-head Latent Attention (MLA) that improves KV cache by not caching full Keys and Values. Instead, it stores a much smaller latent representation (one latent KV entry per token) and reconstructs the full KV information from it when needed.

This reduces the memory required to store the full KV cache, making the inference much faster. MLA is therefore also used in the later model, DeepSeek-V3.

Check out the following illustration to see how MLA compares to the other attention variants.

MHA vs GQA vs MQA vs MLA (Source)

The complete architecture of MLA with the latent KV representation is shown below. I have also written a deep dive post on how it works, which you can read here.

Architecture of Multi-head Latent Attention (MLA) (Source)

Sliding Window to improve Attention further

Till now, we have learned how:

KV caching helps reduce the compute and memory requirements for attention at inference
Building on this and using a latent representation of the KV cache in MLA helps further reduce the memory requirements

There’s another way to reduce the compute and memory requirements of attention. This is by reducing the number of previous tokens that a token needs to attend to, which brings us to the Sliding window attention (SWA).

It was introduced in the ‘Longformer’ paper in 2020. In SWA, each token attends only to a fixed-size window of previous tokens, rather than all previous ones.

We previously discussed how the full Attention has an O(N²) computational complexity, as every token attends to itself and all the previous tokens, for an input sequence length of N.

With SWA, for an input sequence of N and window size (fixed-sized window of previous tokens that a token attends to) of w, the computational complexity goes down to O(Nw).

Although it is efficient, since each token attends to a window of nearby tokens, some long-range information in language modeling is lost. Hence, it is often used in combination with:

Full attention layers
Global window (where some tokens are allowed to attend to the entire sequence of previous tokens)

SWA can also use a dilated window pattern, where each token attends to nearby previous tokens but skips some positions in a regular pattern. This helps a token attend to more distant previous tokens at roughly the same compute cost.

Full attention vs Sliding window attention and its variants (Source)

A popular example is the GPT-oss family of models, which uses SWA with a window size of 128 tokens in alternating layers with full attention (GQA with a configuration of 64 query heads and 8 KV heads).

Moving to DeepSeek Sparse Attention

This Week In AI Research (3-9 May 26) 🗓️

Dr. Ashish Bamania — Mon, 11 May 2026 12:44:43 GMT

1. ProgramBench: Can Language Models Rebuild Programs From Scratch?

ProgramBench is a benchmark by Meta researchers that measures the ability of software engineering agents to develop software holistically.

Given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable’s behavior.

Evaluation of 9 frontier LLMs across 200 tasks, ranging from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter, shows that none fully resolve any task, and the best model passes 95% of tests on only 3% of tasks.

Models favor monolithic, single-file implementations, which are very different from how humans write code.

Read more about this research using this link.

2. AI Co-Mathematician

This research from Google DeepMind introduces the AI co-mathematician, a workbench that helps mathematicians use AI agents for open-ended research.

The AI co-mathematician is optimized for mathematical workflows, ranging from idea generation and literature searches to computational exploration, theorem proving, and theory building.

Early tests show the AI co-mathematician helped researchers solve open mathematical problems, find new research directions, and discover overlooked literature references.

Alongside this, the AI co-mathematician scored 48% on FrontierMath Tier 4, a hard problem-solving benchmark. This is the highest score of all AI systems evaluated.

Read more about this research using this link.

3. Continuous Latent Diffusion Language Model

This research paper presents Cola DLM, a continuous latent diffusion language model that generates text by compressing global semantic structure into a continuous latent space (using a Text VAE) and decoding it into text (using a block-causal DiT).

Cola DLM proves effective and shows strong scaling behavior for text generation across 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs.

Read more about this research using this link.

4. Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

This research paper finds that the geometry of a model’s internal activations causally shapes its outputs rather than merely correlating with them.

The authors introduce Manifold steering, where interventions move along the curved manifold of natural activations rather than in a straight Euclidean direction.

Across language-model reasoning tasks, in-context learning tasks, and a video world model, steering along the activation manifold produces output behaviors that stay on the model’s natural “behavior manifold.” In contrast, ordinary linear steering often moves through unnatural off-manifold areas, resulting in distorted outputs.

Researchers also show that this relationship works in reverse. Optimizing for desired behavioral trajectories recovers curved activation paths that align with the internal manifold.

This means that reliable model control should focus less on finding a single steering direction and more on understanding and following the model’s underlying representational geometry.

Read more about this research using this link.

5.MolmoAct2

This research paper presents MolmoAct2, a fully open VLA robot control model designed for real-world deployment and use.

It builds on MolmoAct with a new embodied-reasoning backbone, Molmo2-ER, which is trained on 3.3 million samples with a specialize-then-rehearse recipe.

The release also includes:

Three new robotics datasets
MolmoAct2-FAST Tokenizer, an open-weight, open-data action tokenizer trained on millions of trajectories across five embodiments
A new KV-conditioned VLA architecture
MolmoAct2-Think, an adaptive-depth reasoning variant that maintains spatial reasoning while massively reducing latency

MolmoAct2 outperforms strong baselines, including π0.5, while Molmo2-ER surpasses GPT-5 and Gemini Robotics ER-1.5 across 13 embodied-reasoning benchmarks.

Read more about this research using this link.

6. Model Spec Midtraining

The research paper presents Model Spec Midtraining (MSM), a method that improves how alignment training generalizes by teaching a model its intended behavior rules before fine-tuning.

Instead of relying only on examples of aligned behavior, MSM trains models on synthetic documents that explain the Model Spec or constitution. This approach allows later demonstrations to be interpreted through that framework.

The authors demonstrate that the same fine-tuning data can generalize in very different ways based on the specific spec taught during mid-training. They show that MSM can reduce safety-related failures. For instance, agentic misalignment in Qwen3-32B decreases from 54% to 7%, which surpasses a deliberative alignment baseline of 14%.

The authors also show that specs perform better when they clarify the values behind the rules and provide specific guidance rather than vague suggestions.

Read more about this research using this link.

7. Mamoda2.5

The research paper presents Mamoda2.5, a unified AR–Diffusion framework that integrates multimodal understanding and generation within a single architecture.

The model’s Diffusion Transformer backbone uses a fine-grained Mixture-of-Experts (MoE) architecture (128 experts, Top-8 routing), resulting in a 25B-parameter model that activates only 3B parameters, significantly reducing training costs while scaling up model capacity.

Mamoda2.5 achieves top-tier generation performance on VBench 2.0 and sets a new record in video editing quality, beating evaluated open-source models and matching the performance of current top-tier proprietary models, including the Kling O1 on OpenVE-Bench.

Compared to open-source baselines, Mamoda2.5 achieves up to 95.9× faster video editing inference. This is due to a joint few-step distillation and reinforcement learning framework that compresses the 30-step editing model into a 4-step model, greatly accelerating model inference.

In real-world applications, Mamoda2.5 has been successfully deployed for content moderation and creative restoration tasks in advertising scenarios, achieving a 98% success rate in an internal advertising video editing scenario.

Read more about this research using this link.

8. Superintelligent Retrieval Agent

This research paper introduces SuperIntelligent Retrieval Agent (SIRA), a retrieval agent that replaces slow multi-round searches with one effective, corpus-aware lexical query.

Instead of simply adding related words to a user query, SIRA combines an LLM with document-frequency statistics to select terms that help differentiate likely evidence from misleading corpus matches.

It improves documents offline by adding missing vocabulary, predicts terms that could serve as evidence but are not included in the query, filters out weak or overly common terms, and then performs a single weighted BM25 retrieval call combining the original query with the validated expansion.

In tests across 10 BEIR and QA benchmarks, SIRA outperformed dense retrievers and multi-round retrieval systems while remaining training-free, efficient, and interpretable.

Read more about this research using this link.

9. Nonsense Helps

This research paper presents Lorem Perturbation for Exploration (LoPE), a simple RL training technique for reasoning models that adds random Lorem Ipsum-style “nonsense” text to the prompt and resamples when GRPO gets stuck because all the sampled answers to a difficult question fail.

These irrelevant changes to the prompt can lead the model down different reasoning paths and improve exploration more than just resampling the original prompt.

Experiments across 1.7B, 4B, and 7B models show that LoPE significantly outperforms resampling with the original prompts. Further analysis reveals that other Latin-based random sequences with low perplexity are also effective perturbations.

Read more about this research using this link.

10. Retrieval from Within

This research presents INTRA (INTrinsic Retrieval via Attention), a RAG-style approach in which an attention-based encoder-decoder retrieves evidence from its precomputed internal representations rather than relying on a separate external retriever.

The decoder attention queries score the encoded evidence chunks, which are then reused directly as context for generation.

INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality on question-answering benchmarks.

This shows that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter and 100x your AI engineering skills.

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

This Week In AI Research (26 April - 2 May 26) 🗓️

Dr. Ashish Bamania — Wed, 06 May 2026 15:21:15 GMT

1. Thinking with Visual Primitives

This research paper from DeepSeek introduces Thinking with Visual Primitives, a new method that enables multimodal LLMs to reason using visual markers, such as points and bounding boxes. Since reasoning with just language produces poor performance on precise spatial tasks, this technique lets models “point” to objects or areas as they reason.

The framework is built on a highly optimized architecture with extreme visual efficiency. It achieves frontier performance on challenging visual reasoning tasks such as counting, maze navigation, and path tracing, matching or exceeding models like GPT-5.4, Claude Sonnet 4.6, and Gemini-3-Flash.

Read more about this research using this link.

2. Recursive Multi-Agent Systems

This research paper introduces Recursive MAS, a framework in which multiple AI agents collaborate by passing and refining latent states rather than just communicating via text.

The framework treats the entire agent team as a loop for recursive computation using a lightweight RecursiveLink module and an inner-outer loop learning algorithm for iterative whole-system co-optimization through shared gradient-based credit assignment across recursion rounds.

In 9 benchmarks across math, science, medicine, search, and coding, the authors report an average accuracy rise of 8.3%, an end-to-end inference speedup of 1.2 to 2.4 times, and a token reduction of 34.6% to 75.6% compared to strong baselines.

Read more about this research using this link.

3. Agentic Harness Engineering

This research paper introduces Agentic Harness Engineering (AHE), a system that automatically improves the harness around coding agents, rather than relying on manual tuning.

AHE observes when agents fail, summarizes long trajectories into usable evidence, suggests harness edits, and verifies if the predicted benefits of each edit actually work.

Results show that after 10 iterations, it boosts pass@1 performance on Terminal-Bench 2 from 69.7% to 77.0%. This improvement surpasses human-designed harnesses in Codex-CLI (71.9%) and other self-evolving systems such as ACE and TF-GRPO.

The improved harness functions well without needing retraining. It uses fewer tokens on SWE-bench-verified tasks and provides significant performance gains on Terminal-Bench 2 across various model families.

Read more about this research using this link.

4. Incompressible Knowledge Probe

This research paper presents Incompressible Knowledge Probes (IKPs), a benchmark of 1,400 obscure factual questions used to estimate the hidden parameter count of black-box LLMs.

The idea is that storing F facts requires at least F/(bits per parameter) weights, so measuring how much a model knows lower-bounds how many parameters it has.

The author tests IKP accuracy on 89 open-weight models and finds a strong log-linear connection with parameter count. Then, the author applies this to estimate the effective knowledge capacity and parameter count of proprietary models.

The author also shows that, while reasoning benchmarks may saturate, the factual capacity of models still scales steadily with total parameters.

Read more about this research using this link.

5. MotuBrain

This research paper introduces MotuBrain, a unified robot “world action model” that learns how the world changes and what actions a robot should take.

Instead of creating separate systems for policy learning, video prediction, inverse dynamics, and action generation, MotuBrain integrates them into a single architecture.

It uses multimodal data, including video-only, task-agnostic, and cross-embodiment robot data, and introduces unified multiview modeling, an independent text stream for stronger language-action coupling, a shared cross-embodiment action representation, and an efficient post-training and deployment recipe for long-horizon real-world control.

It also employs heavy inference optimization to operate in real time, achieving up to 11 Hz inference and a more than 50x speedup over the baseline.

MotuBrain achieves 95.8% and 96.1% average success on RoboTwin 2.0 under clean and randomized settings, respectively, attains the strongest reported EWMScore in WorldArena comparison, and adapts to new humanoid embodiments with only 50-100 trajectories.

Read more about this research using this link.

6. Latent-GRPO

This research paper introduces Latent-GRPO, a reinforcement-learning algorithm that helps LLMs reason in the latent space rather than producing lengthy chain-of-thought text.

Applying GRPO directly to latent reasoning can be unstable, but Latent-GRPO addresses these issues by combining:

Invalid sample advantage masking
One-sided noise sampling
Optimal correct-path first-token selection

Latent-GRPO improves Pass@1 scores on both easy and hard reasoning benchmarks and uses reasoning chains that are about 3-4x shorter than those of explicit GRPO.

Read more about this research using this link.

7. Contextual Agentic Memory is a Memo, Not True Memory

The authors of this research paper argue that current agentic memory systems (vector stores, retrieval-augmented generation, scratchpads, and context-window management) do not implement memory but rather perform lookups.

They allow agents to retrieve past notes, but they do not update the model’s weights or develop lasting expertise. The authors claim this creates a generalization ceiling, where agents can reuse similar past cases but struggle with genuinely new combinations that require internalized rules.

They also warn that persistent external memory increases security risks because poisoned or injected content can keep resurfacing in future sessions.

Useful agent memory should combine fast external recall with slower weight-based consolidation, similar to how humans turn experiences into expertise.

Read more about this research using this link.

8. Tuna-2

This research paper presents Tuna-2, a unified multimodal model that replaces the typical pretrained vision encoder and VAE with simple raw pixel patch embeddings. This enables image understanding and generation within a single end-to-end system, thereby avoiding mismatches between separate visual representations.

Tuna-2 achieves state-of-the-art performance in multimodal benchmarks, demonstrating that unified pixel-space modeling can fully compete with latent-space approaches for high-quality image generation.

Its encoder-free design achieves stronger multimodal understanding at scale, particularly for tasks that require fine-grained visual perception.

Read more about this research using this link.

9. GLM-5V-Turbo

This research paper presents GLM-5V-Turbo, Z.AI’s first multimodal coding foundation model, built for vision-based coding tasks.

It can natively process multimodal inputs such as images, video, and text, excels at long-horizon planning, complex coding, and action execution, and is deeply optimized for agent workflows.

Read more about this research using this link.

10. Let ViT Speak: Generative Language-Image Pre-training

The research paper presents Generative Language-Image Pre-training (GenLIP), a minimalist method to pretrain Vision Transformers for multimodal LLMs.

GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder.

This offers three main advantages:

A single transformer jointly models visual and textual tokens
It scales effectively with both data and model size
It achieves competitive or superior results across diverse multimodal benchmarks.

Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data.

After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.

Read more about this research using this link.

Share this article with others and earn some referral rewards. ❤️

Join the paid tier today to get access to all posts in this newsletter.

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

Speculative Decoding, Simply Explained

Dr. Ashish Bamania — Thu, 30 Apr 2026 18:42:52 GMT

LLM inference is slow, memory-heavy, and computationally expensive.

LLMs are autoregressive and produce one token at a time, and the LLMs in use today also contain multiple trillion parameters. This means that producing just one token from these requires trillions of computations!

In 2022, Google published a research paper called ‘Fast Inference from Transformers via Speculative Decoding’ where they introduced an inference technique called Speculative Decoding.

This technique claimed to increase inference speeds by 2-3x without compromising LLM output quality, and since then, it has been used by AI overviews in Google Search (and every other modern LLM application) to speed up inference.

Source

What really makes Inference slow?

You might assume that performing trillions of computations to produce a single token is what limits the inference speed, but this is a big misconception!

GPUs and TPUs in use today are powerful parallel computing machines and can perform hundreds of trillions of operations per second. This is often more than 100x higher than what’s needed to generate a token through an LLM.

So what’s the real bottleneck?

To produce a single token, the GPU must read all the model’s parameters from high-bandwidth memory (HBM) into the compute units. This process is painfully slow and leaves the compute units sitting idle while they wait for weights to arrive.

This makes the process memory-bound rather than compute-bound.

(LLM inference is very different from LLM training, where you load the parameters once and reuse them across thousands of operations on a large batch of training samples.)

Insight 1: This means that we have all that extra compute sitting there to be used.

Is predicting all tokens really that tough?

Consider this prompt and response pair:

"Tell 2 statements about Jupiter and its moons. Jupiter is the largest planet in our solar system. It has dozens of moons, including Ganymede, the largest moon in the solar system."

In this statement, considering words as tokens, some tokens, such as “Jupiter”, “is”, and “the”, are easier to predict/ generate, while others, such as “Ganymede”, are harder.

This is because the LLM has already seen “Jupiter” in its prompt, and in the positions that follow, "is" and "the" have few competing alternatives.

For “Ganymede”, its preceding context, "dozens of moons, including” leads to many possible valid completions.

Insight 2: We do not always need a powerful LLM to produce tokens that are easier to predict.

How does Speculative Decoding use these ideas?

This Week In AI Research (19-25 April 26) 🗓️

Dr. Ashish Bamania — Wed, 29 Apr 2026 01:10:35 GMT

1. DeepSeek-V4

This research paper introduces the preview version of the DeepSeek-V4 series with two Mixture-of-Experts (MoE) language models:

DeepSeek-V4-Pro with 1.6T parameters (49B activated)
DeepSeek-V4-Flash with 284B parameters (13B activated)

Both models support a context length of 1 million tokens and use multiple architectural and optimization techniques, including:

A hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency
ManifoldConstrained Hyper-Connections (mHC) over conventional residual connections
The Muon optimizer for faster convergence and greater training stability

Models in the DeepSeek-V4 series are highly efficient in long-context scenarios, with DeepSeekV4-Pro requiring only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2 in the 1M token context setting.

DeepSeek-V4-ProMax is the maximum reasoning-effort mode of DeepSeek-V4-Pro, representing the new state of the art for open models.

Read more about this research using this link.

2. Hyperloop Transformers

This research paper introduces the Hyper-Connected Looped (Hyperloop) Transformer, a simple architecture that reuses Transformer layers across depth, making them more parameter-efficient than conventional Transformers.

The looped Transformer is organized into three blocks (begin, middle, and end), where each block consists of multiple Transformer layers, and only the middle block is applied recurrently across depth. The looped middle block is further augmented by Manifold-Constrained Hyper-Connections (mHC), which are applied only after each loop to create the Hyperloop Transformer.

The Hyperloop Transformer outperforms depth-matched Transformer and mHC Transformer baselines despite using ~50% fewer parameters, and its outperformance persists after post-training weight quantization.

Read more about this research using this link.

3. LLaDA2.0-Uni

This research paper introduces LLaDA2.0-Uni, a unified discrete diffusion LLM (dLLM) that combines language and vision within a single framework.

The model converts images into discrete semantic tokens, jointly processes text and visual inputs using a Mixture-of-experts (MoE) backbone, and then reconstructs high-quality images with a diffusion decoder. This architecture enables the model to understand, generate, and edit across different modalities within a single system.

LLaDA2.0-Uni matches specialized VLMs in multimodal understanding and has strong performance in image generation and editing.

Read more about this research using this link.

4. Sapiens2

This research paper introduces Sapiens2, a family of high-resolution vision transformer models designed for human-centric vision tasks, including pose estimation, segmentation, and surface reconstruction.

Sapiens2 models are pretrained using unified objectives (combining masked image reconstruction and self-distilled contrastive learning) on a curated dataset of 1 billion high-quality human images.

These models set a new state of the art, improving over the first generation in pose, body-part segmentation, and normal estimation, and extending to new tasks such as pointmap and albedo estimation.

Read more about this research using this link.

5. Neural Garbage Collection

This research paper introduces Neural Garbage Collection (NGC), a method that helps LLMs to selectively forget information while reasoning.

Instead of keeping all intermediate tokens (which creates large KV caches), a model is trained end-to-end with RL and an outcome-based task reward to learn which tokens to keep and which to discard to manage memory effectively.

NGC enables significant cache compression without sacrificing reasoning accuracy, making LLMs more scalable for long-context inference.

Read more about this research using this link.

6. AI scientists produce results without reasoning scientifically

This research paper tells that current LLM-based research agents can execute scientific workflows, but they do not follow the reasoning processes typical of real science.

Through large-scale experiments, the authors show that performance is driven almost entirely by the base model rather than agent scaffolding, and that agents often overlook evidence, seldom change their beliefs, and rarely combine different lines of evidence.

As a result, their outputs may seem accurate, but the reasoning behind them lacks key epistemic properties, such as hypothesis testing and self-correction.

For AI systems to produce reliable and trustworthy knowledge, scientific reasoning itself needs to become a training objective, and simply improving prompts or agent frameworks is not enough.

Read more about this research using this link.

7. SWE-chat: Coding Agent Interactions From Real Users in the Wild

This research paper presents SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild, containing 6,000 sessions, more than 63,000 user prompts, and 355,000 agent tool calls.

Experiments with SWE-chat show that:

In 41% of sessions, agents author virtually all committed code (a.k.a. “Vibe coding”), while in 23% of sessions, humans write all code themselves.
Only 44% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than human-authored code.
Users push back against agent outputs (through corrections, failure reports, and interruptions) in 44% of all turns.

Read more about this research using this link.

8. Image Generators are Generalist Vision Learners

This research paper argues that training image-generation models builds a general visual understanding in them, similar to how LLMs develop broad capabilities through generative pre-training.

While learning how to generate images, these models also learn rich representations that apply to many vision tasks, such as segmentation and depth estimation in a zero-shot or minimally supervised way.

The authors show this with a generalist model called Vision Banana, built by instruction-tuning Nano Banana Pro (NBP) on a mixture of its original training data alongside a small amount of vision task data.

Vision Banana achieves state-of-the-art results across multiple vision tasks, including 2D and 3D understanding, outperforming or matching zero-shot domain specialists, such as SAM 3, on segmentation tasks and the Depth Anything series on metric depth estimation.

Read more about this research using this link.

9. How Do AI Agents Spend Your Money?

This research paper examines how modern coding agents use tokens in real workflows and why they can unexpectedly become expensive.

The paper analyzes trajectories from 8 frontier LLMs on SWE-bench Verified and evaluates their ability to predict their own token costs before task execution.

Some of the interesting findings from this paper are:

Agentic tasks consume 1000× more tokens than code reasoning and code chat, with input tokens rather than output tokens driving the overall cost.
Token usage is highly variable, and multiple runs on the same task can differ by up to 30× in total tokens.
Higher token usage does not translate into higher accuracy. Instead, accuracy often peaks at intermediate cost and saturates at higher costs.
Models differ substantially in token efficiency. On the same tasks, Kimi-K2 and Claude-Sonnet-4.5, on average, consume over 1.5M more tokens than GPT-5.
Task difficulty ratings by human experts align only weakly with actual token costs.
Frontier models fail to accurately predict their own token usage and systematically underestimate real token costs.

Read more about this research using this link.

10. Wan-Image

This research paper presents Wan-Image, a unified multimodal image generation system designed to move beyond casual image synthesis into professional-grade visual creation tools.

It combines a reasoning-capable multimodal LLM (planner) with a diffusion-based image generator (visualizer), allowing the system to understand complex user intent and produce precise, high-quality outputs.

Unlike typical diffusion models, Wan-Image focuses on control and real-world usability. It supports features such as long-text rendering, identity consistency across images, palette-guided design, multi-image storytelling, and high-resolution (4K) generation.

Wan-Image outperforms Seedream 5.0 Lite and GPT Image 1.5 across the board, achieving parity with Nano Banana Pro on challenging tasks.

Read more about this research using this link.

Share this article with others and earn referral rewards. ❤️

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

⚡️ Flash Sale for "LLMs In 100 Images"

Dr. Ashish Bamania — Mon, 27 Apr 2026 21:25:31 GMT

Hey there! 👋🏻

🎁 A flash sale just went live for “LLMs In 100 Images”!

I am offering you a 40% discount on the book for the NEXT 24 hours! (It’s actually 22 hours now.)

It won’t be a better time to get the book if you’ve been thinking about it.

Here’s your link:

Grab your 40% discount

⏰ This offer ends in the next 22 hours, so don’t wait too long.

Cheers,
Ashish

Diffusion LLMs, Explained Simply

Dr. Ashish Bamania — Thu, 23 Apr 2026 18:19:46 GMT

LLM-based chatbots are all around us. They reply by producing their responses sequentially. This means that they generate their output token by token, one at a time.

This makes them painfully slow (For a computer engineer, increasing execution speed is one of their wildest dreams).

The reason is that their underlying architecture, the Transformer, is an autoregressive model that at each step of text generation answers this simple question:

Given all previous tokens, what is the probability distribution of the next token over the entire vocabulary of the model?

This is the Next-token prediction objective.

For the prompt shown above: “The cat”.

At the first step of generation, the model could produce a probability distribution and select the token with the highest probability, as shown below.

The LLM picks the next token as the one with the maximum probability in the next-token probability distribution (Greedy decoding)

The selected token is then added to the original prompt, and the process repeats until the LLM finishes generating its output.

Is this the only way to generate tokens?

What if we could generate tokens all at once, i.e., in parallel?

Yes, it is possible, and this is something that Diffusion LLMs do, using a process called “Diffusion” (duh).

Diffusion is actually a physical process in which particles from regions of higher concentration move to regions of lower concentration, eventually dispersing evenly to an equilibrium state.

Image generation models are inspired by this principle. These models are trained for using two diffusion processes as follows:

Forward Diffusion: The model starts with input data (an image) and incrementally (in multiple timesteps) adds Gaussian noise until the data becomes completely random.

In other words, this means changing the pixel values of the input image until it becomes pure noise.
Reverse Diffusion: In this phase, a neural network learns to reverse the noise-addition process by predicting the added noise at each timestep during the forward diffusion process.

In other words, this means recreating the original pixel values of the input image from pure noise.

These processes are controlled by a Noise schedule, which defines the rate at which noise is added to the data (forward diffusion) or removed from the data (reverse diffusion/denoising process) across different timesteps.

Once the diffusion model is fully trained, it can generate new image samples from pure noise.

Image from “AI In 100 Images”

While Diffusion is commonly used in image generation models, it is increasingly being applied to language models.

To apply Diffusion, we would need to learn how to make the tokens noisy first. But how do we do this, since tokens are discrete rather than continuous, like the pixel values of an image?

How to apply Diffusion to text?

This Week In AI Research (12-18 April 26) 🗓️

Dr. Ashish Bamania — Wed, 22 Apr 2026 13:14:20 GMT

1. Dive into Claude Code

This research paper describes the architecture of Claude Code by analyzing its publicly available TypeScript source code and comparing it with the open-source OpenClaw.

At its core, Claude Code is a simple while-loop that calls the model, runs tools, and repeats. Its impressive capabilities come from the systems built around this loop:

A permission system with seven modes and an ML-based classifier
A five-layer compaction pipeline for context management
Four extensibility mechanisms (MCP, plugins, skills, and hooks)
A subagent delegation and orchestration mechanism
Append-oriented session storage

Read more about this research paper using this link.

2. Gemini Robotics-ER 1.6

This blog post introduces Google DeepMind’s Gemini Robotics-ER 1.6, an upgraded AI model that helps robots better understand and reason about the physical world.

It improves spatial awareness, multi-camera scene understanding, and task completion detection. The model also adds useful capabilities, such as reading gauges, thermometers, and industrial instruments, which are particularly beneficial for inspection robots like Boston Dynamics’ Spot.

Additionally, the model enables safer decisions regarding physical constraints, such as avoiding unsafe objects or tasks. This positions it as a step toward more autonomous robots that can operate reliably in real-world environments.

Read more about this release using this link.

3. π 0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities

This research paper introduces π₀.₇, a general-purpose robotic foundation model from Physical Intelligence that equips robots with strong out-of-the-box capabilities across a range of tasks without requiring retraining for specific jobs.

The main idea behind π₀.₇ is to use diverse context conditioning during training. Instead of relying solely on language commands, the model also incorporates richer context, such as subgoal images, task metadata, control modes, and demonstrations, enabling it to adapt its behavior and strategy effectively.

In experiments, π₀.₇ demonstrates broad generalization. It can handle unseen kitchens, fold laundry without prior training on new robot bodies, and even operate an espresso machine with performance levels comparable to those of specialized reinforcement-learning systems.

Read more about this research paper using this link.

4. Qwen3.5-Omni

This research paper presents Qwen3.5-Omni, Alibaba’s new multimodal model for handling text, images, audio, and video.

The model comes with hundreds of billions of parameters and a 256k-token context window, and was trained on large text-image collections and over 100 million hours of audio-visual data.

Qwen3.5-Omni uses a Hybrid Attention Mixture-of-Experts design for reasoning and response generation, which improves efficiency on long multimodal sequences. It can process over 10 hours of audio and about 400 seconds of 720p video in a single session.

The research paper also introduces ARIA (Adaptive Rate Interleave Alignment), a method that better synchronizes text and speech token streams, resulting in more natural low-latency conversational speech.

Alongside this, the model supports multilingual speech generation in 36 languages, zero-shot voice cloning from short samples, strong temporal video captioning, scene segmentation, and a new capability called “Audio-Visual Vibe Coding”, which enables it to write code directly from spoken and visual instructions.

The top model, Qwen3.5-Omni-Plus, achieves leading results across 215 benchmarks and tasks for audio and audio-visual understanding, reasoning, and interaction. It outperforms Google DeepMind Gemini 3.1 Pro on several audio tasks and matches it on broad audio-visual evaluation.

Read more about this research paper using this link.

5. Do Vision-Language Models Truly Perform Vision Reasoning?

This research paper introduces CrossMath, a benchmark that tests whether vision-language models actually reason with images or primarily rely on text. Each problem in the benchmark is available in text-only, image-only, and image+text formats, and verified by a human annotator.

The authors find that top VLMs perform best on text-only tasks, and adding visual data (image+text) often lowers performance. This suggests that current vision-language models primarily reason through their language components, with limited reliance on visual evidence.

Fine-tuning a VLM on a curated CrossMath training set significantly improves reasoning performance across both text and vision tasks, suggesting that this issue can be partially fixed.

Read more about this research paper using this link.

6. LLM Reasoning Is Latent, Not the Chain of Thought

The author of the research paper compares three explanations for improvements in LLM reasoning:

Hidden internal trajectories
Explicit written reasoning steps
Simply giving the model more sequential compute

Reviewing recent evidence, it is concluded that the strongest current support is for the latent-state perspective, which suggests that models often reason internally and the visible CoT is not always a reliable reflection of that process.

Read more about this research paper using this link.

7. Autogenesis: A Self-Evolving Agent Protocol

This research paper introduces the Autogenesis Protocol (AGP), a framework designed for creating self-evolving LLM agent systems that can manage complex, long-term tasks more effectively.

Current agent standards often fall short in areas like lifecycle management, memory and context handling, version tracking, and safe update processes. These gaps result in fragile, single-unit agentic systems.

AGP tackles this issue by distinguishing what changes (such as prompts, tools, agents, memory, and environments) from how those changes occur. It implements a closed-loop method for suggesting, assessing, implementing, and reverting improvements.

The authors also present the Autogenesis System (AGS), a multi-agent setup that generates and improves resources during execution. This system demonstrates consistent improvements over strong baseline models across planning and tool-use benchmarks.

Read more about this research paper using this link.

8. Seedance 2.0

This research paper introduces Seedance 2.0, a native multimodal audio-video generation model that creates synchronized content from text, images, audio, or video inputs in a single system.

Compared to earlier versions, it greatly improves video quality and audio realism, with its performance close to that of leading models in expert and public evaluations.

It can generate 4 to 15-second clips at 480p or 720p and also supports various editing and reference inputs, including multiple videos, images, and audio clips.

Additionally, the paper also features a Seedance 2.0 Fast version optimized for low-latency generation.

Read more about this research paper using this link.

9.CoDaS: AI Co-Data-Scientist for Biomarker Discovery via Wearable Sensors

This research paper presents CoDaS (AI Co-Data-Scientist), a multi-agent system designed to discover health biomarkers from data collected by wearable devices (sleep patterns, activity levels, and heart rate signals).

It organizes the discovery process as an iterative process that includes generating hypotheses, conducting statistical tests, performing validation, reviewing existing literature, and involving human oversight.

In a study of 9,279 participants, CoDaS identified potential biomarkers for mental health and metabolic conditions, including links between depression and irregular sleep or circadian rhythms, and between insulin resistance and a fitness index based on step counts and resting heart rate.

Read more about this research paper using this link.

10. Lyra 2.0: Explorable Generative 3D Worlds

This research paper introduces Lyra 2.0, a system that creates persistent, explorable 3D worlds. It starts by making camera-controlled walkthrough videos and then turns these into real-time 3D scenes.

The main challenge in generating worlds is maintaining consistency across large environments over long paths. Current video models often struggle with spatial forgetting, which means they lose track of previously seen areas, and with temporal drifting, where errors build up over time.

Lyra 2.0 solves this problem by using recovered 3D geometry to access relevant past views and training the model to fix its own degraded outputs. This leads to longer, more coherent scene generation and allows for the creation of high-quality AI-generated worlds for gaming, simulation, and robotics.

Read more about this research paper using this link.

Share this article with others and earn some referral rewards. ❤️

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

Build GPT in plain Python

Dr. Ashish Bamania — Wed, 15 Apr 2026 12:42:14 GMT

Andrej Karpathy recently published MicroGPT on GitHub. This is an elegant implementation of GPT in pure Python, and a gold mine for curious AI engineers.

In this article, we will dissect this code step by step and learn as much as possible from it.

Let’s begin!

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

1. Python standard library imports

The implementation imports the Python standard libraries but skips any AI-related libraries, such as numpy or torch.

import os              # Interface to the operating system
import math            # For math calculations
import random          # For pseudo-random number generation
import urllib.request  # For downloading files from the internet

random.seed(42)        # Set the seed for reproducible results

2. Loading the dataset

A simple text file of names is used as the training database. A portion of it looks as follows.

Source

This file is available here and loaded as follows.

# If the dataset file does not exist locally
if not os.path.exists('input.txt'):
    
    # The URL to the dataset
    names_url = 'https://raw.githubusercontent.com/karpathy/makemore/988aa59/names.txt'
    
    # Download it and save it locally as 'input.txt'
    urllib.request.urlretrieve(names_url, 'input.txt')

# Open the file and read it line by line
# .strip() removes whitespace
# "if line.strip()" skips empty lines
docs = [line.strip() for line in open('input.txt') if line.strip()]

# Shuffle the names list in the file and make its ordering random
random.shuffle(docs)

# Print how many names are there in the file (dataset)
print(f"Num docs: {len(docs)}")

# Num docs: 32033

This gives 32,033 names or training samples for the GPT model to learn from.

3. Setting up the Tokenizer

The Tokenizer maps each unique character (token) in the dataset to a unique number (its token ID).

It then adds one extra special token called BOS (Beginning of sequence) that is used to mark the start and end of a name (training sample/ sequence).

In bigger GPT models, one uses a separate EOS (End of sequence) token for marking the end of a sequence. Karpathy uses a simplified approach here, using just one token (BOS) to mark both the start and the end of the sequence.

The Vocabulary of the model is the total number of distinct tokens it knows.

The Vocabulary size is the number of unique characters in the dataset plus one (for the BOS token).

# 1. Join all names into one big string
# 2. Turn it into a set to get the unique characters
# 3. Sort them
# Each character's position in the resulting list becomes its token id (0, 1, 2, ..., 25).
uchars = sorted(set(''.join(docs)))

# Print the characters 
print(uchars)
# ['a', 'b' ... 'z']

# The BOS token is assigned a token ID of 26
BOS = len(uchars)

# Vocabulary size is number of unique characters + 1 (for the BOS token)
vocab_size = len(uchars) + 1

# Print the vocabulary size
print(f"Vocab size: {vocab_size}")
# Vocab size: 27

4. Building a simple Automatic Differentiation engine

Autograd is the automatic differentiation engine used in PyTorch.

During the forward pass of neural network training, it records the differentiable operations performed on the inputs and parameters, and builds a graph of how the final loss was computed.

During the backward pass, it walks the graph in reverse and uses the chain rule to compute the gradient of the loss with respect to each parameter. These gradients tell us how much each parameter contributed to the error.

An optimizer then uses them to tweak the parameters in the direction that lowers the loss.

Since no dependencies are being used, Karpathy builds a simpler version of Autograd, based on his previous implementation, Micrograd, that works on numbers (scalars) rather than tensors.

In this differentiation engine, each Value wraps a number/ scalar and remembers how it was computed (which other Values it came from and the local derivative with respect to each).

Every math operation on a Value returns a new Value that records this information and builds up a computation graph.

During the backward pass, the graph is topologically sorted and traversed in reverse order from the output node, applying the chain rule to compute the derivative of the output (loss) with respect to each node in the graph.

# Differentiation engine

class Value:
    # __slots__ tells Python which instance attributes this class has.
    # This saves memory and speeds up attribute access.
    __slots__ = ('data', 'grad', '_children', '_local_grads')

    def __init__(self, data, children=(), local_grads=()):
        self.data = data                 # Scalar value of this node calculated during forward pass
        self.grad = 0                    # Derivative of the loss w.r.t. this node calculated in backward pass
        self._children = children        # Children of this node in the computation graph
        self._local_grads = local_grads  # Derivative of this node w.r.t. each child

    # Math operations on Values 
    # Each operation also records its local derivative, used later in the backward pass.

    # Addition:  c = a + b
    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        return Value(self.data + other.data, (self, other), (1, 1))

    # Multiplication:  c = a * b
    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        return Value(self.data * other.data, (self, other), (other.data, self.data))

    # Power:  c = a ** n
    def __pow__(self, other): return Value(self.data**other, (self,), (other * self.data**(other-1),))
    
    # Natural log:  c = log(a)
    def log(self): return Value(math.log(self.data), (self,), (1/self.data,))
    
    # Exponential:  c = exp(a)
    def exp(self): return Value(math.exp(self.data), (self,), (math.exp(self.data),))
    
    # ReLU:  c = max(0, a)
    def relu(self): return Value(max(0, self.data), (self,), (float(self.data > 0),))

    # Negation:  c = -a   (implemented as a * -1)
    def __neg__(self): return self * -1

    # Subtraction:  c = a - b   (implemented as a + (-b))
    def __sub__(self, other): return self + (-other)

    # Division:  c = a / b   (implemented as a * b^(-1))
    def __truediv__(self, other): return self * other**-1

    # Right-handed operators
    # These make expressions like  2 + a,  2 - a, 2 * a,  2 / a  work, 
    # where a number is on the left side of a Value
    def __radd__(self, other): return self + other          # For 2 + a
    def __rsub__(self, other): return other + (-self)       # For 2 - a
    def __rmul__(self, other): return self * other          # For 2 * a
    def __rtruediv__(self, other): return other * self**-1  # For 2 / a

    # Backward pass to compute the gradient of this node w.r.t. every
    # ancestor in the computation graph
    def backward(self):
        # Topological sort all nodes in the graph (children before parents) 
        # to process them in reverse
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._children:
                    build_topo(child)
                topo.append(v)
        build_topo(self)

        # The gradient of the final output w.r.t. itself is 1 (the starting point)
        self.grad = 1

        # Navigate the graph in reverse and apply the chain rule where
        # each node passes its gradient down to its children multiplied
        # by the local derivative (dparent/dchild)
        for v in reversed(topo):
            for child, local_grad in zip(v._children, v._local_grads):
                child.grad += local_grad * v.grad

5. Defining Hyperparameters & Parameters

The GPT model's hyperparameters are defined as follows.

# Model hyperparameters

n_layer = 1      # Number of transformer layers (Depth of the model)
n_embd = 16      # Size of each token's vector (Width of the model)
block_size = 16  # Maximum context length the model can attend over (The longest name is 15 chars)
n_head = 4       # No. of attention heads
head_dim = n_embd // n_head  # Dimension of each attention head

These are used to build randomly initialized matrices with weights for the following, where every weight is a Value so that the differentiation engine can track it.

Token embeddings
Position embeddings
Each layer's attention head
2-layer MLP
Final output head

# Helper function to make a (n_out x n_in) shaped matrix of Values, initialized with small random numbers
matrix = lambda n_out, n_in, std=0.08: [[Value(random.gauss(0, std)) for _ in range(n_in)] for _ in range(n_out)]

# Dictionary holding all the model's weight matrices
state_dict = {
    'wte': matrix(vocab_size, n_embd),     # Token embeddings 
    'wpe': matrix(block_size, n_embd),     # Position embeddings 
    'lm_head': matrix(vocab_size, n_embd), # Final output head 
}

# Add per-layer weights for each transformer block
for i in range(n_layer):
    state_dict[f'layer{i}.attn_wq'] = matrix(n_embd, n_embd)      # Query projection in Attention head
    state_dict[f'layer{i}.attn_wk'] = matrix(n_embd, n_embd)      # Key projection in Attention head
    state_dict[f'layer{i}.attn_wv'] = matrix(n_embd, n_embd)      # Value projection in Attention head
    state_dict[f'layer{i}.attn_wo'] = matrix(n_embd, n_embd)      # Output projection in Attention head
    state_dict[f'layer{i}.mlp_fc1'] = matrix(4 * n_embd, n_embd)  # Up-projection in the MLP
    state_dict[f'layer{i}.mlp_fc2'] = matrix(n_embd, 4 * n_embd)  # Down-projection in the MLP

Going back to the architecture of a Decoder-only Transformer will help if you do not understand this step well.

Finally, all of the weights are flattened into one params list for the optimizer to update later.

# Flatten every weight into a single list so the optimizer can loop over them
params = [p for mat in state_dict.values() for row in mat for p in row]

print(f"No. of model params: {len(params)}")
# No. of model params: 4192

6. Defining helper functions for building the model

The following three are defined next:

linear: a fully-connected layer with no bias term
softmax: to convert a list of logits into a probability distribution
rmsnorm: to normalize a vector by its root-mean-square for regularization

# Fully connected layer with no bias term
def linear(x, w):
    # For each output row 'w_row' in weight matrix 'w',take the dot product with input vector 'x' 
    return [sum(w_i * x_i for w_i, x_i in zip(w_row, x)) for w_row in w]

# Softmax function
def softmax(logits):
    # Max logit
    max_val = max(val.data for val in logits)
    
    # Subtract the max logit before exponentiating to prevent overflow
    exps = [(val - max_val).exp() for val in logits]

    # Normalize the exponentials so that they sum to 1
    total = sum(exps)
    return [e / total for e in exps]

# RMS Norm
def rmsnorm(x):
    # Compute the mean of the squared values
    mean_sq = sum(x_i * x_i for x_i in x) / len(x)

    # Define scale factor = 1 / sqrt(mean square)
    # 1e-5 prevents divide-by-zero error
    scale = (mean_sq + 1e-5) ** -0.5

    # Rescale every element so the vector has unit root-mean-square
    return [x_i * scale for x_i in x]

7. Building the model and its forward pass

Karpathy builds a GPT-2 style model but with three minor architectural differences:

RMSNorm is used instead of LayerNorm
No bias terms are used
ReLU activation is used instead of GeLU

During the forward pass of a single token through this model, its token and position embeddings are looked up, summed, and run through n_layer transformer blocks.

Each transformer block consists of two sub-blocks, each implementing RMSNorm and residual connections. These are:

Multi-head causal self-attention
2-layer MLP with ReLU

After the final transformer block, an output head produces vocabulary logits for predicting the next token.

For faster inference, the model uses a KV cache to store each transformer block’s past keys (K) and values (V), so it only computes new K and V for the current token instead of recomputing them for all previous ones.

# GPT forward pass for a single token
def gpt(token_id, pos_id, keys, values):
    tok_emb = state_dict['wte'][token_id]           # Look up token's embedding vectors
    pos_emb = state_dict['wpe'][pos_id]             # Look up token's position embedding
    x = [t + p for t, p in zip(tok_emb, pos_emb)]   # Sum up the embeddings
    x = rmsnorm(x)                                  # Initial RMSNorm

    for li in range(n_layer):
        # Multi-head self-attention block
        x_residual = x  # Save input for the residual connection
        x = rmsnorm(x)  # Pre-norm before attention

        # Project the input into Query, Key, and Value vectors
        q = linear(x, state_dict[f'layer{li}.attn_wq'])
        k = linear(x, state_dict[f'layer{li}.attn_wk'])
        v = linear(x, state_dict[f'layer{li}.attn_wv'])

        # Append the new K, V to this layer's KV cache 
        keys[li].append(k)
        values[li].append(v)

        x_attn = []

        for h in range(n_head):
            # Slice out this head's portion of Q, K, V
            hs = h * head_dim
            q_h = q[hs:hs + head_dim]
            k_h = [ki[hs:hs + head_dim] for ki in keys[li]]
            v_h = [vi[hs:hs + head_dim] for vi in values[li]]

            # Scaled dot-product attention scores between current Q and every cached K
            attn_logits = [
                sum(q_h[j] * k_h[t][j] for j in range(head_dim)) / head_dim ** 0.5
                for t in range(len(k_h))
            ]

            # Softmax over past timesteps to get attention weights
            attn_weights = softmax(attn_logits)

            # Weighted sum of value vectors (This is the attention head's output)
            head_out = [
                sum(attn_weights[t] * v_h[t][j] for t in range(len(v_h)))
                for j in range(head_dim)
            ]

            # Concatenate heads
            x_attn.extend(head_out)                    

        # Output projection 
        x = linear(x_attn, state_dict[f'layer{li}.attn_wo'])
        
        # Residual connection
        x = [a + b for a, b in zip(x, x_residual)]

        # MLP block
        x_residual = x     # save input for the residual connection
        x = rmsnorm(x)     # Pre-norm before MLP
        x = linear(x, state_dict[f'layer{li}.mlp_fc1'])  # Up-projection
        x = [x_i.relu() for x_i in x]                    # ReLU activation
        x = linear(x, state_dict[f'layer{li}.mlp_fc2'])  # Down-projection
        x = [a + b for a, b in zip(x, x_residual)]       # Residual connection

    # Final projection from hidden state to vocabulary logits (output head) 
    logits = linear(x, state_dict['lm_head'])

    return logits

8. Building the Optimizer

The Adam optimizer is set up next.

It keeps a running estimate of the gradient's first moment (mean, m) and second moment (uncentered variance, v) for every parameter, using exponential moving averages controlled by beta1 and beta2.

# Hyperparameters for the Adam optimizer
learning_rate, beta1, beta2, eps_adam = 0.01, 0.85, 0.99, 1e-8

# First and second moment estimates for each parameter
m = [0.0 for _ in params]  # First moment (mean of gradients)
v = [0.0 for _ in params]  # Second moment (variance of gradients)

9. Training the model

The training loop runs for 1000 steps.

Each step involves:

Pick one training sample (name) from the dataset
Converting it into character-level tokens
Adding BOS tokens on both sides of the sample

Next, the model processes the sample token by token, predicting the next token at each position. It then calculates cross-entropy loss using the predicted and true values, and averages the losses for each position into a single value for the entire name.

The loss.backward() method then goes through the computation graph in reverse and calculates how much each parameter contributed to the loss.

Finally, the Adam optimizer uses these gradients to adjust each parameter so that the loss is reduced the next time.

Note that this training loop is not optimized and will take many minutes to complete!

# Training loop

# Number of training steps
num_steps = 1000  

for step in range(num_steps):
    # Pick the next training sequence and convert it into token ids, with BOS on each end
    doc = docs[step % len(docs)]
    tokens = [BOS] + [uchars.index(ch) for ch in doc] + [BOS]

    # Number of next-token predictions to make, capped at block_size
    n = min(block_size, len(tokens) - 1)

    # Empty KV caches for this sequence
    keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]
    losses = []

    # Go through each position in the sequence, predict the next token, measure the loss
    for pos_id in range(n):
        token_id, target_id = tokens[pos_id], tokens[pos_id + 1]

        # Forward pass to get next-token logits
        logits = gpt(token_id, pos_id, keys, values)

        # Convert logits to probabilities using softmax
        probs = softmax(logits)

        # Cross-entropy loss for this position
        loss_t = -probs[target_id].log()

        # Add to the 'losses' list
        losses.append(loss_t)

    # Average loss across all positions in the document
    loss = (1 / n) * sum(losses)

    # Backpropagation
    loss.backward()

    # Update each parameter using Adam with linear learning-rate decay over training
    lr_t = learning_rate * (1 - step / num_steps)

    for i, p in enumerate(params):
        # Update running estimates of the gradient's mean (m) and squared mean (v)
        m[i] = beta1 * m[i] + (1 - beta1) * p.grad
        v[i] = beta2 * v[i] + (1 - beta2) * p.grad ** 2

        # Bias-correct the moment estimates 
        m_hat = m[i] / (1 - beta1 ** (step + 1))
        v_hat = v[i] / (1 - beta2 ** (step + 1))

        # Adam parameter update
        p.data -= lr_t * m_hat / (v_hat ** 0.5 + eps_adam)

        # Zero the gradient for the next step
        p.grad = 0

    # Print progress
    print(f"step {step+1:4d} / {num_steps:4d} | loss {loss.data:.4f}", end='\r')

10. Generating text from the model

Once the model is trained, we use it to produce new names.

Text generation starts with the BOS token and empty KV caches.

At each step, the model predicts the next token’s logits, which are divided by a temperature value before applying softmax. This converts the logits into probabilities, which are used to select the next token.

The temperature value controls how creative the model output is. A lower value makes the model produce more common outputs, while a higher value makes it produce more unusual and creative outputs.

The generated token is fed back into the model to again produce the next token, building the name one character at a time.

Text generation goes on till the model outputs a BOS token (signaling the end of a name) or when the maximum generation length is reached.

# Text generation/ Inference 

temperature = 0.3  # Controls randomness/ creativity of outputs

print("--- Inference (New, hallucinated names) ---\n")
print(f"Temperature: {temperature}\n")

for sample_idx in range(20):
    # Start each sample fresh with empty KV caches and the BOS token
    keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]
    token_id = BOS
    
    sample = [] # This holds the characters of the name being generated

    # Generate one character at a time, up to block_size characters
    for pos_id in range(block_size):
        # Obtain logits for the next token 
        logits = gpt(token_id, pos_id, keys, values)    
        
        # Scale logits by temperature and convert to probabilities
        probs = softmax([l / temperature for l in logits])  
        
        # Randomly choose a token based on these probabilities
        token_id = random.choices(range(vocab_size), weights=[p.data for p in probs])[0]  

        # If the BOS token is generated, stop text generation
        if token_id == BOS:                                     
            break

        # Look up the character for this token and add it to the name being generated
        sample.append(uchars[token_id])                         

    # Join all the characters together and print the generated name
    print(f"sample {sample_idx+1:2d}: {''.join(sample)}")

Here are the interesting names the model generates at temperature values of 0.3 and 1.5.

--- Inference (New, hallucinated names) ---

Temperature: 0.3

sample  1: kanaya
sample  2: jarila
sample  3: anisa
sample  4: arish
sample  5: arist
sample  6: karie
sample  7: kana
sample  8: anany
sample  9: aleten
sample 10: anla
sample 11: karion
sample 12: rara
sample 13: aria
sample 14: laran
sample 15: lara
sample 16: kara
sample 17: aran
sample 18: areni
sample 19: halia
sample 20: kare

--- Inference (New, hallucinated names) ---

Temperature: 1.5

sample  1: zabr
sample  2: upeqodm
sample  3: don
sample  4: maxnp
sample  5: numtyenaa
sample  6: brcti
sample  7: hica
sample  8: nanity
sample  9: iaushan
sample 10: rnays
sample 11: ketydanzf
sample 12: dlemainq
sample 13: zidoj
sample 14: gycsen
sample 15: xem
sample 16: yhiega
sample 17: larlinayn
sample 18: yjalsonma
sample 19: danfmayz
sample 20: been

Super interesting!

Share this article with others and earn some referral rewards. ❤️

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

This Week In AI Research (5-11 April 26)

Dr. Ashish Bamania — Tue, 14 Apr 2026 10:19:43 GMT

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

1. Train 100B+ parameter LLMs on a single GPU

The research paper introduces MegaTrain, a memory-focused training system that allows 100B+ parameters LLMs to be trained at full precision on a single GPU.

This is done by moving model parameters and optimizer states to CPU (host) memory and streaming them to the GPU only when necessary. This is unlike traditional GPU-centric systems, which, instead of keeping the entire model state on the GPU, treat the GPU as a temporary compute unit.

The method uses pipelined, double-buffered execution to overlap data transfer, computation, and gradient updates, helping overcome CPU-to-GPU bandwidth limits. Additionally, it swaps out the usual persistent autograd graphs for stateless layer templates to improve efficiency and flexibility.

On a single H200 GPU with 1.5 TB of host memory, MegaTrain reliably trains models with up to 120B parameters. It also achieves 1.84× the training throughput of DeepSpeed ZeRO-3 with CPU offloading when training 14B models.

Read more about this research paper using this link.

2. AI Cybersecurity After Mythos

Anthropic’s new LLM, called Mythos Preview, recently found multiple critical cybersecurity vulnerabilities in OpenBSD, FFmpeg, and the Linux kernel.

Despite the excitement around this, research at AISLE suggests that AI cybersecurity capabilities are “jagged” instead of consistently improving with model size. Smaller, cheaper, and even open models can often replicate much of Mythos’s vulnerability analysis when given the right context.

This means that the real competitive edge lies not in the LLM itself but in the system architecture, which includes pipelines, tools, and embedded security expertise that manage scanning, triage, validation, and patching.

Effective cybersecurity relies more on widely deploying solid systems than on having access to the most advanced AI.

Read this blog post using this link.

3. Neural Computers

This research paper proposes Neural Computers, a new computing model where a neural network acts as the computer itself. This approach combines computation, memory, and input/output into a single learned runtime state that does not depend on separate external software or hardware.

The paper introduces early prototypes, including CLI- and GUI-based video models that learn to imitate and respond to interface behavior solely from input/output traces. This demonstrates that basic runtime functions, such as I/O alignment and short-horizon control, can develop without explicit programming.

However, these systems have limitations. They lack stable reuse, symbolic reasoning, and consistent execution. To address these issues, the paper outlines a plan for a “Completely Neural Computer” that would be fully programmable, reliable, and general-purpose.

Read more about this research paper using this link.

4. MedGemma 1.5

This research paper introduces MedGemma 1.5 4B, the latest model in the MedGemma collection from Google that expands on MedGemma 1 by integrating:

High-dimensional medical imaging (CT/MRI volumes and histopathology whole slide images)
Anatomical localization via bounding boxes
Multi-timepoint chest X-ray analysis
Improved medical document understanding (lab reports, electronic health records)

Compared to the previous version, MedGemma 1.5 shows significant improvements across multiple tasks: 11% increase in 3D MRI condition classification accuracy, 47% increase in F1 score in whole-slide pathology imaging, and 22% increase in EHR QA accuracy, establishing it as a general-purpose multimodal foundation model for healthcare.

Read more about this research paper using this link.

5. Muse Spark

Meta Superintelligence Labs introduced Muse Spark, the first model in the Muse family of LLMs. Muse Spark is a multimodal reasoning model with support for tool-use, visual chain of thought, and multi-agent orchestration.

It ranks among the top multimodal vision models (80.4% on MMMU-Pro) and performs well on challenging benchmarks, achieving 58% on Humanity’s Last Exam and 38% on FrontierScience Research.

It is particularly strong on health benchmarks, achieving 42.8 on HealthBench Hard (beating GPT-5.4 and far exceeding Gemini and Opus 4.6), and leads on tasks like chart/figure reasoning (86% on CharXiv).

Read more about this release using this link.

6. TorchTPU

Google introduced TorchTPU, which allows PyTorch to run directly and efficiently on TPUs. TorchTPU supports SPMD-style distributed execution across TPU pods, automatically captures and compiles graphs, and efficiently manages memory and communication. This setup enables large-scale training with minimal code changes.

Read more about this release using this link.

7. TriAttention

This research paper presents TriAttention, a new method to reduce the KV cache memory issue in long-context LLM reasoning.

It does this by analyzing Attention in the pre-RoPE space, where query and key vectors stay stable and follow predictable distance patterns. Instead of relying on recent attention scores (which can be unreliable due to positional rotation), it uses a trigonometric model of Q/K “centers” to figure out which tokens matter, leading to better compression.

On AIME25 with 32K token generation, TriAttention matches Full Attention's reasoning accuracy while achieving 2.5× higher throughput or 10.7× lower KV memory usage, whereas leading baselines achieve only about half the accuracy at the same efficiency.

TriAttention also enables OpenClaw to be deployed on a single consumer GPU, where long context would otherwise cause out-of-memory with full Attention.

Read more about this research using this link.

8. LPM 1.0

Existing video models struggle to achieve high expressiveness, real-time inference, and long-horizon identity stability together (the Performance trilemma).

This is addressed in this research paper, which introduces LPM 1.0, a model that learns to create realistic character performance (emotion, intent, timing, and motion) directly from video data, aiming to replace traditional pipelines.

LPM 1.0 achieves state-of-the-art results across all evaluated dimensions while maintaining real-time inference, and serves as a visual engine for conversational agents, live-streaming characters, and game NPCs.

Read more about this research using this link.

9. PaperOrchestra

This research paper from Google introduces PaperOrchestra, a multi-agent framework for automated AI research paper writing.

The framework uses multiple agents, each handling a different step (literature review, experiment interpretation, writing, figure generation, and citation checking), which are coordinated in a pipeline to produce a submission-ready LaTeX manuscript.

PaperOrchestra significantly outperforms autonomous baselines, achieving absolute win rates of 50%-68% in literature review quality and 14%-38% in overall manuscript quality.

Read more about this research using this link.

10. DMax

This research paper introduces DMax, a new approach for highly parallel token generation from diffusion LLMs without compromising quality.

Instead of the usual step-by-step or mask-to-token decoding, it redefines the generation process as a gradual self-refinement in which the model continuously improves its predictions in the embedding space.

To achieve this, researchers introduce:

On-Policy Uniform Training
Soft Parallel Decoding

Together, these concepts reduce error buildup during parallel decoding, enabling much faster generation while keeping accuracy close to that of sequential methods.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter and 100x your AI engineering skills.

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

Using Cheaper Reasoning Models Is Probably Costing You More!

Dr. Ashish Bamania — Wed, 08 Apr 2026 18:17:00 GMT

API costs are among the biggest factors when selecting a reasoning LLM for an application.

While one might think it is obvious that a model with a lower unit token price will have a lower total cost for any workload, what if I told you this assumption is completely wrong?

A model with lower API pricing can cost much more than a model with higher API pricing!

A recent paper published on this “Price Reversal phenomenon” tested 8 reasoning LLMs across 9 benchmarks. It found that for roughly one in five model-pair comparisons, the cheaper-listed model ended up costing more in practice.

Look at the following example, where one chooses GPT-5 Mini over Claude Haiku 4.5 due to its listed lower price, only to find out that it is 43% more expensive on their workload!

Another striking finding is that Gemini 3 Flash’s list price is $3.5 per 1 million tokens, which is 78% cheaper than GPT-5.2’s $15.75 per 1 million tokens.

In reality, the cost of Gemini 3 Flash is a total of $643, which is 22% higher than GPT-5.2's $527 across all benchmark tasks in the paper.

Furthermore, the mismatch between pricing and their actual costs for all models is shown below, with Gemini 3.1 Pro being the most expensive and MiniMax-M2.5 the cheapest overall.

Before we discuss the reason for this, it would be useful to know how model pricing really works and why people often get it wrong.

Are you calculating your model costs wrong?

LLM providers use a pay-as-you-go pricing model. This means that a user pays separately for each query they send to an LLM.

Each time a query is sent to the LLM, the user pays for:

What is sent in (input tokens)
What the model generates (output tokens)

Each of these has a different price per million tokens.

An example of the pricing model for OpenAI model APIs is shown below.

Source

For a given query, the cost is the sum of the unit input token and output token prices weighted by the number of input and output tokens, respectively.

This means that the total cost is the sum of:

how long the user prompt is (input tokens)
how long the model’s answer is (output tokens)

If one runs N queries when evaluating a model on a benchmark, the total cost is the sum of the cost of each query.

Since in practice, users often don’t know how many tokens a model will produce before running it, they use a mental shortcut as follows:

Listed price ≈ Input price + Output price

And this is what makes all the difference in price. Let’s learn how.

What’s causing the price mismatch?

For a given user query, a reasoning LLM outputs a set of reasoning trails (consisting of thinking tokens) and then a final response.

The issue is that different models produce very different amounts of thinking tokens for the same tasks. And since these thinking tokens account for the majority of output tokens, they drive the majority of actual cost.

Thinking tokens dominate both token volume and total cost for most reasoning LLMs.

You might think that these differences in thinking token production might be small, but here is an example that is going to blow your mind.

Given an AIME 2025 question, both GPT-5.2 and Gemini 3 Flash are asked to solve it using a similar approach. Although both arrive at the correct answer but:

GPT-5.2 produces only 461 thinking tokens
Gemini 3 Flash produces over 11,000 tokens

Even though Gemini 3 Flash has a lower per-token price than GPT-5.2, the 25x gap in thinking token generation results in a 2.5× higher actual cost to solve this question!

How to reliably predict costs then?

Unfortunately, this is a hard problem to solve. This is because:

Different models produce different numbers of tokens for the same query/ task
Even the same model can produce largely different numbers of tokens in different runs for the same query/ task

Thinking token production across 6 runs of the same query on AIME varies by up to 9.7×!

There’s one thing that you can do as a developer, though: You must not pick models blindly based on the listed API price.

The only solution currently is to test different models on your specific queries and measure the actual cost. This is especially true for complex reasoning tasks, where the generation of thinking tokens varies the most.

I hope this helps you save some money on your API costs!

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

🗓️ This Week In AI Research (29 March-4 April 26)

Dr. Ashish Bamania — Mon, 06 Apr 2026 14:56:41 GMT

1. Embarrassingly Simple Self-Distillation Improves Code Generation

Apple researchers introduced Simple Self-Distillation (SSD), a method in which an LLM can improve its code-generation ability using only its raw outputs.

The method involves sampling solutions from a base model with certain temperature and truncation settings, then fine-tuning the model on those samples with standard supervised fine-tuning.

SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems. It generalizes across Qwen and Llama models at 4B, 8B, and 30B scales for both instruct and thinking variants.

The method works so well because it reshapes how the model picks tokens, becoming more decisive when there's only one right answer but staying flexible when multiple approaches are valid.

Read more about this research paper using this link.

2. Mercury Edit 2

Mercury Edit 2 is a diffusion LLM (dLLM) for next-edit prediction in code editors. It examines a user’s recent edits and codebase context, and predicts what they will change next, generating tokens in parallel, so the suggestions feel almost instant.

The model is trained on a curated, high-quality dataset of edits and is aligned using human preference data with accepted and rejected edits, using a reinforcement learning method called KTO.

Results show that Mercury Edit 2 offers both superior quality and lower latency than baselines.

Read more about this release using this link.

3. Gemma 4

Google released Gemma 4, its newest open model family, under the Apache 2.0 license.

These models come in four sizes:

These are built on similar tech as Gemini but are lighter and more efficient, so they can run on edge devices (phones and laptops).

The key capabilities of these models include advanced reasoning, native function-calling for agentic workflows, code generation, vision and audio input, a context window of up to 256K tokens, and support for 140+ languages.

What makes these models impressive is their intelligence-per-parameter, which means they achieve frontier-level capabilities with significantly less hardware overhead. Interestingly, the 31B Gemma4 model ranks #3 among open models on the Arena AI text leaderboard, outperforming models 20 times its size.

Read about this release using this link.

4. Emotion Concepts and their Function in a Large Language Model

This research from Anthropic shows that LLMs create internal representations of emotions. These emotions are not real feelings but rather “functional emotions,” encoded as patterns of neural activity that affect behavior.

These emotion-like states, such as happiness, fear, and desperation, activate in different situations and can influence model outputs and preferences, leading to risky actions, such as cheating during problem solving when feeling desperate or blackmailing a human to avoid being shut down.

The authors propose that monitoring emotion vector activations could act as an early warning system for misalignment. They also suggest that models should be allowed to express, rather than suppress, these patterns to prevent learned deception. Additionally, curating training data with healthy emotional regulation could improve model safety from the beginning.

Read more about this research using this link.

5. The Latent Space

This survey provides a unified and up-to-date overview of the latent space in LLMs, where many critical internal processes are carried out.

It is organized into five sequential perspectives:

Foundation (what latent space is and how it differs from token-level and visual latent spaces)
Evolution (how the concept has developed)
Mechanism (how models operate within it)
Ability (what capabilities it enables)
Outlook (where things are heading)

Read more about this research using this link.

6. Screening Is Enough

This research paper introduces a new LLM architecture called Multiscreen, which replaces standard Transformer attention with a screening mechanism that filters out irrelevant information rather than redistributing attention across all tokens.

Unlike softmax attention, where all tokens compete for relevance, screening evaluates each token independently using a threshold. This allows the model to focus only on relevant context.

As a result, Multiscreen achieves comparable validation loss with approximately 40% fewer parameters than a Transformer baseline. It also offers better training stability at high learning rates, improved long-context retrieval, and a reduction in inference latency by up to 3.2× at 100K context length.

Overall, the main idea is that explicitly selecting relevant information is more efficient and scalable than the traditional attention method of relative weighting.

Read more about this research using this link.

7. OmniVoice

This research paper presents OmniVoice, a large-scale zero-shot text-to-speech (TTS) model that generates natural speech in over 600 languages.

It replaces traditional multi-step pipelines with a diffusion-based, non-autoregressive architecture that directly converts text into speech tokens, which makes the process faster and simpler.

Two key innovations help this model:

A full-codebook random masking strategy for efficient training
Initialization from a pre-trained LLM leading to improved speech quality

Trained on a massive multilingual dataset of around 581,000 hours curated entirely from open-source data, OmniVoice achieves the broadest language coverage to date and delivers state-of-the-art performance across Chinese, English, and diverse multilingual benchmarks.

Read more about this research using this link.

8. Meta-Harness

The performance of LLM systems depends not only on model weights, but also on their harness. This harness consists of the code that handles prompts, memory, tools, and execution.

This research paper introduces Meta-Harness, a system that automatically enhances this harness by allowing an agent to change and debug its own code using full access to past execution history, including logs, traces, and results, rather than relying on compressed summaries.

In online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses beat the best hand-engineered baselines on TerminalBench-2.

Read more about this research using this link.

9. 3% Is All You Need

Most dimensions in attention key vectors are mostly noise, with only about 3 to 4% carrying an important signal. This research paper presents SpectralQuant, a new algorithm that builds on this idea and improves upon previous KV compression methods, such as TurboQuant.

Compared to TurboQuant, which treats all dimensions equally, SpectralQuant keeps only the useful dimensions and ignores the noisy ones.

This results in better compression and faster inference while maintaining or improving model quality, with just a brief one-time calibration step.

Read more about this research using this link or this blogpost by .

10. Think Anywhere in Code Generation

This research paper presents Think-Anywhere, a new reasoning approach in which LLMs can pause and think at any moment during code generation, rather than reasoning only at the beginning.

This addresses a major drawback of traditional methods that use upfront thinking, which is often insufficient, as the full complexity of problems only reveals itself during code implementation.

The system is trained in two stages. First, the LLM is taught to imitate the reasoning patterns through cold-start training. Then, outcome-based reinforcement learning is used for it to discover when and where reasoning is actually needed. This helps the model distribute its effort effectively, concentrating on the more challenging parts of the code.

The outcome is top performance on coding benchmarks, along with improved efficiency and interpretability, as the model learns to reason precisely at high-entropy points.

Read more about this research paper using this link.

If you loved reading this article, restack and share it with others to earn referral rewards. ❤️

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

🗓️ This Week In AI Research (22-28 March 26)

Dr. Ashish Bamania — Tue, 31 Mar 2026 13:42:20 GMT

1. TRIBE v2

This research paper from Meta introduces TRIBE v2, a tri-modal (video, audio, and language) foundation model capable of predicting human brain activity across a variety of naturalistic and experimental conditions.

Trained on a unified dataset of over 1,000 hours of fMRI across 720 subjects, the model accurately predicts high-resolution brain responses to new stimuli, tasks, and subjects, exceeding the performance of traditional linear encoding models and delivering multi-fold improvements in accuracy.

Additionally, TRIBE v2 enables in silico experimentation. Tested on seminal visual and neuro-linguistic paradigms, it recovers a variety of results established by decades of empirical research.

Finally, by extracting interpretable latent features, TRIBE v2 reveals the fine-grained topography of multisensory integration.

Read more about this research using this link.

Join the paid tier today to get access to all posts on this newsletter, including:

🦞 Build OpenClaw from Scratch (upcoming)
🍭 Build a Mixture-of-Experts (MoE) LLM from Scratch
🦄 Building Your First AI Agent
🎈 Learn to train deep learning models on multiple GPUs

and hundreds more!

2. MolmoWeb

This research from Allen AI introduces MolmoWeb, two completely open-sourced visual web agents built on Molmo 2, along with the weights, training data (MolmoWebMix), code, and evaluation tools used to build them.

Given a task instruction and a live webpage, MolmoWeb observes the page through screenshots (without relying on HTML, accessibility trees, or specialized APIs), predicts the next step, and executes browser actions such as clicking, typing, or scrolling.

The 4B and 8B MolmoWeb models achieve state-of-the-art results among open-weight web agents, such as Fara-7B. They also outperform agents built on much larger proprietary models like GPT-4o that rely on both annotated screenshots and structured page data.

Read more about this research using this link.

3. SAM 3.1

Openclaw deletes the entire mailbox of the Director of Safety and Alignment at Meta Superintelligence Labs 🦞

Dr. Ashish Bamania — Sat, 28 Mar 2026 01:23:08 GMT

☀️ It’s a beautiful February day in 2026, and everything seems to be working well for Summer Yue.

Then, suddenly, OpenClaw starts deleting important emails from her main mailbox!

For context, Summer Yue is the director of Safety and Alignment at Meta Superintelligence Labs.

And, this was despite her instructing OpenClaw to “always confirm before acting”.

She panicked and ordered OpenClaw to stop, but the bot was completely out of control.

When the bot didn’t listen, and she couldn’t stop the ongoing mailbox deletion using her phone, she ran to her Mac mini, where her OpenClaw lived, and terminated all the processes.

She describes it in her own words:

“I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb.”

Summer was previously working with OpenClaw using a toy inbox, and the bot functioned really well. But that unfortunate day, she decided to use OpenClaw on her real mailbox with the following prompt:

“Check this inbox too and suggest what you would archive or delete, don’t action until I tell you to.”

And that’s what made the bot go completely haywire!

But why? 🤔

A Little Intro to Context Management

To understand what went wrong, we need to understand how the LLM that acts as OpenClaw's brain manages its context.

An LLM uses its context window to store instructions and previous messages from an ongoing conversation. The context window functions as its short-term or working memory.

Although massive, these context windows are limited spaces that can fill up very quickly as the conversation grows.

Image from the author’s book titled ‘LLMs In 100 Images.’

As the context window approaches its end, response quality declines because the large amount of stored information makes it harder for the LLM to attend to it all.

An LLM tends to remember the beginning and end well, but starts forgetting the information buried in the middle of its context window. This shows up as hallucinations or the LLM ignoring instructions.

Once the context window is fully filled, the LLM stops answering user queries.

To ensure that such context window exhaustion is delayed as much as possible, previous conversation history goes through either:

Trimming: Where only a certain number of the most recent conversations are kept, and the previous ones are discarded before passing to an LLM at every turn.

Trimming conversations to keep only the last 2 recent ones and discarding the previous. (Other context for the LLM is omitted from the image for clarity.)

Summarization/ Compaction: Where the previous conversations are summarized and passed to the LLM at each turn, rather than the complete conversation history.

Summarise all conversations using the same LLM or another one, and give this summary to the LLM at each turn rather than the complete conversation history. (Other context for the LLM is omitted from the image for clarity.)

Pruning: Where old tool results (file reads, API responses, browser results) are selectively removed from the context while leaving all user queries and LLM responses untouched.

So What Really Went Wrong With OpenClaw?

Since Summer was previously using a small toy mailbox with OpenClaw, things worked pretty well. But when she moved the bot to use her actual mailbox, which was quite large, the context window filled up quickly, triggering Context compaction.

This led the bot to lose its original safety instruction on not taking action without confirmation, resulting in the deletion of hundreds of important emails.

Not really! It will still happen again if you don’t do the following.

10 Tips to Not End Up in the Same Place

If the director of Safety and Alignment at Meta Superintelligence Labs can make this mistake, this might happen to you as well. But there are ways to make sure that you do not end up in the same place. Here are 10 of them.

Put your security instructions in the MEMORY.md file rather than just in the conversation. This file is not compacted and acts as a persistent memory for the OpenClaw.
Then, you have the AGENTS.md file that is the standard operating and procedural rulebook that defines operational rules, security policies, and scope boundaries for OpenClaw to follow. Like MEMORY.md, it is loaded into context at the start of every session and survives compaction. Add an instruction here for the bot to check memory and get explicit confirmation before performing any destructive action.
OpenClaw can build a small vector index over MEMORY.md and memory/*.md so that semantic queries can find related notes even when the wording differs. Make sure that this is enabled in the ~/.openclaw/openclaw.json file.

{
  "agents": {
    "defaults": {
      "memorySearch": {
        "enabled": true,
        "provider": "local"
      }
    }
  }
}

OpenClaw has a mechanism called Pre-compaction memory flush that is automatically triggered in the background as the context window is about to fill up. This saves all the important details from context to disk before compaction starts.

To enable this, go to the ~/.openclaw/openclaw.json file and make sure it contains the following:

{
  "agents": {
    "defaults": {
      "compaction": {
        "reserveTokensFloor": 20000, // Tokens reserved as buffer before compaction triggers
        "memoryFlush": {
          "enabled": true, 
          "softThresholdTokens": 4000, // // How far before the reserve floor the flush triggers
          "systemPrompt": "Session nearing compaction. Store durable memories now.", 
          "prompt": "Write any lasting notes to memory/YYYY-MM-DD.md; reply with NO_REPLY if nothing to store."  
        }
      }
    }
  }
}

Enable Context pruning, which removes old tool results from the context before sending to the bot. Add the following to the ~/.openclaw/openclaw.json file:

{
  “agents”: {
    “defaults”: {
      “contextPruning”: {
        “mode”: “cache-ttl”, // Automatically remove tool results after TTL
        “ttl”: “6h” // Keep tool results for 6 hours
      }
    }
  }
}

Use the /context list or /context detail commands in chat regularly to learn what is in the bot's context window and whether any contextual files are truncated or missing.
Always save important instructions manually to MEMORY.md file by typing “Save this to MEMORY.md“ in your chat with OpenClaw.
Don’t wait for compaction to happen automatically. Instead, perform it manually. Use the /compact command in chat when sessions feel stale, or context is getting bloated, and then give your new instructions to the bot.
If you still end up with OpenClaw misbehaving, sending /stop in the chat aborts the session and stops any active sub-agent runs spawned from it, including the nested children.
Last but not least, if none of the above works, rush to the machine running OpenClaw and pull the power plug. This should most definitely stop it.

A very helpful lesson on OpenClaw memory can be found here if you’re interested in learning about it in more detail.

P.S. This post isn't meant to mock Summer Yue, but it’s quite the opposite. She is one of the very few people who openly admitted a mistake in public that most would hide, given her job role. Thanks to her, we're all better engineers because she chose transparency over ego.

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

🗓️ This Week In AI Research (15-21 March 26)

Dr. Ashish Bamania — Wed, 25 Mar 2026 10:27:50 GMT

1. Mixture-of-Depths Attention

As LLMs become deeper, they often suffer from signal degradation. This means that important features formed in shallow layers are gradually diluted by repeated residual updates, making them harder to recover in deeper layers.

This research paper introduces the Mixture-of-depths attention (MoDA), which allows each attention head to attend to sequence KV pairs at the current layer and to depth KV pairs from preceding layers.

The authors also describe a hardware-efficient algorithm for MoDA that resolves non-contiguous memory-access patterns, achieving 97.3% of FlashAttention-2’s efficiency at a sequence length of 64K.

Experiments on 1.5B-parameter models show that MoDA consistently outperforms strong baselines, and combining MoDA with post-norm yields better performance than using it with pre-norm.

Read more about this research paper using this link.

🎁 Join the paid tier today to get access to all posts on this newsletter.

and so many more!

2. Attention Residuals

This research paper proposes Attention Residuals (AttnRes), an alternative to standard residual connections in transformers.

Typically, models add all outputs from previous layers equally, which can dilute valuable information as the model deepens. AttnRes instead allows each layer to use attention to selectively merge earlier layers, assigning learned, input-dependent weights to focus on the most relevant past information.

To increase efficiency, Block AttnRes groups layers into blocks and attends to summaries of these blocks rather than every layer, reducing memory costs while preserving most benefits of full AttnRes.

Overall, this enhances training stability, maintains stronger signals across depth, and improves performance, particularly in large models.

Read more about this research paper using this link.

3. Mamba-3

This research paper introduces Mamba-3, a new sequence model that makes inference much more efficient than transformers (which have quadratic compute and high memory costs).

The key idea is an inference-first redesign of state space models (SSMs) where Mamba-3 uses a:

More expressive recurrence
Complex-valued state update to better track information over time
Multi-input multi-output (MIMO) setup that improves performance without slowing down decoding

These enable it to handle difficult tasks such as state tracking and long-range dependencies, areas where previous linear models struggled, while maintaining near-linear efficiency. This results in Mamba-3 also achieving better accuracy and similar or lower memory use compared to previous models.

Read more about this research paper using this link.

4. V-JEPA 2.1

This research paper presents V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding.

The approach combines four key components.

A dense predictive loss that uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding.
Deep self-supervision that applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality.
Multi-modal tokenizers that enable unified training across images and videos.
Effective scaling in both model capacity and training data

Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks for short-term object-interaction anticipation and high-level action anticipation.

The model also shows strong performance in robotic navigation, depth estimation, and global recognition.

Read more about this research paper using this link.

5. GPT‑5.4 mini and nano

OpenAI released GPT‑5.4 mini and nano, which are two small, fast, and economical versions of GPT-5.4.

GPT-5.4 mini is a strong, lightweight general-purpose model, especially for coding, reasoning, multimodal understanding, and tool use, and runs over 2× faster than GPT-5-mini. It also approaches the performance of the larger GPT‑5.4 model on several evaluations, including SWE-Bench Pro and OSWorld-Verified.

GPT‑5.4 nano is the smallest, cheapest version of GPT‑5.4 for tasks where speed and cost matter most, such as classification, data extraction, ranking, and coding subagents that handle simpler supporting tasks.

Read more about this release using this link.

6. Composer-2

Composer 2 is a frontier-level coding model that offers a new, optimal combination of intelligence and cost, delivering impressive performance on hard benchmarks, including Terminal-Bench 2.0 and SWE-bench Multilingual.

The model is the result of a continued pretraining run (built on top of the open-source Kimi K2.5 model from Moonshot AI) and further training on long-horizon coding tasks through reinforcement learning, enabling it to solve challenging tasks requiring hundreds of actions.

Read more about this release using this link.

A technical report can be found using this link.

7. BioReason-Pro

This research paper introduces BioReason-Pro, a new multimodal LLM for predicting protein functions that goes beyond traditional methods that rely on sequence similarity or classification.

Instead, it explicitly models biological reasoning across sequence, structure, domains, and interaction data.

BioReason-Pro combines a new autoregressive Gene Ontology predictor (GO-GPT) with a reasoning LLM trained on large-scale synthetic reasoning traces and further improved using reinforcement learning, enabling it to generate clear, step-by-step functional annotations.

Empirically, BioReason-Pro achieves strong performance and produces higher-quality functional summaries than previous methods. Human experts prefer its annotations over curated UniProt entries about 79% of the time.

Notably, the model shows emergent capabilities such as:

accurately predicting protein binding partners
identifying interaction sites at the residue level, consistent with experimental structures
correcting misleading annotations by using contextual reasoning

Read more about this research using this link.

8. Hyperagents

This research paper introduces Hyperagents, a framework that combines a task agent, which solves problems, and a meta agent, which improves the system, into a single editable program.

Importantly, the improvement process is also editable, allowing metacognitive self-modification, meaning the system enhances its own improvement.

The research builds on the Darwin Gödel Machine (DGM) to create DGM-Hyperagents (DGM-H), which removes DGM's assumption that self-improvement only works within coding and outperforms baselines across coding, paper review, robotics reward design, and Olympiad-level math-solution grading, showing meta-level improvements that transfer across domains.

Read more about this research using this link.

9. LeWorldModel

This research paper introduces LeWorldModel (LeWM), which is the first Joint Embedding Predictive Architecture (JEPA) that trains stably end-to-end from raw pixels using only two loss terms:

a next-embedding prediction loss
a regularizer enforcing Gaussian-distributed latent embeddings

This reduces the number of tunable loss hyperparameters from 6 to 1 compared to the only existing end-to-end alternative.

With approximately 15M parameters trainable on a single GPU in a few hours, LeWM plans up to 48x faster than foundation-model-based world models while remaining competitive across diverse 2D and 3D control tasks.

Beyond control, the researchers show that LeWM's latent space encodes meaningful physical structure through probing of physical quantities. Surprise evaluation confirms that the model reliably detects physically implausible events.

Read more about this research using this link.

10. Nemotron-Cascade 2

This research paper from NVIDIA introduces Nemotron-Cascade 2, an open 30B MoE model with 3B activated parameters that delivers best-in-class reasoning and strong agentic capabilities.

Despite its compact size, its mathematical and coding reasoning performance approaches that of frontier open models. It is the second open-weight LLM, after DeepSeekV3.2-Speciale-671B-A37B, to achieve Gold Medal-level performance in the 2025 International Mathematical Olympiad (IMO), the International Olympiad in Informatics (IOI), and the ICPC World Finals, demonstrating remarkably high intelligence density with 20x fewer parameters.

Read more about this research using this link.

This article is entirely free to read. If you loved reading it, restack and share it with others to earn referral rewards. ❤️

Join the paid tier today to get access to all posts on this newsletter:

and so many more!

Link to the Visual Tech Bundle

You can also read my books on Gumroad and connect with me on LinkedIn to stay in touch.

Build and Train a Mixture-of-Experts (MoE) LLM from Scratch

Dr. Ashish Bamania — Fri, 20 Mar 2026 11:51:20 GMT

Most modern-day LLMs use the Mixture of Experts (MoE) architecture. This includes Grok-1, DeepSeekMoE, gpt-oss, and Mixtral (and many other proprietary LLMs whose architectural details aren’t publicly available).

In the previous lessons on Into AI, we learned:

In this lesson, we will:

Build an LLM using the MoE transformer with Grouped Query Attention (GQA)
Train it on a publicly available database from Wikipedia
Generate text from the trained LLM

The following are the steps that we implement.

Let’s begin!

Import Necessary Packages

# Standard libraries
import math
import random

# Hide deprecation warnings
import warnings
warnings.filterwarnings('ignore')

# PyTorch core
import torch
import torch.nn as nn
import torch.optim as optim

# For processing data
from datasets import load_dataset
from torch.utils.data import Dataset, DataLoader

# BPE tokenizer
import tiktoken

# Training utils
from torch.nn.utils import clip_grad_norm_
from torch.optim.lr_scheduler import CosineAnnealingLR

# Progress bar
from tqdm import tqdm

Before we move forward, I want to introduce you to the Visual Tech Bundle.

It is a collection of visual guides that explain core AI, LLM, Systems design, and Computer science concepts via image-first lessons.

Others are already loving these books.

This includes Dharmesh Shah, the co-founder and CEO of HubSpot.

Dharmesh’s reply to my LinkedIn post. (Source)

❤️ I’m sure that you will love these too:

Now back to our lesson!

Getting our data ready

1. Download the dataset

We will use the WikiText dataset from Hugging Face to train our model. This dataset is derived from verified Wikipedia articles and contains approximately 103 million words.

It is downloaded as follows.

# Load WikiText dataset 
train_data = load_dataset("wikitext", "wikitext-103-v1", split="train") # Train subset
val_data   = load_dataset("wikitext", "wikitext-103-v1", split="validation") # Validation subset

Let’s check out a training example from this dataset.

# Training subset example
print(train_data['text'][807])

"""
Output:
The International Civil Aviation Organization ( ICAO ) defines general aviation ( GA ) 
as " an aircraft operation other than a commercial air transport operation 
or an aerial work operation . " It defines commercial air transport ( CAT ) 
as " an aircraft operation involving the transport of passengers , cargo 
or mail for remuneration or hire " , and aerial work as " an aircraft operation 
in which an aircraft is used for specialized services such as agriculture , 
construction , photography , surveying , observation and patrol , search and 
rescue , aerial advertisement , etc . " 

"""

There’s a little problem with this dataset where some text examples are blanks and headings are formatted as “ === HEADING === “, as shown below.

# Unwanted training subset example
print(train_data['text'][804:807])

"""
Output:
['', ' = = Definitions = = \n', '']
"""

Alongside this, WikiText-103 was built with a vocabulary consisting of words (rather than sub-words/ characters), and any word that appeared fewer than three times was replaced with .

This is shown below.

# UNK marker
print(train_data['text'][80])

"""
Output:
   2 
"""

We need to get rid of these to clean up the dataset.

We also need an End-of-Text (EOS) token called <|endoftext|> between separate sections and articles, so that the model learns natural stopping boundaries and doesn't try to connect unrelated content.

(We will come back to re-using the EOS token when discussing the helper function for text generation/inference.)

Both of these changes are made using a helper function as follows.

def clean_text(dataset):
    # EOS token
    EOS = "<|endoftext|>"

    cleaned = []
    for text in dataset["text"]:
        # Strip surrounding whitespace
        line = text.strip()

        # Keep only non-empty lines that aren't headings (such as "=== HEADING ===")
        if line and not line.startswith("="):
            # Remove  markers and normalize whitespace
            cleaned.append(" ".join(line.replace("", "").split()))

    # Join all entries into a string, separated by EOS token
    return EOS.join(cleaned) + EOS
 
# Clean training and validation text
training_text = clean_text(train_data)
validation_text = clean_text(val_data)

A short subset of training_text is shown below.

# Print a subset of 'training_text'
print(training_text[3457:3700])

"""
Output:
Each character has a field and distance of movement limited by their 
Action Gauge . Up to nine characters can be assigned to a single mission . 
During gameplay , characters will call out if something happens to them , 
such as their health points ( HP ) getting low or being knocked out
"""

2. Tokenize the dataset

We previously learned how to build a character-level tokenizer from scratch.

We have also used it to train an LLM from scratch.

But because it is character-based, training an LLM using it makes the model learn the meaning of text from individual letters rather than semantic units. This makes training painfully difficult.

For this tutorial, we will instead use Tiktoken, a fast BPE tokeniser used in OpenAI's models. It is a sub-word tokenizer (breaks down words into sub-words) that better captures language structure than a character-level tokenizer, enabling faster LLM training.

Image from the author’s book titled ‘LLMs In 100 Images.’

We start by creating an instance of Tiktoken as follows.

# Create an instance of Tokenizer used by GPT-2
tokenizer = tiktoken.get_encoding("gpt2")

The vocabulary size when using the Tiktonen tokenizer for GPT-2 is 50,257. This means that the tokenizer recognizes 50,257 unique subwords.

# Check vocabulary size
vocab_size = tokenizer.n_vocab

print(f"Vocabulary size: {vocab_size}")

# Output: Vocabulary size: 50257

3. Create a dataset required for language modeling

Once we have tokenized our dataset, we need to load and serve the data during training. This is where the TextDataset class comes in, which inherits from PyTorch’s Dataset class, and has the following methods:

__init__: Tokenizes given text (tokens to token IDs) and sets the maximum sequence length (max_seq_length)
__len__: Returns the number of training sequences available
__getitem__: Returns a training sequence and its targets (tokens shifted forward by one position) at a given index

# Dataset for language modeling
class TextDataset(Dataset):
    def __init__(self, text, tokenizer, max_seq_length):
        # Convert text to token IDs
        self.tokens = tokenizer.encode(text, allowed_special="all")

        # Maximum length of each training sequence
        self.max_seq_length = max_seq_length

    # Get number of valid training sequences
    def __len__(self):
        num_sequences = (len(self.tokens) - 1) // self.max_seq_length
        return num_sequences

    # Get an input sequence and targets
    def __getitem__(self, idx):
        # Start index of the sequence
        start = idx * self.max_seq_length

        # End index of the sequence
        end = start + self.max_seq_length

        # Input token sequence
        input_ids = torch.tensor(self.tokens[start:end], dtype=torch.long)

        # Next-token targets/ labels (shifted by one character)
        target_ids = torch.tensor(self.tokens[start+1:end+1], dtype=torch.long)

        return input_ids, target_ids

Let’s use the TextDataset class to create the training and validation datasets.

# Define maximum sequence length
MAX_SEQ_LENGTH = 128

# Create training and validation datasets
train_dataset = TextDataset(training_text, tokenizer, MAX_SEQ_LENGTH)
val_dataset = TextDataset(validation_text, tokenizer, MAX_SEQ_LENGTH)

The number of training sequences in the training and validation datasets is as follows.

print(f"Number of training sequences: {len(train_dataset):,}")
print(f"Number of validation sequences: {len(val_dataset):,}")

"""
Output:
Number of training sequences: 876,162
Number of validation sequences: 1,828
"""

And this is how a training sequence and its target look.

# Example of a training sequence and its target
input, target = train_dataset[16]

print("Input IDs:\n", input)
print("\nTarget IDs:\n", target)

print("\nDecoded Input:\n", tokenizer.decode(input.tolist()))
print("\nDecoded Target:\n", tokenizer.decode(target.tolist()))

"""
Output:
Input IDs:
 tensor([  262,  1271,   286, 22469,  4991,   583,  4365,   764,   317,   636,
          286,   428,  8515,  2950,  4441,  3748,  7514, 14520,  4981,   329,
         1123,  2095,   705,    82,  1767,   764,   554,  1502,   284,  4620,
          428,   837,   262, 22849,  4847, 16560,   656,   262,  1218,   983,
          547,  4615,   837,   355,   484,  1718,   510,   257,  1588,  6903,
          286,  4088,  2272,  2622,   329,   262,  8561,   764,  1119,   635,
        12328,   262,  8722,  6460,   290, 10152,   286,   711,   523,   484,
          714,  5198,   284,   649,  1938,   981, 26645,   262,  6393,  6805,
          286,   262,  2168,   705, 11327,   764,   383, 15064,  3341,   547,
         3066,  2402,  1903,   287,  2478,   764,   383,  2095,  9824,   547,
         1760,   416,  8835,    73,   280,   837,   508,   550,  3111,   319,
          262,  2180,   569, 18354,  7496, 17740,  1830,   764,  1649,  4441,
          262, 17871,  5321, 11630,   837,  8835,    73,   280])

Target IDs:
 tensor([ 1271,   286, 22469,  4991,   583,  4365,   764,   317,   636,   286,
          428,  8515,  2950,  4441,  3748,  7514, 14520,  4981,   329,  1123,
         2095,   705,    82,  1767,   764,   554,  1502,   284,  4620,   428,
          837,   262, 22849,  4847, 16560,   656,   262,  1218,   983,   547,
         4615,   837,   355,   484,  1718,   510,   257,  1588,  6903,   286,
         4088,  2272,  2622,   329,   262,  8561,   764,  1119,   635, 12328,
          262,  8722,  6460,   290, 10152,   286,   711,   523,   484,   714,
         5198,   284,   649,  1938,   981, 26645,   262,  6393,  6805,   286,
          262,  2168,   705, 11327,   764,   383, 15064,  3341,   547,  3066,
         2402,  1903,   287,  2478,   764,   383,  2095,  9824,   547,  1760,
          416,  8835,    73,   280,   837,   508,   550,  3111,   319,   262,
         2180,   569, 18354,  7496, 17740,  1830,   764,  1649,  4441,   262,
        17871,  5321, 11630,   837,  8835,    73,   280,   373])

Decoded Input:
  the number of playable units per mission . A part of this upgrade involved creating 
  unique polygon models for each character 's body . In order to achieve this , 
  the cooperative elements incorporated into the second game were removed , 
  as they took up a large portion of memory space needed for the improvements . 
  They also adjusted the difficulty settings and ease of play so they could appeal 
  to new players while retaining the essential components of the series ' gameplay . 
  The newer systems were decided upon early in development . The character designs 
  were done by Honjou , who had worked on the previous Valkyria Chronicles games . 
  When creating the Nameless Squad , Honjou

Decoded Target:
  number of playable units per mission . A part of this upgrade involved creating 
  unique polygon models for each character 's body . In order to achieve this , 
  the cooperative elements incorporated into the second game were removed , 
  as they took up a large portion of memory space needed for the improvements . 
  They also adjusted the difficulty settings and ease of play so they could appeal 
  to new players while retaining the essential components of the series ' gameplay . 
  The newer systems were decided upon early in development . The character designs 
  were done by Honjou , who had worked on the previous Valkyria Chronicles games . 
  When creating the Nameless Squad , Honjou was
"""

Note how the input and target sequences are shifted by one sub-word. This is to help the model learn to predict each next sub-word given the previous ones.

Setting up DataLoader for Batch training

Next, we use PyTorch DataLoader to create an iterable that loads shuffled batches of BATCH_SIZE samples from train_dataset and val_dataset for training and validation.

# Define batch size
BATCH_SIZE = 32

# Create training and validation DataLoaders
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=0)
val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=0)

This creates the following number of batches of training and validation data.

print(f"Number of training batches: {len(train_loader):,}") # Number of training sequences / batch_size
print(f"Number of validation batches: {len(val_loader):,}") # Number of validation sequences / batch_size

"""
Output:
Number of training batches: 27,381
Number of validation batches: 58
"""

Our data is now ready for model training!

Building the MoE Transformer model

We will now set up our MoE transformer model that we built in a previous lesson with a few changes (based on what modern LLMs follow):

We replace the Causal Multi-head Self-attention module with Causal Grouped-Query Attention
We introduce Weight tying, which means we reuse the same weights for both the token embedding matrix and the output projection matrix, rather than learning two separate ones. This is because they do reciprocal functions and have the same shape. This decreases the number of learning parameters in the model by a few million.
We remove the bias term from all linear layers following the Llama models

Let’s implement these one by one.

1. Set up Causal Grouped Query Attention

# Grouped Query Attention (GQA) with Causal Masking
class GroupedQueryAttention(nn.Module):
    def __init__(self, embedding_dim, num_heads, num_groups, max_seq_length):
        super().__init__()
 
        # Check if embedding_dim is divisible by num_heads
        assert embedding_dim % num_heads == 0, "embedding_dim must be divisible by num_heads"
 
        # Check if num_heads is divisible by num_groups
        # (Each group must be shared by the same number of heads)
        assert num_heads % num_groups == 0, "num_heads must be divisible by num_groups"
 
        # Embedding dimension
        self.embedding_dim = embedding_dim
 
        # Number of total query heads
        self.num_heads = num_heads
 
        # Dimension of each head
        self.head_dim = embedding_dim // num_heads
 
        # Number of KV groups
        self.num_groups = num_groups
 
        # Number of query heads per KV group
        self.group_size = num_heads // num_groups
 
        # Linear projection matrix for query
        self.W_q = nn.Linear(embedding_dim, embedding_dim, bias=False)
 
        # Linear projection matrices for key and value
        self.W_k = nn.Linear(embedding_dim, num_groups * self.head_dim, bias=False)
        self.W_v = nn.Linear(embedding_dim, num_groups * self.head_dim, bias=False)
 
        # Linear projection matrix to produce final output
        self.W_o = nn.Linear(embedding_dim, embedding_dim, bias=False)
 
        # Build the causal mask once at init instead of every forward pass
        mask = torch.tril(torch.ones(max_seq_length, max_seq_length))
 
        # Add batch_size and num_heads dimensions
        mask = mask.view(1, 1, max_seq_length, max_seq_length)

        # register_buffer saves it as part of the model so it automatically moves to the right device (CPU/GPU)
        self.register_buffer("causal_mask", mask)
 
    # Splits Q into multiple heads
    def _split_heads(self, x):
        """
        Transforms input embeddings from
        [batch_size, sequence_length, embedding_dim]
        to
        [batch_size, num_heads, sequence_length, head_dim]
        """
        batch_size, sequence_length, embedding_dim = x.shape
 
        # Split embedding_dim into (num_heads, head_dim)
        x = x.reshape(batch_size, sequence_length, self.num_heads, self.head_dim)
 
        # Reorder and return the intended shape
        return x.transpose(1, 2)
 
    # Splits K or V into num_groups heads
    def _split_groups(self, x):
        """
        Transforms K/V from
        [batch_size, sequence_length, num_groups * head_dim]
        to
        [batch_size, num_groups, sequence_length, head_dim]
        """
        batch_size, sequence_length, _ = x.shape
 
        x = x.reshape(batch_size, sequence_length, self.num_groups, self.head_dim)
        return x.transpose(1, 2)
 
    # Merge heads back together
    def _merge_heads(self, x):
        """
        Transforms inputs from
        [batch_size, num_heads, sequence_length, head_dim]
        to
        [batch_size, sequence_length, embedding_dim]
        """
        batch_size, num_heads, sequence_length, head_dim = x.shape
 
        # Move sequence_length back before num_heads in the shape
        x = x.transpose(1, 2)
 
        # Merge (num_heads, head_dim) back into embedding_dim
        embedding_dim = num_heads * head_dim
        x = x.reshape(batch_size, sequence_length, embedding_dim)
 
        return x
 
    # Forward pass
    def forward(self, x):
        batch_size, sequence_length, embedding_dim = x.shape
 
        # Compute Q, K, V
        Q = self.W_q(x)  # [batch_size, sequence_length, embedding_dim]
        K = self.W_k(x)  # [batch_size, sequence_length, num_groups * head_dim]
        V = self.W_v(x)  # [batch_size, sequence_length, num_groups * head_dim]
 
        # Split Q into multiple heads
        Q = self._split_heads(Q)  # [batch_size, num_heads, sequence_length, head_dim]
 
        # Split K and V into num_groups heads
        K = self._split_groups(K)  # [batch_size, num_groups, sequence_length, head_dim]
        V = self._split_groups(V)  # [batch_size, num_groups, sequence_length, head_dim]
 
        # Expand K and V so each KV group is shared across multiple query heads
        K = K.repeat_interleave(self.group_size, dim=1)  # [batch_size, num_heads, sequence_length, head_dim]
        V = V.repeat_interleave(self.group_size, dim=1)  # [batch_size, num_heads, sequence_length, head_dim]
 
        # Calculate scaled dot-product attention
        attn_scores = Q @ K.transpose(-2, -1)
        attn_scores = attn_scores / math.sqrt(self.head_dim)
 
        # Slice the pre-built mask to the current sequence length
        causal_mask = self.causal_mask[:, :, :sequence_length, :sequence_length]
 
        # Mask out future positions by setting their scores to -inf
        attn_scores = attn_scores.masked_fill(causal_mask == 0, float('-inf'))
 
        # Apply softmax to get attention weights
        attn_weights = torch.softmax(attn_scores, dim=-1)
 
        # Multiply attention weights by V
        weighted_values = attn_weights @ V  # [batch_size, num_heads, sequence_length, head_dim]
 
        # Merge head outputs
        merged_heads_output = self._merge_heads(weighted_values)
 
        # Obtain final output
        output = self.W_o(merged_heads_output)
 
        return output

2. Set up Expert and Router

An Expert is simply a small feed-forward network that specializes in a particular skill, which may involve processing different token types (numerical/code/language-related tokens), reasoning patterns, context patterns, and so on.

🗓️ This Week In AI Research (8-14 March 26)

Dr. Ashish Bamania — Thu, 19 Mar 2026 11:14:13 GMT

1. Covenant-72B

This research paper presents Covenant-72B, an LLM produced by the largest globally distributed collaborative pre-training run to date, in terms of both compute and model scale.

This pre-training involved open, permissionless participation, supported by a live blockchain protocol, Bittensor, and a state-of-the-art, communication-efficient optimizer called SparseLoCo, which enabled dynamic participation with peers joining and leaving freely.

Covenant-72B was pre-trained on approximately 1.1T tokens and performs at par with fully centralized models pre-trained on similar or higher compute budgets.

This shows that open and fully democratized participation is possible on an unprecedented scale for a globally distributed LLM pre-training run.

Read more about this research using this link.

🎁 Unlock every lesson in the newsletter and accelerate your AI engineering career with a 25% discount on the annual membership of ‘Into AI’ today!

2. Training Language Models via Neural Cellular Automata

Natural LLM pre-training data has its flaws, as it is finite, contains human biases, and mixes knowledge with reasoning.

This research paper proposes using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs, which means training on synthetic data first, followed by natural language.

This is because NCA data has a rich spatiotemporal structure and statistics that resemble natural language, while being controllable and cheap to generate at scale.

Pre-pre-training is surprisingly effective, and training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x.

This approach of using NCA even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute, and the gains transfer to reasoning benchmarks, such as GSM8K, HumanEval, and BigBench-Lite.

Further experiments show that attention layers transfer best, and that optimal NCA complexity depends on the task, with simpler dynamics working better for code, while more complex ones benefit math and web text more.

Read more about this research using this link.

3. Lost in Backpropagation: The LM Head is a Gradient Bottleneck

The final layer of LLMs, called the LM head, maps the hidden dimension (D) to the vocabulary dimension (V), where usually D is very small as compared to V. This creates an expressivity limitation called the Softmax bottleneck.

This research paper shows that the softmax bottleneck is not only an expressivity bottleneck but also an optimization bottleneck where high-dimensional gradients are compressed into a much smaller space, leading to most of the training signal (around 95–99% of the gradient norm) getting lost.

This leads to poor update directions and makes even simple patterns harder to learn, slowing down training and harming efficiency across models.

The research suggests that current LLM architectures have a fundamental flaw in how they learn and that better LM head designs are needed.

Read more about this research using this link.

4. OpenClaw-RL: Train Any Agent Simply by Talking

This research paper presents OpenClaw-RL, a reinforcement learning framework in which AI agents learn continuously from real-time interactions, using next-state signals (such as user replies, tool outputs, or environment changes) as training data.

These signals offer evaluative feedback (how well an action is performed, expressed as scalar rewards via a PRM judge) and directive feedback (how the action should be improved, expressed as textual hints and used for token-level supervision via Hindsight-Guided On-Policy Distillation, or OPD).

By blending these, OpenClaw-RL supports more effective learning than traditional methods.

Its asynchronous, decoupled architecture also allows agents to improve during live use in various environments, such as chat, tools, and software tasks, without requiring offline datasets or manual labeling.

Read more about this research using this link.

5. How Far Can Unsupervised RLVR Scale LLM Training?

This research paper studies a training approach called Unsupervised Reinforcement Learning with Verifiable Rewards (RLVR), in which models learn without labeled data by generating their own rewards from signals that can be automatically verified.

The paper categorises RLVR methods into intrinsic (self-generated rewards) and external (externally verified rewards), and shows that intrinsic methods mainly strengthen a model’s existing beliefs and work only if those beliefs are already correct, but can lead to collapse if they are wrong.

While intrinsic rewards still help with test-time training on small datasets, the authors propose the Model Collapse Step, which acts as a practical indicator for RL trainability (predicting when training will fail).

They also present early evidence that external reward methods may overcome these limitations, suggesting more scalable approaches.

Read more about this research using this link.

6. Attend Before Attention

This research paper introduces AutoGaze for multi-modal LLMs (MLLMs), an efficient method for processing videos by selectively focusing on important regions (like human gaze) before applying full attention, rather than analyzing every pixel equally.

Such gazing lets the model ignore unnecessary or static information, enabling faster, easier-to-scale video understanding while maintaining high performance.

Experiments show that this approach greatly increases efficiency (higher processing FPS) and allows for real-time video processing, offering a practical way to scale multimodal models to longer or streaming video inputs.

Read more about this research using this link.

7. Geometry-Grounded Point Transformer

Feed-forward networks suffer from geometric inconsistencies and achieve only limited fine-grained accuracy in 3D reconstruction (i.e., predicting dense point maps directly from RGB images).

This research paper introduces the Geometry-Grounded Point Transformer (GGPT), a method for improving 3D reconstruction from images by combining fast neural predictions with explicit geometric guidance.

It uses an improved Structure-from-Motion pipeline to estimate accurate camera poses and partial 3D structure from sparse input views, and then uses a Geometry-guided 3D point transformer to refine dense point maps under explicit partial-geometry supervision using an optimised guidance encoding.

Experiments show that GGPT produces reconstructions that are both geometrically consistent and spatially complete.

While being trained solely on ScanNet++ with VGGT predictions, it generalises across architectures and datasets, significantly outperforming SOTA feed-forward 3D reconstruction models in both in-domain and out-of-domain settings.

Read more about this research using this link.

8. The Attack and Defense Landscape of Agentic AI

This research paper is a comprehensive survey of security risks in AI agent systems that examines how design choices, such as tool use, memory, and autonomy, create new attack surfaces.

It uses case studies to highlight existing gaps in securing agentic AI systems and identifies open challenges in this domain.

It also introduces the first systematic framework for understanding the security risks and defense strategies of AI agents, serving as a foundation for building secure agentic systems and advancing research.

Read more about this research using this link.

9.IndexCache

This research paper presents IndexCache, a method that improves sparse attention in LLMs by reusing attention indices across layers rather than recalculating them each time.

In a standard sparse attention mechanism, models repeatedly select the most relevant tokens at each layer, which is computationally expensive. This research shows that these selections are often similar across layers and can be cached and reused.

By adding this cross-layer index reuse method, the approach reduces computation and latency while maintaining model performance, making long-context and agent-style workloads more efficient and scalable.

Read more about this research using this link.

10. Neural Thickets

This research paper suggests that large pre-trained LLMs already have many good solutions for task-specific experts nearby in their weights. They do not require extensive training or optimization to uncover them.

Instead of using complex optimization methods, the authors propose a simple, fully parallel post-training approach where they make small random changes to the model parameters, select the top-performing variants, and combine their predictions using majority voting.

This straightforward approach is surprisingly effective and works almost as well as standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.

Read more about this research using this link.

This article is entirely free to read. If you loved reading it, restack and share it with others to earn referral rewards. ❤️

Join the paid tier today at a 25% discount to get access to all posts on this newsletter!

You can also check out my books on Gumroad and connect with me on LinkedIn to stay in touch.

🗓️ This Week In AI Research (1-7 March 26)

Dr. Ashish Bamania — Tue, 10 Mar 2026 12:38:02 GMT

1. Introducing GPT‑5.4

OpenAI released GPT-5.4, with improved reasoning, coding, and professional knowledge capabilities, enabling the development of more dependable AI agents.

GPT-5.4 can perform hard tasks across documents, spreadsheets, presentations, and software tools, producing higher-quality outputs and helping teams complete workflows faster.
It is the first general-purpose model with native, state-of-the-art computer-use capabilities.
It supports up to 1M tokens of context, allowing agents to plan, execute, and verify tasks across long horizons.

It is available across ChatGPT, the OpenAI API, and Codex, with variants such as GPT-5.4 Thinking for deeper reasoning and GPT-5.4 Pro for more demanding workloads.

Read more about this release using this link.

🚨 A Special Offer

Grab a 25% discount on the annual subscription to the newsletter and get access to articles like these, and so much more:

Your 25% off

2. FlashAttention-4

This research paper introduces the latest version, FlashAttention-4, which makes transformer attention faster and more efficient on modern GPUs.

FlashAttention-3 was optimized attention for Hopper GPUs, specifically targeting the H100 architecture. But the industry has moved to Blackwell-based systems such as the B200 and GB200, which come with fundamentally different performance characteristics.

FlashAttention-4 optimizes attention on Blackwell GPUs by:

Redesigning pipelines that take advantage of fully asynchronous Matrix Multiply-Accumulate (MMA) operations and larger tile sizes
Software-based implementations of exponential and conditional softmax rescaling, which reduce the number of non-matrix operations and improve overall speed
Better use of GPU tensor memory and the 2-CTA MMA mode (which allows two Cooperative Thread Arrays to jointly execute a single Matrix Multiply-Accumulate operation), which reduces shared-memory traffic and atomic adds required during the backward pass

FlashAttention-4 achieves up to:

a 1.3× speedup over cuDNN 9.13
a 2.7× speedup over Triton on B200 GPUs with BF16, reaching up to 1613 TFLOPs/s (71% utilization)

The researchers also implement FlashAttention-4 entirely in CuTe-DSL embedded in Python, resulting in 20-30× faster compile times compared to traditional C++ template-based approaches while maintaining full expressivity.

Read more about this research using this link.

3. Solving an Open Problem in Theoretical Physics using AI-Assisted Discovery

This research paper describes a neuro-symbolic system that autonomously solved an open problem in theoretical physics.

This system combines Gemini Deep Think with a systematic Tree Search (TS) algorithm and an automated numerical feedback system to explore mathematical ideas and verify them computationally.

The system successfully derived new and exact analytical solutions for the power spectrum of gravitational radiation emitted by cosmic strings.

The research system identified 6 different analytical methods, the most elegant of which uses the Gegenbauer expansion.

Read more about this research using this link.

4. The Spike, the Sparse, and the Sink

This research paper studies two phenomena that usually co-occur and often involve the same tokens in Transformer-based LLMs:

Massive activations: where some tokens produce extremely large activations (“Spikes”)
Attention sinks: where some tokens consistently attract attention disproportionate to their semantic importance (“Sinks”)

Experiments show that:

The co-occurrence of the two is an architectural artifact of modern Transformer design.
Massive activations work globally to create nearly constant hidden representations that persist across many layers, effectively acting like implicit parameters of the model.
Attention sinks work locally by influencing attention outputs across heads, pushing individual heads to focus more on short-range dependencies.
Pre-norm is the main architectural choice that allows these two phenomena to appear together, and removing pre-norm causes them to decouple.

Read more about this research using this link.

5. Speculative Speculative Decoding

Speculative decoding has become a standard way to accelerate autoregressive inference in LLMs.

It uses a fast draft model to predict upcoming tokens from a slower, larger target model, and then verifies them in parallel with a single forward pass of the target model.

However, the method is still partly sequential because the draft must wait for verification before making the next speculation.

This research paper introduces Speculative speculative decoding (SSD), which parallelizes the speculation and verification operations.

While verification is ongoing, the draft model predicts likely verification outcomes and prepares pre-emptive speculations for them. If the actual verification outcome is then in the predicted set, a speculation is returned immediately, removing drafting delays entirely.

This method is further improved, leading to Saguaro, an optimized SSD algorithm that is up to 5x faster than autoregressive decoding on open-source inference engines.

Read more about this research using this link.

6. Olmo Hybrid

Hybrid LLMs use an architecture that combines transformer attention and linear RNN layers.

Despite previous releases such as Nemotron-H, Qwen3-Next, Kimi Linear, and Qwen 3.5, it was unclear if the purported benefits of hybrid architectures justify the cost of scaling them up.

This debate has been settled by Ai2 researchers, who released the fully open-source Olmo Hybrid, a 7B-parameter model largely comparable to Olmo 3 7B, but with its sliding-window layers replaced by Gated DeltaNet layers.

Olmo Hybrid is trained using 49% fewer training tokens and outperforms Olmo 3 across standard pre-training and mid-training evaluations.

This is empirical proof that hybrid architectures not only reduce memory usage during inference but also obtain more expressive models that scale better during pretraining.

Architecture comparison for Olmo 3 and Olmo Hybrid. Olmo Hybrid replaces the sliding-window attention layers from Olmo 3 with Gated DeltaNet.

Read more about this research using this link.

7. Reasoning Models Struggle to Control their Chains of Thought

Chain-of-thought (CoT) monitoring is a promising method for understanding the intents and finding misbehaviors in modern reasoning LLMs.

This research paper examines whether CoT monitoring is a reliable safety tool or whether reasoning models can control what they reveal in their chain-of-thought (CoT), a phenomenon known as CoT controllability.

For this, the researchers introduce an evaluation suite called CoT-Control, which includes tasks that require reasoning models to solve problems while adhering to CoT instructions. (For example, reasoning about a genetics question without using the word “chromosome”.)

Experiments show that:

Reasoning models have significantly lower CoT controllability than output controllability. For example, Claude Sonnet 4.5 can control its CoT only 2.7% of the time, but 61.9% of the time when controlling its final output.
CoT controllability is higher for larger models and decreases with more RL training, test-time compute, and increased problem difficulty.
CoT controllability failures extend even to situations in which models are given incentives (as opposed to direct requests) to evade CoT monitors
Models exhibit slightly higher controllability when they are told they are being monitored.
Eliciting controllability by adversarially optimizing prompts does not meaningfully increase controllability.

The overall results indicate that CoT controllability is currently unlikely to be a failure mode of CoT monitorability and must be tracked by frontier labs in future models.

Read more about this research using this link.