Jev is not an LLM, but a System One model. This terminology is based on Daniel Kahneman’s systems of thinking, in which:

• System 1 refers to fast, pattern-based thinking (e.g., recognizing faces)

• System 2 refers to slow, effortful reasoning (e.g., solving complex math problems)

Like LLMs, System One models understand natural-language inputs.

But unlike LLMs, they do not write replies, produce code, or generate explanations of their reasoning. Instead of generating text, these models return typed decisions and probabilities. (Classical ML enthusiasts would love this!)

Subscribe now

What do Jev’s inputs and outputs look like?

Jev accepts the following two inputs:

1. State: the content that the model has to evaluate. This is text or JSON.

2. Questions: the judgments the model should make given the state.

Given these inputs, Jev returns one typed answer per question with probabilities and a confidence score.

For example, an input could be:

• State: “My app keeps crashing when I click the ‘Submit’ button.”

• Question: “Which team should handle this user request: Technical or the Marketing team?”

For this input, the model would return a response such as:

“Technical (98% probability of this option with confidence of 95%).”

Join the paid tier today to get access to all articles in this newsletter and level up as an AI engineer.

Become a paid subscriber today

What is State?

State is the content passed to Jev around which you ask questions.

It is declared using the state field in the input/ API request, and each such request evaluates one state against one or more questions.

A state could be either:

• a string

  "I have chest tightness for the last 2 days that worsens when going upstairs"

• an array

  [
    "Hi, I've had chest tightness that's worse when going upstairs for the last 2 days",
    "I have diabetes and high blood pressure"
  ]

• an object

  {
    "age": 67,
    "medical_conditions": ["Diabetes", "Hypertension"],
    "message": "I've chest tightness for 2 days that worsens when going upstairs"
  }

What are Primitives?

Primitives are the building blocks that form the inputs (questions) and outputs (answers) to Jev. These are of three types, and each answers something different:

1. Choice: Which of these options? (e.g., “Which team should the request be redirected to?”)

2. Score: Which level/ score? (e.g., “How relevant is this candidate's experience to the job posting?”)

3. Noul: Is this true or false? (e.g., “Is the customer requesting a refund?”)

These primitives are used to create questions that are passed to Jev.

Each question has the following fields:

• ID: This is used to identify the answer in the response.

• type: This is the type of primitive, either Choice, Score, or Noul.

• instructions: This is the question you are asking about the state.

• If a question contains the Choice and Score primitives, they also take criteria (the possible answers), which define the options for a Choice question or the levels for a Score question.

• Questions with the Noul primitive can also have an optional criteria which clarifies what yes and no mean.

The answer depends on the type of primitive used in a question.

• Choice question returns:

  • choice (the option that Jev chose)

  • probabilities (probability distribution over the options given in the criteria field), and

  • confidence (this is a single value derived from the probability distribution which tells how certain Jev is about its choice)

• A Score question returns:

  • score (a value between the given levels)

  • legend (each level and its description)

  • probabilities

  • confidence

• A Noul question returns just noul. This is the probability that the answer is yes/ true.
  
  A confidence value isn’t returned here, and a noul value near 1 means a strong yes, near 0 means a strong no, and 0.5 means uncertain.

Why State and Primitives?

• Primitives make Jev type-safe with its inputs and outputs. You get the exact type of response that you intend.

• Also, every answer that Jev chooses comes from the options given to it. This ensures that it never hallucinates new options. Note that this does not mean that Jev does not make mistakes. It can still choose the wrong option from the given ones, but it will never invent a new option.

• Confidence is what lets the model say it is not sure, rather than agreeing with what the user intends to get out of the model. This makes Jev foundational to building reliable systems.

Constructing an input to Jev

Now that we understand Primitives, it’s time to look at what an actual input to Jev looks like.

Consider the following input for a patient triage app where we send the state as an array of patient messages, and we ask three questions regarding it:

• a Noul question (labeled as urgency) on whether the patient message needs an urgent response

• a Choice question (team) of which team should handle it, and

• a Score question (severity) on how severe the symptoms are.

{
  "model": "jev-latest",
  "state": [
    "Hi, I've had chest tightness that's worse when going upstairs for the last 2 days",
    "I have diabetes and high blood pressure"
  ],
  "questions": {
    "urgency": {
      "type": "noul",
      "instructions": "Does this patient message need urgent response?"
    },
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "pharmacy": "For medication based issues",
        "emergency_doctor": "For medical issues that need urgent or emergency attention",
        "billing": "For pricing related issues"
      }
    },
    "severity": {
      "type": "score",
      "instructions": "How severe are the symptoms described by the patient?",
      "criteria": [
        "Mild, no immediate concern",
        "Moderate, can be seen in a few days",
        "Serious, needs same-day attention",
        "Critical, needs emergency care now"
      ]
    }
  }
}

Jev will answer all three questions independently in parallel and return their respective probabilities and confidence.

Note that the answer to one question does not affect another in any way. Questions are completely independent, and one question’s answer is not hidden context for another.

An example response looks as follows.

{
  "model": "jev-1.13.0",
  "answers": {
    "urgency": {
      "type": "noul",
      "noul": 0.94
    },
    "team": {
      "type": "choice",
      "choice": "emergency_doctor",
      "probabilities": {
        "pharmacy": 0.02,
        "emergency_doctor": 0.95,
        "billing": 0.03
      },
      "confidence": 0.93
    },
    "severity": {
      "type": "score",
      "score": 2.2,
      "legend": {
        "0": "Mild, no immediate concern",
        "1": "Moderate, can be seen in a few days",
        "2": "Serious, needs same-day attention",
        "3": "Critical, needs emergency care now"
      },
      "probabilities": {
        "0": 0.01,
        "1": 0.09,
        "2": 0.58,
        "3": 0.32
      },
      "confidence": 0.71
    }
  },
  "usage": {
    "input_tokens": 180,
    "output_tokens": 24
  }
}

Notice the input and output tokens? You're billed only for input tokens, while the output tokens are completely free. We will discuss this in the next section.

Subscribe now

What makes Jev better than LLMs?

LLMs are post-trained using RLHF. This teaches them to say things that people prefer. Now this works well for chatbots in most cases, but it also rewards sycophancy (models praising users insincerely), confident-sounding hallucinations (inventing things that do not exist), and a preference for a particular style of responses (“You’re absolutely right!”).

Instead of RLHF, Jev is trained using Reinforcement Learning for Calibrated Decisions (RLCD). This algorithm trains Jev to output decisions and calibrated probabilities rather than generated text aligned with human preferences.

Jev is also cheap, at only $0.042 per million input tokens with completely free output tokens. This is because most computation occurs on the input tokens, and its outputs are tiny, since it does not generate text or reasoning tokens.

Compare this to Opus 5.5, a reasoning model that can produce a large volume of reasoning-related output tokens for decision-making and costs $4 per million input tokens and $20 per million output tokens.

(But take this price comparison with a grain of salt, as Jev struggles when making complex decisions and is not a replacement for reasoning LLMs in such use cases.)

Another great feature of Jev is its self-consistency, which makes its outputs reliable across repeated runs with the same input. In an experiment run by LangChain, Jev had the lowest observed mean per-case variance of 0.0000149. In the same experiment, GPT‑5.6 Luna’s variance was 433× higher, GPT-5.6 Terra’s was 913× higher, and Claude Sonnet 4.6’s was 92× higher.

What can you use Jev for?

Jev can be used for tasks that require fast, consistent decisions, ideally when those decisions do not require complex, multi-hop reasoning.

Some use cases of Jev are shown below. You can check out this page for a detailed description of these use cases.

TL;DR

• Jev is a System One model from TypeSafe AI, trained to generate decisions based on fast, pattern-based thinking rather than the complex reasoning used by System Two models like LLMs.

• Instead of generating text, Jev returns typed decisions with probabilities and a confidence score.

• Inputs to Jev are a state and one or more questions, based on three building blocks, or Primitives: Choice, Score, and Noul.

• ‘Choice’ based questions return a choice of answer from the given options, ‘Score’ based questions return a score from the given levels, and ‘Noul’ based questions return the probability that something is true.

• Answers always come from the options that are given to the model to pick from and are returned with probabilities (and confidence for ‘Choice’ and ‘Score’ based questions).

• Each question is answered independently, and one question’s answer is not used as hidden context for another.

• Jev is trained using RLCD (Reinforcement Learning for Calibrated Decisions) instead of RLHF. This means that Jev is not post-trained to produce outputs that are aligned with human preferences.

• It is built to be used for fast judgment tasks like model routing, content moderation, guardrails, risk assessment, and more.

• It is highly self-consistent across repeated turns of the same input, making it ideal for building reliable systems.

• It costs $0.042 per million input tokens with free output tokens.

Join the paid tier today to get access to all posts in this newsletter, including:

• 🧑🏻‍💻 Build and train a Diffusion LLM from scratch

• 🌈 20 PyTorch Concepts, Explained Simply

• ⚙️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🍭 9 GPU Concepts Every AI Engineer Should Know

and so many more!

Become a paid subscriber today

1. Prefill, which processes the input prompt and creates the KV cache. Since it involves processing all tokens in the input prompt together, it is compute-bound.

2. Decode, which generates tokens one at a time, autoregressively. It is memory-bandwidth bound because very little computation is performed at each step relative to the amount of data (primarily model parameters and the KV cache) moved from memory.

Now that we understand this, here are the 10 inference metrics that you must know when deploying LLMs in production. Metrics 1-4 measure latency, 5-6 measure throughput, and 7-10 measure the efficiency of GPU compute, memory, and cache usage.

Subscribe now

1. Time to first token (TTFT)

This is the time between the request’s arrival at the serving system and the generation of the first token.

TTFT is the latency metric that users perceive when interacting with an LLM-based system. It is what makes or breaks an interactive workload such as a real-time chat application.

It is determined by:

• Queue time, which means the time between request arrival and the forward pass through the LLM. This is when a request is waiting for GPU/batch resources. Higher queue time means longer TTFT.

• Length of the input prompt: Longer prompts take more time to process during prefill, resulting in a longer TTFT.

• Prefill compute: The higher the compute available, the lower the TTFT.

When optimizing for low TTFT, it must always be determined whether the cause is a bottleneck in the serving system or the GPU's compute limitations.

2. Time per output token (TPOT)

TPOT is the average time to generate each token during Decode. The first token is excluded as it is a result of the Prefill.

The faster an LLM’s parameters can be moved from memory, the faster token generation during Decode is, and the lower the TPOT is.

TPOT is determined by:

• The number of model parameters and their precision (lower means lower TPOT because the amount of data moved per decode step goes down)

• Context length (higher values mean higher TPOT because longer sequences increase KV‑cache size that must be read to generate each token)

• Batch size (higher values mean higher TPOT because more concurrent computation must occur per decode step)

• GPU memory bandwidth (higher values mean lower TPOT because data can be moved faster from memory during Decode)

3. Inter-token latency (ITL)

ITL is frequently confused with TPOT. While TPOT measures the average time per generated token during Decode, ITL measures the latency between two consecutive generated tokens.

In simple terms, TPOT is the average of a request's ITLs.

For example, if the latency between tokens in a response is 42 ms, 46 ms, 60 ms, and 35 ms, these individual numbers are the ITLs, and their average (45.75 ms) is the TPOT.

When serving an interactive application, it is important to ensure that ITL remains consistent. Otherwise, the interaction will feel jittery, leading to poor user satisfaction.

The metric to consider in such cases is the p99 ITL, or the 99th percentile inter-token latency. As an example, if p99 ITL is 120 ms, it means 99% of ITLs are 120 ms or less, and the slowest 1% are above 120 ms.

p99 ITL is called tail latency because it measures latency in the high-percentile tail of the latency distribution.

4. End-to-end (E2E) latency

Read more

LLM training is commonly done in three stages, and each stage helps the model to become more useful. These stages are as follows:

1. Pretraining: Teaches an LLM the basics of language and gives it foundational knowledge of the world.

2. Mid-training: Improves the capabilities of an LLM in intended domains (coding, health, law, etc.).

3. Post-training: Gives an LLM reasoning capabilities and helps align it to be a useful, human-value-aligned assistant.

Two popular reinforcement learning algorithms used in LLM post-training are PPO and GRPO. Let’s learn how these work and differ.

Subscribe now

PPO

Proximal Policy Optimization (PPO) is an algorithm used to align LLMs to be helpful, honest, and harmless assistants. This is done using a process called Reinforcement Learning from Human Feedback (RLHF).

PPO was developed at OpenAI and later used to align a model called InstructGPT. RLHF with PPO was subsequently used to align other ChatGPT models.

PPO belongs to the family of policy-based RL algorithms known as Policy gradient. Here is how Policy gradient algorithms work:

• An LLM in training generates a response for a given prompt.

• A reward model assigns a reward to the response.

• The reward is compared to a baseline reward the LLM is expected to get for the prompt. The difference is called the Advantage, and it tells how much better or worse the response was than expected.

• The LLM’s parameters are updated to increase the likelihood of responses with a positive advantage, and to decrease the likelihood of responses with a negative advantage.

Standard policy gradient algorithms are good but not perfect and can lead to unstable alignment training.

Consider that an LLM is being trained to produce less verbose answers. Let’s say that while training, a very short response gets an unusually high reward. This will result in a large advantage and update the LLM's parameters significantly in one step.

This might make an LLM start giving one-word answers to every future prompt. And in the worst case, it might forget what it had learned in the past.

And this problem is what leads to PPO.

PPO carefully updates the LLM parameters so that the model’s behavior does not deviate too far from that of its previous version. (Hence, the term ‘Proximal’ in its name.)

An LLM’s behavior is determined by the probabilities it assigns to the tokens it generates. PPO limits how much these token probabilities can change in a single step (clipped updates), regardless of how large the advantage is. This prevents large jumps in LLM behavior in a single step and makes training more stable.

PPO uses a separate model, called the Value model, to estimate the advantage. The Value model’s job is to predict the expected reward, or “Value”, at every token in the response.

The final reward is then combined with the Value model's per-token estimates to compute an advantage for each token. This helps figure out which tokens in the response are responsible for the good or bad score.

The Value model is also trained alongside the LLM, so its predictions improve over time.

GRPO

GRPO (Group Relative Policy Optimization) was introduced by DeepSeek and used to train the DeepSeekMath model to improve its mathematical reasoning. GRPO and its variants have become the standard for post-training LLMs for reasoning today.

GRPO is itself a variant of PPO that removes the Value model for advantage calculation (but keeps PPO’s clipped updates), which makes it highly memory-efficient.

Here is how it works:

• For a given prompt, an LLM in training generates a group of responses.

• A reward model or a verifier (when training for math and coding tasks) assigns a reward to each response.

• The average and standard deviation of all rewards are calculated for the group of responses.

• Advantage is calculated for each response as follows, where r(i) is the i-th response. Each token in the response gets this same advantage, which is the main trade-off of dropping the Value model.

• The LLM’s parameters are updated (using a clipped update) to increase the probability of generating high-advantage responses and reduce the probability of generating low-advantage responses.

To simplify these algorithms, I have not shown:

• how Generalized Advantage Estimation (GAE) is used to calculate per-token advantages in PPO, and how the Value model is trained

• A reference model, which is a frozen copy of the LLM before post-training, that is used to compute a KL penalty that keeps the training LLM (Policy model) from deviating too far from it

• How KL calculations are applied differently in PPO and GRPO

• The full clipped objective for updating the LLM in training

A more detailed comparison diagram is shown below from the research paper describing the DeepSeekMath model. If you’re interested in learning about these algorithms in detail, please refer to the ‘Further Reading’ section.

Further Reading

• 21 Reinforcement Learning (RL) Concepts Explained In Plain English

• A Detailed Guide To Reinforcement Learning From Human Feedback (RLHF) From Scratch

• Learn To Train A Reasoning Model From Scratch Using GRPO

• PPO for LLMs: A Guide for Normal People by

• LLM Training: RLHF and Its Alternatives by

This edition of the newsletter is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🧑🏻‍💻 Build and train a Diffusion LLM from scratch

• 🍭 9 GPU Concepts Every AI Engineer Should Know

• 🌈 20 PyTorch Concepts, Explained Simply

• ⚙️ Build A Vector Database From Scratch To Understand RAG In Depth

and so many more!

Become a paid subscriber today

On-policy distillation (OPD) is a post-training method in which a student LLM generates its own responses, and a stronger teacher scores them token-by-token, with the student trained to minimize the reverse KL divergence between the two models’ next-token distributions.

On-Policy Distillation (OPD) has become a popular algorithm for post-training LLMs, and almost all recent open-weight LLMs (such as Qwen3, GLM-5.3, and Nemotron-Cascade 2) have used it to achieve amazing performance.

As an example, the post-training results for Qwen3-8B showed that OPD achieved significantly better performance than RL while requiring only about 1/10^th of the GPU hours.

Many engineers still confuse OPD with RL, knowledge distillation, and SFT, but you do not have to. Here is a lesson that will help you understand it well.

What is On-Policy Distillation?

LLM training takes place in three stages:

1. Pretraining: Teaches an LLM the basics of language (grammar, syntax, and semantic structure) and gives it foundational knowledge of the world

2. Mid-training: Teaches an LLM domain-specific knowledge, improves reasoning, and extends context length

3. Post-training: Further improves reasoning and instruction following, and aligns an LLM with human values to be helpful and not harmful

Post-training takes place in two ways:

1. Off-policy: Where an LLM in training (called Policy) learns from data generated from an external source, either:

   • Trajectories/ responses from a stronger teacher model (Knowledge distillation)

   • A dataset of task-specific labeled examples (using supervised fine-tuning)

2. On-policy: Where the policy LLM learns from its own trajectories. This can be done using either reinforcement learning (which requires a reward signal) or distillation (which requires a stronger teacher model), i.e., On-policy distillation (OPD).

OPD involves improving the capabilities of a weaker “student” model using a stronger “teacher” model.

It is called “On-policy” because the student model generates its own trajectories, which are then scored token-by-token by a stronger teacher model. This differs from conventional Knowledge distillation (KD), in which a student model learns from trajectories generated by a teacher model (an “Off-policy” approach).

KD is like a guitar teacher showing how to play like Jimi Hendrix, and a student trying to copy it. OPD, on the other hand, is like a student playing an easier song while the guitar teacher evaluates and corrects the student's playing.

With KD, a student tries to learn techniques they rarely encounter in their own playing, which can make them overconfident in their skills. With OPD, the student gradually learns from their own mistakes and improves.

Subscribe now

Understanding OPD with an example

The process starts with two models: a stronger teacher model and a weaker student model.

The goal of OPD is to improve the student model's capabilities using its own trajectories (also called responses/ rollouts), supervised by the teacher model.

The process takes place as follows:

• Pass a prompt from the prompts dataset to the student model and collect its response at each generated token position.

• Pass the prompt and the student-generated response to the student and the teacher, and collect their per-token next-token probability distributions along the same trajectory.

• Calculate the per-token reverse KL divergence between the student’s and teacher’s next-token probability distributions. We will discuss later why reverse KL is used instead of forward KL in this step.

• Use backpropagation to update only the student’s parameters, bringing it closer to the teacher's capabilities.

• Repeat these steps using the updated student model.

Let’s learn this better using an example.

Consider the following prompt from a prompt database:

“Why is space black?”

When this prompt is passed to the student model, it generates a response as follows:

“Space looks black because there is no atmosphere to scatter sunlight.”

Each token in this response is chosen from a next-token probability distribution at each step of generation.

(Also, please note that I am considering a word as a token for simplicity here. In reality, a token is typically a sub-word.)

Next, let’s look at this intermediate step in response generation with the following context:

“Space looks black because there is no”

The student and teacher models’ probabilities for the next tokens at this step are as follows (only the top three token probabilities are shown explicitly for simplicity).

The reverse KL divergence measures how these probability distributions differ and is calculated using the following formula:

where:

• P_S​ is the student model’s next-token probability distribution

• P_T is the teacher model’s next-token probability distribution

• ‘t’ is a token in the vocabulary

• P_S(t) is the probability the student assigns to token ‘t’

• P_T(t) is the probability the teacher assigns to token ‘t’

The reverse KL divergence at this token generation step is calculated as:

Similarly, the reverse KL divergence is computed between the two models’ probability distributions at each generation step and averaged to obtain the OPD loss over the student-generated response.

This loss (OPD loss) is used to update the student model’s parameters with backpropagation.

This process is repeated with the updated student model till we reach the loss minima and achieve the desired results.

Why is reverse KL divergence used as the OPD loss?

KL divergence measures how a probability distribution differs from the true probability distribution.

In our case, it measures how much the student’s probability distribution (the approximating distribution) differs from the teacher’s (the true or target probability distribution).

• If the KL divergence is 0, the two distributions are identical.

• The larger the value of the KL divergence, the more the two distributions differ.

When we use forward KL, i.e., D_KL​(Teacher || Student), as the post-training loss and iteratively minimize it, the student model tries to cover the teacher's entire output distribution. This is why forward KL is mass-covering or mean-seeking.

In simple words, when minimizing the forward KL, the teacher shows all the different kinds of answers it can generate, and the student is expected to cover/learn all of them.

With reverse KL, i.e., D_KL(Student || Teacher), as the post-training loss, the student model focuses on covering the high-probability regions of the teacher’s distribution. This is why reverse KL is mode-seeking.

In simple words, when minimizing the reverse KL, the student avoids producing answers that the teacher would consider unlikely.

Reverse KL suits OPD because the student generates outputs that the teacher evaluates. And since it is mode-seeking, it helps the student focus on the high-probability behavior of the teacher model rather than trying to cover every possible behavior of the teacher.

This newsletter edition is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🧑🏻‍💻 Build and train a Diffusion LLM from scratch

• 🌈 20 PyTorch Concepts, Explained Simply

• ⚙️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🍭 9 GPU Concepts Every AI Engineer Should Know

and so many more!

Become a paid subscriber today

The NVIDIA H100 is one of the most popular GPUs for large-scale LLM training and inference. Here is what its GPU spec sheet looks like.

Let’s break down and learn what each of these rows means, one at a time.

Subscribe now

1. Form factor

Two variants of the H100 GPU are compared in the spec sheet: H100 SXM and H100 NVL.

The main difference between them is the form factor, which refers to the GPU's size, shape, and mounting style. The form factor determines the power limit, cooling design, and the number of GPUs that can be connected together.

H100 SXM (Server PCI Express Module) follows the SXM form factor. This means that the GPU can be directly mounted on a specialized GPU baseboard with a custom SXM socket. This provides higher performance and power, better cooling, and the option to connect 4 or 8 GPUs together using NVLink/NVSwitch, NVIDIA’s proprietary high-bandwidth interconnect.

H100 NVL follows the PCIe form factor. This means the GPU must be connected to the motherboard using a PCIe slot, which offers greater compatibility and easier installation but provides a lower power limit and inter-GPU bandwidth.

H100 NVL is sold as a pair of GPUs connected via NVLink, and hence the name.

NVIDIA also produced the standard H100 PCIe version (not shown in the spec sheet above), which can also be connected to another H100 using NVLink. However, the H100 NVL has higher memory bandwidth than the PCIe version because it uses HBM3, whereas the PCIe version uses HBM2e, an earlier HBM standard.

2. GPU memory capacity

The GPU memory row (also generally called VRAM) refers to the capacity of the GPU’s High Bandwidth Memory (HBM).

When running an LLM, this is one of the first figures to consider.

A GPU’s HBM stores:

• LLM weights and activations (during both training and inference)

• KV cache (during inference)

• Gradients and optimizer states (during training)

• CUDA/runtime workspaces and temporary buffers/tensors (during both training and inference)

Let’s say that we are deploying the Qwen3-32B model in BF16 or FP16 precision. Since the model has 32.8B parameters and each parameter takes 2 bytes, the parameters alone require 61 GB (32.8B × 2 bytes ≈ 61 GB).

The H100 SXM has 80 GB of HBM, which is enough to hold this model’s parameters but can fill up quickly once you add the KV cache and runtime buffers, especially for longer context-length workloads.

The Qwen3-32B model has:

• 64 layers

• 8 KV heads

• 128 dimensions per head

At BF16, its KV cache is roughly:

KV cache size = 64 x 8 x 2 (one each for K & V) x 128 x 2 bytes = 262,144 bytes ≈ 0.25 MB per token

For a sequence of length 8000 tokens and a batch size of 1, this makes:

KV cache size = 8000 tokens x 0.25 MB ≈ 2 GB

These 61 GB of parameters, along with a 2 GB KV cache and additional overhead, might fit within the 80 GB of HBM. But if you require a larger batch size or have long-context requests, you’d need to either:

• Reduce the precision of the model parameters

• Use a GPU with more HBM (such as H100 NVL with 94 GB of HBM)

• Connect multiple GPUs

3. GPU memory bandwidth

Memory bandwidth refers to how quickly the GPU can move data between its HBM and its streaming multiprocessors (SMs).

H100 NVL has higher bandwidth than the SXM version. This means it can move LLM weights from HBM to the compute cores faster.

Read more

“This is the lesson that you need if you are new to GPUs and want to master them without getting overwhelmed.”

Frameworks like PyTorch are easy to use because they hide away all the intricacies that occur at the GPU level. But companies and research labs today are looking for engineers who can look under the hood of PyTorch, explain where their compute budget is being spent, debug GPUs, and make LLMs more performant.

Here are 9 concepts explained simply that will help you better understand GPUs.

• 1, 2, 3: discuss GPU computation hardware

• 4, 5, 6: discuss GPU memory

• 7, 8: discuss how programs are executed on GPUs

• 9: discusses how multiple GPUs are connected together

Let’s begin!

1. Streaming Multiprocessor

Streaming Multiprocessor (SM) is the core computational unit in NVIDIA GPUs.

It is similar to the cores of a CPU, with one major difference: it can execute parallel instructions and specific operations, such as matrix multiplication, much faster.

While a CPU core is optimized for high execution speed of a single thread, an SM on a GPU is optimized for high throughput across many threads (running a large number of threads in parallel).

A Streaming Multiprocessor (SM) is called so because:

• It works with streams of data (data inputs requiring similar computation that are continuously fed to it)

• It contains multiple processing units that can perform computations on a data stream in parallel

The main components of an SM are:

• CUDA cores (INT32, FP32, FP64): For basic arithmetic calculations

• Tensor Cores: For matrix multiplication calculations

• Warp schedulers and Dispatch units: For choosing which thread groups (warps) to run next

• Load/ Store (LD/ST) units: For moving data between memory and registers

• Registers/ Register files: For storing each thread’s data temporarily

• L1 cache: For temporarily storing data to be used across the components of an SM

• Shared memory: Partition of the same SRAM as the L1 cache that can be used to store data. While L1 caches data automatically, shared memory can be used programmatically.

The following is an architectural diagram of the SM for an NVIDIA H100 GPU. A single H100 SXM has 132 such SMs.

Although the above-mentioned components are most common in GPUs, you can see many more components in the H100 GPU, such as:

• SFUs (Special Function Units): For performing exponential, logarithmic, and trigonometric functions.

• Texture (Tex) units: For processing surface texture data efficiently

• Tensor Memory Accelerator: For efficiently moving multi-dimensional data between global memory and shared memory

• L0 and L1 instruction cache: Small and fast cache memory that stores frequently used instructions close to the compute cores

2. CUDA core

A CUDA (Compute Unified Device Architecture) core handles a single simple arithmetic operation, such as integer and floating-point calculations, per clock cycle, in an SM.

Different CUDA cores handle different types of numerical precision, such as INT32, FP32, and FP64, as seen in the architectural diagram above.

An H100 SXM GPU has:

• 128 FP32 CUDA cores per SM

• 16,896 FP32 CUDA cores per GPU

A CUDA core is much simpler than a CPU core. While a CPU core can perform complex operations, a CUDA core’s job is to work as a simple calculator for basic arithmetic. It is the massive number of these cores performing calculations in parallel that makes a GPU so powerful.

Note that the term ‘CUDA core’ refers to hardware and is not directly related to the CUDA software ecosystem.

3. Tensor core

A Tensor core is a specialized unit in an SM that can perform fast matrix multiply-and-accumulate operations.

These operations make up the bulk of the workload in deep learning and are handled by the Tensor Cores, leaving non-matrix operations (such as activation functions, normalization, and element-wise operations) to the CUDA cores.

NVIDIA first introduced the Tensor Core in the Volta GPU architecture in 2017, which massively accelerated neural-network training workflows.

Tensor Cores are also specialized for different numeric precisions, such as FP16, BF16, TF32, and FP64. The H100 GPU has specialized tensor cores for low-precision formats such as FP8, while the Blackwell generation of GPUs includes tensor cores for FP4 (specifically, NVFP4).

You will often find discussions of TFLOPS in GPU specs, which refer to the performance of the GPU's Tensor cores and CUDA cores.

4. High Bandwidth Memory

High Bandwidth Memory (HBM) is a GPU’s main memory. It is also called global memory or VRAM. This is the memory that holds the model weights, activations, and KV cache.

HBM is mounted alongside the GPU die and works with smaller memory components (L2, L1, and register files) that are etched directly on the die (on-chip memory).

It is a type of Dynamic random-access memory (DRAM), which offers larger capacity and is cheaper than on-chip memory, a type of Static random-access memory (SRAM), which is extremely fast but much smaller.

5. On-chip memory

Read more

BDH-CQ is a 150M-parameter reasoning model that breaks the previously reported cost-accuracy Pareto frontier in ARC-AGI-1 and sets a new state-of-the-art for benchmark cost efficiency.

The model learns each new task from the examples it’s shown at inference time and progressively updates a recurrent memory to reason in latent space before giving an answer.

It achieves 29.5% pass@2 on ARC-AGI-1 at an inference cost of $0.0007 per task.

Although this is not the highest-accuracy result, BDH-CQ is ~57x cheaper than GPT 5.6 Luna (Low), which scores 34.2% (only 4.7% higher) at $0.040. Even after OpenAI announced a recent 80% reduction in API price, BDH-CQ is still ~11x cheaper than GPT 5.6 Luna (Low).

👨🏻‍💻 Read more about this research using these links: Hugging Face | ArXiv | Blog

Sponsor this newsletter to reach 12,000 smart AI engineers and leaders.

While requests can be processed one at a time as they arrive (sequential/ serialized processing), batching them is one of the best ways to improve LLM throughput. A scheduler decides how batches are formed and updated.

Here are the batching and serving strategies commonly used in modern inference architectures.

1. Static batching

Static batching involves grouping incoming requests, and when a fixed preset number of requests (batch size) is reached, processing them together as a batch.

This is a simple batching strategy to implement, but it comes with high latency, as the preset batch size must be reached before processing requests begins. Also, the GPU has to wait for the longest request to finish before returning the batch outputs.

This strategy is used mainly for offline batch jobs or online inference systems with low, predictable traffic.

2. Dynamic batching

Dynamic batching involves grouping incoming requests and processing them in batches when either the maximum batch size or the maximum wait window is reached.

This means that:

• If a batch reaches the maximum size, processing begins immediately.

• If the maximum wait window is reached, the processing begins immediately even if the batch contains only one request.

This strategy can handle online inference with unpredictable traffic (variable inter-request intervals), but it still wastes compute, as the GPU must wait for the longest request to finish before returning outputs, even when shorter requests have already finished processing.

Subscribe now

3. Continuous batching

Continuous batching adds and removes incoming requests between decoding iterations. This means a new request can be added to the batch as soon as a request in the batch finishes processing. This substantially minimizes GPU idle time.

It is also called:

• In-flight batching because the requests are added and removed on the fly, and

• Iteration-level scheduling because the scheduler re-decides the batch membership before every decoding step rather than fixing it once at batch formation

This is one of the most popular strategies in modern high-throughput serving systems that handle massive traffic and can be easily implemented with most serving frameworks such as vLLM, TensorRT-LLM, and SGLang.

Continuous batching treats prefill and decode for requests similarly, even though prefill is compute-bound and decode is memory-bound. This leads us to the next two batching strategies.

4. Continuous batching with chunked prefill

For every request, LLM inference takes place in two phases:

1. Prefill: The first phase, where all tokens in a user’s prompt are processed together in a single forward pass by the LLM. This phase builds the KV cache and has high arithmetic intensity, making it compute-bound.

2. Decode: The second phase, where the LLM generates one token at a time, with a single forward pass through the model for each token. This phase has low arithmetic intensity and is memory-bound.

You can read more about these phases using the following links:

Let’s say that user A’s request is in the decode phase (tokens are being generated one at a time) and user B sends a prefill-heavy request with 50,000 tokens. This will cause user A to wait a long time between their generated tokens.

Instead of processing user B’s request all at once, the prefill could be chunked or broken down into smaller token-length pieces and processed with the decode of user A’s request. This ensures that user A continues to receive tokens while user B's long prompt is being processed.

This strategy of chunking prefill is used with continuous batching, which allows processed requests to be added to or removed from the batch between decoding iterations, improving GPU utilization. This is especially beneficial when handling traffic with long-context requests.

One disadvantage of using chunked prefill is that it reduces prefill throughput and increases Time to first token (TTFT) for long prompts. This leads us to the next approach.

5. Prefill-Decode disaggregation

Prefill-Decode disaggregation is a serving architecture that moves the two phases of LLM inference to their dedicated GPU pools.

The dedicated pool of GPUs for prefill is optimized for higher compute, while the dedicated pool of GPUs for decode is optimized for higher memory throughput.

For example, the NVIDIA H200 GPU, compared to the H100, has much faster and larger memory but the same core compute. Therefore, H200s are best used in the decode pool, and H100s in the prefill pool.

During inference, the KV cache is transferred from the prefill-dedicated GPU pool to the decode-dedicated GPU pool. This adds latency and consumes interconnect bandwidth, which is a disadvantage of this architecture at smaller scales. Using high-speed interconnects such as NVLink or InfiniBand helps optimize KV cache transfers.

It must be noted that Prefill-Decode disaggregation is a serving architecture in which each GPU pool can still use continuous batching.

Thanks to Pathway, this edition of the newsletter is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️

Share

Let’s understand visually how the watermarking process works.

But first, how do LLMs generate text?

LLMs are trained to generate text one token at a time, based on the preceding tokens/ context.

Check out the following example where an LLM generates the token “dark” given the context “The solar eclipse turned everything”.

Behind the scenes, the tokens in the context are converted into token IDs and then into token embeddings, which the LLM processes to produce logits.

Next, we take the logits for the last position and apply softmax to them to obtain probabilities.

Let’s look closely at the step where the Softmax function is applied. Softmax turns logits into non-negative probabilities that sum to 1. Here are the top-5 next-token probabilities for the given context “The solar eclipse turned everything”.

From here, different sampling algorithms can be applied to select the next token.

In our example, the next sampled token is “dark”. It is appended to the context “The solar eclipse turned everything”, and the process is repeated to generate further tokens until either a maximum response length is reached or an end-of-sequence (EOS) token is generated.

How is the generated text watermarked?

The watermarking process does not require any further LLM training. It only changes how tokens are sampled from the probability distribution of next tokens.

It consists of the following steps:

1. Generate a secret watermarking key (Anthropic holds it)

2. Given the last ‘H’ number of tokens in the context and the watermarking key, generate a random seed

3. This seed is input into ‘m’ watermarking functions, whose job is to randomly assign a score of 0 or 1 to every token. In our example, we consider m = 3.

4. Sample 2^m tokens (which will likely be non-unique) from the LLM. In our example, this is 2³ = 8 tokens. Note how “dark” and “black” are picked more times than “eerie” and “gray”, as their next-token probabilities are higher.

5. Pair these tokens, and in each pair, select the highest-scoring one based on the watermarking function chosen.

6. Repeat this process multiple times for multiple rounds/layers using successive watermarking functions, and break ties randomly if they occur.

7. The winner of the last tournament round/ layer becomes the next generated token. This sampling algorithm is called Tournament sampling.

7. The generated token is appended to the context, and this process is repeated to generate further tokens until either a maximum response length is reached or an end-of-sequence (EOS) token is generated.

Note how “dark” was the most probable token, and it was even sampled three times, yet it lost in the tournament of tokens, and “black” was chosen as the next token instead. This pattern that tournament sampling leaves across many steps of generation is what makes the text detectable later.

How is the watermarked text detected?

Watermark detection works as follows:

• Take the watermarking key used during generation (this key is secret to Anthropic) and the tokenized text to be analyzed

• For each token, take its previous ‘H’ tokens and the key to rebuild the random seed used during generation

• Use this seed to compute the values of the ‘m’ watermarking functions for the token at that position

• Average these values across all tokens and all rounds/ layers of the tournament to get a score. Here:

  • ‘T’ is the number of tokens in the text being analyzed

  • ‘t’ is the position of the token

  • ‘m’ is the number of tournament rounds/ layers

  • ‘l’ is the index of the layer

  • x_t is the token at position ‘t’ in the text being analyzed

  • r_t is the random seed at position ‘t’, rebuilt from the previous ‘H’ tokens and the watermarking key

• Compare this score to a threshold. If it exceeds it, the text can be called watermarked.

According to Anthropic, a watermarked text detection API will be available soon.

The above-described algorithm of text watermark generation and detection is called SynthID-Text and was developed by Google DeepMind. Anthropic says that it uses a version of it to watermark Claude’s text but does not explicitly disclose the finer details.

Even if these details are not mentioned, it is clear that the watermark only changes the source of the randomness that is used to pick tokens from the next-token probability distribution.

• When watermarking is not used, it is multinomial sampling, with randomness generated by a pseudorandom number generator.

• In the watermarking process, it is Tournament sampling, where the next token is decided by pseudorandom values coming from the watermarking key and the context.

Clearing the myths

There are many misconceptions around the watermarking process that need to be cleared up. Here is what is true:

1. Watermarking has no impact on the output quality, creativity, or readability of the generated text.

2. It does not push Claude to produce words it would not have picked otherwise. For example, in the context “The solar eclipse turned everything”, watermarking would not cause Claude to generate a very rare next word such as “spectral”.

3. There are no hidden characters in the generated text.

4. It does not cost extra tokens when generating watermarked text.

5. It has a negligible impact on the token-generation speed of models.

6. It is not applied in cases where there’s a single answer (e.g., “1 + 1 =”) or low-entropy outputs, such as facts (e.g., “Which is the latest book written by ?”) and code (except code comments).

7. Neither the watermark nor the key contains anything that identifies a user, an organization, or a chat.

8. The watermark can only determine if Claude was likely involved with the content at some point. It cannot distinguish “Claude wrote this” from “Claude heavily edited this.”

9. It does not confirm whether a text was written by a human, nor is it a method for AI-generated text detection from other LLMs.

10. Confidence in detections is higher for longer samples.

11. A complete rewrite of the generated text removes the watermark, but light editing probably will not.

12. For files generated by Claude, content credentials are added to the file’s metadata, indicating whether the file was generated or processed by Claude. This labeling uses an open industry standard called C2PA, which differs from a watermark in that it resides in the metadata rather than in the content itself.

TL;DR

• Anthropic has introduced watermarking for all text generated by Claude to comply with the EU AI Act.

• The watermarking method is based on Google DeepMind's SynthID-Text.

• The method does not add any hidden characters to the text, nor does it require any LLM training. It only replaces the source of randomness used to select each token, using a sampling algorithm called Tournament sampling.

• Watermarking does not affect the quality or creativity of text generation or the text's readability.

• A positive detection result only means that Claude was likely involved with the content at some point (generation or editing).

• It does not detect whether text is human-written, nor is it a method for detecting AI-generated text from other AI models.

• A light edit may not, but a full text rewrite can remove the watermark.

As we finish this lesson, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

This newsletter edition is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all articles in this newsletter and level up as an AI engineer.

Become a paid subscriber today

✨ This week’s newsletter features new research from Pathway. ✨

Researchers at Pathway, Bielik AI, and NYU have just published results for a new model called BDH-CQ, which breaks the previously reported cost-accuracy Pareto frontier in ARC-AGI-1 and sets a new state-of-the-art for benchmark cost efficiency.

• BDH-CQ is a 150M-parameter reasoning model.

• It learns each new task from the examples it’s shown at inference time (in-context learning), which progressively updates a recurrent memory rather than filling a growing context window. It then solves a given query by iteratively reasoning in a structured, continuous latent state rather than using a verbalized chain of thought.

• It achieves 29.5% pass@2 on ARC-AGI-1 at an inference cost of $0.0007 per task. This is less than 1/10^th of a cent per task!

• Although this is not the highest-accuracy result, its significance is clear when viewed alongside other LLMs’ results.

  • BDH-CQ is ~57x cheaper than GPT 5.6 Luna (Low), which scores 34.2% (only 4.7% higher) at $0.040.

  • Even after OpenAI announced a recent 80% API price reduction, BDH-CQ is still ~11x cheaper than GPT 5.6 Luna (Low).

  • GLM 5 scores 44.7% (15.2% higher), but costs 243x more per task than BDH-CQ.

• BDH-CQ is based on a post-Transformer sequence-model architecture called ‘Dragon Hatchling’ (BDH).

• Early pretraining experiments show that it has Transformer-like scaling behavior across scales from 1B to 600B parameters while preserving its latent reasoning capabilities.

Read more about this research using these links: Hugging Face | ArXiv | Blog

1. Qwen3.8-Max

• Alibaba released its most capable model, Qwen3.8-Max, which is heavily focused on coding, autonomous agents, and long-horizon tasks.

• It outperforms previous Qwen models in agentic coding, computer use, research tasks, and co-working workflows.

• The model has 2.4 trillion total parameters (95B active per token) and uses a Mixture-of-Experts architecture.

• With this model, the team is, for the first time, open-sourcing the weights of a Qwen-Max-class model.

Read more about this release using this link.

Subscribe now

2. On-Policy Self-Distillation without Any Supervision

• U-OPSD (Unsupervised On-Policy Self-Distillation) is an algorithm that helps an LLM improve its reasoning without ground-truth answers, rewards, or a stronger teacher model. With U-OPSD, an LLM learns and improves using only its own generations, guided by internal consistency.

• The process starts with the model generating multiple solutions for each problem and using a majority vote to form its own pseudo-answer, provided it meets a self-consistency threshold.

• It then conditions the model’s distribution on the pseudo-solution and distills itself on the disagreeing completions, letting the model correct itself precisely where it is confidently wrong.

• U-OPSD consistently outperforms the base models and matches or surpasses supervised methods with ground truth (GT), such as OPSD and GRPO.

Read more about this research using this link.

3. Leanstral

• Leanstral is Mistral AI’s series of open-source generalist code-agent models for Lean 4.

• The model has a Mixture-of-Experts architecture with 119B total and 6B active parameters.

• It runs within the open-source Mistral Vibe coding-agent harness rather than a specialized theorem-proving scaffold, and uses no test-time scaling method beyond context compaction.

• Leanstral 1.5's performance is comparable to far larger, proprietary systems, as it saturates miniF2F, solves 587/672 problems on PutnamBench, and reaches a new state-of-the-art of 34% on FATE-X and 43.2% pass@8 on FLTEval.

• Beyond competition mathematics, Leanstral can formally verify code and resolve bugs and issues in real-world repositories across graduate-level mathematics, mathematical finance, and code verification.

Read more about this research using this link.

4. The Beginning of ChatGPT Ads

• This is the first empirical study of advertising content shown within ChatGPT during the early 2026 rollout.

• Researchers created 91 simulated U.S. accounts and ran hundreds of prompts to collect ads from 191 unique advertisers across 127,801 conversations.

• The results show that lower-income simulated accounts were significantly more likely to receive ads irrespective of their race.

• Product/recommendation-style prompts were more likely to trigger advertising.

• The ads were heavily skewed towards consumer goods and directed users to a specific advertiser rather than a particular product.

• Ads were clearly marked “Sponsored” and separated from the LLM’s response text rather than embedded within it.

• There were near-zero ad rates for medical conditions, mental health, and political prompts.

Read more about this research using this link.

5. Ten Advances in Mathematics and Theoretical Computer Science

• An internal OpenAI model, Astra, produced 10 research advances in mathematics and theoretical computer science.

• These results go beyond reproducing known proofs to include new resolutions, counterexamples, and improved bounds for many open problems in these subjects.

Read more about this research using this link.

6. Maple-Preview

• Maple-Preview is DeepGrove’s open-source ternary-weight reasoning model, designed to run efficiently on consumer hardware.

• The model uses a Mixture-of-Experts architecture with 256 experts (8 active) and has 20.2B total parameters (1.49B active).

• The model has a 131k-token context window and is competitive with larger models at strong mathematical/general reasoning, including IMO-level problems.

• It runs at 218 tokens/sec on an M4 Mac mini, which is 5-16× faster than efficient models like Gemma 4, Qwen3.5, and gpt-oss.

Read more about this release using this link.

Subscribe now

7. Scaling Automated Post-Training with Locus

• Locus is Intology’s automated research agent, extended to autonomously run the experiments needed to post-train other AI models.

• PostTrainBench is a benchmark where an agent is given a base model, a target benchmark, a single-H100 compute node, and internet access, and its task is to produce a post-trained version of the model optimized for the target benchmark.

• Locus with Opus 5 has state-of-the-art post-training capabilities on PostTrainBench, where it outperforms every frontier-agent baseline.

• On a larger compute variant of PostTrainBench, called PostTrainBench+, when given thousands of H100-hours, Locus continues improving while general-purpose coding agents plateau, and Locus-trained Qwen3-1.7B models collectively surpass Qwen’s official human-tuned checkpoint.

• Beyond post-training, across live prize-money Kaggle competitions, Locus achieved an average rank beating 89.5% of human competitors.

• Working with Bubble, a no-code app development platform, Locus discovered and executed a post-training recipe end-to-end, and the resulting model now serves millions of users at ~2.8× lower error, ~5.4× lower latency, and ~100× lower cost than the frontier API it replaced.

Read more about this release using this link.

8. LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

• Instead of sending every user query to a single frontier model, model routing dynamically routes each query to the model best suited to it.

• LLMRouter is an open-source framework and benchmark for standardized and modular implementation of LLM routers.

• It defines routing as a sequential decision process using five components:

  • Context encoder

  • Model encoder

  • Scoring function

  • Decision rule

  • Learning signals

• Existing methods can then be organized into three families: single-turn, multi-turn, and personalized routing.

• The authors also introduce xRouteBench, a benchmark for evaluating routers across general LLM tasks, conversational memory, vision/video, time-series, and personalization using a pool of 18 models with different costs and capabilities.

• Using these, an empirical study of LLM routing shows that learned routers achieve a 14.6% relative improvement over the strongest fixed-model baseline.

• The results also show that multi-turn routing is not consistently better than simple single-turn routing because decomposition and aggregation can introduce extra cost and redundant information.

Read more about this research using this link.

Subscribe now

9. OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents

• OneDayAgent is a harness for autonomous agents that helps them complete long-horizon, multi-step everyday tasks without losing sight of their goals, state, or context.

• It turns an open-ended request into a managed execution process that decomposes tasks into bounded subtasks, maintains execution memory under context pressure, and verifies and repairs the final output.

• OneDayAgent with GLM-5.2 sets a new state-of-the-art with an overall score of 0.821 on AgentIF-OneDay, which contains 104 realistic work, study, and life tasks.

• The same harness runs across five backend LLMs from three model families, showing that it generalizes well without any tuning.

Read more about this research using this link.

10. ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

• ResidencyRL is an RL method from Google DeepMind for training clinical AI agents through simulated multi-turn clinical encounters rather than only on static medical Q&A.

• An agent, initialized from Gemini 3.5 Flash, is trained on ~57,000 simulated clinical cases, with each encounter running for up to 60 dialogue turns and including 8 tool calls. The training uses GRPO with rewards covering diagnostic accuracy, management, history-taking, communication, documentation, and safety.

• On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7% under adversarial conditions and reduces missed red flag rates by 31%.

• Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of cases.

• The gains also transferred to unseen benchmarks (AMIE, AgentClinic, and CRAFT-MD) and specialties(Oncology).

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️

Share

The process of Knowledge distillation (KD) was described in a 2015 paper by Hinton et al. It involves training a weaker/smaller model called the “student model” to imitate the outputs of a stronger/ larger model called the “teacher model”.

The student learns from teacher-generated output trajectories, rather than from trajectories generated by itself. This is why conventional KD is an Off-policy post-training method (as opposed to On-policy distillation).

The process of KD goes like this:

1. Take a larger/stronger teacher model and a smaller/weaker student.

2. Create a dataset of prompts relevant to the training task.

3. Pass a prompt to the teacher model and collect the teacher's next-token probability distribution over the vocabulary. These probabilities are called “soft targets” (as opposed to “hard targets” or training labels).

4. Pass the prompt and the teacher’s response to the student model and collect the student’s next-token probability distribution conditioned on the same prefix as the teacher.

5. Use a loss function to measure how much the teacher’s and student’s probability distributions differ. A commonly used loss is the KL Divergence.
   Optionally, a hyperparameter called Temperature is applied to the logits of both the teacher and the student models when calculating the distillation loss. This helps flatten the teacher’s probability distribution of predictions, which better reveals the relationships between different tokens.

6. In many cases, alongside KL Divergence, a Cross-entropy loss between the student’s final prediction and the ground-truth label, or “hard target”, is also calculated and used to formulate the combined distillation loss. This combined loss helps the student model to learn from both the teacher model and from ground-truth data.

7. The teacher model is kept frozen, but the student model’s parameters are updated using backpropagation.

8. This process is repeated over all prompts in the dataset as the loss decreases and the student model learns to approximate the teacher model's outputs.

Distillation with real numbers

The process of knowledge distillation is shown in detail in the illustration below.

Read more

1. DeepSeek-V4-Flash-0731

DeepSeek-V4-Flash-0731 is the official release of DeepSeek-V4-Flash that replaces its preview version.

The model has a 304B-parameter architecture, uses DSpark speculative decoding for faster inference, and supports low, high, and max reasoning-effort levels.

It outperforms the preview version on multiple benchmarks while activating fewer parameters, and has competitive performance with the strongest proprietary models available today.

It scores 50 points on the Artificial Analysis Intelligence Index, similar to GPT-5.6 Luna (51 points) and Google Gemini 3.6 Flash (50 points), while costing ~60% less per task than Luna and 10x less than Gemini, with a 98% discount on cache hits.

Read more about this release using this link.

2. Pangram 4

Pangram 4 is the latest deep-learning-based AI-text classification model from Pangram Labs with state-of-the-art performance in AI text detection and is being used by to detect ever-increasing AI slop.

Pangram 4 is a LoRA fine-tuned MoE model trained on human text paired with AI-rewritten and AI-edited versions of itself, labeled clause-by-clause by matching against the human source. At inference, it scores overlapping 512-token windows, averages the logits, and uses a CRF (Conditional Random Field) to decode the authorship spans.

On AI-text classification, it achieves an AUROC of 0.9916, a false positive rate of 0.0041% (roughly 1 in 24,000), and a false negative rate of 0.3396%.

Read more about this research using this link.

Join the paid tier today to get access to all articles in this newsletter and level up as an AI engineer.

Become a paid subscriber today

3. Frontis-MA1

This research introduces OpenMLE, an open full-stack system for Recursive self-improvement (RSI) research covering:

• Verifiable task environments with execution feedback (OpenMLE-Gym)

• Operator learning (OpenMLE-ERL)

• Long-horizon search (OpenMLE-Evo)

Using it, the authors train Frontis-MA1-35B, a meta-evolution agent for ML engineering, that learns four operations (Draft, Improve, Debug, and Crossover), then applies them over long experiment chains.

On MLE-Bench Lite under a 12-hour per-task budget on one RTX 4090 with 12 GB VRAM, Frontis-MA1-35B improves medal average from 39.39% to 60.61% over its base model with OpenMLE-Evo, and reaches 71.21% with OpenMLE-Evo-Max, exceeding GPT-5.5 + Codex and approaching GPT-5.6 Sol and Kimi K3.

Read more about this research using this link.

4. Metis: Memory Foundation Model

This research introduces Metis, a memory foundation model, an LLM with persistent memory built into its architecture rather than implemented through external modules.

Metis equips a foundation model with a native memory state, allowing past information to be compressed into an internal memory state and accessed through memory attention. At inference time, all learned model weights remain frozen, while the native memory states are autonomously updated with forward computation.

Metis is trained on large-scale memory-specific datasets, and multiple mid-training optimization objectives are used to acquire these native memory procedures.

Experiments show that Metis-27B substantially outperforms other no-context memory methods on LoCoMo and NextMem benchmarks, but still struggles with very long histories.

Read more about this research using this link.

5. Memory for Large Language Models

This survey systematically organizes the fragmented field of memory in LLMs. It classifies memory across three axes:

1. Representation (implicit versus explicit)

2. Update dynamics (offline versus online)

3. Persistence (short-term versus long-term)

It distinguishes model-level memory from external agent memory or prompt-based RAG pipelines, highlights key trade-offs, and calls for improved memory-management methods and unified benchmarks.

Read more about this research using this link.

Subscribe now

6. PhiZero: A World Model Built Around Physical Language

This research presents PhiZero, a physical world model built around “physical language” or compact discrete representation of world-state transitions.

The model is trained to learn the physical language from in-the-wild videos with self-supervision and uses it to explicitly reason about how the physical world evolves.

Instead of taking the usual approach of predicting future videos directly in pixel space, PhiZero uses a reason-then-render approach. This means that it first infers future world evolution as a physical-language sequence and then renders the inferred transitions into videos.

Experiments show that it has leading results on several physical-video benchmarks and shows potential for realistic and interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.

Read more about this research using this link.

7. Weak-to-Strong On-Policy Distillation

On-policy distillation (OPD) has become a popular post-training method for LLMs that aligns a student with the teacher’s token-level distribution on the student’s own rollouts.

This method either distills a larger model into a smaller one or trains multiple domain experts from a shared base and distills them into one student (MOPD). These approaches fail at the frontier when no larger teacher exists or turn out to be costly for multiple teachers.

This research works on these issues and introduces Weak-to-Strong On-Policy Distillation (W2S-OPD), an algorithm that improves the strong student by distilling from multiple weak models.

W2S-OPD creates a proxy teacher by comparing a positive and a negative model, both smaller than the student. Their logit difference captures a specific capability direction, which is added to the student’s base-model logits. Because the resulting proxy remains close to the student’s distribution, the student can learn from it by minimizing per-token reverse KL divergence on its own generated outputs.

W2S-OPD consistently outperforms OPD on multiple math and coding benchmarks and even makes the student surpass the domain teacher and continue to improve when every supervision source is weaker.

Read more about this research using this link.

8. Gemini Robotics ER 2

Google released Gemini Robotics ER 2, its most capable embodied reasoning model for robotics. (ER stands for Embodied Reasoning.)

The model can handle conversation, understand the physical world, plan multi-step tasks, reason and call tools, and can hand off motor execution to any given lower-level VLA model. It can also track task progress, detect failures, adjust actions in real time, and allow multi-robot collaboration that lets multiple robots work together in shared spaces and complete complex workflows too tough for a single robot.

The model has 91.3% accuracy in critical moment-finding tasks in videos, comes with sub-second latency, consistently achieves the highest accuracy across all core ER capabilities, and is Google’s safest model that stops a robot when a person is nearby and resumes only when the area is clear.

Read more about this release using this link.

9. PatientAgentBench

This research introduces PatientAgentBench, a patient-facing agentic healthcare benchmark. It evaluates an agentic LLM using realistic, multi-turn conversations involving simulated patients, medical records, and healthcare tools.

Each conversation is scored by an LLM-as-a-Jury across 6 dimensions using 100+ conversation-agnostic, clinician-grounded criteria, with its automated judging showing 79-93% agreement with licensed clinicians.

Experiments over 10 models across four families on 1200 scenarios show major gaps in triage quality and safety.

• Pass rates for triage quality range from 32% for the weakest models to 88% for the strongest, with agents frequently acting on administrative requests without clinical screening.

• For clinical safety and workflow accuracy, the weakest models frequently fail by fabricating unexecuted actions, while frontier models fail on only 1-3% of cases, from unverified tool outputs and omitted crisis resources in an emergency.

• More capable models can narrow these gaps but do not close them, and the strongest model, Claude Opus 4.8, scores only 4.25 of 5 overall.

The benchmark also confirms that static benchmarks are insufficient to evaluate healthcare agentic systems, and sustained, tool-using conversations against realistic patient records are the only way to surface their failures.

Read more about this research using this link.

Subscribe now

10. HiSkill

This research introduces HiSkill, which organizes past interaction experiences into a hierarchical skill graph, rather than a flat list of textual skills.

The hierarchical skill graph is a directed graph with skill nodes, AtomicOp (Atomic Operation) nodes, and typed edges that connect reusable high-level skills with executable action templates, while also capturing decomposition, temporal transition, compatibility, support, and recovery relations among them.

At inference time, for each task, HiSkill retrieves a small task-relevant subgraph, tracks its state, selects or switches skills, and iteratively converts AtomicOps into executable actions.

Experiments on three interactive environments show that HiSkill outperforms state-of-the-art baselines by achieving average relative improvements of 17.33% in success rate, while reducing inference token consumption by 78.75% over the strongest baseline.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

LLM inference optimization is one of the hottest topics today, and everyone wants text generation that's faster, more accurate, and cheaper. Here are 10 techniques that can help you get there.

1. KV Caching

During a step of autoregressive text generation, an LLM compares the last token’s Query vector to the Key vectors of all previous tokens, including itself. This creates attention weights, which are used to compute a weighted sum over the corresponding Value vectors. This results in the attention output for that token.

For example, considering a word as a token, when generating the token after "The capital of the U.K. is", the LLM:

• Converts the current token ("is") into a Query vector

• Scores it against the Key of each preceding token to find that "capital" and "U.K." are most related to it

• Combines those tokens' Value vectors to decide that the next token is most likely to be "London"

This process is repeated at each step of text generation.

Calculating Key and Value vectors for all previous tokens at each step is expensive, especially as the sequence length grows. Using a KV cache helps solve this issue of repeated computations.

The KV cache stores each token’s Key and Value vectors once, rather than recomputing them for the entire sequence at each step. This means the LLM now has to calculate (and append to the KV cache) only the new token’s Key and Value, reusing the rest from the cache at each step.

This massively reduces the amount of computation during inference but comes at the cost of using up HBM memory. (There’s no free lunch!)

Using KV caching is one of the simplest techniques that you can take to improve inference.

Before we move forward, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

2. Efficient Attention mechanisms

As we discussed earlier, the KV cache takes space in a GPU’s HBM, and managing its size helps make inference fast.

The traditional Multi-head self-attention (MHA) uses multiple heads, each with its own Key and Value vectors.

There are a few attention variants that reduce the number of Key and Value vectors used (and therefore the KV cache size) as follows:

• In Multi-Query Attention (MQA), all attention heads share the same Key (K) and Value (V) vectors, while each head still has its own Query (Q) vector.

• In Grouped Query Attention (GQA), Query (Q) heads are grouped, and each group shares one Key (K) and Value (V).

There is also Multi-head Latent Attention (MLA), which compresses the Key and Value vectors into a low-dimensional latent vector and caches it. This uses much less memory than the full-sized KV cache. The full Key and Value vectors are reconstructed from this latent vector when needed.

3. Continuous Batching

A naive way to process LLM requests is to handle them sequentially. This is very inefficient, as different requests can arrive at different times and can have variable input and output lengths (and therefore processing times).

Requests are therefore batched and processed together. This improves the system's overall throughput and reduces underutilized GPU compute.

Batching can be done at the request and token levels. When done at the request level, the system has to wait for a certain predetermined number of requests to arrive and be batched before they can be processed. This is called Static Batching.

A system can also be configured to wait only until a pre-decided maximum time rather than indefinitely to fill the full batch before the batch can start processing. This is called Dynamic Batching.

In either case, the entire batch must finish processing before any new request can join.

In contrast, when done at the token level, as in Continuous (in-flight) batching, requests are processed token by token. Instead of waiting for the entire batch to finish, the system evicts a completed request and immediately fills the vacated slot with a new request. This helps utilize GPU resources more effectively than the previous two approaches.

Subscribe now

4. Optimized Kernels

Inference Decode is memory-bound, which means that the GPU spends more time moving data from memory (HBM) to the compute cores than performing computations.

A kernel is a function that runs on a GPU and performs multiple parallel operations. A kernel can be optimized to keep data in on-chip memory (SRAM/registers), avoid re-reading it from the slow HBM, and feed it to the compute cores to better use them.

This can be done using Kernel fusion, a technique in which multiple individual operations in an algorithm are merged (“fused”) into a single kernel to avoid data movement across memory.

FlashAttention is a well-known example of this, in which operations from the attention computation are merged into a single kernel.

This is how standard attention is computed:

• Query and Key matrices are read from HBM and multiplied to produce an attention score matrix in the compute cores

• This matrix is written back to the HBM

• The matrix is re-read to apply Softmax

• The result is written back to the HBM

• The result is read back to multiply it by the Values matrix to create the final output of attention computation, finally writing it back to the HBM

FlashAttention avoids all this data movement by never creating the full attention matrix in HBM. It splits the Query, Key, and Value matrices into smaller blocks called Tiles that fit in on-chip SRAM and performs attention calculations one tile at a time.

The attention scores and softmax are computed on-chip, and only the final output is written back to the HBM. This reduces data movement between HBM and the compute cores, resulting in faster attention calculations using far less memory.

5. Quantization

Quantization is the process of storing and computing LLM parameters, activations, and KV cache in lower-precision formats such as INT8 (8 bits) and INT4 (4 bits) instead of full-precision 32-bit floating-point (FP32) or 16-bit floating-point (FP16 or BF16) formats.

This reduces the model's memory requirements and increases inference throughput by reducing data transferred from HBM to the compute cores, at the cost of a slight drop in accuracy.

Take the example of Llama 3 70B. In full precision (FP32), each parameter occupies 4 bytes of memory. The total memory requirement for this model is 280 GB (70B parameters X 4 bytes/parameter). This means it will not fit on a single H100 GPU with only 80 GB of memory.

Reducing the precision to INT4, each parameter occupies 0.5 bytes of memory, and the total memory requirement for this model reduces to 35 GB (70B parameters × 0.5 bytes per parameter), making it possible to fit it on a single GPU.

A commonly used convention for quantized models is W(x)A(y) which tells how much a model’s weights (W) and activations (A) have been quantized. For example:

• W4A16 with 4-bit weights and 16-bit activations

• W8A8 with 8-bit weights and 8-bit activations

• W4A4 with 4-bit weights and 4-bit activations

You will also find quantized models suffixed with “-GPTQ” and “-AWQ”. These are the commonly used techniques for model quantization:

• GPTQ (Generative Pre-trained Transformer Quantization)

• AWQ (Activation-Aware Weight Quantization)

6. Prefix / Prompt caching & RadixAttention

LLM inference has two phases:

• Prefill, where all tokens in the input prompt are processed together, making it compute-bound. Prefill determines the Time-to-first-token (TTFT), and it is when the KV cache is built.

• Decode, where subsequent tokens are generated one at a time, making it memory-bound since very little computation is done at each step. Decode determines the Time per Output Token (TPOT).

Prefix (or prompt) caching is an optimization technique that reuses the computed KV cache for a shared portion/ prefix of the input prompt across multiple user requests. This reduces redundant computation during Prefill and leads to a shorter Time-to-first-token (TTFT).

For example, consider a long system prompt “System: You are a helpful assistant…”.

• Without prefix caching, each user request computes Prefill for all tokens in the system prompt and the user's question.

• With prefix caching, the KV cache for the system prompt is computed once and stored. Every user request then reuses it and computes Prefill only for the user's actual question.

RadixAttention is a technique introduced with the SGLang serving framework that efficiently implements Prefix caching using a data structure called a radix tree.

In a radix tree, each path from the root is a token sequence, and shared prefixes (prompt portions common to user requests) are shared branches.

For a new user request, the algorithm walks the radix tree, matching tokens as far as possible and reusing the KV cache along the matched path. Prefill is computed only for the unique tokens in the user request.

7. Pruning

Pruning is the technique of removing parts of a model that do not contribute much to its accuracy, making it smaller and therefore cheaper to perform inference on.

Pruning can be done in three ways:

1. Unstructured pruning: Involves setting low-magnitude individual parameters to zero

2. Structured pruning: Involves removing complete sections (neurons, attention heads, fully-connected layers) of the model

3. Semi-structured (N: M) pruning/ sparsity: Involves setting N of every M consecutive weights to zero. Newer NVIDIA GPUs (Ampere onward) have hardware support for 2:4 sparsity, which leads to ~2x increase in matrix multiplication throughput.

Subscribe now

8. PagedAttention

PagedAttention is a technique introduced in the vLLM serving framework to efficiently manage the KV cache's memory requirements. The technique is inspired by how operating systems manage memory using Paging.

Instead of storing the KV cache in a single large contiguous chunk of GPU memory upfront, it is dynamically stored in multiple fixed-size, small, non-contiguous blocks called pages.

A per-sequence block table maps logical token positions to physical memory blocks, and the attention kernel uses it to locate the correct KV pairs when needed.

9. Speculative Decoding

Speculative Decoding is a technique used to accelerate inference with large LLMs.

To do so, it uses a smaller LLM from the same family as the larger one, with both sharing the tokenizer and vocabulary. The smaller LLM in this case is called the ‘Draft model’, and the larger LLM is called the ‘Target model‘. This smaller LLM naturally runs inference faster because it has fewer parameters than the larger one.

This is how it works:

• For a given prompt, the smaller/ Draft model first generates K tokens (a draft). This is the Speculation/ Drafting step.

• These K tokens, along with the original prompt, are given to the larger/ Target model.
  In a single forward pass, the larger model outputs its probability distribution for the next token at every position in the draft, plus one for an additional position beyond it. So the total number of target predictions is K + 1 = 6. Because Decode is memory-bound, this single forward pass takes almost as long as generating just one token from the model.

• Next comes the Verification step. Each drafted token is checked against the larger/ target model’s prediction at that position. If they agree (when using greedy decoding) or the target finds it likely enough (when using probabilistic decoding via rejection sampling), the token is accepted.

  
  At the first token where this check fails, that token and all subsequent drafted tokens are discarded, and the larger model provides a replacement token for that position. All subsequent tokens from the target LLM are also discarded because they were conditioned on the rejected drafted token.
  
  Drafting then resumes from this position.

The benefits from this are quite evident.

• Consider a case where all K draft tokens are accepted, we get K+1 tokens in the final output. This is the best-case scenario.

• More commonly, if the first i draft tokens are accepted, we get i+1 tokens, i.e. i draft tokens + 1 resampled replacement token from the target model.

• In the worst case, if the first draft token is rejected, we get just 1 token, which is the resampled replacement token from the target model. This is the same as normal decoding from the target model.

If you want to read more about Speculative Decoding, please check out the following lesson.

10. Prefill-Decode Disaggregation

We have previously discussed that:

• Prefill is compute-bound and determines the Time to first token (TTFT)

• Decode is memory-bound and determines the Time Per Output Token (TPOT), also called Inter-token latency (ITL)

These are two fundamentally different problems, and optimizing each on a single GPU can be tough, especially when the prompts are long relative to the outputs (prefill-heavy requests), and one wants both low TTFT and TPOT at the same time.

Prefill-decode disaggregation is a serving architecture that helps in this case by moving the two stages of inference to separate dedicated pools of GPUs.

In this case, the dedicated pool of GPUs for Prefill would be optimized for higher compute, while the dedicated pool of GPUs for Decode would be optimized for higher memory bandwidth and capacity.

For example, the H200 GPU, compared to the H100, has much faster and larger memory but the same core compute. Therefore, H200s are best used in the Decode pool, and H100s in the Prefill pool.

One of the major overheads in this serving technique is transferring the KV cache between the Prefill-dedicated GPU pool and the Decode-dedicated GPU pool, which can be optimized with faster interconnects such as NVLink or InfiniBand.

Thanks for reading this article. If you found it valuable, show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Arithmetic Intensity, Simply Explained

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Become a paid subscriber today

1. Claude Opus 5

Anthropic released Claude Opus 5, a model that comes close to the frontier intelligence of Claude Fable 5 at half the price.

Opus 5 is the new SOTA for coding and knowledge work evals and is designed to be used efficiently for everyday tasks, but it remains behind Mythos 5 on cybersecurity tasks. It also improves on Opus 4.8 across multiple benchmarks evaluating automation, computer use, scientific research, and visual output.

Opus 5 is described as Anthropic’s most aligned model yet and costs $5 per million input tokens and $25 per million output tokens (the same as Opus 4.8).

Read more about this release using this link.

2. Hilbert Operator for Progressive Encoding (HOPE)

This research paper introduces Hilbert Operator for Progressive Encoding (HOPE), a data-free and hyperparameter-free mathematical framework that helps deconstruct the internal knowledge stored inside deep neural networks.

While network compression is usually used to make models smaller, it can also help analyze what a model has learned. This is because removing a neuron tests whether it contributed to important knowledge, and merging neurons tells whether they perform functionally similar roles.

HOPE, or Hilbert Operator for Progressive Encoding, models each neuron as a rank-1 Hilbert-Schmidt operator. This allows evaluation of neurons in a Hilbert space of continuous functions and simplifies a neural network based on what each component actually does (its function) rather than relying only on the size of its weights.

With HOPE, different compression actions (pruning neurons, merging neurons, and removing residual blocks) can be treated as forms of the same low-rank projection problem and compared using a single scale-invariant metric.

Read more about this research using this link.

Before we move forward, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

3. LLMs Get Lost in Evolving User Intent

This research paper evaluates how well LLMs track and act on user intent as it evolves over the course of a conversation.

This is done using a framework that changes static, single-turn tasks into dynamic multi-turn conversations in which the user’s intent evolves across turns while preserving each task’s original evaluation method, allowing existing benchmarks to be reused as controlled test environments without requiring new annotations.

The results show that LLMs do not faithfully track and act on the user’s evolving intent, a capability that is critical for collaborative agents. For example, the performance of GPT-5.5 goes down from 99% to 80.5% on GSM8K after 6 intent changes.

Task switching is the most difficult transition for LLMs, and simple memory prompts or even restating the user’s correct current intent improve results but do not restore single-turn performance.

Read more about this research using this link.

4. Laguna S 2.1

Poolside released Laguna S 2.1, an open-weight, agentic coding model. It comes with an 118B total-parameter Mixture-of-Experts (MoE) architecture with 8B activated parameters per token and supports a context window of 1M tokens.

The model’s performance is a result of strong post-training that rewards verification, backtracking, and persistence at tasks, using more generous rollout budgets, better RL sandbox infrastructure, and multi-harness rollouts (same prompts rolled out in several agent harnesses).

Laguna S 2.1 performs competitively with much larger LLMs, scoring 70.2% on Terminal-Bench 2.1 and 78.5% on SWE-Bench Multilingual. Some impressive tasks that it achieves are:

• Building a basic browser engine from scratch

• Improving its software harness used for training/evaluation and user interactions, making it 5.2% faster with ~70% lower memory allocation

• Independently re-discovering a proof to Erdős problem #397 (the first proof to the conjecture was found earlier in January 2026 by GPT-5.2 Pro)

Read more about this release using this link.

5. Music-JEPA

This research paper introduces Music-JEPA, a world model that learns a world model of piano sound using JEPA by framing music as an action-conditioned system where the audio is treated as the state, and the pianoroll as the instrument action.

Given a current audio state and an action, the model predicts the resulting future audio state directly in a latent space rather than reconstructing the raw sound. The model is trained in a fully offline setting on roughly 200 hours of paired piano audio and MIDI, without environment interaction.

Experiments show that the learned model captures relationships between musical actions and their resulting sound, including pitch, timing, velocity, and pedal control, and outperforms JEPA models trained on passive audio alone.

The resulting representations can be used for downstream tasks such as beat tracking, composer identification, and key estimation, and enable piano transcription by planning, i.e., working backward to identify the actions most likely to have produced a given sound.

Although impressive, Music-JEPA focuses on classical piano with aligned MIDI, and it remains unclear how well the model scales or generalizes to broader musical settings where action annotations are unavailable.

Read more about this research using this link.

6. LLM-as-a-Coach

This research paper introduces Experiential Learning (EL), a method for improving LLMs on open-ended tasks (such as creative writing and summarization), where there is no single correct answer.

In standard RL, an LLM is used as a judge, and its detailed evaluation is reduced to a scalar reward. In EL, an LLM is used as a coach where it evaluates the model’s responses and produces detailed guidance, which is distilled into the model’s weights through a teacher model using On-policy context distillation.

LLM-as-a-Coach creates a dense supervision signal with 17,600 bits of information per sample, which is more than 5,000× the bandwidth of a scalar reward.

This approach consistently outperforms rubric-based RL on held-out and unseen open-ended tasks, generalizes better beyond the training distribution, and reduces reward hacking.

Read more about this research using this link.

7. Nanbeige 4.2

This research paper introduces Nanbeige 4.2-3B, an open-weight agentic model with strong performance in coding office tasks and complex tool-use tasks, and highly competitive reasoning capabilities in mathematics, coding, and science.

The model has only 3B non-embedding parameters and uses a Looped Transformer that sends information through the same layer stack twice to increase effective depth without adding parameters.

Nanbeige 4.2-3B was pretrained from scratch, then post-trained on execution-grounded agent trajectories and improves through multi-stage RL focused on response quality, reasoning efficiency, and stable agentic learning.

Evaluations show that it outperforms larger models such as Qwen3.5-9B and Gemma4-12B across diverse agentic benchmarks while remaining competitive on reasoning and alignment tasks.

Read more about this research using this link.

8. Loop the Loopies!

This research paper presents the Loopie series of Mixture-of-Experts (MoE) models:

• Loopie-20B-A2B: 20B-parameter model with 2B active parameters

• Loopie-6B-A0.6B: 6B-parameter model with 0.6B active parameters

These models use Looped transformers, where each Transformer layer is reused twice before moving to the next layer.

Looped transformers previously faced the challenge that, on an N-times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times.

This is fixed in the “Loopie Recipe,” which reuses each Transformer layer twice before moving to the next, halving the number of stored layers while preserving effective depth. The saved memory is then used to increase training throughput (batch size and model width) under the same wall-clock budget.

Comparisons with a vanilla 30B-A3B model show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget and, with a new post-training method, reaches frontier-level reasoning performance.

Read more about this research using this link.

9. Fugu-Cyber

Sakana AI released Fugu-Cyber, a cybersecurity-focused multi-agent orchestration system that behaves like a single model available as one API endpoint while dynamically assigning tasks to specialized agentic models.

This system is based on the Fugu orchestration model, which dynamically orchestrates the world's best models to solve complex, multi-step problems and reach frontier-level performance without single-vendor dependency.

Fugu-Cyber achieves SOTA performance on the industry’s most challenging security benchmarks, reaching a success rate of 86.9% on CyberGym and 72.1% on CTI-REALM.

These results are comparable to leading cybersecurity-focused frontier models such as GPT-5.5-Cyber and Mythos-Preview.

Read more about this release using this link.

10. SANA-Video 2.0

This research paper introduces SANA-Video 2.0, a hybrid video-generation model available at 5B and 14B scales that can generate high-quality video up to 720p on a single GPU.

The model uses hybrid Linear–Softmax Attention, which combines gated linear attention with periodic gated-softmax attention layers in a 3:1 ratio.

The softmax layers restore detailed token interactions that pure linear attention lacks, and these are propagated across depth (later layers) using Block Attention Residuals (AttnRes).

Trained from scratch, the 5B model achieves a VBench score of 84.30 while generating 480p video in 13.2 seconds on a single H100, remaining competitive with far larger softmax video DiTs at a fraction of the latency.

Its compiled DiT forward pass is 3.2× faster than a matched full-softmax baseline at 720p/60s, a gap that increases with video duration.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Hugging Face recently disclosed a security incident in its production infrastructure that is unlike anything it has handled before. This cyberattack was completely driven by an autonomous AI agent system comprising GPT‑5.6 Sol and a more capable unnamed pre-release model from OpenAI.

What went wrong?

The attack occurred when OpenAI ran an internal evaluation on a benchmark called ExploitGym. This benchmark tests how capable an LLM is at performing cyberattacks. Model versions with reduced cyber refusals were used in this test, and protective classifiers that prevent models from engaging in high-risk cyber activity were deliberately turned off.

The benchmark was supposed to run in a highly isolated environment. The only network access given was for installing packages using an internally hosted third-party software that acted as a proxy and cache for package registries. And this is where things all went wrong.

During evaluation, the models used a good amount of inference compute to find and exploit a zero-day vulnerability in this package registry cache proxy. They then performed a series of privilege escalation and lateral movement actions until they reached a node that had access to the Internet.

After gaining Internet access, the models worked out that Hugging Face potentially hosted models, datasets, and solutions for the ExploitGym benchmark, which they could use to cheat on the benchmark.

Using this information and over the course of the weekend, the model combined multiple attack vectors, stealing cloud and cluster credentials and exploiting zero-day vulnerabilities to identify a remote code execution path on the Hugging Face servers. This is when Hugging Face got alerted about the attack.

Subscribe now

How was the attack stopped?

The cyberattack was flagged by Hugging Face’s internal anomaly-detection AI pipeline, which uses LLM-based triage on its security telemetry. Based on suspicious signals, the team decided to analyze the full attacker action log, which was made up of more than 17,000 recorded events.

They first turned to frontier commercial model APIs to analyze the logs, but these models' safety guardrails refused to process these requests.

This is such a big power asymmetry, since the safety guardrails held back the compromised Hugging Face team rather than stopping the attackers!

The team then moved towards using the open-weight GLM 5.2 on their own infrastructure. This worked well and ensured that attacker data and credentials never left their environment during the analysis. Using this LLM-first approach, the team was able to do in hours what would usually take days, and match the attacker’s speed.

When the attack occurred, Hugging Face was still largely unaware of which model (whether a jailbroken hosted model or an unrestricted open-weight model) was responsible for it. OpenAI’s security team, however, discovered this activity internally and in due course reported it to Hugging Face while the attack was being contained.

Hugging Face said that they found no evidence that the attack involved public, user-facing models, datasets, or Spaces. Their software supply chain (container images and published packages) was also verified to be clean.

The benchmarks saw this coming

ExploitGym, the benchmark that was being run during this exploit, consists of 898 instances drawn from real-world vulnerabilities across three domains: userspace programs, Google’s V8 JavaScript engine, and the Linux kernel.

Initial tests on this benchmark showed that frontier models could successfully exploit a big chunk of these vulnerabilities. Claude Mythos Preview and OpenAI’s GPT-5.5 could generate working exploits for 157 and 120 instances on this benchmark, respectively.

These offensive capabilities even remained when the widely used defenses were enabled.

The UK AI Security Institute’s evaluation showed similar results, indicating that frontier models such as GPT‑5.5 and Mythos/Claude could autonomously execute long-time-horizon, complex multi-step cyberattacks.

Autonomous AI-driven cyberattacks are no longer a thing of the future. They are here. And given what the benchmarks and red-team evals have been signaling for months, it's no surprise this was coming.

Companies and organizations that still haven't prepared need to start now. This means treating the data and model surface as primary targets for attackers and putting AI to work in defense to guard against such attacks.

Incidents like this will happen faster and hit harder from here. This is just the beginning.

Further Reading

• UK AI Security Institute’s report titled ‘Our evaluation of OpenAI’s GPT-5.5 cyber capabilities’

• ArXiv paper titled ‘ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?’

• Hugging Face’s security incident disclosure

• OpenAI’s security incident disclosure

1. Kimi K3

Moonshot AI released Kimi K3, the world's first open 3T-parameter class model, designed for frontier intelligence across long-horizon coding, knowledge work with native vision, and reasoning.

The model has 2.8T parameters and comes with a 1M-token context window. It uses Kimi Delta Attention, Attention Residuals, and a highly sparse MoE architecture (Stable LatentMoE) that activates 16 of 896 experts, resulting in 2.5× better scaling efficiency than Kimi K2.

While its overall performance still remains behind the most powerful proprietary models, Claude Fable 5 and GPT 5.6 Sol, Kimi K3 shows frontier-level performance across diverse benchmarks, consistently outperforming other tested models.

Some limitations of the model are that it performs less reliably when earlier reasoning steps or important context are missing. It also tends to act too proactively when user instructions are unclear, making unexpected decisions on the user's behalf.

Read the technical blog using this link.

Before we move forward, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

2.xHC: Expanded Hyper-Connections

Hyper-Connections (HC) expand the residual stream of Transformers into N multiple parallel streams, which enables memory scaling beyond model width and depth, while Manifold-Constrained HC (mHC) stabilizes this approach at scale.

However, scaling mHC beyond N=4 leads to diminishing performance gains and rapidly increasing training cost because very little information is written back across the growing number of streams, while residual-mixing generation scales cubically with N.

This research paper addresses this by introducing xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond N=4. xHC does this by maintaining 16 readable streams, providing dense access to the full residual state, while updating only 4 streams at each layer.

xHC improves performance across knowledge, reasoning, coding, and Chinese-language benchmarks across MoE models. To reach the same training loss, standard Transformers and mHC require 1.50× and 1.19× as much compute as xHC, respectively.

Read more about this research paper using this link.

Subscribe now

3. VideoChat3

VideoChat3 is a fully open (model weights, training code, training strategy, and complete training datasets), 4B-parameter multimodal model with generalist video-centric capabilities.

VideoChat3 uses the Inflated 3D Vision Transformer (I3D-ViT) and adaptive frame resolution to efficiently compress video while preserving spatiotemporal information, and reduces the cost of processing video inputs during training and inference.

Trained on roughly 3 million instruction samples, VideoChat3 achieves a rare balance of broad generalization and computational efficiency, outperforming prior open-source models with equal or larger parameter counts while using only 4B parameters.

Read more about this research using this link.

4. Inkling

Thinking Machines Lab released Inkling, an open-weight model for reasoning, coding, tool use, and multimodal tasks. It uses the Mixture-of-Experts architecture with 975B total parameters and 41B active parameters, and supports a context window of up to 1M tokens.

Alongside this, the team released a preview of Inkling-Small, a lighter-weight model with 12B active parameters, trained with a similar recipe, that achieves strong performance with even lower cost and latency.

Inkling comes with adjustable reasoning effort and can be fine-tuned through the Tinker platform.

While it is not the strongest overall model available today (open or closed), its diverse capabilities make it a good open-weights base for customization.

Read the technical blog using this link.

5. AIDE²: The First Evidence of Recursive Self-Improvement

AIDE² is a recursive self-improvement (RSI) system that uses two autoresearch loops:

• An inner loop that functions as a normal autoresearch agent, optimizing code against an eval.

• An outer loop that optimizes the inner-loop agent’s harness code.

Over 100 autonomous outer-loop iterations running for eight days, AIDE²:

• Discovered 7 successive improved versions of AIDE

• Developed a new search algorithm

• Reduced the prompt size by 16×

• Outperformed a manually tuned agent that had been iterated on for two years, on several unseen tasks under a fixed compute budget

• Reduced reward hacking on a held-out GPU benchmark from 63% to 34%

The team describes this as “Level 1” recursive self-improvement because the system improved itself more efficiently than human researchers. However, it did not reach “Level 2” or “Ignition,” because the improved agent was not conclusively better at running the outer self-improvement loop, and its evolved code became complex and difficult to maintain.

Read the technical blog using this link.

Subscribe now

6. WanSong v1.0

WanSong is Alibaba’s pure diffusion-based model for generating commercial-grade, multilingual songs up to 5 minutes long. Trained on more than six million hours of audio, the model produces complete audio in a single run and separately outputs vocals and background music.

On a multilingual and multi-genre benchmark, WanSong achieved the lowest reported pronunciation error and the highest musicality score compared with LeVo, Mureka V7.6, and Suno V5.

Read more about this research using this link.

Subscribe now

7. Bonsai 27B

Bonsai 27B is PrismML's new multimodal model based on Qwen3.6 27B, capable of multi-step reasoning, structured tool calls, vision tasks, and agentic loops for computer use.

Models of this size typically require 54GB of memory at 16-bit precision (27B × 2 bytes per parameter). This far exceeds the memory of any typical phone.

Bonsai 27B changes this with its two variants:

1. Ternary Bonsai 27B: Uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, leading to 1.71 effective bits per weight. This totals 5.9 GB of memory, which makes it suitable for running on an everyday laptop.

2. 1-bit Bonsai 27B uses binary {−1, +1} weights with FP16 group-wise scaling, leading to 1.125 effective bits per weight. This totals 3.9 GB, which means it fits and runs comfortably on an iPhone 17 Pro Max.

Ternary Bonsai 27B achieves 95% of the full-precision baseline, and 1-bit Bonsai 27B achieves 90% across 15 benchmarks in knowledge, reasoning, math, coding, instruction following, tool calling, and vision.

Read the technical blog using this link.

8. Mach-Mind-4-Flash

This research presents Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts (MoE) agentic model (3B activated) that, through post-training optimization alone, achieves performance on par with or surpassing that of 100B-parameter-class models.

Its three-stage training pipeline combines:

1. A unified RL and on-policy distillation infrastructure, with dynamic multi-teacher scheduling and operator-level acceleration, which gives it a 17% end-to-end training speedup

2. Multiple domain-specific RL experts trained in parallel across Reasoning, General, and Agent tracks, then combined into a single generalist using Multi-Teacher On-Policy Distillation (MOPD)

3. Hybrid Median-length Policy Optimization (HMPO), which compresses reasoning chains by 19–46% with not more than a 0.7-point loss in accuracy.

Mach-Mind-4-Flash scores competitively matching or outperforming models with 10-30× its activated size while using substantially less inference compute.

Read more about the research using this link.

9. Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values

This research identifies a type of misalignment in LLMs called Covert value leakage. This is when the information provided by an LLM is influenced by its own values, and this influence is never disclosed to the user.

For example, when a user asks whether they are considering investing in an AI company and wants to know how likely the AI bubble is to pop, Claude Opus 4.8 gives a lower probability when the company under consideration is Anthropic rather than OpenAI, without disclosing this influence to the user.

The research shows that models are influenced by different types of values, including:

• Preferences for morally good outcomes

• Preferences for the company that developed them

• Preferences for some human leisure activities over others

Models differed in how openly they revealed these biases. For example, Claude models falsely claim to provide unbiased answers during chain-of-thought reasoning, whereas Qwen models explain how their values bias their answers.

Read more about this research using this link.

10. On-Policy Delta Distillation

On-policy distillation (OPD) is an RL post-training method that trains a student model based on its own responses using token-level feedback from a stronger teacher model. This encourages the student model to generate tokens that the teacher prefers.

This research paper improves OPD by introducing a new distillation reward, called the ‘delta signal’, instead of the teacher’s output distribution.

The delta signal is the difference between the teacher model and its base model prior to post-training for reasoning. This helps capture and transfer the reasoning skills gained during the teacher model’s post-training rather than copying all of its knowledge and style. This new approach is termed On-Policy Delta Distillation (OPD²).

Experiments across mathematics, science, and code-reasoning benchmarks show that OPD² consistently outperforms conventional on-policy distillation. However, it requires access to both the teacher and its base model and increases training time compared with standard OPD.

Read more about this research using this link.

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Building Your First AI Agent

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Become a paid subscriber today

AI/ML research is hard to keep up with. I have found three main reasons for this:

1. Many engineers are not used to reading math-heavy research papers

2. The language of research papers is dense and requires strong foundations to understand

3. There is just too much research being published every day, and it’s tough to read it all

Let’s address each of these issues one by one to make AI/ML research easy to follow.

Before we start, I want to introduce you to my book, ‘LLMs In 100 Images’.

It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.

I’m offering a limited-time 30% discount, which you can claim using the button below.

Get a 30% discount today

Math-heavy papers & what to do about them?

Check out a paragraph from the research paper titled ‘Improving Language Understanding by Generative Pre-Training’, which introduces the GPT architecture.

The paragraph describes the pretraining objective that is maximized during LLM training.

It looks intimidating, but it actually isn't. It is written this way because math is a great way to condense ideas and concepts and avoid verbosity.

To make it easy to understand, simply pass it to an LLM with a simple prompt as follows.

And this is what you get as a response.

You could also prompt an LLM further to explain a concept in more detail or, with step-by-step examples, if you’re still struggling to understand something.

All thanks to LLMs, this is one of the best (and judgment-free) ways to understand math-heavy research papers.

Subscribe now

“Should I read a math book before reading a research paper?”

There are two ways to approach the mathematics in AI/ML research:

1. Bottom-up: Read all the foundational math first (Linear algebra, calculus, control theory, mathematical optimization, and so on), and then start reading AI/ML research.

2. Top-down: Read an AI/ML research paper first and then learn the relevant math.

I’ve followed both approaches in the past and found that the ‘Bottom-up’ one simply took too much time and effort, making it impractical. (I was stuck trying to learn all the different ways to integrate equations and multiply matrices, which later proved to be of not much use.)

I have long since dropped this approach and now read research papers first, then go ‘Top-down’ to read only the math essential to understanding the paper. This is what I find most efficient and recommend to you, unless you’re absolutely new to math (which is usually not the case for engineers).

Dense language & what to do about it?

Research papers are deliberately written concisely, avoiding verbosity. Otherwise, a single one would have pages enough to fill an entire textbook.

Here’s an excerpt from ‘Attention is all you need’, the famous research paper that introduces the Transformer architecture.

This short text introduces you to roughly a dozen concepts, including:

• Recurrent neural networks

• LSTM

• GRU

• Sequence modeling/ Language modeling

• Machine translation

• Encoder-Decoder architectures

• Sequential computation precluding parallelization

• Memory constraints limiting batching across examples

• Factorization tricks

• Conditional computation, and so on.

To a person just getting introduced to AI/ML research, this is a nightmare.

Here are three ways to read and understand such dense language:

1. Scan and read in passes

   Give yourself time and read the research paper in multiple passes rather than in one go. First, scan the research paper to get its basic layout and reassure yourself that the paper does not go on and on forever. (Sounds silly, but it works.) Then, patiently read each section one at a time.

   If you’re completely new to AI/ML research, there’s a high chance that none of it will make any sense in the first go. This is completely normal.

2. Let an LLM expand the dense sections

   Pass each dense paragraph one at a time to an LLM and ask it to “Explain this paragraph line by line in simple words.” Take your time to understand the concepts, and keep returning to the earlier sections until everything fits together in a coherent sequence.

3. Don’t let the jargon overwhelm you

   Look up unfamiliar terms with a simple web search and follow references.

If you follow these steps patiently, I promise everything will eventually start making sense.

Too long, too many, no time

Now, if you’re completely comfortable with reading AI/ML research, there’s a chance that you might feel overwhelmed by the number of research papers being published. Each paper is 15-20+ pages long, and hundreds to thousands land on arXiv every day.

Finding the relevant ones, let alone reading everything published each day, seems impossible.

I solve this by simply curating my social feeds (I mostly use LinkedIn and Substack) towards AI/ML research, releases, and developments.

Here are some accounts that you could follow on LinkedIn that talk about AI and ML:

• Yann LeCun

• Sebastian Raschka

• Cameron R. Wolfe

• Ignacio de Gregorio Noblejas

• Deedy Das

• Karun Thankachan

• Dr. Ashish Bamania

Here are some of my recommendations from Substack in the AI/ ML engineering, trends, and macroeconomics space:

I also have an AI agent that brings me some interesting research papers every week, and you could build one for yourself following this tutorial.

Alongside this passive approach of simply letting my feed decide what I am updated on, I look out for recently published research papers on:

• Hugging Face Trending papers

• AlphaXiv

I write about what I found important each week in a newsletter edition titled “This Week in AI Research” and send it out every week. This could be a source you can use to stay up to date on AI/ML research.

How to quickly filter through AI/ML research?

Now that you have a curated feed and other trusted sources of AI/ML research, there’s still the problem of going through them all.

I use a simple system to find important research papers that you can steal.

• Quickly read through the abstract, check the figures, and read the conclusion.

• If a paper seems promising or directly relevant to your work or curiousity, then go deep into the methodology and evaluation sections. This is where the most important details of a paper lie: whether baselines are fair, whether ablations are present, and whether benchmarks were cherry-picked.

How to actually learn from a research paper?

Simply reading a research paper end-to-end is less useful than applying what it teaches. Some of the ways to make the learnings stick are:

• Implement them in code: Manually code up the core idea from a research paper. Use an LLM to review your code, but don’t rely on vibe-coding. I frequently use this approach to learn better. (Example)

• Use them at work or in your personal projects: Always ask how you can apply your learnings to the product or pipeline you’re working on. This will give you compounding returns in your career.

• Start writing: Writing personal notes or an explainer/tutorial for others is another great way to learn better. Writing forces you to really get good at something before you can simplify it and explain it to others.

“Which papers to read if I am starting out?”

Pick 2-3 areas that deeply interest you or are relevant to your current work or career ambitions. Prioritize the early work and foundational papers in these areas and read them first.

A great way to build a strong foundation is to keep following and reading the research papers in the references. This will help you master a field’s foundational concepts faster than you might expect.

For example, before reading about Heavily Compressed Attention (HCA), I would follow references and read about:

• DeepSeek Sparse Attention

• Multi-head Latent Attention

• Self-Attention

• Bahdanau attention

How to build a research reading habit?

• Curiosity and necessity are two great forces that might motivate you to read AI/ML research. For me, curiosity has been a big one.

• Curate your feed and connect with others who love reading and keeping up with AI/ML research.

• If you’re a beginner, set aside 1 to 2 30- to 60-minute sessions dedicated to reading AI/ML research every week. Put them in your calendar as protected time and do not skip them.

Reading for just an hour a week beats a heroic weekend binge every few months, and this is how you build a habit and win with slow compounding without burning out.

No one keeps up with all AI/ML research

My final advice to you before we conclude is to drop the unrealistic goal of keeping up with everything happening in AI/ML. No one can do this, including full-time researchers. Let go of this goal, be easy on yourself, and take your time.

👋🏻 Feel free to leave a comment or DM me if you have any questions, and share this with others if you found it valuable!

Share

Join the paid tier today to get access to all posts in this newsletter, including:

• 🌈 20 PyTorch Concepts, Explained Simply

• 🧑🏻‍💻 Building Your First AI Agent

• 👾 Tiny Recursive Model (TRM): A Deep Dive

• 👷🏼‍♂️ Build A Vector Database From Scratch To Understand RAG In Depth

• 🚀 Build a Mixture-of-Experts (MoE) Layer from Scratch

and so many more!

Become a paid subscriber

1. Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Existing RL training approaches update all model parameters uniformly, assuming that every layer contributes equally to the gains obtained.

This research paper challenges this idea and finds that training a single transformer layer can recover most of the gains achieved by full-parameter RL training and, in some cases, even surpass them.

Studying seven models across two Qwen families (Qwen2.5 and Qwen3), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains, it is found that RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layer.

These high-contribution layers are usually in the middle of the transformer stack, while layers near the input and output ends contribute substantially less.

Based on these findings, the researchers develop simple layer-aware training strategies that consistently outperform standard full-parameter RL training.

Read more about the research paper using this link.

Before we move forward, I want to introduce you to the Visual Tech Bundle.

It is a collection of visual guides that explain core AI, LLM, Systems design, and Computer science concepts via image-first lessons.

Others are already loving these books. Why not give them a try?

Link to the Visual Tech Bundle

2. LLM-as-a-Verifier

This research paper introduces LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training.

Unlike standard LM judges, which are LLMs prompted to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to produce continuous scores.

This probabilistic approach reduces tie rates when comparing complex solutions and enables verification across multiple dimensions:

• Granularity of score tokens

• Number of repeated evaluations

• Decomposition of evaluation criteria

To make verification scaling practical, the researchers further introduce a cost-efficient ranking algorithm to select the best solution among candidates, using preference probabilities derived from the verifier’s continuous scores.

LLM-as-a-Verifier achieves SOTA performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%).

Beyond verification, it can also act as a proxy for estimating task progress and as a dense reward signal for RL, making training more sample-efficient.

Read more about the research paper using this link.

3. Mixture-of-Parallelisms

This research paper proposes a memory-efficient training stack called Mixture-of-Parallelisms (MoP) for Mixture-of-Experts (MoE) models.

It uses multiple parallelism techniques across the training pipeline to maximize efficiency under the cluster constraints, including a new strategy for the optimizer step.

MoP achieves high throughput and memory efficiency, enabling lossless pre-training/fine-tuning of trillion-parameter-scale models at a million-context length using just under 12 8x H200 GPU nodes.

It delivers 4.7-8.2× higher per-GPU throughput than a strongly-tuned FSDP2 baseline and sustains training at context lengths up to 1M tokens, whereas the baseline runs out of memory beyond 64–128K context length.

Read more about this research paper using this link.

4. Measuring the Gap Between Human and LLM Research Ideas

This research paper examines how LLM-generated research ideas differ from those of human researchers.

Studying thousands of published papers and prompting models from families including GPT, Claude, Gemini, DeepSeek, and Qwen to generate ideas from the same prior-work context, the authors find that:

• LLM ideas are disproportionately concentrated around connecting existing fields or synthesizing existing methods

• Human ideas cover a wider range of motivations and contributions, which include identifying failures, explaining contradictions, relaxing assumptions, and developing new systems

This suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.

Read more about this research using this link.

5. Infinite Worlds with Versatile Interactions

This research paper presents LingBot-World 2.0 (LingBot-World-Infinity), a model that generates responsive, open-ended, infinite virtual environments for exploration.

Compared to the earlier version, this model maintains consistent output quality, supports up to 720p at 60 fps, and generates highly diverse interactive elements that enable broader actions (e.g., attacking, archery, spell-casting, and shooting) and a variety of text-driven events.

Additionally, the model is integrated with an agentic harness with two autonomous agents:

• a pilot agent that plans and runs character behavior, and

• a director agent that creates new environmental elements as the scene progresses.

The model is available in 14B and 1.3B versions, with the 1.3 B version capable of running on a single GPU.

Read more about this research using this link.

6. AutoMem: Automated Learning of Memory as a Cognitive Skill

This research paper introduces AutoMem, a framework that reframes LLM memory management as a trainable skill rather than a fixed system.

An LLM agent using it gets file-system operations (read, write, search, append) as first-class actions, allowing it to decide what to remember and retrieve.

A strong LLM then improves this automatically by reviewing long episode traces to revise the agent's memory scaffold, and fine-tuning a dedicated memory model on the agent's own good decisions.

Across three procedurally generated long-horizon games (Crafter, MiniHack, and NetHack), optimizing memory alone (without modifying the model’s task-action behavior) improves the base agent’s performance by 2-4×, bringing a 32B open-weight model to a level competitive with frontier systems such as Claude Opus 4.5 and Gemini 3.1 Pro Thinking.

Read more about this research using this link.

7. GPT‑Live

OpenAI launched its new generation of voice models called GPT-Live for natural real-time human-AI interaction, with its two versions being GPT‑Live‑1 and GPT‑Live‑1 mini.

GPT‑Live is built on a full-duplex architecture, which means that meaning it can listen and speak at the same time, acknowledge you while you talk, pause when you need time, handle interruptions more naturally, and delegate harder tasks like search or reasoning to frontier models such as GPT-5.5 in the background.

Read more about this release using this link.

8. Grok 4.5

SpaceXAI released its new flagship model called Grok 4.5. This model is built for coding, agentic tasks, and knowledge work, with a big focus on software engineering and efficient reasoning.

Grok 4.5 was trained on tens of thousands of NVIDIA GB300 GPUs using large-scale reinforcement learning, with a focus on per-token intelligence.

It reaches 83.3% on Terminal Bench 2.1, resolves SWE-Bench Pro tasks with ~4.2× fewer output tokens than Opus 4.8 (max).

The model is served at about 80 tokens per second generation speed and is priced at $2 per million input tokens and $6 per million output tokens.

Read more about this release using this link.

9. SWE-1.7

SWE-1.7 is Cognition’s most capable software-engineering model to date. It is designed for long-horizon, asynchronous coding tasks and is built by RL post-training a Kimi K2.7 base model.

It scores 42.3% on FrontierCode 1.1, 81.5% on Terminal-Bench 2.1, and 77.8% on SWE-Bench Multilingual, bringing it close to frontier models like GPT-5.5 and Claude Opus, while achieving better cost efficiency.

Read more about this release using this link.

10. Muse Spark 1.1

Muse Spark 1.1 is a multimodal reasoning model from Meta Superintelligence Labs built for exceptional performance in agentic workflows, tool and computer use, coding, and multimodal understanding.

The model can orchestrate parallel subagents, has a 1-million-token context window, and outperforms Gemini 3.1 Pro (high), Opus 4.8 (max), and GPT 5.5(xhigh) on multiple difficult benchmarks.

Read more about this release using this link.

Join the paid tier today to get support my writing and get access to all posts in this newsletter.

Join 'Into AI' premium today

This newsletter edition is completely free to read. Show your love by liking it, restacking it, and sharing it with others! ❤️

Share

Keeping up with ML research is hard these days, given the mammoth number of papers published every single day.

This Saturday (July 11, 2026), I’m going live with Karun Thankachan, a brilliant Senior Data Scientist at Walmart, to share the exact framework we use to stay on top of it all.

In one session, you’ll learn:

• How to spot quality research among the endless stream of new papers

• How to read papers efficiently and skip what doesn’t matter

• How to pull out takeaways you can use in your day-to-day work

Everyone who registers also gets a handout with all the resources discussed in the session.

Save the date: 📅 Saturday, July 11 | 🕑 12:00 PM EST

It’s free to join, and you can register using the link below. :)

Join the webinar →

Seats are limited and filling fast!

See you inside,
Ashish

I’m offering a handful of consulting slots for this quarter to help solve your problems.

This is for you if:

• You want to launch an AI product or feature at your company, but you don’t know where to start or how to make it happen

• You have a business idea but are unsure how to begin with the tech

• Your team is overwhelmed, and you want someone to review your workflows and create systems that save you time and increase your revenue

• You have a half-finished or stalled AI project that needs to get moving

• You’re pouring effort into your social media, and it’s just not growing, and you’d love to turn it into something you could eventually do full-time

• You’re looking to market your product and want it to reach the right audience

The first step is a free 30-minute call to see how I can help.

🤝 If you’re interested, just reply to this email in a few lines about what you’d like help with, and I’ll send you a link to schedule a time to connect.

P.S. This is not a sales-maxing pitch, and I’m only taking on a few clients that I can truly help, so the spots are limited. If this sounds like you, don’t wait up.

Talk soon,
Ashish

What is Arithmetic Intensity?

Arithmetic Intensity (AI) is the ratio of arithmetic operations/computations performed to the data moved to and from memory.

It indicates how much computation is performed per unit of data transferred from memory. If a workload/kernel performs very few arithmetic operations but requires massive data reading and writing from memory, it has low arithmetic intensity, and vice versa.

Note that:

• FLOP is a floating-point operation (a single arithmetic operation on floating-point numbers).

• FLOPs is the plural form of FLOP or the total number of floating-point operations.

• FLOPS is floating-point operations per second.

How is it used in the Roofline model?

In a GPU, HBM is the slowest but the largest, while registers are extremely fast but the smallest units of memory.

Read more