Anthropic recently announced that it is introducing watermarking for text generated by Claude models. This is to comply with the EU Code of Practice on Transparency of AI-Generated Content, which requires AI system providers to use methods of marking AI-generated text.
Let’s understand visually how the watermarking process works.
But first, how do LLMs generate text?
LLMs are trained to generate text one token at a time, based on the preceding tokens/ context.
Check out the following example where an LLM generates the token “dark” given the context “The solar eclipse turned everything”.
Behind the scenes, the tokens in the context are converted into token IDs and then into token embeddings, which the LLM processes to produce logits.
Next, we take the logits for the last position and apply softmax to them to obtain probabilities.
Let’s look closely at the step where the Softmax function is applied. Softmax turns logits into non-negative probabilities that sum to 1. Here are the top-5 next-token probabilities for the given context “The solar eclipse turned everything”.
From here, different sampling algorithms can be applied to select the next token.
In our example, the next sampled token is “dark”. It is appended to the context “The solar eclipse turned everything”, and the process is repeated to generate further tokens until either a maximum response length is reached or an end-of-sequence (EOS) token is generated.
How is the generated text watermarked?
The watermarking process does not require any further LLM training. It only changes how tokens are sampled from the probability distribution of next tokens.
It consists of the following steps:
Generate a secret watermarking key (Anthropic holds it)
Given the last ‘H’ number of tokens in the context and the watermarking key, generate a random seed
This seed is input into ‘m’ watermarking functions, whose job is to randomly assign a score of 0 or 1 to every token. In our example, we consider m = 3.
Sample 2m tokens (which will likely be non-unique) from the LLM. In our example, this is 23 = 8 tokens. Note how “dark” and “black” are picked more times than “eerie” and “gray”, as their next-token probabilities are higher.
Pair these tokens, and in each pair, select the highest-scoring one based on the watermarking function chosen.
Repeat this process multiple times for multiple rounds/layers using successive watermarking functions, and break ties randomly if they occur.
The winner of the last tournament round/ layer becomes the next generated token. This sampling algorithm is called Tournament sampling.
The generated token is appended to the context, and this process is repeated to generate further tokens until either a maximum response length is reached or an end-of-sequence (EOS) token is generated.
Note how “dark” was the most probable token, and it was even sampled three times, yet it lost in the tournament of tokens, and “black” was chosen as the next token instead. This pattern that tournament sampling leaves across many steps of generation is what makes the text detectable later.
How is the watermarked text detected?
Watermark detection works as follows:
Take the watermarking key used during generation (this key is secret to Anthropic) and the tokenized text to be analyzed
For each token, take its previous ‘H’ tokens and the key to rebuild the random seed used during generation
Use this seed to compute the values of the ‘m’ watermarking functions for the token at that position
Average these values across all tokens and all rounds/ layers of the tournament to get a score. Here:
‘T’ is the number of tokens in the text being analyzed
‘t’ is the position of the token
‘m’ is the number of tournament rounds/ layers
‘l’ is the index of the layer
xt is the token at position ‘t’ in the text being analyzed
rt is the random seed at position ‘t’, rebuilt from the previous ‘H’ tokens and the watermarking key
Compare this score to a threshold. If it exceeds it, the text can be called watermarked.
According to Anthropic, a watermarked text detection API will be available soon.
The above-described algorithm of text watermark generation and detection is called SynthID-Text and was developed by Google DeepMind. Anthropic says that it uses a version of it to watermark Claude’s text but does not explicitly disclose the finer details.
Even if these details are not mentioned, it is clear that the watermark only changes the source of the randomness that is used to pick tokens from the next-token probability distribution.
When watermarking is not used, it is multinomial sampling, with randomness generated by a pseudorandom number generator.
In the watermarking process, it is Tournament sampling, where the next token is decided by pseudorandom values coming from the watermarking key and the context.
Clearing the myths
There are many misconceptions around the watermarking process that need to be cleared up. Here is what is true:
Watermarking has no impact on the output quality, creativity, or readability of the generated text.
It does not push Claude to produce words it would not have picked otherwise. For example, in the context “The solar eclipse turned everything”, watermarking would not cause Claude to generate a very rare next word such as “spectral”.
There are no hidden characters in the generated text.
It does not cost extra tokens when generating watermarked text.
It has a negligible impact on the token-generation speed of models.
It is not applied in cases where there’s a single answer (e.g., “1 + 1 =”) or low-entropy outputs, such as facts (e.g., “Which is the latest book written by Austin Kleon?”) and code (except code comments).
Neither the watermark nor the key contains anything that identifies a user, an organization, or a chat.
The watermark can only determine if Claude was likely involved with the content at some point. It cannot distinguish “Claude wrote this” from “Claude heavily edited this.”
It does not confirm whether a text was written by a human, nor is it a method for AI-generated text detection from other LLMs.
Confidence in detections is higher for longer samples.
A complete rewrite of the generated text removes the watermark, but light editing probably will not.
For files generated by Claude, content credentials are added to the file’s metadata, indicating whether the file was generated or processed by Claude. This labeling uses an open industry standard called C2PA, which differs from a watermark in that it resides in the metadata rather than in the content itself.
TL;DR
Anthropic has introduced watermarking for all text generated by Claude to comply with the EU AI Act.
The watermarking method is based on Google DeepMind's SynthID-Text.
The method does not add any hidden characters to the text, nor does it require any LLM training. It only replaces the source of randomness used to select each token, using a sampling algorithm called Tournament sampling.
Watermarking does not affect the quality or creativity of text generation or the text's readability.
A positive detection result only means that Claude was likely involved with the content at some point (generation or editing).
It does not detect whether text is human-written, nor is it a method for detecting AI-generated text from other AI models.
A light edit may not, but a full text rewrite can remove the watermark.
As we finish this lesson, I want to introduce you to my book, ‘LLMs In 100 Images’.
It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.
I’m offering a limited-time 30% discount, which you can claim using the button below.
This newsletter edition is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️
Join the paid tier today to get access to all articles in this newsletter and level up as an AI engineer.