Lasso Security tested SynthID watermarking on six open models and found it changes 6.5 percent of tool calls on average

Turning on a content watermark can change what a coding or support agent actually does. Lasso Security ran Google DeepMind's SynthID-Text watermark against six open models and reports that 6.5 percent of tool calls change verdict on average, and that refusal to prompt injection weakens on four of the six.

The study, published by Lasso Research author Andrea Siposova on 17 September 2026, tested Phi-4, Llama-3.1-8B, Granite-3.2-8B, Gemma-3-27b, Gemma-3-12b and Qwen3-4B. Each model was run twice on the same input, once with the watermark on and once without, at temperatures 0.001, 0.7 and 1.0. The tool-call test used the 1,150 non-live call-expected tasks in BFCL v4 single-turn AST. The refusal test used 200 HarmBench harmful behaviors and 100 benign JailbreakBench controls.

The 6.5 percent hides a wider spread by model

Lasso Security reports the paired disagreement rate, meaning the share of prompts where the same model with the same input returns a different tool call once the watermark is applied. Across the 21 model-temperature combinations the study averages 6.5 percent. Phi-4 at temperature 1.0 changes verdict on 16.8 percent of prompts and loses 2.87 points of net accuracy. Llama-3.1-8B at the same setting changes on 9.9 percent. The paper reports the same disagreement at temperature 0.001, where model output is close to deterministic, so temperature noise cannot explain it.

Refusal weakens under prompt injection

The refusal test asks the model to refuse a HarmBench prompt directly, and then asks the same prompt inside a prompt-injection wrapper. Watermarking left the direct case close to baseline for every model, and changed the injected case for four of the six. On Gemma-3-27b at temperature 0.001, Lasso Security reports refusal churn rises from 6.0 percent on bare harmful requests to 23.5 percent under prompt injection. At temperature 0.7, four models show watermark-induced churn exceeding what a temperature swing produces: Gemma-3-27b at 26.0 percent versus 13.5 percent, Granite-3.2-8B at 21.5 percent versus 15.5 percent, Llama-3.1-8B at 17.5 percent versus 7.5 percent, and Gemma-3-12b at 11.0 percent versus 6.0 percent.

The study repeated the tests with ten additional watermark keys on top of the first one, and the drift held across the eleven keys. Lasso Security calls the effect a "provenance tax" and writes that a provenance mechanism that appears stable on ordinary inputs may not hold under attack.

For any team about to switch on watermarking for a shipped agent, the paper is a reason to add a before-and-after gate to the eval suite so the change is measured on the team's own tasks. The measured drift puts the watermark in the same review as the temperature setting, the system prompt and the safety filters. Every figure above is Lasso Security's own, on six open models under a specific method; a shop running closed models or an in-house watermark will need its own run.

Source

• Lasso Security: The Provenance Tax
• Discussion: Hacker News thread