Hugging Face used reinforcement learning to teach a model to paint watercolours

Hugging Face published a training write-up on 3 September 2026 that answers a question anyone working with reinforcement learning has probably asked: can you run RL on taste, when there is no test to pass and no math answer to grade? The team's answer is yes, with a specific recipe, and they released the code, the trained adapters, and the training data alongside the post.

The setup

The task is to write a JavaScript program using the p5.brush library, which simulates watercolour painting in a headless browser. The model produces code, Chromium renders the image, and a reward function scores the result. The base model is Qwen3.5-35B-A3B, fine-tuned with LoRA using GRPO through TRL, with OpenEnv managing the reinforcement learning loop. Training ran on H200 hardware through Hugging Face Jobs, at 18 to 34 hours per run.

The reward is a composite. A small gate term (weight 0.05) checks that the code compiles and does not cheat. A length term (weight 0.05) prefers longer code. The two terms that count are a pairwise vision judge (Qwen3-VL-30B, weight 0.60) that compares the output against a hand-curated reference pool of 178 paintings, and HPSv3 (weight 0.30), a 7B preference model that scores aesthetics directly.

The measured outcomes

Three runs used different reward mixes. The hps-only run, driven purely by the preference model, moved from a group-mean reward of 0.58 to 0.71 over 60 steps, but Hugging Face notes that "three quarters of the rise in group mean comes from bad paintings becoming rare" rather than from top-tier improvement. The judge-led run, at 0.60 pairwise judge and 0.30 HPSv3, moved from 0.45 to 0.72 over 110 steps. The hps-led run, with the weights flipped, moved from 0.57 to 0.82 over 110 steps and gave the best aesthetic convergence.

The pairwise judge also changed a concrete behaviour: paint coverage roughly doubled in the judge-led runs (0.11 to 0.23, 0.13 to 0.30) while the hps-only run barely moved on that axis.

The configuration lessons

The post is candid about what did not work at first. Learning rate at 2e-5 was too low and pushed to 5e-5. The default linear scheduler burned through most of the learning rate by mid-run, so the team switched to constant_with_warmup. Setting scale_rewards to group let gate rejections shrink the useful rewards, so they set it to none. And hand-picked LoRA target modules missed layers in the MoE architecture, so they set all-linear. These are the kinds of adjustments that separate a training run that stops improving from one that keeps learning.

Most teams thinking about reinforcement learning for their own work have met the same problem: the interesting signals in a product are subjective. Does this code review comment sound helpful. Does this generated UI look right. Does this summary match our tone. This recipe is a working example of what a preference-model approach actually looks like end to end, with the failure modes named.

Two lessons apply across use cases. A preference model alone tends to remove the worst outputs before it improves the best; adding a pairwise judge grounded in a curated reference pool pulls the top of the distribution up as well. And the reference pool has to be built by someone with taste, 178 examples here, which sets the upper limit on what the trained model can reach.

Source

Primary source: Training a coding model to paint watercolours with TRL and OpenEnv on the Hugging Face blog.