
The question
“Vibe coding” means building software by describing what you want in plain language and letting an AI write the code. It is spreading fast. So are fully automated pipelines in which AI agents instruct other AI agents. Two questions follow. Is there still a unique benefit to having a human in the loop? And if so, which job should the human do?
What we did
We designed a coding task that anyone can do and that can be scored objectively: recreate a target picture (for example, a robin) as SVG code, a text format that draws images.
Each iteration has three roles:
- An instructor looks at the target and the current best drawing and writes a short instruction, such as “make the tail longer.”
- A code generator (an LLM) writes new SVG code from that instruction.
- A selector picks the best of the candidate drawings, which becomes the starting point for the next iteration.
A chain runs for 15 iterations. Independent raters score how closely each drawing matches the target. Across 20 experiments with 737 human participants, we varied whether a human or an AI filled each role.

What we found
1. Human-led coding improves; AI-led coding collapses
In the first iterations, human-led and AI-led chains scored about the same. By the final iteration, human-led chains were far ahead (difference = 1.01 rating points, Cohen’s d = 1.49). AI-led chains (with GPT-5 as instructor and selector) actually got worse over time. Switching to Claude or Gemini didn’t help, and neither did letting the AI see the code, the image, or both. AI-led chains still failed to improve.

2. People give short, general edits; AI over-describes
Human instructions averaged about 18 words, such as “remove the tail and make the body bigger.” AI instructions averaged about 755 words of detailed description. Human instructions also drew on a shared vocabulary across very different targets: general editing moves that work almost anywhere. Even when AI instructions were capped at 10–30 words, they stayed overly specific to the target and did not catch up.

3. Let humans steer and let AI judge
Swapping in an AI selector while keeping a human instructor performed about as well as an all-human team. The reverse did not work: an AI instructor needed a human selector to avoid collapse. In chains where humans and AI took turns, scores fell steadily as the AI’s share grew. The human contribution that matters most is the high-level direction.

My take
Links
BibTeX
@article{hu2026vibecoding,
title = {Why Human Guidance Matters in Collaborative Vibe Coding},
author = {Hu, Haoyu and Marjieh, Raja and Collins, Katherine M. and Li, Chenyi and
Griffiths, Thomas L. and Sucholutsky, Ilia and Jacoby, Nori},
journal = {arXiv preprint arXiv:2602.10473},
year = {2026},
doi = {10.48550/arXiv.2602.10473}
}