TL;DR
When groups take turns “vibe coding” a picture by telling an AI what to change, human instructions keep improving the result, while AI instructions make it collapse. The best hybrid lets humans give the instructions and leaves judging the results to AI.
Example progressions led by humans vs. by AI (top), scores across iterations (bottom left), and what happens when the roles are split between humans and AI (bottom right). Figures from the paper.
Example progressions led by humans vs. by AI (top), scores across iterations (bottom left), and what happens when the roles are split between humans and AI (bottom right). Figures from the paper.

The question

“Vibe coding” means building software by describing what you want in plain language and letting an AI write the code. It is spreading fast. So are fully automated pipelines in which AI agents instruct other AI agents. Two questions follow. Is there still a unique benefit to having a human in the loop? And if so, which job should the human do?

What we did

We designed a coding task that anyone can do and that can be scored objectively: recreate a target picture (for example, a robin) as SVG code, a text format that draws images.

Each iteration has three roles:

  • An instructor looks at the target and the current best drawing and writes a short instruction, such as “make the tail longer.”
  • A code generator (an LLM) writes new SVG code from that instruction.
  • A selector picks the best of the candidate drawings, which becomes the starting point for the next iteration.

A chain runs for 15 iterations. Independent raters score how closely each drawing matches the target. Across 20 experiments with 737 human participants, we varied whether a human or an AI filled each role.

The vibe-coding loop: an instructor writes natural-language edits, an AI turns them into SVG code, and a selector keeps the best result for the next round.
The vibe-coding loop: an instructor writes natural-language edits, an AI turns them into SVG code, and a selector keeps the best result for the next round.

What we found

1. Human-led coding improves; AI-led coding collapses

In the first iterations, human-led and AI-led chains scored about the same. By the final iteration, human-led chains were far ahead (difference = 1.01 rating points, Cohen’s d = 1.49). AI-led chains (with GPT-5 as instructor and selector) actually got worse over time. Switching to Claude or Gemini didn’t help, and neither did letting the AI see the code, the image, or both. AI-led chains still failed to improve.

Similarity to the target across iterations. Human-led chains (red) improve; AI-led chains (blue) decline.
Similarity to the target across iterations. Human-led chains (red) improve; AI-led chains (blue) decline.

2. People give short, general edits; AI over-describes

Human instructions averaged about 18 words, such as “remove the tail and make the body bigger.” AI instructions averaged about 755 words of detailed description. Human instructions also drew on a shared vocabulary across very different targets: general editing moves that work almost anywhere. Even when AI instructions were capped at 10–30 words, they stayed overly specific to the target and did not catch up.

(A) Example human vs. AI instructions. (B) Human instructions cluster together across targets; AI instructions spread apart. (C) Capping AI instruction length doesn't close the gap. (D) Semantic profile of the two instruction styles.
(A) Example human vs. AI instructions. (B) Human instructions cluster together across targets; AI instructions spread apart. (C) Capping AI instruction length doesn’t close the gap. (D) Semantic profile of the two instruction styles.

3. Let humans steer and let AI judge

Swapping in an AI selector while keeping a human instructor performed about as well as an all-human team. The reverse did not work: an AI instructor needed a human selector to avoid collapse. In chains where humans and AI took turns, scores fell steadily as the AI’s share grew. The human contribution that matters most is the high-level direction.

(A) Removing the selection step hurts human-led chains but not AI-led ones. (B) A human instructor with an AI selector does as well as an all-human team.
(A) Removing the selection step hurts human-led chains but not AI-led ones. (B) A human instructor with an AI selector does as well as an all-human team.

My take

BibTeX
@article{hu2026vibecoding,
  title   = {Why Human Guidance Matters in Collaborative Vibe Coding},
  author  = {Hu, Haoyu and Marjieh, Raja and Collins, Katherine M. and Li, Chenyi and
             Griffiths, Thomas L. and Sucholutsky, Ilia and Jacoby, Nori},
  journal = {arXiv preprint arXiv:2602.10473},
  year    = {2026},
  doi     = {10.48550/arXiv.2602.10473}
}