The state of preview deployments
Nine months ago we started counting what happens to a preview deployment after it is created. Not just whether somebody clicked it, but who, how long they stayed, and whether a comment they left changed the code that shipped.
The numbers
2.3 million preview URLs created
61% opened by someone other than the author
18% received at least one comment
Comments on a preview led to a code change 4.2× more often than comments on the diff alone
Reviewers are not who you think
The largest group of people opening previews were not engineers. Designers, product managers and support leads together opened more previews than the authors' own teammates.
The preview link turned out to be the most-read artefact in the pull request, and the least-written-about.
What we're changing
We are moving preview comments next to the code they refer to, adding visual diffs between the preview and production, and making it possible to require a sign-off from a named reviewer before merge.
Visual diffs
Every preview now records a screenshot of each changed route. When the preview is rebuilt we compare, and highlight what moved.
Required sign-offs
A repository can name a reviewer group whose approval on the preview is required. It shows up as a status check, like a test.