Prow: Testing the way to Kubernetes Next


December 26th, 2017

Prow - extended nautical metaphor. Go Gopher originally by Renee French, SVG version by Takuya Ueda, modified under the CC BY 3.0 license. Ship's wheel from Kubernetes logo by Tim Hockin.

The Kubernetes project does a lot of testing, on the order of 10000 jobs per day covering everything from build and unit tests, to end-to-end testing on real clusters deployed from source all the way up to ~5000 node scalability and performance tests.

a grafana dashboard screenshot showing over 10,000 jobs / day, with around an 80% pass-rate

Velodrome job metrics

The system handling all of this leverages Kubernetes, naturally, and of-course has a number of nautically-named components. This system is Prow, and is used to manage automatic validation and merging of human-approved pull requests and to verify branch-health leading up to each release.

With Prow each job is a single-container pod, created in a dedicated build and test cluster by “plank”, a micro-service running in the services cluster. Each Prow component (roughly outlined above, along with TestGrid) is a small Go service structured around managing these one-off single-pod “ProwJobs”.

Using Kubernetes frees us from worrying about most of the resource management and scheduling / bin-packing of these jobs once they have been created and has generally been a pleasant experience.

Prow / “hook” also provides a number of GitHub automation plugins< used to provide things like issue and pull request slash commands for applying and removing labels, opening and closing issues, etc. This has been particularly helpful since GitHub’s permissions model is not particularly granular and we’d like contributors to be able to label issues without write permissions. πŸ™ƒ

If any of this sounds interesting to you come check out Prow’s source code and join our SIG Testing meetings for more.


✎ Update β€’ August 2018
Work on open-sourcing TestGrid still ongoing. TestGrid must be ported off of many Google internal libraries and is primarily staffed by one engineer currently, who must also maintain it, so this is taking longer than we hoped. We still intend to open source TestGrid and are making progress, slowly.
✎ Update β€’ August 2019
The TestGrid repo is now created at github.com/GoogleCloudPlatform/testgrid, Michelle received approval and has started migrating the code. πŸŽ‰