Community Leadership Summit Wiki

The OpenStack Continuous Integration Process and How It Shapes the Community

178pages on
this wiki

Mark Atwood and Elizabeth Krumbach

OpenStack has 500-700 contributors per 30 day window, 200 contributing companies, 170  commits per day

every commit is fully tested in a 2.5 hour cycle

this process formed in response to earlier anti-patterns eg Java

how it works:

check subproj out of git

make your change


(CLA must be signed somewhere in here, or you don't get an SSH key)

pushes the contribution to a special tree managed by robots, smoke tests are run automatically, results sent up to the metadata

Gerrit is a code review system bought by Google and rewritten in Java, works well with Git - event stream: ssh to a port, send JSON commands

(OpenStack's version may become a fork as Google is not taking their patches.)

anyone can do a +1 or -1 code review for tech nature, comments, git commit message well formatted? "no assholes" rule applies although nitpicking is allowed

first level reviewers eventually get social cred and may be recommended by the core reviewers - based on merit, not employer

one core reviewer (preferably 2) for each project give a +2, then the change moves into a more extensive test system (2.5 hours)

there are no bypasses, process is the same for all

"everybody and nobody" has the right to commit

tip always works - you can always check out a release and it will work

merges happen all the time 

git with gitreview plug-in

Zuul is the gatekeeper

with multiple patches, Zuul spins up open compute instances and test trunk + A, trunk + A + B, etc.

if all works, all get committed

if not, continue combination testing until successful combinations found - those are passed

the whole system is another OpenStack project, OpenStack Infra, which can also be changed via the same process

how many core reviewers? depends on the project - 12-18 in some

OpenStack is a loosely-coupled set of components speaking to each other via REST

very decentralized - has to be, due to the membership of the large OpenStack project

talking to MediaWiki and KDE about using this system to review

this system leads to continuous deployment

where do the tests come from?

added as part of dev - every feature includes test cases

there is a QA team that keeps tests under control so the gate process doesn't become too cumbersome

    • this is documented at **

use a work in progress tag for intermediate review, non-final code

there is a technical committee that decides eg what we're going to test on, therefore what will be in the gate

onboarding mechanisms? one recently had a hackathon / trainathon for the CI system

other projects do their own onboarding

the automation makes minor initial contribs easy so people can onboard themselves

there is a practice system that resets every 24 hours

"how to get involved" page on the wiki links to the Garrett page

you need to know git, and do gitreview rather than git pull to submit it to Garrett

newbies don't need to be aware of what's going on behind the scenes

if there's a problem, system logs will be sent

there are dashboards to see status, most changes are accepted within hours, also sends status emails for each step

LaunchPad for bugs, ID, and blueprints

much of this is lifted from the Ubuntu process

docs maintained as part of source code in git means you can reject a push which does not include docs - docs are done via git

if something is not touched in 7 days after a -1, it is marked as "abandoned", but can be resurrected if needed - this is just to keep it out of review queues if nothing is happening on it

review day every 2 weeks - see - to clean up review lists

OpenStack uses github as a public mirror - pull requests not accepted directly from github

Monty Taylor architected this

"We have contributors from the NSA."

level 1 reviewers are not necessarily core contributors; level 2 reviewers typically are

OpenStack has many contributors who are paid to contribute as their day jobs

engagement of corp QA not so great, but they do contribute to Tempest which is a QA project

(UTest as a crowd source for testing - check it out)

How to set up your own? see the runbook - would take a skilled sysadmin half day of work BUT it does assume that you're running on an OpenStack system, could be difficult if you're not

OpenStack installation is not easy at all


data localization is a growing issue with recent US NSA revelations -> local clouds will be popular

there is not a public OpenStack cloud in every jurisdiction yet, but there is growing impetus for it

  1. openstackinfra on freenode

parallelizable tests preferred, but WIP

assuming that computation is infinite and free

it does break sometimes, ie because of a new dependency

Around Wikia's network

Random Wiki