Page MenuHomeSoftware Heritage

Survey of continuous integration systems
Closed, ResolvedPublic

Description

We need to make an assessment of the existing continuous integration tools, and the amount of integration we can get with our current setup.

References : https://secure.phabricator.com/T9456 (Upstream phabricator integration with external build tools)

For this evaluation, I have focused on self-hosted tools as they are the most flexible and the most easily trustable (allowing us to do things like package builds etc. within our infra).

Here are the contenders for this evaluation:

  • Harbormaster (Phabricator built-in CI tool)
  • Jenkins
  • buildbot
  • Drone CI (https://drone.io/)
  • GitLab CI

Here's a comparison table on the solutions

CriterionPlain HarbormasterJenkinsBuildbotDrone CIGitLab CI
Installation (master)Built into phabricatorUpstream Debian packages; Docker ImagesDebian packagesDocker Images (only option supported upstream)Built into GitLab (Upstream Debian packages available)
Installation (workers)Manual (and fairly rough)Jenkins agents (Jenkins can also execute tasks on the master)Debian-packaged buildbot worker daemonUse the master host itself as a docker providerDebian packages for the runner, uses docker in backend
Phabricator integrationBuilt-inWith third party plugin (supports build on diffs, maybe on revisions?)Could not find anyone who did it. webhooks or polling ?Could not find anyone who did it. webhooks ?Only integrated with GitLab
Community supportlowhigh (lots of installs of all sizes)med-high (lots of high profile, heavily custom installs)low-ishmed-high (lots of small installs)
Configuration format Bring your own scripts - Actual jobs setup in phabricator interface Default: database in the CI master; Can do declarative configuration in-repository Jenkinsfile; Can do declarative configuration generation out-of-repository with jenkins-job-builderDeclared as python scriptsIn-repository yaml fileIn-repository yaml file
Debian package building integration Bring your own scripts https://jenkins-debian-glue.org/ used by grml, apt.postgresql.org, ...i3 does it: https://i3wm.org/docs/buildbot.htmlBring your own scriptsBring your own scripts; Debian uses GitLab for its development now and some people probably have the proper glue setup already
CriterionPlain HarbormasterJenkinsBuildbotDrone CIGitLab CI

Event Timeline

olasd created this task.Apr 13 2018, 11:08 AM
olasd changed the task status from Open to Work in Progress.
olasd triaged this task as High priority.
olasd closed this task as Resolved.Oct 23 2018, 11:25 AM

I think we went with Jenkins + jenkins-job-builder at this point...