We need to make an assessment of the existing continuous integration tools, and the amount of integration we can get with our current setup.
References : https://secure.phabricator.com/T9456 (Upstream phabricator integration with external build tools)
For this evaluation, I have focused on self-hosted tools as they are the most flexible and the most easily trustable (allowing us to do things like package builds etc. within our infra).
Here are the contenders for this evaluation:
- Harbormaster (Phabricator built-in CI tool)
- Jenkins
- buildbot
- Drone CI (https://drone.io/)
- GitLab CI
Here's a comparison table on the solutions
| Criterion | Plain Harbormaster | Jenkins | Buildbot | Drone CI | GitLab CI |
|---|---|---|---|---|---|
| Installation (master) | Built into phabricator | Upstream Debian packages; Docker Images | Debian packages | Docker Images (only option supported upstream) | Built into GitLab (Upstream Debian packages available) |
| Installation (workers) | Manual (and fairly rough) | Jenkins agents (Jenkins can also execute tasks on the master) | Debian-packaged buildbot worker daemon | Use the master host itself as a docker provider | Debian packages for the runner, uses docker in backend |
| Phabricator integration | Built-in | With third party plugin (supports build on diffs, maybe on revisions?) | Could not find anyone who did it. webhooks or polling ? | Could not find anyone who did it. webhooks ? | Only integrated with GitLab |
| Community support | low | high (lots of installs of all sizes) | med-high (lots of high profile, heavily custom installs) | low-ish | med-high (lots of small installs) |
| Configuration format | Bring your own scripts - Actual jobs setup in phabricator interface | Default: database in the CI master; Can do declarative configuration in-repository Jenkinsfile; Can do declarative configuration generation out-of-repository with jenkins-job-builder | Declared as python scripts | In-repository yaml file | In-repository yaml file |
| Debian package building integration | Bring your own scripts | https://jenkins-debian-glue.org/ used by grml, apt.postgresql.org, ... | i3 does it: https://i3wm.org/docs/buildbot.html | Bring your own scripts | Bring your own scripts; Debian uses GitLab for its development now and some people probably have the proper glue setup already |
| Criterion | Plain Harbormaster | Jenkins | Buildbot | Drone CI | GitLab CI |