Page MenuHomeSoftware Heritage

bin/install: install packages one by one and enforce prerequisites
ClosedPublic

Authored by zack on Feb 26 2021, 2:26 PM.

Details

Summary

This is trying to make bin/install (again) a click-button developer setup
installation tool, simplifying the process for newcomers and making it faster
for everyone.

To that end:

  • we ensure that pip is up to date before using it for anything else
  • we install the prerequisites (flake8, mypy, etc.) to avoid having to name

them one by one in user documentation

  • we install packages one by one, rather them all together, to avoid that pip

failures force users to restart the whole process from scratch. In passing,
this makes also things much easier for the new pip resolver. To avoid
dependency inconsistencies we install packages in bin/pip-swh-packages, which
is already supposed to be topologically sorted

  • we also install packages only *once*, rather than first without -with-testing

and later with it (as currently documented in the developer setup), because the
"[testing]" markers are extra requires, which always depend on the base
package (hopefully)

When this gets landed, the developer setup can be shortened to just say "run
bin/install".

bin/pip-swh-packages: remove [extras] markers before topo-sorting

Diff Detail

Repository
rDENV Development environment
Branch
feature/dev-setup-pip-improvements
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 19518
Build 30279: arc lint + arc unit

Event Timeline

zack requested review of this revision.Feb 26 2021, 2:26 PM
ardumont added inline comments.
bin/install
5

pifpaf can go away now (can be cleaned up later though)

Yes, that sounds more reasonable now.

Thanks.

Wondering if that should go into the swh-environment build which seems to try to do something similar.

Aside that this build frequently fails due to some repository being red often) that migth help in detecting misdeclared dependency.

[1] https://jenkins.softwareheritage.org/job/DENV/job/tests/configure

This revision is now accepted and ready to land.Feb 26 2021, 2:42 PM
bin/install
14

I'm also wondering whether we should add checks on runtime dependencies (the ones that could not be dealt with pip and which are referenced in the docs [1]).

A simple check and warning if missing.

[1] https://docs.softwareheritage.org/devel/developer-setup.html#install-required-dependencies

bin/install
14

Not a bad idea, but they are all system dependencies, I have no idea if there is a reasonable way to check for their presence without having to discriminate debian/redhat/etc. (which clearly we do not want to do).

I've took care of pre-commit though, but I think it's needed at bin/update time (which happens before), so it's not really helping either.