Page MenuHomeSoftware Heritage

bin/install: install packages one by one and enforce prerequisites

Authored by zack on Feb 26 2021, 2:26 PM.



This is trying to make bin/install (again) a click-button developer setup
installation tool, simplifying the process for newcomers and making it faster
for everyone.

To that end:

  • we ensure that pip is up to date before using it for anything else
  • we install the prerequisites (flake8, mypy, etc.) to avoid having to name

them one by one in user documentation

  • we install packages one by one, rather them all together, to avoid that pip

failures force users to restart the whole process from scratch. In passing,
this makes also things much easier for the new pip resolver. To avoid
dependency inconsistencies we install packages in bin/pip-swh-packages, which
is already supposed to be topologically sorted

  • we also install packages only *once*, rather than first without -with-testing

and later with it (as currently documented in the developer setup), because the
"[testing]" markers are extra requires, which always depend on the base
package (hopefully)

When this gets landed, the developer setup can be shortened to just say "run

bin/pip-swh-packages: remove [extras] markers before topo-sorting

Diff Detail

rDENV Development environment
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

zack requested review of this revision.Feb 26 2021, 2:26 PM
ardumont added inline comments.

pifpaf can go away now (can be cleaned up later though)

Yes, that sounds more reasonable now.


Wondering if that should go into the swh-environment build which seems to try to do something similar.

Aside that this build frequently fails due to some repository being red often) that migth help in detecting misdeclared dependency.


This revision is now accepted and ready to land.Feb 26 2021, 2:42 PM

I'm also wondering whether we should add checks on runtime dependencies (the ones that could not be dealt with pip and which are referenced in the docs [1]).

A simple check and warning if missing.



Not a bad idea, but they are all system dependencies, I have no idea if there is a reasonable way to check for their presence without having to discriminate debian/redhat/etc. (which clearly we do not want to do).

I've took care of pre-commit though, but I think it's needed at bin/update time (which happens before), so it's not really helping either.