bin/install: install packages one by one and enforce prerequisites
ClosedPublic
Actions

Authored by zack on Feb 26 2021, 2:26 PM.

Details

Reviewers

Group Reviewers

Commits

rDENV4f3600624565: bin/pip-swh-packages: remove [extras] markers before topo-sorting
rDENVe6fbb0f3bccf: bin/install: install packages one by one and enforce prerequisites

Summary

This is trying to make bin/install (again) a click-button developer setup
installation tool, simplifying the process for newcomers and making it faster
for everyone.

To that end:

we ensure that pip is up to date before using it for anything else

we install the prerequisites (flake8, mypy, etc.) to avoid having to name

them one by one in user documentation

we install packages one by one, rather them all together, to avoid that pip

failures force users to restart the whole process from scratch. In passing,
this makes also things much easier for the new pip resolver. To avoid
dependency inconsistencies we install packages in bin/pip-swh-packages, which
is already supposed to be topologically sorted

we also install packages only *once*, rather than first without -with-testing

and later with it (as currently documented in the developer setup), because the
"[testing]" markers are extra requires, which always depend on the base
package (hopefully)

When this gets landed, the developer setup can be shortened to just say "run
bin/install".

bin/pip-swh-packages: remove [extras] markers before topo-sorting

Diff Detail

Repository

rDENV Development environment

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

zack created this revision.Feb 26 2021, 2:26 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 26 2021, 2:26 PM

zack requested review of this revision.Feb 26 2021, 2:26 PM

Harbormaster completed remote builds in B19518: Diff 18439.Feb 26 2021, 2:26 PM

zack added a child revision: D5155: developer setup: simplify package installation by delegating to bin/install.Feb 26 2021, 2:27 PM

ardumont added a subscriber: ardumont.Feb 26 2021, 2:33 PM

ardumont added inline comments.

bin/install
4	pifpaf can go away now (can be cleaned up later though)

Yes, that sounds more reasonable now.

Thanks.

Wondering if that should go into the swh-environment build which seems to try to do something similar.

Aside that this build frequently fails due to some repository being red often) that migth help in detecting misdeclared dependency.

[1] https://jenkins.softwareheritage.org/job/DENV/job/tests/configure

This revision is now accepted and ready to land.Feb 26 2021, 2:42 PM

ardumont added inline comments.Feb 26 2021, 2:47 PM

bin/install
13	I'm also wondering whether we should add checks on runtime dependencies (the ones that could not be dealt with pip and which are referenced in the docs [1]). A simple check and warning if missing. [1] https://docs.softwareheritage.org/devel/developer-setup.html#install-required-dependencies

zack added inline comments.Feb 26 2021, 2:49 PM

bin/install
13	Not a bad idea, but they are all system dependencies, I have no idea if there is a reasonable way to check for their presence without having to discriminate debian/redhat/etc. (which clearly we do not want to do). I've took care of pre-commit though, but I think it's needed at bin/update time (which happens before), so it's not really helping either.