Page MenuHomeSoftware Heritage

pip install swh.model[cli] no longer provides a usable "swh" command
Closed, MigratedEdits Locked

Description

As per subject. Here's my test case:

$ mkvirtualenv -p `which python3` test
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/zack/.virtualenvs/test/bin/python3
Also creating executable in /home/zack/.virtualenvs/test/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/zack/.virtualenvs/test/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/zack/.virtualenvs/test/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/zack/.virtualenvs/test/bin/preactivate
virtualenvwrapper.user_scripts creating /home/zack/.virtualenvs/test/bin/postactivate
virtualenvwrapper.user_scripts creating /home/zack/.virtualenvs/test/bin/get_env_details
$ pip install 'swh.model[cli]'
Collecting swh.model[cli]
  Using cached swh.model-0.0.56-py3-none-any.whl (93 kB)
Collecting python-dateutil
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Processing ./.cache/pip/wheels/33/84/29/2fff179e8de0b5e0d17ee4a570536f547b809d62ada8d6d861/vcversioner-2.16.0.0-cp37-none-any.whl
Collecting hypothesis
  Using cached hypothesis-5.5.4-py3-none-any.whl (288 kB)
Collecting attrs
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting dulwich; extra == "cli"
  Using cached dulwich-0.19.15-py3-none-any.whl (419 kB)
Collecting Click; extra == "cli"
  Using cached Click-7.0-py2.py3-none-any.whl (81 kB)
Collecting six>=1.5
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting sortedcontainers<3.0.0,>=2.1.0
  Using cached sortedcontainers-2.1.0-py2.py3-none-any.whl (28 kB)
Collecting urllib3>=1.24.1
  Using cached urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Collecting certifi
  Using cached certifi-2019.11.28-py2.py3-none-any.whl (156 kB)
Installing collected packages: six, python-dateutil, vcversioner, sortedcontainers, attrs, hypothesis, urllib3, certifi, dulwich, Click, swh.model
Successfully installed Click-7.0 attrs-19.3.0 certifi-2019.11.28 dulwich-0.19.15 hypothesis-5.5.4 python-dateutil-2.8.1 six-1.14.0 sortedcontainers-2.1.0 swh.model-0.0.56 urllib3-1.25.8 vcversioner-2.16.0.0
$ swh identify foo.txt 
zsh: command not found: swh
$ swh-identify foo.txt
Traceback (most recent call last):
  File "/home/zack/.local/bin/swh-identify", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3251, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'swh.model' distribution was not found and is required by the application
$ 

Additional notes:

  • there are at least two separate issues here: (1) swh identify backtrace, and (2) swh not being installed (or in the PATH?)
  • the cached swh.model release in the above log (0.0.56) is the most recent on pypi at the time of writing
  • I've also tried pip install swh.model, without [cli], obtaining the same result
  • pip install is the intended way to compute PID locally, so this stuff should work reliably. It would be nice to have periodic automated testing for this

Event Timeline

zack triaged this task as High priority.Feb 18 2020, 9:43 AM
zack created this task.
zack updated the task description. (Show Details)
zack updated the task description. (Show Details)

Your swh-identify backtrace looks like a $PATH caching issue (see also the path of the swh-identify script in the first line). I can't reproduce it in a fresh venv and swh-identify works fine.

As for the swh command not being installed: the swh.core dependency in swh.model has been dropped in v0.0.38, released in June. The actual commit introducing the change, rDMODb3250d29742514, was done to work around the longstanding pip dependency resolution bug when you end up depending on different sets of extras for a given package.

I think re-introducing the swh.core dependency in swh.model[cli] should be fine, as none of our other packages actually depend on that. I'm submitting a diff to that effect.

zack lowered the priority of this task from High to Normal.Feb 18 2020, 12:55 PM
In T2288#42054, @olasd wrote:

Your swh-identify backtrace looks like a $PATH caching issue (see also the path of the swh-identify script in the first line). I can't reproduce it in a fresh venv and swh-identify works fine.

Ah, thank you! After a rehash indeed swh-identify works again, I'm hence retitling and lowering the priority of this issue, focusing on the second point:

As for the swh command not being installed: the swh.core dependency in swh.model has been dropped in v0.0.38, released in June. The actual commit introducing the change, rDMODb3250d29742514, was done to work around the longstanding pip dependency resolution bug when you end up depending on different sets of extras for a given package.

zack renamed this task from pip install swh.model[cli] no longer provides a usable "swh identify" to pip install swh.model[cli] no longer provides a usable "swh" command.Feb 18 2020, 12:55 PM