Page MenuHomeSoftware Heritage
Create Task
Maniphest T3484

Fix the release builds for swh-search
Closed, MigratedEdits Locked
Actions

Description

The build is failing due to a missing module during the yarn build

pypi build:

17:33:59  + yarn build
17:33:59  yarn run v1.22.5
17:33:59  $ yarn build-so && yarn build-wasm
17:33:59  $ yarn generate && cd query_language && python3 build.py
17:33:59  $ cd query_language && tree-sitter generate --no-bindings && echo 'Generated parser files '
17:34:00  Generated parser files 
17:34:00  Traceback (most recent call last):
17:34:00    File "build.py", line 1, in <module>
17:34:00      from tree_sitter import Language
17:34:00  ModuleNotFoundError: No module named 'tree_sitter'
17:34:00  error Command failed with exit code 1.

Debian build, 2 issues:

16:42:50  running ts_build_export
16:42:50  running ts_install
16:42:50  sh: 1: yarn: not found
16:42:50  running ts_build
16:42:50  running ts_build_so
16:42:50  sh: 1: yarn: not found
16:42:50  running ts_build_wasm
16:42:50  sh: 1: yarn: not found
16:42:50  static files generated. copying them to package dir
16:42:50  cp: cannot stat 'query_language/swh_ql.so': No such file or directory
16:42:50  cp: cannot stat 'query_language/swh_ql.wasm': No such file or directory
16:42:51  ============================= test session starts ==============================
16:42:51  platform linux -- Python 3.9.2, pytest-6.0.2, py-1.10.0, pluggy-0.13.0
16:42:51  rootdir: /<<PKGBUILDDIR>>, configfile: pytest.ini
16:42:51  plugins: swh.journal-0.8.0, swh.core-0.14.4, mock-1.10.4, hypothesis-5.43.3
16:42:51  collected 131 items / 2 errors / 129 selected
16:42:51  Hint: make sure your test modules/packages have valid Python names.
16:42:51  Traceback:
16:42:51  /usr/lib/python3.9/importlib/__init__.py:127: in import_module
16:42:51      return _bootstrap._gcd_import(name[level:], package, level)
16:42:51  swh/search/tests/test_translator.py:3: in <module>
16:42:51      from swh.search.translator import Translator
16:42:51  swh/search/translator.py:4: in <module>
16:42:51      from tree_sitter import Language, Parser
16:42:51  E   ModuleNotFoundError: No module named 'tree_sitter'
16:42:51  =============================== warnings summary ===============================

Revisions and Commits

rCDFJ Dockerfiles for Jenkins
D6085rCDFJ2fea9ce49664 Install a missing python module for the swh-search build
rDSEA Archive search
D6086rDSEA84115fa41877 Document the dependency on the tree-sitter python module

Related Objects
Search...

Event Timeline

vsellier changed the task status from Open to Work in Progress.Aug 13 2021, 10:35 AM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier moved this task from Backlog to in-progress on the System administration board.
vsellier added a revision: D6085: Install a missing python module for the swh-search build.Aug 13 2021, 4:14 PM
vsellier added a revision: D6086: Document the dependency on the tree-sitter python module.Aug 13 2021, 4:18 PM
vsellier added a commit: rDSEA84115fa41877: Document the dependency on the tree-sitter python module.Aug 13 2021, 4:30 PM
vsellier added a commit: rCDFJ2fea9ce49664: Install a missing python module for the swh-search build.
ardumont added a comment.Aug 13 2021, 4:45 PM

So that unstuck the pypi part ;)
Now on with unstucking the debian build [1]

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DSEA/job/gbp-buildpackage/56/console

vlorentz added a subscriber: vlorentz.Aug 13 2021, 5:24 PM

D6088 should help

vlorentz added a comment.EditedAug 13 2021, 5:25 PM

you also need to add yarn as a build-dep for debian packages

vsellier renamed this task from Fix the pypi-upload build for swh-search to Fix the release builds for swh-search.Aug 16 2021, 2:54 PM
vsellier updated the task description. (Show Details)
vsellier added a comment.Aug 17 2021, 5:57 PM

The pypi build is still working well with the 2 last diff.
Now there is a new error during the debian ones:

dh: warning: Compatibility levels before 10 are deprecated (level 9 in use)
   dh_auto_clean -O--buildsystem=pybuild
dh_auto_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use)
I: pybuild base:232: python3.9 setup.py clean 
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101bcd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101beb0>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b850>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b730>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b610>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
ERROR: Could not find a version that satisfies the requirement tree-sitter==0.19.0
ERROR: No matching distribution found for tree-sitter==0.19.0
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/setuptools/installer.py", line 75, in fetch_build_egg
    subprocess.check_call(cmd)
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3.9', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpdrbws3hq', '--quiet', 'tree-sitter==0.19.0']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/setup.py", line 157, in <module>
    setup(
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 152, in setup
    _install_setup_requires(attrs)
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 147, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 686, in fetch_build_eggs
    resolved_dists = pkg_resources.working_set.resolve(
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 766, in resolve
    dist = best[req.key] = env.best_match(
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1051, in best_match
    return self.obtain(req, installer)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1063, in obtain
    return installer(requirement)
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 745, in fetch_build_egg
    return fetch_build_egg(self, req)
  File "/usr/lib/python3/dist-packages/setuptools/installer.py", line 77, in fetch_build_egg
    raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/usr/bin/python3.9', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpdrbws3hq', '--quiet', 'tree-sitter==0.19.0']' returned non-zero exit status 1.
E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean 
dh_auto_clean: error: pybuild --clean --test-pytest -i python{version} -p 3.9 returned exit code 13
make: *** [debian/rules:8: clean] Error 25
dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2
vsellier added a comment.Aug 17 2021, 6:13 PM

https://wiki.debian.org/Python/LibraryStyleGuide#Build-Depends

One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based packages have the nasty habit of downloading dependencies from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends > is up-to-date), then distribute will not try to download the package, otherwise it will try to download it. This is a huge no-no, and pybuild internally sets the http_proxy and https_proxy environment variables (to 127.0.0.1:9) to prevent this from happening.

well it seems there will no choice but to prepare a debian package for tree-sitter :(

vsellier mentioned this in rCJSWH9a81390eda36: Declare debian package build for tree-sitter.Aug 18 2021, 4:11 PM
vsellier closed this task as Resolved.Sep 1 2021, 3:21 PM
vsellier claimed this task.
vsellier moved this task from in-progress to done on the System administration board.

The build is now fixed and the v0.11.4 version is ready to be deployed on the environments

gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:03 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal