Page MenuHomeSoftware Heritage

Remove .wasm from all distributions, and .so from source distributions
ClosedPublic

Authored by vlorentz on Aug 25 2021, 11:38 AM.

Details

Summary

.so is still built, but only for binary distributions (it can be built
from a source distribution).

To do this, we now include intermediary files (parser.c in particular)
in the source distribution.
They are not really source file, but this allows removing the dependency
on a nodejs runtime to install from pypi.

Diff Detail

Repository
rDSEA Archive search
Branch
dist
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 23143
Build 36098: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 36097: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D6131 (id=22183)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit 38cd164fa371a6a2edcbc7bdf47345d75ab4ac99
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this will allow (in a future commit)
    to remove the dependency on a nodejs runtime to install from pypi.

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/272/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/272/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 25 2021, 11:39 AM
Harbormaster failed remote builds in B23141: Diff 22183!

Build has FAILED

Patch application report for D6131 (id=22184)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit b47019c5c620d6e8fff676ea3e644304f0faa1f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/273/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/273/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 25 2021, 11:39 AM
Harbormaster failed remote builds in B23142: Diff 22184!

Build is green

Patch application report for D6131 (id=22185)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit bc087727deaa405e05f48e52104e75214416973a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:42:28 2021 +0200

    Fix jenkins build?

commit b47019c5c620d6e8fff676ea3e644304f0faa1f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/274/ for more details.

Build has FAILED

Patch application report for D6131 (id=22186)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit 0b81fbc932bc9444248a504826c43e2967aede51
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:42:28 2021 +0200

    Fix jenkins build?

commit b47019c5c620d6e8fff676ea3e644304f0faa1f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/275/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/275/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 25 2021, 11:47 AM
Harbormaster failed remote builds in B23144: Diff 22186!

Build has FAILED

Patch application report for D6131 (id=22187)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit ea382f9e5429b66e32682ff3a97d7eb2a9e5c05d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/276/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/276/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 25 2021, 11:51 AM
Harbormaster failed remote builds in B23145: Diff 22187!

looks like setuptools sometimes copies it itself? anyway, override it

Build has FAILED

Patch application report for D6131 (id=22188)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit fff010523772a97372eae7f62e1fffa29b2045b4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/277/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/277/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 25 2021, 11:54 AM
Harbormaster failed remote builds in B23146: Diff 22188!

Build has FAILED

Patch application report for D6131 (id=22189)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit 91422ad3d4e2d86b55e6eb2a90010d2363b597ae
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/278/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/278/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 25 2021, 11:57 AM
Harbormaster failed remote builds in B23147: Diff 22189!

should have tested before committing...

Build is green

Patch application report for D6131 (id=22190)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit e24beeb72047534dd87a3638618c26b4c9edf3f0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/279/ for more details.

I still got an error when packaging swh-search, see below:

(swh) ✔ ~/swh/swh-environment/swh-search [arcpatch-D6131 L|…1⚑ 3] 
12:13 $ python3 setup.py sdist bdist_wheel
running sdist
running egg_info
writing swh.search.egg-info/PKG-INFO
writing dependency_links to swh.search.egg-info/dependency_links.txt
writing entry points to swh.search.egg-info/entry_points.txt
writing requirements to swh.search.egg-info/requires.txt
writing top-level names to swh.search.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
warning: no files found matching 'version.txt'
adding license file 'LICENSE'
adding license file 'AUTHORS'
writing manifest file 'swh.search.egg-info/SOURCES.txt'
running check
creating swh.search-0.11.3.dev2+g7f5026d
creating swh.search-0.11.3.dev2+g7f5026d/docs
creating swh.search-0.11.3.dev2+g7f5026d/docs/_static
creating swh.search-0.11.3.dev2+g7f5026d/docs/_templates
creating swh.search-0.11.3.dev2+g7f5026d/es_config
creating swh.search-0.11.3.dev2+g7f5026d/swh
creating swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
creating swh.search-0.11.3.dev2+g7f5026d/swh/search
creating swh.search-0.11.3.dev2+g7f5026d/swh/search/api
creating swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language
creating swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language/test
creating swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language/test/corpus
creating swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying files to swh.search-0.11.3.dev2+g7f5026d...
copying .gitignore -> swh.search-0.11.3.dev2+g7f5026d
copying .pre-commit-config.yaml -> swh.search-0.11.3.dev2+g7f5026d
copying AUTHORS -> swh.search-0.11.3.dev2+g7f5026d
copying CODE_OF_CONDUCT.md -> swh.search-0.11.3.dev2+g7f5026d
copying CONTRIBUTORS -> swh.search-0.11.3.dev2+g7f5026d
copying LICENSE -> swh.search-0.11.3.dev2+g7f5026d
copying MANIFEST.in -> swh.search-0.11.3.dev2+g7f5026d
copying Makefile -> swh.search-0.11.3.dev2+g7f5026d
copying Makefile.local -> swh.search-0.11.3.dev2+g7f5026d
copying README.md -> swh.search-0.11.3.dev2+g7f5026d
copying mypy.ini -> swh.search-0.11.3.dev2+g7f5026d
copying package.json -> swh.search-0.11.3.dev2+g7f5026d
copying pyproject.toml -> swh.search-0.11.3.dev2+g7f5026d
copying pytest.ini -> swh.search-0.11.3.dev2+g7f5026d
copying requirements-swh.txt -> swh.search-0.11.3.dev2+g7f5026d
copying requirements-test.txt -> swh.search-0.11.3.dev2+g7f5026d
copying requirements.txt -> swh.search-0.11.3.dev2+g7f5026d
copying setup.cfg -> swh.search-0.11.3.dev2+g7f5026d
copying setup.py -> swh.search-0.11.3.dev2+g7f5026d
copying tox.ini -> swh.search-0.11.3.dev2+g7f5026d
copying yarn.lock -> swh.search-0.11.3.dev2+g7f5026d
copying docs/.gitignore -> swh.search-0.11.3.dev2+g7f5026d/docs
copying docs/Makefile -> swh.search-0.11.3.dev2+g7f5026d/docs
copying docs/cli.rst -> swh.search-0.11.3.dev2+g7f5026d/docs
copying docs/conf.py -> swh.search-0.11.3.dev2+g7f5026d/docs
copying docs/index.rst -> swh.search-0.11.3.dev2+g7f5026d/docs
copying docs/query-language.rst -> swh.search-0.11.3.dev2+g7f5026d/docs
copying docs/_static/.placeholder -> swh.search-0.11.3.dev2+g7f5026d/docs/_static
copying docs/_templates/.placeholder -> swh.search-0.11.3.dev2+g7f5026d/docs/_templates
copying es_config/elasticsearch.keystore -> swh.search-0.11.3.dev2+g7f5026d/es_config
copying es_config/elasticsearch.yml -> swh.search-0.11.3.dev2+g7f5026d/es_config
copying es_config/jvm.options -> swh.search-0.11.3.dev2+g7f5026d/es_config
copying es_config/log4j2.properties -> swh.search-0.11.3.dev2+g7f5026d/es_config
copying swh/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh
copying swh.search.egg-info/PKG-INFO -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
copying swh.search.egg-info/SOURCES.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
copying swh.search.egg-info/dependency_links.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
copying swh.search.egg-info/entry_points.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
copying swh.search.egg-info/not-zip-safe -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
copying swh.search.egg-info/requires.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
copying swh.search.egg-info/top_level.txt -> swh.search-0.11.3.dev2+g7f5026d/swh.search.egg-info
copying swh/search/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/cli.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/elasticsearch.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/in_memory.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/interface.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/journal_client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/metrics.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/py.typed -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/translator.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/utils.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search
copying swh/search/api/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/api
copying swh/search/api/client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/api
copying swh/search/api/server.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/api
copying swh/search/query_language/.gitignore -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language
copying swh/search/query_language/grammar.js -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language
copying swh/search/query_language/sample_query -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language
copying swh/search/query_language/tokens.js -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language
copying swh/search/query_language/test/corpus/combinations.txt -> swh.search-0.11.3.dev2+g7f5026d/swh/search/query_language/test/corpus
copying swh/search/tests/__init__.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/conftest.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_api_client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_cli.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_elasticsearch.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_in_memory.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_init.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_journal_client.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_search.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_server.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
copying swh/search/tests/test_translator.py -> swh.search-0.11.3.dev2+g7f5026d/swh/search/tests
Writing swh.search-0.11.3.dev2+g7f5026d/setup.cfg
running ts_install
yarn install v1.22.5
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.06s.
Copying parser files
Getting path
Generating
Creating tar archive
removing 'swh.search-0.11.3.dev2+g7f5026d' (and everything under it)
running bdist_wheel
running build
running build_py
running ts_build
running ts_build_so
error: [Errno 2] No such file or directory: 'swh/search/query_language/src/parser.c'

Build is green

Patch application report for D6131 (id=22192)

Rebasing onto 26f800cde3...

Current branch diff-target is up to date.
Changes applied before test
commit 7da09cf5e7d546bb9e5f62a18ec10a018ab223fb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/280/ for more details.

This revision is now accepted and ready to land.Aug 25 2021, 1:43 PM

fix support for old yarn versions (without -s)

Build is green

Patch application report for D6131 (id=22233)

Rebasing onto 3893e39ede...

First, rewinding head to replay your work on top of it...
Applying: Remove .wasm from all distributions, and .so from source distributions
Changes applied before test
commit c6ac6a1a7c23d98327e54fb558de32b6b87e0ad0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/284/ for more details.

Build is green

Patch application report for D6131 (id=22236)

Rebasing onto 3893e39ede...

Current branch diff-target is up to date.
Changes applied before test
commit 3940daa889ceefbcdbbe7e4b096511777b8ef5f1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 25 11:38:20 2021 +0200

    Remove .wasm from all distributions, and .so from source distributions
    
    .so is still built, but only for binary distributions (it can be built
    from a source distribution).
    
    To do this, we now include intermediary files (parser.c in particular)
    in the source distribution.
    They are not really source file, but this allows to remove the dependency
    on a nodejs runtime to install from pypi.

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/285/ for more details.