Page MenuHomeSoftware Heritage
Differential D7995

Arch Linux loader
ClosedPublic
Actions

Authored by franckbret on Jun 16 2022, 9:38 AM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4233: Ingest Arch Linux
Commits
rDLDBASEb6af2638c11a: Arch Linux loader
Summary

Fetch Arch linux packages from lister discovered origins.
For each origin it get versions from extra_loader_arguments['artifacts']

Arch Linux package can comes as .xz or .zst file archive.
Support for .zst (Zstandard compression) has been requested with D7993.
CI should fail until D7993 is merged.

Related to T4233

Diff Detail

Event Timeline

franckbret created this revision.Jun 16 2022, 9:38 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 16 2022, 9:38 AM
swh-public-ci added a comment.Jun 16 2022, 9:42 AM

Build has FAILED

Patch application report for D7995 (id=28801)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: WIP: Arch Linux loader
Changes applied before test
commit ef2f9c304bd148af8cf4c04ed3d00b9898befd27
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    WIP: Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    CI should fail until D7993 is merged.
    
    Related to T4233

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/798/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/798/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2022, 9:42 AM
Harbormaster failed remote builds in B29884: Diff 28801!
franckbret updated this revision to Diff 28813.Jun 16 2022, 12:40 PM

Updating D7995: WIP: Arch Linux loader

Remove commented dead code.
Also now that D7993 is merged, let's check if CI succeed.

swh-public-ci added a comment.Jun 16 2022, 12:44 PM

Build has FAILED

Patch application report for D7995 (id=28813)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: WIP: Arch Linux loader
Changes applied before test
commit bdf578055066fa892094278ede9c53b580526bf2
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    WIP: Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    CI should fail until D7993 is merged.
    
    Related to T4233

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/799/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/799/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2022, 12:44 PM
Harbormaster failed remote builds in B29899: Diff 28813!
swh-public-ci added a comment.Jun 16 2022, 1:48 PM

Build is green

Patch application report for D7995 (id=28813)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: WIP: Arch Linux loader
Changes applied before test
commit 9560d51de7122de560aa4007e6bebda3fe8435c1
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    WIP: Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    CI should fail until D7993 is merged.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/800/ for more details.

Harbormaster completed remote builds in B29899: Diff 28813.Jun 16 2022, 1:48 PM
franckbret requested review of this revision.Jun 16 2022, 1:48 PM
franckbret updated this revision to Diff 28816.Jun 16 2022, 1:58 PM

Updating D7995: Arch Linux loader

Remove commented lines from fake_arch.sh script
Pin swh-core requirements to swh.core >= 2.12
Remove WIP from title

swh-public-ci added a comment.Jun 16 2022, 2:02 PM

Build is green

Patch application report for D7995 (id=28816)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: Arch Linux loader
Changes applied before test
commit cb8ff1f51ca723ad020d195dee873f786ec13e3d
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/801/ for more details.

Harbormaster completed remote builds in B29902: Diff 28816.Jun 16 2022, 2:02 PM
vlorentz added a subscriber: vlorentz.Jun 16 2022, 2:07 PM

LGTM

swh/loader/package/arch/loader.py
122–135

please add this to https://forge.softwareheritage.org/source/swh-loader-core/browse/master/docs/package-loader-specifications.rst

123–125

for consistency with D7998

ardumont added a subscriber: ardumont.Jun 16 2022, 2:08 PM
ardumont added inline comments.
swh/loader/package/arch/loader.py
60–63

with:

from ..pattern import CredentialsType
from typing import Optional

for the types

I recall that breaks otherwise when deployed.

ardumont added inline comments.Jun 16 2022, 2:10 PM
swh/loader/package/arch/loader.py
60–63

wait, nvm, we are in package loader...
sorry for the hit and miss ;)

vlorentz added inline comments.Jun 16 2022, 2:17 PM
swh/loader/package/arch/loader.py
60–63

even better ^

that's what we decided to do with all loaders now

ardumont added inline comments.Jun 16 2022, 2:19 PM
swh/loader/package/arch/loader.py
60–63

+1 ^

104

To ensure we dot not wrongly assume too much.

franckbret added inline comments.Jun 16 2022, 2:20 PM
swh/loader/package/arch/loader.py
60–63

;-)

123–125

Ok

Do you think it valuable to add in this arch message {p.arch} and {p.repo}?

ardumont added a comment.Jun 16 2022, 2:21 PM

lgtm as well.

You are missing the entrypoint entry so the loader can be called through the swh loader cli. [1]
I'm ok if you want to open a new diff for that part though, as you wish.

[1] https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-tutorial.html#creating-the-files-hierarchy

franckbret added inline comments.Jun 16 2022, 2:25 PM
swh/loader/package/arch/loader.py
118

@ardumont @vlorentz Does it add any value to add intrinsic_metadata to p_info object?

ardumont added inline comments.Jun 16 2022, 2:29 PM
swh/loader/package/arch/loader.py
123–125

It does not hurt. Although, if you look at the exising loader packages [1], we limit the
information stored in the crafted message. I guess that information ends up being in the
intrinsic metadata anyway so i'd say it's fine as is.

@vlorentz any more thoughts on the matter?

[1] https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-specifications.html#release-fields

vlorentz added inline comments.Jun 16 2022, 2:31 PM
swh/loader/package/arch/loader.py
118

They are more readily available when in the release message right now, because metadata views are really bad.

123–125

Hmm, I don't know. But if we remove them here, then we should remove them from the Crates loader as well.

What do the descriptions look like in practice, for Arch and Crates?

ardumont added inline comments.Jun 16 2022, 2:34 PM
swh/loader/package/arch/loader.py
123–125

to clarify, i was just saying "keep the actual implementation" to keep the crafted message minimalistic like the other package loader and not add the suggested p.arch and p.repo.

ardumont added inline comments.Jun 16 2022, 2:42 PM
swh/loader/package/arch/loader.py
122–135

that will show up in the doc when we land and tag [1]

[1] https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-specifications.html#release-fields

ardumont mentioned this in T4233: Ingest Arch Linux.Jun 16 2022, 2:47 PM
franckbret updated this revision to Diff 28823.Jun 16 2022, 3:13 PM
franckbret marked 6 inline comments as done.

Updating D7995: WIP: Arch Linux loader

Add Arch Linux release fields entry to package loader specification
+ some small fixes after review

swh-public-ci added a comment.Jun 16 2022, 3:17 PM

Build has FAILED

Patch application report for D7995 (id=28823)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: Arch Linux loader
Changes applied before test
commit 8a808bbb36b8617f35156c45bf163c658365382f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/804/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/804/console

Harbormaster failed remote builds in B29909: Diff 28823!Jun 16 2022, 3:17 PM
franckbret updated this revision to Diff 28824.Jun 16 2022, 3:32 PM

Updating D7995: Arch Linux loader

Fix tests

swh-public-ci added a comment.Jun 16 2022, 3:36 PM

Build is green

Patch application report for D7995 (id=28824)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: Arch Linux loader
Changes applied before test
commit 98c25f32623ebdaaebddea5c7c8647bc71cae8e4
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/805/ for more details.

Harbormaster completed remote builds in B29910: Diff 28824.Jun 16 2022, 3:36 PM
franckbret marked an inline comment as done.Jun 16 2022, 3:39 PM
franckbret retitled this revision from WIP: Arch Linux loader to Arch Linux loader.
ardumont accepted this revision.Jun 16 2022, 3:44 PM
This revision is now accepted and ready to land.Jun 16 2022, 3:44 PM
franckbret updated this revision to Diff 28832.Jun 17 2022, 9:33 AM

Updating D7995: Arch Linux loader

Rebase from master

This revision was landed with ongoing or failed builds.Jun 17 2022, 9:33 AM
Closed by commit rDLDBASEb6af2638c11a: Arch Linux loader (authored by franckbret). · Explain Why
This revision was automatically updated to reflect the committed changes.
franckbret added a commit: rDLDBASEb6af2638c11a: Arch Linux loader.
swh-public-ci added a comment.Jun 17 2022, 9:36 AM

Build is green

Patch application report for D7995 (id=28832)

Rebasing onto d925d06e6f...

Current branch diff-target is up to date.
Changes applied before test
commit b6af2638c11ac597649378407dd1a4f1d7ab3bd0
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/806/ for more details.

Harbormaster completed remote builds in B29917: Diff 28832.Jun 17 2022, 9:37 AM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
docs/
9 lines--
2 lines--
1 line--
swh/
loader/
package/
arch/
17 lines100%
Loading...
136 lines100%
Loading...
14 lines100%
Loading...
tests/
97%
Loading...
data/
86 lines--
https_archive.archlinux.org/
--
--
https_uk.mirror.archlinuxarm.org/
--
231 lines97%
Loading...
35 lines100%
Loading...

Diff 28832

View Options

docs/package-loader-specifications.rst

Loading...
View Options

requirements-swh.txt

Loading...
View Options

setup.py

Loading...
View Options

swh/loader/package/arch/__init__.py

Loading...
View Options

swh/loader/package/arch/loader.py

Loading...
View Options

swh/loader/package/arch/tasks.py

Loading...
View Options

swh/loader/package/arch/tests/__init__.py

Loading...
View Options

swh/loader/package/arch/tests/data/fake_arch.sh

Loading...
View Options

swh/loader/package/arch/tests/data/https_archive.archlinux.org/packages_d_dialog_dialog-1:1.3_20190211-1-x86_64.pkg.tar.xz

Loading...
View Options

swh/loader/package/arch/tests/data/https_archive.archlinux.org/packages_d_dialog_dialog-1:1.3_20220414-1-x86_64.pkg.tar.zst

Loading...
View Options

swh/loader/package/arch/tests/data/https_uk.mirror.archlinuxarm.org/aarch64_core_gzip-1.12-1-aarch64.pkg.tar.xz

Loading...
View Options

swh/loader/package/arch/tests/test_arch.py

Loading...
View Options

swh/loader/package/arch/tests/test_tasks.py

Loading...
About · Developer information · Legal