Page MenuHomeSoftware Heritage

Arch Linux loader
ClosedPublic

Authored by franckbret on Jun 16 2022, 9:38 AM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4233: Ingest Arch Linux
Commits
rDLDBASEb6af2638c11a: Arch Linux loader
Summary

Fetch Arch linux packages from lister discovered origins.
For each origin it get versions from extra_loader_arguments['artifacts']

Arch Linux package can comes as .xz or .zst file archive.
Support for .zst (Zstandard compression) has been requested with D7993.
CI should fail until D7993 is merged.

Related to T4233

Diff Detail

Event Timeline

Build has FAILED

Patch application report for D7995 (id=28801)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: WIP: Arch Linux loader
Changes applied before test
commit ef2f9c304bd148af8cf4c04ed3d00b9898befd27
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    WIP: Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    CI should fail until D7993 is merged.
    
    Related to T4233

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/798/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/798/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2022, 9:42 AM
Harbormaster failed remote builds in B29884: Diff 28801!

Updating D7995: WIP: Arch Linux loader

Remove commented dead code.
Also now that D7993 is merged, let's check if CI succeed.

Build has FAILED

Patch application report for D7995 (id=28813)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: WIP: Arch Linux loader
Changes applied before test
commit bdf578055066fa892094278ede9c53b580526bf2
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    WIP: Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    CI should fail until D7993 is merged.
    
    Related to T4233

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/799/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/799/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 16 2022, 12:44 PM
Harbormaster failed remote builds in B29899: Diff 28813!

Build is green

Patch application report for D7995 (id=28813)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: WIP: Arch Linux loader
Changes applied before test
commit 9560d51de7122de560aa4007e6bebda3fe8435c1
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    WIP: Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    CI should fail until D7993 is merged.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/800/ for more details.

Updating D7995: Arch Linux loader

Remove commented lines from fake_arch.sh script
Pin swh-core requirements to swh.core >= 2.12
Remove WIP from title

Build is green

Patch application report for D7995 (id=28816)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: Arch Linux loader
Changes applied before test
commit cb8ff1f51ca723ad020d195dee873f786ec13e3d
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/801/ for more details.

ardumont added inline comments.
swh/loader/package/arch/loader.py
60–63

with:

from ..pattern import CredentialsType
from typing import Optional

for the types

I recall that breaks otherwise when deployed.

swh/loader/package/arch/loader.py
60–63

wait, nvm, we are in package loader...
sorry for the hit and miss ;)

swh/loader/package/arch/loader.py
60–63

even better ^

that's what we decided to do with all loaders now

swh/loader/package/arch/loader.py
60–63

+1 ^

104

To ensure we dot not wrongly assume too much.

swh/loader/package/arch/loader.py
60–63

;-)

123–125

Ok

Do you think it valuable to add in this arch message {p.arch} and {p.repo}?

lgtm as well.

You are missing the entrypoint entry so the loader can be called through the swh loader cli. [1]
I'm ok if you want to open a new diff for that part though, as you wish.

[1] https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-tutorial.html#creating-the-files-hierarchy

swh/loader/package/arch/loader.py
118

@ardumont @vlorentz Does it add any value to add intrinsic_metadata to p_info object?

swh/loader/package/arch/loader.py
123–125

It does not hurt. Although, if you look at the exising loader packages [1], we limit the
information stored in the crafted message. I guess that information ends up being in the
intrinsic metadata anyway so i'd say it's fine as is.

@vlorentz any more thoughts on the matter?

[1] https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-specifications.html#release-fields

swh/loader/package/arch/loader.py
118

They are more readily available when in the release message right now, because metadata views are really bad.

123–125

Hmm, I don't know. But if we remove them here, then we should remove them from the Crates loader as well.

What do the descriptions look like in practice, for Arch and Crates?

swh/loader/package/arch/loader.py
123–125

to clarify, i was just saying "keep the actual implementation" to keep the crafted message minimalistic like the other package loader and not add the suggested p.arch and p.repo.

swh/loader/package/arch/loader.py
122–135
franckbret marked 6 inline comments as done.

Updating D7995: WIP: Arch Linux loader

Add Arch Linux release fields entry to package loader specification
+ some small fixes after review

Build has FAILED

Patch application report for D7995 (id=28823)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: Arch Linux loader
Changes applied before test
commit 8a808bbb36b8617f35156c45bf163c658365382f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/804/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/804/console

Updating D7995: Arch Linux loader

Fix tests

Build is green

Patch application report for D7995 (id=28824)

Rebasing onto d925d06e6f...

First, rewinding head to replay your work on top of it...
Applying: Arch Linux loader
Changes applied before test
commit 98c25f32623ebdaaebddea5c7c8647bc71cae8e4
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/805/ for more details.

franckbret retitled this revision from WIP: Arch Linux loader to Arch Linux loader.
This revision is now accepted and ready to land.Jun 16 2022, 3:44 PM

Updating D7995: Arch Linux loader

Rebase from master

This revision was landed with ongoing or failed builds.Jun 17 2022, 9:33 AM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D7995 (id=28832)

Rebasing onto d925d06e6f...

Current branch diff-target is up to date.
Changes applied before test
commit b6af2638c11ac597649378407dd1a4f1d7ab3bd0
Author: Franck Bret <franck.bret@octobus.net>
Date:   Thu Jun 16 09:21:45 2022 +0200

    Arch Linux loader
    
    Fetch Arch linux packages from lister discovered origins.
    For each origin it get versions from extra_loader_arguments['artifacts']
    
    Arch Linux package can comes as .xz or .zst file archive.
    Support for .zst (Zstandard compression) has been requested with D7993.
    
    Related to T4233

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/806/ for more details.