⚙ D8033 Arch User Repository (AUR) lister

franckbret created this revision.Jun 24 2022, 12:26 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 24 2022, 12:26 PM

Build has FAILED

Patch application report for D8033 (id=28931)

Rebasing onto 1bf11aa26d...

Current branch diff-target is up to date.

Changes applied before test

commit 0238f136b5df5e684b393301d01319ee29abf423
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    [WIP] Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/549/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/549/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 24 2022, 12:30 PM

Harbormaster failed remote builds in B30024: Diff 28931!

Updating D8033: [WIP] Arch User Repository (AUR) lister

Fix issue with 'last_modified' date timezone adding timezone.utc offset.

Build is green

Patch application report for D8033 (id=28933)

Rebasing onto 1bf11aa26d...

Current branch diff-target is up to date.

Changes applied before test

commit e91114aca59c17fb4b7e48028ac1687df580aa6d
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    [WIP] Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/550/ for more details.

Harbormaster completed remote builds in B30026: Diff 28933.Jun 24 2022, 1:00 PM

franckbret requested review of this revision.Jun 24 2022, 1:00 PM

@ardumont @vlorentz Hi, here is a first implementation of the Arch User Repository (AUR).

It's about 83841 packages today, https://aur.archlinux.org/packages.
I get the packages list through https://aur.archlinux.org/packages-meta-ext-v1.json.gz wich is about 6,6Mb gzip file.

There is an rpc named Aurweb RPC interface, see https://wiki.archlinux.org/title/Aurweb_RPC_interface but its recommended to download a json.gz file which contains some metadata for each packages. See https://lists.archlinux.org/pipermail/aur-general/2021-November/036659.html

The main difference from Arch linux is that the end user builds its own packages with the help of makepkg + pacman. See https://wiki.archlinux.org/title/Arch_User_Repository for more details.
It mainly stands on git repositories containing PKGBUILD file and .PKGINFO, see https://aur.archlinux.org/cgit/aur.git/log/?h=hg-evolve for an example.

There is no real direct way for the lister to discover where to download oldest versions of a package. There is a canonical url for each package in its page description but its the latest snapshot url, no way to know which version it is when downloading from this link.

For now I did not implement scrapping or whatever to get oldest versions of a package because I did not find a common way to do that. There is a git log on package page but messages are obviously inconsistent ( https://aur.archlinux.org/cgit/aur.git/log/?h=hg-evolve ).

Maybe we can play with git to found some kind of version diff history in .SRCINFO file as I've tried before for Arch PKGINFO file, but with this amount of repositories it will probably take a lot of time and resources. The .SRCINFO contains generally a link to the 'real' package corresponding version. See https://aur.archlinux.org/cgit/aur.git/tree/.SRCINFO?h=hg-evolve

What do you think?
Is it ok to say that the origin is a git repository url in this case?
If yes, will the loader really be a Package loader or VCS git loader?

ardumont added inline comments.Jun 27 2022, 4:03 PM

swh/lister/aur/tests/__init__.py
2	no need, we usually have it empty.

In D8033#208769, @franckbret wrote:

There is no real direct way for the lister to discover where to download oldest versions of a package. There is a canonical url for each package in its page description but its the latest snapshot url, no way to know which version it is when downloading from this link.

If they are not directly available, then it doesn't make sense to have them in snapshots anyway. We'll just have successive visits of the loader as history.

For now I did not implement scrapping or whatever to get oldest versions of a package because I did not find a common way to do that. There is a git log on package page but messages are obviously inconsistent ( https://aur.archlinux.org/cgit/aur.git/log/?h=hg-evolve ).

Maybe we can play with git to found some kind of version diff history in .SRCINFO file as I've tried before for Arch PKGINFO file, but with this amount of repositories it will probably take a lot of time and resources. The .SRCINFO contains generally a link to the 'real' package corresponding version. See https://aur.archlinux.org/cgit/aur.git/tree/.SRCINFO?h=hg-evolve

What do you think?
Is it ok to say that the origin is a git repository url in this case?
If yes, will the loader really be a Package loader or VCS git loader?

clearly git loader, as there can be branches and whatnot; history also matters here.

However, the main content of these repositories is the PKGBUILD, which (among other things) fetches the code from somewhere else (tarball, git commit, ...), and the PKGBUILD alone is not very useful without that code. Therefore, it looks like we should implement something like T3923, to fetch the actual code.

Updating D8033: [WIP] Arch User Repository (AUR) lister

Fix an issue with requests usage when downloading packages archives ensuring it does not decode the binary directly. (tests and CI were fine, but got the bug while testing runner on docker)

Build is green

Patch application report for D8033 (id=28971)

Rebasing onto 1bf11aa26d...

Current branch diff-target is up to date.

Changes applied before test

commit 4729b0aae165aab640677adefc2f6ceb90072bcd
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/551/ for more details.

Harbormaster completed remote builds in B30059: Diff 28971.Jun 28 2022, 12:52 PM

In D8033#209040, @franckbret wrote:

but got the bug while testing runner on docker

what was the bug, exactly? I don't see how the new code changes the behavior.

Also, please remove Updating D8033: [WIP] Arch User Repository (AUR) lister from change descriptions; it's redundant and makes the "History" tab of https://forge.softwareheritage.org/D8033#toc less readable

franckbret retitled this revision from [WIP] Arch User Repository (AUR) lister to Arch User Repository (AUR) lister.Jun 29 2022, 3:03 PM

In D8033#209048, @vlorentz wrote:

In D8033#209040, @franckbret wrote:

but got the bug while testing runner on docker

what was the bug, exactly? I don't see how the new code changes the behavior.

Also, please remove Updating D8033: [WIP] Arch User Repository (AUR) lister from change descriptions; it's redundant and makes the "History" tab of https://forge.softwareheritage.org/D8033#toc less readable

@vlorentz The change force requests.get to not automatically decompress the response. This way we are sure we always get the raw archive as is. See https://requests.readthedocs.io/en/latest/user/quickstart/#raw-response-content

ardumont added inline comments.Jun 30 2022, 10:37 AM

swh/lister/aur/lister.py
22	No need if you always return one element?
32
35
49	I gather that's a working directory. If so, you'll need to have some post-listing routine cleanup.
107–122	I've also adapted according to my type suggestion early on.

ardumont added inline comments.Jun 30 2022, 11:18 AM

swh/lister/aur/lister.py
22	i mean, your code below returns page of one element if i read it correctly.
124–139	in effect, you only read one origin per "page".

@franckbret ping on earlier comments

franckbret mentioned this in D8101: crates.lister: Implement incremental mode.Jul 19 2022, 3:44 PM

Some typo and consistency fixes after code review

swh/lister/aur/lister.py
22	yep

franckbret added inline comments.Jul 19 2022, 4:35 PM

swh/lister/aur/lister.py
49	Yes It is. Have you an example of post-listing routine action? Same pattern used for arch and crates, I guess I need to do cleanup to

Build is green

Patch application report for D8033 (id=29383)

Rebasing onto 1bf11aa26d...

Current branch diff-target is up to date.

Changes applied before test

commit d4851a18c7bf97dcfffdb8362dcf4c6a720bde02
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/554/ for more details.

Harbormaster completed remote builds in B30457: Diff 29383.Jul 19 2022, 4:36 PM

franckbret added inline comments.Jul 20 2022, 9:52 AM

swh/lister/aur/lister.py
49	Is finalize() suitable for that purpose?

vlorentz added inline comments.Jul 20 2022, 10:10 AM

swh/lister/aur/lister.py
49	yes

Split artifacts data to artifacts and aur_metadata

Build is green

Patch application report for D8033 (id=29732)

Rebasing onto cee6bcb514...

First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister

Changes applied before test

commit a33fbf745785ebe179bce4fbbbbb52a17fabce65
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/586/ for more details.

Harbormaster completed remote builds in B30784: Diff 29732.Aug 16 2022, 12:37 PM

Add a finalize method that remove temporary directory once the lister complete

Build is green

Patch application report for D8033 (id=29755)

Rebasing onto cee6bcb514...

First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister

Changes applied before test

commit 280be6e9530238a385843ee09ee5040ca0d60473
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/587/ for more details.

Harbormaster completed remote builds in B30807: Diff 29755.Aug 16 2022, 1:11 PM

Add a directory cleanup test

Build is green

Patch application report for D8033 (id=29760)

Rebasing onto cee6bcb514...

First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister

Changes applied before test

commit 3e0c54bf111c316ead8a6804060f3432d8f3209f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/588/ for more details.

Harbormaster completed remote builds in B30812: Diff 29760.Aug 16 2022, 3:35 PM

Document the module

Build has FAILED

Patch application report for D8033 (id=29766)

Rebasing onto cee6bcb514...

First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister

Changes applied before test

commit aeb60cc266f0fdf11e8650c9f7710ec00cf2443f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/589/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/589/console

Harbormaster failed remote builds in B30818: Diff 29766!Aug 16 2022, 5:33 PM

Fix documentation build

Build has FAILED

Patch application report for D8033 (id=29767)

Rebasing onto cee6bcb514...

First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister

Changes applied before test

commit 0b722b297b17060de8eca33bc7811039e0bbd3df
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/590/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/590/console

Harbormaster failed remote builds in B30819: Diff 29767!Aug 16 2022, 6:16 PM

Fix docstrings errors

Build is green

Patch application report for D8033 (id=29768)

Rebasing onto cee6bcb514...

First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister

Changes applied before test

commit 3962b637421577c9fec8a0f0f1c8c7661b1bfa88
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/591/ for more details.

Harbormaster completed remote builds in B30820: Diff 29768.Aug 16 2022, 7:16 PM

Looks good! Two more nitpicks before this is good to merge:

swh/lister/aur/__init__.py
20–23	I don't think that's accurate; we do archive split packages, but only their "pkgbase" because that is the only one that actually has source code.
77–96	why is `"version"` in both?

Rephrase the documentation part about regular and split packages

Build is green

Patch application report for D8033 (id=29781)

Rebasing onto 6a53a6ad06...

First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister

Changes applied before test

commit 96683382dc91d6927998797ba5a3cd609319eed4
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/594/ for more details.

Harbormaster completed remote builds in B30833: Diff 29781.Aug 17 2022, 11:57 AM

franckbret marked an inline comment as done.Aug 17 2022, 12:01 PM

franckbret added inline comments.

swh/lister/aur/__init__.py
20–23	I rephrase it, does it looks ok for you?
77–96	Because its the associative key for lines for both dict which is needed for the loader the get the corresponding informations. Btw there will be only one line in the case of aur. I think it's more explicit to keep it as is, but if you want me to remove it, I sure can.

Can we merge this one?

vlorentz accepted this revision.Aug 19 2022, 10:30 AM

vlorentz added inline comments.

swh/lister/aur/__init__.py
77–96	nevermind, it's fine

This revision is now accepted and ready to land.Aug 19 2022, 10:30 AM

Rebase on master

Build is green

Patch application report for D8033 (id=29847)

Rebasing onto 6a53a6ad06...

Current branch diff-target is up to date.

Changes applied before test

commit 97b353bf0b9a726ebb3d414f809aeac26a229f21
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/597/ for more details.

Harbormaster completed remote builds in B30899: Diff 29847.Aug 19 2022, 12:49 PM

Closed by commit rDLS97b353bf0b9a: Arch User Repository (AUR) lister (authored by franckbret). · Explain WhyAug 19 2022, 12:51 PM

This revision was automatically updated to reflect the committed changes.

franckbret added a commit: rDLS97b353bf0b9a: Arch User Repository (AUR) lister.

franckbret mentioned this in T4466: Ingest Aur repository (Arch User Repository).Aug 26 2022, 4:30 PM

Arch User Repository (AUR) lister
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D8033 (id=28931)

Changes applied before test

Patch application report for D8033 (id=28933)

Changes applied before test

Patch application report for D8033 (id=28971)

Changes applied before test

Patch application report for D8033 (id=29383)

Changes applied before test

Patch application report for D8033 (id=29732)

Changes applied before test

Patch application report for D8033 (id=29755)

Changes applied before test

Patch application report for D8033 (id=29760)

Changes applied before test

Patch application report for D8033 (id=29766)

Changes applied before test

Patch application report for D8033 (id=29767)

Changes applied before test

Patch application report for D8033 (id=29768)

Changes applied before test

Patch application report for D8033 (id=29781)

Changes applied before test

Patch application report for D8033 (id=29847)

Changes applied before test

Revision Contents
Changeset List

Diff 29383

setup.py

swh/lister/aur/init.py

swh/lister/aur/lister.py

swh/lister/aur/tasks.py

swh/lister/aur/tests/init.py

swh/lister/aur/tests/data/fake_aur_packages.sh

swh/lister/aur/tests/data/https_aur.archlinux.org/packages-meta-v1.json.gz

swh/lister/aur/tests/test_lister.py

swh/lister/aur/tests/test_tasks.py

Arch User Repository (AUR) listerClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D8033 (id=28931)

Changes applied before test

Patch application report for D8033 (id=28933)

Changes applied before test

Patch application report for D8033 (id=28971)

Changes applied before test

Patch application report for D8033 (id=29383)

Changes applied before test

Patch application report for D8033 (id=29732)

Changes applied before test

Patch application report for D8033 (id=29755)

Changes applied before test

Patch application report for D8033 (id=29760)

Changes applied before test

Patch application report for D8033 (id=29766)

Changes applied before test

Patch application report for D8033 (id=29767)

Changes applied before test

Patch application report for D8033 (id=29768)

Changes applied before test

Patch application report for D8033 (id=29781)

Changes applied before test

Patch application report for D8033 (id=29847)

Changes applied before test

Revision ContentsChangeset List

Diff 29383

setup.py

swh/lister/aur/__init__.py

swh/lister/aur/lister.py

swh/lister/aur/tasks.py

swh/lister/aur/tests/__init__.py

swh/lister/aur/tests/data/fake_aur_packages.sh

swh/lister/aur/tests/data/https_aur.archlinux.org/packages-meta-v1.json.gz

swh/lister/aur/tests/test_lister.py

swh/lister/aur/tests/test_tasks.py

Arch User Repository (AUR) lister
ClosedPublic
Actions

Revision Contents
Changeset List

swh/lister/aur/init.py

swh/lister/aur/tests/init.py