Page MenuHomeSoftware Heritage
Differential D8033

Arch User Repository (AUR) lister
ClosedPublic
Actions

Authored by franckbret on Jun 24 2022, 12:26 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDLS97b353bf0b9a: Arch User Repository (AUR) lister
Summary

Add 'aur' module to swh-lister with data fixtures and tests.
For now, origin url are package vcs (Git) url.

Diff Detail

Repository
rDLS Listers
Branch
aur
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30024
Build 46938: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 46937: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
365 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.aur.tests.test_lister::test_aur_lister
datadir = '/var/lib/jenkins/workspace/DLS/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/lister/aur/tests/data' requests_mock_datadir = <requests_mock.mocker.Mocker object at 0x7ff428cdaa90> swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7ff428cdaeb8>
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.lister::swh.lister.arch.lister.size_to_bytes
1,784 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_lister::test_arch_lister
1,006 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_tasks::test_arch_lister
3,596 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_tasks::test_arch_ping
View Full Test Results (1 Failed · 215 Passed)

Event Timeline

franckbret created this revision.Jun 24 2022, 12:26 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 24 2022, 12:26 PM
swh-public-ci added a comment.Jun 24 2022, 12:30 PM

Build has FAILED

Patch application report for D8033 (id=28931)

Rebasing onto 1bf11aa26d...

Current branch diff-target is up to date.
Changes applied before test
commit 0238f136b5df5e684b393301d01319ee29abf423
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    [WIP] Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/549/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/549/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 24 2022, 12:30 PM
Harbormaster failed remote builds in B30024: Diff 28931!
franckbret updated this revision to Diff 28933.Jun 24 2022, 12:56 PM

Updating D8033: [WIP] Arch User Repository (AUR) lister

Fix issue with 'last_modified' date timezone adding timezone.utc offset.

swh-public-ci added a comment.Jun 24 2022, 1:00 PM

Build is green

Patch application report for D8033 (id=28933)

Rebasing onto 1bf11aa26d...

Current branch diff-target is up to date.
Changes applied before test
commit e91114aca59c17fb4b7e48028ac1687df580aa6d
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    [WIP] Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/550/ for more details.

Harbormaster completed remote builds in B30026: Diff 28933.Jun 24 2022, 1:00 PM
franckbret requested review of this revision.Jun 24 2022, 1:00 PM
franckbret added subscribers: vlorentz, ardumont.Jun 24 2022, 1:52 PM

@ardumont @vlorentz Hi, here is a first implementation of the Arch User Repository (AUR).

It's about 83841 packages today, https://aur.archlinux.org/packages.
I get the packages list through https://aur.archlinux.org/packages-meta-ext-v1.json.gz wich is about 6,6Mb gzip file.

There is an rpc named Aurweb RPC interface, see https://wiki.archlinux.org/title/Aurweb_RPC_interface but its recommended to download a json.gz file which contains some metadata for each packages. See https://lists.archlinux.org/pipermail/aur-general/2021-November/036659.html

The main difference from Arch linux is that the end user builds its own packages with the help of makepkg + pacman. See https://wiki.archlinux.org/title/Arch_User_Repository for more details.
It mainly stands on git repositories containing PKGBUILD file and .PKGINFO, see https://aur.archlinux.org/cgit/aur.git/log/?h=hg-evolve for an example.

There is no real direct way for the lister to discover where to download oldest versions of a package. There is a canonical url for each package in its page description but its the latest snapshot url, no way to know which version it is when downloading from this link.

For now I did not implement scrapping or whatever to get oldest versions of a package because I did not find a common way to do that. There is a git log on package page but messages are obviously inconsistent ( https://aur.archlinux.org/cgit/aur.git/log/?h=hg-evolve ).

Maybe we can play with git to found some kind of version diff history in .SRCINFO file as I've tried before for Arch PKGINFO file, but with this amount of repositories it will probably take a lot of time and resources. The .SRCINFO contains generally a link to the 'real' package corresponding version. See https://aur.archlinux.org/cgit/aur.git/tree/.SRCINFO?h=hg-evolve

What do you think?
Is it ok to say that the origin is a git repository url in this case?
If yes, will the loader really be a Package loader or VCS git loader?

ardumont added inline comments.Jun 27 2022, 4:03 PM
swh/lister/aur/tests/__init__.py


3
no need, we usually have it empty.
vlorentz added a comment.Jun 27 2022, 4:03 PM


In D8033#208769, @franckbret wrote:


There is no real direct way for the lister to discover where to download oldest versions of a package. There is a canonical url for each package in its page description but its the latest snapshot url, no way to know which version it is when downloading from this link.





If they are not directly available, then it doesn't make sense to have them in snapshots anyway. We'll just have successive visits of the loader as history.



For now I did not implement scrapping or whatever to get oldest versions of a package because I did not find a common way to do that. There is a git log on package page but messages are obviously inconsistent ( https://aur.archlinux.org/cgit/aur.git/log/?h=hg-evolve ).



Maybe we can  play with git to found some kind of version diff history in .SRCINFO file as I've tried before for Arch PKGINFO file, but with this amount of repositories it will probably take a lot of time and resources. The .SRCINFO contains generally a link to the 'real' package corresponding version. See https://aur.archlinux.org/cgit/aur.git/tree/.SRCINFO?h=hg-evolve



What do you think? 

Is it ok to say that the origin is a git repository url in this case?

If yes, will the loader really be a Package loader or VCS git loader?



clearly git loader, as there can be branches and whatnot; history also matters here.



However, the main content of these repositories is the PKGBUILD, which (among other things) fetches the code from somewhere else (tarball, git commit, ...), and the PKGBUILD alone is not very useful without that code. Therefore, it looks like we should implement something like T3923, to fetch the actual code.
franckbret updated this revision to Diff 28971.Jun 28 2022, 12:48 PM
Updating D8033: [WIP] Arch User Repository (AUR) lister



Fix an issue with requests usage when downloading packages archives ensuring it does not decode the binary directly. (tests and CI were fine, but got the bug while testing runner on docker)
swh-public-ci added a comment.Jun 28 2022, 12:52 PM
Build is green



Patch application report for D8033 (id=28971)



Rebasing onto 1bf11aa26d...



Current branch diff-target is up to date.





Changes applied before test



commit 4729b0aae165aab640677adefc2f6ceb90072bcd
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/551/ for more details.
Harbormaster completed remote builds in B30059: Diff 28971.Jun 28 2022, 12:52 PM
vlorentz added a comment.Jun 28 2022, 1:17 PM


In D8033#209040, @franckbret wrote:


but got the bug while testing runner on docker





what was the bug, exactly? I don't see how the new code changes the behavior.



Also, please remove Updating D8033: [WIP] Arch User Repository (AUR) lister from change descriptions; it's redundant and makes the "History" tab of https://forge.softwareheritage.org/D8033#toc less readable
franckbret retitled this revision from [WIP] Arch User Repository (AUR) lister to Arch User Repository (AUR) lister.Jun 29 2022, 3:03 PM
franckbret added a comment.Jun 29 2022, 3:11 PM


In D8033#209048, @vlorentz wrote:




In D8033#209040, @franckbret wrote:


but got the bug while testing runner on docker





what was the bug, exactly? I don't see how the new code changes the behavior.



Also, please remove Updating D8033: [WIP] Arch User Repository (AUR) lister from change descriptions; it's redundant and makes the "History" tab of https://forge.softwareheritage.org/D8033#toc less readable





@vlorentz The change force requests.get to not automatically decompress the response. This way we are sure we always get the raw archive as is. See https://requests.readthedocs.io/en/latest/user/quickstart/#raw-response-content
ardumont added inline comments.Jun 30 2022, 10:37 AM
swh/lister/aur/lister.py


22
No need if you always return one element?


32


35


49
I gather that's a working directory. If so, you'll need to have some post-listing routine cleanup.


107–122
I've also adapted according to my type suggestion early on.
ardumont added inline comments.Jun 30 2022, 11:18 AM
swh/lister/aur/lister.py


22
i mean, your code below returns page of one element if i read it correctly.


124–139
in effect, you only read one origin per "page".
ardumont added a comment.Jul 12 2022, 9:38 AM
@franckbret ping on earlier comments
franckbret mentioned this in D8101: crates.lister: Implement incremental mode.Jul 19 2022, 3:44 PM
franckbret updated this revision to Diff 29383.Jul 19 2022, 4:32 PM
franckbret marked 7 inline comments as done.
Some typo and consistency fixes after code review
swh/lister/aur/lister.py


22
yep
franckbret added inline comments.Jul 19 2022, 4:35 PM
swh/lister/aur/lister.py


49
Yes It is. Have you an example of post-listing routine action?

Same pattern used for arch and crates, I guess I need to do cleanup to
swh-public-ci added a comment.Jul 19 2022, 4:36 PM
Build is green



Patch application report for D8033 (id=29383)



Rebasing onto 1bf11aa26d...



Current branch diff-target is up to date.





Changes applied before test



commit d4851a18c7bf97dcfffdb8362dcf4c6a720bde02
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/554/ for more details.
Harbormaster completed remote builds in B30457: Diff 29383.Jul 19 2022, 4:36 PM
franckbret added inline comments.Jul 20 2022, 9:52 AM
swh/lister/aur/lister.py


49
Is finalize() suitable for that purpose?
vlorentz added inline comments.Jul 20 2022, 10:10 AM
swh/lister/aur/lister.py


49
yes
franckbret updated this revision to Diff 29732.Aug 16 2022, 12:33 PM
Split artifacts data to artifacts and aur_metadata
swh-public-ci added a comment.Aug 16 2022, 12:37 PM
Build is green



Patch application report for D8033 (id=29732)



Rebasing onto cee6bcb514...



First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister





Changes applied before test



commit a33fbf745785ebe179bce4fbbbbb52a17fabce65
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/586/ for more details.
Harbormaster completed remote builds in B30784: Diff 29732.Aug 16 2022, 12:37 PM
franckbret updated this revision to Diff 29755.Aug 16 2022, 12:43 PM
Add a finalize method that remove temporary directory once the lister complete
swh-public-ci added a comment.Aug 16 2022, 1:11 PM
Build is green



Patch application report for D8033 (id=29755)



Rebasing onto cee6bcb514...



First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister





Changes applied before test



commit 280be6e9530238a385843ee09ee5040ca0d60473
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/587/ for more details.
Harbormaster completed remote builds in B30807: Diff 29755.Aug 16 2022, 1:11 PM
franckbret updated this revision to Diff 29760.Aug 16 2022, 3:30 PM
Add a directory cleanup test
swh-public-ci added a comment.Aug 16 2022, 3:35 PM
Build is green



Patch application report for D8033 (id=29760)



Rebasing onto cee6bcb514...



First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister





Changes applied before test



commit 3e0c54bf111c316ead8a6804060f3432d8f3209f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/588/ for more details.
Harbormaster completed remote builds in B30812: Diff 29760.Aug 16 2022, 3:35 PM
franckbret updated this revision to Diff 29766.Aug 16 2022, 5:28 PM
franckbret marked 2 inline comments as done.
Document the module
swh-public-ci added a comment.Aug 16 2022, 5:33 PM
Build has FAILED



Patch application report for D8033 (id=29766)



Rebasing onto cee6bcb514...



First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister





Changes applied before test



commit aeb60cc266f0fdf11e8650c9f7710ec00cf2443f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/589/

See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/589/console
Harbormaster failed remote builds in B30818: Diff 29766!Aug 16 2022, 5:33 PM
franckbret updated this revision to Diff 29767.Aug 16 2022, 6:12 PM
Fix documentation build
swh-public-ci added a comment.Aug 16 2022, 6:16 PM
Build has FAILED



Patch application report for D8033 (id=29767)



Rebasing onto cee6bcb514...



First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister





Changes applied before test



commit 0b722b297b17060de8eca33bc7811039e0bbd3df
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/590/

See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/590/console
Harbormaster failed remote builds in B30819: Diff 29767!Aug 16 2022, 6:16 PM
franckbret updated this revision to Diff 29768.Aug 16 2022, 7:11 PM
Fix docstrings errors
swh-public-ci added a comment.Aug 16 2022, 7:16 PM
Build is green



Patch application report for D8033 (id=29768)



Rebasing onto cee6bcb514...



First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister





Changes applied before test



commit 3962b637421577c9fec8a0f0f1c8c7661b1bfa88
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/591/ for more details.
Harbormaster completed remote builds in B30820: Diff 29768.Aug 16 2022, 7:16 PM
vlorentz added a comment.Aug 17 2022, 11:22 AM
Looks good! Two more nitpicks before this is good to merge:
swh/lister/aur/__init__.py


20–23
I don't think that's accurate; we do archive split packages, but only their "pkgbase" because that is the only one that actually has source code.


77–96
why is "version" in both?
franckbret updated this revision to Diff 29781.Aug 17 2022, 11:53 AM
Rephrase the documentation part about regular and split packages
swh-public-ci added a comment.Aug 17 2022, 11:57 AM
Build is green



Patch application report for D8033 (id=29781)



Rebasing onto 6a53a6ad06...



First, rewinding head to replay your work on top of it...
Applying: Arch User Repository (AUR) lister





Changes applied before test



commit 96683382dc91d6927998797ba5a3cd609319eed4
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/594/ for more details.
Harbormaster completed remote builds in B30833: Diff 29781.Aug 17 2022, 11:57 AM
franckbret marked an inline comment as done.Aug 17 2022, 12:01 PM
franckbret added inline comments.
swh/lister/aur/__init__.py


20–23
I rephrase it, does it looks ok for you?


77–96
Because its the associative key for lines for both dict which is needed for the loader the get the corresponding informations.

Btw there will be only one line in the case of aur. I think it's more explicit to keep it as is, but if you want me to remove it, I sure can.
franckbret marked an inline comment as done.Aug 18 2022, 4:05 PM
Can we merge this one?
vlorentz accepted this revision.Aug 19 2022, 10:30 AM
vlorentz added inline comments.
swh/lister/aur/__init__.py


77–96
nevermind, it's fine
This revision is now accepted and ready to land.Aug 19 2022, 10:30 AM
franckbret updated this revision to Diff 29847.Aug 19 2022, 12:45 PM
Rebase on master
swh-public-ci added a comment.Aug 19 2022, 12:49 PM
Build is green



Patch application report for D8033 (id=29847)



Rebasing onto 6a53a6ad06...



Current branch diff-target is up to date.





Changes applied before test



commit 97b353bf0b9a726ebb3d414f809aeac26a229f21
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Jun 24 12:19:15 2022 +0200

    Arch User Repository (AUR) lister
    
    Add 'aur' module to swh-lister with data fixtures and tests.
    For now, origin url are package vcs (Git) url.



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/597/ for more details.
Harbormaster completed remote builds in B30899: Diff 29847.Aug 19 2022, 12:49 PM
Closed by commit rDLS97b353bf0b9a: Arch User Repository (AUR) lister (authored by franckbret).  ·  Explain WhyAug 19 2022, 12:51 PM
This revision was automatically updated to reflect the committed changes.
franckbret added a commit: rDLS97b353bf0b9a: Arch User Repository (AUR) lister.
franckbret mentioned this in T4466: Ingest Aur repository (Arch User Repository).Aug 26 2022, 4:30 PM
Revision Contents
Changeset List
PathSize
1 line
swh/
lister/
aur/
12 lines
139 lines
19 lines
tests/
3 lines
data/
27 lines
https_aur.archlinux.org/
104 lines
31 lines
Diff 28931
View Options
setup.py
Loading...
View Options
swh/lister/aur/__init__.py
Loading...
View Options
swh/lister/aur/lister.py
Loading...
View Options
swh/lister/aur/tasks.py
Loading...
View Options
swh/lister/aur/tests/__init__.py
Loading...
View Options
swh/lister/aur/tests/data/fake_aur_packages.sh
Loading...
View Options
swh/lister/aur/tests/data/https_aur.archlinux.org/packages-meta-v1.json.gz
Loading...
View Options
swh/lister/aur/tests/test_lister.py
Loading...
View Options
swh/lister/aur/tests/test_tasks.py
Loading...
About · Developer information · Legal