Page MenuHomeSoftware Heritage

aur: Change origin URL + add git origin + reduce disk writes
ClosedPublic

Authored by anlambert on Sep 2 2022, 1:49 PM.

Details

Summary

Simplify code for downloading packages index as gzip and deflate
transfer-encodings are automatically decoded by requests, also
do not stream response for a couple of megabytes.

Use https://aur.archlinux.org/packages/{package_name} as origin
URL and also yield a git origin for each package repository URL.

Add more debug logs to track lister execution.

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D8383 (id=30256)

Rebasing onto 7638f2028b...

Current branch diff-target is up to date.
Changes applied before test
commit 87e31bb8922b46ec6dceb55dc2eab367ed7e2843
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 2 13:44:19 2022 +0200

    aur: Improve lister implementation
    
    Simplify code for downloading packages index as gzip and deflate
    transfer-encodings are automatically decoded by requests, also
    do not stream response for a couple of megabytes.
    
    Use https://aur.archlinux.org/packages/{package_name} as origin
    URL and also yield a git origin for each package repository URL.
    
    Add more debug logs to track lister execution.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/639/ for more details.

Simplify code for downloading packages index as gzip and deflate
transfer-encodings are automatically decoded by requests, also
do not stream response for a couple of megabytes.

Use https://aur.archlinux.org/packages/{package_name} as origin
URL and also yield a git origin for each package repository URL.

this should be two different commits.

Anyway, please change the diff/commit title to make it clear it changes the produced origin URLs (it's not just an implementation change). eg.

  1. aur: Change origin URL pattern
  2. aur: Store package index in memory instead of disk
swh/lister/aur/lister.py
67–68

Update: Split changes into multiple commits:

  • aur: Store packages index in memory instead of disk
  • aur: Mofify origin URL for listed packages
  • aur: Create also a git origin for each listed package repository
This revision is now accepted and ready to land.Sep 2 2022, 3:51 PM
vlorentz retitled this revision from aur: Improve lister implementation to aur: Change origin URL + add git origin + reduce disk writes.Sep 2 2022, 3:52 PM

Build is green

Patch application report for D8383 (id=30261)

Rebasing onto 7638f2028b...

Current branch diff-target is up to date.
Changes applied before test
commit 23375f8720fbb2874f974081c2cb4bd3586180fc
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 2 15:45:07 2022 +0200

    aur: Create also a git origin for each listed package repository
    
    It will enable to archive the history of the PKGBUILD file associated
    to the AUR package.

commit 54cbaaca437c3f2b6b428262c5a2020deab40c1f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 2 15:42:01 2022 +0200

    aur: Mofify origin URL for listed packages
    
    Use https://aur.archlinux.org/packages/<package_name> instead
    of https://aur.archlinux.org/<package_name>.git

commit 92baa2b45c2436636bdcf6c0708dee573c21ffae
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 2 15:36:20 2022 +0200

    aur: Store packages index in memory instead of disk
    
    Simplify code for downloading packages index as gzip and deflate
    transfer-encodings are automatically decoded by requests, also
    do not stream response for a couple of megabytes and store
    HTTP responses in memory.
    
    Also add more debug logs to track lister execution.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/640/ for more details.

Build is green

Patch application report for D8383 (id=30262)

Rebasing onto 7638f2028b...

Current branch diff-target is up to date.
Changes applied before test
commit b6c69e5075bb742be89f02ee9321c4ce969dec0a
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 2 15:45:07 2022 +0200

    aur: Create also a git origin for each listed package repository
    
    It will enable to archive the history of the PKGBUILD file associated
    to the AUR package.

commit d76fbb3447b25a21ba44e6b6f60c223eabfdc554
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 2 15:42:01 2022 +0200

    aur: Modify origin URL for listed packages
    
    Use https://aur.archlinux.org/packages/<package_name> instead
    of https://aur.archlinux.org/<package_name>.git

commit 92baa2b45c2436636bdcf6c0708dee573c21ffae
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 2 15:36:20 2022 +0200

    aur: Store packages index in memory instead of disk
    
    Simplify code for downloading packages index as gzip and deflate
    transfer-encodings are automatically decoded by requests, also
    do not stream response for a couple of megabytes and store
    HTTP responses in memory.
    
    Also add more debug logs to track lister execution.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/641/ for more details.