Created these in Hex.pm official github repo:

• PR to add order param in /packages API
• Issue for missing inserted_at values in release metadata. A corresponding PR was created by the maintainer to fix this issue. This shall be helpful for the hex.pm loader.

order sounds best. Do you want to do it?

but if you change it, then pagination is unusable because it's offset-based.

• Fix license headers
• Fix test cases to match Fedora lister output

• Use intrinsic version (and other suggestions by @anlambert)
• Update tests for the same

• Rebase against master

I have a question: The .tar files obtained from this have a contents.tar.gz which contains the real source code. Is there a way to extract this internal .tar file using the current tar loader?

• Rebase against master

Minor fixes in the loader docstrings

• Add tests for handling of HTTP errors and sha1 checksum (increase test coverage)

• Extract .tar.gz as a seperate branch (and other suggestions made by @anlambert)
• Remove .tar.gz extraction logic from extract_rpm_package function. Previously, I was just replacing .tar.gz with its extracted folder but now we are creating a separate branch as well.
• Updating relevant tests for the same

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

• Send origins in the last iteration
• Change origin url format
• Use intrinsic version for pkg version key

• Improvements suggested by @anlambert

• remove unused code

• remove ununsed code
• rebase with master

feat: Remove microseconds from buildTime metadata to match real values from fedora lister

feat: Parse buildTime as UTC and store in ISO format

• Use list comprehension for side_effect
• Trigger rebuild

feat: Add edition and build time in package metadata

• Squash commits
• Improve comments, fix vars, and add docs

• feat: Make the lister incremental and use build_time as release date
• tests: Refactor to use fewer lines of code and improve readability

• feat: Use subprocess.check_call and extract .tar obtained from .rpm
• Remove extra TODOs and FIXMEs
• Add missing copyrights

• feat: Use subprocess.check_call and extract .tar obtained from .rpm

P1090 should be helpful

• refactor(fedora): Improve helper functions in test_lister.py
• squash commits

tests: Mock urllib.request.urlopen instead of repomd.load function

• feat: Add release field in extra_loader_args and hardcode test cases
• rebase against master

• feat: Add release field in extra_loader_args and hardcode test cases

Support listing multiple editions and old releases via archives.fedoraprojects.org

• Add support for listing multiple editions based on release
• Migrate to archives.fedoraproject.org (contains rpms for old releases)
• Seperate full and incremental listing tests

Please keep it open since the search isn't fast enough at the moment.

Some observations:

• There are multiple dirs in the fedora index where I see .rpm files. Ex: /releases/35/Everything/x86_64/debug/tree/ and /releases/35/Everything/source/tree/. I verified the .rpm file names and many of them are different. So I need to extend this lister to cover all such dirs.
• Default URL of the lister should be changed to archives.fedoraproject.org since the folders related to earlier releases of fedora are empty in the current index.

• gitea: fix test name
• squash and reword commits

• gitea: skip on status code 500

@ardumont because of T4423 we will face 500 for https://try.gogs.io/api/v1/repos/search?page=17 no matter how many times we retry (until it is fixed by Gogs maintainers). So should we not ignore the page and move on to the next one?

thanks. will try to reproduce this locally.

• test_lister: remove unused code
• squash commits

• remove defusedxml from requirements.txt
• use docstrings for lister vars
• directly pass default value of releases
• fix typo in tasks.py
• compare scheduler_origins with pkg names

• refactor(fedora): Improve vars, update comments, and update tests

Squash commits and rebase

• Introduce 4th page in incremental listing test
• Improve comments

• Introduce last_seen_repo_id in GogsListerState to properly set the value of self.updated in`finalize()`
• Update incremental listing test to test happy flow (instead of the buggy T4423)

test: fix test failures

• make last_seen_next_link = None if the last page is encountered
• in test_gogs_incremental_lister, throw 400 in the 1st listing attempt and succeed in the 2nd one.

fix: Improvements for edge cases and coverage

• state.last_seen_next_link should be None if last page is encountered (no next link)
• assert page.repos is not None in get_origins_from_page
• Improve comments and formatting

test: Add test for incremental gogs lister and improve existing tests

• Add test for incremental listing in gogs
• Use 3 mocked pages instead of 2 in existing tests
• In test_gogs_list_http_error, successfully parse 1 page before throwing HTTPError
• Minor improvement in variables names (P1, P2, P3)

• test: Improve variable names and comments in gogs lister tests

worth opening a dedicated forge issue

Done. T4423

I just ran the lister using swh lister run -l gogs url=https://try.gogs.io/api/v1/ api_token=xxx

• minor update to GogsLister docstring
• squash commit (again)

• update GogsLister docstring
• squash commits