Details

Reviewers

vlorentz
anlambert

Group Reviewers

Reviewers

Maniphest Tasks

T4448: Implementation of Fedora Lister

Commits

rDLS6ad61aec2394: feat(fedora): Introduce fedora lister

Summary

Lister to ingest fedora mirrors (.rpm)

Diff Detail

Repository

rDLS Listers

Branch

feat/fedora-lister

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 31309
Build 48976: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 48975: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	0 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.fedora.lister .tox/py3/lib/python3.7/site-packages/swh/lister/fedora/lister.py:12: in <module> import repomd E ModuleNotFoundError: No module named 'repomd'
	0 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.fedora.tasks .tox/py3/lib/python3.7/site-packages/swh/lister/fedora/tasks.py:9: in <module> from .lister import FedoraLister .tox/py3/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
	0 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.fedora.tests.test_lister .tox/py3/lib/python3.7/site-packages/swh/lister/fedora/tests/test_lister.py:8: in <module> import defusedxml E ModuleNotFoundError: No module named 'defusedxml'

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

feat: Add release field in extra_loader_args and hardcode test cases
rebase against master

Build is green

Patch application report for D8386 (id=31284)

Rebasing onto 05cd1de1cd...

Current branch diff-target is up to date.

Changes applied before test

commit f438b781c6cbee7d2d6d453b399d0da9ca38ad1f
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 12 11:41:06 2022 +0530

    feat: Add release field in extra_loader_args and hardcode test cases
    
    fix: Minor changes to polish the code

commit 97e7075fe9dfcc845a62ee6c58342152680ec376
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Fri Oct 7 11:12:11 2022 +0530

    feat: Support listing multiple editions and old releases via archives.fedoraprojects.org

commit 07a09440f5030d990ba4a881eb1d76c124b33eee
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/786/ for more details.

Harbormaster completed remote builds in B32235: Diff 31284.Oct 12 2022, 8:34 AM

KShivendu added inline comments.Oct 12 2022, 10:42 AM

swh/lister/fedora/tests/test_lister.py
33	I tried but stopped since it would be ugly since `repomd` uses `urllib.request.urlopen` instead of `requests` library (`requests_mock` doesn't work) If you think it's still worth the shot, I'll do it :)
159–161	Changing `pkg._version_info.set("ver", "1.0.0")` affected it (not `pkg_versions`)

vlorentz added inline comments.Oct 12 2022, 10:52 AM

swh/lister/fedora/tests/test_lister.py
33	yes, please. There are already examples of it in `swh-loader-core/swh/loader/package/tests/test_utils.py` and `swh-storage/swh/storage/tests/migrate_extrinsic_metadata/test_pypi.py`
159–161	Ah, I see. Thanks for the extra comments

tests: Mock urllib.request.urlopen instead of repomd.load function

Build is green

Patch application report for D8386 (id=31312)

Rebasing onto 05cd1de1cd...

Current branch diff-target is up to date.

Changes applied before test

commit 3b7da8cdaec22abf3ed181d44b46b53cf2d384c0
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/789/ for more details.

Harbormaster completed remote builds in B32265: Diff 31312.Oct 13 2022, 7:38 AM

refactor(fedora): Improve helper functions in test_lister.py
squash commits

Build is green

Patch application report for D8386 (id=31313)

Rebasing onto 05cd1de1cd...

Current branch diff-target is up to date.

Changes applied before test

commit b6d3a5e8f512a31818e3b051513dc9fa37bba4ac
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/790/ for more details.

Harbormaster completed remote builds in B32266: Diff 31313.Oct 13 2022, 8:06 AM

lgtm, but we still need to reach a decision on using rpm:// for origin URLs. I opened T4632 so it does not block this diff.

swh/lister/fedora/tests/test_lister.py
21–26

feat: Add edition and build time in package metadata

Build has FAILED

Patch application report for D8386 (id=31776)

Rebasing onto 60707a45dd...

First, rewinding head to replay your work on top of it...
Applying: feat(fedora): Introduce fedora lister

Changes applied before test

commit 57cf513c510b502aa042bef6f4ebf3f5130054e0
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/849/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/849/console

Harbormaster failed remote builds in B32729: Diff 31776!Nov 6 2022, 7:16 AM

Use list comprehension for side_effect
Trigger rebuild

Build has FAILED

Patch application report for D8386 (id=31777)

Rebasing onto 60707a45dd...

First, rewinding head to replay your work on top of it...
Applying: feat(fedora): Introduce fedora lister

Changes applied before test

commit 80a8c517a3362b7f68e2f0539ecb1df24ebdeebe
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/850/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/850/console

Harbormaster failed remote builds in B32730: Diff 31777!Nov 6 2022, 8:13 AM

feat: Parse buildTime as UTC and store in ISO format

Build is green

Patch application report for D8386 (id=31801)

Rebasing onto e8699422d7...

First, rewinding head to replay your work on top of it...
Applying: feat(fedora): Introduce fedora lister

Changes applied before test

commit b374e5fb9c6bb0a3664799329e35cdbdb5bfd36a
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/851/ for more details.

Harbormaster completed remote builds in B32742: Diff 31801.Nov 8 2022, 12:48 PM

remove ununsed code
rebase with master

remove unused code

Build is green

Patch application report for D8386 (id=31803)

Rebasing onto e8699422d7...

Current branch diff-target is up to date.

Changes applied before test

commit 7de16c7a8828a85f55d986524836f6e2d56b07e2
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/852/ for more details.

Harbormaster completed remote builds in B32744: Diff 31803.Nov 8 2022, 1:41 PM

Build is green

Patch application report for D8386 (id=31804)

Rebasing onto e8699422d7...

Current branch diff-target is up to date.

Changes applied before test

commit f442e9191538aae3aaaf646089d90319fe6e84af
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/853/ for more details.

Harbormaster completed remote builds in B32745: Diff 31804.Nov 8 2022, 1:43 PM

@KShivendu , after testing the lister in docker environment there is room for improvements before we can accept that diff.

I added a bunch of inline comments about what should be changed.

I also paste below the diff of my changes after hacking on that diff.

diff --git a/swh/lister/fedora/lister.py b/swh/lister/fedora/lister.py
index 7552bc0..96180b6 100644
--- a/swh/lister/fedora/lister.py
+++ b/swh/lister/fedora/lister.py
@@ -1,4 +1,4 @@
-# Copyright (C) 2017-2021 The Software Heritage developers
+# Copyright (C) 2022 The Software Heritage developers
 # See the AUTHORS file at the top-level directory of this distribution
 # License: GNU General Public License version 3, or any later version
 # See top-level LICENSE file for more information
@@ -7,6 +7,7 @@ from dataclasses import dataclass, field
 from datetime import datetime, timezone
 import logging
 from typing import Any, Dict, Iterator, List, Set, Type
+from urllib.error import HTTPError
 from urllib.parse import urljoin
 
 import repomd
@@ -90,15 +91,8 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
         )
 
         self.releases = releases
-
         self.listed_origins: Dict[FedoraOrigin, ListedOrigin] = {}
         "will hold all listed origins info"
-        self.sent_origins: Set[FedoraOrigin] = set()
-        """will contain origin urls that have already been listed
-         in a previous page (fedora release)"""
-        self.origins_to_update: Dict[FedoraOrigin, ListedOrigin] = {}
-        """will contain already listed package info that need to be
-         sent to the scheduler for update in the commit_page method"""
         self.package_versions: Dict[PkgName, Set[PkgVersion]] = {}
         "will contain the lister state after a call to run"
 
@@ -114,11 +108,11 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
             self.url,
             f"{release}/{edition}/source/SRPMS/"
             if release < 24
-            else f"{release}/Everything/source/tree/",
+            else f"{release}/{edition}/source/tree/",
         )
 
         repo = repomd.load(index_url)  # throws error if no repomd.xml is not found
-        logging.debug(
+        logger.debug(
             "Fetched metadata from url: %s, found %d packages", index_url, len(repo)
         )
         # TODO: Extract more fields like "provides" and "requires" from *primary.xml
@@ -133,18 +127,27 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
                 logger.debug("Listing fedora release %s edition %s", release, edition)
                 self.current_release = release
                 self.current_edition = edition
-                yield self.page_request(release, edition)
+                try:
+                    yield self.page_request(release, edition)
+                except HTTPError as http_error:
+                    if http_error.getcode() == 404:
+                        logger.debug(
+                            "No packages metadata found for fedora release %s edition %s",
+                            release,
+                            edition,
+                        )
+                        continue
+                    raise
 
     def origin_url_for_package(self, package_name: PkgName) -> FedoraOrigin:
         """Return the origin url for the given package"""
-        return f"rpm://{self.instance}/packages/{package_name}"
+        return f"https://packages.fedoraproject.org/pkgs/{package_name}"
 
     def get_origins_from_page(self, page: FedoraPageType) -> Iterator[ListedOrigin]:
         """Convert a page of fedora package sources into an iterator of ListedOrigin."""
         assert self.lister_obj.id is not None
 
         origins_to_send = {}
-        self.origins_to_update = {}
 
         # iterate on each package's metadata
         for pkg_metadata in page:
@@ -171,6 +174,7 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
                     url=origin_url,
                     visit_type="rpm",
                     extra_loader_arguments={"packages": {}},
+                    last_update=package_build_time,
                 )
 
                 # origin will be yielded at the end of that method
@@ -178,12 +182,7 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
                 # init set that will contain all listed package versions
                 self.package_versions[package_name] = set()
 
-            # package has already been listed in a previous or current page (release)
-            elif origin_url not in origins_to_send:
-                # if package has been listed in previously, its new versions
-                # will be added to its ListedOrigin object but the update will
-                # be sent to the scheduler in the commit_page method
-                self.origins_to_update[origin_url] = self.listed_origins[origin_url]
+            origins_to_send[origin_url] = self.listed_origins[origin_url]
 
             # update package versions data in parameter that will be provided
             # to the rpm loader
@@ -198,16 +197,25 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
                 "buildTime": package_build_time.isoformat(),
             }
 
+            if package_build_time > self.listed_origins[origin_url].last_update:
+                self.listed_origins[origin_url].last_update = package_build_time
+
             # add package version key to the set of found versions
             self.package_versions[package_name].add(package_version_key)
 
-        # update already counted origins with changes since last page
-        self.sent_origins.update(origins_to_send.keys())
+            # package has already been listed during a previous listing process
+            if package_name in self.state.package_versions:
+                new_versions = (
+                    self.package_versions[package_name]
+                    - self.state.package_versions[package_name]
+                )
+                # no new versions so far, no need to send the origin to the scheduler
+                if not new_versions:
+                    origins_to_send.pop(origin_url, None)
 
         logger.debug(
-            "Found %s new packages, %s packages with new versions.",
+            "Found %s packages to update (new ones or packages with new versions).",
             len(origins_to_send),
-            len(self.origins_to_update),
         )
         logger.debug(
             "Current total number of listed packages is equal to %s.",
@@ -216,15 +224,7 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
 
         yield from origins_to_send.values()
 
-    def get_origins_to_update(self) -> Iterator[ListedOrigin]:
-        yield from self.origins_to_update.values()
-
-    def commit_page(self, page: FedoraPageType):
-        """Send to scheduler already listed origins where new versions have been found
-        in current page."""
-        self.send_origins(self.get_origins_to_update())
-
     def finalize(self):
         # set mapping between listed package names and versions as lister state
         self.state.package_versions = self.package_versions
-        self.updated = len(self.sent_origins) > 0
+        self.updated = len(self.listed_origins) > 0
diff --git a/swh/lister/fedora/tasks.py b/swh/lister/fedora/tasks.py
index 250472c..18c8a60 100644
--- a/swh/lister/fedora/tasks.py
+++ b/swh/lister/fedora/tasks.py
@@ -2,7 +2,7 @@
 # License: GNU General Public License version 3, or any later version
 # See top-level LICENSE file for more information
 
-from typing import Dict, List, Optional
+from typing import Dict
 
 from celery import shared_task
 
@@ -10,17 +10,9 @@ from .lister import FedoraLister
 
 
 @shared_task(name=__name__ + ".FullFedoraRelister")
-def list_fedora_full(
-    url: str,
-    instance: Optional[str] = None,
-    releases: List[str] = None,
-) -> Dict[str, int]:
+def list_fedora_full(**lister_args) -> Dict[str, int]:
     """Full update of a Fedora instance"""
-    lister = FedoraLister.from_configfile(
-        url=url,
-        instance=instance,
-        releases=releases,
-    )
+    lister = FedoraLister.from_configfile(**lister_args)
     return lister.run().dict()
 
 
diff --git a/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz b/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz
index fa5b748..2e4e030 100644
Binary files a/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz and b/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz differ
diff --git a/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36.xml.gz b/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36.xml.gz
index 3d9afb7..b42ef89 100644
Binary files a/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36.xml.gz and b/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36.xml.gz differ
diff --git a/swh/lister/fedora/tests/test_lister.py b/swh/lister/fedora/tests/test_lister.py
index 3936615..4552daf 100644
--- a/swh/lister/fedora/tests/test_lister.py
+++ b/swh/lister/fedora/tests/test_lister.py
@@ -37,7 +37,7 @@ def rpm_url(release, path):
 @pytest.fixture
 def pkg_versions():
     return {
-        "rpm://fedora/packages/0install": {
+        "https://packages.fedoraproject.org/pkgs/0install": {
             "26/Everything/2.11": {
                 "name": "0install",
                 "version": "2.11",
@@ -47,7 +47,7 @@ def pkg_versions():
                 "url": rpm_url(26, "0/0install-2.11-4.fc26.src.rpm"),
             }
         },
-        "rpm://fedora/packages/0xFFFF": {
+        "https://packages.fedoraproject.org/pkgs/0xFFFF": {
             "36/Everything/0.9": {
                 "name": "0xFFFF",
                 "version": "0.9",
@@ -65,7 +65,7 @@ def pkg_versions():
                 "url": rpm_url(26, "0/0xFFFF-0.3.9-15.fc26.src.rpm"),
             },
         },
-        "rpm://fedora/packages/2ping": {
+        "https://packages.fedoraproject.org/pkgs/2ping": {
             "36/Everything/4.5.1": {
                 "name": "2ping",
                 "version": "4.5.1",
@@ -79,13 +79,17 @@ def pkg_versions():
 
 
 def run_lister(
-    lister: FedoraLister,
     swh_scheduler: SchedulerInterface,
     releases: List[Release],
     pkg_versions: dict,
     origin_count: int,
 ):
     """Runs the lister and tests that the listed origins are correct."""
+    lister = FedoraLister(
+        scheduler=swh_scheduler,
+        releases=releases,
+    )
+
     stats = lister.run()
     scheduler_origins = swh_scheduler.get_listed_origins(lister.lister_obj.id).results
     lister_state = lister.get_state_from_scheduler()
@@ -120,15 +124,11 @@ def test_full_lister_fedora(
     """
     releases = [26, 36]
 
-    lister = FedoraLister(
-        scheduler=swh_scheduler,
-        releases=releases,
-    )
     get_editions_patch = mocker.patch("swh.lister.fedora.lister.get_editions")
     get_editions_patch.return_value = ["Everything"]
 
     mock_repomd(datadir, mocker)
-    run_lister(lister, swh_scheduler, releases, pkg_versions, origin_count=3)
+    run_lister(swh_scheduler, releases, pkg_versions, origin_count=3)
 
 
 def test_incremental_lister(
@@ -141,25 +141,22 @@ def test_incremental_lister(
     Simulates an incremental listing of packages for fedora releases.
     """
     releases = [26, 36]
-    lister = FedoraLister(
-        scheduler=swh_scheduler,
-        releases=releases,
-    )
+
     get_editions_patch = mocker.patch("swh.lister.fedora.lister.get_editions")
     get_editions_patch.return_value = ["Everything"]
 
     # First run
     mock_repomd(datadir, mocker)
-    run_lister(lister, swh_scheduler, releases, pkg_versions, origin_count=3)
+    run_lister(swh_scheduler, releases, pkg_versions, origin_count=3)
     # Second run (no updates)
     mock_repomd(datadir, mocker)
-    run_lister(lister, swh_scheduler, releases, pkg_versions, origin_count=0)
+    run_lister(swh_scheduler, releases, pkg_versions, origin_count=0)
 
     # Use an altered version of primary36.xml in which we updated the version
     # of package 0xFFFF to 0.10:
     mock_repomd(datadir, mocker, use_altered_fedora36=True)
     # Add new version to the set of expected pkg versions:
-    pkg_versions["rpm://fedora/packages/0xFFFF"].update(
+    pkg_versions["https://packages.fedoraproject.org/pkgs/0xFFFF"].update(
         {
             "36/Everything/0.10": {
                 "name": "0xFFFF",
@@ -168,10 +165,10 @@ def test_incremental_lister(
                 "edition": "Everything",
                 "buildTime": "2022-01-19T19:13:53+00:00",
                 # .rpm URL remains same as per primary.xml file:
-                "url": rpm_url(36, "0/0xFFFF-0.9-4.fc36.src.rpm"),
+                "url": rpm_url(36, "0/0xFFFF-0.10-1.fc36.src.rpm"),
             }
         }
     )
 
     # Third run (0xFFFF in fedora36 editions got updated, but no new origins were found)
-    run_lister(lister, swh_scheduler, releases, pkg_versions, origin_count=0)
+    run_lister(swh_scheduler, releases, pkg_versions, origin_count=1)
diff --git a/swh/lister/fedora/tests/test_tasks.py b/swh/lister/fedora/tests/test_tasks.py
index 2ac04f7..7fd4236 100644
--- a/swh/lister/fedora/tests/test_tasks.py
+++ b/swh/lister/fedora/tests/test_tasks.py
@@ -32,9 +32,7 @@ def test_full_listing(lister, swh_scheduler_celery_app, swh_scheduler_celery_wor
     res.wait()
     assert res.successful()
 
-    actual_kwargs = dict(**kwargs, instance=None, releases=None)
-
-    lister.from_configfile.assert_called_once_with(**actual_kwargs)
+    lister.from_configfile.assert_called_once_with(**kwargs)
     lister.run.assert_called_once_with()

Please note that I have also modified the gzipped primary36-altered.xml file the following way:

<?xml version="1.0" encoding="UTF-8"?>
<metadata xmlns="http://linux.duke.edu/metadata/common" xmlns:rpm="http://linux.duke.edu/metadata/rpm" packages="14">
  <package type="rpm">
    <name>0xFFFF</name>
    <arch>src</arch>
    <version epoch="0" ver="0.10" rel="4.fc36" />
    <checksum type="sha256" pkgid="YES">45eee8d990d502324ae665233c320b8a5469c25d735f1862e094c1878d6ff2cd</checksum>
    <summary>The Open Free Fiasco Firmware Flasher</summary>
    <description>The 'Open Free Fiasco Firmware Flasher' aka 0xFFFF utility implements
a free (GPL3) userspace handler for the NOLO bootloader and related
utilities for the Nokia Internet Tablets like flashing setting device
options, packing/unpacking FIASCO firmware format and more.</description>
    <packager>Fedora Project</packager>
    <url>https://talk.maemo.org/showthread.php?t=87996</url>
    <time file="1643111870" build="1642619633" />
    <size package="73742" installed="69776" archive="70152" />
    <location href="Packages/0/0xFFFF-0.10-1.fc36.src.rpm" />
    <format>
      <rpm:license>GPLv3</rpm:license>
      <rpm:vendor>Fedora Project</rpm:vendor>
      <rpm:group>Unspecified</rpm:group>
      <rpm:buildhost>buildvm-a32-17.iad2.fedoraproject.org</rpm:buildhost>
      <rpm:sourcerpm></rpm:sourcerpm>
      <rpm:header-range start="4504" end="7345" />
      <rpm:provides>
        <rpm:entry name="0xFFFF" flags="EQ" epoch="0" ver="0.10" rel="4.fc36" />
      </rpm:provides>
      <rpm:requires>
        <rpm:entry name="gcc" />
        <rpm:entry name="libusb-devel" />
        <rpm:entry name="make" />
      </rpm:requires>
    </format>
  </package>
  <package type="rpm">
    <name>0xFFFF</name>
    <arch>src</arch>
    <version epoch="0" ver="0.9" rel="4.fc36" />
    <checksum type="sha256" pkgid="YES">45eee8d990d502324ae665233c320b8a5469c25d735f1862e094c1878d6ff2cd</checksum>
    <summary>The Open Free Fiasco Firmware Flasher</summary>
    <description>The 'Open Free Fiasco Firmware Flasher' aka 0xFFFF utility implements
a free (GPL3) userspace handler for the NOLO bootloader and related
utilities for the Nokia Internet Tablets like flashing setting device
options, packing/unpacking FIASCO firmware format and more.</description>
    <packager>Fedora Project</packager>
    <url>https://talk.maemo.org/showthread.php?t=87996</url>
    <time file="1643111870" build="1642619633" />
    <size package="73742" installed="69776" archive="70152" />
    <location href="Packages/0/0xFFFF-0.9-4.fc36.src.rpm" />
    <format>
      <rpm:license>GPLv3</rpm:license>
      <rpm:vendor>Fedora Project</rpm:vendor>
      <rpm:group>Unspecified</rpm:group>
      <rpm:buildhost>buildvm-a32-17.iad2.fedoraproject.org</rpm:buildhost>
      <rpm:sourcerpm></rpm:sourcerpm>
      <rpm:header-range start="4504" end="7345" />
      <rpm:provides>
        <rpm:entry name="0xFFFF" flags="EQ" epoch="0" ver="0.9" rel="4.fc36" />
      </rpm:provides>
      <rpm:requires>
        <rpm:entry name="gcc" />
        <rpm:entry name="libusb-devel" />
        <rpm:entry name="make" />
      </rpm:requires>
    </format>
  </package>
  <package type="rpm">
    <name>2ping</name>
    <arch>src</arch>
    <version epoch="0" ver="4.5.1" rel="2.fc36" />
    <checksum type="sha256" pkgid="YES">2ce028d944ebea1cab8c6203c9fed882792478b42fc34682b886a9db16e9de28</checksum>
    <summary>Bi-directional ping utility</summary>
    <description>2ping is a bi-directional ping utility. It uses 3-way pings (akin to TCP SYN,
SYN/ACK, ACK) and after-the-fact state comparison between a 2ping listener and
a 2ping client to determine which direction packet loss occurs.</description>
    <packager>Fedora Project</packager>
    <url>https://www.finnie.org/software/2ping</url>
    <time file="1643111886" build="1642619541" />
    <size package="86521" installed="82546" archive="82932" />
    <location href="Packages/2/2ping-4.5.1-2.fc36.src.rpm" />
    <format>
      <rpm:license>GPLv2+</rpm:license>
      <rpm:vendor>Fedora Project</rpm:vendor>
      <rpm:group>Unspecified</rpm:group>
      <rpm:buildhost>buildvm-x86-32.iad2.fedoraproject.org</rpm:buildhost>
      <rpm:sourcerpm></rpm:sourcerpm>
      <rpm:header-range start="4504" end="7637" />
      <rpm:provides>
        <rpm:entry name="2ping" flags="EQ" epoch="0" ver="4.5.1" rel="2.fc36" />
      </rpm:provides>
      <rpm:requires>
        <rpm:entry name="python3-devel" />
        <rpm:entry name="python3-pytest" />
        <rpm:entry name="python3-setuptools" />
        <rpm:entry name="systemd" />
      </rpm:requires>
    </format>
  </package>
</metadata>

swh/lister/fedora/lister.py
1	Copyright (C) 2022
96–101	You can remove these instance variables. I guess you inspire from the debian lister to implement the fedora one but since then debian lister has been updated in rDLS5426883c49ce0fc4a442b705ae573f868b9f7a62 to something simpler as base lister class gained the ability to return accurate origin counts in the `run` method.
117	else f"{release}/{edition}/source/tree/",
121	s/logging/logger/
136	You should handle 404 errors here as it exists cases where no packages metadata are available for a given fedora release and edition. For instance: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/27/Workstation/source/tree/ => 200 https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/36/Workstation/source/tree/ => 404 try: yield self.page_request(release, edition) except HTTPError as http_error: if http_error.getcode() == 404: logger.debug( "No packages metadata found for fedora release %s edition %s", release, edition, ) continue raise
140	I think it is better to use a real HTTPS origin URL in the form `https://packages.fedoraproject.org/pkgs/{package_name}` instead of such URL with non standard scheme. I know we used something similar in the debian lister but thinking it back it was a mistake (see T4632#98164).
147	to remove
169–174	You should set the `last_update` parameter of `ListedOrigin` constructor to `package_build_time` here. Setting a last update date to `ListedOrigin` instances is quite important as it really improves the scheduling of loading tasks for fedora packages, meaning only packages with new versions since last listing will be scheduled to be loaded again.
181–186	replace this block with: origins_to_send[origin_url] = self.listed_origins[origin_url]
203	After updating package versions, you should also update the `last_update` value of associated `ListedOrigin`: if package_build_time > self.listed_origins[origin_url].last_update: self.listed_origins[origin_url].last_update = package_build_time As in the debian lister, you can also discard `ListedOrigin` instances with no new versions to be sent to the scheduler: # package has already been listed during a previous listing process if package_name in self.state.package_versions: new_versions = ( self.package_versions[package_name] - self.state.package_versions[package_name] ) # no new versions so far, no need to send the origin to the scheduler if not new_versions: origins_to_send.pop(origin_url, None)
204–205	to remove
207–211	Replace this debug log with: logger.debug( "Found %s packages to update (new ones or packages with new versions).", len(origins_to_send), )
219–225	to remove
230	self.updated = len(self.listed_origins) > 0
swh/lister/fedora/tasks.py
12–24	It is better to not hardcode task arguments as in other lister tasks, use this instead: @shared_task(name=__name__ + ".FullFedoraRelister") def list_fedora_full(lister_args) -> Dict[str, int]: """Full update of a Fedora instance""" lister = FedoraLister.from_configfile(lister_args) return lister.run().dict()
swh/lister/fedora/tests/test_lister.py
81–88	Instead of reusing the same lister instance, you should create a new one each time you want to execute a listing, this is the behavior we have in production. def run_lister( swh_scheduler: SchedulerInterface, releases: List[Release], pkg_versions: dict, origin_count: int, ): """Runs the lister and tests that the listed origins are correct.""" lister = FedoraLister( scheduler=swh_scheduler, releases=releases, )

This revision now requires changes to proceed.Nov 9 2022, 2:46 PM

@KShivendu I forgot to mention in my review that we should also get the checksums associated to a rpm archive, the loader will then use it to check download integrity.

You can get those the following way:

15:16 $ git diff
diff --git a/swh/lister/fedora/lister.py b/swh/lister/fedora/lister.py
index 06ee3b1..e9719e1 100644
--- a/swh/lister/fedora/lister.py
+++ b/swh/lister/fedora/lister.py
@@ -48,6 +48,15 @@ def get_last_modified(pkg: repomd.Package) -> datetime:
     return datetime.utcfromtimestamp(int(ts)).replace(tzinfo=timezone.utc)
 
 
+def get_checksums(pkg: repomd.Package) -> Dict[str, str]:
+    """Get checksums associated to rpm archive."""
+    cs = pkg._element.find("common:checksum", namespaces=repomd._ns)
+    cs_type = cs.get("type")
+    if cs_type == "sha":
+        cs_type = "sha1"
+    return {cs_type: cs.text}
+
+
 @dataclass
 class FedoraListerState:
     """State of Fedora lister"""
@@ -141,7 +150,7 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
 
     def origin_url_for_package(self, package_name: PkgName) -> FedoraOrigin:
         """Return the origin url for the given package"""
-        return f"rpm://{self.instance}/packages/{package_name}"
+        return f"https://packages.fedoraproject.org/pkgs/{package_name}"
 
     def get_origins_from_page(self, page: FedoraPageType) -> Iterator[ListedOrigin]:
         """Convert a page of fedora package sources into an iterator of ListedOrigin."""
@@ -195,6 +204,7 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
                 "release": self.current_release,
                 "edition": self.current_edition,
                 "buildTime": package_build_time.isoformat(),
+                "checksums": get_checksums(pkg_metadata),
             }
 
             if package_build_time > self.listed_origins[origin_url].last_update:

Improvements suggested by @anlambert

Build is green

Patch application report for D8386 (id=31821)

Rebasing onto e1f3f87c73...

First, rewinding head to replay your work on top of it...
Applying: feat(fedora): Introduce fedora lister

Changes applied before test

commit 3e00e4507550933cdd3d8c735fa0249fdd9a476d
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/856/ for more details.

Harbormaster completed remote builds in B32761: Diff 31821.Nov 10 2022, 6:31 AM

@KShivendu thanks for the adaptations !

I did more testing on the lister yesterday evening and I found other room for improvements, notably:

we should update origin URL pattern to the following: https://src.fedoraproject.org/rpms/{pkg_name} (see T4632#98214)
we should use pkg_metadata.vr as package_version_key
we should yield ListedOrigin instances only when all fedora releases and editions have been processed in order to send a single time a package to load with all of its versions, it makes the listing process faster and avoids multiple visits of a same package to load all of its versions

I am also wondering if we should not extract pkg_metadata.packager to set it as release author, its value is most of the time Fedora Projectbut it makes sense.

I have implemented the changes listed above the following way:

diff --git a/swh/lister/fedora/lister.py b/swh/lister/fedora/lister.py
index 880beb2..5970edb 100644
--- a/swh/lister/fedora/lister.py
+++ b/swh/lister/fedora/lister.py
@@ -103,8 +103,11 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
 
         self.listed_origins: Dict[FedoraOrigin, ListedOrigin] = {}
         "will hold all listed origins info"
+        self.origins_to_send: Set[FedoraOrigin] = set()
+        "will hold updated origins since last listing"
         self.package_versions: Dict[PkgName, Set[PkgVersion]] = {}
         "will contain the lister state after a call to run"
+        self.last_page = False
 
     def state_from_dict(self, d: Dict[str, Any]) -> FedoraListerState:
         return FedoraListerState(package_versions={k: set(v) for k, v in d.items()})
@@ -120,8 +123,12 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
             if release < 24
             else f"{release}/{edition}/source/tree/",
         )
-
         repo = repomd.load(index_url)  # throws error if no repomd.xml is not found
+
+        self.last_page = (
+            release == self.releases[-1] and edition == get_editions(release)[-1]
+        )
+
         logger.debug(
             "Fetched metadata from url: %s, found %d packages", index_url, len(repo)
         )
@@ -151,13 +158,13 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
 
     def origin_url_for_package(self, package_name: PkgName) -> FedoraOrigin:
         """Return the origin url for the given package"""
-        return f"https://packages.fedoraproject.org/pkgs/{package_name}"
+        return f"https://src.fedoraproject.org/rpms/{package_name}"
 
     def get_origins_from_page(self, page: FedoraPageType) -> Iterator[ListedOrigin]:
         """Convert a page of fedora package sources into an iterator of ListedOrigin."""
         assert self.lister_obj.id is not None
 
-        origins_to_send = {}
+        origins_to_send = set()
 
         # iterate on each package's metadata
         for pkg_metadata in page:
@@ -170,9 +177,7 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
             # build origin url
             origin_url = self.origin_url_for_package(package_name)
             # create package version key as expected by the fedora (rpm) loader
-            package_version_key = (
-                f"{self.current_release}/{self.current_edition}/{package_version}"
-            )
+            package_version_key = pkg_metadata.vr
 
             # this is the first time a package is listed
             if origin_url not in self.listed_origins:
@@ -185,13 +190,11 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
                     extra_loader_arguments={"packages": {}},
                     last_update=package_build_time,
                 )
-
-                # origin will be yielded at the end of that method
-                origins_to_send[origin_url] = self.listed_origins[origin_url]
                 # init set that will contain all listed package versions
                 self.package_versions[package_name] = set()
 
-            origins_to_send[origin_url] = self.listed_origins[origin_url]
+            # origin will be yielded at the end of that method
+            origins_to_send.add(origin_url)
 
             # update package metadata in parameter that will be provided
             # to the rpm loader
@@ -222,7 +225,7 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
                 )
                 # no new versions so far, no need to send the origin to the scheduler
                 if not new_versions:
-                    origins_to_send.pop(origin_url, None)
+                    origins_to_send.remove(origin_url)
 
         logger.debug(
             "Found %s packages to update (new ones or packages with new versions).",
@@ -233,7 +236,12 @@ class FedoraLister(Lister[FedoraListerState, FedoraPageType]):
             len(self.listed_origins),
         )
 
-        yield from origins_to_send.values()
+        self.origins_to_send.update(origins_to_send)
+
+        if self.last_page:
+            # yield listed origins when all fedora releases and editions processed
+            for origin_url in self.origins_to_send:
+                yield self.listed_origins[origin_url]
 
     def finalize(self):
         # set mapping between listed package names and versions as lister state
diff --git a/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz b/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz
index 5bb38d8..b018474 100644
Binary files a/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz and b/swh/lister/fedora/tests/data/archives.fedoraproject.org/primary36-altered.xml.gz differ
diff --git a/swh/lister/fedora/tests/test_lister.py b/swh/lister/fedora/tests/test_lister.py
index bee161a..b859287 100644
--- a/swh/lister/fedora/tests/test_lister.py
+++ b/swh/lister/fedora/tests/test_lister.py
@@ -37,8 +37,8 @@ def rpm_url(release, path):
 @pytest.fixture
 def pkg_versions():
     return {
-        "https://packages.fedoraproject.org/pkgs/0install": {
-            "26/Everything/2.11": {
+        "https://src.fedoraproject.org/rpms/0install": {
+            "2.11-4.fc26": {
                 "name": "0install",
                 "version": "2.11",
                 "release": 26,
@@ -50,8 +50,8 @@ def pkg_versions():
                 },
             }
         },
-        "https://packages.fedoraproject.org/pkgs/0xFFFF": {
-            "26/Everything/0.3.9": {
+        "https://src.fedoraproject.org/rpms/0xFFFF": {
+            "0.3.9-15.fc26": {
                 "name": "0xFFFF",
                 "version": "0.3.9",
                 "release": 26,
@@ -62,7 +62,7 @@ def pkg_versions():
                     "sha256": "96f9c163c0402d2b30e5343c8397a6d50e146c85a446804396b119ef9698231f"
                 },
             },
-            "36/Everything/0.9": {
+            "0.9-4.fc36": {
                 "name": "0xFFFF",
                 "version": "0.9",
                 "release": 36,
@@ -74,8 +74,8 @@ def pkg_versions():
                 },
             },
         },
-        "https://packages.fedoraproject.org/pkgs/2ping": {
-            "36/Everything/4.5.1": {
+        "https://src.fedoraproject.org/rpms/2ping": {
+            "4.5.1-2.fc36": {
                 "name": "2ping",
                 "version": "4.5.1",
                 "release": 36,
@@ -165,16 +165,16 @@ def test_incremental_lister(
     # of package 0xFFFF to 0.10:
     mock_repomd(datadir, mocker, use_altered_fedora36=True)
     # Add new version to the set of expected pkg versions:
-    pkg_versions["https://packages.fedoraproject.org/pkgs/0xFFFF"].update(
+    pkg_versions["https://src.fedoraproject.org/rpms/0xFFFF"].update(
         {
-            "36/Everything/0.10": {
+            "0.10-4.fc36": {
                 "name": "0xFFFF",
                 "version": "0.10",
                 "release": 36,
                 "edition": "Everything",
                 "buildTime": "2022-01-19T19:13:53+00:00",
                 # .rpm URL remains same as per primary.xml file:
-                "url": rpm_url(36, "0/0xFFFF-0.10-1.fc36.src.rpm"),
+                "url": rpm_url(36, "0/0xFFFF-0.10-4.fc36.src.rpm"),
                 "checksums": {
                     "sha256": "45eee8d990d502324ae665233c320b8a5469c25d735f1862e094c1878d6ff2cd"
                 },

Then in docker I could list source packages of all fedora releases using this command:

(swh) ✘-INT ~/swh/swh-environment/docker [master|⚑ 84] 
13:30 $ docker compose exec swh-scheduler swh scheduler task add -p oneshot list-fedora-full releases="[7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36]"

The listing took around 5 minutes:

docker-swh-lister-1  | [2022-11-10 10:48:33,222: INFO/MainProcess] Task swh.lister.fedora.tasks.FullFedoraRelister[3f13583a-9fc5-4a42-a5ce-4739f6b1407d] received
docker-swh-lister-1  | [2022-11-10 10:48:33,223: INFO/MainProcess] listers@fcedc344d050 ready.
docker-swh-lister-1  | [2022-11-10 10:48:33,327: DEBUG/ForkPoolWorker-1] Loading config file /lister.yml
docker-swh-lister-1  | [2022-11-10 10:48:33,377: DEBUG/ForkPoolWorker-1] Listing fedora release 7 edition Everything
docker-swh-lister-1  | [2022-11-10 10:48:34,867: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/7/Everything/source/SRPMS/, found 4227 packages
docker-swh-lister-1  | [2022-11-10 10:48:35,739: DEBUG/ForkPoolWorker-1] Found 4226 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:35,739: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 4226.
docker-swh-lister-1  | [2022-11-10 10:48:35,739: DEBUG/ForkPoolWorker-1] Listing fedora release 7 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:36,105: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 7 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:36,105: DEBUG/ForkPoolWorker-1] Listing fedora release 8 edition Everything
docker-swh-lister-1  | [2022-11-10 10:48:37,549: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/8/Everything/source/SRPMS/, found 4834 packages
docker-swh-lister-1  | [2022-11-10 10:48:38,370: DEBUG/ForkPoolWorker-1] Found 4834 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:38,371: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 4910.
docker-swh-lister-1  | [2022-11-10 10:48:38,372: DEBUG/ForkPoolWorker-1] Listing fedora release 8 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:38,648: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 8 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:38,649: DEBUG/ForkPoolWorker-1] Listing fedora release 9 edition Everything
docker-swh-lister-1  | [2022-11-10 10:48:40,073: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/9/Everything/source/SRPMS/, found 5547 packages
docker-swh-lister-1  | [2022-11-10 10:48:40,922: DEBUG/ForkPoolWorker-1] Found 5547 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:40,923: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 5746.
docker-swh-lister-1  | [2022-11-10 10:48:40,927: DEBUG/ForkPoolWorker-1] Listing fedora release 9 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:41,873: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/9/Fedora/source/SRPMS/, found 1314 packages
docker-swh-lister-1  | [2022-11-10 10:48:42,097: DEBUG/ForkPoolWorker-1] Found 1314 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:42,097: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 5746.
docker-swh-lister-1  | [2022-11-10 10:48:42,097: DEBUG/ForkPoolWorker-1] Listing fedora release 10 edition Everything
docker-swh-lister-1  | [2022-11-10 10:48:43,676: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/10/Everything/source/SRPMS/, found 6406 packages
docker-swh-lister-1  | [2022-11-10 10:48:44,639: DEBUG/ForkPoolWorker-1] Found 6406 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:44,639: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 6682.
docker-swh-lister-1  | [2022-11-10 10:48:44,640: DEBUG/ForkPoolWorker-1] Listing fedora release 10 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:45,601: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/10/Fedora/source/SRPMS/, found 1375 packages
docker-swh-lister-1  | [2022-11-10 10:48:45,985: DEBUG/ForkPoolWorker-1] Found 1375 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:45,986: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 6682.
docker-swh-lister-1  | [2022-11-10 10:48:45,986: DEBUG/ForkPoolWorker-1] Listing fedora release 11 edition Everything
docker-swh-lister-1  | [2022-11-10 10:48:47,828: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/11/Everything/source/SRPMS/, found 7446 packages
docker-swh-lister-1  | [2022-11-10 10:48:48,998: DEBUG/ForkPoolWorker-1] Found 7446 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:48,999: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 7837.
docker-swh-lister-1  | [2022-11-10 10:48:49,000: DEBUG/ForkPoolWorker-1] Listing fedora release 11 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:49,963: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/11/Fedora/source/SRPMS/, found 1393 packages
docker-swh-lister-1  | [2022-11-10 10:48:50,252: DEBUG/ForkPoolWorker-1] Found 1393 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:50,252: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 7837.
docker-swh-lister-1  | [2022-11-10 10:48:50,253: DEBUG/ForkPoolWorker-1] Listing fedora release 12 edition Everything
docker-swh-lister-1  | [2022-11-10 10:48:52,018: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/12/Everything/source/SRPMS/, found 8518 packages
docker-swh-lister-1  | [2022-11-10 10:48:53,149: DEBUG/ForkPoolWorker-1] Found 8518 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:53,150: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 9045.
docker-swh-lister-1  | [2022-11-10 10:48:53,152: DEBUG/ForkPoolWorker-1] Listing fedora release 12 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:54,105: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/12/Fedora/source/SRPMS/, found 1475 packages
docker-swh-lister-1  | [2022-11-10 10:48:54,461: DEBUG/ForkPoolWorker-1] Found 1475 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:54,462: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 9045.
docker-swh-lister-1  | [2022-11-10 10:48:54,462: DEBUG/ForkPoolWorker-1] Listing fedora release 13 edition Everything
docker-swh-lister-1  | [2022-11-10 10:48:56,878: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/13/Everything/source/SRPMS/, found 9152 packages
docker-swh-lister-1  | [2022-11-10 10:48:58,780: DEBUG/ForkPoolWorker-1] Found 9152 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:48:58,780: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 9826.
docker-swh-lister-1  | [2022-11-10 10:48:58,784: DEBUG/ForkPoolWorker-1] Listing fedora release 13 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:48:59,848: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/13/Fedora/source/SRPMS/, found 1538 packages
docker-swh-lister-1  | [2022-11-10 10:49:00,079: DEBUG/ForkPoolWorker-1] Found 1538 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:00,079: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 9826.
docker-swh-lister-1  | [2022-11-10 10:49:00,079: DEBUG/ForkPoolWorker-1] Listing fedora release 14 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:02,443: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/14/Everything/source/SRPMS/, found 9746 packages
docker-swh-lister-1  | [2022-11-10 10:49:03,942: DEBUG/ForkPoolWorker-1] Found 9746 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:03,942: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 10560.
docker-swh-lister-1  | [2022-11-10 10:49:03,944: DEBUG/ForkPoolWorker-1] Listing fedora release 14 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:49:04,999: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/14/Fedora/source/SRPMS/, found 1730 packages
docker-swh-lister-1  | [2022-11-10 10:49:05,358: DEBUG/ForkPoolWorker-1] Found 1730 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:05,358: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 10560.
docker-swh-lister-1  | [2022-11-10 10:49:05,361: DEBUG/ForkPoolWorker-1] Listing fedora release 15 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:07,495: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/15/Everything/source/SRPMS/, found 10412 packages
docker-swh-lister-1  | [2022-11-10 10:49:08,988: DEBUG/ForkPoolWorker-1] Found 10412 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:08,988: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 11436.
docker-swh-lister-1  | [2022-11-10 10:49:08,991: DEBUG/ForkPoolWorker-1] Listing fedora release 15 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:49:10,043: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/15/Fedora/source/SRPMS/, found 1855 packages
docker-swh-lister-1  | [2022-11-10 10:49:10,563: DEBUG/ForkPoolWorker-1] Found 1855 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:10,563: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 11436.
docker-swh-lister-1  | [2022-11-10 10:49:10,564: DEBUG/ForkPoolWorker-1] Listing fedora release 16 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:12,919: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/16/Everything/source/SRPMS/, found 10929 packages
docker-swh-lister-1  | [2022-11-10 10:49:14,882: DEBUG/ForkPoolWorker-1] Found 10929 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:14,882: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 12193.
docker-swh-lister-1  | [2022-11-10 10:49:14,884: DEBUG/ForkPoolWorker-1] Listing fedora release 16 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:49:15,975: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/16/Fedora/source/SRPMS/, found 1961 packages
docker-swh-lister-1  | [2022-11-10 10:49:16,487: DEBUG/ForkPoolWorker-1] Found 1961 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:16,488: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 12193.
docker-swh-lister-1  | [2022-11-10 10:49:16,492: DEBUG/ForkPoolWorker-1] Listing fedora release 17 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:18,735: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/17/Everything/source/SRPMS/, found 11614 packages
docker-swh-lister-1  | [2022-11-10 10:49:20,412: DEBUG/ForkPoolWorker-1] Found 11614 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:20,412: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 13255.
docker-swh-lister-1  | [2022-11-10 10:49:20,414: DEBUG/ForkPoolWorker-1] Listing fedora release 17 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:49:21,503: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/17/Fedora/source/SRPMS/, found 2131 packages
docker-swh-lister-1  | [2022-11-10 10:49:21,916: DEBUG/ForkPoolWorker-1] Found 2131 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:21,916: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 13255.
docker-swh-lister-1  | [2022-11-10 10:49:21,921: DEBUG/ForkPoolWorker-1] Listing fedora release 18 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:24,553: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/18/Everything/source/SRPMS/, found 12614 packages
docker-swh-lister-1  | [2022-11-10 10:49:26,531: DEBUG/ForkPoolWorker-1] Found 12614 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:26,532: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 14538.
docker-swh-lister-1  | [2022-11-10 10:49:26,534: DEBUG/ForkPoolWorker-1] Listing fedora release 18 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:49:27,631: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/18/Fedora/source/SRPMS/, found 2629 packages
docker-swh-lister-1  | [2022-11-10 10:49:28,190: DEBUG/ForkPoolWorker-1] Found 2629 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:28,190: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 14538.
docker-swh-lister-1  | [2022-11-10 10:49:28,191: DEBUG/ForkPoolWorker-1] Listing fedora release 19 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:30,623: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/19/Everything/source/SRPMS/, found 13606 packages
docker-swh-lister-1  | [2022-11-10 10:49:32,993: DEBUG/ForkPoolWorker-1] Found 13606 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:32,994: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 15720.
docker-swh-lister-1  | [2022-11-10 10:49:32,997: DEBUG/ForkPoolWorker-1] Listing fedora release 19 edition Fedora
docker-swh-lister-1  | [2022-11-10 10:49:34,170: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/19/Fedora/source/SRPMS/, found 2484 packages
docker-swh-lister-1  | [2022-11-10 10:49:34,839: DEBUG/ForkPoolWorker-1] Found 2484 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:34,839: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 15720.
docker-swh-lister-1  | [2022-11-10 10:49:34,840: DEBUG/ForkPoolWorker-1] Listing fedora release 20 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:37,503: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Everything/source/SRPMS/, found 14364 packages
docker-swh-lister-1  | [2022-11-10 10:49:39,537: DEBUG/ForkPoolWorker-1] Found 14364 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:39,537: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 16734.
docker-swh-lister-1  | [2022-11-10 10:49:39,539: DEBUG/ForkPoolWorker-1] Listing fedora release 20 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:39,812: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 20 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:39,812: DEBUG/ForkPoolWorker-1] Listing fedora release 20 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:40,083: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 20 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:40,084: DEBUG/ForkPoolWorker-1] Listing fedora release 21 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:42,993: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/21/Everything/source/SRPMS/, found 15842 packages
docker-swh-lister-1  | [2022-11-10 10:49:45,314: DEBUG/ForkPoolWorker-1] Found 15842 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:45,314: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 18527.
docker-swh-lister-1  | [2022-11-10 10:49:45,317: DEBUG/ForkPoolWorker-1] Listing fedora release 21 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:45,595: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 21 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:45,595: DEBUG/ForkPoolWorker-1] Listing fedora release 21 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:45,867: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 21 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:45,867: DEBUG/ForkPoolWorker-1] Listing fedora release 22 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:48,721: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/22/Everything/source/SRPMS/, found 16565 packages
docker-swh-lister-1  | [2022-11-10 10:49:51,243: DEBUG/ForkPoolWorker-1] Found 16565 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:51,243: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 19423.
docker-swh-lister-1  | [2022-11-10 10:49:51,247: DEBUG/ForkPoolWorker-1] Listing fedora release 22 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:51,520: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 22 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:51,521: DEBUG/ForkPoolWorker-1] Listing fedora release 22 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:51,791: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 22 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:51,791: DEBUG/ForkPoolWorker-1] Listing fedora release 23 edition Everything
docker-swh-lister-1  | [2022-11-10 10:49:55,055: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/23/Everything/source/SRPMS/, found 17080 packages
docker-swh-lister-1  | [2022-11-10 10:49:57,912: DEBUG/ForkPoolWorker-1] Found 17080 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:49:57,912: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 20143.
docker-swh-lister-1  | [2022-11-10 10:49:57,915: DEBUG/ForkPoolWorker-1] Listing fedora release 23 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:58,189: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 23 edition Server
docker-swh-lister-1  | [2022-11-10 10:49:58,189: DEBUG/ForkPoolWorker-1] Listing fedora release 23 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:58,459: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 23 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:49:58,459: DEBUG/ForkPoolWorker-1] Listing fedora release 24 edition Everything
docker-swh-lister-1  | [2022-11-10 10:50:02,596: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/24/Everything/source/tree/, found 18021 packages
docker-swh-lister-1  | [2022-11-10 10:50:05,321: DEBUG/ForkPoolWorker-1] Found 18021 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:05,321: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 21447.
docker-swh-lister-1  | [2022-11-10 10:50:05,323: DEBUG/ForkPoolWorker-1] Listing fedora release 24 edition Server
docker-swh-lister-1  | [2022-11-10 10:50:06,462: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/24/Server/source/tree/, found 1331 packages
docker-swh-lister-1  | [2022-11-10 10:50:06,946: DEBUG/ForkPoolWorker-1] Found 1331 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:06,946: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 21447.
docker-swh-lister-1  | [2022-11-10 10:50:06,947: DEBUG/ForkPoolWorker-1] Listing fedora release 24 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:50:07,956: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/24/Workstation/source/tree/, found 1485 packages
docker-swh-lister-1  | [2022-11-10 10:50:08,183: DEBUG/ForkPoolWorker-1] Found 1485 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:08,183: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 21447.
docker-swh-lister-1  | [2022-11-10 10:50:08,184: DEBUG/ForkPoolWorker-1] Listing fedora release 25 edition Everything
docker-swh-lister-1  | [2022-11-10 10:50:11,347: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/25/Everything/source/tree/, found 18428 packages
docker-swh-lister-1  | [2022-11-10 10:50:14,622: DEBUG/ForkPoolWorker-1] Found 18428 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:14,622: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 22136.
docker-swh-lister-1  | [2022-11-10 10:50:14,626: DEBUG/ForkPoolWorker-1] Listing fedora release 25 edition Server
docker-swh-lister-1  | [2022-11-10 10:50:15,604: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/25/Server/source/tree/, found 1387 packages
docker-swh-lister-1  | [2022-11-10 10:50:16,144: DEBUG/ForkPoolWorker-1] Found 1387 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:16,144: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 22136.
docker-swh-lister-1  | [2022-11-10 10:50:16,149: DEBUG/ForkPoolWorker-1] Listing fedora release 25 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:50:17,190: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/25/Workstation/source/tree/, found 1501 packages
docker-swh-lister-1  | [2022-11-10 10:50:17,397: DEBUG/ForkPoolWorker-1] Found 1501 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:17,397: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 22136.
docker-swh-lister-1  | [2022-11-10 10:50:17,398: DEBUG/ForkPoolWorker-1] Listing fedora release 26 edition Everything
docker-swh-lister-1  | [2022-11-10 10:50:21,339: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/26/Everything/source/tree/, found 19195 packages
docker-swh-lister-1  | [2022-11-10 10:50:23,925: DEBUG/ForkPoolWorker-1] Found 19195 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:23,925: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 23182.
docker-swh-lister-1  | [2022-11-10 10:50:23,929: DEBUG/ForkPoolWorker-1] Listing fedora release 26 edition Server
docker-swh-lister-1  | [2022-11-10 10:50:26,823: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/26/Server/source/tree/, found 1500 packages
docker-swh-lister-1  | [2022-11-10 10:50:27,381: DEBUG/ForkPoolWorker-1] Found 1500 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:27,381: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 23182.
docker-swh-lister-1  | [2022-11-10 10:50:27,388: DEBUG/ForkPoolWorker-1] Listing fedora release 26 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:50:28,415: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/26/Workstation/source/tree/, found 1554 packages
docker-swh-lister-1  | [2022-11-10 10:50:28,619: DEBUG/ForkPoolWorker-1] Found 1554 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:28,619: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 23182.
docker-swh-lister-1  | [2022-11-10 10:50:28,620: DEBUG/ForkPoolWorker-1] Listing fedora release 27 edition Everything
docker-swh-lister-1  | [2022-11-10 10:50:32,811: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/27/Everything/source/tree/, found 19664 packages
docker-swh-lister-1  | [2022-11-10 10:50:35,485: DEBUG/ForkPoolWorker-1] Found 19664 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:35,485: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 23782.
docker-swh-lister-1  | [2022-11-10 10:50:35,491: DEBUG/ForkPoolWorker-1] Listing fedora release 27 edition Server
docker-swh-lister-1  | [2022-11-10 10:50:36,545: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/27/Server/source/tree/, found 1528 packages
docker-swh-lister-1  | [2022-11-10 10:50:37,098: DEBUG/ForkPoolWorker-1] Found 1528 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:37,098: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 23782.
docker-swh-lister-1  | [2022-11-10 10:50:37,107: DEBUG/ForkPoolWorker-1] Listing fedora release 27 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:50:38,231: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/27/Workstation/source/tree/, found 1558 packages
docker-swh-lister-1  | [2022-11-10 10:50:38,489: DEBUG/ForkPoolWorker-1] Found 1558 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:38,489: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 23782.
docker-swh-lister-1  | [2022-11-10 10:50:38,489: DEBUG/ForkPoolWorker-1] Listing fedora release 28 edition Everything
docker-swh-lister-1  | [2022-11-10 10:50:44,190: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/28/Everything/source/tree/, found 21209 packages
docker-swh-lister-1  | [2022-11-10 10:50:47,509: DEBUG/ForkPoolWorker-1] Found 21209 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:47,510: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 25438.
docker-swh-lister-1  | [2022-11-10 10:50:47,515: DEBUG/ForkPoolWorker-1] Listing fedora release 28 edition Server
docker-swh-lister-1  | [2022-11-10 10:50:48,543: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/28/Server/source/tree/, found 1534 packages
docker-swh-lister-1  | [2022-11-10 10:50:49,169: DEBUG/ForkPoolWorker-1] Found 1534 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:49,169: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 25438.
docker-swh-lister-1  | [2022-11-10 10:50:49,176: DEBUG/ForkPoolWorker-1] Listing fedora release 28 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:50:50,224: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/28/Workstation/source/tree/, found 1666 packages
docker-swh-lister-1  | [2022-11-10 10:50:50,564: DEBUG/ForkPoolWorker-1] Found 1666 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:50,564: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 25438.
docker-swh-lister-1  | [2022-11-10 10:50:50,565: DEBUG/ForkPoolWorker-1] Listing fedora release 28 edition Modular
docker-swh-lister-1  | [2022-11-10 10:50:51,126: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/28/Modular/source/tree/, found 11 packages
docker-swh-lister-1  | [2022-11-10 10:50:51,168: DEBUG/ForkPoolWorker-1] Found 9 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:51,168: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 25439.
docker-swh-lister-1  | [2022-11-10 10:50:51,171: DEBUG/ForkPoolWorker-1] Listing fedora release 29 edition Everything
docker-swh-lister-1  | [2022-11-10 10:50:56,762: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/source/tree/, found 21847 packages
docker-swh-lister-1  | [2022-11-10 10:50:59,560: DEBUG/ForkPoolWorker-1] Found 21847 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:50:59,560: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26298.
docker-swh-lister-1  | [2022-11-10 10:50:59,566: DEBUG/ForkPoolWorker-1] Listing fedora release 29 edition Server
docker-swh-lister-1  | [2022-11-10 10:51:00,657: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Server/source/tree/, found 1666 packages
docker-swh-lister-1  | [2022-11-10 10:51:01,213: DEBUG/ForkPoolWorker-1] Found 1666 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:01,213: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26298.
docker-swh-lister-1  | [2022-11-10 10:51:01,219: DEBUG/ForkPoolWorker-1] Listing fedora release 29 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:02,378: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Workstation/source/tree/, found 1672 packages
docker-swh-lister-1  | [2022-11-10 10:51:02,700: DEBUG/ForkPoolWorker-1] Found 1672 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:02,700: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26298.
docker-swh-lister-1  | [2022-11-10 10:51:02,701: DEBUG/ForkPoolWorker-1] Listing fedora release 29 edition Modular
docker-swh-lister-1  | [2022-11-10 10:51:03,560: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Modular/source/tree/, found 708 packages
docker-swh-lister-1  | [2022-11-10 10:51:03,723: DEBUG/ForkPoolWorker-1] Found 289 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:03,724: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26302.
docker-swh-lister-1  | [2022-11-10 10:51:03,724: DEBUG/ForkPoolWorker-1] Listing fedora release 30 edition Everything
docker-swh-lister-1  | [2022-11-10 10:51:08,169: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/Everything/source/tree/, found 21292 packages
docker-swh-lister-1  | [2022-11-10 10:51:11,240: DEBUG/ForkPoolWorker-1] Found 21292 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:11,240: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26968.
docker-swh-lister-1  | [2022-11-10 10:51:11,246: DEBUG/ForkPoolWorker-1] Listing fedora release 30 edition Server
docker-swh-lister-1  | [2022-11-10 10:51:12,287: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/Server/source/tree/, found 1690 packages
docker-swh-lister-1  | [2022-11-10 10:51:12,860: DEBUG/ForkPoolWorker-1] Found 1690 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:12,860: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26968.
docker-swh-lister-1  | [2022-11-10 10:51:12,861: DEBUG/ForkPoolWorker-1] Listing fedora release 30 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:13,962: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/Workstation/source/tree/, found 1689 packages
docker-swh-lister-1  | [2022-11-10 10:51:14,197: DEBUG/ForkPoolWorker-1] Found 1689 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:14,197: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26968.
docker-swh-lister-1  | [2022-11-10 10:51:14,198: DEBUG/ForkPoolWorker-1] Listing fedora release 30 edition Modular
docker-swh-lister-1  | [2022-11-10 10:51:15,033: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/Modular/source/tree/, found 472 packages
docker-swh-lister-1  | [2022-11-10 10:51:15,178: DEBUG/ForkPoolWorker-1] Found 325 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:15,178: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 26978.
docker-swh-lister-1  | [2022-11-10 10:51:15,178: DEBUG/ForkPoolWorker-1] Listing fedora release 31 edition Everything
docker-swh-lister-1  | [2022-11-10 10:51:18,462: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/31/Everything/source/tree/, found 21192 packages
docker-swh-lister-1  | [2022-11-10 10:51:21,368: DEBUG/ForkPoolWorker-1] Found 21192 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:21,368: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 28265.
docker-swh-lister-1  | [2022-11-10 10:51:21,374: DEBUG/ForkPoolWorker-1] Listing fedora release 31 edition Server
docker-swh-lister-1  | [2022-11-10 10:51:22,287: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/31/Server/source/tree/, found 942 packages
docker-swh-lister-1  | [2022-11-10 10:51:22,791: DEBUG/ForkPoolWorker-1] Found 942 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:22,791: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 28265.
docker-swh-lister-1  | [2022-11-10 10:51:22,791: DEBUG/ForkPoolWorker-1] Listing fedora release 31 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:23,062: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 31 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:23,063: DEBUG/ForkPoolWorker-1] Listing fedora release 31 edition Modular
docker-swh-lister-1  | [2022-11-10 10:51:24,286: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/31/Modular/source/tree/, found 944 packages
docker-swh-lister-1  | [2022-11-10 10:51:24,538: DEBUG/ForkPoolWorker-1] Found 409 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:24,538: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 28275.
docker-swh-lister-1  | [2022-11-10 10:51:24,538: DEBUG/ForkPoolWorker-1] Listing fedora release 32 edition Everything
docker-swh-lister-1  | [2022-11-10 10:51:28,463: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/32/Everything/source/tree/, found 20934 packages
docker-swh-lister-1  | [2022-11-10 10:51:32,016: DEBUG/ForkPoolWorker-1] Found 20934 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:32,016: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 29106.
docker-swh-lister-1  | [2022-11-10 10:51:32,020: DEBUG/ForkPoolWorker-1] Listing fedora release 32 edition Server
docker-swh-lister-1  | [2022-11-10 10:51:32,950: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/32/Server/source/tree/, found 937 packages
docker-swh-lister-1  | [2022-11-10 10:51:33,501: DEBUG/ForkPoolWorker-1] Found 937 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:33,501: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 29106.
docker-swh-lister-1  | [2022-11-10 10:51:33,502: DEBUG/ForkPoolWorker-1] Listing fedora release 32 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:33,772: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 32 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:33,773: DEBUG/ForkPoolWorker-1] Listing fedora release 32 edition Modular
docker-swh-lister-1  | [2022-11-10 10:51:34,667: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/32/Modular/source/tree/, found 933 packages
docker-swh-lister-1  | [2022-11-10 10:51:34,878: DEBUG/ForkPoolWorker-1] Found 401 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:34,879: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 29106.
docker-swh-lister-1  | [2022-11-10 10:51:34,879: DEBUG/ForkPoolWorker-1] Listing fedora release 33 edition Everything
docker-swh-lister-1  | [2022-11-10 10:51:40,151: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/33/Everything/source/tree/, found 20965 packages
docker-swh-lister-1  | [2022-11-10 10:51:44,717: DEBUG/ForkPoolWorker-1] Found 20965 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:44,717: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 30073.
docker-swh-lister-1  | [2022-11-10 10:51:44,721: DEBUG/ForkPoolWorker-1] Listing fedora release 33 edition Server
docker-swh-lister-1  | [2022-11-10 10:51:45,722: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/33/Server/source/tree/, found 932 packages
docker-swh-lister-1  | [2022-11-10 10:51:46,541: DEBUG/ForkPoolWorker-1] Found 932 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:46,541: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 30073.
docker-swh-lister-1  | [2022-11-10 10:51:46,542: DEBUG/ForkPoolWorker-1] Listing fedora release 33 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:46,828: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 33 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:46,829: DEBUG/ForkPoolWorker-1] Listing fedora release 33 edition Modular
docker-swh-lister-1  | [2022-11-10 10:51:47,685: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/33/Modular/source/tree/, found 653 packages
docker-swh-lister-1  | [2022-11-10 10:51:47,809: DEBUG/ForkPoolWorker-1] Found 235 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:47,810: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 30074.
docker-swh-lister-1  | [2022-11-10 10:51:47,810: DEBUG/ForkPoolWorker-1] Listing fedora release 34 edition Everything
docker-swh-lister-1  | [2022-11-10 10:51:52,827: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/34/Everything/source/tree/, found 22063 packages
docker-swh-lister-1  | [2022-11-10 10:51:57,302: DEBUG/ForkPoolWorker-1] Found 22063 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:57,302: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 31966.
docker-swh-lister-1  | [2022-11-10 10:51:57,310: DEBUG/ForkPoolWorker-1] Listing fedora release 34 edition Server
docker-swh-lister-1  | [2022-11-10 10:51:58,274: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/34/Server/source/tree/, found 897 packages
docker-swh-lister-1  | [2022-11-10 10:51:58,944: DEBUG/ForkPoolWorker-1] Found 897 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:51:58,944: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 31966.
docker-swh-lister-1  | [2022-11-10 10:51:58,944: DEBUG/ForkPoolWorker-1] Listing fedora release 34 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:59,232: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 34 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:51:59,233: DEBUG/ForkPoolWorker-1] Listing fedora release 34 edition Modular
docker-swh-lister-1  | [2022-11-10 10:52:00,098: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/34/Modular/source/tree/, found 723 packages
docker-swh-lister-1  | [2022-11-10 10:52:00,236: DEBUG/ForkPoolWorker-1] Found 358 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:52:00,236: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 31966.
docker-swh-lister-1  | [2022-11-10 10:52:00,236: DEBUG/ForkPoolWorker-1] Listing fedora release 35 edition Everything
docker-swh-lister-1  | [2022-11-10 10:52:06,527: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/35/Everything/source/tree/, found 22573 packages
docker-swh-lister-1  | [2022-11-10 10:52:11,325: DEBUG/ForkPoolWorker-1] Found 22573 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:52:11,325: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 33044.
docker-swh-lister-1  | [2022-11-10 10:52:11,330: DEBUG/ForkPoolWorker-1] Listing fedora release 35 edition Server
docker-swh-lister-1  | [2022-11-10 10:52:12,269: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/35/Server/source/tree/, found 915 packages
docker-swh-lister-1  | [2022-11-10 10:52:12,924: DEBUG/ForkPoolWorker-1] Found 915 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:52:12,924: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 33044.
docker-swh-lister-1  | [2022-11-10 10:52:12,925: DEBUG/ForkPoolWorker-1] Listing fedora release 35 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:52:13,211: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 35 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:52:13,211: DEBUG/ForkPoolWorker-1] Listing fedora release 35 edition Modular
docker-swh-lister-1  | [2022-11-10 10:52:14,003: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/35/Modular/source/tree/, found 335 packages
docker-swh-lister-1  | [2022-11-10 10:52:14,073: DEBUG/ForkPoolWorker-1] Found 187 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:52:14,073: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 33044.
docker-swh-lister-1  | [2022-11-10 10:52:14,073: DEBUG/ForkPoolWorker-1] Listing fedora release 36 edition Everything
docker-swh-lister-1  | [2022-11-10 10:52:19,630: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/36/Everything/source/tree/, found 23028 packages
docker-swh-lister-1  | [2022-11-10 10:52:22,848: DEBUG/ForkPoolWorker-1] Found 23028 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:52:22,849: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 33826.
docker-swh-lister-1  | [2022-11-10 10:52:22,857: DEBUG/ForkPoolWorker-1] Listing fedora release 36 edition Server
docker-swh-lister-1  | [2022-11-10 10:52:23,797: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/36/Server/source/tree/, found 930 packages
docker-swh-lister-1  | [2022-11-10 10:52:24,463: DEBUG/ForkPoolWorker-1] Found 930 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:52:24,463: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 33826.
docker-swh-lister-1  | [2022-11-10 10:52:24,463: DEBUG/ForkPoolWorker-1] Listing fedora release 36 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:52:24,744: DEBUG/ForkPoolWorker-1] No packages metadata found for fedora release 36 edition Workstation
docker-swh-lister-1  | [2022-11-10 10:52:24,745: DEBUG/ForkPoolWorker-1] Listing fedora release 36 edition Modular
docker-swh-lister-1  | [2022-11-10 10:52:25,497: DEBUG/ForkPoolWorker-1] Fetched metadata from url: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/36/Modular/source/tree/, found 221 packages
docker-swh-lister-1  | [2022-11-10 10:52:25,566: DEBUG/ForkPoolWorker-1] Found 187 packages to update (new ones or packages with new versions).
docker-swh-lister-1  | [2022-11-10 10:52:25,566: DEBUG/ForkPoolWorker-1] Current total number of listed packages is equal to 33826.
docker-swh-lister-1  | [2022-11-10 10:53:41,962: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@6b56c5e66722', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 19.2.1', 'product': 'RabbitMQ', 'version': '3.6.16'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
docker-swh-lister-1  | [2022-11-10 10:53:41,963: DEBUG/ForkPoolWorker-1] using channel_id: 1
docker-swh-lister-1  | [2022-11-10 10:53:41,964: DEBUG/ForkPoolWorker-1] Channel open
docker-swh-lister-1  | [2022-11-10 10:53:41,966: INFO/ForkPoolWorker-1] Task swh.lister.fedora.tasks.FullFedoraRelister[3f13583a-9fc5-4a42-a5ce-4739f6b1407d] succeeded in 308.63382762594847s: {'pages': 70, 'origins': 33826}

This revision now requires changes to proceed.Nov 10 2022, 1:33 PM

Send origins in the last iteration
Change origin url format
Use intrinsic version for pkg version key

Build is green

Patch application report for D8386 (id=31840)

Rebasing onto e1f3f87c73...

First, rewinding head to replay your work on top of it...
Applying: feat(fedora): Introduce fedora lister

Changes applied before test

commit aba06d6d95a4e1b2942391ab94804523d1b32204
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/857/ for more details.

Harbormaster completed remote builds in B32777: Diff 31840.Nov 11 2022, 7:10 AM

@anlambert

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

Add tests for handling of HTTP errors and sha1 checksum (increase test coverage)

Build is green

Patch application report for D8386 (id=31861)

Rebasing onto ea146ce297...

First, rewinding head to replay your work on top of it...
Applying: feat(fedora): Introduce fedora lister

Changes applied before test

commit 0c82f65dba89f31364672f6e00e8897159094234
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/860/ for more details.

Harbormaster completed remote builds in B32796: Diff 31861.Nov 15 2022, 9:15 AM

In D8386#229677, @KShivendu wrote:

@anlambert

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

This seems like a misfeature in the webapp:

https://archive.softwareheritage.org/api/1/snapshot/158a3f36b0bd3da461fb7458de44cfa2c94e4270/

The snapshot has multiple branches, with the same version suffix, pointing at the same objects (because the exact same version of the package is present in multiple Ubuntu suites).

I'm not 100% sure how we should be fixing that, but that bug shouldn't prevent you from giving the fedora snapshots the "semantically correct" structure.

In D8386#229882, @olasd wrote:

In D8386#229677, @KShivendu wrote:

@anlambert

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

This seems like a misfeature in the webapp:

https://archive.softwareheritage.org/api/1/snapshot/158a3f36b0bd3da461fb7458de44cfa2c94e4270/

The snapshot has multiple branches, with the same version suffix, pointing at the same objects (because the exact same version of the package is present in multiple Ubuntu suites).

I'm not 100% sure how we should be fixing that, but that bug shouldn't prevent you from giving the fedora snapshots the "semantically correct" structure.

I also noticed that yesterday evening and I was also wondering what is the best way to fix that. I see two possible options:

We change the names of the keys in snapshot branches dictionary in order to use the intrinsic version of a debian package instead of its extrinsic one (meaning releases/bionic-security/main/1.14.0-0ubuntu1.10 should rather be releases/1.14.0-0ubuntu1.10)
We update the webapp to filter duplicated releases before display as the release name is used instead of the snapshot branches key associated to the release

I would rather go for the second one as keeping the debian/ubuntu suites and components in keys of snapshot branches dictionary seems of interest.
We could do the same for the fedora case as based on my tests it is quite common that extrinsic versions in the form [0-9].[0-9].[0-9]-[0-9].fc[0-9]+
target the same intrinsic version [0-9].[0-9].[0-9]-[0-9].

Looks good to me, thanks !

This revision is now accepted and ready to land.Nov 15 2022, 10:59 AM

In D8386#229890, @anlambert wrote:

In D8386#229882, @olasd wrote:

In D8386#229677, @KShivendu wrote:

@anlambert

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

This seems like a misfeature in the webapp:

https://archive.softwareheritage.org/api/1/snapshot/158a3f36b0bd3da461fb7458de44cfa2c94e4270/

The snapshot has multiple branches, with the same version suffix, pointing at the same objects (because the exact same version of the package is present in multiple Ubuntu suites).

I'm not 100% sure how we should be fixing that, but that bug shouldn't prevent you from giving the fedora snapshots the "semantically correct" structure.

I also noticed that yesterday evening and I was also wondering what is the best way to fix that. I see two possible options:

We change the names of the keys in snapshot branches dictionary in order to use the intrinsic version of a debian package instead of its extrinsic one (meaning releases/bionic-security/main/1.14.0-0ubuntu1.10 should rather be releases/1.14.0-0ubuntu1.10)

We update the webapp to filter duplicated releases before display as the release name is used instead of the snapshot branches key associated to the release

I would rather go for the second one as keeping the debian/ubuntu suites and components in keys of snapshot branches dictionary seems of interest.
We could do the same for the fedora case as based on my tests it is quite common that extrinsic versions in the form [0-9].[0-9].[0-9]-[0-9].fc[0-9]+
target the same intrinsic version [0-9].[0-9].[0-9]-[0-9].

We should move this branch (ha) of discussion to a separate task. I would really want us to keep (and display) the information about what suites have been detected to contain what package.

How does the webapp get the release name showed in the menu? From the release object, or from the branch name?

I think it would make sense to display the branch name after the release name, if it didn't match the usual refs/tags/$(release name) pattern

In D8386#229912, @olasd wrote:

In D8386#229890, @anlambert wrote:

In D8386#229882, @olasd wrote:

In D8386#229677, @KShivendu wrote:

@anlambert

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

This seems like a misfeature in the webapp:

https://archive.softwareheritage.org/api/1/snapshot/158a3f36b0bd3da461fb7458de44cfa2c94e4270/

The snapshot has multiple branches, with the same version suffix, pointing at the same objects (because the exact same version of the package is present in multiple Ubuntu suites).

I'm not 100% sure how we should be fixing that, but that bug shouldn't prevent you from giving the fedora snapshots the "semantically correct" structure.

I also noticed that yesterday evening and I was also wondering what is the best way to fix that. I see two possible options:

We change the names of the keys in snapshot branches dictionary in order to use the intrinsic version of a debian package instead of its extrinsic one (meaning releases/bionic-security/main/1.14.0-0ubuntu1.10 should rather be releases/1.14.0-0ubuntu1.10)

We update the webapp to filter duplicated releases before display as the release name is used instead of the snapshot branches key associated to the release

I would rather go for the second one as keeping the debian/ubuntu suites and components in keys of snapshot branches dictionary seems of interest.
We could do the same for the fedora case as based on my tests it is quite common that extrinsic versions in the form [0-9].[0-9].[0-9]-[0-9].fc[0-9]+
target the same intrinsic version [0-9].[0-9].[0-9]-[0-9].

We should move this branch (ha) of discussion to a separate task. I would really want us to keep (and display) the information about what suites have been detected to contain what package.

How does the webapp get the release name showed in the menu? From the release object, or from the branch name?

I think it would make sense to display the branch name after the release name, if it didn't match the usual refs/tags/$(release name) pattern

The webapp currently gets the release name from the release object not from the branch name, so the confusion.
I was also converging to that third option as it seems simpler than the two I mentioned above.
I think adding a new column with the branch name in the releases list view is the right way to go here.

KShivendu mentioned this in T4686: Duplicated branch names are rendered by the webapp.Nov 15 2022, 11:21 AM

Rebase against master

Build is green

Patch application report for D8386 (id=31873)

Rebasing onto ea146ce297...

Current branch diff-target is up to date.

Changes applied before test

commit 6ad61aec23943bdbcc14cfab9c8d34c9fc60ff32
Author: KShivendu <shivendu@iitbhilai.ac.in>
Date:   Wed Oct 5 15:44:10 2022 +0530

    feat(fedora): Introduce fedora lister
    
    Summary: Lister to ingest fedora mirrors (.rpm)
    
    Reviewers: #reviewers, vlorentz
    
    Subscribers: vlorentz, olasd
    
    Maniphest Tasks: T4448
    
    Differential Revision: https://forge.softwareheritage.org/D8386

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/862/ for more details.

Harbormaster completed remote builds in B32808: Diff 31873.Nov 15 2022, 11:30 AM

Closed by commit rDLS6ad61aec2394: feat(fedora): Introduce fedora lister (authored by KShivendu). · Explain WhyNov 15 2022, 11:31 AM

This revision was automatically updated to reflect the committed changes.

KShivendu added a commit: rDLS6ad61aec2394: feat(fedora): Introduce fedora lister.

feat(fedora): Introduce fedora listerClosedPublicActions

Details

Diff Detail

Unit TestsFailedView All

Event Timeline

Patch application report for D8386 (id=31284)

Changes applied before test

Patch application report for D8386 (id=31312)

Changes applied before test

Patch application report for D8386 (id=31313)

Changes applied before test

Patch application report for D8386 (id=31776)

Changes applied before test

Patch application report for D8386 (id=31777)

Changes applied before test

Patch application report for D8386 (id=31801)

Changes applied before test

Patch application report for D8386 (id=31803)

Changes applied before test

Patch application report for D8386 (id=31804)

Changes applied before test

Patch application report for D8386 (id=31821)

Changes applied before test

Patch application report for D8386 (id=31840)

Changes applied before test

Patch application report for D8386 (id=31861)

Changes applied before test

Patch application report for D8386 (id=31873)

Changes applied before test

Revision ContentsChangeset List

Diff 30274

README.md

mypy.ini

setup.py

swh/lister/fedora/__init__.py

swh/lister/fedora/lister.py

swh/lister/fedora/tasks.py

swh/lister/fedora/tests/__init__.py

swh/lister/fedora/tests/data/https_eu.edge.kernel.org_fedora_36/primary.xml

swh/lister/fedora/tests/test_lister.py

swh/lister/fedora/tests/test_tasks.py

swh/lister/tests/test_cli.py

feat(fedora): Introduce fedora lister
ClosedPublic
Actions

Unit TestsFailed
View All

Revision Contents
Changeset List

swh/lister/fedora/init.py

swh/lister/fedora/tests/init.py