Details

Reviewers

Group Reviewers

Commits

rDLSa60e0bbc4139: lister.debian: Fix task creation
rDLSb745c5a735ba: lister.debian: Default to run a listing on debian distribution
rDLSf87279240722: debian.lister: Send origin url as load-debian task parameter

Summary

Instead of the old origin dict. That's what the debian loaders (old and new)
expect.

@douardda, in the end, there was only this discrepancy in the data structure
sent to the loader(s), as shown by the example:

{
    'url': 'deb://Debian/packages/wolf4sdl',  # <- prior to this diff, this was a dict with keys {url, type}
    'date': '2019-11-04T13:26:29.908746+00:00',
    'packages': {
        'stretch/contrib/1.7+svn262+dfsg1-4': {
            'id': 145,
            'name': 'wolf4sdl',
            'files': {
                'wolf4sdl_1.7+svn262+dfsg1-4.dsc': {
                    'uri': 'http://deb.debian.org/debian//pool/contrib/w/wolf4sdl/wolf4sdl_1.7+svn262+dfsg1-4.dsc',
                    'name': 'wolf4sdl_1.7+svn262+dfsg1-4.dsc',
                    'size': 2049,
                    'md5sum': '399d1bcf461dd9059fc79e73f3f54efc',
                    'sha256': '3d1e09e1b633ee3954f24279ddeabc1d2ad4d58f58b493d777478d58c28894a5'
                },
                'wolf4sdl_1.7+svn262+dfsg1.orig.tar.gz': {
                    'uri': 'http://deb.debian.org/debian//pool/contrib/w/wolf4sdl/wolf4sdl_1.7+svn262+dfsg1.orig.tar.gz',
                    'name': 'wolf4sdl_1.7+svn262+dfsg1.orig.tar.gz',
                    'size': 681684,
                    'md5sum': '854cbc8882d4761c230dd84be3e352c1',
                    'sha256': '00e49d840ad6537a7f461645363f2e54303902e4a414b1af8ecd90a2f800aafd'
                },
                'wolf4sdl_1.7+svn262+dfsg1-4.debian.tar.xz': {
                    'uri': 'http://deb.debian.org/debian//pool/contrib/w/wolf4sdl/wolf4sdl_1.7+svn262+dfsg1-4.debian.tar.xz',
                    'name': 'wolf4sdl_1.7+svn262+dfsg1-4.debian.tar.xz',
                    'size': 23212,
                    'md5sum': '7bf98fd34c30e4497ba197ada943bcc7',
                    'sha256': 'afb975f352debc5aba0232f4de01556db0e7c4bd31824e04381afcddf9e0b3d0'
                }
            },
            'version': '1.7+svn262+dfsg1-4',
            'revision_id': None
        }
    }
}

Note: Output from integration test run.

Test Plan

tox

Diff Detail

Repository

rDLS Listers

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 8808
Build 12837: tox-on-jenkins	Jenkins
Build 12836: arc lint + arc unit

Event Timeline

ardumont created this revision.Nov 4 2019, 2:52 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/469/ for more details.

Harbormaster completed remote builds in B8801: Diff 7634.Nov 4 2019, 2:56 PM

one docker-dev run agrees.

Here is another sample:

{
    'url': 'deb://Debian/packages/golang-github-jwilder-encoding'
    'date': '2019-11-04T14:34:38.449084+00:00',
    'packages': {
        'buster/main/0.0~git20170811.b4e1701-3': {
            'files': {
                'golang-github-jwilder-encoding_0.0~git20170811.b4e1701-3.debian.tar.xz': {
                    'md5sum': '487b69abdb59247bf237620bbdea6767',
                    'name': 'golang-github-jwilder-encoding_0.0~git20170811.b4e1701-3.debian.tar.xz',
                    'sha256': 'b73b092f435ab87ccfd791483d51642a67f0df05ebacdb06dbc42fe496a62b7b',
                    'size': 2048,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20170811.b4e1701-3.debian.tar.xz'},
                'golang-github-jwilder-encoding_0.0~git20170811.b4e1701-3.dsc': {
                    'md5sum': '3ee14ce9c4097b67fe71d92fc966ff6e',
                    'name': 'golang-github-jwilder-encoding_0.0~git20170811.b4e1701-3.dsc',
                    'sha256': 'e6f0fd15fc79e4bb73210396bfadad8784345ad7b18ee377852e3684b1e6d421',
                    'size': 2435,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20170811.b4e1701-3.dsc'
                },
                'golang-github-jwilder-encoding_0.0~git20170811.b4e1701.orig.tar.xz': {
                    'md5sum': '99b34637a73dd8e3b10ddb9bedb15272',
                    'name': 'golang-github-jwilder-encoding_0.0~git20170811.b4e1701.orig.tar.xz',
                    'sha256': 'e2b1b67686aa0977d9ddb5889d022f2d6e564ea8f700229c2278e28f617cc373',
                    'size': 8060,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20170811.b4e1701.orig.tar.xz'}},
            'id': 139820,
            'name': 'golang-github-jwilder-encoding',
            'revision_id': None,
            'version': '0.0~git20170811.b4e1701-3'
        },
        'stretch/main/0.0~git20160426.0.ac74639-1': {
            'files': {
                'golang-github-jwilder-encoding_0.0~git20160426.0.ac74639-1.debian.tar.xz': {
                    'md5sum': '885aabea0abf77306d54248864584113',
                    'name': 'golang-github-jwilder-encoding_0.0~git20160426.0.ac74639-1.debian.tar.xz',
                    'sha256': 'fd710bee0649912f3aa7401ab9ee1f990949c56db500e134571c17461776d6f9',
                    'size': 1888,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20160426.0.ac74639-1.debian.tar.xz'
                },
                'golang-github-jwilder-encoding_0.0~git20160426.0.ac74639-1.dsc': {
                    'md5sum': '1a0d5617dec8c7a124710973a457bcdd',
                    'name': 'golang-github-jwilder-encoding_0.0~git20160426.0.ac74639-1.dsc',
                    'sha256': '846445d5492e9eb32bc7dbfd5f5f99ab908edef312f224970822849d17fcf0b8',
                    'size': 2347,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20160426.0.ac74639-1.dsc'
                },
                'golang-github-jwilder-encoding_0.0~git20160426.0.ac74639.orig.tar.xz': {
                    'md5sum': 'f1e6dfd7d37b55143af6abfe04bafae0',
                    'name': 'golang-github-jwilder-encoding_0.0~git20160426.0.ac74639.orig.tar.xz',
                    'sha256': '72c1968569c1bb52d8d08f5eaf3ec89e8836eb2ba07df5e290df9574065def09',
                    'size': 7588,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20160426.0.ac74639.orig.tar.xz'
                }
            },
            'id': 113918,
            'name': 'golang-github-jwilder-encoding',
            'revision_id': None,
            'version': '0.0~git20160426.0.ac74639-1'
        },
        'stretch/main/0.0~git20160927.0.4dada27-1': {
            'files': {
                'golang-github-jwilder-encoding_0.0~git20160927.0.4dada27-1.debian.tar.xz': {
                    'md5sum': 'eba1023fe0782eabc335894a87ff5ff5',
                    'name': 'golang-github-jwilder-encoding_0.0~git20160927.0.4dada27-1.debian.tar.xz',
                    'sha256': '347feaafe36d26712043281403b21988c6723891b7cc0c335da9d8261cf58d71',
                    'size': 1908,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20160927.0.4dada27-1.debian.tar.xz'
                },
                'golang-github-jwilder-encoding_0.0~git20160927.0.4dada27-1.dsc': {
                    'md5sum': 'fa9e70d4dbea46fc6e684feb0642c49e',
                    'name': 'golang-github-jwilder-encoding_0.0~git20160927.0.4dada27-1.dsc',
                    'sha256': '4e5cabf0f1b1ff9d8674589870ff6a030be1f95e2ff7ad7615f13b5bc54e4356',
                    'size': 2404,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20160927.0.4dada27-1.dsc'
                },
                'golang-github-jwilder-encoding_0.0~git20160927.0.4dada27.orig.tar.xz': {
                    'md5sum': 'c0183cb3d69dc0256e42c6f63a9e9fa2',
                    'name': 'golang-github-jwilder-encoding_0.0~git20160927.0.4dada27.orig.tar.xz',
                    'sha256': '17adca7ed379b2f087127a5e955d562d575a989fff0459193d63aee8424d18da',
                    'size': 7604,
                    'uri': 'http://deb.debian.org/debian//pool/main/g/golang-github-jwilder-encoding/golang-github-jwilder-encoding_0.0~git20160927.0.4dada27.orig.tar.xz'
                }
            },
            'id': 113919,
            'name': 'golang-github-jwilder-encoding',
            'revision_id': None,
            'version': '0.0~git20160927.0.4dada27-1'
        }
    },
}

Note: local docker-compose.override.yml with the local repositories mounted (swh-lister)

(Referencing it here to not forget)

I had to amend the local code to:

add a retries_left to something not null (or else the task creation fails) [1]

have a default distribution set to Debian in the load method for the lister [2]

scheduler: add a task-type 'load-deb-package' [3]

[1] or else the task creation fails because retries_left has a not null constraint...

[2] cli would not work for the lister debian as is.

swh lister run --lister debian

Because that calls the run method without any arguments (because it's a kind of generic code).
In the current code, the debian implementation wants a distribution parameter though.

[3] because in production the expected task type is 'load-deb-package'.
Somehow load-debian-package task type exists within docker-dev. I have yet to search why.

lister.debian: Fix [1] task creation
lister.debian: Default to run a listing on debian distribution [2]