Page MenuHomeSoftware Heritage

debian.lister: Add integration test which checks scheduled tasks
ClosedPublic

Authored by ardumont on Oct 12 2019, 5:12 AM.

Details

Summary

Simple checks

Related T2032

Related D2133

Test Plan

tox

Diff Detail

Repository
rDLS Listers
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8503
Build 12347: tox-on-jenkinsJenkins
Build 12346: arc lint + arc unit

Event Timeline

swh/lister/debian/tests/conftest.py
19

I want a smaller distribution dataset here for test purposes.

38

This can go away if we agree to add that behavior within requests_mock_datadir directly (D2133).

swh/lister/debian/tests/test_lister.py
32

And now we can:

import ipdb
ipdb.set_trace()

\m/

That could also allow db introspection (if we can print the db_url used by test, cf. swh_listers)

Indeed.

You are connected to database "tests" as user "postgres" on host "127.0.0.1" at port "32127".
tests=# select * from area;
 id | distribution_id |      name       | active
----+-----------------+-----------------+--------
  1 |               1 | stretch/main    | t
  2 |               1 | stretch/contrib | t
(2 rows)

tests=# select name, version, directory from package limit 3;
   name   | version  |      directory
----------+----------+----------------------
 0ad      | 0.0.21-2 | pool/main/0/0ad
 0ad-data | 0.0.21-1 | pool/main/0/0ad-data
 0xffff   | 0.7-2    | pool/main/0/0xffff
(3 rows)
35

btw, @douardda, here is a sample of kwargs:

{'date': '2019-10-12T05:58:09.165557+00:00',
 'origin': {'type': 'deb', 'url': 'deb://Debian/packages/cicero'},
 'packages': {'stretch/contrib/0.7.2-3': {'files': {'cicero_0.7.2-3.diff.gz': {'md5sum': 'a93661b6a48db48d59ba7d26796fc9ce',
                                                                               'name': 'cicero_0.7.2-3.diff.gz',
                                                                               'sha256': 'f039c9642fe15c75bed5254315e2a29f9f2700da0e29d9b0729b3ffc46c8971c',
                                                                               'size': 3964,
                                                                               'uri': 'http://deb.debian.org/debian//pool/contrib/c/cicero/cicero_0.7.2-3.diff.gz'},
                                                    'cicero_0.7.2-3.dsc': {'md5sum': 'd5dac83eb9cfc9bb52a15eb618b4670a',
                                                                           'name': 'cicero_0.7.2-3.dsc',
                                                                           'sha256': '35b7f1048010c67adfd8d70e4961aefd8800eb9a83a4d1cc68088da0009d9a03',
                                                                           'size': 1864,
                                                                           'uri': 'http://deb.debian.org/debian//pool/contrib/c/cicero/cicero_0.7.2-3.dsc'},
                                                    'cicero_0.7.2.orig.tar.gz': {'md5sum': '4353dede07c5728319ba7f5595a7230a',
                                                                                 'name': 'cicero_0.7.2.orig.tar.gz',
                                                                                 'sha256': '63f40f2436ea9f67b44e2d4bd669dbabe90e2635a204526c20e0b3c8ee957786',
                                                                                 'size': 96527,
                                                                                 'uri': 'http://deb.debian.org/debian//pool/contrib/c/cicero/cicero_0.7.2.orig.tar.gz'}},
                                          'id': 23,
                                          'name': 'cicero',
                                          'revision_id': None,
                                          'version': '0.7.2-3'}}}

I don't recall that what we saw (and it's way better as that seems consistent with the current loader ;).

I don't recall that what we saw (and it's way better as that seems consistent with the current loader ;).

Indeed, here is what we saw:

{'args': [],
 'kwargs': {'date': '2018-10-28T10:40:10.006139+00:00',
            'origin': {'type': 'deb',
                       'url': 'deb://Debian-Security/packages/nginx'},
            'packages': {
                'jessie-kfreebsd/updates/main/1.6.2-5+deb8u5': {
                    'id': 160353,
                    'name': 'nginx',
                    'revision_id': '5fc3d32414a6782aafb39c57d2077fb476cda588',
                    'version': '1.6.2-5+deb8u5'},
                'jessie/updates/main/1.6.2-5+deb8u5': {
                    'id': 159043,
                    'name': 'nginx',
                    'revision_id': '5fc3d32414a6782aafb39c57d2077fb476cda588',
                    'version': '1.6.2-5+deb8u5'},
                'wheezy/updates/main/1.2.1-2.2+wheezy4+deb7u1': {
                    'id': 157309,
                    'name': 'nginx',
                    'revision_id': 'e22cddb727238a4307bb36686e81b5af681466b7',
                    'version': '1.2.1-2.2+wheezy4+deb7u1'}}}}
swh/lister/debian/tests/test_lister.py
35

Indeed, we did not see any 'files' declarations in scheduler's tasks we found in the db.

Use latest pytest_plugin fixture with http support

swh/lister/debian/tests/conftest.py
38

done

This revision is now accepted and ready to land.Oct 17 2019, 2:12 PM