Page MenuHomeSoftware Heritage

debian.lister: Add integration test which checks scheduled tasks
ClosedPublic

Authored by ardumont on Oct 12 2019, 5:12 AM.

Details

Summary

Simple checks

Related T2032

Related D2133

Test Plan

tox

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

swh/lister/debian/tests/conftest.py
19

I want a smaller distribution dataset here for test purposes.

38

This can go away if we agree to add that behavior within requests_mock_datadir directly (D2133).

swh/lister/debian/tests/test_lister.py
32

And now we can:

import ipdb
ipdb.set_trace()

\m/

That could also allow db introspection (if we can print the db_url used by test, cf. swh_listers)

Indeed.

You are connected to database "tests" as user "postgres" on host "127.0.0.1" at port "32127".
tests=# select * from area;
 id | distribution_id |      name       | active
----+-----------------+-----------------+--------
  1 |               1 | stretch/main    | t
  2 |               1 | stretch/contrib | t
(2 rows)

tests=# select name, version, directory from package limit 3;
   name   | version  |      directory
----------+----------+----------------------
 0ad      | 0.0.21-2 | pool/main/0/0ad
 0ad-data | 0.0.21-1 | pool/main/0/0ad-data
 0xffff   | 0.7-2    | pool/main/0/0xffff
(3 rows)
35

btw, @douardda, here is a sample of kwargs:

{'date': '2019-10-12T05:58:09.165557+00:00',
 'origin': {'type': 'deb', 'url': 'deb://Debian/packages/cicero'},
 'packages': {'stretch/contrib/0.7.2-3': {'files': {'cicero_0.7.2-3.diff.gz': {'md5sum': 'a93661b6a48db48d59ba7d26796fc9ce',
                                                                               'name': 'cicero_0.7.2-3.diff.gz',
                                                                               'sha256': 'f039c9642fe15c75bed5254315e2a29f9f2700da0e29d9b0729b3ffc46c8971c',
                                                                               'size': 3964,
                                                                               'uri': 'http://deb.debian.org/debian//pool/contrib/c/cicero/cicero_0.7.2-3.diff.gz'},
                                                    'cicero_0.7.2-3.dsc': {'md5sum': 'd5dac83eb9cfc9bb52a15eb618b4670a',
                                                                           'name': 'cicero_0.7.2-3.dsc',
                                                                           'sha256': '35b7f1048010c67adfd8d70e4961aefd8800eb9a83a4d1cc68088da0009d9a03',
                                                                           'size': 1864,
                                                                           'uri': 'http://deb.debian.org/debian//pool/contrib/c/cicero/cicero_0.7.2-3.dsc'},
                                                    'cicero_0.7.2.orig.tar.gz': {'md5sum': '4353dede07c5728319ba7f5595a7230a',
                                                                                 'name': 'cicero_0.7.2.orig.tar.gz',
                                                                                 'sha256': '63f40f2436ea9f67b44e2d4bd669dbabe90e2635a204526c20e0b3c8ee957786',
                                                                                 'size': 96527,
                                                                                 'uri': 'http://deb.debian.org/debian//pool/contrib/c/cicero/cicero_0.7.2.orig.tar.gz'}},
                                          'id': 23,
                                          'name': 'cicero',
                                          'revision_id': None,
                                          'version': '0.7.2-3'}}}

I don't recall that what we saw (and it's way better as that seems consistent with the current loader ;).

I don't recall that what we saw (and it's way better as that seems consistent with the current loader ;).

Indeed, here is what we saw:

{'args': [],
 'kwargs': {'date': '2018-10-28T10:40:10.006139+00:00',
            'origin': {'type': 'deb',
                       'url': 'deb://Debian-Security/packages/nginx'},
            'packages': {
                'jessie-kfreebsd/updates/main/1.6.2-5+deb8u5': {
                    'id': 160353,
                    'name': 'nginx',
                    'revision_id': '5fc3d32414a6782aafb39c57d2077fb476cda588',
                    'version': '1.6.2-5+deb8u5'},
                'jessie/updates/main/1.6.2-5+deb8u5': {
                    'id': 159043,
                    'name': 'nginx',
                    'revision_id': '5fc3d32414a6782aafb39c57d2077fb476cda588',
                    'version': '1.6.2-5+deb8u5'},
                'wheezy/updates/main/1.2.1-2.2+wheezy4+deb7u1': {
                    'id': 157309,
                    'name': 'nginx',
                    'revision_id': 'e22cddb727238a4307bb36686e81b5af681466b7',
                    'version': '1.2.1-2.2+wheezy4+deb7u1'}}}}
swh/lister/debian/tests/test_lister.py
35

Indeed, we did not see any 'files' declarations in scheduler's tasks we found in the db.

Use latest pytest_plugin fixture with http support

swh/lister/debian/tests/conftest.py
38

done

This revision is now accepted and ready to land.Oct 17 2019, 2:12 PM