Page MenuHomeSoftware Heritage

journal_data: Use datetime for origin-visit date field
ClosedPublic

Authored by ardumont on Jun 8 2020, 6:11 PM.

Details

Summary

This is preparatory work for D3241

Test Plan

tox is unhappy
i've yet to understand why

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build has FAILED

Patch application report for D3246 (id=11505)

Rebasing onto 64cd01fc7c...

Current branch diff-target is up to date.
Changes applied before test
commit fe9d6aa50073a82ef4767c700b927db5287dd2e3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Jun 8 18:10:52 2020 +0200

    journal_data: Use datetime for origin-visit date field
    
    This is preparatory work for D3241

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/82/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/82/console

Could you amend the commit message to mention it's because we want to phase out the ISO string?

This revision is now accepted and ready to land.Jun 8 2020, 6:42 PM

Could you amend the commit message to mention it's because we want to phase out the ISO string?

Sure thing.

That's the next thing I will do when I understand and fix the test failure
I don't get it for now [1]

[1] D3241#inline-22403

That's the next thing I will do when I understand and fix the test failure

ok then, test needs the date to be string and not datetime so that's the missing cog:

$ PATH=/usr/bin:$PATH tox -e py3 -- -x -s --no-cov -k test_kafka_writer
...
> /home/tony/work/inria/repo/swh/swh-environment/swh-journal/.tox/py3/lib/python3.7/site-packages/swh/journal/pytest_plugin.py(98)assert_all_objects_consumed()
-> assert value in received_values, (
(Pdb) value
{'origin': 'https://somewhere.org/den/fox', 'date': datetime.datetime(2013, 5, 7, 4, 20, 39, 369271, tzinfo=datetime.timezone.utc), 'snapshot': None, 'status': 'ongoing', 'metadata': {'foo': 'bar'}, 'type': 'git', 'visit': 1}
(Pdb) value in received_values
False
(Pdb) received_values
({'origin': 'https://somewhere.org/den/fox', 'date': '2018-11-27 17:20:39+00:00', 'status': 'ongoing', 'type': 'git', 'snapshot': None, 'metadata': {'baz': 'qux'}, 'visit': 2}, {'origin': 'https://somewhere.org/den/fox', 'date': '2018-11-27 17:20:39+00:00', 'status': 'full', 'type': 'git', 'snapshot': b"t,\xdck\xe7\xbfn\x89[\x05R'\xc20\x00p\xf0V\xe0{", 'metadata': {'baz': 'qux'}, 'visit': 3}, {'origin': 'https://somewhere.org/den/fox', 'date': '2013-05-07 04:20:39.369271+00:00', 'status': 'ongoing', 'type': 'git', 'snapshot': None, 'metadata': {'foo': 'bar'}, 'visit': 1}, {'origin': 'https://overtherainbow.org/fox/den', 'date': '2015-11-27 17:20:39+00:00', 'status': 'partial', 'type': 'hg', 'snapshot': b'\xec\xeeH9z\x92\xb0\xd04\xe9u*\x17E\x9f6\x91\xa7>\xf9', 'metadata': {'something': 'wrong occurred'}, 'visit': 2}, {'origin': 'https://overtherainbow.org/fox/den', 'date': '2014-11-27 17:20:39+00:00', 'status': 'ongoing', 'type': 'hg', 'snapshot': None, 'metadata': {'baz': 'qux'}, 'visit': 1})

Values in received_values in that context for origin-visit have their date "stringified":

> /home/tony/work/inria/repo/swh/swh-environment/swh-journal/.tox/py3/lib/python3.7/site-packages/swh/journal/pytest_plugin.py(86)assert_all_objects_consumed()
-> for value in received_values:
(Pdb) value
{'origin': 'https://somewhere.org/den/fox', 'date': '2018-11-27 17:20:39+00:00', 'status': 'ongoing', 'type': 'git', 'snapshot': None, 'metadata': {'baz': 'qux'}, 'visit': 2}
  • Fix test
  • Rework commit message to explicit the phase out iso8601 string date field intent

Build is green

Patch application report for D3246 (id=11508)

Rebasing onto 64cd01fc7c...

Current branch diff-target is up to date.
Changes applied before test
commit ee934c7de8d6a71b930c3376d7d7a4c958eb9a6a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Jun 8 18:10:52 2020 +0200

    test: Use origin-visit date field as datetime to phase out iso8601 str
    
    This is also preparatory work for D3241

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/83/ for more details.