Page MenuHomeSoftware Heritage

Fix the bitbucket lister;
ClosedPublic

Authored by douardda on Jan 31 2019, 4:15 PM.

Details

Summary
  • Handle properly the date-like bounds
  • Fix the SWHIndexingLister.db_partition_indices the db query actually returns a table-like (2d) structure.

Also

  • Log errors when fetching an url in SWHListerHttpTransport
  • Use the named logger instead of the root logger in lister_base.py

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Jan 31 2019, 4:15 PM
vlorentz added inline comments.
requirements.txt
7 ↗(On Diff #3324)

Did you check if you can use arrow (which we already depend on) instead?

douardda marked an inline comment as done.Jan 31 2019, 6:09 PM
douardda added inline comments.
requirements.txt
7 ↗(On Diff #3324)

nope, old habits I guess... let me check...

ah, doc says

Some ISO-8601 compliant strings are recognized and parsed without a format string

so maybe it's enough... maybe

ardumont accepted this revision.Feb 1 2019, 8:51 AM
ardumont added a subscriber: ardumont.

Just a couple of non blocking questions.

swh/lister/bitbucket/lister.py
46 ↗(On Diff #3324)

What is that error?

Also why not a ValueError?

This revision is now accepted and ready to land.Feb 1 2019, 8:51 AM

What is that error?

Don't remember, looks pretty much like a "debug" statement in fact. Let me check that

douardda marked an inline comment as done.Feb 1 2019, 11:09 AM
douardda added inline comments.
swh/lister/bitbucket/lister.py
46 ↗(On Diff #3324)

Ok, in fact this looks very much like a "debug" statement before the hunk below in swh/lister/core/indexing_lister.py

douardda marked an inline comment as done.Feb 1 2019, 11:20 AM
douardda added inline comments.
requirements.txt
7 ↗(On Diff #3324)

https://github.com/crsmithdev/arrow/issues/91 makes me feel I want to stick to iso8601 for now

douardda updated this revision to Diff 3335.Feb 1 2019, 11:21 AM

remove a debug piece of code

douardda updated this revision to Diff 3336.Feb 1 2019, 11:40 AM

Do really remove a debug piece of code and fix the test

douardda updated this revision to Diff 3367.Feb 1 2019, 3:38 PM

rebased

This revision was automatically updated to reflect the committed changes.