Page MenuHomeSoftware Heritage
Differential D6819

Add test_revision_add_old, xfail on postgresql
AbandonedPublic
Actions

Authored by vlorentz on Dec 9 2021, 5:05 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

What should we do about this?

Diff Detail

Repository
rDSTO Storage manager
Branch
weird-git
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 25553
Build 39950: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 39949: arc lint + arc unit

Event Timeline

vlorentz created this revision.Dec 9 2021, 5:05 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 9 2021, 5:05 PM
vlorentz updated this revision to Diff 24699.Dec 9 2021, 5:05 PM

remove stuff that shouldn't be committed

swh-public-ci added a comment.Dec 9 2021, 5:06 PM

Build has FAILED

Patch application report for D6819 (id=24698)

Could not rebase; Attempt merge onto 7cb4128e40...

Updating 7cb4128e..5d7b43ab
Fast-forward
 swh/storage/postgresql/converters.py            |   5 +-
 swh/storage/tests/storage_data.py               |  26 ++++
 swh/storage/tests/storage_tests.py              |  17 +++
 swh/storage/tests/test_postgresql.py            |   7 +
 swh/storage/tests/test_postgresql_converters.py | 187 +++++++++++++++++-------
 5 files changed, 190 insertions(+), 52 deletions(-)
Changes applied before test
commit 5d7b43abfd0fe7dd4876102218af2f0f8731d335
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 17:04:40 2021 +0100

    Add test_revision_add_old, xfail on postgresql

commit 110e2c3cbd6fca00a0e8cce5e43cf81c7fd216cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 16:25:20 2021 +0100

    postgresql: Fix one-by-one error in db_to_date on negative dates
    
    Using `int()` on `date.timestamp()` rounded it up (toward zero),
    but the semantics of `model.Timestamp` is that the actual time is
    `ts.seconds + ts.microseconds/1000000`, so all negative dates were
    shifted one second up.
    
    In particular, this causes dates from
    `1969-12-31T23:59:59.000001` to `1969-12-31T23:59:59.999999`
    (inclusive) to smash into dates from
    `1970-01-01T00:00:00.000001` to `1970-01-01T00:00:00.999999`,
    which is how I discovered the issue.

commit 34ca67e21f68de920f603223f629070683018c2a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 16:23:44 2021 +0100

    postgresql: Add tests for db_to_date.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1499/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1499/console

Harbormaster failed remote builds in B25551: Diff 24698!Dec 9 2021, 5:06 PM
swh-public-ci added a comment.Dec 9 2021, 5:07 PM

Build has FAILED

Patch application report for D6819 (id=24699)

Could not rebase; Attempt merge onto 7cb4128e40...

Updating 7cb4128e..0bdc7e8b
Fast-forward
 swh/storage/postgresql/converters.py            |   5 +-
 swh/storage/tests/storage_data.py               |  24 +++
 swh/storage/tests/storage_tests.py              |  17 +++
 swh/storage/tests/test_postgresql.py            |   7 +
 swh/storage/tests/test_postgresql_converters.py | 187 +++++++++++++++++-------
 5 files changed, 188 insertions(+), 52 deletions(-)
Changes applied before test
commit 0bdc7e8b9e11ed42396740c8f915d9a01a5de53e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 17:04:40 2021 +0100

    Add test_revision_add_old, xfail on postgresql

commit 110e2c3cbd6fca00a0e8cce5e43cf81c7fd216cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 16:25:20 2021 +0100

    postgresql: Fix one-by-one error in db_to_date on negative dates
    
    Using `int()` on `date.timestamp()` rounded it up (toward zero),
    but the semantics of `model.Timestamp` is that the actual time is
    `ts.seconds + ts.microseconds/1000000`, so all negative dates were
    shifted one second up.
    
    In particular, this causes dates from
    `1969-12-31T23:59:59.000001` to `1969-12-31T23:59:59.999999`
    (inclusive) to smash into dates from
    `1970-01-01T00:00:00.000001` to `1970-01-01T00:00:00.999999`,
    which is how I discovered the issue.

commit 34ca67e21f68de920f603223f629070683018c2a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 16:23:44 2021 +0100

    postgresql: Add tests for db_to_date.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1500/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1500/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 9 2021, 5:07 PM
Harbormaster failed remote builds in B25552: Diff 24699!
vlorentz updated this revision to Diff 24700.Dec 9 2021, 5:10 PM

ditto

swh-public-ci added a comment.Dec 9 2021, 5:18 PM

Build is green

Patch application report for D6819 (id=24700)

Could not rebase; Attempt merge onto 7cb4128e40...

Updating 7cb4128e..7765309a
Fast-forward
 swh/storage/postgresql/converters.py            |   5 +-
 swh/storage/tests/storage_data.py               |  23 +++
 swh/storage/tests/storage_tests.py              |  17 +++
 swh/storage/tests/test_postgresql.py            |   7 +
 swh/storage/tests/test_postgresql_converters.py | 187 +++++++++++++++++-------
 5 files changed, 187 insertions(+), 52 deletions(-)
Changes applied before test
commit 7765309a7e3b8a2d2e8f59fd434849fe247e28a3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 17:04:40 2021 +0100

    Add test_revision_add_old, xfail on postgresql

commit 110e2c3cbd6fca00a0e8cce5e43cf81c7fd216cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 16:25:20 2021 +0100

    postgresql: Fix one-by-one error in db_to_date on negative dates
    
    Using `int()` on `date.timestamp()` rounded it up (toward zero),
    but the semantics of `model.Timestamp` is that the actual time is
    `ts.seconds + ts.microseconds/1000000`, so all negative dates were
    shifted one second up.
    
    In particular, this causes dates from
    `1969-12-31T23:59:59.000001` to `1969-12-31T23:59:59.999999`
    (inclusive) to smash into dates from
    `1970-01-01T00:00:00.000001` to `1970-01-01T00:00:00.999999`,
    which is how I discovered the issue.

commit 34ca67e21f68de920f603223f629070683018c2a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Dec 9 16:23:44 2021 +0100

    postgresql: Add tests for db_to_date.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1501/ for more details.

Harbormaster completed remote builds in B25553: Diff 24700.Dec 9 2021, 5:18 PM
vlorentz requested review of this revision.Dec 9 2021, 5:18 PM
vlorentz added a child revision: D6821: Add tests checking round-tripping of dir/rev/rel/snp objects generated by Hypothesis.Dec 9 2021, 5:28 PM
olasd added a subscriber: olasd.Dec 13 2021, 10:52 AM

Can you get a cite for the "loss of precision" on very old timestamps? timestamp with time zone is supposed to have a microsecond of precision in its whole range of operation, which goes from year -4713 to year 294276.

vlorentz abandoned this revision.EditedDec 13 2021, 1:12 PM

Actually, it was a "bug" of psycopg2 < 2.9.0 with non-integer timezones: https://www.psycopg.org/docs/usage.html#time-zones-handling

I'll submit a new diff to bump the psycopg2 dependency and remove the xfail

vlorentz mentioned this in D6825: Add test_revision_add_fractional_timezone.Dec 13 2021, 1:23 PM
vlorentz mentioned this in D6821: Add tests checking round-tripping of dir/rev/rel/snp objects generated by Hypothesis.Dec 13 2021, 1:23 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
storage/
tests/
23 lines100%
Loading...
17 lines99%
Loading...
7 lines99%
Loading...

Diff 24700

View Options

swh/storage/tests/storage_data.py

Loading...
View Options

swh/storage/tests/storage_tests.py

Loading...
View Options

swh/storage/tests/test_postgresql.py

Loading...
About · Developer information · Legal