Page MenuHomeSoftware Heritage

Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone()
ClosedPublic

Authored by vlorentz on Jan 13 2022, 12:22 PM.

Details

Summary

swh-model 5.0.0 removes these arguments from the constructor.

Depends on D6923.

Test Plan

Will fail because this depends on D6936, which isn't released yet.

Diff Detail

Repository
rDLDG Git loader
Branch
weird-git
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 26051
Build 40714: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 40713: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D6939 (id=25129)

Could not rebase; Attempt merge onto 64050aa08b...

Merge made by the 'recursive' strategy.
 requirements-swh.txt                    |   2 +-
 swh/loader/git/converters.py            | 114 +++++++++++++--
 swh/loader/git/tests/test_converters.py | 246 ++++++++++++++++++++++++++++----
 3 files changed, 319 insertions(+), 43 deletions(-)
Changes applied before test
commit f993bc53a5b44e6930300dcb1b7a7a22cfec2d2d
Merge: 64050aa 5d2549a
Author: Jenkins user <jenkins@localhost>
Date:   Thu Jan 13 11:22:17 2022 +0000

    Merge branch 'diff-target' into HEAD

commit 5d2549aceb51315d08d0208fb9e970379ea9e5d7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 17:08:04 2022 +0100

    Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone()
    
    swh-model 5.0.0 removes these arguments from the constructor.

commit c2947c14bd7d7ae9209555caf4455f52fe109ab2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 11:58:29 2022 +0100

    converters: Write raw_manifest of Directory objects
    
    This allows representing git trees with disordered entries,
    as the "normal" data model requires them to be sorted.

commit 6808ddef4ff9a5f330ae384f3654d3acc38a5fb8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest on revisions and releases
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.
    
    This commit is restricted to revisions and releases for now,
    a future commit will add directories.

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/176/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/176/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 13 2022, 12:22 PM
Harbormaster failed remote builds in B25999: Diff 25129!
  • Fix fallback timestamp conversion in dulwich_tsinfo_to_timestamp.

Build has FAILED

Patch application report for D6939 (id=25171)

Could not rebase; Attempt merge onto 64050aa08b...

Merge made by the 'recursive' strategy.
 requirements-swh.txt                    |   2 +-
 swh/loader/git/converters.py            | 114 +++++++++++--
 swh/loader/git/tests/test_converters.py | 279 ++++++++++++++++++++++++++++----
 3 files changed, 352 insertions(+), 43 deletions(-)
Changes applied before test
commit e0b5e4d9132a3fe9c674d7fe82f11c16f66471bd
Merge: 64050aa c8703db
Author: Jenkins user <jenkins@localhost>
Date:   Fri Jan 14 11:40:38 2022 +0000

    Merge branch 'diff-target' into HEAD

commit c8703db672566658bcf3a1145c892492254a5793
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 14 12:40:14 2022 +0100

    Fix fallback timestamp conversion in dulwich_tsinfo_to_timestamp.

commit 5d2549aceb51315d08d0208fb9e970379ea9e5d7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 17:08:04 2022 +0100

    Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone()
    
    swh-model 5.0.0 removes these arguments from the constructor.

commit c2947c14bd7d7ae9209555caf4455f52fe109ab2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 11:58:29 2022 +0100

    converters: Write raw_manifest of Directory objects
    
    This allows representing git trees with disordered entries,
    as the "normal" data model requires them to be sorted.

commit 6808ddef4ff9a5f330ae384f3654d3acc38a5fb8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest on revisions and releases
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.
    
    This commit is restricted to revisions and releases for now,
    a future commit will add directories.

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/177/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/177/console

lower swh-model version requirement

lower swh-model version requirement

Build is green

Patch application report for D6939 (id=25187)

Rebasing onto 64050aa08b...

Current branch diff-target is up to date.
Changes applied before test
commit 4b77e567f80b2105b517e245ad9aaaf4a58a51dc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 17:08:04 2022 +0100

    Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone()
    
    swh-model 5.0.0 removes these arguments from the constructor.

commit dc63db4c0c43e012167bd1d17eade5f6839e16fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 11:58:29 2022 +0100

    converters: Write raw_manifest of Directory objects
    
    This allows representing git trees with disordered entries,
    as the "normal" data model requires them to be sorted.

commit 44cc7d083327449594e93e8ad2d01abb6982be82
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest on revisions and releases
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.
    
    This commit is restricted to revisions and releases for now,
    a future commit will add directories.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/178/ for more details.

Build is green

Patch application report for D6939 (id=25188)

Could not rebase; Attempt merge onto 64050aa08b...

Updating 64050aa..4b77e56
Fast-forward
 requirements-swh.txt                    |   2 +-
 swh/loader/git/converters.py            | 114 +++++++++++--
 swh/loader/git/tests/test_converters.py | 279 ++++++++++++++++++++++++++++----
 3 files changed, 352 insertions(+), 43 deletions(-)
Changes applied before test
commit 4b77e567f80b2105b517e245ad9aaaf4a58a51dc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 17:08:04 2022 +0100

    Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone()
    
    swh-model 5.0.0 removes these arguments from the constructor.

commit dc63db4c0c43e012167bd1d17eade5f6839e16fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jan 12 11:58:29 2022 +0100

    converters: Write raw_manifest of Directory objects
    
    This allows representing git trees with disordered entries,
    as the "normal" data model requires them to be sorted.

commit 44cc7d083327449594e93e8ad2d01abb6982be82
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest on revisions and releases
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.
    
    This commit is restricted to revisions and releases for now,
    a future commit will add directories.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/179/ for more details.

olasd added inline comments.
swh/loader/git/converters.py
152–153

Should this be a warning?

swh/loader/git/tests/test_converters.py
319–320

It's not clear what makes this manifest "not understood" by the custom parser?

swh/loader/git/converters.py
152–153

I don't know; it will most likely trigger the Expected revision to have id %s, but got %s. Recording raw_manifest warning anyway.

swh/loader/git/tests/test_converters.py
319–320

I didn't set it, so the manifest is just b""

Thanks!

swh/loader/git/converters.py
152–153

Yeah, that's true.

swh/loader/git/tests/test_converters.py
319–320

Ah, I guess it's worth a comment then, because from a casual look this is a "well-formed" commit object.

This revision is now accepted and ready to land.Jan 17 2022, 4:44 PM