Page MenuHomeSoftware Heritage
Differential D6894

converters: Write object_bytes and raw_manifest on revisions and releases
ClosedPublic
Actions

Authored by vlorentz on Jan 7 2022, 5:28 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T3752: Store/represent time offsets as strings
Commits
rDLDG44cc7d083327: converters: Write object_bytes and raw_manifest on revisions and releases
Summary

This allows representing all git objects instead of rejecting
objects that do not fit in our "normal" data model.

This diff is restricted to revisions and releases for now,
a future diff will add directories.

Depends on D6890.

Test Plan

will fail until D6890 is released.

Diff Detail

Repository
rDLDG Git loader
Branch
weird-git
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 25875
Build 40437: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 40436: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_converters.TestConverters::test_weird_tag
self = <swh.loader.git.tests.test_converters.TestConverters object at 0x7f491bc9d048> def test_weird_tag(self):
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_converters.TestConverters::test_author_line_to_author
14 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_converters.TestConverters::test_blob_to_content
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_converters.TestConverters::test_commit_to_revision
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_converters.TestConverters::test_commit_to_revision_with_extra_headers
View Full Test Results (1 Failed · 109 Passed)

Event Timeline

vlorentz created this revision.Jan 7 2022, 5:28 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 7 2022, 5:28 PM
vlorentz published this revision for review.Jan 7 2022, 5:28 PM
swh-public-ci added a comment.Jan 7 2022, 5:30 PM

Build has FAILED

Patch application report for D6894 (id=25005)

Rebasing onto 0cc96c25ab...

Current branch diff-target is up to date.
Changes applied before test
commit cecc39e23675b249b00d89048ab7ab86bb12a565
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest.
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/159/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/159/console

Harbormaster failed remote builds in B25875: Diff 25005!Jan 7 2022, 5:30 PM
vlorentz updated this revision to Diff 25006.Jan 7 2022, 5:33 PM

bump dep

swh-public-ci added a comment.Jan 7 2022, 5:34 PM

Build has FAILED

Patch application report for D6894 (id=25006)

Rebasing onto 0cc96c25ab...

Current branch diff-target is up to date.
Changes applied before test
commit 380504edc486914d5cddec17dab234ddbc55daa8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest.
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/160/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/160/console

Harbormaster failed remote builds in B25876: Diff 25006!Jan 7 2022, 5:34 PM
olasd added a subscriber: olasd.Jan 10 2022, 2:36 PM

Should we submit a change to dulwich to get access to, at least, the raw author/committer lines (or even better raw unparsed headers)?

requirements-swh.txt
3 ↗(On Diff #25006)

Spurious v

swh/loader/git/converters.py
271

s/Adding/Recording/ ?

vlorentz updated this revision to Diff 25014.Jan 10 2022, 3:53 PM

apply comments

swh-public-ci added a comment.Jan 10 2022, 3:54 PM

Build has FAILED

Patch application report for D6894 (id=25014)

Rebasing onto 0cc96c25ab...

Current branch diff-target is up to date.
Changes applied before test
commit 0ae25b4d7055a369a10e7a8754cccf3c00ab8b22
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest.
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/161/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/161/console

Harbormaster failed remote builds in B25884: Diff 25014!Jan 10 2022, 3:54 PM
vlorentz added a comment.Jan 10 2022, 3:55 PM

I don't know, this seems like a very niche feature

vlorentz marked 2 inline comments as done.Jan 10 2022, 3:56 PM
swh-public-ci added a comment.Jan 10 2022, 4:14 PM

Build has FAILED

Patch application report for D6894 (id=25014)

Rebasing onto 0cc96c25ab...

Current branch diff-target is up to date.
Changes applied before test
commit 0ae25b4d7055a369a10e7a8754cccf3c00ab8b22
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest.
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/162/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/162/console

vlorentz updated this revision to Diff 25041.Jan 10 2022, 6:41 PM

rebase

swh-public-ci added a comment.Jan 10 2022, 6:44 PM

Build is green

Patch application report for D6894 (id=25041)

Rebasing onto 4dfd2ed12f...

Current branch diff-target is up to date.
Changes applied before test
commit 96929ff0e50a219b51e183ebcb647be0a4c59239
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest.
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/171/ for more details.

Harbormaster completed remote builds in B25909: Diff 25041.Jan 10 2022, 6:44 PM
vlorentz planned changes to this revision.Jan 12 2022, 10:39 AM
vlorentz edited the summary of this revision. (Show Details)Jan 12 2022, 11:15 AM
vlorentz updated this revision to Diff 25073.Jan 12 2022, 11:15 AM

actually implement this for commits as well

swh-public-ci added a comment.Jan 12 2022, 11:17 AM

Build is green

Patch application report for D6894 (id=25073)

Rebasing onto 64050aa08b...

First, rewinding head to replay your work on top of it...
Applying: converters: Write object_bytes and raw_manifest.
Changes applied before test
commit 85afafeb03e70d8dba097a792467adf21d52c292
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest.
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.
    
    This commit is restricted to revisions and releases for now,
    a future commit will add directories.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/173/ for more details.

Harbormaster completed remote builds in B25948: Diff 25073.Jan 12 2022, 11:17 AM
vlorentz retitled this revision from converters: Write object_bytes and raw_manifest. to converters: Write object_bytes and raw_manifest on revisions and releases.Jan 12 2022, 11:56 AM
vlorentz updated this revision to Diff 25082.Jan 12 2022, 11:57 AM

fix commit msg

vlorentz added a child revision: D6923: converters: Write raw_manifest of Directory objects.Jan 12 2022, 11:58 AM
swh-public-ci added a comment.Jan 12 2022, 11:59 AM

Build is green

Patch application report for D6894 (id=25082)

Rebasing onto 64050aa08b...

First, rewinding head to replay your work on top of it...
Applying: converters: Write object_bytes and raw_manifest on revisions and releases
Changes applied before test
commit 36ab86c7fc17052b4ad69e50d19f10a326dc8e16
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest on revisions and releases
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.
    
    This commit is restricted to revisions and releases for now,
    a future commit will add directories.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/174/ for more details.

Harbormaster completed remote builds in B25956: Diff 25082.Jan 12 2022, 11:59 AM
vlorentz added a task: T3752: Store/represent time offsets as strings.Jan 12 2022, 3:24 PM
olasd accepted this revision.Jan 17 2022, 4:22 PM

This all looks fine to me, thanks!

swh/loader/git/converters.py
142–144

So, we'll be dropping these arguments once we drop them off of storage, correct?

This revision is now accepted and ready to land.Jan 17 2022, 4:22 PM
vlorentz added inline comments.Jan 17 2022, 4:27 PM
swh/loader/git/converters.py
142–144

yes

vlorentz updated this revision to Diff 25226.Jan 18 2022, 10:57 AM

rebase

swh-public-ci added a comment.Jan 18 2022, 11:00 AM

Build is green

Patch application report for D6894 (id=25226)

Rebasing onto 64050aa08b...

Current branch diff-target is up to date.
Changes applied before test
commit 44cc7d083327449594e93e8ad2d01abb6982be82
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 7 17:27:10 2022 +0100

    converters: Write object_bytes and raw_manifest on revisions and releases
    
    This allows representing all git objects instead of rejecting
    objects that do not fit in our "normal" data model.
    
    This commit is restricted to revisions and releases for now,
    a future commit will add directories.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/180/ for more details.

Harbormaster completed remote builds in B26087: Diff 25226.Jan 18 2022, 11:00 AM
Closed by commit rDLDG44cc7d083327: converters: Write object_bytes and raw_manifest on revisions and releases (authored by vlorentz). · Explain WhyJan 18 2022, 11:02 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLDG44cc7d083327: converters: Write object_bytes and raw_manifest on revisions and releases.

Revision Contents
Changeset List

PathSize
swh/
loader/
git/
74 lines
tests/
70 lines

Diff 25005

View Options

swh/loader/git/converters.py

Loading...
View Options

swh/loader/git/tests/test_converters.py

Loading...
About · Developer information · Legal