Page MenuHomeSoftware Heritage
Differential D3454

Adapt the git loader to swh-model >0.4
ClosedPublic
Actions

Authored by douardda on Jul 7 2020, 7:03 PM.

Details

Reviewers
olasd
ardumont
anlambert
Group Reviewers
Reviewers
Commits
rDLDG0394f0f85edf: Adapt the git loader to swh-model >0.4
Summary

in which extra_headers is now a top-level attribute of a Revision.

Add tests that actually include having extra_header in loaded git revisions.
For this, the fast-import based dump of the test git repo is replaced by
a bundle dump (required to ensure things like gpg signatures are kept).
A few git revisions including gpg signature, encoding and mergetag
headers are also added to the git repo.

Diff Detail

Repository
rDLDG Git loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Jul 7 2020, 7:03 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 7 2020, 7:03 PM
swh-public-ci added a comment.Jul 7 2020, 7:04 PM

Build is green

Patch application report for D3454 (id=12229)

Rebasing onto fc2ec733aa...

First, rewinding head to replay your work on top of it...
Applying: Adapt the git loader to swh-model >0.4
Changes applied before test
commit 5ba5d65e90e336adc232838163a6154a32d90fdc
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Jul 7 15:48:40 2020 +0200

    Adapt the git loader to swh-model >0.4
    
    in which extra_headers is now a top-level attribute of a Revision.
    
    Add tests that actually include having extra_header in loaded git revisions.
    For this, the fast-import based dump of the test git repo is replaced by
    a bundle dump (required to ensure things like gpg signatures are kept).
    A few git revisions including gpg signature, encoding and mergetag
    headers are also added to the git repo.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/33/ for more details.

Harbormaster completed remote builds in B13449: Diff 12229.Jul 7 2020, 7:05 PM
olasd accepted this revision.Jul 7 2020, 9:08 PM
olasd added a subscriber: olasd.

Nice!

This revision is now accepted and ready to land.Jul 7 2020, 9:08 PM
olasd added a comment.Jul 7 2020, 9:09 PM

Oh, that needs a requirements-swh.txt bump as well, I guess.

douardda added a comment.Jul 8 2020, 9:19 AM
In D3454#84921, @olasd wrote:

Oh, that needs a requirements-swh.txt bump as well, I guess.

yes but I need to tag swh-storage, since it also requires the new revision api from the storage.

ardumont accepted this revision.Jul 8 2020, 9:48 AM
ardumont added a subscriber: ardumont.

Cool ;)

ardumont added inline comments.Jul 8 2020, 9:53 AM
swh/loader/git/converters.py
116

That might help:
https://github.com/dulwich/dulwich/blob/a8c7a6e3173537798c6a8f77591aac9ced88a135/dulwich/objects.py#L1229-L1252

It seems extra fields are supplementary tuple of freeform information on the commit.
So for the test, take an existing revision from the bundle, amend it with some
those extra fields and check that they get back within the extra headers.
And that would be enough?

douardda updated this revision to Diff 12241.Jul 8 2020, 12:25 PM

bump dependencies

swh-public-ci added a comment.Jul 8 2020, 12:26 PM

Build has FAILED

Patch application report for D3454 (id=12241)

Rebasing onto fc2ec733aa...

First, rewinding head to replay your work on top of it...
Applying: Adapt the git loader to swh-model >0.4
Using index info to reconstruct a base tree...
M	requirements-swh.txt
Falling back to patching base and 3-way merge...
Removing swh/loader/git/tests/data/git-repos/example-submodule.fast-export.xz
Auto-merging requirements-swh.txt
CONFLICT (content): Merge conflict in requirements-swh.txt
Patch failed at 0001 Adapt the git loader to swh-model >0.4

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Rebase failed (ret=1)!




Could not rebase; Attempt merge onto fc2ec733aa...



Already up to date.





Changes applied before test



commit f6a523cd2fcaa5e0d63f57602e35922269dcb054
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Jul 7 15:48:40 2020 +0200

    Adapt the git loader to swh-model >0.4
    
    in which extra_headers is now a top-level attribute of a Revision.
    
    Add tests that actually include having extra_header in loaded git revisions.
    For this, the fast-import based dump of the test git repo is replaced by
    a bundle dump (required to ensure things like gpg signatures are kept).
    A few git revisions including gpg signature, encoding and mergetag
    headers are also added to the git repo.



Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/34/

See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/34/console
Harbormaster failed remote builds in B13460: Diff 12241!Jul 8 2020, 12:26 PM
douardda updated this revision to Diff 12248.Jul 8 2020, 2:00 PM
fix MANIFEST.in to include the git bundle file
swh-public-ci added a comment.Jul 8 2020, 2:02 PM
Build is green



Patch application report for D3454 (id=12248)



Rebasing onto fc2ec733aa...



First, rewinding head to replay your work on top of it...
Applying: Adapt the git loader to swh-model >0.4
Using index info to reconstruct a base tree...
M	MANIFEST.in
M	requirements-swh.txt
Falling back to patching base and 3-way merge...
Removing swh/loader/git/tests/data/git-repos/example-submodule.fast-export.xz
Auto-merging requirements-swh.txt
CONFLICT (content): Merge conflict in requirements-swh.txt
Auto-merging MANIFEST.in
CONFLICT (content): Merge conflict in MANIFEST.in
Patch failed at 0001 Adapt the git loader to swh-model >0.4

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".



Rebase failed (ret=1)!






Could not rebase; Attempt merge onto fc2ec733aa...



Already up to date.





Changes applied before test



commit 338d8bb46cc97c407d2b14cc5be99fe8f5456fea
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Jul 7 15:48:40 2020 +0200

    Adapt the git loader to swh-model >0.4
    
    in which extra_headers is now a top-level attribute of a Revision.
    
    Add tests that actually include having extra_header in loaded git revisions.
    For this, the fast-import based dump of the test git repo is replaced by
    a bundle dump (required to ensure things like gpg signatures are kept).
    A few git revisions including gpg signature, encoding and mergetag
    headers are also added to the git repo.



See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/35/ for more details.
Harbormaster completed remote builds in B13467: Diff 12248.Jul 8 2020, 2:02 PM
douardda updated this revision to Diff 12249.Jul 8 2020, 2:12 PM
rebase
swh-public-ci added a comment.Jul 8 2020, 2:13 PM
Build is green



Patch application report for D3454 (id=12249)



Rebasing onto fc2ec733aa...



Current branch diff-target is up to date.





Changes applied before test



commit 0394f0f85edf6d78548980cd5db0c4436ad7488e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Jul 7 15:48:40 2020 +0200

    Adapt the git loader to swh-model >0.4
    
    in which extra_headers is now a top-level attribute of a Revision.
    
    Add tests that actually include having extra_header in loaded git revisions.
    For this, the fast-import based dump of the test git repo is replaced by
    a bundle dump (required to ensure things like gpg signatures are kept).
    A few git revisions including gpg signature, encoding and mergetag
    headers are also added to the git repo.



See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/36/ for more details.
Harbormaster completed remote builds in B13468: Diff 12249.Jul 8 2020, 2:13 PM
anlambert accepted this revision.Jul 8 2020, 2:53 PM
anlambert added a subscriber: anlambert.
anlambert added inline comments.
swh/loader/git/converters.py


121–123
I think you can remove these lines.


137
extra_headers=tuple(git_metadata),
douardda updated this revision to Diff 12252.Jul 8 2020, 3:11 PM
small simplification as suggested by anlambert
swh-public-ci added a comment.Jul 8 2020, 3:13 PM
Build is green



Patch application report for D3454 (id=12252)



Rebasing onto fc2ec733aa...



Current branch diff-target is up to date.





Changes applied before test



commit 0394f0f85edf6d78548980cd5db0c4436ad7488e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Jul 7 15:48:40 2020 +0200

    Adapt the git loader to swh-model >0.4
    
    in which extra_headers is now a top-level attribute of a Revision.
    
    Add tests that actually include having extra_header in loaded git revisions.
    For this, the fast-import based dump of the test git repo is replaced by
    a bundle dump (required to ensure things like gpg signatures are kept).
    A few git revisions including gpg signature, encoding and mergetag
    headers are also added to the git repo.



See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/37/ for more details.
Harbormaster completed remote builds in B13471: Diff 12252.Jul 8 2020, 3:13 PM
Closed by commit rDLDG0394f0f85edf: Adapt the git loader to swh-model >0.4 (authored by douardda).  ·  Explain WhyJul 8 2020, 3:13 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDLDG0394f0f85edf: Adapt the git loader to swh-model >0.4.
Revision Contents
Changeset List
Diff 12253
View Options
MANIFEST.in
Loading...
View Options
requirements-swh.txt
Loading...
View Options
swh/loader/git/converters.py
Loading...
View Options
swh/loader/git/tests/data/git-repos/example-submodule.bundle
Loading...
View Options
swh/loader/git/tests/data/git-repos/example-submodule.fast-export.xz
Loading...
View Options
swh/loader/git/tests/test_converters.py
Loading...
About · Developer information · Legal