Page MenuHomeSoftware Heritage

Add support for (topological) branches and merges in generate_repo.py
ClosedPublic

Authored by douardda on Jun 17 2021, 8:16 PM.

Details

Summary

include an example input file (currently not used in actual tests).

Depends on D5891.

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build has FAILED

Patch application report for D5892 (id=21102)

Could not rebase; Attempt merge onto c9d1369ba1...

Updating c9d1369..413c2a4
Fast-forward
 mypy.ini                                           |   3 +
 requirements-test.txt                              |   1 +
 swh/provenance/tests/conftest.py                   |  38 ++++-----
 swh/provenance/tests/data/cmdbts2.msgpack          | Bin 17734 -> 14470 bytes
 swh/provenance/tests/data/generate_repo.py         |  16 +++-
 .../tests/data/generate_storage_from_git.py        |  86 +++------------------
 swh/provenance/tests/data/out-of-order.msgpack     | Bin 6653 -> 6325 bytes
 swh/provenance/tests/data/repo_with_merges.yaml    |  73 +++++++++++++++++
 8 files changed, 117 insertions(+), 100 deletions(-)
 create mode 100644 swh/provenance/tests/data/repo_with_merges.yaml
Changes applied before test
commit 413c2a48eff28dc2236cbb96b1cc701c62b99897
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jun 17 14:52:54 2021 +0200

    Add support for (topological) branches and merges in generate_repo.py
    
    include an example input file (currently not used in actual tests).

commit e1a003a28f2ce1319f65af1bc54dac6003cc4910
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jun 17 19:53:06 2021 +0200

    Refactor the generate_storage_from_git dataset creation tool
    
    use the newly introduced swh.journal StreamJournalWriter to generate the
    msgpack-file with the content of the storage serialized, instead of
    reimplementing a similar logic by hand.
    
    Doing so, the msgpack file format is a little bit different, so current
    datasets need to be regenerated.
    
    From the 'consumer' side (the fill_storage fixture), simply use
    swh.storage's process_replay_objects (once again, instead of reinventing
    the wheel).
    
    This is needed because we will need to handle much more model object types
    for testing the origin-revision layer.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/156/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/156/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 17 2021, 8:17 PM
Harbormaster failed remote builds in B22090: Diff 21102!

Build is green

Patch application report for D5892 (id=21102)

Could not rebase; Attempt merge onto 8ff1ab5860...

Merge made by the 'recursive' strategy.
 mypy.ini                                           |   3 +
 requirements-test.txt                              |   1 +
 swh/provenance/tests/conftest.py                   |  38 ++++-----
 swh/provenance/tests/data/cmdbts2.msgpack          | Bin 17734 -> 14470 bytes
 swh/provenance/tests/data/generate_repo.py         |  16 +++-
 .../tests/data/generate_storage_from_git.py        |  86 +++------------------
 swh/provenance/tests/data/out-of-order.msgpack     | Bin 6653 -> 6325 bytes
 swh/provenance/tests/data/repo_with_merges.yaml    |  73 +++++++++++++++++
 8 files changed, 117 insertions(+), 100 deletions(-)
 create mode 100644 swh/provenance/tests/data/repo_with_merges.yaml
Changes applied before test
commit 4f26e661d3f66dd68025184875c8b628fd650402
Merge: 8ff1ab5 413c2a4
Author: Jenkins user <jenkins@localhost>
Date:   Fri Jun 18 10:32:24 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 413c2a48eff28dc2236cbb96b1cc701c62b99897
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jun 17 14:52:54 2021 +0200

    Add support for (topological) branches and merges in generate_repo.py
    
    include an example input file (currently not used in actual tests).

commit e1a003a28f2ce1319f65af1bc54dac6003cc4910
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jun 17 19:53:06 2021 +0200

    Refactor the generate_storage_from_git dataset creation tool
    
    use the newly introduced swh.journal StreamJournalWriter to generate the
    msgpack-file with the content of the storage serialized, instead of
    reimplementing a similar logic by hand.
    
    Doing so, the msgpack file format is a little bit different, so current
    datasets need to be regenerated.
    
    From the 'consumer' side (the fill_storage fixture), simply use
    swh.storage's process_replay_objects (once again, instead of reinventing
    the wheel).
    
    This is needed because we will need to handle much more model object types
    for testing the origin-revision layer.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/159/ for more details.

This revision is now accepted and ready to land.Jun 21 2021, 3:39 PM
This revision was landed with ongoing or failed builds.Jun 21 2021, 4:48 PM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D5892 (id=21161)

Could not rebase; Attempt merge onto 8ff1ab5860...

Updating 8ff1ab5..6734fd3
Fast-forward
 mypy.ini                                           |   3 +
 requirements-test.txt                              |   1 +
 swh/provenance/tests/conftest.py                   |  38 ++++-----
 swh/provenance/tests/data/cmdbts2.msgpack          | Bin 17734 -> 14470 bytes
 swh/provenance/tests/data/generate_repo.py         |  16 +++-
 .../tests/data/generate_storage_from_git.py        |  86 +++------------------
 swh/provenance/tests/data/out-of-order.msgpack     | Bin 6653 -> 6325 bytes
 swh/provenance/tests/data/repo_with_merges.yaml    |  73 +++++++++++++++++
 8 files changed, 117 insertions(+), 100 deletions(-)
 create mode 100644 swh/provenance/tests/data/repo_with_merges.yaml
Changes applied before test
commit 6734fd36b87233aa7520c90e7b6ad999d75103cf
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jun 17 14:52:54 2021 +0200

    Add support for (topological) branches and merges in generate_repo.py
    
    include an example input file (currently not used in actual tests).

commit 7886bf494ab8dd0405ad391a0ca57eed1ee7149c
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jun 17 19:53:06 2021 +0200

    Refactor the generate_storage_from_git dataset creation tool
    
    use the newly introduced swh.journal StreamJournalWriter to generate the
    msgpack-file with the content of the storage serialized, instead of
    reimplementing a similar logic by hand.
    
    Doing so, the msgpack file format is a little bit different, so current
    datasets need to be regenerated.
    
    From the 'consumer' side (the fill_storage fixture), simply use
    swh.storage's process_replay_objects (once again, instead of reinventing
    the wheel).
    
    This is needed because we will need to handle much more model object types
    for testing the origin-revision layer.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/174/ for more details.