Page MenuHomeSoftware Heritage
Differential D4311

Add testing repository builder
ClosedPublic
Actions

Authored by acezar on Oct 19 2020, 10:30 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDLDHGff11f77f1b49: Add testing repository builder
Summary

This build script purpose is to create example repositories from bash scripts
and extract assertion data from them into json files.

Advantages:

  • the bash script documents the repository creation
  • automating creation allow easy repository update
  • automation extraction allow easier update of assertion data

Diff Detail

Repository
rDLDHG Mercurial loader
Branch
D4311
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16657
Build 25682: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 25681: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
182 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_identify::test_all
datadir = '/var/lib/jenkins/workspace/DLDHG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/mercurial/tests/data' tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_all0')
167 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_identify::test_all_revisions
datadir = '/var/lib/jenkins/workspace/DLDHG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/mercurial/tests/data' tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_all_revisions0')
164 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_identify::test_single_revision
datadir = '/var/lib/jenkins/workspace/DLDHG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/mercurial/tests/data' tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_single_revision0')
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_converters.TestParseAuthorConverters::test_parse_author_2
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_converters.TestParseAuthorConverters::test_parse_author_3
View Full Test Results (3 Failed · 17 Passed)

Event Timeline

acezar created this revision.Oct 19 2020, 10:30 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 19 2020, 10:30 PM
swh-public-ci added a comment.Oct 19 2020, 10:31 PM

Build has FAILED

Patch application report for D4311 (id=15239)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..07f1b37
Fast-forward
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 494 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 187 +++++++++++
 swh/loader/mercurial/tests/test_identify.py |  88 +++++
 4 files changed, 771 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit 07f1b3730f6659133a2431c45d6b01696fe9da45
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    Depends on D4216

commit b543125f433ab3022cc80c5b76e3a2eab0162b61
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/62/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/62/console

Harbormaster failed remote builds in B16472: Diff 15239!Oct 19 2020, 10:31 PM
acezar added a child revision: D4312: Add `Hg20BundleLoader` tests from json files.Oct 19 2020, 10:47 PM
douardda added a subscriber: douardda.Oct 23 2020, 11:21 AM

please do not put the "depends on Dxxx" line in the git commit message.

douardda added a comment.Oct 23 2020, 11:25 AM

Also the commit message should give a bit more information on what this new script is needed for, maybe with a usage example.
This defines a bunch of commands. When and how should "I" use them?

swh/loader/mercurial/tests/data/build.py
2

why 2018? Is this copied from elsewhere in the swh space?

douardda requested changes to this revision.Oct 23 2020, 11:25 AM
This revision now requires changes to proceed.Oct 23 2020, 11:25 AM
acezar updated this revision to Diff 15402.Oct 26 2020, 4:46 PM
acezar edited the summary of this revision. (Show Details)

Add and fix documentation

swh-public-ci added a comment.Oct 26 2020, 4:47 PM

Build has FAILED

Patch application report for D4311 (id=15402)

Rebasing onto bd914dec39...

Current branch diff-target is up to date.
Changes applied before test
commit 838149f98d435fce7d809fbe1c324c4c42dc2ba7
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/67/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/67/console

Harbormaster failed remote builds in B16631: Diff 15402!Oct 26 2020, 4:47 PM
acezar updated this revision to Diff 15416.Oct 27 2020, 11:05 AM

Fighting arc

swh-public-ci added a comment.Oct 27 2020, 11:06 AM

Build has FAILED

Patch application report for D4311 (id=15416)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..5b02566
Fast-forward
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 532 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 187 ++++++++++
 swh/loader/mercurial/tests/test_identify.py |  88 +++++
 4 files changed, 809 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit 5b0256683349ad892ffed749e7dc45d8b768e5fb
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/72/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/72/console

Harbormaster failed remote builds in B16643: Diff 15416!Oct 27 2020, 11:06 AM
acezar updated this revision to Diff 15427.Oct 27 2020, 2:52 PM

Get back the right commit

swh-public-ci added a comment.Oct 27 2020, 2:53 PM

Build has FAILED

Patch application report for D4311 (id=15427)

Rebasing onto bd914dec39...

Current branch diff-target is up to date.
Changes applied before test
commit 838149f98d435fce7d809fbe1c324c4c42dc2ba7
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/76/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/76/console

Harbormaster failed remote builds in B16656: Diff 15427!Oct 27 2020, 2:53 PM
acezar updated this revision to Diff 15428.Oct 27 2020, 2:53 PM

arc fail again...

swh-public-ci added a comment.Oct 27 2020, 2:54 PM

Build has FAILED

Patch application report for D4311 (id=15428)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..838149f
Fast-forward
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 532 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 265 ++++++++++++++
 swh/loader/mercurial/tests/test_identify.py |  88 +++++
 4 files changed, 887 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit 838149f98d435fce7d809fbe1c324c4c42dc2ba7
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/77/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/77/console

Harbormaster failed remote builds in B16657: Diff 15428!Oct 27 2020, 2:55 PM
acezar added inline comments.Oct 27 2020, 4:01 PM
swh/loader/mercurial/tests/data/build.py
2

copied from test_loader.py

acezar added inline comments.Oct 27 2020, 4:10 PM
swh/loader/mercurial/tests/data/build.py
2

Some extraction:

  • swh/loader/mercurial/archive_extract.py: # Copyright (C) 2017-2018
  • swh/loader/mercurial/bundle20_reader.py: # Copyright (C) 2017
  • swh/loader/mercurial/chunked_reader.py: # Copyright (C) 2017
  • swh/loader/mercurial/cli.py: # Copyright (C) 2018
  • swh/loader/mercurial/converters.py: # Copyright (C) 2015-2017
  • swh/loader/mercurial/identify.py: # Copyright (C) 2020
  • swh/loader/mercurial/__init__.py: # Copyright (C) 2019
  • swh/loader/mercurial/loader.py: # Copyright (C) 2017-2020
  • swh/loader/mercurial/objects.py: # Copyright (C) 2017-2018
  • swh/loader/mercurial/tasks.py: # Copyright (C) 2017-2019
  • swh/loader/mercurial/tests/conftest.py: # Copyright (C) 2019-2020
  • swh/loader/mercurial/tests/test_converters.py: # Copyright (C) 2018
  • swh/loader/mercurial/tests/test_loader.py: # Copyright (C) 2018-2020
  • swh/loader/mercurial/tests/test_tasks.py: # Copyright (C) 2018-2020
acezar updated this revision to Diff 15448.Oct 27 2020, 4:47 PM

Update

swh-public-ci added a comment.Oct 27 2020, 4:49 PM

Build has FAILED

Patch application report for D4311 (id=15448)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..838149f
Fast-forward
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 532 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 265 ++++++++++++++
 swh/loader/mercurial/tests/test_identify.py |  88 +++++
 4 files changed, 887 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit 838149f98d435fce7d809fbe1c324c4c42dc2ba7
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/80/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/80/console

Harbormaster failed remote builds in B16676: Diff 15448!Oct 27 2020, 4:49 PM
acezar updated this revision to Diff 15452.Oct 27 2020, 4:49 PM

Update

swh-public-ci added a comment.Oct 27 2020, 4:51 PM

Build has FAILED

Patch application report for D4311 (id=15452)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..b4e053f
Fast-forward
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 532 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 265 ++++++++++++++
 swh/loader/mercurial/tests/test_identify.py |  93 +++++
 4 files changed, 892 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit b4e053fdcdd4cd7cdaef915de79c89829186b067
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit e75b99287ba58bdbc6cb95703f11303c561a8f34
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/82/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/82/console

Harbormaster failed remote builds in B16680: Diff 15452!Oct 27 2020, 4:51 PM
douardda added inline comments.Oct 28 2020, 10:09 AM
swh/loader/mercurial/tests/data/build.py
2

This very file is new, thus the copyright starts in 2020. Did I miss something?

acezar marked 2 inline comments as done.Oct 29 2020, 10:11 AM
acezar added inline comments.
swh/loader/mercurial/tests/data/build.py
2

Was my understanding of the issue which was wrong.

swh-public-ci added a comment.Oct 30 2020, 5:11 PM

Build has FAILED

Patch application report for D4311 (id=15452)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..b4e053f
Fast-forward
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 532 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 265 ++++++++++++++
 swh/loader/mercurial/tests/test_identify.py |  93 +++++
 4 files changed, 892 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit b4e053fdcdd4cd7cdaef915de79c89829186b067
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit e75b99287ba58bdbc6cb95703f11303c561a8f34
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/87/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/87/console

douardda accepted this revision.Nov 4 2020, 9:25 AM

I'm fine with this, can be landed as soon as tests are ok.

This revision is now accepted and ready to land.Nov 4 2020, 9:25 AM
acezar updated this revision to Diff 15777.Nov 10 2020, 2:39 PM
acezar marked an inline comment as done.

Make sure the diff is up to date.

Maybe I messed something with arc again, since the tests works for me and
another dev on 2 different setups: one with docker the other without docker.

swh-public-ci added a comment.Nov 10 2020, 2:41 PM

Build is green

Patch application report for D4311 (id=15777)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..ea7cc14
Fast-forward
 requirements.txt                            |   1 +
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 533 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 265 ++++++++++++++
 swh/loader/mercurial/tests/test_identify.py |  93 +++++
 5 files changed, 894 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit ea7cc149edbee8778121fc37a099edaedc584f60
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit 00b709bae8c01854fa9eb37e2b5d9989fa68a17d
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/88/ for more details.

Harbormaster completed remote builds in B16988: Diff 15777.Nov 10 2020, 2:41 PM
acezar updated this revision to Diff 15784.Nov 10 2020, 2:52 PM

Update

Seems that I messed my arc diff --update ci should passe with this.

swh-public-ci added a comment.Nov 10 2020, 2:53 PM

Build is green

Patch application report for D4311 (id=15784)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..ea7cc14
Fast-forward
 requirements.txt                            |   1 +
 setup.py                                    |   2 +
 swh/loader/mercurial/identify.py            | 533 ++++++++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py    | 265 ++++++++++++++
 swh/loader/mercurial/tests/test_identify.py |  93 +++++
 5 files changed, 894 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit ea7cc149edbee8778121fc37a099edaedc584f60
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit 00b709bae8c01854fa9eb37e2b5d9989fa68a17d
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/89/ for more details.

Harbormaster completed remote builds in B16995: Diff 15784.Nov 10 2020, 2:53 PM
acezar updated this revision to Diff 16416.Nov 27 2020, 5:26 PM

Cannot push the merge because commit message as be changed
(Probably to remove arcanist data)

swh-public-ci added a comment.Nov 27 2020, 5:27 PM

Build is green

Patch application report for D4311 (id=16416)

Rebasing onto a2e9cf1691...

Current branch diff-target is up to date.
Changes applied before test
commit ff11f77f1b493bd1c8ed257e790ded8da276101c
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/125/ for more details.

Harbormaster completed remote builds in B17605: Diff 16416.Nov 27 2020, 5:27 PM
Closed by commit rDLDHGff11f77f1b49: Add testing repository builder (authored by acezar). · Explain WhyNov 27 2020, 5:27 PM
This revision was automatically updated to reflect the committed changes.
acezar added a commit: rDLDHGff11f77f1b49: Add testing repository builder.

Revision Contents
Changeset List

PathSize
swh/
loader/
mercurial/
tests/
data/
265 lines

Diff 15428

View Options

swh/loader/mercurial/tests/data/build.py

Loading...
About · Developer information · Legal