Page MenuHomeSoftware Heritage
Differential D5040

loader.git: Explicit the failure test cases
ClosedPublic
Actions

Authored by ardumont on Feb 8 2021, 5:50 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3030: Improve loaders to deal with new visit status events
Commits
rDLDG11074801b0bb: loader.git: Explicit the failure test cases
Summary

With the new loader.core 0.17, failed or partial status changed slightly.
This adds the necessary tests to explicit those.

Related to T3030

Test Plan

tox
(failing until swh.loader.core > 0.16 release)

Diff Detail

Repository
rDLDG Git loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Feb 8 2021, 5:50 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 8 2021, 5:50 PM
swh-public-ci added a comment.Feb 8 2021, 5:51 PM

Build has FAILED

Patch application report for D5040 (id=17980)

Rebasing onto 220678c753...

Current branch diff-target is up to date.
Changes applied before test
commit aa71a198881188d7e42ea82d9896fcb394e6d1d4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 8 17:36:57 2021 +0100

    svn.loader: Mark visit status as failed
    
    Related to T3030

Link to build: https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/101/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/101/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 8 2021, 5:51 PM
Harbormaster failed remote builds in B19101: Diff 17980!
swh-public-ci added a comment.Feb 11 2021, 11:57 AM

Build is green

Patch application report for D5040 (id=17980)

Rebasing onto 220678c753...

Current branch diff-target is up to date.
Changes applied before test
commit aa71a198881188d7e42ea82d9896fcb394e6d1d4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 8 17:36:57 2021 +0100

    svn.loader: Mark visit status as failed
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/103/ for more details.

Harbormaster completed remote builds in B19101: Diff 17980.Feb 11 2021, 11:57 AM
ardumont requested review of this revision.Feb 11 2021, 11:57 AM
ardumont updated this revision to Diff 18053.Feb 11 2021, 12:03 PM

Rework commit message

swh-public-ci added a comment.Feb 11 2021, 12:05 PM

Build is green

Patch application report for D5040 (id=18053)

Rebasing onto 220678c753...

Current branch diff-target is up to date.
Changes applied before test
commit ddc18d08514725fe7fb59857c8181e2cdaa47e3e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 8 17:36:57 2021 +0100

    svn.loader: Mark visit status as failed
    
    With the latest swh.loader.core 0.17, a partial without without snapshot is failed.
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/105/ for more details.

Harbormaster completed remote builds in B19165: Diff 18053.Feb 11 2021, 12:05 PM
ardumont edited the summary of this revision. (Show Details)Feb 11 2021, 12:08 PM
ardumont updated this revision to Diff 18061.Feb 11 2021, 2:16 PM

Make that a git diff

ardumont retitled this revision from svn.loader: Mark visit status as failed to loader.git: Explicit the failure test cases.Feb 11 2021, 2:17 PM
ardumont edited the summary of this revision. (Show Details)
swh-public-ci added a comment.Feb 11 2021, 2:17 PM

Build is green

Patch application report for D5040 (id=18061)

Rebasing onto 824880e7d7...

Current branch diff-target is up to date.
Changes applied before test
commit 11074801b0bb928c62442d4dd74d514cbbdbd0bf
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 8 19:07:04 2021 +0100

    loader.git: Explicit the failure test cases
    
    With the new loader.core 0.17, failed or partial status changed slightly.
    This adds the necessary tests to explicit those.
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/77/ for more details.

Harbormaster completed remote builds in B19173: Diff 18061.Feb 11 2021, 2:17 PM
vlorentz accepted this revision.Feb 11 2021, 2:23 PM
This revision is now accepted and ready to land.Feb 11 2021, 2:23 PM
Closed by commit rDLDG11074801b0bb: loader.git: Explicit the failure test cases (authored by ardumont). · Explain WhyFeb 11 2021, 2:32 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLDG11074801b0bb: loader.git: Explicit the failure test cases.

Revision Contents
Changeset List

PathSize
2 lines
swh/
loader/
git/
tests/
41 lines

Diff 18063

View Options

requirements-swh.txt

swh.core >= 0.0.7 swh.core >= 0.0.7
swh.loader.core >= 0.16.0 swh.loader.core >= 0.17.0
swh.model >= 0.4.0 swh.model >= 0.4.0
swh.scheduler >= 0.0.39 swh.scheduler >= 0.0.39
swh.storage >= 0.22.0 swh.storage >= 0.22.0
View Options

swh/loader/git/tests/test_from_disk.py

# Copyright (C) 2018-2020 The Software Heritage developers # Copyright (C) 2018-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import copy import copy
import datetime import datetime
import os.path import os.path
from unittest import TestCase from unittest import TestCase
▲ Show 20 Lines • Show All 151 Lines • ▼ Show 20 Lines def test_load_unchanged(self):
assert_last_visit_matches( assert_last_visit_matches(
self.loader.storage, self.loader.storage,
self.repo_url, self.repo_url,
status="full", status="full",
type="git", type="git",
snapshot=SNAPSHOT1.id, snapshot=SNAPSHOT1.id,
) )
def test_load_visit_without_snapshot_so_status_failed(self):
# unfortunately, monkey-patch the hard way, self.loader is already instantiated
# (patching won't work self.loader is already instantiated)
# Make get_contents fail for some reason
self.loader.get_contents = None
res = self.loader.load()
assert res == {"status": "failed"}
assert_last_visit_matches(
self.loader.storage,
self.repo_url,
status="failed",
type="git",
snapshot=None,
)
def test_load_visit_with_snapshot_so_status_partial(self):
# unfortunately, monkey-patch the hard way, self.loader is already instantiated
# (patching won't work self.loader is already instantiated)
# fake store_metadata raising for some reason, so we could have a snapshot id
# at this point in time
self.loader.store_metadata = None
# fake having a snapshot so the visit status is partial
self.loader.loaded_snapshot_id = hash_to_bytes(
"a23699280a82a043f8c0994cf1631b568f716f95"
)
res = self.loader.load()
assert res == {"status": "failed"}
assert_last_visit_matches(
self.loader.storage,
self.repo_url,
status="partial",
type="git",
snapshot=None,
)
class FullGitLoaderTests(CommonGitLoaderTests): class FullGitLoaderTests(CommonGitLoaderTests):
"""Tests for GitLoader (from disk or not). Includes the common ones, and """Tests for GitLoader (from disk or not). Includes the common ones, and
add others that only work with a local dir. add others that only work with a local dir.
""" """
def test_load_changed(self): def test_load_changed(self):
▲ Show 20 Lines • Show All 228 Lines • Show Last 20 Lines
About · Developer information · Legal