Page MenuHomeSoftware Heritage
Differential D2405

svn.loader: Migrate to new buffer/filter behavior
ClosedPublic
Actions

Authored by ardumont on Dec 6 2019, 2:51 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Commits
rDLDSVN4d036c1a0a3d: svn.loader: Migrate to new buffer/filter behavior
Summary

This migrates the code to the latest (yet unpublished) loader-core.

Note: build will fail expectedly for now.

Depends D2403

Test Plan

tox

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 9705
Build 14306: tox-on-jenkinsJenkins
Build 14305: arc lint + arc unit

Event Timeline

ardumont created this revision.Dec 6 2019, 2:51 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 6 2019, 2:51 PM
ardumont mentioned this in D2403: core.loader: Drop deprecated behavior.Dec 6 2019, 2:51 PM
swh-public-ci added a comment.Dec 6 2019, 2:52 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/44/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/44/console

Harbormaster failed remote builds in B9576: Diff 8493!Dec 6 2019, 2:52 PM
ardumont edited the summary of this revision. (Show Details)Dec 6 2019, 2:53 PM
anlambert accepted this revision.Dec 6 2019, 3:02 PM
anlambert added a subscriber: anlambert.

Looks good !

swh/loader/svn/tests/test_loader.py
37

You can safely remove that entry too as its use is dropped in D2403

This revision is now accepted and ready to land.Dec 6 2019, 3:02 PM
ardumont added inline comments.Dec 6 2019, 3:05 PM
swh/loader/svn/tests/test_loader.py
37

right! thanks!

ardumont updated this revision to Diff 8554.Dec 10 2019, 9:47 AM

Missing update?

swh-public-ci added a comment.Dec 10 2019, 9:49 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/45/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/45/console

Harbormaster failed remote builds in B9634: Diff 8554!Dec 10 2019, 9:49 AM
ardumont updated this revision to Diff 8613.Dec 12 2019, 12:20 PM

Rebase and plug to master

swh-public-ci added a comment.Dec 12 2019, 12:22 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/46/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/46/console

Harbormaster failed remote builds in B9695: Diff 8613!Dec 12 2019, 12:22 PM
ardumont updated this revision to Diff 8622.Dec 12 2019, 2:09 PM

Update requirements

ardumont updated this revision to Diff 8623.EditedDec 12 2019, 2:10 PM

Update to the latest bug-free version

swh-public-ci added a comment.Dec 12 2019, 2:11 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/47/ for more details.

Harbormaster completed remote builds in B9704: Diff 8622.Dec 12 2019, 2:11 PM
swh-public-ci added a comment.Dec 12 2019, 2:12 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/48/ for more details.

Harbormaster completed remote builds in B9705: Diff 8623.Dec 12 2019, 2:12 PM
Closed by commit rDLDSVN4d036c1a0a3d: svn.loader: Migrate to new buffer/filter behavior (authored by ardumont). · Explain WhyDec 12 2019, 2:13 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLDSVN4d036c1a0a3d: svn.loader: Migrate to new buffer/filter behavior.

Revision Contents
Changeset List

PathSize
2 lines
swh/
loader/
svn/
8 lines
tests/
34 lines
39 lines

Diff 8623

View Options

requirements-swh.txt

# Copyright (C) 2015-2020 The Software Heritage developers # Copyright (C) 2015-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from collections import defaultdict from collections import defaultdict
import datetime import datetime
from datetime import timedelta from datetime import timedelta
import inspect import inspect
import itertools import itertools
import math import math
import random import random
from typing import Any, ClassVar, Dict, Iterator, Optional from typing import Any, ClassVar, Dict, Iterator, Optional
from unittest.mock import MagicMock
import attr import attr
from hypothesis import HealthCheck, given, settings, strategies from hypothesis import HealthCheck, given, settings, strategies
import pytest import pytest
from swh.model import from_disk from swh.model import from_disk
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
from swh.model.hypothesis_strategies import objects from swh.model.hypothesis_strategies import objects
▲ Show 20 Lines • Show All 345 Lines • ▼ Show 20 Lines def test_content_add_metadata_collision(self, swh_storage, sample_data):
content_hex_hashes(cont1.hashes()), content_hex_hashes(cont1.hashes()),
content_hex_hashes(cont1b.hashes()), content_hex_hashes(cont1b.hashes()),
] ]
assert exc.colliding_content_hashes() == [ assert exc.colliding_content_hashes() == [
cont1.hashes(), cont1.hashes(),
cont1b.hashes(), cont1b.hashes(),
] ]
def test_content_add_objstorage_first(self, swh_storage, sample_data):
"""Tests the objstorage is written to before the DB and journal"""
cont = sample_data.content
swh_storage.objstorage.content_add = MagicMock(side_effect=Exception("Oops"))
# Try to add, but the objstorage crashes
try:
swh_storage.content_add([cont])
except Exception:
pass
# The DB must be written to after the objstorage, so the DB should be
# unchanged if the objstorage crashed
assert swh_storage.content_get_data(cont.sha1) is None
# The journal too
assert list(swh_storage.journal_writer.journal.objects) == []
def test_skipped_content_add(self, swh_storage, sample_data): def test_skipped_content_add(self, swh_storage, sample_data):
contents = sample_data.skipped_contents[:2] contents = sample_data.skipped_contents[:2]
cont = contents[0] cont = contents[0]
cont2 = attr.evolve(contents[1], blake2s256=None) cont2 = attr.evolve(contents[1], blake2s256=None)
contents_dict = [c.to_dict() for c in [cont, cont2]] contents_dict = [c.to_dict() for c in [cont, cont2]]
missing = list(swh_storage.skipped_content_missing(contents_dict)) missing = list(swh_storage.skipped_content_missing(contents_dict))
▲ Show 20 Lines • Show All 3,592 Lines • Show Last 20 Lines
View Options

swh/loader/svn/loader.py

# Copyright (C) 2015-2020 The Software Heritage developers # Copyright (C) 2015-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from collections import defaultdict from collections import defaultdict
import datetime import datetime
from datetime import timedelta from datetime import timedelta
import inspect import inspect
import itertools import itertools
import math import math
import random import random
from typing import Any, ClassVar, Dict, Iterator, Optional from typing import Any, ClassVar, Dict, Iterator, Optional
from unittest.mock import MagicMock
import attr import attr
from hypothesis import HealthCheck, given, settings, strategies from hypothesis import HealthCheck, given, settings, strategies
import pytest import pytest
from swh.model import from_disk from swh.model import from_disk
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
from swh.model.hypothesis_strategies import objects from swh.model.hypothesis_strategies import objects
▲ Show 20 Lines • Show All 345 Lines • ▼ Show 20 Lines def test_content_add_metadata_collision(self, swh_storage, sample_data):
content_hex_hashes(cont1.hashes()), content_hex_hashes(cont1.hashes()),
content_hex_hashes(cont1b.hashes()), content_hex_hashes(cont1b.hashes()),
] ]
assert exc.colliding_content_hashes() == [ assert exc.colliding_content_hashes() == [
cont1.hashes(), cont1.hashes(),
cont1b.hashes(), cont1b.hashes(),
] ]
def test_content_add_objstorage_first(self, swh_storage, sample_data):
"""Tests the objstorage is written to before the DB and journal"""
cont = sample_data.content
swh_storage.objstorage.content_add = MagicMock(side_effect=Exception("Oops"))
# Try to add, but the objstorage crashes
try:
swh_storage.content_add([cont])
except Exception:
pass
# The DB must be written to after the objstorage, so the DB should be
# unchanged if the objstorage crashed
assert swh_storage.content_get_data(cont.sha1) is None
# The journal too
assert list(swh_storage.journal_writer.journal.objects) == []
def test_skipped_content_add(self, swh_storage, sample_data): def test_skipped_content_add(self, swh_storage, sample_data):
contents = sample_data.skipped_contents[:2] contents = sample_data.skipped_contents[:2]
cont = contents[0] cont = contents[0]
cont2 = attr.evolve(contents[1], blake2s256=None) cont2 = attr.evolve(contents[1], blake2s256=None)
contents_dict = [c.to_dict() for c in [cont, cont2]] contents_dict = [c.to_dict() for c in [cont, cont2]]
missing = list(swh_storage.skipped_content_missing(contents_dict)) missing = list(swh_storage.skipped_content_missing(contents_dict))
▲ Show 20 Lines • Show All 3,592 Lines • Show Last 20 Lines
View Options

swh/loader/svn/tests/conftest.py

# Copyright (C) 2015-2020 The Software Heritage developers # Copyright (C) 2015-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from collections import defaultdict from collections import defaultdict
import datetime import datetime
from datetime import timedelta from datetime import timedelta
import inspect import inspect
import itertools import itertools
import math import math
import random import random
from typing import Any, ClassVar, Dict, Iterator, Optional from typing import Any, ClassVar, Dict, Iterator, Optional
from unittest.mock import MagicMock
import attr import attr
from hypothesis import HealthCheck, given, settings, strategies from hypothesis import HealthCheck, given, settings, strategies
import pytest import pytest
from swh.model import from_disk from swh.model import from_disk
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
from swh.model.hypothesis_strategies import objects from swh.model.hypothesis_strategies import objects
▲ Show 20 Lines • Show All 345 Lines • ▼ Show 20 Lines def test_content_add_metadata_collision(self, swh_storage, sample_data):
content_hex_hashes(cont1.hashes()), content_hex_hashes(cont1.hashes()),
content_hex_hashes(cont1b.hashes()), content_hex_hashes(cont1b.hashes()),
] ]
assert exc.colliding_content_hashes() == [ assert exc.colliding_content_hashes() == [
cont1.hashes(), cont1.hashes(),
cont1b.hashes(), cont1b.hashes(),
] ]
def test_content_add_objstorage_first(self, swh_storage, sample_data):
"""Tests the objstorage is written to before the DB and journal"""
cont = sample_data.content
swh_storage.objstorage.content_add = MagicMock(side_effect=Exception("Oops"))
# Try to add, but the objstorage crashes
try:
swh_storage.content_add([cont])
except Exception:
pass
# The DB must be written to after the objstorage, so the DB should be
# unchanged if the objstorage crashed
assert swh_storage.content_get_data(cont.sha1) is None
# The journal too
assert list(swh_storage.journal_writer.journal.objects) == []
def test_skipped_content_add(self, swh_storage, sample_data): def test_skipped_content_add(self, swh_storage, sample_data):
contents = sample_data.skipped_contents[:2] contents = sample_data.skipped_contents[:2]
cont = contents[0] cont = contents[0]
cont2 = attr.evolve(contents[1], blake2s256=None) cont2 = attr.evolve(contents[1], blake2s256=None)
contents_dict = [c.to_dict() for c in [cont, cont2]] contents_dict = [c.to_dict() for c in [cont, cont2]]
missing = list(swh_storage.skipped_content_missing(contents_dict)) missing = list(swh_storage.skipped_content_missing(contents_dict))
▲ Show 20 Lines • Show All 3,592 Lines • Show Last 20 Lines
View Options

swh/loader/svn/tests/test_loader.py

# Copyright (C) 2015-2020 The Software Heritage developers # Copyright (C) 2015-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from collections import defaultdict from collections import defaultdict
import datetime import datetime
from datetime import timedelta from datetime import timedelta
import inspect import inspect
import itertools import itertools
import math import math
import random import random
from typing import Any, ClassVar, Dict, Iterator, Optional from typing import Any, ClassVar, Dict, Iterator, Optional
from unittest.mock import MagicMock
import attr import attr
from hypothesis import HealthCheck, given, settings, strategies from hypothesis import HealthCheck, given, settings, strategies
import pytest import pytest
from swh.model import from_disk from swh.model import from_disk
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
from swh.model.hypothesis_strategies import objects from swh.model.hypothesis_strategies import objects
▲ Show 20 Lines • Show All 345 Lines • ▼ Show 20 Lines def test_content_add_metadata_collision(self, swh_storage, sample_data):
content_hex_hashes(cont1.hashes()), content_hex_hashes(cont1.hashes()),
content_hex_hashes(cont1b.hashes()), content_hex_hashes(cont1b.hashes()),
] ]
assert exc.colliding_content_hashes() == [ assert exc.colliding_content_hashes() == [
cont1.hashes(), cont1.hashes(),
cont1b.hashes(), cont1b.hashes(),
] ]
def test_content_add_objstorage_first(self, swh_storage, sample_data):
"""Tests the objstorage is written to before the DB and journal"""
cont = sample_data.content
swh_storage.objstorage.content_add = MagicMock(side_effect=Exception("Oops"))
# Try to add, but the objstorage crashes
try:
swh_storage.content_add([cont])
except Exception:
pass
# The DB must be written to after the objstorage, so the DB should be
# unchanged if the objstorage crashed
assert swh_storage.content_get_data(cont.sha1) is None
# The journal too
assert list(swh_storage.journal_writer.journal.objects) == []
def test_skipped_content_add(self, swh_storage, sample_data): def test_skipped_content_add(self, swh_storage, sample_data):
contents = sample_data.skipped_contents[:2] contents = sample_data.skipped_contents[:2]
cont = contents[0] cont = contents[0]
cont2 = attr.evolve(contents[1], blake2s256=None) cont2 = attr.evolve(contents[1], blake2s256=None)
contents_dict = [c.to_dict() for c in [cont, cont2]] contents_dict = [c.to_dict() for c in [cont, cont2]]
missing = list(swh_storage.skipped_content_missing(contents_dict)) missing = list(swh_storage.skipped_content_missing(contents_dict))
▲ Show 20 Lines • Show All 3,592 Lines • Show Last 20 Lines
About · Developer information · Legal