Page MenuHomeSoftware Heritage
Differential D4371

WIP: archive + meta directory sharding
AbandonedPublic
Actions

Authored by haltode on Oct 28 2020, 1:42 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Maniphest Tasks
T2694: FUSE: add sharding support for top-level dirs
Summary

Tested by hand.

Closes T2694.

TODO:

  • Update unit tests to use default sharding parameters
  • Update documentation/docstring
  • Code refactoring(?)/cleaning

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/archive-meta-sharding
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16697
Build 25746: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 25745: arc lint + arc unit

Event Timeline

haltode created this revision.Oct 28 2020, 1:42 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 28 2020, 1:42 PM
swh-public-ci added a comment.Oct 28 2020, 1:43 PM

Build is green

Patch application report for D4371 (id=15472)

Rebasing onto f9de49aca7...

Current branch diff-target is up to date.
Changes applied before test
commit d04da6529e4990e08ef830459fa542fe69756d5b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 28 13:30:25 2020 +0100

    WIP: sharding

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/130/ for more details.

Harbormaster completed remote builds in B16697: Diff 15472.Oct 28 2020, 1:43 PM
haltode updated this revision to Diff 15473.Oct 28 2020, 1:46 PM

Update commit message

swh-public-ci added a comment.Oct 28 2020, 1:47 PM

Build is green

Patch application report for D4371 (id=15473)

Rebasing onto f9de49aca7...

Current branch diff-target is up to date.
Changes applied before test
commit 2c8729dff1266c161af3d5e1c1cf4a730afc5927
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 28 13:30:25 2020 +0100

    WIP: sharding
    
    Closes T2694.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/131/ for more details.

Harbormaster completed remote builds in B16698: Diff 15473.Oct 28 2020, 1:47 PM
haltode edited the summary of this revision. (Show Details)Oct 28 2020, 2:20 PM
haltode edited the summary of this revision. (Show Details)Oct 29 2020, 10:05 AM
haltode added a task: T2694: FUSE: add sharding support for top-level dirs.
haltode abandoned this revision.Nov 10 2020, 12:44 PM

Moved on to only sharding history/ directory instead of archive/ and meta/ (see D4416).

zack mentioned this in T2771: FUSE: rethink the visibility of files under archive/ and meta/, and possibly add a new cache/ entrypoint.Nov 11 2020, 3:11 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
fuse/
1 line61%
Loading...
fs/
21 lines93%
Loading...
68 lines74%
Loading...
63 lines87%
Loading...
1 line82%
Loading...
tests/
1 line92%
Loading...

Diff 15472

View Options

swh/fuse/cli.py

Show First 20 Lines • Show All 330 Lines • ▼ Show 20 Lines def load(self) -> Dict[str, str]:
- Call :meth:`process_data` to optionally run processing between - Call :meth:`process_data` to optionally run processing between
:meth:`fetch_data` and :meth:`store_data` :meth:`fetch_data` and :meth:`store_data`
- Call :meth:`store_data` to store the data - Call :meth:`store_data` to store the data
- Call :meth:`cleanup` to clean up any eventual state put in place - Call :meth:`cleanup` to clean up any eventual state put in place
in :meth:`prepare` method. in :meth:`prepare` method.
""" """
sentry_sdk.set_tag("swh.loader.origin_url", self.origin.url[0:200])
sentry_sdk.set_tag("swh.loader.visit_type", self.visit_type)
try: try:
with self.statsd_timed("pre_cleanup"): with self.statsd_timed("pre_cleanup"):
self.pre_cleanup() self.pre_cleanup()
except Exception: except Exception:
msg = "Cleaning up dangling data failed! Continue loading." msg = "Cleaning up dangling data failed! Continue loading."
self.log.warning(msg) self.log.warning(msg)
sentry_sdk.capture_exception() sentry_sdk.capture_exception()
▲ Show 20 Lines • Show All 290 Lines • Show Last 20 Lines
View Options

swh/fuse/fs/artifact.py

Show First 20 Lines • Show All 330 Lines • ▼ Show 20 Lines def load(self) -> Dict[str, str]:
- Call :meth:`process_data` to optionally run processing between - Call :meth:`process_data` to optionally run processing between
:meth:`fetch_data` and :meth:`store_data` :meth:`fetch_data` and :meth:`store_data`
- Call :meth:`store_data` to store the data - Call :meth:`store_data` to store the data
- Call :meth:`cleanup` to clean up any eventual state put in place - Call :meth:`cleanup` to clean up any eventual state put in place
in :meth:`prepare` method. in :meth:`prepare` method.
""" """
sentry_sdk.set_tag("swh.loader.origin_url", self.origin.url[0:200])
sentry_sdk.set_tag("swh.loader.visit_type", self.visit_type)
try: try:
with self.statsd_timed("pre_cleanup"): with self.statsd_timed("pre_cleanup"):
self.pre_cleanup() self.pre_cleanup()
except Exception: except Exception:
msg = "Cleaning up dangling data failed! Continue loading." msg = "Cleaning up dangling data failed! Continue loading."
self.log.warning(msg) self.log.warning(msg)
sentry_sdk.capture_exception() sentry_sdk.capture_exception()
▲ Show 20 Lines • Show All 290 Lines • Show Last 20 Lines
View Options

swh/fuse/fs/entry.py

Show First 20 Lines • Show All 330 Lines • ▼ Show 20 Lines def load(self) -> Dict[str, str]:
- Call :meth:`process_data` to optionally run processing between - Call :meth:`process_data` to optionally run processing between
:meth:`fetch_data` and :meth:`store_data` :meth:`fetch_data` and :meth:`store_data`
- Call :meth:`store_data` to store the data - Call :meth:`store_data` to store the data
- Call :meth:`cleanup` to clean up any eventual state put in place - Call :meth:`cleanup` to clean up any eventual state put in place
in :meth:`prepare` method. in :meth:`prepare` method.
""" """
sentry_sdk.set_tag("swh.loader.origin_url", self.origin.url[0:200])
sentry_sdk.set_tag("swh.loader.visit_type", self.visit_type)
try: try:
with self.statsd_timed("pre_cleanup"): with self.statsd_timed("pre_cleanup"):
self.pre_cleanup() self.pre_cleanup()
except Exception: except Exception:
msg = "Cleaning up dangling data failed! Continue loading." msg = "Cleaning up dangling data failed! Continue loading."
self.log.warning(msg) self.log.warning(msg)
sentry_sdk.capture_exception() sentry_sdk.capture_exception()
▲ Show 20 Lines • Show All 290 Lines • Show Last 20 Lines
View Options

swh/fuse/fs/mountpoint.py

Show First 20 Lines • Show All 330 Lines • ▼ Show 20 Lines def load(self) -> Dict[str, str]:
- Call :meth:`process_data` to optionally run processing between - Call :meth:`process_data` to optionally run processing between
:meth:`fetch_data` and :meth:`store_data` :meth:`fetch_data` and :meth:`store_data`
- Call :meth:`store_data` to store the data - Call :meth:`store_data` to store the data
- Call :meth:`cleanup` to clean up any eventual state put in place - Call :meth:`cleanup` to clean up any eventual state put in place
in :meth:`prepare` method. in :meth:`prepare` method.
""" """
sentry_sdk.set_tag("swh.loader.origin_url", self.origin.url[0:200])
sentry_sdk.set_tag("swh.loader.visit_type", self.visit_type)
try: try:
with self.statsd_timed("pre_cleanup"): with self.statsd_timed("pre_cleanup"):
self.pre_cleanup() self.pre_cleanup()
except Exception: except Exception:
msg = "Cleaning up dangling data failed! Continue loading." msg = "Cleaning up dangling data failed! Continue loading."
self.log.warning(msg) self.log.warning(msg)
sentry_sdk.capture_exception() sentry_sdk.capture_exception()
▲ Show 20 Lines • Show All 290 Lines • Show Last 20 Lines
View Options

swh/fuse/fuse.py

Show First 20 Lines • Show All 330 Lines • ▼ Show 20 Lines def load(self) -> Dict[str, str]:
- Call :meth:`process_data` to optionally run processing between - Call :meth:`process_data` to optionally run processing between
:meth:`fetch_data` and :meth:`store_data` :meth:`fetch_data` and :meth:`store_data`
- Call :meth:`store_data` to store the data - Call :meth:`store_data` to store the data
- Call :meth:`cleanup` to clean up any eventual state put in place - Call :meth:`cleanup` to clean up any eventual state put in place
in :meth:`prepare` method. in :meth:`prepare` method.
""" """
sentry_sdk.set_tag("swh.loader.origin_url", self.origin.url[0:200])
sentry_sdk.set_tag("swh.loader.visit_type", self.visit_type)
try: try:
with self.statsd_timed("pre_cleanup"): with self.statsd_timed("pre_cleanup"):
self.pre_cleanup() self.pre_cleanup()
except Exception: except Exception:
msg = "Cleaning up dangling data failed! Continue loading." msg = "Cleaning up dangling data failed! Continue loading."
self.log.warning(msg) self.log.warning(msg)
sentry_sdk.capture_exception() sentry_sdk.capture_exception()
▲ Show 20 Lines • Show All 290 Lines • Show Last 20 Lines
View Options

swh/fuse/tests/conftest.py

Show First 20 Lines • Show All 330 Lines • ▼ Show 20 Lines def load(self) -> Dict[str, str]:
- Call :meth:`process_data` to optionally run processing between - Call :meth:`process_data` to optionally run processing between
:meth:`fetch_data` and :meth:`store_data` :meth:`fetch_data` and :meth:`store_data`
- Call :meth:`store_data` to store the data - Call :meth:`store_data` to store the data
- Call :meth:`cleanup` to clean up any eventual state put in place - Call :meth:`cleanup` to clean up any eventual state put in place
in :meth:`prepare` method. in :meth:`prepare` method.
""" """
sentry_sdk.set_tag("swh.loader.origin_url", self.origin.url[0:200])
sentry_sdk.set_tag("swh.loader.visit_type", self.visit_type)
try: try:
with self.statsd_timed("pre_cleanup"): with self.statsd_timed("pre_cleanup"):
self.pre_cleanup() self.pre_cleanup()
except Exception: except Exception:
msg = "Cleaning up dangling data failed! Continue loading." msg = "Cleaning up dangling data failed! Continue loading."
self.log.warning(msg) self.log.warning(msg)
sentry_sdk.capture_exception() sentry_sdk.capture_exception()
▲ Show 20 Lines • Show All 290 Lines • Show Last 20 Lines
About · Developer information · Legal