Changeset View
Changeset View
Standalone View
Standalone View
swh/loader/core/loader.py
Show First 20 Lines • Show All 330 Lines • ▼ Show 20 Lines | def load(self) -> Dict[str, str]: | ||||
- Call :meth:`process_data` to optionally run processing between | - Call :meth:`process_data` to optionally run processing between | ||||
:meth:`fetch_data` and :meth:`store_data` | :meth:`fetch_data` and :meth:`store_data` | ||||
- Call :meth:`store_data` to store the data | - Call :meth:`store_data` to store the data | ||||
- Call :meth:`cleanup` to clean up any eventual state put in place | - Call :meth:`cleanup` to clean up any eventual state put in place | ||||
in :meth:`prepare` method. | in :meth:`prepare` method. | ||||
""" | """ | ||||
sentry_sdk.set_tag("swh.loader.origin_url", self.origin.url[0:200]) | |||||
sentry_sdk.set_tag("swh.loader.visit_type", self.visit_type) | |||||
try: | try: | ||||
with self.statsd_timed("pre_cleanup"): | with self.statsd_timed("pre_cleanup"): | ||||
self.pre_cleanup() | self.pre_cleanup() | ||||
except Exception: | except Exception: | ||||
msg = "Cleaning up dangling data failed! Continue loading." | msg = "Cleaning up dangling data failed! Continue loading." | ||||
self.log.warning(msg) | self.log.warning(msg) | ||||
sentry_sdk.capture_exception() | sentry_sdk.capture_exception() | ||||
▲ Show 20 Lines • Show All 290 Lines • Show Last 20 Lines |