Page MenuHomeSoftware Heritage

Core LoaderFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

loader plumbings used by other loaders

Recent Activity

Fri, Nov 20

ardumont closed T2537: Extend new deposit endpoint to support metadata-only deposits, a subtask of T1021: SWORD deposit of metadata about an existing SWH object, as Resolved.
Fri, Nov 20, 10:20 AM · Core Loader, SWORD deposit

Wed, Nov 18

ardumont added a subtask for T1021: SWORD deposit of metadata about an existing SWH object: T2537: Extend new deposit endpoint to support metadata-only deposits.
Wed, Nov 18, 11:33 AM · Core Loader, SWORD deposit

Nov 3 2020

ardumont moved T871: Migrate swh-storage api functions relative to indexers to swh-indexer from Backlog to Archived on the SWORD deposit board.
Nov 3 2020, 4:07 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont moved T872: Deploy and restart indexers from Backlog to Archived on the SWORD deposit board.
Nov 3 2020, 4:07 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

Oct 26 2020

douardda added projects to T2585: Properly handle deleted/no longer available origins: Core Loader, Lister.
Oct 26 2020, 2:41 PM · Lister, Core Loader

Sep 25 2020

ardumont closed D4012: core.loader: Log information about origin currently being ingested.
Sep 25 2020, 9:04 AM · Core Loader
ardumont added a comment to D4012: core.loader: Log information about origin currently being ingested.

Our logging handler swh.core.logger.JournalHandler already knows how to pull some metadata from the celery tasks:
...

Sep 25 2020, 9:02 AM · Core Loader
olasd accepted D4012: core.loader: Log information about origin currently being ingested.
Sep 25 2020, 9:00 AM · Core Loader
swh-public-ci added a comment to D4012: core.loader: Log information about origin currently being ingested.

Build is green

Sep 25 2020, 8:58 AM · Core Loader
ardumont added inline comments to D4012: core.loader: Log information about origin currently being ingested.
Sep 25 2020, 8:56 AM · Core Loader
ardumont updated the diff for D4012: core.loader: Log information about origin currently being ingested.

Adapt according to suggestion

Sep 25 2020, 8:56 AM · Core Loader

Sep 24 2020

olasd added a comment to D4012: core.loader: Log information about origin currently being ingested.

To be clear, my main issue today, when I try to look through our logs to
investigate or plain read what's going on (after a deployment for example), I
don't have any clues immediately...

In my mind, the kibana information is not enough by itself, so i think i need
to cross information with say sentry to have some more context... It's
currently quite frustrating... up to an eventual point of, "oh well, I have
some other urgent matters somewhere else..." (sometimes I push through but
sometimes, I fail).

Sep 24 2020, 4:59 PM · Core Loader
swh-public-ci added a comment to D4012: core.loader: Log information about origin currently being ingested.

Build is green

Sep 24 2020, 4:06 PM · Core Loader
ardumont updated the summary of D4012: core.loader: Log information about origin currently being ingested.
Sep 24 2020, 4:05 PM · Core Loader
ardumont updated the diff for D4012: core.loader: Log information about origin currently being ingested.

Simplify to just one log statement

Sep 24 2020, 4:05 PM · Core Loader
ardumont added inline comments to D4012: core.loader: Log information about origin currently being ingested.
Sep 24 2020, 3:58 PM · Core Loader
ardumont added a comment to D4012: core.loader: Log information about origin currently being ingested.

ok, i'll adapt
i would not be against a nudge in the right direction to actually improve the logging

Sep 24 2020, 3:56 PM · Core Loader
douardda added a comment to D4012: core.loader: Log information about origin currently being ingested.
In D4012#99525, @olasd wrote:

I don't think the origin url and visit type should be sent in the task result; they're arguments of the task already.

If we want them logged by the worker when the task ends (which I agree would be useful), then we should improve logging on the worker/celery side to show some of the task arguments (for instance, if there's a "url" argument) instead / in addition of the task id.

Sep 24 2020, 3:28 PM · Core Loader
douardda added inline comments to D4012: core.loader: Log information about origin currently being ingested.
Sep 24 2020, 3:26 PM · Core Loader
ardumont added a comment to D4012: core.loader: Log information about origin currently being ingested.

I don't think the origin url and visit type should be sent in the task
result; they're arguments of the task already.

Sep 24 2020, 10:33 AM · Core Loader
ardumont added inline comments to D4012: core.loader: Log information about origin currently being ingested.
Sep 24 2020, 10:22 AM · Core Loader
olasd added a comment to D4012: core.loader: Log information about origin currently being ingested.

I don't think the origin url and visit type should be sent in the task result; they're arguments of the task already.

Sep 24 2020, 9:14 AM · Core Loader

Sep 23 2020

zack triaged T2634: swh-core: missing test dependency on requests as Low priority.
Sep 23 2020, 2:23 PM · Easy hack, Core & foundations

Sep 22 2020

olasd added a comment to T1461: Add loader-related metrics to swh-loader-core.

I think the second point mostly happened: the storage is returning statistics to the loader, but the loaders don't generally collect them.

Sep 22 2020, 6:13 PM · Core Loader, Metrics/monitoring
olasd updated the task description for T1461: Add loader-related metrics to swh-loader-core.
Sep 22 2020, 6:11 PM · Core Loader, Metrics/monitoring
olasd updated the task description for T1461: Add loader-related metrics to swh-loader-core.
Sep 22 2020, 6:10 PM · Core Loader, Metrics/monitoring
olasd placed T1461: Add loader-related metrics to swh-loader-core up for grabs.
Sep 22 2020, 6:10 PM · Core Loader, Metrics/monitoring
ardumont updated the summary of D4012: core.loader: Log information about origin currently being ingested.
Sep 22 2020, 6:07 PM · Core Loader

Aug 31 2020

vlorentz added a comment to T2471: NPM package angular-ts-manage fails to be properly loaded.

I'm afraid the only way to properly solve this is to wait until we stop writing metadata to the revision table

Aug 31 2020, 4:38 PM · Core Loader

Aug 19 2020

vlorentz triaged T2523: Archive opensource.samsung.com as Normal priority.
Aug 19 2020, 7:40 PM · Lister, Archive coverage

Jul 27 2020

ardumont closed T2481: Migrate dvcs loader tests code to pytest as Resolved.
Jul 27 2020, 3:21 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a parent task for T2481: Migrate dvcs loader tests code to pytest: T2221: Development workflow & code quality.
Jul 27 2020, 3:20 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 20 2020

ardumont closed T2483: tests: Make check-snapshot utility test function recursively check targetted object exists, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Jul 20 2020, 9:17 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2483: tests: Make check-snapshot utility test function recursively check targetted object exists as Resolved.
Jul 20 2020, 9:17 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Jul 20 2020, 9:16 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as Resolved.
Jul 20 2020, 9:16 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 17 2020

ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3551: tests: Reuse pytest fixtures from swh.loader.core.
Jul 17 2020, 12:12 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3550: tests: Reuse pytest fixtures from swh.loader.core.
Jul 17 2020, 12:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3549: tests: Reuse pytest fixtures from swh.loader.core.
Jul 17 2020, 12:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 16 2020

ardumont closed T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Jul 16 2020, 3:18 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest as Resolved.
Jul 16 2020, 3:18 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest: D3522: tests: Drop no longer used loader base tests code.
Jul 16 2020, 11:45 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3515: pytest_plugin: Avoid fixture client to declare optional dependency.
Jul 16 2020, 10:30 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 10 2020

ardumont added revisions to T2483: tests: Make check-snapshot utility test function recursively check targetted object exists: D3502: check_snapshot: Check existence down to the first level directories, D3503: check_snapshot: Check existence down to contents.
Jul 10 2020, 7:01 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont removed revisions from T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3503: check_snapshot: Check existence down to contents, D3502: check_snapshot: Check existence down to the first level directories.
Jul 10 2020, 7:01 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3503: check_snapshot: Check existence down to contents.
Jul 10 2020, 6:14 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3502: check_snapshot: Check existence down to the first level directories.
Jul 10 2020, 5:40 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont reopened T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Open.
Jul 10 2020, 3:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont reopened T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as "Open".

Reopening as i'm still refactoring/cleaning up more modules.

Jul 10 2020, 3:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3495: Expose a pytest_plugin module so other loaders can reuse for tests.
Jul 10 2020, 3:42 PM · SVN Loader, Mercurial loader, Git loader, Core Loader