Page MenuHomeSoftware Heritage
Feed Advanced Search

Tue, Feb 11

ardumont accepted D2650: Add missing __init__.py file and fix tests.

Thanks for catching this.

Tue, Feb 11, 6:19 PM
ardumont accepted D2648: Move unrelated line out of try block..
Tue, Feb 11, 4:00 PM
ardumont committed rDSTO652ecf0aeb68: storages: Refactor journal operations with a dedicated writer collab (authored by ardumont).
storages: Refactor journal operations with a dedicated writer collab
Tue, Feb 11, 3:55 PM
ardumont closed D2634: storages: Refactor journal operations with a dedicated writer collaborator.
Tue, Feb 11, 3:55 PM
ardumont updated the diff for D2634: storages: Refactor journal operations with a dedicated writer collaborator.

Rebase and plug to master branch

Tue, Feb 11, 3:44 PM

Mon, Feb 10

ardumont updated the diff for D2634: storages: Refactor journal operations with a dedicated writer collaborator.

Remove unnecessary copy instruction (already converting from content object to
a new dict to ease the data key removal)

Mon, Feb 10, 7:16 PM
ardumont added inline comments to D2634: storages: Refactor journal operations with a dedicated writer collaborator.
Mon, Feb 10, 7:05 PM
ardumont updated the diff for D2634: storages: Refactor journal operations with a dedicated writer collaborator.
  • Add missing types on remaining collaborator method signatures
  • Remove unneeded to_dict conversion as the journal is already swh.model.model.BaseModel aware.
Mon, Feb 10, 7:04 PM
ardumont committed rDSTOa2e565d8b388: d/control: Update runtime dependency (authored by ardumont).
d/control: Update runtime dependency
Mon, Feb 10, 6:43 PM
ardumont retitled D2634: storages: Refactor journal operations with a dedicated writer collaborator from storages: Refactor journal operations with a dedicated writer collab to storages: Refactor journal operations with a dedicated writer collaborator.
Mon, Feb 10, 5:30 PM
ardumont updated the diff for D2634: storages: Refactor journal operations with a dedicated writer collaborator.
  • Rebase on latest master (includes D2640)
  • Squash commits
Mon, Feb 10, 5:26 PM
ardumont committed rDSTOde7797df0f0b: tests: Remove print statement (authored by ardumont).
tests: Remove print statement
Mon, Feb 10, 5:18 PM

Sat, Feb 8

ardumont updated the diff for D2634: storages: Refactor journal operations with a dedicated writer collaborator.
  • writer: Make the writer able to deal with BaseModel objects
Sat, Feb 8, 11:26 AM
ardumont updated the diff for D2634: storages: Refactor journal operations with a dedicated writer collaborator.
  • writer: Keep the signature to only what's needed
Sat, Feb 8, 10:41 AM
ardumont added a comment to D2634: storages: Refactor journal operations with a dedicated writer collaborator.

Also, you could make the journal writer support model objects, so you don't have to use .to_dict() every time?

Sat, Feb 8, 10:30 AM
ardumont added inline comments to D2634: storages: Refactor journal operations with a dedicated writer collaborator.
Sat, Feb 8, 10:25 AM

Fri, Feb 7

ardumont added inline comments to D2622: Add type annotations to indexer classes.
Fri, Feb 7, 6:50 PM
ardumont accepted D2640: Add a validation proxy for _add() methods..
Fri, Feb 7, 6:46 PM
ardumont updated the diff for D2643: storages: Refactor objstorage operations with a dedicated collaborator.

Fix

Fri, Feb 7, 6:25 PM
ardumont updated the diff for D2643: storages: Refactor objstorage operations with a dedicated collaborator.

Fix

Fri, Feb 7, 6:20 PM
ardumont created D2643: storages: Refactor objstorage operations with a dedicated collaborator.
Fri, Feb 7, 6:07 PM
ardumont updated the summary of D2634: storages: Refactor journal operations with a dedicated writer collaborator.
Fri, Feb 7, 5:54 PM
ardumont accepted D2642: Use skipped_content_add instead of content_add_metadata for skipped content..
Fri, Feb 7, 5:45 PM
ardumont accepted D2641: Make OriginVisit.snapshot optional..
Fri, Feb 7, 5:45 PM
ardumont accepted D2639: Make 'visible' the default status for present Contents..
Fri, Feb 7, 4:10 PM
ardumont accepted D2638: Make content length mandatory..
Fri, Feb 7, 4:10 PM
ardumont added a comment to D2634: storages: Refactor journal operations with a dedicated writer collaborator.

Also, you could make the journal writer support model objects, so you don't have to use .to_dict() every time?

Fri, Feb 7, 3:45 PM
ardumont updated the diff for D2634: storages: Refactor journal operations with a dedicated writer collaborator.
  • storages: Keep journal_writer as collaborator's name
  • Rebase on latest master
Fri, Feb 7, 3:40 PM
ardumont committed rDOBJS18020c3a56b3: objstorage: Return summary of batched object added to the store (authored by ardumont).
objstorage: Return summary of batched object added to the store
Fri, Feb 7, 3:28 PM
ardumont closed D2637: objstorage: Return summary of batched object added to the store.
Fri, Feb 7, 3:28 PM
ardumont added inline comments to D2634: storages: Refactor journal operations with a dedicated writer collaborator.
Fri, Feb 7, 3:26 PM
ardumont added inline comments to D2634: storages: Refactor journal operations with a dedicated writer collaborator.
Fri, Feb 7, 3:25 PM
ardumont added inline comments to D2637: objstorage: Return summary of batched object added to the store.
Fri, Feb 7, 3:18 PM
ardumont updated the diff for D2637: objstorage: Return summary of batched object added to the store.

Fix

Fri, Feb 7, 3:17 PM
ardumont updated the diff for D2637: objstorage: Return summary of batched object added to the store.
  • Rename content to object
  • Fix to compute correctly the mean
Fri, Feb 7, 12:06 PM
ardumont added inline comments to D2637: objstorage: Return summary of batched object added to the store.
Fri, Feb 7, 12:02 PM
ardumont created D2637: objstorage: Return summary of batched object added to the store.
Fri, Feb 7, 11:35 AM
ardumont committed rDDOC5bedc683a93e: docs/developer-setup: Add missing lzip dependency (authored by ardumont).
docs/developer-setup: Add missing lzip dependency
Fri, Feb 7, 11:19 AM
ardumont closed D2635: docs/developer-setup: Add missing lzip dependency.
Fri, Feb 7, 11:19 AM · Development documentation, Core & foundations
ardumont updated the summary of D2635: docs/developer-setup: Add missing lzip dependency.
Fri, Feb 7, 10:47 AM · Development documentation, Core & foundations
ardumont updated the summary of D2635: docs/developer-setup: Add missing lzip dependency.
Fri, Feb 7, 10:44 AM · Development documentation, Core & foundations
ardumont created D2635: docs/developer-setup: Add missing lzip dependency.
Fri, Feb 7, 10:42 AM · Development documentation, Core & foundations
ardumont committed rDLDG9dc60db00cef: loaders: Remove content size computation during conversion (authored by ardumont).
loaders: Remove content size computation during conversion
Fri, Feb 7, 3:32 AM
ardumont added a comment to D2614: scheduler.backend_es: Leave index opened when streaming bulk.

yeah, i don't get what you mean ;)
as usual, i just compacted code within methods but that may be irrelevant in the current context (although again, that's not in the current diff's scope either ;)

Fri, Feb 7, 3:20 AM
ardumont accepted D2622: Add type annotations to indexer classes.

Thanks, looks good.

Fri, Feb 7, 3:13 AM
ardumont accepted D2628: Unify exception raised by invalid input to API endpoints..
Fri, Feb 7, 3:00 AM
ardumont created D2634: storages: Refactor journal operations with a dedicated writer collaborator.
Fri, Feb 7, 2:49 AM

Thu, Feb 6

ardumont closed T1351: (periodically) ingest GNU package releases as Resolved.
Thu, Feb 6, 7:16 PM · Archive coverage
ardumont closed T1723: GNU Loader, a subtask of T1351: (periodically) ingest GNU package releases, as Resolved.
Thu, Feb 6, 7:16 PM · Archive coverage
ardumont closed T1723: GNU Loader as Resolved.
Thu, Feb 6, 7:16 PM · Archive coverage
ardumont closed T1962: Mercurial loader sometimes hangs forever during hg clone as Resolved.
Thu, Feb 6, 7:15 PM · Mercurial loader
ardumont closed T1626: loader-core: Unify counter with the storage's results as Resolved.

It's been done and removed... so closed.

Thu, Feb 6, 7:12 PM · Core Loader
ardumont committed rDLDHG34037a2b08a5: mercurial.loader: Simplify content conversion (authored by ardumont).
mercurial.loader: Simplify content conversion
Thu, Feb 6, 6:44 PM
ardumont closed D2633: mercurial.loader: Simplify content conversion.
Thu, Feb 6, 6:43 PM
ardumont committed rDLDSVN23865d22f0d5: loader.svn: Call storage's (skipped_)content_add endpoints (authored by ardumont).
loader.svn: Call storage's (skipped_)content_add endpoints
Thu, Feb 6, 6:42 PM
ardumont closed D2632: loader.svn: Call storage's (skipped_)content_add endpoints.
Thu, Feb 6, 6:42 PM
ardumont updated the diff for D2633: mercurial.loader: Simplify content conversion.

Update requirements

Thu, Feb 6, 6:37 PM
ardumont updated the diff for D2632: loader.svn: Call storage's (skipped_)content_add endpoints.

Update requirements

Thu, Feb 6, 6:35 PM
ardumont created D2633: mercurial.loader: Simplify content conversion.
Thu, Feb 6, 5:15 PM
ardumont created D2632: loader.svn: Call storage's (skipped_)content_add endpoints.
Thu, Feb 6, 5:14 PM
ardumont committed rDLDBASE588b06d8adf4: tests: Use new get_storage signature (authored by ardumont).
tests: Use new get_storage signature
Thu, Feb 6, 3:28 PM
ardumont committed rDLDBASE38c09a87ab04: loader.core.converters: Prefer the with open pattern to read file (authored by ardumont).
loader.core.converters: Prefer the with open pattern to read file
Thu, Feb 6, 3:28 PM
ardumont committed rDLDBASEfa65c2afe991: test_converters: Add coverage on prepare_contents method (authored by ardumont).
test_converters: Add coverage on prepare_contents method
Thu, Feb 6, 3:28 PM
ardumont committed rDLDBASE883858517030: test_converters: Migrate to pytest (authored by ardumont).
test_converters: Migrate to pytest
Thu, Feb 6, 3:28 PM
ardumont committed rDLDBASEda15a25eaf24: loader.core/package: Call storage's (skipped_)content_add endpoints (authored by ardumont).
loader.core/package: Call storage's (skipped_)content_add endpoints
Thu, Feb 6, 3:28 PM
ardumont closed D2631: loader.core/package: Call storage's (skipped_)content_add endpoints.
Thu, Feb 6, 3:28 PM
ardumont added inline comments to D2631: loader.core/package: Call storage's (skipped_)content_add endpoints.
Thu, Feb 6, 3:21 PM
ardumont added inline comments to D2631: loader.core/package: Call storage's (skipped_)content_add endpoints.
Thu, Feb 6, 3:20 PM
ardumont created D2631: loader.core/package: Call storage's (skipped_)content_add endpoints.
Thu, Feb 6, 3:15 PM
ardumont accepted D2626: Split 'content_add' method into 'content_add' and 'skipped_content_add'..
Thu, Feb 6, 2:33 PM
ardumont accepted D2630: Re-raise StorageArgumentException through API calls..
Thu, Feb 6, 2:31 PM
ardumont accepted D2626: Split 'content_add' method into 'content_add' and 'skipped_content_add'..
Thu, Feb 6, 12:33 PM
ardumont closed T2185: Make webapp0 use Cassandra as storage backend., a subtask of T1892: Cassandra as a storage backend, as Resolved.
Thu, Feb 6, 12:31 PM · meta-task, Storage manager
ardumont closed T2185: Make webapp0 use Cassandra as storage backend. as Resolved.
Thu, Feb 6, 12:31 PM · Storage manager
ardumont accepted D2627: In case of errors, return a simple dictionary instead of pickled exception..
Thu, Feb 6, 12:07 PM
ardumont added a comment to D2629: dataset: add graph export based on kafka.

Missing tests?

Thu, Feb 6, 12:06 PM
ardumont added a comment to D2614: scheduler.backend_es: Leave index opened when streaming bulk.

okay-ish but lifecycle of ES related services/objects is unclear to me.

Thu, Feb 6, 12:01 PM

Wed, Feb 5

ardumont added a comment to D2626: Split 'content_add' method into 'content_add' and 'skipped_content_add'..

oh yeah, i accepted the review in a first drafted version...

Wed, Feb 5, 6:17 PM
ardumont committed rDCIDX2174f424c1ee: indexer.storage: Fix missing import (authored by ardumont).
indexer.storage: Fix missing import
Wed, Feb 5, 6:13 PM
ardumont added a comment to D2627: In case of errors, return a simple dictionary instead of pickled exception..

starts to look, better ;)

Wed, Feb 5, 6:06 PM
ardumont accepted D2625: Show error text in apiget in case of unexpected status code..
Wed, Feb 5, 6:00 PM
ardumont added a comment to D2626: Split 'content_add' method into 'content_add' and 'skipped_content_add'..

Not finished the review but i feeling side-tracked by other stuff so submitting remarks for now, only small ones.

Wed, Feb 5, 5:55 PM
ardumont accepted D2623: Split Content class into two classes, for missing and non-missing contents..
Wed, Feb 5, 4:09 PM
ardumont added a comment to D2623: Split Content class into two classes, for missing and non-missing contents..
  • rename missing -> skipped

it avoid confusion with the terminology used in swh-storage,
as "content missing" means we never saw that content before;
while "skipped content" means we saw it but didn't ingest it
for some reason.

Wed, Feb 5, 4:08 PM
ardumont changed the status of T2185: Make webapp0 use Cassandra as storage backend., a subtask of T1892: Cassandra as a storage backend, from Open to Work in Progress.
Wed, Feb 5, 4:05 PM · meta-task, Storage manager
ardumont changed the status of T2185: Make webapp0 use Cassandra as storage backend. from Open to Work in Progress.
Wed, Feb 5, 4:05 PM · Storage manager
ardumont added a comment to T2185: Make webapp0 use Cassandra as storage backend..

storage02.euwest.azure exposes a rpc server using cassandra as storage backend.
webapp0 has been updated to use it.

Wed, Feb 5, 4:04 PM · Storage manager
ardumont closed T2183: Switch webapp0 to use swh-search instead of postgresql search. as Resolved.
Wed, Feb 5, 3:02 PM · Archive search, Storage manager
ardumont closed T2183: Switch webapp0 to use swh-search instead of postgresql search., a subtask of T2185: Make webapp0 use Cassandra as storage backend., as Resolved.
Wed, Feb 5, 3:02 PM · Storage manager
ardumont closed T2183: Switch webapp0 to use swh-search instead of postgresql search., a subtask of T2182: Switch production swh-web to use swh-search instead of postgresql search., as Resolved.
Wed, Feb 5, 3:02 PM · Archive search, Storage manager
ardumont committed rSPRE13403ace3042: azure: Add storage02.euwest.azure (authored by ardumont).
azure: Add storage02.euwest.azure
Wed, Feb 5, 2:43 PM
ardumont added a project to T2266: Use python-cassandra 3.21: Storage manager.
Wed, Feb 5, 1:50 PM · Storage manager
ardumont accepted D2626: Split 'content_add' method into 'content_add' and 'skipped_content_add'..
Wed, Feb 5, 12:27 PM
ardumont added a comment to D2626: Split 'content_add' method into 'content_add' and 'skipped_content_add'..

BUILD has failed

Wed, Feb 5, 12:25 PM
ardumont committed rSPREc71aca6fe291: azure: Define storage node within terraform manifests (authored by ardumont).
azure: Define storage node within terraform manifests
Wed, Feb 5, 12:07 PM
ardumont closed D2621: azure: Add new storage01.euwest.azure node.
Wed, Feb 5, 12:07 PM
ardumont added a comment to D2621: azure: Add new storage01.euwest.azure node.

Creation went smoothly for storage01.euwest.azure up to puppet run \m/

Wed, Feb 5, 12:07 PM
ardumont updated the diff for D2621: azure: Add new storage01.euwest.azure node.
  • Migrate to new dsl (for_each)
  • Use the right debian template name
  • Remove unnecessary disk definition (which failed the template creation)
  • Rename the resource to the correct name euwest-server`s`
Wed, Feb 5, 12:06 PM
ardumont updated the summary of D2621: azure: Add new storage01.euwest.azure node.
Wed, Feb 5, 12:04 PM
ardumont accepted D2623: Split Content class into two classes, for missing and non-missing contents..
Wed, Feb 5, 11:19 AM