Page MenuHomeSoftware Heritage
Feed Advanced Search

Jan 28 2020

ardumont added inline comments to D2585: Add tests for db_transaction and db_transaction_generator..
Jan 28 2020, 2:42 PM
ardumont updated the diff for D2592: webapp0: Switch to cassandra storage.

Fix python3-cassandra package name

Jan 28 2020, 2:32 PM
ardumont updated the diff for D2592: webapp0: Switch to cassandra storage.
  • Properly switch to webapp0's storage to cassandra storage
  • Add conditional to install the python3-cassandra when needed
Jan 28 2020, 2:32 PM
ardumont added a comment to D2592: webapp0: Switch to cassandra storage.

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

Jan 28 2020, 2:18 PM
ardumont updated the test plan for D2592: webapp0: Switch to cassandra storage.
Jan 28 2020, 2:17 PM
ardumont updated the test plan for D2592: webapp0: Switch to cassandra storage.
Jan 28 2020, 2:15 PM
ardumont requested changes to D2589: Add return type to get_storage..
Jan 28 2020, 2:13 PM
ardumont accepted D2585: Add tests for db_transaction and db_transaction_generator..
Jan 28 2020, 2:10 PM
ardumont updated the task description for T2244: Use swh-model for passing objects instead of dicts.
Jan 28 2020, 1:32 PM · Core & foundations
ardumont committed rDLDBASEae17e430d5d0: npm.loader: Skip artifacts with no intrinsic metadata (authored by ardumont).
npm.loader: Skip artifacts with no intrinsic metadata
Jan 28 2020, 12:28 PM
ardumont committed rDLDBASE6f3d6446fd0a: pypi.loader: Skip artifacts with no intrinsic metadata (authored by ardumont).
pypi.loader: Skip artifacts with no intrinsic metadata
Jan 28 2020, 12:28 PM
ardumont closed D2579: package.loader: Skip artifacts with no intrinsic metadata.
Jan 28 2020, 12:27 PM
ardumont updated the diff for D2579: package.loader: Skip artifacts with no intrinsic metadata.

Add tests on both loaders

Jan 28 2020, 12:06 PM
ardumont added a comment to D2592: webapp0: Switch to cassandra storage.

For some unknown reasons yet, i don't see the python-cassandra-driver (buster) uploaded in pergamon.
Don't know if it's related or not, It's marked as source, i need to continue the work on that [2]

Jan 28 2020, 11:34 AM
ardumont retitled D2592: webapp0: Switch to cassandra storage from storage0: Switch to cassandra storage to webapp0: Switch to cassandra storage.
Jan 28 2020, 11:01 AM
ardumont updated the diff for D2592: webapp0: Switch to cassandra storage.

Switch to cassandra storage only for webapp0

Jan 28 2020, 11:00 AM
ardumont added a comment to D2592: webapp0: Switch to cassandra storage.

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

Jan 28 2020, 10:35 AM
ardumont added a comment to D2592: webapp0: Switch to cassandra storage.

I also wanted to make sure i did not misconfigure. Nobody shouted at it so it seemed fine from that standpoint ;)

Jan 28 2020, 10:17 AM

Jan 27 2020

ardumont committed rPCASS7ffce5bcd79c: d/changelog: New release for buster (authored by ardumont).
d/changelog: New release for buster
Jan 27 2020, 5:28 PM
ardumont committed rDDOCdb1725926770: docs: Update swh-docker-dev mentions to swh-environment/docker (authored by ardumont).
docs: Update swh-docker-dev mentions to swh-environment/docker
Jan 27 2020, 5:14 PM
ardumont closed D2590: docs: Update swh-docker-dev mentions to swh-environment/docker.
Jan 27 2020, 5:14 PM
ardumont updated the diff for D2590: docs: Update swh-docker-dev mentions to swh-environment/docker.

Remove unneeded possessive form

Jan 27 2020, 5:14 PM
ardumont created D2592: webapp0: Switch to cassandra storage.
Jan 27 2020, 4:04 PM
ardumont accepted D2583: Implement content_update for the in-mem storage..
Jan 27 2020, 3:56 PM
ardumont accepted D2591: Install Cassandra in base-buster image..
Jan 27 2020, 1:34 PM
ardumont updated the diff for D2590: docs: Update swh-docker-dev mentions to swh-environment/docker.

Adapt according to review

Jan 27 2020, 12:34 PM
ardumont added inline comments to D2590: docs: Update swh-docker-dev mentions to swh-environment/docker.
Jan 27 2020, 12:31 PM
ardumont added a comment to T2183: Switch webapp0 to use swh-search instead of postgresql search..

It's deployed btw.

Jan 27 2020, 12:01 PM · Archive search, Storage manager
ardumont created D2590: docs: Update swh-docker-dev mentions to swh-environment/docker.
Jan 27 2020, 11:55 AM
ardumont updated the summary of D2566: Add Cassandra backend..
Jan 27 2020, 11:41 AM
ardumont added a comment to D2585: Add tests for db_transaction and db_transaction_generator..

looks like D2585 and D2586 are entangled together.

Jan 27 2020, 11:12 AM
ardumont accepted D2586: Make db_transaction* remove db/cur from the signature..

I agree with the reasoning in the description.

Jan 27 2020, 11:11 AM
ardumont added inline comments to D2587: Move Storage documentation and endpoint paths to a new StorageInterface class.
Jan 27 2020, 11:10 AM
ardumont updated subscribers of D2583: Implement content_update for the in-mem storage..
Jan 27 2020, 10:00 AM
ardumont accepted D2587: Move Storage documentation and endpoint paths to a new StorageInterface class.
Jan 27 2020, 9:47 AM
ardumont added a comment to D2587: Move Storage documentation and endpoint paths to a new StorageInterface class.

I started the review and asked for docstring updates.
I was gonna ask to add types as well.
But i stopped.

Jan 27 2020, 9:45 AM
ardumont accepted D2588: Rename in_memory.Storage to in_memory.InMemoryStorage..
Jan 27 2020, 9:40 AM
ardumont accepted D2584: Remove cur/db arguments from the in-mem storage..
Jan 27 2020, 9:00 AM

Jan 24 2020

ardumont committed rDSTOc8389c226868: 146: Fix typo (authored by ardumont).
146: Fix typo
Jan 24 2020, 2:54 PM
ardumont updated the summary of D2579: package.loader: Skip artifacts with no intrinsic metadata.
Jan 24 2020, 2:42 PM
ardumont committed rDSTO2ebcdf323a20: pgstorage: Empty temp tables instead of dropping them (authored by ardumont).
pgstorage: Empty temp tables instead of dropping them
Jan 24 2020, 2:38 PM
ardumont closed D2580: pgstorage: Empty temp tables instead of dropping them.
Jan 24 2020, 2:38 PM
ardumont added a comment to D2580: pgstorage: Empty temp tables instead of dropping them.

See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/892/console

Jan 24 2020, 2:37 PM
ardumont created D2581: search.elasticsearch: Increase request timeout.
Jan 24 2020, 2:30 PM
ardumont closed T2167: Deploy swh-search, a subtask of T1910: Redesign origin search using a dedicated component (swh-search), as Resolved.
Jan 24 2020, 1:46 PM · Archive search, Storage manager
ardumont closed T2167: Deploy swh-search, a subtask of T2183: Switch webapp0 to use swh-search instead of postgresql search., as Resolved.
Jan 24 2020, 1:46 PM · Archive search, Storage manager
ardumont closed T2167: Deploy swh-search as Resolved.

From triggering a search on webapp0, we have timeout connection from storage0 now.

Jan 24 2020, 1:46 PM · System administration, Archive search
ardumont accepted D2573: Add tox_env parameter to tox jobs..

looks good (iic)

Jan 24 2020, 1:38 PM
ardumont updated subscribers of D2580: pgstorage: Empty temp tables instead of dropping them.

Build has FAILED

yeah, well, that tends to happen when the environment to build cassandra job is no longer there...

Jan 24 2020, 1:37 PM
ardumont committed rSPPRIVCdb05975be07a: sentry: Add swh-search's dsn setup (authored by ardumont).
sentry: Add swh-search's dsn setup
Jan 24 2020, 1:23 PM
ardumont added a comment to D2580: pgstorage: Empty temp tables instead of dropping them.

Build has FAILED

Jan 24 2020, 1:13 PM
ardumont updated the diff for D2580: pgstorage: Empty temp tables instead of dropping them.

Plug to master branch

Jan 24 2020, 12:29 PM
ardumont added a comment to D2580: pgstorage: Empty temp tables instead of dropping them.

accepted through irc ;)

Jan 24 2020, 12:28 PM
ardumont updated the summary of D2580: pgstorage: Empty temp tables instead of dropping them.
Jan 24 2020, 12:24 PM
ardumont closed D2578: Deploy swh search on storage0 and make webapp0 use it for search.

landed.

Jan 24 2020, 12:16 PM · Archive search
ardumont updated the diff for D2580: pgstorage: Empty temp tables instead of dropping them.

Drop duplicated comment

Jan 24 2020, 12:14 PM
ardumont updated the summary of D2580: pgstorage: Empty temp tables instead of dropping them.
Jan 24 2020, 12:11 PM
ardumont updated the summary of D2580: pgstorage: Empty temp tables instead of dropping them.
Jan 24 2020, 12:10 PM
ardumont updated the summary of D2580: pgstorage: Empty temp tables instead of dropping them.
Jan 24 2020, 12:09 PM
ardumont created D2580: pgstorage: Empty temp tables instead of dropping them.
Jan 24 2020, 12:06 PM
ardumont added inline comments to D2578: Deploy swh search on storage0 and make webapp0 use it for search.
Jan 24 2020, 11:39 AM · Archive search
ardumont updated subscribers of D2578: Deploy swh search on storage0 and make webapp0 use it for search.
Jan 24 2020, 11:37 AM · Archive search
ardumont updated the summary of D2579: package.loader: Skip artifacts with no intrinsic metadata.
Jan 24 2020, 10:48 AM
ardumont created D2579: package.loader: Skip artifacts with no intrinsic metadata.
Jan 24 2020, 10:46 AM
ardumont added inline comments to D2578: Deploy swh search on storage0 and make webapp0 use it for search.
Jan 24 2020, 9:42 AM · Archive search
ardumont changed the status of T2167: Deploy swh-search from Open to Work in Progress.
Jan 24 2020, 9:38 AM · System administration, Archive search
ardumont changed the status of T2167: Deploy swh-search, a subtask of T1910: Redesign origin search using a dedicated component (swh-search), from Open to Work in Progress.
Jan 24 2020, 9:38 AM · Archive search, Storage manager
ardumont changed the status of T2167: Deploy swh-search, a subtask of T2183: Switch webapp0 to use swh-search instead of postgresql search., from Open to Work in Progress.
Jan 24 2020, 9:38 AM · Archive search, Storage manager
ardumont added a comment to T2185: Make webapp0 use Cassandra as storage backend..

Cool, looks like this is all ready within our code base:

Jan 24 2020, 9:38 AM · Storage manager
ardumont claimed T2167: Deploy swh-search.
Jan 24 2020, 9:37 AM · System administration, Archive search
ardumont renamed T1910: Redesign origin search using a dedicated component (swh-search) from Redesign origin search using a dedicated component to Redesign origin search using a dedicated component (swh-search).
Jan 24 2020, 9:28 AM · Archive search, Storage manager
ardumont added inline comments to D2578: Deploy swh search on storage0 and make webapp0 use it for search.
Jan 24 2020, 9:21 AM · Archive search
ardumont updated subscribers of D2578: Deploy swh search on storage0 and make webapp0 use it for search.
Jan 24 2020, 9:14 AM · Archive search
ardumont updated the test plan for D2578: Deploy swh search on storage0 and make webapp0 use it for search.
Jan 24 2020, 8:45 AM · Archive search

Jan 23 2020

ardumont created D2578: Deploy swh search on storage0 and make webapp0 use it for search.
Jan 23 2020, 5:10 PM · Archive search
ardumont added a comment to T546: Update debian loader to register origin_visit's state.

right, it is.

Jan 23 2020, 4:01 PM · Debian loader
ardumont closed T2238: Configure Sentry environments as Resolved.
Jan 23 2020, 3:16 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont committed rDLDBASEcd33b4bced7f: package.loader: Fix edge case when some listing returns no content (authored by ardumont).
package.loader: Fix edge case when some listing returns no content
Jan 23 2020, 3:08 PM
ardumont closed D2577: package.loader: Fix edge case when some listing returns no content somehow.
Jan 23 2020, 3:08 PM
ardumont updated the diff for D2577: package.loader: Fix edge case when some listing returns no content somehow.

Plug to master branch

Jan 23 2020, 3:07 PM
ardumont added a comment to T2238: Configure Sentry environments.

Finalizing deployment. Was missing a scheduler release and then a bug fix in there.

Jan 23 2020, 2:57 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont updated the diff for D2577: package.loader: Fix edge case when some listing returns no content somehow.

Improve instructions

Jan 23 2020, 2:54 PM
ardumont added inline comments to D2577: package.loader: Fix edge case when some listing returns no content somehow.
Jan 23 2020, 2:52 PM
ardumont created D2577: package.loader: Fix edge case when some listing returns no content somehow.
Jan 23 2020, 2:41 PM
ardumont committed rDSCHf6cc23122ec4: sentry: Fix initialization init_sentry call (authored by ardumont).
sentry: Fix initialization init_sentry call
Jan 23 2020, 2:24 PM
ardumont committed rDDEP4b5b68a543c1: tests: Adapt init_sentry api change call with environment parameter (authored by ardumont).
tests: Adapt init_sentry api change call with environment parameter
Jan 23 2020, 2:03 PM
ardumont closed T2181: Set SWH_MAIN_PACKAGE and SWH_SENTRY_ENVIRONMENT for all services as Resolved.

Deployed.

Jan 23 2020, 12:09 PM · Metrics/monitoring, Sprint 2019/12 (Monitor and Conquer), System administration
ardumont closed T2181: Set SWH_MAIN_PACKAGE and SWH_SENTRY_ENVIRONMENT for all services, a subtask of T2238: Configure Sentry environments, as Resolved.
Jan 23 2020, 12:09 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont added a subtask for T2238: Configure Sentry environments: T2181: Set SWH_MAIN_PACKAGE and SWH_SENTRY_ENVIRONMENT for all services.
Jan 23 2020, 11:13 AM · Sprint 2019/12 (Monitor and Conquer)
ardumont added a parent task for T2181: Set SWH_MAIN_PACKAGE and SWH_SENTRY_ENVIRONMENT for all services: T2238: Configure Sentry environments.
Jan 23 2020, 11:13 AM · Metrics/monitoring, Sprint 2019/12 (Monitor and Conquer), System administration
ardumont added a comment to D2576: sentry: Define setup for swh services (servers, workers, ...).

Well the nice thing is that i saw duplication (loader package, indexer) i can trim so good!

Jan 23 2020, 10:34 AM
ardumont added inline comments to D2576: sentry: Define setup for swh services (servers, workers, ...).
Jan 23 2020, 10:33 AM
ardumont added a comment to T1788: indexer-license: Investigate timeouts.

sample with our shiny sentry: https://sentry.softwareheritage.org/share/issue/f4a40625783b4a5588980005ddc5a5e6/

Jan 23 2020, 9:05 AM · Indexer
ardumont closed D2576: sentry: Define setup for swh services (servers, workers, ...).

closed by 0dd87a05c11e85fc80c34f4c42093db8f80b6c40

Jan 23 2020, 8:57 AM
ardumont updated the diff for D2576: sentry: Define setup for swh services (servers, workers, ...).

Rebase on latest staging

Jan 23 2020, 8:55 AM

Jan 22 2020

ardumont updated the diff for D2576: sentry: Define setup for swh services (servers, workers, ...).
  • Fix unneeded expansion
  • Remove obsolete staging entry about prometheus
  • Add missing quote
Jan 22 2020, 7:05 PM
ardumont added inline comments to D2576: sentry: Define setup for swh services (servers, workers, ...).
Jan 22 2020, 7:01 PM
ardumont updated the test plan for D2576: sentry: Define setup for swh services (servers, workers, ...).
Jan 22 2020, 6:57 PM
ardumont updated the diff for D2576: sentry: Define setup for swh services (servers, workers, ...).

Fix indexer fossology license typo

Jan 22 2020, 6:56 PM
ardumont added a revision to T2181: Set SWH_MAIN_PACKAGE and SWH_SENTRY_ENVIRONMENT for all services: D2576: sentry: Define setup for swh services (servers, workers, ...).
Jan 22 2020, 6:50 PM · Metrics/monitoring, Sprint 2019/12 (Monitor and Conquer), System administration