Page MenuHomeSoftware Heritage

Sprint 2019 03Goal
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

aka 'The Mirror'

Recent Activity

Jan 11 2021

vlorentz set the icon for Sprint 2019 03 to Goal.
Jan 11 2021, 1:46 PM

Sep 24 2020

douardda closed T1601: Journal client of swh-storage mirrors as Resolved.

I guess this can be closed now

Sep 24 2020, 2:42 PM · Sprint 2019 03

Jan 27 2020

vlorentz closed T1602: Analyze kakfa storage requirements as Resolved.
Jan 27 2020, 4:32 PM · Sprint 2019 03

Jun 18 2019

olasd closed T1600: Write a storage backend that writes to kafka as Resolved.
Jun 18 2019, 2:54 PM · Sprint 2019 03
vlorentz reassigned T1601: Journal client of swh-storage mirrors from vlorentz to douardda.
Jun 18 2019, 1:26 PM · Sprint 2019 03

Jun 4 2019

douardda closed T1620: Sprint deliverable: deploy a kafka-in-the-loop mirror instance as Resolved.

For the record, we do have a docker based deployment for this, but not a 'production' (aka based on debian packages) one. So I close this task for now. The task is now to document this (T1782).

Jun 4 2019, 10:19 AM · System administration, Sprint 2019 03

May 14 2019

olasd closed T1604: Improve kafka deployment as Resolved.

So that took a few tries in puppet, but adding new brokers to the kafka deployment should now be seamless.

May 14 2019, 7:49 PM · System administration, Sprint 2019 03

Apr 18 2019

douardda closed T1599: Analyze objstorage's Azure updateness as Resolved.
Apr 18 2019, 10:35 AM · Sprint 2019 03

Apr 12 2019

ardumont closed T1603: kafka storage backfiller as Resolved.

Closed by D1345

Apr 12 2019, 3:50 PM · Journal, Sprint 2019 03
ardumont closed T1598: Add stasd probes in swh-storage as Resolved.
Apr 12 2019, 3:49 PM · Sprint 2019 03

Apr 10 2019

vlorentz moved T1607: Graph object count per object storage backend from Backlog to in progress on the Sprint 2019 03 board.
Apr 10 2019, 10:52 AM · Metrics/monitoring, Sprint 2019 03
vlorentz moved T1621: Add metrics to the currently deployed kafka cluster from Backlog to in progress on the Sprint 2019 03 board.
Apr 10 2019, 10:52 AM · System administration, Metrics/monitoring, Sprint 2019 03
vlorentz moved T1620: Sprint deliverable: deploy a kafka-in-the-loop mirror instance from Backlog to done on the Sprint 2019 03 board.
Apr 10 2019, 10:51 AM · System administration, Sprint 2019 03
vlorentz moved T1601: Journal client of swh-storage mirrors from Backlog to done on the Sprint 2019 03 board.
Apr 10 2019, 10:51 AM · Sprint 2019 03
vlorentz claimed T1601: Journal client of swh-storage mirrors.
Apr 10 2019, 10:51 AM · Sprint 2019 03
vlorentz moved T1603: kafka storage backfiller from Backlog to in progress on the Sprint 2019 03 board.
Apr 10 2019, 10:50 AM · Journal, Sprint 2019 03
vlorentz moved T1600: Write a storage backend that writes to kafka from in progress to done on the Sprint 2019 03 board.
Apr 10 2019, 10:50 AM · Sprint 2019 03

Apr 2 2019

zack added a project to T1603: kafka storage backfiller: Journal.
Apr 2 2019, 2:44 PM · Journal, Sprint 2019 03
zack added a project to T1620: Sprint deliverable: deploy a kafka-in-the-loop mirror instance: System administration.
Apr 2 2019, 2:43 PM · System administration, Sprint 2019 03
zack added a project to T1604: Improve kafka deployment: System administration.
Apr 2 2019, 2:43 PM · System administration, Sprint 2019 03
zack added projects to T1621: Add metrics to the currently deployed kafka cluster: Metrics/monitoring, System administration.
Apr 2 2019, 2:43 PM · System administration, Metrics/monitoring, Sprint 2019 03
olasd changed the status of T1621: Add metrics to the currently deployed kafka cluster from Open to Work in Progress.
Apr 2 2019, 12:56 PM · System administration, Metrics/monitoring, Sprint 2019 03
olasd triaged T1620: Sprint deliverable: deploy a kafka-in-the-loop mirror instance as Normal priority.
Apr 2 2019, 12:53 PM · System administration, Sprint 2019 03
zack renamed T1603: kafka storage backfiller from kafka storage backfill to kafka storage backfiller.
Apr 2 2019, 12:37 PM · Journal, Sprint 2019 03
ardumont claimed T1603: kafka storage backfiller.
Apr 2 2019, 12:11 PM · Journal, Sprint 2019 03
ardumont updated the task description for T1603: kafka storage backfiller.
Apr 2 2019, 12:10 PM · Journal, Sprint 2019 03

Apr 1 2019

douardda moved T1599: Analyze objstorage's Azure updateness from Backlog to in progress on the Sprint 2019 03 board.
Apr 1 2019, 11:54 AM · Sprint 2019 03

Mar 29 2019

ardumont added a comment to T1598: Add stasd probes in swh-storage.

Btw, i'll remove the @timed get_storage as this is just noisy on the graph ;)
(That concurs with what you said orally)

Mar 29 2019, 10:56 AM · Sprint 2019 03
ardumont added a comment to T1598: Add stasd probes in swh-storage.

Some more granular probes inside the main storage implementation would be helpful, e.g. counting the actual number of objects inserted by each request.

Mar 29 2019, 10:47 AM · Sprint 2019 03

Mar 28 2019

olasd added a comment to T1598: Add stasd probes in swh-storage.

The current state of these probes has been deployed in production (and the workers restarted).

Mar 28 2019, 6:38 PM · Sprint 2019 03
olasd closed T1608: Write all new objects to azure synchronously as Resolved.

This is now deployed on the main storage. Let's see how it holds up.

Mar 28 2019, 6:10 PM · Sprint 2019 03
olasd closed T1608: Write all new objects to azure synchronously, a subtask of T1599: Analyze objstorage's Azure updateness, as Resolved.
Mar 28 2019, 6:10 PM · Sprint 2019 03

Mar 27 2019

olasd closed T1609: Mitigate azure objstorage http(s) pipelining issues by backporting urllib3 patch, a subtask of T1608: Write all new objects to azure synchronously, as Resolved.
Mar 27 2019, 12:14 PM · Sprint 2019 03
olasd closed T1609: Mitigate azure objstorage http(s) pipelining issues by backporting urllib3 patch as Resolved by committing rSPSITE30b0c3819523: Enable some backported packages by default system-wide.
Mar 27 2019, 12:14 PM · Sprint 2019 03
olasd added a comment to T1609: Mitigate azure objstorage http(s) pipelining issues by backporting urllib3 patch.

@olasd, urllib3 1.124.1 is available on stretch-backports [1], including the commit you mentioned.

[1] https://packages.debian.org/stretch-backports/python-urllib3

Mar 27 2019, 10:55 AM · Sprint 2019 03
anlambert added a comment to T1609: Mitigate azure objstorage http(s) pipelining issues by backporting urllib3 patch.

@olasd, urllib3 1.24.1 is available on stretch-backports [1], including the commit you mentioned.

Mar 27 2019, 10:55 AM · Sprint 2019 03

Mar 26 2019

olasd triaged T1609: Mitigate azure objstorage http(s) pipelining issues by backporting urllib3 patch as High priority.
Mar 26 2019, 7:05 PM · Sprint 2019 03
olasd added a subtask for T1608: Write all new objects to azure synchronously: T1577: Compare/benchmark objstorage backends .
Mar 26 2019, 6:58 PM · Sprint 2019 03
olasd triaged T1608: Write all new objects to azure synchronously as High priority.
Mar 26 2019, 6:57 PM · Sprint 2019 03
olasd changed the status of T1607: Graph object count per object storage backend, a subtask of T1599: Analyze objstorage's Azure updateness, from Open to Work in Progress.
Mar 26 2019, 6:52 PM · Sprint 2019 03
olasd changed the status of T1607: Graph object count per object storage backend from Open to Work in Progress.

https://grafana.softwareheritage.org/d/jScG7g6mk/objstorage-object-counts shows the data that we're currently able to collect.

Mar 26 2019, 6:52 PM · Metrics/monitoring, Sprint 2019 03
vlorentz added a revision to T1600: Write a storage backend that writes to kafka: D1294: Add a new JournalWriter interface, which is notified by swh-storage before writing to pgsql..
Mar 26 2019, 5:10 PM · Sprint 2019 03
olasd triaged T1607: Graph object count per object storage backend as High priority.
Mar 26 2019, 4:31 PM · Metrics/monitoring, Sprint 2019 03
vlorentz claimed T1600: Write a storage backend that writes to kafka.
Mar 26 2019, 3:15 PM · Sprint 2019 03
vlorentz moved T1600: Write a storage backend that writes to kafka from Backlog to in progress on the Sprint 2019 03 board.
Mar 26 2019, 3:14 PM · Sprint 2019 03
vlorentz merged T1606: Make swh-storage write to Kafka ahead of Postgresql into T1600: Write a storage backend that writes to kafka.
Mar 26 2019, 3:14 PM · Sprint 2019 03
vlorentz merged task T1606: Make swh-storage write to Kafka ahead of Postgresql into T1600: Write a storage backend that writes to kafka.
Mar 26 2019, 3:14 PM · Sprint 2019 03
vlorentz moved T1606: Make swh-storage write to Kafka ahead of Postgresql from Backlog to in progress on the Sprint 2019 03 board.
Mar 26 2019, 3:13 PM · Sprint 2019 03
vlorentz claimed T1606: Make swh-storage write to Kafka ahead of Postgresql.
Mar 26 2019, 3:13 PM · Sprint 2019 03
vlorentz triaged T1606: Make swh-storage write to Kafka ahead of Postgresql as Normal priority.
Mar 26 2019, 3:13 PM · Sprint 2019 03