Page MenuHomeSoftware Heritage
Feed Advanced Search

Yesterday

vlorentz added a comment to T2003: Content replayer may try to copy objects before they are available in an objstorage.

馃憤 to both your messages

Tue, Sep 17, 2:04 PMJournal
olasd added a comment to T2003: Content replayer may try to copy objects before they are available in an objstorage.

I agree with @douardda that the "failed content" queue + separate processor approach would be the most sensible.

Tue, Sep 17, 1:33 PMJournal
douardda added a comment to T2003: Content replayer may try to copy objects before they are available in an objstorage.

My guts on this task tell me that what we need (what we really really need) is a 3rd solution:

Tue, Sep 17, 12:20 PMJournal
vlorentz triaged T2003: Content replayer may try to copy objects before they are available in an objstorage as Normal priority.
Tue, Sep 17, 11:58 AMJournal

Tue, Sep 3

olasd closed T528: swh-journal: Create a journal client listing objects of a given type, a subtask of T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage, as Resolved.
Tue, Sep 3, 1:22 PMJournal
olasd closed T528: swh-journal: Create a journal client listing objects of a given type as Resolved.

We do have the infra for journal clients now.

Tue, Sep 3, 1:22 PMJournal

Fri, Aug 23

olasd closed T1017: Estimate for Kafka cluster specifications, a subtask of T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage, as Resolved.
Fri, Aug 23, 6:42 PMJournal

Jul 3 2019

ardumont closed D1683: swh-indexer-journal-client: Adapt systemd and configuration according to latest version.
Jul 3 2019, 6:09 PMJournal, Indexer
ardumont updated the diff for D1683: swh-indexer-journal-client: Adapt systemd and configuration according to latest version.
  • swh-indexer-journal-client: Update cli to latest version
  • swh-indexer-journal-client: Fix configuration change
Jul 3 2019, 6:09 PMJournal, Indexer
vlorentz accepted D1683: swh-indexer-journal-client: Adapt systemd and configuration according to latest version.
Jul 3 2019, 4:44 PMJournal, Indexer
ardumont retitled D1683: swh-indexer-journal-client: Adapt systemd and configuration according to latest version from swh-indexer-journal-client: Adapt configuration to swh-indexer-journal-client: Adapt systemd and configuration according to latest version.
Jul 3 2019, 4:38 PMJournal, Indexer
ardumont added projects to D1683: swh-indexer-journal-client: Adapt systemd and configuration according to latest version: Indexer, Journal.
Jul 3 2019, 4:34 PMJournal, Indexer
ardumont placed T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage up for grabs.
Jul 3 2019, 3:26 PMJournal
ardumont placed T1278: swh-journal: the monitoring tool question! up for grabs.
Jul 3 2019, 3:26 PMJournal
ardumont placed T1279: swh-journal: The migration problem up for grabs.
Jul 3 2019, 3:26 PMJournal
ardumont closed D1678: Stop using the 'hidden' click option.
Jul 3 2019, 10:26 AMJournal
swh-public-ci added a comment to D1678: Stop using the 'hidden' click option.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/176/ for more details.

Jul 3 2019, 10:23 AMJournal
ardumont updated the diff for D1678: Stop using the 'hidden' click option.

Update help message to add deprecated notice (as reasonably proposed)

Jul 3 2019, 10:19 AMJournal
douardda added a comment to D1678: Stop using the 'hidden' click option.

Maybe modify help messages the way vlorentz did in D1674

Jul 3 2019, 10:04 AMJournal
douardda accepted D1678: Stop using the 'hidden' click option.
Jul 3 2019, 10:04 AMJournal

Jul 2 2019

swh-public-ci added a comment to D1678: Stop using the 'hidden' click option.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/174/ for more details.

Jul 2 2019, 7:15 PMJournal
ardumont added a project to D1678: Stop using the 'hidden' click option: Journal.
Jul 2 2019, 7:13 PMJournal

Jun 28 2019

douardda added a comment to T1828: Improve directory journal backfill performance.

1 month is good enough. Let's stick to this.

Jun 28 2019, 10:13 AMMirror, Journal

Jun 25 2019

olasd changed the status of T1828: Improve directory journal backfill performance from Open to Work in Progress.

With 16 processes in parallel still, adding more CPUs gives an ETA of ~1 month, which stays pretty bad.

Jun 25 2019, 6:31 PMMirror, Journal
olasd added a comment to T1828: Improve directory journal backfill performance.

Running the directory backfiller (single instance) against belvedere yields an ETA of 250 days, which is around a 3x speedup from somerset.

Jun 25 2019, 2:54 PMMirror, Journal
douardda added a comment to T1828: Improve directory journal backfill performance.

have we now any insight on the behavior of the backfiller against belvedere?

Jun 25 2019, 9:27 AMMirror, Journal
douardda added a comment to T1827: Tweak content backfill order to help content replayer.

I'm enclined to prefer option 2, since performance is an issue we cannot underestimate...

Jun 25 2019, 9:25 AMMirror, Journal

Jun 18 2019

olasd triaged T1828: Improve directory journal backfill performance as High priority.
Jun 18 2019, 3:57 PMMirror, Journal
olasd triaged T1827: Tweak content backfill order to help content replayer as High priority.
Jun 18 2019, 3:44 PMMirror, Journal

Apr 29 2019

vlorentz closed T1541: journal publisher may crash on HTTP timeouts as Wontfix.

Closing because we removed the journal publisher. (My first WONTFIX <3)

Apr 29 2019, 2:26 PMJournal

Apr 12 2019

ardumont retitled D1345: swh.journal: Add backfiller implementation from swh.journal: Bootstrap backfiller to swh.journal: Add backfiller implementation.
Apr 12 2019, 3:50 PMStorage manager, Journal
ardumont closed T1603: kafka storage backfiller as Resolved.

Closed by D1345

Apr 12 2019, 3:50 PMJournal, Sprint 2019 03
ardumont closed D1345: swh.journal: Add backfiller implementation.
Apr 12 2019, 2:21 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/145/ for more details.

Apr 12 2019, 2:14 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Rebase to latest master and branch diff to master

Apr 12 2019, 2:10 PMStorage manager, Journal
olasd accepted D1345: swh.journal: Add backfiller implementation.
Apr 12 2019, 2:08 PMStorage manager, Journal
vlorentz closed T1628: replayer: non reliability when fetching items from kafka as Resolved.
Apr 12 2019, 10:01 AMJournal

Apr 11 2019

swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/144/ for more details.

Apr 11 2019, 6:33 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Fix pep8 violation (missing one blank line)

Apr 11 2019, 6:29 PMStorage manager, Journal
Harbormaster failed remote builds in B5345: Diff 4537 for D1345: swh.journal: Add backfiller implementation!
Apr 11 2019, 6:27 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 11 2019, 6:27 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Rebase on latest master

Apr 11 2019, 6:26 PMStorage manager, Journal
ardumont updated subscribers of D1345: swh.journal: Add backfiller implementation.
Apr 11 2019, 10:00 AMStorage manager, Journal

Apr 10 2019

vlorentz added a revision to T1628: replayer: non reliability when fetching items from kafka: D1390: Make the replayer commit *after* adding objects to the replica..
Apr 10 2019, 11:41 AMJournal
vlorentz moved T1603: kafka storage backfiller from Backlog to in progress on the Sprint 2019 03 board.
Apr 10 2019, 10:50 AMJournal, Sprint 2019 03

Apr 9 2019

Harbormaster failed remote builds in B5285: Diff 4477 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 6:07 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 6:07 PMStorage manager, Journal
ardumont added inline comments to D1345: swh.journal: Add backfiller implementation.
Apr 9 2019, 6:07 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.
  • backfill: Use the right logging instruction
  • backfill: Fix logging statement
Apr 9 2019, 6:04 PMStorage manager, Journal
anlambert added inline comments to D1345: swh.journal: Add backfiller implementation.
Apr 9 2019, 6:02 PMStorage manager, Journal
Harbormaster failed remote builds in B5283: Diff 4475 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 6:01 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 6:01 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Rebase on latest master

Apr 9 2019, 6:01 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 6:00 PMStorage manager, Journal
Harbormaster failed remote builds in B5282: Diff 4474 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 6:00 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.
  • backfill: Improve module docstrings
  • backfill: Flush when data is sent
  • backfill: Define a _compute_shit_bits function to reuse code
Apr 9 2019, 5:59 PMStorage manager, Journal
ardumont added inline comments to D1345: swh.journal: Add backfiller implementation.
Apr 9 2019, 5:46 PMStorage manager, Journal
anlambert added inline comments to D1345: swh.journal: Add backfiller implementation.
Apr 9 2019, 4:48 PMStorage manager, Journal
Harbormaster failed remote builds in B5238: Diff 4430 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 4:26 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 4:26 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.
  • Remove publisher notion
Apr 9 2019, 4:22 PMStorage manager, Journal
Harbormaster failed remote builds in B5219: Diff 4412 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 12:30 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 12:30 PMStorage manager, Journal
Harbormaster failed remote builds in B5218: Diff 4411 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 12:27 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 12:27 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

oops, the rebase should have been on cleanup-publisher branch

Apr 9 2019, 12:22 PMStorage manager, Journal
Harbormaster failed remote builds in B5215: Diff 4409 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 12:22 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 12:22 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Rebase on master

Apr 9 2019, 12:22 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Rebase on cleanup_publisher branch

Apr 9 2019, 12:19 PMStorage manager, Journal
ardumont added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 12:03 PMStorage manager, Journal
Harbormaster failed remote builds in B5210: Diff 4402 for D1345: swh.journal: Add backfiller implementation!
Apr 9 2019, 11:57 AMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 9 2019, 11:57 AMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Rebase on latest master

Apr 9 2019, 11:54 AMStorage manager, Journal
zack triaged T1628: replayer: non reliability when fetching items from kafka as Normal priority.
Apr 9 2019, 9:59 AMJournal

Apr 8 2019

swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/124/ for more details.

Apr 8 2019, 3:58 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/123/ for more details.

Apr 8 2019, 3:55 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.
  • backfill: Make partition key a singleton
Apr 8 2019, 3:54 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.
  • backfill: Fix origin_visit support to use the right format
  • backfill: Flush the messages to the journal when done with the batch
Apr 8 2019, 3:52 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/122/ for more details.

Apr 8 2019, 3:19 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Fix pep8 violation

Apr 8 2019, 3:16 PMStorage manager, Journal
Harbormaster failed remote builds in B5193: Diff 4387 for D1345: swh.journal: Add backfiller implementation!
Apr 8 2019, 3:12 PMStorage manager, Journal
swh-public-ci added a comment to D1345: swh.journal: Add backfiller implementation.

Build has FAILED

Apr 8 2019, 3:12 PMStorage manager, Journal
ardumont updated the summary of D1345: swh.journal: Add backfiller implementation.
Apr 8 2019, 3:12 PMStorage manager, Journal
ardumont updated the diff for D1345: swh.journal: Add backfiller implementation.

Add support for snapshot and directory

Apr 8 2019, 3:12 PMStorage manager, Journal

Apr 6 2019

ardumont added projects to D1345: swh.journal: Add backfiller implementation: Journal, Storage manager.
Apr 6 2019, 12:01 AMStorage manager, Journal

Apr 4 2019

olasd updated the task description for T1624: Normalize data sent from clients to the storage.
Apr 4 2019, 12:31 PMStorage manager, Journal
olasd triaged T1624: Normalize data sent from clients to the storage as Normal priority.
Apr 4 2019, 12:31 PMStorage manager, Journal

Apr 2 2019

zack added a project to T1603: kafka storage backfiller: Journal.
Apr 2 2019, 2:44 PMJournal, Sprint 2019 03
ardumont closed T1276: swh-journal: Add tests as Resolved.
Apr 2 2019, 12:12 PMContinuous Integration, Journal
ardumont closed T1276: swh-journal: Add tests, a subtask of T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage, as Resolved.
Apr 2 2019, 12:12 PMJournal

Mar 25 2019

ardumont closed D1239: swh-journal: Add tests to the journal publisher using kafka instance.
Mar 25 2019, 3:53 PMJournal
swh-public-ci added a comment to D1239: swh-journal: Add tests to the journal publisher using kafka instance.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/63/ for more details.

Mar 25 2019, 3:51 PMJournal
ardumont updated the diff for D1239: swh-journal: Add tests to the journal publisher using kafka instance.
  • tests: Inline the kafka_consumer pytest fixture
  • tests: Simplify initialization test to use fixtures consistently
Mar 25 2019, 3:50 PMJournal
Harbormaster failed remote builds in B4784: Diff 4075 for D1239: swh-journal: Add tests to the journal publisher using kafka instance!
Mar 25 2019, 11:35 AMJournal
swh-public-ci added a comment to D1239: swh-journal: Add tests to the journal publisher using kafka instance.

Build has FAILED

Mar 25 2019, 11:35 AMJournal
ardumont updated the diff for D1239: swh-journal: Add tests to the journal publisher using kafka instance.

Rework commit messages

Mar 25 2019, 11:33 AMJournal

Mar 22 2019

swh-public-ci added a comment to D1239: swh-journal: Add tests to the journal publisher using kafka instance.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/61/ for more details.

Mar 22 2019, 4:16 PMJournal
ardumont updated the diff for D1239: swh-journal: Add tests to the journal publisher using kafka instance.
  • test_publisher_kafka: Rename test_publisher to make a nicer/smaller diff
Mar 22 2019, 4:14 PMJournal
swh-public-ci added a comment to D1239: swh-journal: Add tests to the journal publisher using kafka instance.

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/60/ for more details.

Mar 22 2019, 4:13 PMJournal