Page MenuHomeSoftware Heritage
Feed Advanced Search

Nov 5 2015

ardumont changed the status of T183: gnu/old-gnu injection to production from Open to Work in Progress.
Nov 5 2015, 10:57 AM · Directory loader, Tarball loader
ardumont changed the status of T183: gnu/old-gnu injection to production, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, from Open to Work in Progress.
Nov 5 2015, 10:57 AM · Tarball loader, Directory loader
ardumont changed the status of T183: gnu/old-gnu injection to production from Work in Progress to Open.
Nov 5 2015, 10:57 AM · Directory loader, Tarball loader
ardumont changed the status of T183: gnu/old-gnu injection to production, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, from Work in Progress to Open.
Nov 5 2015, 10:57 AM · Tarball loader, Directory loader
ardumont changed the status of T183: gnu/old-gnu injection to production from Open to Work in Progress.
Nov 5 2015, 10:56 AM · Directory loader, Tarball loader
ardumont changed the status of T183: gnu/old-gnu injection to production, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, from Open to Work in Progress.
Nov 5 2015, 10:56 AM · Tarball loader, Directory loader
ardumont added a comment to T183: gnu/old-gnu injection to production.
  • old-gnu tarball messages sent to queue
  • gnu tarball messages sent to queue
Nov 5 2015, 10:48 AM · Directory loader, Tarball loader
olasd added a comment to T163: Add entry points to create / request entities in swh.storage.

The first two points have been implemented in rDSTOc779ebdff8d2

Nov 5 2015, 10:30 AM · Storage manager

Nov 3 2015

olasd closed T121: Populate fetch_history as Resolved.

This has been done: All the GitHub repositories where the injection was successful or unsuccessful now have an entry in fetch_history.

Nov 3 2015, 7:08 PM · Storage manager
ardumont renamed T183: gnu/old-gnu injection to production from Go to production to gnu/old-gnu injection to production.
Nov 3 2015, 3:02 PM · Directory loader, Tarball loader
ardumont raised the priority of T183: gnu/old-gnu injection to production from Normal to High.
Nov 3 2015, 1:53 PM · Directory loader, Tarball loader
ardumont created T183: gnu/old-gnu injection to production.
Nov 3 2015, 1:53 PM · Directory loader, Tarball loader
ardumont removed a parent task for T182: investigate gnu.org tarball ingestion failures: T90: swh-loader-tar - Load gnu tarballs in swh-storage.
Nov 3 2015, 1:52 PM · Origin-GNU, Directory loader, Tarball loader
ardumont removed a subtask for T90: swh-loader-tar - Load gnu tarballs in swh-storage: T182: investigate gnu.org tarball ingestion failures.
Nov 3 2015, 1:52 PM · Tarball loader, Directory loader
ardumont lowered the priority of T177: Investigate `Not implemented 'compression type 6 implode'` from Normal to Low.
Nov 3 2015, 1:52 PM · Directory loader, Tarball loader
ardumont lowered the priority of T182: investigate gnu.org tarball ingestion failures from Normal to Low.
Nov 3 2015, 1:51 PM · Origin-GNU, Directory loader, Tarball loader
ardumont lowered the priority of T182: investigate gnu.org tarball ingestion failures from High to Normal.
Nov 3 2015, 12:24 PM · Origin-GNU, Directory loader, Tarball loader
ardumont created T182: investigate gnu.org tarball ingestion failures.
Nov 3 2015, 12:03 PM · Origin-GNU, Directory loader, Tarball loader
ardumont closed T178: Replay loading mirror gnu and old-gnu from scratch, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Nov 3 2015, 12:02 PM · Tarball loader, Directory loader
ardumont closed T178: Replay loading mirror gnu and old-gnu from scratch as Resolved.
Nov 3 2015, 12:02 PM · Directory loader, Tarball loader
olasd claimed T163: Add entry points to create / request entities in swh.storage.
Nov 3 2015, 10:18 AM · Storage manager
olasd raised the priority of T163: Add entry points to create / request entities in swh.storage from Normal to High.

This is getting necessary for the Debian and GitHub listers

Nov 3 2015, 10:18 AM · Storage manager

Nov 2 2015

ardumont added a comment to T178: Replay loading mirror gnu and old-gnu from scratch.
  • db cleanup
  • old gnu messages sent to broker (started: Mon Nov 2 17:10:36 CET 2015)
  • gnu messages sent to broker (started: Mon Nov 2 17:17:12 CET 2015)
  • loading all gnu tarballs
Nov 2 2015, 5:18 PM · Directory loader, Tarball loader
ardumont created T178: Replay loading mirror gnu and old-gnu from scratch.
Nov 2 2015, 5:06 PM · Directory loader, Tarball loader
ardumont removed a parent task for T177: Investigate `Not implemented 'compression type 6 implode'`: T90: swh-loader-tar - Load gnu tarballs in swh-storage.
Nov 2 2015, 5:04 PM · Directory loader, Tarball loader
ardumont removed a subtask for T90: swh-loader-tar - Load gnu tarballs in swh-storage: T177: Investigate `Not implemented 'compression type 6 implode'`.
Nov 2 2015, 5:04 PM · Tarball loader, Directory loader
ardumont closed T175: Redeploy and trigger sublist of tarballs impacted by errors as Resolved.
Nov 2 2015, 5:03 PM · Directory loader, Tarball loader
ardumont closed T175: Redeploy and trigger sublist of tarballs impacted by errors, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Nov 2 2015, 5:03 PM · Tarball loader, Directory loader
ardumont closed T176: Fix `Permission denied: write to /usr/local/lib/gcl-3...` as Resolved.
Nov 2 2015, 4:17 PM · Directory loader, Tarball loader
ardumont closed T176: Fix `Permission denied: write to /usr/local/lib/gcl-3...` , a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Nov 2 2015, 4:17 PM · Tarball loader, Directory loader
ardumont added a comment to T176: Fix `Permission denied: write to /usr/local/lib/gcl-3...` .

9a3349e fixes it

Nov 2 2015, 4:17 PM · Directory loader, Tarball loader
ardumont created T177: Investigate `Not implemented 'compression type 6 implode'`.
Nov 2 2015, 11:36 AM · Directory loader, Tarball loader
ardumont created T176: Fix `Permission denied: write to /usr/local/lib/gcl-3...` .
Nov 2 2015, 11:34 AM · Directory loader, Tarball loader
ardumont closed T174: Analyse and fix last errors as Resolved.
Nov 2 2015, 11:34 AM · Directory loader, Tarball loader
ardumont closed T174: Analyse and fix last errors, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Nov 2 2015, 11:34 AM · Tarball loader, Directory loader
ardumont added a comment to T174: Analyse and fix last errors.
  • Do not skip tarball containing empty upper root directory
  • Fix broken link size and data computation
Nov 2 2015, 11:21 AM · Directory loader, Tarball loader
zack placed T29: evaluate conffile/argparse Python module up for grabs.
Nov 2 2015, 10:52 AM · Core & foundations
ardumont raised the priority of T174: Analyse and fix last errors from Normal to High.
Nov 2 2015, 10:48 AM · Directory loader, Tarball loader
ardumont created T175: Redeploy and trigger sublist of tarballs impacted by errors.
Nov 2 2015, 10:43 AM · Directory loader, Tarball loader
ardumont created T174: Analyse and fix last errors.
Nov 2 2015, 10:43 AM · Directory loader, Tarball loader
ardumont closed T155: Deploy and load tarballs from worker01 as Resolved.
Nov 2 2015, 10:42 AM · Directory loader, Tarball loader
ardumont closed T155: Deploy and load tarballs from worker01, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Nov 2 2015, 10:42 AM · Tarball loader, Directory loader

Oct 30 2015

ardumont closed T165: web app initialized twice(?) as Resolved by committing rDWAPPSffde8f3fac08: Fix duplicate print statement error at dev start-up.
Oct 30 2015, 4:28 PM · Web app
ardumont added a comment to T165: web app initialized twice(?).

Ok, so finally, bad use of logging and print statement.

Oct 30 2015, 4:27 PM · Web app
ardumont added a comment to T165: web app initialized twice(?).

Also, if i let the debug flag but deactivate the reload behavior (use_reloader=False to app.run call)...

Oct 30 2015, 4:12 PM · Web app
ardumont added a comment to T165: web app initialized twice(?).

I think that, since it's in debug mode (automatically refresh when a change happens), something might errorneously trick the app to reload itself...

Oct 30 2015, 3:33 PM · Web app
ardumont added a comment to T165: web app initialized twice(?).

Strange behavior indeed.

Oct 30 2015, 3:26 PM · Web app
ardumont added a comment to T155: Deploy and load tarballs from worker01.
  • random archive message generation from producer
  • reading content optimization on swh-loader-dir
  • db cleaned up
Oct 30 2015, 2:47 PM · Directory loader, Tarball loader
ardumont closed T170: Random order in tarball messages computation as Resolved.
Oct 30 2015, 12:07 PM · Directory loader, Tarball loader
ardumont closed T170: Random order in tarball messages computation, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 30 2015, 12:07 PM · Tarball loader, Directory loader
ardumont closed T171: Fix content loading only when needed as Resolved.
Oct 30 2015, 12:06 PM · Directory loader
ardumont closed T171: Fix content loading only when needed, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 30 2015, 12:06 PM · Tarball loader, Directory loader
ardumont created T171: Fix content loading only when needed.
Oct 30 2015, 12:02 PM · Directory loader
ardumont created T170: Random order in tarball messages computation.
Oct 30 2015, 12:02 PM · Directory loader, Tarball loader

Oct 29 2015

ardumont added a comment to T155: Deploy and load tarballs from worker01.

messages sent:

  • old-gnu
  • gnu sent
Oct 29 2015, 3:48 PM · Directory loader, Tarball loader
ardumont closed T156: Package latest swh-storage as Resolved.
Oct 29 2015, 2:10 PM · Tarball loader, Directory loader
ardumont closed T156: Package latest swh-storage, a subtask of T155: Deploy and load tarballs from worker01, as Resolved.
Oct 29 2015, 2:10 PM · Directory loader, Tarball loader
ardumont closed T157: Package latest swh-loader-dir as Resolved.
Oct 29 2015, 10:51 AM · Directory loader, Tarball loader
ardumont closed T157: Package latest swh-loader-dir, a subtask of T155: Deploy and load tarballs from worker01, as Resolved.
Oct 29 2015, 10:51 AM · Directory loader, Tarball loader
ardumont closed T158: Package latest swh-loader-tar, a subtask of T155: Deploy and load tarballs from worker01, as Resolved.
Oct 29 2015, 10:46 AM · Directory loader, Tarball loader
ardumont closed T158: Package latest swh-loader-tar as Resolved.
Oct 29 2015, 10:46 AM · Tarball loader, Directory loader
zack closed T36: performance estimation: how long will it take to git-bulk-load all the GitHub repos we have as Resolved.
Oct 29 2015, 10:04 AM · Git loader

Oct 28 2015

zack added a comment to T166: functional tests for the web UI, with actual data.

FWIW, my preference is for (2), i.e., object mocking.

Oct 28 2015, 7:01 PM · Web app
zack created T166: functional tests for the web UI, with actual data.
Oct 28 2015, 7:01 PM · Web app
zack renamed T48: upload-based content search from Upload a file, computes its sha1 hash and sha256 hash and check its existence in swh's storage to upload-based content search.
Oct 28 2015, 6:58 PM · Web app
zack created T165: web app initialized twice(?).
Oct 28 2015, 6:53 PM · Web app
olasd added a comment to T119: snapshot.debian.org producer.

We should likely do the snapshot.d.o import in several passes:

Oct 28 2015, 6:47 PM · Origin-Debian, Debian loader
olasd added a parent task for T163: Add entry points to create / request entities in swh.storage: T164: Port GitHub lister to swh.storage instead of sqlalchemy.
Oct 28 2015, 6:42 PM · Storage manager
olasd added a subtask for T164: Port GitHub lister to swh.storage instead of sqlalchemy: T163: Add entry points to create / request entities in swh.storage.
Oct 28 2015, 6:42 PM · GitHub lister
olasd created T164: Port GitHub lister to swh.storage instead of sqlalchemy.
Oct 28 2015, 6:42 PM · GitHub lister
olasd created T163: Add entry points to create / request entities in swh.storage.
Oct 28 2015, 6:41 PM · Storage manager
olasd added a comment to T62: DB schema: add directory→tarball provenance information.

We're settling on a new table:

Oct 28 2015, 5:31 PM · Storage manager
ardumont removed a parent task for T62: DB schema: add directory→tarball provenance information: T90: swh-loader-tar - Load gnu tarballs in swh-storage.
Oct 28 2015, 4:58 PM · Storage manager
ardumont removed a subtask for T90: swh-loader-tar - Load gnu tarballs in swh-storage: T62: DB schema: add directory→tarball provenance information.
Oct 28 2015, 4:58 PM · Tarball loader, Directory loader
olasd added a comment to T121: Populate fetch_history.

fetch_history is now getting populated by swh.loader.git.

Oct 28 2015, 3:32 PM · Storage manager
ardumont created T158: Package latest swh-loader-tar.
Oct 28 2015, 12:07 PM · Tarball loader, Directory loader
ardumont added a parent task for T157: Package latest swh-loader-dir: T155: Deploy and load tarballs from worker01.
Oct 28 2015, 12:07 PM · Directory loader, Tarball loader
ardumont added a subtask for T155: Deploy and load tarballs from worker01: T157: Package latest swh-loader-dir.
Oct 28 2015, 12:07 PM · Directory loader, Tarball loader
ardumont removed a parent task for T157: Package latest swh-loader-dir: T156: Package latest swh-storage.
Oct 28 2015, 12:07 PM · Directory loader, Tarball loader
ardumont removed a subtask for T156: Package latest swh-storage: T157: Package latest swh-loader-dir.
Oct 28 2015, 12:07 PM · Tarball loader, Directory loader
ardumont created T157: Package latest swh-loader-dir.
Oct 28 2015, 12:06 PM · Directory loader, Tarball loader
ardumont created T156: Package latest swh-storage.
Oct 28 2015, 12:06 PM · Tarball loader, Directory loader
ardumont created T155: Deploy and load tarballs from worker01.
Oct 28 2015, 12:06 PM · Directory loader, Tarball loader
ardumont closed T142: reload missing tarballs, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Invalid.
Oct 28 2015, 12:05 PM · Tarball loader, Directory loader
ardumont closed T142: reload missing tarballs as Invalid.
Oct 28 2015, 12:05 PM · Directory loader, Tarball loader
ardumont closed T154: Send tarball checksums along the revision, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 28 2015, 12:04 PM · Tarball loader, Directory loader
ardumont closed T154: Send tarball checksums along the revision as Resolved.
Oct 28 2015, 12:04 PM · Directory loader, Tarball loader
ardumont created T154: Send tarball checksums along the revision.
Oct 28 2015, 11:30 AM · Directory loader, Tarball loader
ardumont closed T22: DB schema: add metadata field to revision table as Resolved by committing rDSTOba31ca32cb01: Add metadata field to revision table.
Oct 28 2015, 11:29 AM · Storage manager
ardumont closed T22: DB schema: add metadata field to revision table, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 28 2015, 11:29 AM · Tarball loader, Directory loader

Oct 27 2015

ardumont claimed T22: DB schema: add metadata field to revision table.
Oct 27 2015, 5:25 PM · Storage manager
ardumont added a parent task for T22: DB schema: add metadata field to revision table: T90: swh-loader-tar - Load gnu tarballs in swh-storage.
Oct 27 2015, 4:03 PM · Storage manager
ardumont added a parent task for T148: Populate fetch_history: T90: swh-loader-tar - Load gnu tarballs in swh-storage.
Oct 27 2015, 4:03 PM · Tarball loader
ardumont added subtasks for T90: swh-loader-tar - Load gnu tarballs in swh-storage: T149: Populate fetch_history table, T148: Populate fetch_history, T22: DB schema: add metadata field to revision table.
Oct 27 2015, 4:03 PM · Tarball loader, Directory loader
ardumont added a parent task for T149: Populate fetch_history table: T90: swh-loader-tar - Load gnu tarballs in swh-storage.
Oct 27 2015, 4:03 PM · Directory loader
ardumont closed T148: Populate fetch_history as Resolved by committing rDLDTARa370bb45254a: Refactor - Create an actual tarloader.
Oct 27 2015, 2:47 PM · Tarball loader
ardumont closed T148: Populate fetch_history, a subtask of T121: Populate fetch_history, as Resolved.
Oct 27 2015, 2:47 PM · Storage manager
olasd closed T30: port ghlister to swh.* namespace and conventions as Resolved by committing rDLScabf83613c76: ghlister: move to swh.lister.github as per swh convention.
Oct 27 2015, 2:38 PM · GitHub lister
olasd closed T30: port ghlister to swh.* namespace and conventions, a subtask of T37: port ghlister to swh task interface - daily updates, as Resolved.
Oct 27 2015, 2:38 PM · GitHub lister
olasd closed T30: port ghlister to swh.* namespace and conventions, a subtask of T38: port ghlister to swh task interface - list all / catch up, as Resolved.
Oct 27 2015, 2:38 PM · GitHub lister