Page MenuHomeSoftware Heritage
Feed Advanced Search

Oct 27 2015

zack placed T38: port ghlister to swh task interface - list all / catch up up for grabs.
Oct 27 2015, 2:04 PM · GitHub lister
zack placed T37: port ghlister to swh task interface - daily updates up for grabs.
Oct 27 2015, 2:03 PM · GitHub lister
zack placed T30: port ghlister to swh.* namespace and conventions up for grabs.
Oct 27 2015, 2:03 PM · GitHub lister
ardumont closed T149: Populate fetch_history table as Resolved by committing rDLDDIRf2240d1fc22f: Populate fetch history when loading a directory.
Oct 27 2015, 1:14 PM · Directory loader
ardumont closed T149: Populate fetch_history table, a subtask of T121: Populate fetch_history, as Resolved.
Oct 27 2015, 1:14 PM · Storage manager
ardumont removed a parent task for T149: Populate fetch_history table: T148: Populate fetch_history.
Oct 27 2015, 10:38 AM · Directory loader
ardumont removed a subtask for T148: Populate fetch_history: T149: Populate fetch_history table.
Oct 27 2015, 10:38 AM · Tarball loader
ardumont renamed T149: Populate fetch_history table from Populate optionally fetch_history table (default to true but overridable) to Populate fetch_history table.
Oct 27 2015, 10:38 AM · Directory loader
ardumont updated the task description for T149: Populate fetch_history table.
Oct 27 2015, 10:37 AM · Directory loader
ardumont added a parent task for T149: Populate fetch_history table: T148: Populate fetch_history.
Oct 27 2015, 10:11 AM · Directory loader
ardumont added a subtask for T148: Populate fetch_history: T149: Populate fetch_history table.
Oct 27 2015, 10:11 AM · Tarball loader
ardumont removed a subtask for T149: Populate fetch_history table: T148: Populate fetch_history.
Oct 27 2015, 10:11 AM · Directory loader
ardumont removed a parent task for T148: Populate fetch_history: T149: Populate fetch_history table.
Oct 27 2015, 10:11 AM · Tarball loader
ardumont added a parent task for T148: Populate fetch_history: T149: Populate fetch_history table.
Oct 27 2015, 10:11 AM · Tarball loader
ardumont added a subtask for T149: Populate fetch_history table: T148: Populate fetch_history.
Oct 27 2015, 10:11 AM · Directory loader
ardumont removed a subtask for T149: Populate fetch_history table: T148: Populate fetch_history.
Oct 27 2015, 10:10 AM · Directory loader
ardumont removed a parent task for T148: Populate fetch_history: T149: Populate fetch_history table.
Oct 27 2015, 10:10 AM · Tarball loader
ardumont added a parent task for T148: Populate fetch_history: T149: Populate fetch_history table.
Oct 27 2015, 10:08 AM · Tarball loader
ardumont added a subtask for T149: Populate fetch_history table: T148: Populate fetch_history.
Oct 27 2015, 10:08 AM · Directory loader
ardumont renamed T149: Populate fetch_history table from Use optionally fetch_history table (default to true but overridable) to Populate optionally fetch_history table (default to true but overridable).
Oct 27 2015, 10:07 AM · Directory loader
ardumont renamed T148: Populate fetch_history from Use fetch_history table to Populate fetch_history.
Oct 27 2015, 10:06 AM · Tarball loader
ardumont renamed T148: Populate fetch_history from swh-loader-tar - Use fetch history to Use fetch_history table.
Oct 27 2015, 10:05 AM · Tarball loader
ardumont created T149: Populate fetch_history table.
Oct 27 2015, 10:04 AM · Directory loader
ardumont created T148: Populate fetch_history.
Oct 27 2015, 10:03 AM · Tarball loader

Oct 26 2015

ardumont closed T147: Migrate code to use latest schema improvments from T3 as Invalid.
Oct 26 2015, 4:59 PM · Directory loader, Tarball loader
ardumont closed T147: Migrate code to use latest schema improvments from T3, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Invalid.
Oct 26 2015, 4:59 PM · Tarball loader, Directory loader
ardumont created T147: Migrate code to use latest schema improvments from T3.
Oct 26 2015, 4:56 PM · Directory loader, Tarball loader
olasd closed T3: Discuss the project <-> origin mapping as Resolved by committing rDSTO5c2a6fc2dc6b: sql: update schema of organizations and projects.
Oct 26 2015, 4:26 PM · Storage manager
olasd added a comment to T121: Populate fetch_history.

rDSTO04bb5cb is a first approach for this.

Oct 26 2015, 4:24 PM · Storage manager
ardumont added a subtask for T90: swh-loader-tar - Load gnu tarballs in swh-storage: T62: DB schema: add directory→tarball provenance information.
Oct 26 2015, 3:25 PM · Tarball loader, Directory loader
ardumont added a parent task for T62: DB schema: add directory→tarball provenance information: T90: swh-loader-tar - Load gnu tarballs in swh-storage.
Oct 26 2015, 3:25 PM · Storage manager
ardumont closed T140: analysis errors on first loading batch, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 26 2015, 3:20 PM · Tarball loader, Directory loader
ardumont closed T140: analysis errors on first loading batch as Resolved.
Oct 26 2015, 3:20 PM · Directory loader, Tarball loader
ardumont closed T141: compute missing tarballs list for reloading the ones missing, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 26 2015, 3:19 PM · Tarball loader, Directory loader
ardumont closed T141: compute missing tarballs list for reloading the ones missing as Resolved.
Oct 26 2015, 3:19 PM · Directory loader, Tarball loader
ardumont closed T143: understand why errors does not appear in log db as Resolved.
Oct 26 2015, 3:19 PM · Directory loader, Tarball loader
ardumont closed T143: understand why errors does not appear in log db, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 26 2015, 3:19 PM · Tarball loader, Directory loader
ardumont added a comment to T143: understand why errors does not appear in log db.

They do, i just need to:

  • fix the typo in logger dir (done)
  • look at the right column jsonb data
Oct 26 2015, 3:19 PM · Directory loader, Tarball loader
ardumont closed T136: Load tarballs in a production-like swh-storage instance on worker01 as Resolved.
Oct 26 2015, 12:33 PM · Directory loader, Tarball loader
ardumont closed T136: Load tarballs in a production-like swh-storage instance on worker01, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 26 2015, 12:33 PM · Tarball loader, Directory loader
ardumont created T143: understand why errors does not appear in log db.
Oct 26 2015, 12:33 PM · Directory loader, Tarball loader
ardumont created T142: reload missing tarballs.
Oct 26 2015, 12:30 PM · Directory loader, Tarball loader
ardumont created T141: compute missing tarballs list for reloading the ones missing.
Oct 26 2015, 12:30 PM · Directory loader, Tarball loader
ardumont created T140: analysis errors on first loading batch.
Oct 26 2015, 12:30 PM · Directory loader, Tarball loader

Oct 23 2015

ardumont added a comment to T136: Load tarballs in a production-like swh-storage instance on worker01.
  • old-gnu messages loaded in broker
  • loading gnu messages in broker

in progress...

Oct 23 2015, 5:22 PM · Directory loader, Tarball loader
ardumont created T136: Load tarballs in a production-like swh-storage instance on worker01.
Oct 23 2015, 5:22 PM · Directory loader, Tarball loader
ardumont renamed T90: swh-loader-tar - Load gnu tarballs in swh-storage from swh-loader-tar - Load a tarball in swh-storage to swh-loader-tar - Load tarballs in swh-storage.
Oct 23 2015, 5:21 PM · Tarball loader, Directory loader
ardumont closed T106: Test on worker01: producing and consuming one swh_loader_tar message, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 5:21 PM · Tarball loader, Directory loader
ardumont closed T106: Test on worker01: producing and consuming one swh_loader_tar message as Resolved.
Oct 23 2015, 5:21 PM · Tarball loader, Directory loader
olasd added a comment to T62: DB schema: add directory→tarball provenance information.

A few proposals:

Oct 23 2015, 3:12 PM · Storage manager
ardumont renamed T106: Test on worker01: producing and consuming one swh_loader_tar message from Test on worker01 producing and consuming one swh_loader_tar message is ok to Test on worker01: producing and consuming one swh_loader_tar message.
Oct 23 2015, 2:30 PM · Tarball loader, Directory loader
olasd added a comment to T3: Discuss the project <-> origin mapping.

@zack points out that organization does not feel like the right term anymore.

Oct 23 2015, 2:22 PM · Storage manager
ardumont closed T105: make worker01 stop consuming swh_loader_git messages, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 12:07 PM · Tarball loader, Directory loader
ardumont closed T105: make worker01 stop consuming swh_loader_git messages as Resolved.
Oct 23 2015, 12:07 PM · Tarball loader, Directory loader
ardumont added a project to T105: make worker01 stop consuming swh_loader_git messages: Tarball loader.
Oct 23 2015, 12:01 PM · Tarball loader, Directory loader
ardumont added a project to T106: Test on worker01: producing and consuming one swh_loader_tar message: Tarball loader.
Oct 23 2015, 12:01 PM · Tarball loader, Directory loader
ardumont closed T127: release swh-loader-tar, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 12:01 PM · Tarball loader, Directory loader
ardumont closed T127: release swh-loader-tar as Resolved.
Oct 23 2015, 12:01 PM · Tarball loader, Directory loader
ardumont added a project to T90: swh-loader-tar - Load gnu tarballs in swh-storage: Tarball loader.
Oct 23 2015, 12:00 PM · Tarball loader, Directory loader
ardumont added a project to T127: release swh-loader-tar: Tarball loader.
Oct 23 2015, 11:59 AM · Tarball loader, Directory loader
ardumont closed T104: release swh-loader-dir, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 11:44 AM · Tarball loader, Directory loader
ardumont renamed T104: release swh-loader-dir from release of swh-loader-dir when ready to release swh-loader-dir.
Oct 23 2015, 11:44 AM · Directory loader
olasd added a comment to T3: Discuss the project <-> origin mapping.
In T3#1275, @olasd wrote:
  • GitHub
    • GitHub git hosting -> GitHub lister, generates suborganizations for "GitHub organizations"
      • hylang -> no specific lister, "shallow" organization
      • zacchiro
      • olasd
      • ...
    • GitHub asset hosting -> T17
Oct 23 2015, 11:43 AM · Storage manager
ardumont closed T126: release swh-core, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 11:20 AM · Tarball loader, Directory loader
ardumont closed T126: release swh-core as Resolved.
Oct 23 2015, 11:20 AM · Directory loader
ardumont created T127: release swh-loader-tar.
Oct 23 2015, 11:19 AM · Tarball loader, Directory loader
ardumont created T126: release swh-core.
Oct 23 2015, 11:19 AM · Directory loader
ardumont closed T98: Produce list of tarballs message as Resolved.
Oct 23 2015, 10:53 AM · Directory loader
ardumont closed T125: Permits loading of tarball subsets from a file as Resolved.
Oct 23 2015, 10:53 AM · Directory loader
ardumont closed T125: Permits loading of tarball subsets from a file, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 10:53 AM · Tarball loader, Directory loader
ardumont closed T124: Tryouts run on gnu to complete the tarball listing exhaustiveness as Resolved.
Oct 23 2015, 10:53 AM · Directory loader
ardumont closed T124: Tryouts run on gnu to complete the tarball listing exhaustiveness, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 10:53 AM · Tarball loader, Directory loader
ardumont closed T123: Tryouts run on old-gnu to complete the tarball listing exhaustiveness as Resolved.
Oct 23 2015, 10:52 AM · Directory loader
ardumont closed T123: Tryouts run on old-gnu to complete the tarball listing exhaustiveness, a subtask of T90: swh-loader-tar - Load gnu tarballs in swh-storage, as Resolved.
Oct 23 2015, 10:52 AM · Tarball loader, Directory loader
ardumont created T125: Permits loading of tarball subsets from a file.
Oct 23 2015, 10:51 AM · Directory loader
ardumont created T124: Tryouts run on gnu to complete the tarball listing exhaustiveness.
Oct 23 2015, 10:50 AM · Directory loader
ardumont created T123: Tryouts run on old-gnu to complete the tarball listing exhaustiveness.
Oct 23 2015, 10:50 AM · Directory loader
olasd added a comment to T3: Discuss the project <-> origin mapping.

I've had some thoughts about this problem, and here are my propositions.

Oct 23 2015, 10:01 AM · Storage manager

Oct 22 2015

olasd created T121: Populate fetch_history.
Oct 22 2015, 2:31 PM · Storage manager
olasd created T120: regular debian archive producer.
Oct 22 2015, 12:33 PM · Debian loader, Origin-Debian
olasd created T119: snapshot.debian.org producer.
Oct 22 2015, 12:32 PM · Origin-Debian, Debian loader
olasd created T118: Create swh-loader-debian.
Oct 22 2015, 12:30 PM · Debian loader, Origin-Debian
zack closed T117: factor out common code from git/dir loader as Invalid.

duplicate of T89

Oct 22 2015, 10:54 AM · Directory loader, Git loader
zack renamed T89: refactor common behavior between git loader and dir loader from Extract common behavior between swh-loader-git and swh-loader-dir ~> swh-loader-core to refactor common behavior between git loader and dir loader.
Oct 22 2015, 10:53 AM · Git loader, Directory loader
zack created T117: factor out common code from git/dir loader.
Oct 22 2015, 10:50 AM · Directory loader, Git loader

Oct 21 2015

olasd added a comment to T116: Add storage endpoints to help with repository updates.

Inspiration :

Oct 21 2015, 6:05 PM · Storage manager
olasd added a subtask for T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too): T116: Add storage endpoints to help with repository updates.
Oct 21 2015, 5:59 PM · Git cloner, Git loader
olasd added a parent task for T116: Add storage endpoints to help with repository updates: T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too).
Oct 21 2015, 5:59 PM · Storage manager
olasd removed a parent task for T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too): T116: Add storage endpoints to help with repository updates.
Oct 21 2015, 5:58 PM · Git cloner, Git loader
olasd removed a subtask for T116: Add storage endpoints to help with repository updates: T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too).
Oct 21 2015, 5:58 PM · Storage manager
olasd added a parent task for T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too): T116: Add storage endpoints to help with repository updates.
Oct 21 2015, 5:58 PM · Git cloner, Git loader
olasd added a subtask for T116: Add storage endpoints to help with repository updates: T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too).
Oct 21 2015, 5:58 PM · Storage manager
olasd triaged T116: Add storage endpoints to help with repository updates as Normal priority.
Oct 21 2015, 5:58 PM · Storage manager
olasd updated the task description for T116: Add storage endpoints to help with repository updates.
Oct 21 2015, 5:57 PM · Storage manager
olasd created T116: Add storage endpoints to help with repository updates.
Oct 21 2015, 5:56 PM · Storage manager
olasd added a comment to T62: DB schema: add directory→tarball provenance information.

We should make sure our abstraction works for things where the original artifact has several files (e.g. a debian source package)

Oct 21 2015, 5:38 PM · Storage manager

Oct 20 2015

olasd added a comment to T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too).

Started playing with dulwich's git smart protocol client.

Oct 20 2015, 7:18 PM · Git cloner, Git loader
zack lowered the priority of T71: update database/storage size estimation using current content of SWH DB from High to Normal.
Oct 20 2015, 10:59 AM · Restricted Project

Oct 19 2015

olasd added a comment to T111: Performance analysis of read queries.

shortcut query:

explain (analyze,buffers,verbose)     select origin, branch, revision                                                                                                          
    from occurrence_history as occ_hist
    where occ_hist.revision = '\x1140811ebed3b33b496a3aed32ed16b068ebffab'
    order by upper(occ_hist.validity)  -- TODO filter by authority?
    limit 1;
Oct 19 2015, 2:34 PM · Storage manager
olasd added a comment to T111: Performance analysis of read queries.

swh_content_find_directory query :

Oct 19 2015, 2:28 PM · Storage manager