Page MenuHomeSoftware Heritage
Feed Advanced Search

Feb 19 2016

olasd changed the status of T52: swh-cron: manifest-based scheduler for recurring tasks from Open to Work in Progress.

An implementation of this is now available in rDSCH.

Feb 19 2016, 12:52 PM

Feb 17 2016

olasd added a comment to T75: Check integrity of directories, revisions, and releases.
In T75#3503, @olasd wrote:

It currently breaks on *completely* empty messages, but the patch seems fairly simple.

Feb 17 2016, 12:18 PM · Archive content, Restricted Project

Feb 11 2016

olasd added a comment to T52: swh-cron: manifest-based scheduler for recurring tasks.

We did some f2f thinking about this, concentrating on the "origin update" part of the mechanism. The shortcoming of our mechanism is that it's completely specific to updating our origins, and we can do something better...

Feb 11 2016, 9:47 PM

Feb 9 2016

olasd closed T68: support for git tags that point to arbitrary git objects, instead of revisions as Resolved.

This is now supported.

Feb 9 2016, 2:26 PM · Git loader
ardumont renamed T308: retrieve content from s3 and store it in SWH storage from swh-loader-antelink bootstrap - Retrieve content from s3 and store inside swh-storage to Retrieve content from s3 and store inside swh-storage.
Feb 9 2016, 12:29 PM · Storage manager, Antelink loader
ardumont updated the task description for T309: Delete duplicated antelink/antepedia content from s3.
Feb 9 2016, 12:21 PM · Antelink loader
ardumont updated the task description for T308: retrieve content from s3 and store it in SWH storage.
Feb 9 2016, 12:20 PM · Storage manager, Antelink loader
ardumont added a project to T310: import antelink metadata: Antelink loader.
Feb 9 2016, 12:20 PM · Antelink loader
ardumont created T310: import antelink metadata.
Feb 9 2016, 12:19 PM · Antelink loader
ardumont added a project to T308: retrieve content from s3 and store it in SWH storage: Storage manager.
Feb 9 2016, 12:18 PM · Storage manager, Antelink loader
ardumont added a project to T309: Delete duplicated antelink/antepedia content from s3: Storage manager.
Feb 9 2016, 12:18 PM · Antelink loader
ardumont added projects to T309: Delete duplicated antelink/antepedia content from s3: Antelink loader, Developers.
Feb 9 2016, 12:18 PM · Antelink loader
ardumont added projects to T308: retrieve content from s3 and store it in SWH storage: Antelink loader, Developers.
Feb 9 2016, 12:18 PM · Storage manager, Antelink loader
ardumont created T309: Delete duplicated antelink/antepedia content from s3.
Feb 9 2016, 12:01 PM · Antelink loader
ardumont created T308: retrieve content from s3 and store it in SWH storage.
Feb 9 2016, 12:00 PM · Storage manager, Antelink loader

Feb 8 2016

zack closed T71: update database/storage size estimation using current content of SWH DB as Resolved.

No need of making estimates anymore, as we know know.
After GitHub + Debian snapshot + GNU we have the following:

Feb 8 2016, 4:30 PM · Restricted Project

Feb 4 2016

zack added a project to T240: content archiver: Storage manager.
Feb 4 2016, 3:00 PM · Storage manager
zack created T304: content integrity checker.
Feb 4 2016, 3:00 PM · Storage manager
olasd closed T7: backup: object storage — 2nd copy after first large batch import as Resolved.

We now have a backup of all the contents that were stored on uffizi at the end of our first batch import.

Feb 4 2016, 2:38 PM

Jan 29 2016

ardumont closed T299: Return the person's identifier along the person's data as Resolved by committing rDWAPPS4f57e6862526: Returns person's identifier on api + Hide person's emails in views endpoint.
Jan 29 2016, 5:43 PM · Storage manager, Web app
ardumont closed T300: Hide person's email on revision/release/person view as Resolved by committing rDWAPPS4f57e6862526: Returns person's identifier on api + Hide person's emails in views endpoint.
Jan 29 2016, 5:43 PM · Web app
ardumont claimed T299: Return the person's identifier along the person's data.
Jan 29 2016, 3:25 PM · Storage manager, Web app
ardumont closed T303: Fix bug in production as Resolved by committing rDSTOfe919314cbae: sql/upgrades/049: 049->050 schema change - Clean up obsolete functions.
Jan 29 2016, 3:23 PM
ardumont created T303: Fix bug in production.
Jan 29 2016, 3:21 PM
ardumont closed T296: Try to decode the content's raw data and fail gracefully as Resolved by committing rDWAPPS1c2f64e21ff4: Try to decode the content's raw data and fail gracefully.
Jan 29 2016, 2:27 PM · Web app
ardumont closed T295: Update /browse/directory/<path>/ to show content when path resolves to a content as Resolved by committing rDWAPPS3ec60a52643b: Unify /directory api to Display content's raw data when path resolves to a file.
Jan 29 2016, 12:39 PM · Web app

Jan 28 2016

olasd created T302: swh-loader-tar origin validities are the current time instead of the mirroring time.
Jan 28 2016, 5:31 PM · Tarball loader
olasd closed T72: occurrence history table: use an array of timestamps instead of intervals as Resolved.

This schema change is now done in production.

Jan 28 2016, 5:23 PM · Storage manager
olasd closed T72: occurrence history table: use an array of timestamps instead of intervals, a subtask of T286: view /browse/origin/ORIGIN_ID/ - calendar-like UI, as Resolved.
Jan 28 2016, 5:23 PM · Web app
ardumont created T301: Improve date parsing error messages.
Jan 28 2016, 4:37 PM · Web app
ardumont added a project to T300: Hide person's email on revision/release/person view: Web app.
Jan 28 2016, 4:36 PM · Web app
ardumont created T300: Hide person's email on revision/release/person view.
Jan 28 2016, 4:36 PM · Web app
ardumont created T299: Return the person's identifier along the person's data.
Jan 28 2016, 4:32 PM · Storage manager, Web app
ardumont added a parent task for T298: Add /api/1/revision/<sha1_git>/raw/: T297: Try to decode the revision's message data and fail gracefully.
Jan 28 2016, 4:30 PM · Web app
ardumont added a subtask for T297: Try to decode the revision's message data and fail gracefully: T298: Add /api/1/revision/<sha1_git>/raw/.
Jan 28 2016, 4:30 PM · Web app
ardumont created T298: Add /api/1/revision/<sha1_git>/raw/.
Jan 28 2016, 4:30 PM · Web app
ardumont created T297: Try to decode the revision's message data and fail gracefully.
Jan 28 2016, 4:29 PM · Web app
ardumont created T296: Try to decode the content's raw data and fail gracefully.
Jan 28 2016, 4:28 PM · Web app
ardumont created T295: Update /browse/directory/<path>/ to show content when path resolves to a content.
Jan 28 2016, 4:23 PM · Web app
ardumont added a subtask for T286: view /browse/origin/ORIGIN_ID/ - calendar-like UI: T72: occurrence history table: use an array of timestamps instead of intervals.
Jan 28 2016, 4:21 PM · Web app
ardumont added a parent task for T72: occurrence history table: use an array of timestamps instead of intervals: T286: view /browse/origin/ORIGIN_ID/ - calendar-like UI.
Jan 28 2016, 4:21 PM · Storage manager
ardumont triaged T288: Open /api/1/revision/origin/<origin_id>/[branch/<branch_name>][ts/<ts>]/log/ as Normal priority.
Jan 28 2016, 4:21 PM · Web app
ardumont added a subtask for T294: Align /browse/directory/<sha1_git>/ template to other /directory/ apis: T293: Open /browse/directory/<sha1_git>/[<path>/] view.
Jan 28 2016, 4:20 PM · Web app
ardumont added a parent task for T293: Open /browse/directory/<sha1_git>/[<path>/] view: T294: Align /browse/directory/<sha1_git>/ template to other /directory/ apis.
Jan 28 2016, 4:20 PM · Web app
ardumont created T294: Align /browse/directory/<sha1_git>/ template to other /directory/ apis.
Jan 28 2016, 4:20 PM · Web app
ardumont created T293: Open /browse/directory/<sha1_git>/[<path>/] view.
Jan 28 2016, 4:18 PM · Web app
ardumont created T292: Open /api/1/directory/<sha1_git>/[<path>/] api.
Jan 28 2016, 4:17 PM · Web app
ardumont created T291: Remove default identifier on api/view.
Jan 28 2016, 4:13 PM · Web app
ardumont created T290: Open /api/1/revision/<sha1_git_root>/history/<sha1_git>/log/ api.
Jan 28 2016, 4:11 PM · Web app
ardumont created T289: Open /browse/revision/<sha1_git_root>/history/<sha1_git>/log/ view.
Jan 28 2016, 4:10 PM · Web app
ardumont created T288: Open /api/1/revision/origin/<origin_id>/[branch/<branch_name>][ts/<ts>]/log/.
Jan 28 2016, 4:09 PM · Web app
ardumont created T287: Open /browse/revision/origin/<origin_id>/[branch/<branch_name>][ts/<ts>]/log/ view.
Jan 28 2016, 4:09 PM · Web app
ardumont created T286: view /browse/origin/ORIGIN_ID/ - calendar-like UI.
Jan 28 2016, 4:07 PM · Web app

Jan 27 2016

ardumont renamed T271: Update clients on impacts + upgrade respective package dependencies from Update api conversion on impacted fields to Update clients on impacts + upgrade respective package dependencies.
Jan 27 2016, 4:51 PM · Git loader, Directory loader, Data Model, Web app
ardumont closed T270: TEXT → BYTEA conversion for columns that represent VCS tags/branches as Resolved by committing rDSTO78024bf20d6f: Deal with occurrence_history.branch, release.name as bytes.
Jan 27 2016, 3:16 PM · Language-SQL, Storage manager
ardumont closed T270: TEXT → BYTEA conversion for columns that represent VCS tags/branches, a subtask of T271: Update clients on impacts + upgrade respective package dependencies, as Resolved.
Jan 27 2016, 3:16 PM · Git loader, Directory loader, Data Model, Web app
ardumont claimed T270: TEXT → BYTEA conversion for columns that represent VCS tags/branches.
Jan 27 2016, 12:01 PM · Language-SQL, Storage manager
zack added a comment to T72: occurrence history table: use an array of timestamps instead of intervals.
In T72#3501, @olasd wrote:

full SQL code with the new schemata for origin_visit, occurrence_history and occurrence. Those three tables are implicitly relevant only for the "Software Heritage" authority.

Jan 27 2016, 10:40 AM · Storage manager

Jan 26 2016

zack added a comment to T75: Check integrity of directories, revisions, and releases.
In T75#3505, @olasd wrote:

Reading the dulwich code a bit further, it turns out that git commits can have more header attributes than we initally expected.

Jan 26 2016, 10:45 PM · Archive content, Restricted Project
olasd added a comment to T75: Check integrity of directories, revisions, and releases.

Reading the dulwich code a bit further, it turns out that git commits can have more header attributes than we initally expected.

Jan 26 2016, 4:26 PM · Archive content, Restricted Project
olasd added a comment to T75: Check integrity of directories, revisions, and releases.

Dulwich seems to handle some of those special cases just fine.

Jan 26 2016, 12:59 PM · Archive content, Restricted Project
olasd added a comment to T72: occurrence history table: use an array of timestamps instead of intervals.

full SQL code with the new schemata for origin_visit, occurrence_history and occurrence. Those three tables are implicitly relevant only for the "Software Heritage" authority.

Jan 26 2016, 12:16 PM · Storage manager
rdicosmo added a comment to T78: Unable to perfectly round-trip releases pointed at by occurrences.

Roberto Di Cosmo (via mobile/cell)
Le 26 janv. 2016 07:47, "olasd (Nicolas Dandrimont)" <
forge@softwareheritage.org> a écrit :

Jan 26 2016, 9:10 AM · Storage manager
olasd closed T78: Unable to perfectly round-trip releases pointed at by occurrences as Resolved.

This has now been deployed in swh.storage v0.0.30: occurrences and releases can now point to arbitrary objects.

Jan 26 2016, 7:46 AM · Storage manager
olasd closed T78: Unable to perfectly round-trip releases pointed at by occurrences, a subtask of T67: prototype: git clone from SWH, as Resolved.
Jan 26 2016, 7:46 AM · Vault, General

Jan 25 2016

ardumont closed T280: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/history/<SHA1>/directory/[<PATH>] view as Resolved by committing rDWAPPS0dc8a834876b: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>]….
Jan 25 2016, 4:53 PM · Web app
ardumont closed T285: Open /browse/revision/<sha1_git>/log/ as Resolved by committing rDWAPPS16f97f8d8e80: Open /browse/revision/<sha1_git>/log/.
Jan 25 2016, 3:53 PM · Web app
ardumont created T285: Open /browse/revision/<sha1_git>/log/.
Jan 25 2016, 3:52 PM · Web app
ardumont closed T281: Open /revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/directory/[<PATH>] view as Resolved by committing rDWAPPS43f9c827cd45: Open /revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/directory/….
Jan 25 2016, 2:04 PM · Web app
ardumont closed T278: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/ view as Resolved by committing rDWAPPSdb71b926e66a: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/ view.
Jan 25 2016, 10:23 AM · Web app
ardumont closed T279: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/history/<SHA1>/ view as Resolved by committing rDWAPPSb5dd1178cc2f: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>]….
Jan 25 2016, 10:23 AM · Web app

Jan 24 2016

zack renamed T283: investigate libvirt I/O slowdown from investigate libvirt I/O slow down to investigate libvirt I/O slowdown.
Jan 24 2016, 10:45 PM · System administration
zack added a comment to T284: investigate XFS tuning slowdown on uffizi w.r.t. banco.

Full benchmark data at https://intranet.softwareheritage.org/index.php?title=User:StefanoZacchiroli/Disk_array_benchmark

Jan 24 2016, 10:43 PM · System administration
zack created T284: investigate XFS tuning slowdown on uffizi w.r.t. banco.
Jan 24 2016, 10:43 PM · System administration
zack added a comment to T283: investigate libvirt I/O slowdown.

Full benchmark data are available at: https://intranet.softwareheritage.org/index.php?title=User:StefanoZacchiroli/Disk_array_benchmark

Jan 24 2016, 10:36 PM · System administration
zack added a comment to T283: investigate libvirt I/O slowdown.

For completeness, here are the slowdown benchmarks for prado's SSD disks (bottom line: the slow down seems to be present there too, but "only" of the order of 20% or so):

Jan 24 2016, 10:35 PM · System administration
zack created T283: investigate libvirt I/O slowdown.
Jan 24 2016, 10:32 PM · System administration
olasd closed T282: reduce icinga/nagios apt check frequency to 24h as Resolved.

Should be fixed.

Jan 24 2016, 5:44 PM

Jan 23 2016

zack created T282: reduce icinga/nagios apt check frequency to 24h.
Jan 23 2016, 7:04 PM
zack closed T274: investigate slow uffizi read performances from the object store as Resolved.
Jan 23 2016, 7:00 PM

Jan 22 2016

zack claimed T274: investigate slow uffizi read performances from the object store.

The main bottleneck turned out to be seek time, that for 1.6B files really adds up.

Jan 22 2016, 6:49 PM
zack changed the status of T7: backup: object storage — 2nd copy after first large batch import from Open to Work in Progress.

By looking at bonnie++ output and doing some math, we have concluded that transfer slowness is essentially dominated by seek time.

Jan 22 2016, 6:47 PM
zack removed a subtask for T7: backup: object storage — 2nd copy after first large batch import: T274: investigate slow uffizi read performances from the object store.
Jan 22 2016, 6:45 PM
zack removed a parent task for T274: investigate slow uffizi read performances from the object store: T7: backup: object storage — 2nd copy after first large batch import.
Jan 22 2016, 6:45 PM
olasd closed T272: Fix double-encoded release.name entries as Resolved.

This has now been done.

Jan 22 2016, 6:31 PM · Data Model, Storage manager
ardumont claimed T278: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/ view.
Jan 22 2016, 5:14 PM · Web app
ardumont closed T277: Open /browse/revision/<SHA1_GIT_ROOT>/history/<SHA1_GIT>/directory/[<PATH>] view as Resolved by committing rDWAPPSa4f574de5736: Open /browse/revision/<SHA1_GIT_ROOT>/history/<SHA1_GIT>/directory/[<PATH>] view.
Jan 22 2016, 5:09 PM · Web app
ardumont closed T276: Open /browse/revision/<sha1_git_root>/directory/<path>/ view as Resolved by committing rDWAPPS45720eb71a04: Open /browse/revision/<sha1_git_root>/directory/<path>/.
Jan 22 2016, 2:58 PM · Web app
olasd added a comment to T72: occurrence history table: use an array of timestamps instead of intervals.

Still thinking about this .

Jan 22 2016, 12:13 PM · Storage manager
ardumont closed T275: Open /browse/revision/<sha1_git_root>/history/<sha1_git>/ view as Resolved by committing rDWAPPS9fd582d85697: Open /browse/revision/<sha1_git_root>/history/<sha1_git>/.
Jan 22 2016, 11:52 AM · Web app
ardumont created T281: Open /revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/directory/[<PATH>] view.
Jan 22 2016, 11:51 AM · Web app
ardumont created T280: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/history/<SHA1>/directory/[<PATH>] view.
Jan 22 2016, 11:50 AM · Web app
ardumont created T279: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/history/<SHA1>/ view.
Jan 22 2016, 11:50 AM · Web app
ardumont created T278: Open /browse/revision/origin/<ORIG_ID>[/branch/<BRANCH>][/ts/<TIMESTAMP>]/ view.
Jan 22 2016, 11:49 AM · Web app
ardumont created T277: Open /browse/revision/<SHA1_GIT_ROOT>/history/<SHA1_GIT>/directory/[<PATH>] view.
Jan 22 2016, 11:48 AM · Web app
ardumont created T276: Open /browse/revision/<sha1_git_root>/directory/<path>/ view.
Jan 22 2016, 11:46 AM · Web app
ardumont created T275: Open /browse/revision/<sha1_git_root>/history/<sha1_git>/ view.
Jan 22 2016, 11:45 AM · Web app
ardumont changed the status of T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too) from Work in Progress to Open.
Jan 22 2016, 10:05 AM · Git cloner, Git loader
ardumont changed the status of T51: smart, all-in-one git cloner/loader/ (+ dealing with updates too), a subtask of T66: clone and load fork GitHub repositories, from Work in Progress to Open.
Jan 22 2016, 10:05 AM · Restricted Project, General
zack lowered the priority of T268: ontology - document on the wiki the data model terminology from High to Normal.
Jan 22 2016, 8:25 AM · General, Restricted Project