Page MenuHomeSoftware Heritage
Feed Advanced Search

Nov 3 2017

zack changed the visibility for F2912180: mark-as-missing.sql.
Nov 3 2017, 2:37 PM
zack changed the visibility for F2912179: rm-oversize-from-origin.
Nov 3 2017, 2:37 PM
zack changed the visibility for F2912178: import-oversize.
Nov 3 2017, 2:37 PM
zack changed the status of T809: move contents larger than current injection limit to separate object storage from Open to Work in Progress.

ok, here are the building blocks I've prepared to resolve this task, as a step by step recipe:

Nov 3 2017, 2:36 PM · Archive content

Nov 2 2017

zack closed T762: integrate lister tutorial into development documentation, a subtask of T509: Generate and publish Software Heritage Development Documentation, as Resolved.
Nov 2 2017, 5:54 PM · Documentation
zack closed T762: integrate lister tutorial into development documentation as Resolved by committing rDLSb8ad57fc05f4: docs: integrate swh-lister tutorial.
Nov 2 2017, 5:54 PM · Documentation
zack committed rDLSb8ad57fc05f4: docs: integrate swh-lister tutorial (authored by zack).
docs: integrate swh-lister tutorial
Nov 2 2017, 5:54 PM
zack committed rDDOC489b015a5e09: docs: integrate architecture page into the index (authored by zack).
docs: integrate architecture page into the index
Nov 2 2017, 11:10 AM
zack committed rDWAPPS09d740f9bdef: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDSTO477bd443be6b: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDVAU787cfd3431c8: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDSCH09b82c918f02: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDOBJSbf31f8d1f536: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDMOD94bd8dd5db6a: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLDTAR1d650ae81b59: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLDSVNf8afc6e2f73a: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLDGe7640dc55e25: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLDHG71fe4fce2b72: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLDDEB2f48abd90b1e: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLDDIR81fe06060d23: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLSfe4725d383ab: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDLDBASE4e3c0a7d1afc: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDCIDXf692f4fd8c02: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDDEP22b07b725d26: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDJNL8dfe5ae4b3c2: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDCOREbb1b36815b67: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM
zack committed rDARC7844ee8d0590: docs: add absolute anchor to documentation index (authored by zack).
docs: add absolute anchor to documentation index
Nov 2 2017, 11:09 AM

Oct 30 2017

zack added a project to T824: sync missing repositories to github: Phabricator.
Oct 30 2017, 1:41 PM · Phabricator, Documentation
zack updated the task description for T825: automatically sync the *list* of repositories to sync to github.
Oct 30 2017, 1:41 PM · Phabricator
zack updated the task description for T825: automatically sync the *list* of repositories to sync to github.
Oct 30 2017, 1:41 PM · Phabricator
zack created T825: automatically sync the *list* of repositories to sync to github.
Oct 30 2017, 1:40 PM · Phabricator
zack created T824: sync missing repositories to github.
Oct 30 2017, 1:39 PM · Phabricator, Documentation
zack committed rSPPROF1b46fe08229e: swh-puppet-master-deploy: port to new location of environments in /etc (authored by zack).
swh-puppet-master-deploy: port to new location of environments in /etc
Oct 30 2017, 1:31 PM
zack created P186 (An Untitled Masterwork).
Oct 30 2017, 1:26 PM
zack added a comment to T761: upgrade all machines to Debian Stretch.

For info: it looks like the fix for T755, which is now deployed on pergamon, requires the version of monitoring-plugins-basic that is on Debian Stretch; previous versions do not have the --only-critical flag.
So, until this is fixed, pending package upgrades on all workers have status "unknown". (Which isn't a big deal.)

Oct 30 2017, 10:53 AM · System administration

Oct 26 2017

zack changed the visibility for F2906089: mimetype-stats.sql.
Oct 26 2017, 12:37 PM
zack added a comment to T817: analyze bogus mimetype values in content_mimetype table.

FTR, the query I've used to generate the stats is:


(the encoding there is needed due to T818)

Oct 26 2017, 12:37 PM · Archive content, Indexer
zack created T817: analyze bogus mimetype values in content_mimetype table.
Oct 26 2017, 12:36 PM · Archive content, Indexer
zack changed the visibility for F2906080: mimetype-stats.txt.
Oct 26 2017, 12:32 PM
zack added a comment to T813: publications page on the website.

can we first check if we agree on the proposal above between us?
then, sure, @anlambert can get Fred in the loop before implementing this

Oct 26 2017, 11:14 AM · Website
zack created T813: publications page on the website.
Oct 26 2017, 10:14 AM · Website

Oct 25 2017

zack committed rMSLD78b94c8d8fbe: biblio module: fix broken markup (authored by zack).
biblio module: fix broken markup
Oct 25 2017, 6:42 PM
zack committed rMSLDab18e58cfef3: update FORCE11 slides for the software citation use case(s) (authored by zack).
update FORCE11 slides for the software citation use case(s)
Oct 25 2017, 6:42 PM
zack committed rMSLD2f3e97edc86f: principles-short: new module for 1-slide version of our principles (authored by zack).
principles-short: new module for 1-slide version of our principles
Oct 25 2017, 6:42 PM
zack committed rMSLD07b44b312d7b: better org-beamer markup for bibliography module (authored by zack).
better org-beamer markup for bibliography module
Oct 25 2017, 2:24 PM
zack committed rMSLD2f354368e83f: fix typo in FORCE11 example referencing directories (authored by zack).
fix typo in FORCE11 example referencing directories
Oct 25 2017, 2:24 PM
zack committed rMSLDb375e7872e3a: draft slides for FORCE11 WG meeting (authored by zack).
draft slides for FORCE11 WG meeting
Oct 25 2017, 2:01 PM
zack committed rMSLD01c4a9fe050f: common: new module for bibliography slides (authored by zack).
common: new module for bibliography slides
Oct 25 2017, 2:01 PM

Oct 22 2017

zack renamed T810: doc generation broken with sphinx < 1.5 from doc generation broken: "%s is not unknown HTTP header" to doc generation broken with sphinx < 1.5.
Oct 22 2017, 12:14 PM · Web app, Documentation
zack added a comment to T810: doc generation broken with sphinx < 1.5.

I did have sphinxcontrib-httpdomain-1.5.0 installed...
Removing it, i now reproduced the error:

Oct 22 2017, 12:14 PM · Web app, Documentation

Oct 21 2017

zack reassigned T810: doc generation broken with sphinx < 1.5 from ardumont to anlambert.

The initial sphinx output is misleading. Even though i don't reproduce the error, on my side (cf. P184), the sphinx message output seems to be scrambled when reaching swh-deposit.

Oct 21 2017, 6:35 PM · Web app, Documentation
zack committed rDSTO7e2e9a999a2d: docs: integrate postgres DB schema in dev docs (authored by zack).
docs: integrate postgres DB schema in dev docs
Oct 21 2017, 5:22 PM
zack committed rDSTOcc8a6187ca7f: sql doc: also generate SVG version of the schema chart (authored by zack).
sql doc: also generate SVG version of the schema chart
Oct 21 2017, 5:22 PM
zack committed rDSTO6ecf3d61db1d: doc: convert archiver blueprint to rst and link it from doc index (authored by zack).
doc: convert archiver blueprint to rst and link it from doc index
Oct 21 2017, 5:22 PM
zack changed the visibility for F2902947: sphinx-err-myex0h6h.log.
Oct 21 2017, 5:14 PM
zack added a comment to T810: doc generation broken with sphinx < 1.5.

here is the traceback

Oct 21 2017, 5:14 PM · Web app, Documentation
zack created T810: doc generation broken with sphinx < 1.5.
Oct 21 2017, 5:13 PM · Web app, Documentation
zack committed rDSTO8915caff4b81: db schema chart: add metadata and statistic clusters (authored by zack).
db schema chart: add metadata and statistic clusters
Oct 21 2017, 3:54 PM
zack committed rDSTO89b9ce4028ac: db schema chart: use more current titles for some clusters (authored by zack).
db schema chart: use more current titles for some clusters
Oct 21 2017, 3:54 PM
zack updated subscribers of T809: move contents larger than current injection limit to separate object storage.
Oct 21 2017, 1:00 PM · Archive content
zack added a comment to T809: move contents larger than current injection limit to separate object storage.

and here's the actual disk usage (!= compressed size)

/srv/softwareheritage/scratch/lists $ xzcat oversize-contents.txt.xz | while read id ; do du $(swh-ls-obj $id) ; done | cut -f 1 | paste -sd+ | bc
16847900696

those are KB, so the total disk usage is ~15.5 TB (not bad!)

Oct 21 2017, 12:59 PM · Archive content

Oct 20 2017

zack changed the status of T755: icinga notification spam: pending package upgrades from Open to Work in Progress.

D258 should take care of this, we'll see for real when it's deployed.

Oct 20 2017, 5:55 PM · System administration
zack committed rSPPROFf1f3a6a2d2d5: make icinga warn only about pending critical upgrades (authored by zack).
make icinga warn only about pending critical upgrades
Oct 20 2017, 5:55 PM
zack closed D258: make icinga warn only about pending critical upgrades.
Oct 20 2017, 5:55 PM
zack added a reviewer for D258: make icinga warn only about pending critical upgrades: System administrators.

Assuming it works, this should give quite a blow to T755.
Relevant doc is at https://www.icinga.com/docs/icinga2/latest/doc/10-icinga-template-library/#apt

Oct 20 2017, 5:50 PM
zack created D258: make icinga warn only about pending critical upgrades.
Oct 20 2017, 5:48 PM
zack added a comment to T809: move contents larger than current injection limit to separate object storage.

as a datapoint, at the time of writing the total (uncompressed) size occupied by content objects that are larger than our current limit is as follows:

Oct 20 2017, 5:39 PM · Archive content
zack created T809: move contents larger than current injection limit to separate object storage.
Oct 20 2017, 4:59 PM · Archive content

Oct 17 2017

zack added inline comments to D255: Adding codemeta.json file to content indexer.
Oct 17 2017, 3:48 PM
zack added a project to T808: phabricator lister: Phabricator forge.
Oct 17 2017, 3:46 PM · Easy hack, Phabricator forge
zack created Phabricator forge.
Oct 17 2017, 3:46 PM
zack renamed FusionForge Forge from Forge FusionForge to FusionForge Forge.
Oct 17 2017, 3:45 PM
zack added a subtask for T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs): T329: hg / mercurial loader.
Oct 17 2017, 3:44 PM · General
zack added a parent task for T329: hg / mercurial loader: T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs).
Oct 17 2017, 3:44 PM · Mercurial loader
zack added a subtask for T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs): T328: svn / subversion loader.
Oct 17 2017, 3:44 PM · General
zack added a parent task for T328: svn / subversion loader: T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs).
Oct 17 2017, 3:44 PM · SVN Loader
zack created T808: phabricator lister.
Oct 17 2017, 3:43 PM · Easy hack, Phabricator forge
zack created T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs).
Oct 17 2017, 3:43 PM · General
zack added a comment to D255: Adding codemeta.json file to content indexer.
In D255#5235, @ardumont wrote:

Shouldn't we have a tool to update this regularly based on the different metadata we already have spread across the repository (setup.py, debian/control, debian/copyright, debian/changelog, git tag, etc...)?

Oct 17 2017, 12:10 PM
zack retitled D255: Adding codemeta.json file to content indexer from Adding codemeta.json file to swh-indexer to Adding codemeta.json file to content indexer.
Oct 17 2017, 11:26 AM
zack retitled D255: Adding codemeta.json file to content indexer from Adding codemeta.json file to document metadata to Adding codemeta.json file to swh-indexer.
Oct 17 2017, 11:26 AM
zack requested changes to D255: Adding codemeta.json file to content indexer.
Oct 17 2017, 11:26 AM

Oct 13 2017

zack committed rDDOC69fcce55b22f: README: no need to make clean before make distclean (authored by zack).
README: no need to make clean before make distclean
Oct 13 2017, 4:11 PM
zack committed rDDOC71b0fbb92d70: README.md: fix whitespace error (cosmetic) (authored by zack).
README.md: fix whitespace error (cosmetic)
Oct 13 2017, 4:09 PM
zack committed rDDOCb4bff0c33ad8: customize doc theme (alabaster) to be more Software Heritage-ish (authored by zack).
customize doc theme (alabaster) to be more Software Heritage-ish
Oct 13 2017, 4:08 PM
zack committed rDDOC39199e618401: Makefile.sphinx: don't build apidoc upon "make apidoc" if SKIP_APIDOC (authored by zack).
Makefile.sphinx: don't build apidoc upon "make apidoc" if SKIP_APIDOC
Oct 13 2017, 2:29 PM
zack added a project to T804: swh-loader-core: doesn't properly batch object insertions (for big requests): Core Loader.
Oct 13 2017, 2:23 PM · Core Loader
zack renamed Core Loader from VCS Loader to Core Loader.
Oct 13 2017, 2:23 PM
zack committed rDDOC73f70bd35405: Makefile.sphinx: exclude (django) migrations submodules (authored by zack).
Makefile.sphinx: exclude (django) migrations submodules
Oct 13 2017, 2:22 PM
zack committed rDDOCe6c4f8d85469: Makefile.sphinx: allow modules to opt-out from apidoc generation (authored by zack).
Makefile.sphinx: allow modules to opt-out from apidoc generation
Oct 13 2017, 2:22 PM
zack committed rDDOC4be294a63ebd: stop playing whacamole with django and load swh-web conf to build doc (authored by zack).
stop playing whacamole with django and load swh-web conf to build doc
Oct 13 2017, 2:03 PM
zack committed rDLDBASEa663c23774ae: sanitize sphinx markup in loader.py (authored by zack).
sanitize sphinx markup in loader.py
Oct 13 2017, 1:38 PM
zack committed rDDOC0175359e9e98: requirements.txt: add dependencies on sphinxcontrib.httpdomain (authored by zack).
requirements.txt: add dependencies on sphinxcontrib.httpdomain
Oct 13 2017, 1:30 PM
zack committed rDWAPPSa76713a6eef1: docs: enable httpdomain sphinx extension (authored by zack).
docs: enable httpdomain sphinx extension
Oct 13 2017, 1:28 PM
zack committed rDDOC438cb539bd40: docs: enable httpdomain sphinx extension (authored by zack).
docs: enable httpdomain sphinx extension
Oct 13 2017, 1:27 PM

Oct 10 2017

zack triaged T802: Better error message when switching branch if the visited tree or file is absent as Low priority.
Oct 10 2017, 10:59 AM · Web app

Oct 9 2017

zack added a subtask for T752: logrotate log spam: "error: error creating output file: FILE File exists": T790: Turn off syslog.
Oct 9 2017, 11:54 AM · System administration
zack added a parent task for T790: Turn off syslog: T752: logrotate log spam: "error: error creating output file: FILE File exists".
Oct 9 2017, 11:54 AM · System administration
zack added a project to T704: beautify wiki URLs: Easy hack.
Oct 9 2017, 11:54 AM · Easy hack, System administration
zack renamed T670: add icinga check for graph-navigation API endpoints from add nagios check for graph-navigation API endpoints to add icinga check for graph-navigation API endpoints.
Oct 9 2017, 11:53 AM · Restricted Project, System administration