Page MenuHomeSoftware Heritage
Feed Advanced Search

Nov 29 2022

vlorentz closed D8888: Drop content_language and content_ctags tables and related SQL functions.
Nov 29 2022, 3:28 PM
vlorentz committed rDCIDXa5ee54aedece: Drop content_language and content_ctags tables and related SQL functions (authored by vlorentz).
Drop content_language and content_ctags tables and related SQL functions
Nov 29 2022, 3:28 PM
vlorentz committed rDGRPHd08d6059ce58: luigi: Add missing docstrings (authored by vlorentz).
luigi: Add missing docstrings
Nov 29 2022, 3:24 PM
vlorentz committed rDGRPH48fca3afc213: luigi: Fix typos in docstring (authored by vlorentz).
luigi: Fix typos in docstring
Nov 29 2022, 2:22 PM
vlorentz closed D8896: luigi: Rename classes to be globally unambiguous.
Nov 29 2022, 2:10 PM
vlorentz committed rDDATASET99a599378f29: luigi: Rename classes to be globally unambiguous (authored by vlorentz).
luigi: Rename classes to be globally unambiguous
Nov 29 2022, 2:10 PM
vlorentz added a comment to D8886: replay: Add more debug logs.

@anlambert Looks like you should use a separate logger, so it integrates with the standard logging config, eg.

Nov 29 2022, 1:48 PM
vlorentz added a comment to D8889: save_code_now: Add from_webhook field to SaveOriginRequest model.

And would it make sense to store which webhook was used?

It can be easily guessed from the origin URL so I do not think we should add that info.

Nov 29 2022, 1:34 PM
vlorentz committed rDGRPH2b634bf4192d: luigi: Simplify compression task (authored by vlorentz).
luigi: Simplify compression task
Nov 29 2022, 11:54 AM
vlorentz committed rDGRPH0a4b706400d4: luigi: Remove option to configure graph name (authored by vlorentz).
luigi: Remove option to configure graph name
Nov 29 2022, 11:54 AM
vlorentz closed D8891: Add luigi task to compress the graph.
Nov 29 2022, 11:54 AM
vlorentz committed rDGRPH60e4707feca6: Add luigi task to compress the graph (authored by vlorentz).
Add luigi task to compress the graph
Nov 29 2022, 11:54 AM
vlorentz requested review of D8896: luigi: Rename classes to be globally unambiguous.
Nov 29 2022, 11:49 AM
vlorentz updated the diff for D8891: Add luigi task to compress the graph.

fix setup.py and doc

Nov 29 2022, 11:23 AM
vlorentz updated the diff for D8891: Add luigi task to compress the graph.

fix tox.ini + remove option to configure graph name

Nov 29 2022, 11:14 AM
vlorentz updated the diff for D8891: Add luigi task to compress the graph.
  • luigi: Simplify compression task
Nov 29 2022, 11:06 AM

Nov 28 2022

vlorentz requested review of D8891: Add luigi task to compress the graph.

tests will pass when swh-dataset is released

Nov 28 2022, 5:34 PM
vlorentz added a revision to T2220: swh-graph in production: D8891: Add luigi task to compress the graph.
Nov 28 2022, 4:34 PM · Roadmap 2022, meta-task, Roadmap 2021, Compressed graph service
vlorentz added a comment to D8889: save_code_now: Add from_webhook field to SaveOriginRequest model.

What is it going to be used for?

Nov 28 2022, 3:42 PM
vlorentz requested review of D8888: Drop content_language and content_ctags tables and related SQL functions.
Nov 28 2022, 3:42 PM
vlorentz added inline comments to D8873: storage: Insert from temporary tables in consistent order.
Nov 28 2022, 12:34 PM
vlorentz closed D8873: storage: Insert from temporary tables in consistent order.
Nov 28 2022, 12:33 PM
vlorentz committed rDCIDXf7833b7e2d98: storage: Insert from temporary tables in consistent order (authored by vlorentz).
storage: Insert from temporary tables in consistent order
Nov 28 2022, 12:33 PM
vlorentz accepted D8887: Migrate to sphinx-design.

ooh, I should use this on https://docs.softwareheritage.org/devel/swh-graph/grpc-api.html too

Nov 28 2022, 12:29 PM
vlorentz committed rDGRPH6572c43136fd: Fix linting (authored by vlorentz).
Fix linting
Nov 28 2022, 12:08 PM
vlorentz reopened T4549: Write a script to generate qualified SWHID from swh-graph as "Open".

We should add some tests so the code doesn't break when we change other stuff. I'll try to do it this week

Nov 28 2022, 11:08 AM · Compressed graph service
vlorentz added inline comments to D8873: storage: Insert from temporary tables in consistent order.
Nov 28 2022, 11:00 AM
vlorentz updated the diff for D8873: storage: Insert from temporary tables in consistent order.

bump schema version and add migration

Nov 28 2022, 11:00 AM
vlorentz renamed T4701: GitHub mirror of swh-graph is not updated from swh-graph mirror not updated to GitHub mirror of swh-graph is not updated.
Nov 28 2022, 10:11 AM · System administration, Compressed graph service
vlorentz added a comment to T4694: Use directory metadata in origin search.

deployment task: https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4682

Nov 28 2022, 10:01 AM · Indexer, Metadata workflow

Nov 24 2022

vlorentz added a revision to T4695: Provide a collaboration graph / dataset: D8883: Add a script to generate a topological sort.
Nov 24 2022, 4:20 PM · Datasets
vlorentz committed rDDATASETf8a13718b69a: luigi: Clean up partial local exports before exporting again (authored by vlorentz).
luigi: Clean up partial local exports before exporting again
Nov 24 2022, 4:12 PM
vlorentz closed D8866: luigi: Add swh-dataset version to export/meta.json.
Nov 24 2022, 4:12 PM
vlorentz closed D8864: luigi: Remove copies of stamp files to/from S3.
Nov 24 2022, 4:12 PM
vlorentz closed D8865: luigi: Add LocalExport task.
Nov 24 2022, 4:12 PM
vlorentz committed rDDATASETb39436e38be5: luigi: Remove copies of stamp files to/from S3 (authored by vlorentz).
luigi: Remove copies of stamp files to/from S3
Nov 24 2022, 4:12 PM
vlorentz committed rDDATASET0bf9c88d9604: luigi: Add LocalExport task (authored by vlorentz).
luigi: Add LocalExport task
Nov 24 2022, 4:12 PM
vlorentz committed rDDATASET9a194f421be3: luigi: Add swh-dataset version to export/meta.json (authored by vlorentz).
luigi: Add swh-dataset version to export/meta.json
Nov 24 2022, 4:12 PM
vlorentz updated the diff for D8866: luigi: Add swh-dataset version to export/meta.json.

rebase

Nov 24 2022, 4:12 PM
vlorentz updated the diff for D8865: luigi: Add LocalExport task.

fix typo

Nov 24 2022, 4:12 PM
vlorentz accepted D8878: vault/assets: Fix cooking task creation when there is a pending one.
Nov 24 2022, 3:57 PM
vlorentz added inline comments to D8878: vault/assets: Fix cooking task creation when there is a pending one.
Nov 24 2022, 3:14 PM
vlorentz requested changes to D8878: vault/assets: Fix cooking task creation when there is a pending one.
Nov 24 2022, 2:33 PM
vlorentz accepted D8878: vault/assets: Fix cooking task creation when there is a pending one.

very nice!

Nov 24 2022, 2:32 PM
vlorentz committed rDGRPHb27c3958cabf: docs: Fix broken reference (authored by vlorentz).
docs: Fix broken reference
Nov 24 2022, 1:47 PM
vlorentz closed D8881: Document the need for higher vm.max_map_count.
Nov 24 2022, 1:47 PM
vlorentz committed rDGRPHdc2cb79bc670: Document the need for higher vm.max_map_count (authored by vlorentz).
Document the need for higher vm.max_map_count
Nov 24 2022, 1:47 PM
vlorentz committed rDGRPH684241e89dc5: cli: Fix support for --graph CLI parameter (authored by vlorentz).
cli: Fix support for --graph CLI parameter
Nov 24 2022, 1:47 PM
vlorentz closed D8880: Fix crash on null Release message.
Nov 24 2022, 1:47 PM
vlorentz committed rDGRPH486312e00eaa: Fix crash on null Release message (authored by vlorentz).
Fix crash on null Release message
Nov 24 2022, 1:47 PM
vlorentz created P1531 (An Untitled Masterwork).
Nov 24 2022, 1:36 PM
vlorentz updated the diff for D8881: Document the need for higher vm.max_map_count.

fix typo, thx

Nov 24 2022, 1:24 PM
vlorentz moved T2202: Collect extrinsic metadata from Work in progress to Pending validation on the Roadmap 2022 board.
Nov 24 2022, 11:24 AM · Roadmap 2022, meta-task, Roadmap 2021, Extrinsic metadata
vlorentz added a task to D8880: Fix crash on null Release message: T2220: swh-graph in production.
Nov 24 2022, 11:24 AM
vlorentz added a task to D8881: Document the need for higher vm.max_map_count: T2220: swh-graph in production.
Nov 24 2022, 11:24 AM
vlorentz added revisions to T2220: swh-graph in production: D8881: Document the need for higher vm.max_map_count, D8880: Fix crash on null Release message.
Nov 24 2022, 11:24 AM · Roadmap 2022, meta-task, Roadmap 2021, Compressed graph service
vlorentz requested review of D8881: Document the need for higher vm.max_map_count.
Nov 24 2022, 11:21 AM
vlorentz requested review of D8880: Fix crash on null Release message.
Nov 24 2022, 11:20 AM
vlorentz accepted D8879: svn: Also escape hash signs in URLs to avoid libsvn assertion.
Nov 24 2022, 11:12 AM
vlorentz added a comment to D8878: vault/assets: Fix cooking task creation when there is a pending one.

Is it visible on the Downloads page even if someone else requested the cooking?

Nov 24 2022, 9:59 AM
vlorentz added a comment to D8879: svn: Also escape hash signs in URLs to avoid libsvn assertion.

Why not urlencode? Couldn't this be an issue with other characters?

Nov 24 2022, 9:58 AM

Nov 23 2022

vlorentz committed rDSNIPcab1eeaba219: recheck_consistency: Fix special-casing of Linux (authored by vlorentz).
recheck_consistency: Fix special-casing of Linux
Nov 23 2022, 11:12 AM
vlorentz committed rDSNIP2197ddd78409: recheck_consistency: Fix lint issues (authored by vlorentz).
recheck_consistency: Fix lint issues
Nov 23 2022, 11:12 AM
vlorentz committed rDSNIP2f435c745c6a: recheck_consistency: Add copyright header (authored by vlorentz).
recheck_consistency: Add copyright header
Nov 23 2022, 11:12 AM
vlorentz triaged T4698: Error when requesting vault cooking for a directory in pending state as Normal priority.
Nov 23 2022, 9:35 AM · Web app
vlorentz renamed T4697: svn loader: FileNotFoundError while creating a directory from FileNotFoundError: [Errno 2] No such file or directory: '/tmp/swh.loader.svn.0g8bqfec-1539429/tmpxz9z024c/tmpo96ke7rq' to svn loader: FileNotFoundError while creating a directory.
Nov 23 2022, 9:30 AM · SVN Loader
vlorentz placed T4697: svn loader: FileNotFoundError while creating a directory up for grabs.
Nov 23 2022, 9:30 AM · SVN Loader
vlorentz requested review of D8873: storage: Insert from temporary tables in consistent order.
Nov 23 2022, 9:26 AM
vlorentz added a revision to T4696: indexer storage: Deadlock in origin_extrinsic_metadata_add: D8873: storage: Insert from temporary tables in consistent order.
Nov 23 2022, 9:21 AM · Indexer
vlorentz triaged T4696: indexer storage: Deadlock in origin_extrinsic_metadata_add as Low priority.
Nov 23 2022, 9:19 AM · Indexer
vlorentz closed D8863: ExtrinsicMetadataIndexer: Add support for metadata with origin in context.
Nov 23 2022, 8:57 AM
vlorentz committed rDCIDX221d48e24252: ExtrinsicMetadataIndexer: Add support for metadata with origin in context (authored by vlorentz).
ExtrinsicMetadataIndexer: Add support for metadata with origin in context
Nov 23 2022, 8:57 AM
vlorentz closed D8861: origin_head: Do not fetch complete snapshots for non-FTP visits.
Nov 23 2022, 8:57 AM
vlorentz committed rDCIDX03b4bb002c87: origin_head: Do not fetch complete snapshots for non-FTP visits (authored by vlorentz).
origin_head: Do not fetch complete snapshots for non-FTP visits
Nov 23 2022, 8:57 AM

Nov 22 2022

vlorentz accepted D8869: cypress: Increase sqlite3 busy timeout.
Nov 22 2022, 12:12 PM

Nov 21 2022

vlorentz accepted D8867: save_origin_webhooks: Handle github ping event.
Nov 21 2022, 7:07 PM
vlorentz accepted D8868: save_origin_webhooks: Handle private repositories.
Nov 21 2022, 7:06 PM
vlorentz requested review of D8866: luigi: Add swh-dataset version to export/meta.json.
Nov 21 2022, 3:55 PM
vlorentz requested review of D8865: luigi: Add LocalExport task.
Nov 21 2022, 3:54 PM
vlorentz requested review of D8864: luigi: Remove copies of stamp files to/from S3.
Nov 21 2022, 3:47 PM
vlorentz requested review of D8863: ExtrinsicMetadataIndexer: Add support for metadata with origin in context.
Nov 21 2022, 3:04 PM
vlorentz accepted D8862: Rename checksum to hash in the schema for consistency.
Nov 21 2022, 3:02 PM
vlorentz added a revision to T4694: Use directory metadata in origin search: D8863: ExtrinsicMetadataIndexer: Add support for metadata with origin in context.
Nov 21 2022, 2:59 PM · Indexer, Metadata workflow
vlorentz added inline comments to D8862: Rename checksum to hash in the schema for consistency.
Nov 21 2022, 2:57 PM
vlorentz added a comment to D8862: Rename checksum to hash in the schema for consistency.

could you pluralize _get_content_by_hash and the ones below?

Nov 21 2022, 2:45 PM
vlorentz requested changes to D8862: Rename checksum to hash in the schema for consistency.
Nov 21 2022, 2:45 PM
vlorentz accepted D8859: Rename the content attribute fileType to mimeType in the schema.

oh you already support that, alright then

Nov 21 2022, 1:53 PM
vlorentz added a comment to D8859: Rename the content attribute fileType to mimeType in the schema.

What about this?

Nov 21 2022, 1:53 PM
vlorentz requested review of D8861: origin_head: Do not fetch complete snapshots for non-FTP visits.
Nov 21 2022, 1:51 PM
vlorentz accepted D8858: Reanme MerkleNode in schema to SWHNode.
Nov 21 2022, 12:15 PM
vlorentz added a comment to D8859: Rename the content attribute fileType to mimeType in the schema.

that's not the actual MIME type though; encoding is only an optional part of it

Nov 21 2022, 12:14 PM
vlorentz triaged T4695: Provide a collaboration graph / dataset as Normal priority.
Nov 21 2022, 12:13 PM · Datasets
vlorentz triaged T4694: Use directory metadata in origin search as Normal priority.
Nov 21 2022, 10:27 AM · Indexer, Metadata workflow
vlorentz closed T4599: Github descriptions are not used to search origins as Resolved.

It works now! eg. https://archive.softwareheritage.org/browse/search/?q=supybot+successor&with_visit=true&with_content=true&search_metadata=true

Nov 21 2022, 10:17 AM · Metadata workflow, Archive search

Nov 17 2022

vlorentz accepted D8855: Makefile: Speedup sphinx-apidoc execution significantly.

FWIW I also submitted a PR to Sphinx to fix the pathological runtimes there too: https://github.com/sphinx-doc/sphinx/pull/10981

Nov 17 2022, 7:53 PM
vlorentz accepted D8663: Hackage: Implement incremental mode.

Looks good, but please make this last change before landing.

Nov 17 2022, 6:42 PM
vlorentz added inline comments to D8379: Hackage: Loads Hackage Listed origins.
Nov 17 2022, 6:39 PM
vlorentz committed rDDEP5dfe8fd61bf0: docs: Fix typo (authored by vlorentz).
docs: Fix typo
Nov 17 2022, 3:28 PM
vlorentz added a member for Reviewers: lunar.
Nov 17 2022, 11:23 AM

Nov 15 2022

vlorentz accepted D8845: Drop unused core.loader.DVCSLoader class.
Nov 15 2022, 12:11 PM