Page MenuHomeSoftware Heritage
Feed All Stories

Today

rdicosmo raised the priority of T1099: support origin blacklist for archive search and browse from Low to High.

This is an important feature: it has been dormant for a while, but we need to actually start implementing it.

Thu, Jul 2, 8:21 PM · General, Web app
ardumont added a comment to D3396: Add raw metadata to the model..

Looks promising ;)

Thu, Jul 2, 7:56 PM
ardumont added inline comments to D3397: loader.svn: Start tests refactoring with pytest.
Thu, Jul 2, 7:33 PM
swh-public-ci added a comment to D3397: loader.svn: Start tests refactoring with pytest.

Build is green

Thu, Jul 2, 7:23 PM
ardumont added a revision to T2462: svn loader: Refactor tests using pytest fixtures: D3397: loader.svn: Start tests refactoring with pytest.
Thu, Jul 2, 7:21 PM · SVN Loader
ardumont created D3397: loader.svn: Start tests refactoring with pytest.
Thu, Jul 2, 7:21 PM
ardumont created P711 svn test refactoring target!.
Thu, Jul 2, 6:06 PM
swh-public-ci added a comment to D3396: Add raw metadata to the model..

Build is green

Thu, Jul 2, 5:55 PM
vlorentz created D3396: Add raw metadata to the model..
Thu, Jul 2, 5:53 PM
zack committed rMSLD121ad4127166: check-in slides for SoHeal 2020 keynote (authored by zack).
check-in slides for SoHeal 2020 keynote
Thu, Jul 2, 5:37 PM
olasd committed rCJSWH4ca591ff184a: Drop obsolete args for the automatic backport job (authored by olasd).
Drop obsolete args for the automatic backport job
Thu, Jul 2, 5:25 PM
anlambert committed rDWAPPS299ed02cbebd: package.json: Upgrade dependencies (authored by anlambert).
package.json: Upgrade dependencies
Thu, Jul 2, 3:52 PM
anlambert committed rDWAPPSa4da1d819a45: common/service: Ensure to get last status when looking up origin visit (authored by anlambert).
common/service: Ensure to get last status when looking up origin visit
Thu, Jul 2, 3:52 PM
anlambert closed D3395: common/service: Ensure to get last status when looking up origin visit.
Thu, Jul 2, 3:52 PM
ardumont accepted D3395: common/service: Ensure to get last status when looking up origin visit.
Thu, Jul 2, 3:25 PM
ardumont committed rDLDBASEaafac87f5c48: Reuse swh.model.from_disk.iter_directory function (authored by ardumont).
Reuse swh.model.from_disk.iter_directory function
Thu, Jul 2, 3:24 PM
ardumont closed D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 3:24 PM
ardumont updated the test plan for D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 3:23 PM
ardumont updated the test plan for D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 3:23 PM
swh-public-ci added a comment to D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.

Build is green

Thu, Jul 2, 3:23 PM
ardumont committed rDLDSVNee23fd758227: Reuse swh.model.from_disk.iter_directory function (authored by ardumont).
Reuse swh.model.from_disk.iter_directory function
Thu, Jul 2, 3:22 PM
ardumont closed D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 3:22 PM
swh-public-ci added a comment to D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function.

Build is green

Thu, Jul 2, 3:21 PM
swh-public-ci added a comment to D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.

Build has FAILED

Thu, Jul 2, 3:20 PM
ardumont committed rDMOD8863b5c186dd: Refactor common loader behavior within from_disk.iter_directory (authored by ardumont).
Refactor common loader behavior within from_disk.iter_directory
Thu, Jul 2, 3:13 PM
ardumont closed D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.
Thu, Jul 2, 3:13 PM
ardumont committed rDMOD363b1659a6f5: Unify object_type some more within the merkle and from_disk modules (authored by ardumont).
Unify object_type some more within the merkle and from_disk modules
Thu, Jul 2, 3:13 PM
ardumont closed D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 3:13 PM
swh-public-ci added a comment to D3395: common/service: Ensure to get last status when looking up origin visit.

Build is green

Thu, Jul 2, 3:12 PM
swh-public-ci added a comment to D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.

Build is green

Thu, Jul 2, 3:11 PM
ardumont updated the diff for D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.
  • Use "refactor" instead of "factorize" term
  • Adapt according to review
Thu, Jul 2, 3:09 PM
swh-public-ci added a comment to D3390: Unify object_type some more within the merkle and from_disk modules.

Build is green

Thu, Jul 2, 3:05 PM
ardumont updated the diff for D3390: Unify object_type some more within the merkle and from_disk modules.

Adapt according to review

Thu, Jul 2, 3:03 PM
ardumont added inline comments to D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 3:02 PM
anlambert created D3395: common/service: Ensure to get last status when looking up origin visit.
Thu, Jul 2, 3:01 PM
anlambert added inline comments to D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 2:57 PM
ardumont retitled D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function from Factorize common loader behavior within swh.model.from_disk.iter_directory function to Refactor common loader behavior within swh.model.from_disk.iter_directory function.
Thu, Jul 2, 2:54 PM
ardumont added a comment to D3390: Unify object_type some more within the merkle and from_disk modules.

kinda hope that solves [1]

Thu, Jul 2, 2:48 PM
ardumont added inline comments to D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.
Thu, Jul 2, 2:45 PM
anlambert accepted D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 2:45 PM
anlambert accepted D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 2:44 PM
anlambert accepted D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.

Looks good, I added some nitpicking comments.

Thu, Jul 2, 2:44 PM
ardumont added inline comments to D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 2:41 PM
anlambert accepted D3390: Unify object_type some more within the merkle and from_disk modules.

LGTM, maybe you could set object_type type as Final like in other model classes (see inline comment).

Thu, Jul 2, 2:35 PM
ardumont added a comment to P709 ERROR swh/core/api/tests/test_async.py - ValueError: option names {'--aiohttp-fast'} already added.

Solution that worked for me:

Thu, Jul 2, 2:24 PM
douardda requested changes to D3394: Improve test coverage and type coverage for copy_to.

Looks globally fine to me, but I have a few comments/requests.

Thu, Jul 2, 2:17 PM
swh-public-ci added a comment to D3394: Improve test coverage and type coverage for copy_to.

Build is green

Thu, Jul 2, 1:34 PM
olasd created D3394: Improve test coverage and type coverage for copy_to.
Thu, Jul 2, 1:32 PM
ardumont added a comment to P709 ERROR swh/core/api/tests/test_async.py - ValueError: option names {'--aiohttp-fast'} already added.

tox run is fine

Thu, Jul 2, 12:40 PM
douardda created P710 (An Untitled Masterwork).
Thu, Jul 2, 12:31 PM
ardumont created P709 ERROR swh/core/api/tests/test_async.py - ValueError: option names {'--aiohttp-fast'} already added.
Thu, Jul 2, 12:25 PM
ardumont updated the task description for T2310: Make origin visits immutable.
Thu, Jul 2, 12:23 PM · Storage manager, Data Model
ardumont retitled D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function from Factorize common loader behavior within from_disk.iter_directory to Factorize common loader behavior within swh.model.from_disk.iter_directory function.
Thu, Jul 2, 12:23 PM
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 12:12 PM
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 12:12 PM
Harbormaster failed remote builds in B13256: Diff 12036 for D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function!
Thu, Jul 2, 12:09 PM
swh-public-ci added a comment to D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function.

Build has FAILED

Thu, Jul 2, 12:09 PM
ardumont created D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 12:08 PM
zack added a comment to T2430: lookup ingested tarballs (or similar source code containers) by container checksum.

@civodul I wanted to raise the topic of storing container metadata (in the style of what tools like pristine-tar do) here too, so thanks for giving me the chance :-)
I agree it might be a technical solution, *but*, I'm not sure I see the point.
Didn't you agree that having a "lookup service" from tarball/container checksums to SWHIDs (the Software Heritage identifiers, that can then be used to lookup stuff in the archive) would be enough to satisfy distro needs?
If yes, then "archiving container metadata" could be replaced by simply having a way to add entries to the lookup table. And allowing distros to do so is option that we can explore. (Once the service exists, of course.)

Thu, Jul 2, 12:07 PM · Data Model
Harbormaster failed remote builds in B13255: Diff 12035 for D3392: loader-core: Reuse swh.model.from_disk.iter_directory function!
Thu, Jul 2, 12:06 PM
swh-public-ci added a comment to D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.

Build has FAILED

Thu, Jul 2, 12:06 PM
ardumont created D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.
Thu, Jul 2, 12:05 PM
swh-public-ci added a comment to D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.

Build is green

Thu, Jul 2, 12:04 PM
ardumont updated the diff for D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.

Make the returned results a tuple of lists

Thu, Jul 2, 12:02 PM
civodul added a comment to T2430: lookup ingested tarballs (or similar source code containers) by container checksum.

Do I get it right that the primary reason why tarballs aren't systematically archived is that doing so would be too expensive storage-wise (no deduplication)?

Thu, Jul 2, 12:00 PM · Data Model
swh-public-ci added a comment to D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.

Build is green

Thu, Jul 2, 12:00 PM
ardumont created D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.
Thu, Jul 2, 11:58 AM
anlambert committed rDWAPPS76d9162807e0: origin_visits/get_origin_visit: Improve default visit picking strategy (authored by anlambert).
origin_visits/get_origin_visit: Improve default visit picking strategy
Thu, Jul 2, 11:39 AM
anlambert closed D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
Thu, Jul 2, 11:39 AM
swh-public-ci added a comment to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.

Build is green

Thu, Jul 2, 11:35 AM
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 11:24 AM
anlambert updated the diff for D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.

Update: Improve tests implementation

Thu, Jul 2, 11:24 AM
ardumont added a comment to D3383: Implement {directory,revision,release,snapshot}_metadata_{add,get}..

I'll rewrite this using swh-model and only two endpoints for all types

Thu, Jul 2, 11:22 AM
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 11:17 AM
vlorentz abandoned D3383: Implement {directory,revision,release,snapshot}_metadata_{add,get}..

I'll rewrite this using swh-model and only two endpoints for all types

Thu, Jul 2, 11:11 AM
swh-public-ci added a comment to D3390: Unify object_type some more within the merkle and from_disk modules.

Build is green

Thu, Jul 2, 11:10 AM
ardumont created D3390: Unify object_type some more within the merkle and from_disk modules.
Thu, Jul 2, 11:09 AM
vlorentz closed D3382: Move tests of content_metadata_* next to origin_metadata_*.

Landed as 248c277445adbae5813ba80ce0618858d8126634.

Thu, Jul 2, 11:05 AM
vlorentz committed rDSTO248c277445ad: Move tests of content_metadata_* next to origin_metadata_* (authored by vlorentz).
Move tests of content_metadata_* next to origin_metadata_*
Thu, Jul 2, 11:04 AM
ardumont updated the task description for T2310: Make origin visits immutable.
Thu, Jul 2, 10:32 AM · Storage manager, Data Model
vlorentz abandoned D3247: [WIP] Add content_metadata_{add,get}..

Replaced by linked diffs

Thu, Jul 2, 10:12 AM

Yesterday

ardumont accepted D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.

nice ;)

Wed, Jul 1, 8:21 PM
swh-public-ci added a comment to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.

Build is green

Wed, Jul 1, 6:26 PM
anlambert updated the diff for D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
  • Use new swh.storage.algos.origin.origin_get_latest_visit_status utility function in revised get_origin_visit implementation
Wed, Jul 1, 6:15 PM
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.

Build is green

Wed, Jul 1, 6:03 PM
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.

more mypy vs. attrs-strict fighting

Wed, Jul 1, 6:02 PM
Harbormaster failed remote builds in B13246: Diff 12027 for D3389: Extract the extra_headers from metadata on the Revision model class!
Wed, Jul 1, 5:41 PM
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.

Build has FAILED

Wed, Jul 1, 5:41 PM
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.

make mypy happy (hopefully)

Wed, Jul 1, 5:39 PM
moranegg triaged T2472: Indexing intrinsic metadata in a deposit using a sub-folder for the content as Normal priority.
Wed, Jul 1, 5:35 PM · SWORD deposit, Metadata workflow
Harbormaster failed remote builds in B13245: Diff 12026 for D3389: Extract the extra_headers from metadata on the Revision model class!
Wed, Jul 1, 5:23 PM
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.

Build has FAILED

Wed, Jul 1, 5:23 PM
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.

restrict extra_headers to (bytes, bytes) only

Wed, Jul 1, 5:23 PM
olasd edited P708 non bytes extra_headers.
Wed, Jul 1, 5:16 PM
olasd updated the language for P708 non bytes extra_headers from autodetect to remarkup.
Wed, Jul 1, 5:12 PM
olasd created P708 non bytes extra_headers.
Wed, Jul 1, 5:12 PM
ardumont updated the task description for T2310: Make origin visits immutable.
Wed, Jul 1, 4:22 PM · Storage manager, Data Model
Harbormaster failed remote builds in B13244: Diff 12025 for D3389: Extract the extra_headers from metadata on the Revision model class!
Wed, Jul 1, 4:05 PM
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.

Build has FAILED

Wed, Jul 1, 4:05 PM
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.

improve bw-compat support, tests and hypothesis strategies

Wed, Jul 1, 4:04 PM