Looks globally fine to me, but I have a few comments/requests.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed All Stories
All Stories
All Stories
Jul 2 2020
Jul 2 2020
Build is green
ardumont added a comment to P709 ERROR swh/core/api/tests/test_async.py - ValueError: option names {'--aiohttp-fast'} already added.
tox run is fine
ardumont retitled D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function from Factorize common loader behavior within from_disk.iter_directory to Factorize common loader behavior within swh.model.from_disk.iter_directory function.
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
Harbormaster failed remote builds in B13256: Diff 12036 for D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function!
swh-public-ci added a comment to D3393: loader.svn: Reuse swh.model.from_disk.iter_directory function.
Build has FAILED
zack added a comment to T2430: lookup ingested tarballs (or similar source code containers) by container checksum.
@civodul I wanted to raise the topic of storing container metadata (in the style of what tools like pristine-tar do) here too, so thanks for giving me the chance :-)
I agree it might be a technical solution, *but*, I'm not sure I see the point.
Didn't you agree that having a "lookup service" from tarball/container checksums to SWHIDs (the Software Heritage identifiers, that can then be used to lookup stuff in the archive) would be enough to satisfy distro needs?
If yes, then "archiving container metadata" could be replaced by simply having a way to add entries to the lookup table. And allowing distros to do so is option that we can explore. (Once the service exists, of course.)
Harbormaster failed remote builds in B13255: Diff 12035 for D3392: loader-core: Reuse swh.model.from_disk.iter_directory function!
swh-public-ci added a comment to D3392: loader-core: Reuse swh.model.from_disk.iter_directory function.
Build has FAILED
swh-public-ci added a comment to D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.
Build is green
ardumont updated the diff for D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.
Make the returned results a tuple of lists
civodul added a comment to T2430: lookup ingested tarballs (or similar source code containers) by container checksum.
Do I get it right that the primary reason why tarballs aren't systematically archived is that doing so would be too expensive storage-wise (no deduplication)?
swh-public-ci added a comment to D3391: Refactor common loader behavior within swh.model.from_disk.iter_directory function.
Build is green
anlambert committed rDWAPPS76d9162807e0: origin_visits/get_origin_visit: Improve default visit picking strategy (authored by anlambert).
origin_visits/get_origin_visit: Improve default visit picking strategy
swh-public-ci added a comment to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
Build is green
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
anlambert updated the diff for D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
Update: Improve tests implementation
ardumont added a comment to D3383: Implement {directory,revision,release,snapshot}_metadata_{add,get}..
I'll rewrite this using swh-model and only two endpoints for all types
ardumont updated the summary of D3390: Unify object_type some more within the merkle and from_disk modules.
I'll rewrite this using swh-model and only two endpoints for all types
swh-public-ci added a comment to D3390: Unify object_type some more within the merkle and from_disk modules.
Build is green
Landed as 248c277445adbae5813ba80ce0618858d8126634.
vlorentz committed rDSTO248c277445ad: Move tests of content_metadata_* next to origin_metadata_* (authored by vlorentz).
Move tests of content_metadata_* next to origin_metadata_*
Replaced by linked diffs
Jul 1 2020
Jul 1 2020
nice ;)
swh-public-ci added a comment to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
Build is green
anlambert updated the diff for D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
- Use new swh.storage.algos.origin.origin_get_latest_visit_status utility function in revised get_origin_visit implementation
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.
Build is green
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.
more mypy vs. attrs-strict fighting
Harbormaster failed remote builds in B13246: Diff 12027 for D3389: Extract the extra_headers from metadata on the Revision model class!
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.
Build has FAILED
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.
make mypy happy (hopefully)
moranegg triaged T2472: Indexing intrinsic metadata in a deposit using a sub-folder for the content as Normal priority.
Harbormaster failed remote builds in B13245: Diff 12026 for D3389: Extract the extra_headers from metadata on the Revision model class!
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.
Build has FAILED
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.
restrict extra_headers to (bytes, bytes) only
Harbormaster failed remote builds in B13244: Diff 12025 for D3389: Extract the extra_headers from metadata on the Revision model class!
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.
Build has FAILED
douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.
improve bw-compat support, tests and hypothesis strategies
In T2344#43031, @moranegg wrote:Great news !!
Does this mean we need to be SWORD 3 compatible?
ardumont committed rDJNLf6ab436d3934: requirements-swh: Add missing version requirement on swh.model (authored by ardumont).
requirements-swh: Add missing version requirement on swh.model
olasd added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.
I'm not sure about popping the extra_headers off of the incoming metadata field right now. This feels like something we want to do long term, but if you do that right now, it means you're putting upgrades of swh.model and swh.storage (with support with the new field) and all loaders in lockstep of one another. If you upgrade swh.model now, you'll lose the extra_headers until swh.storage will be able to store them.
Landed in eda756252f2151ba602ae47646e35fab4a83ebc1
ardumont committed rDJNLeda756252f21: journal_data: Drop obsolete origin_visit fields (authored by ardumont).
journal_data: Drop obsolete origin_visit fields
Build is green
ardumont changed the status of T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive, a subtask of T2202: Collect extrinsic metadata, from Open to Work in Progress.
ardumont changed the status of T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive, a subtask of T2074: Publish extrinsic metadata to swh-journal/Kafka, from Open to Work in Progress.
ardumont changed the status of T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive, a subtask of T2311: Review the deposit of CodeMeta metadata in xml (following SWORD V2 specs) , from Open to Work in Progress.
ardumont changed the status of T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive, a subtask of T1732: Extend metadata for portals depositing software through SWORD, from Open to Work in Progress.
ardumont changed the status of T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive, a subtask of T2328: Collect metadata about software from ScanR, from Open to Work in Progress.
ardumont changed the status of T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive from Open to Work in Progress.
heads up on status, storage related endpoints developed by vlorentz deployed in storage 0.9.0:
- origin-metadata (we already had but it got abstracted)
- content-metadata (new)
ardumont changed the status of T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive, a subtask of T2344: Build a connector for software deposit via Zenodo/InvenioRDM, from Open to Work in Progress.
ardumont committed rDMOD40a40f508b92: model.OriginVisit: Drop obsolete fields (authored by ardumont).
model.OriginVisit: Drop obsolete fields
ardumont committed rDSTOf2619b69b797: Rework 157 migration to ease replication setup (authored by ardumont).
Rework 157 migration to ease replication setup
swh-public-ci added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.
Build is green
anlambert planned changes to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
This needs more work in order to reuse utility functions from storage and rewrite tests using hypothesis.
ardumont added inline comments to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
anlambert committed rDWAPPSb8eb7741843b: Makefile.local: Improve headless cypress tests execution time (authored by anlambert).
Makefile.local: Improve headless cypress tests execution time
anlambert committed rCJSWHb2c90fcea783: jobs/cypress-tests: Improve stability and execution time (authored by anlambert).
jobs/cypress-tests: Improve stability and execution time
ardumont added a comment to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
Looks good, some remarks above though (proposal to improve coverage).
swh-public-ci added a comment to D3388: Makefile.local: Improve headless cypress tests execution time.
Build is green
ardumont retitled D3386: Test to reproduce issue from wip: Test to reproduce issue to Test to reproduce issue.
Agreed on irc that the priority of the task can be decreased, this:
- metadata field is kinda deprecated
- as this will soon get migrated to the metadata storage
- does not impact that many repositories
swh-public-ci added a comment to D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
Build is green
anlambert updated the diff for D3385: origin_visits/get_origin_visit: Improve default visit picking strategy.
Rebase
anlambert committed rDWAPPSc2726ac1249c: service/snapshot_count_branches: Improve empty snapshot detection (authored by anlambert).
service/snapshot_count_branches: Improve empty snapshot detection