Page MenuHomeSoftware Heritage
Differential D4833

WIP: FUSE: use graph/ stream API to retrieve history
Changes PlannedPublicDraft
Actions

Authored by haltode on Jan 10 2021, 12:20 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Maniphest Tasks
T2724: fuse: populate history/ incrementally to avoid temporary hangs on huge repos
Summary

WIP, closes T2724.

This diff was my starting work on solving the task, since quite a lot of changes
are required to solve it. What remains to be done:

  • Add a buffer for data received by the iter_content() requests method to ensure the data flow is only cut after a complete line (and not randomly)
  • Configure the mocking offline tests to use a stream API (similar to what is done in swh-web tests)
  • Add a history/.status file similar to the existing history/by-date/.status
  • Make sure by-date/ by-hash/ by-page/ do not rely on having complete history data

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/stream-history-api
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18262
Build 28194: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28193: arc lint + arc unit

Event Timeline

haltode created this revision.Jan 10 2021, 12:20 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 10 2021, 12:20 PM
haltode updated this revision to Diff 17121.Jan 10 2021, 12:22 PM

Small clean-up

swh-public-ci added a comment.Jan 10 2021, 12:23 PM

Build has FAILED

Patch application report for D4833 (id=17120)

Rebasing onto a75335972e...

Current branch diff-target is up to date.
Changes applied before test
commit 4fe50a2915b20c3b8e6b2a0c9fa6169021728b3e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Jan 6 16:27:25 2021 +0100

    WIP

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/264/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/264/console

Harbormaster failed remote builds in B18262: Diff 17120!Jan 10 2021, 12:23 PM
swh-public-ci added a comment.Jan 10 2021, 12:26 PM

Build has FAILED

Patch application report for D4833 (id=17121)

Rebasing onto a75335972e...

Current branch diff-target is up to date.
Changes applied before test
commit 922a110a59bbb590e16bc4ff855e170a3011e7f3
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Jan 6 16:27:25 2021 +0100

    WIP

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/265/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/265/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 10 2021, 12:26 PM
Harbormaster failed remote builds in B18263: Diff 17121!

Revision Contents
Changeset List

PathSize
swh/
fuse/
8 lines
fs/
75 lines
60 lines
tests/
2 lines

Diff 17120

View Options

swh/fuse/cache.py

Loading...
View Options

swh/fuse/fs/artifact.py

Loading...
View Options

swh/fuse/fuse.py

Loading...
View Options

swh/fuse/tests/conftest.py

Loading...
About · Developer information · Legal