Page MenuHomeSoftware Heritage
Differential D4309

Add flat commit view in a history/ virtual dir
ClosedPublic
Actions

Authored by haltode on Oct 19 2020, 5:27 PM.

Details

Reviewers
seirl
zack
Group Reviewers
Reviewers
Maniphest Tasks
T2659: FUSE: add flat commit view to ease batch commit workloads
Commits
rDFUSEe8cb3c6d7456: tests: add unit test for revision history
rDFUSEa8abee511524: fuse: add flat commit view in a 'history' virtual directory
Summary

Closes T2659.

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/revision-history-dir
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16463
Build 25358: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 25357: arc lint + arc unit

Event Timeline

haltode created this revision.Oct 19 2020, 5:27 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 19 2020, 5:27 PM
swh-public-ci added a comment.Oct 19 2020, 5:28 PM

Build is green

Patch application report for D4309 (id=15225)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit 251e14a6c4ce44303d9b5255e9af92e825274382
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/107/ for more details.

Harbormaster completed remote builds in B16461: Diff 15225.Oct 19 2020, 5:28 PM
haltode updated this revision to Diff 15227.Oct 19 2020, 5:30 PM

Add task in commit message

swh/fuse/tests/data/api_data.py
34–35

Not sure why the ordering has changed now compared to before. Need to investigate.

swh-public-ci added a comment.Oct 19 2020, 5:31 PM

Build is green

Patch application report for D4309 (id=15227)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit 25b5e40008d1899d65f1bb7cd05c901c4c081537
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/108/ for more details.

Harbormaster completed remote builds in B16463: Diff 15227.Oct 19 2020, 5:31 PM
haltode updated this revision to Diff 15243.Oct 20 2020, 9:52 AM

Add history cache and history dir in design.md

swh-public-ci added a comment.Oct 20 2020, 9:53 AM

Build is green

Patch application report for D4309 (id=15243)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit f1ed7873d2d879125ec6b2f5e2575dc3f4ab0daa
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/109/ for more details.

Harbormaster completed remote builds in B16475: Diff 15243.Oct 20 2020, 9:53 AM
haltode marked an inline comment as not done.Oct 20 2020, 9:54 AM
haltode added inline comments.
swh/fuse/fs/artifact.py
141

Update the docstring once changes are done in design.md

haltode updated this revision to Diff 15244.Oct 20 2020, 10:17 AM

Update fuse clean command to remove history cache.

swh-public-ci added a comment.Oct 20 2020, 10:19 AM

Build is green

Patch application report for D4309 (id=15244)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit cc7fc8a28977c354aaf2f3222bb72fff8724c3cd
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/110/ for more details.

Harbormaster completed remote builds in B16476: Diff 15244.Oct 20 2020, 10:19 AM
haltode updated this revision to Diff 15247.Oct 20 2020, 11:19 AM

Do not raise error when missing entry in swh-graph

swh-public-ci added a comment.Oct 20 2020, 11:20 AM

Build is green

Patch application report for D4309 (id=15247)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: WIP: history/ directory
Changes applied before test
commit 7d7096dd5b412cb510a24f0bbd3da17a4162b128
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/112/ for more details.

Harbormaster completed remote builds in B16479: Diff 15247.Oct 20 2020, 11:20 AM
zack resigned from this revision.Oct 20 2020, 12:09 PM
zack added a subscriber: zack.
zack added inline comments.
docs/design.md
105–107 ↗(On Diff #15247)

after checking that we can return commits in topological ordering, we should document that fact here

btw, you can verify this with ls -f

229–233 ↗(On Diff #15247)

better type: rev SWHID → ancestor SWHIDs

better first description sentence: "The history cache map SWHIDs of type rev to a list of rev SWHIDs corresponding to all its revision ancestors, sorted in reverse topological order"

swh/fuse/tests/test_release.py
25

Comparing for set equality is correct, but will make these tests hard to maintain in the longrun.
It would be better to check that each expected element is among the listdir output (i.e., that is a subset).

Note that this is different from the case in which you are testing for the content of the virtual dir associated to a dir SWHID, because in that case any additional object is likely a bug. Here that is not necessarily the case.

zack edited reviewers, added: seirl; removed: zack.Oct 20 2020, 12:10 PM
haltode updated this revision to Diff 15254.Oct 20 2020, 1:16 PM

Better description in design.md

haltode marked 2 inline comments as done.Oct 20 2020, 1:16 PM
swh-public-ci added a comment.Oct 20 2020, 1:17 PM

Build is green

Patch application report for D4309 (id=15254)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: WIP: history/ directory
Changes applied before test
commit 50c7ff126039cf7ab98213d39c986d2127745b1d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/113/ for more details.

Harbormaster completed remote builds in B16486: Diff 15254.Oct 20 2020, 1:17 PM
haltode retitled this revision from WIP: history/ directory to Add flat commit view in a history/ virtual dir.Oct 20 2020, 1:20 PM
haltode updated this revision to Diff 15256.Oct 20 2020, 1:32 PM

Commit message.

swh-public-ci added a comment.Oct 20 2020, 1:33 PM

Build is green

Patch application report for D4309 (id=15256)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: fuse: add flat commit view in a 'history' virtual directory
Changes applied before test
commit e7de0600248419915a6410a8016176386c702686
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/114/ for more details.

Harbormaster completed remote builds in B16488: Diff 15256.Oct 20 2020, 1:33 PM
haltode edited the summary of this revision. (Show Details)Oct 20 2020, 1:38 PM
haltode added a task: T2659: FUSE: add flat commit view to ease batch commit workloads.
haltode updated this revision to Diff 15257.Oct 20 2020, 3:40 PM
  • tests: add unit test for revision history
haltode marked an inline comment as done.Oct 20 2020, 3:41 PM
swh-public-ci added a comment.Oct 20 2020, 3:41 PM

Build is green

Patch application report for D4309 (id=15257)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: fuse: add flat commit view in a 'history' virtual directory
Applying: tests: add unit test for revision history
Changes applied before test
commit 4500e30a11ac8cf7333fcd2bcd97315a3e09beb9
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit fc9eec70e2a6c7e776cfba733087ba16e6f3a72c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/115/ for more details.

Harbormaster completed remote builds in B16489: Diff 15257.Oct 20 2020, 3:41 PM
haltode updated this revision to Diff 15258.Oct 20 2020, 4:00 PM

Request history at the latest once the history/ is accessed, not the revision
directory.

swh-public-ci added a comment.Oct 20 2020, 4:01 PM

Build is green

Patch application report for D4309 (id=15258)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: fuse: add flat commit view in a 'history' virtual directory
Applying: tests: add unit test for revision history
Changes applied before test
commit d5ef144db67ee1861fea578d182b817bc88e506e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit f0df0c0aead3a65e5f6b574d0f0a395c82fd1a30
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/116/ for more details.

Harbormaster completed remote builds in B16490: Diff 15258.Oct 20 2020, 4:01 PM
haltode added inline comments.Oct 21 2020, 11:05 AM
swh/fuse/cache.py
149–150

Update the docstring once changes are done in design.md

haltode added inline comments.Oct 21 2020, 11:14 AM
swh/fuse/fs/artifact.py
207–208

This history field might be a bit redundant because get_history is already utilizing a sqlite cache and we probably want something more general in all __aiter__ (see T2695).

haltode updated this revision to Diff 15266.Oct 21 2020, 11:15 AM

Rebasing on master.

swh-public-ci added a comment.Oct 21 2020, 11:17 AM

Build is green

Patch application report for D4309 (id=15266)

Rebasing onto 3409bd8a09...

Current branch diff-target is up to date.
Changes applied before test
commit 96cb52d890a4d44fb9761251fb224d90686a72eb
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit 9f32b11e52507973e6c5d3b8d6523300b3b99adb
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/119/ for more details.

Harbormaster completed remote builds in B16499: Diff 15266.Oct 21 2020, 11:17 AM
haltode updated this revision to Diff 15274.Oct 21 2020, 1:26 PM

Rebasing on master.

swh-public-ci added a comment.Oct 21 2020, 1:28 PM

Build is green

Patch application report for D4309 (id=15274)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit 91fdf890db384dabfe4c8e9e28e5e1122e32db03
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit e467dd0c4a5f1c7787ed10c514299cc6bbcfcd85
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/121/ for more details.

Harbormaster completed remote builds in B16507: Diff 15274.Oct 21 2020, 1:28 PM
haltode updated this revision to Diff 15280.Oct 21 2020, 2:55 PM

Represent ancestors using graph edges in sqlite.

swh-public-ci added a comment.Oct 21 2020, 2:56 PM

Build has FAILED

Patch application report for D4309 (id=15280)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit 62e96920aef453aae042608ced71d2e97565be6a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit c7026b8cd9f6bdd73dca18714e22900398b621c2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/122/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/122/console

Harbormaster failed remote builds in B16513: Diff 15280!Oct 21 2020, 2:56 PM
haltode updated this revision to Diff 15281.Oct 21 2020, 3:18 PM

Update unit tests for new history cache graph structure

swh-public-ci added a comment.Oct 21 2020, 3:19 PM

Build is green

Patch application report for D4309 (id=15281)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit c54053892233e1f0b56951fe7d7d67a0f744baee
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit c7026b8cd9f6bdd73dca18714e22900398b621c2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/123/ for more details.

Harbormaster completed remote builds in B16514: Diff 15281.Oct 21 2020, 3:19 PM
zack requested changes to this revision.Oct 21 2020, 3:23 PM
zack added inline comments.
swh/fuse/cache.py
156–159

please call the nodes with some meaningful names, likes src and dst, or child parent maybe

also, I don't know how indexes work in sqlite, but you might want to have one on the node you use as query base (src or child)

This revision now requires changes to proceed.Oct 21 2020, 3:23 PM
zack resigned from this revision.Oct 21 2020, 3:23 PM
haltode added inline comments.Oct 21 2020, 3:36 PM
swh/fuse/cache.py
178

Directly convert swhid to string since it is not used anymore for its SWHID method.

haltode updated this revision to Diff 15290.Oct 22 2020, 9:34 AM

Fast graph sql query (union instead of union all).

haltode marked an inline comment as done.Oct 22 2020, 9:34 AM
swh-public-ci added a comment.Oct 22 2020, 9:35 AM

Build is green

Patch application report for D4309 (id=15290)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit babcd40d99b2a265103faa14747eb32477f0be3e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit 8d2e1c203085300686ca9aaf287bcc293c778625
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/124/ for more details.

Harbormaster completed remote builds in B16521: Diff 15290.Oct 22 2020, 9:35 AM
haltode updated this revision to Diff 15330.Oct 22 2020, 3:35 PM

Add missing "if not exists" clause in sqlite index

swh-public-ci added a comment.Oct 22 2020, 3:37 PM

Build is green

Patch application report for D4309 (id=15330)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit 427e5089fcd96eb3ddbd986bcee0d4cec0369675
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit 972399904864fab9153952657114a4c713613d52
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/125/ for more details.

Harbormaster completed remote builds in B16557: Diff 15330.Oct 22 2020, 3:37 PM
seirl accepted this revision.Oct 22 2020, 5:32 PM
This revision is now accepted and ready to land.Oct 22 2020, 5:32 PM
haltode updated this revision to Diff 15344.Oct 22 2020, 5:44 PM
  • Update docstrings
  • Remove RevisionHistory history inner field
swh-public-ci added a comment.Oct 22 2020, 5:45 PM

Build is green

Patch application report for D4309 (id=15344)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit e8cb3c6d7456e9d85a9ba2db7160e436c87f7634
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit a8abee511524058c0c92d0cda69ea24a351eac0d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/126/ for more details.

Harbormaster completed remote builds in B16571: Diff 15344.Oct 22 2020, 5:45 PM
Closed by commit rDFUSEa8abee511524: fuse: add flat commit view in a 'history' virtual directory (authored by haltode). · Explain WhyOct 22 2020, 5:45 PM
This revision was automatically updated to reflect the committed changes.
haltode added a commit: rDFUSEa8abee511524: fuse: add flat commit view in a 'history' virtual directory.
haltode added a commit: rDFUSEe8cb3c6d7456: tests: add unit test for revision history.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
fuse/
41 lines95%
Loading...
1 line61%
Loading...
fs/
23 lines90%
Loading...
23 lines82%
Loading...
tests/
8 lines92%
Loading...
data/
114 lines100%
Loading...
14 lines100%
Loading...
2 lines100%
Loading...
2 lines100%
Loading...

Diff 15227

View Options

swh/fuse/cache.py

Loading...
View Options

swh/fuse/cli.py

Loading...
View Options

swh/fuse/fs/artifact.py

Loading...
View Options

swh/fuse/fuse.py

Loading...
View Options

swh/fuse/tests/conftest.py

Loading...
View Options

swh/fuse/tests/data/api_data.py

Loading...
View Options

swh/fuse/tests/data/gen-api-data.py

Loading...
View Options

swh/fuse/tests/test_release.py

Loading...
View Options

swh/fuse/tests/test_snapshot.py

Loading...
About · Developer information · Legal