Page MenuHomeSoftware Heritage

Add flat commit view in a history/ virtual dir
ClosedPublic

Authored by haltode on Oct 19 2020, 5:27 PM.

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/revision-history-dir
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16463
Build 25358: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 25357: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4309 (id=15225)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit 251e14a6c4ce44303d9b5255e9af92e825274382
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/107/ for more details.

Add task in commit message

swh/fuse/tests/data/api_data.py
34–35

Not sure why the ordering has changed now compared to before. Need to investigate.

Build is green

Patch application report for D4309 (id=15227)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit 25b5e40008d1899d65f1bb7cd05c901c4c081537
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/108/ for more details.

Add history cache and history dir in design.md

Build is green

Patch application report for D4309 (id=15243)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit f1ed7873d2d879125ec6b2f5e2575dc3f4ab0daa
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/109/ for more details.

haltode marked an inline comment as not done.Oct 20 2020, 9:54 AM
haltode added inline comments.
swh/fuse/fs/artifact.py
141

Update the docstring once changes are done in design.md

Update fuse clean command to remove history cache.

Build is green

Patch application report for D4309 (id=15244)

Rebasing onto 3ad12d7aaf...

Current branch diff-target is up to date.
Changes applied before test
commit cc7fc8a28977c354aaf2f3222bb72fff8724c3cd
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/110/ for more details.

Do not raise error when missing entry in swh-graph

Build is green

Patch application report for D4309 (id=15247)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: WIP: history/ directory
Changes applied before test
commit 7d7096dd5b412cb510a24f0bbd3da17a4162b128
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/112/ for more details.

zack added a subscriber: zack.
zack added inline comments.
docs/design.md
105–107 ↗(On Diff #15247)

after checking that we can return commits in topological ordering, we should document that fact here

btw, you can verify this with ls -f

229–233 ↗(On Diff #15247)

better type: rev SWHID → ancestor SWHIDs

better first description sentence: "The history cache map SWHIDs of type rev to a list of rev SWHIDs corresponding to all its revision ancestors, sorted in reverse topological order"

swh/fuse/tests/test_release.py
25

Comparing for set equality is correct, but will make these tests hard to maintain in the longrun.
It would be better to check that each expected element is among the listdir output (i.e., that is a subset).

Note that this is different from the case in which you are testing for the content of the virtual dir associated to a dir SWHID, because in that case any additional object is likely a bug. Here that is not necessarily the case.

Better description in design.md

Build is green

Patch application report for D4309 (id=15254)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: WIP: history/ directory
Changes applied before test
commit 50c7ff126039cf7ab98213d39c986d2127745b1d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    WIP: history/ directory
    
    Closes T2659.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/113/ for more details.

haltode retitled this revision from WIP: history/ directory to Add flat commit view in a history/ virtual dir.Oct 20 2020, 1:20 PM

Build is green

Patch application report for D4309 (id=15256)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: fuse: add flat commit view in a 'history' virtual directory
Changes applied before test
commit e7de0600248419915a6410a8016176386c702686
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/114/ for more details.

  • tests: add unit test for revision history

Build is green

Patch application report for D4309 (id=15257)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: fuse: add flat commit view in a 'history' virtual directory
Applying: tests: add unit test for revision history
Changes applied before test
commit 4500e30a11ac8cf7333fcd2bcd97315a3e09beb9
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit fc9eec70e2a6c7e776cfba733087ba16e6f3a72c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/115/ for more details.

Request history at the latest once the history/ is accessed, not the revision
directory.

Build is green

Patch application report for D4309 (id=15258)

Rebasing onto 3409bd8a09...

First, rewinding head to replay your work on top of it...
Applying: fuse: add flat commit view in a 'history' virtual directory
Applying: tests: add unit test for revision history
Changes applied before test
commit d5ef144db67ee1861fea578d182b817bc88e506e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit f0df0c0aead3a65e5f6b574d0f0a395c82fd1a30
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/116/ for more details.

swh/fuse/cache.py
149–150

Update the docstring once changes are done in design.md

swh/fuse/fs/artifact.py
207–208

This history field might be a bit redundant because get_history is already utilizing a sqlite cache and we probably want something more general in all __aiter__ (see T2695).

Build is green

Patch application report for D4309 (id=15266)

Rebasing onto 3409bd8a09...

Current branch diff-target is up to date.
Changes applied before test
commit 96cb52d890a4d44fb9761251fb224d90686a72eb
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit 9f32b11e52507973e6c5d3b8d6523300b3b99adb
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/119/ for more details.

Build is green

Patch application report for D4309 (id=15274)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit 91fdf890db384dabfe4c8e9e28e5e1122e32db03
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit e467dd0c4a5f1c7787ed10c514299cc6bbcfcd85
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/121/ for more details.

Represent ancestors using graph edges in sqlite.

Build has FAILED

Patch application report for D4309 (id=15280)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit 62e96920aef453aae042608ced71d2e97565be6a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit c7026b8cd9f6bdd73dca18714e22900398b621c2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/122/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/122/console

Update unit tests for new history cache graph structure

Build is green

Patch application report for D4309 (id=15281)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit c54053892233e1f0b56951fe7d7d67a0f744baee
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit c7026b8cd9f6bdd73dca18714e22900398b621c2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/123/ for more details.

zack requested changes to this revision.Oct 21 2020, 3:23 PM
zack added inline comments.
swh/fuse/cache.py
156–159

please call the nodes with some meaningful names, likes src and dst, or child parent maybe

also, I don't know how indexes work in sqlite, but you might want to have one on the node you use as query base (src or child)

This revision now requires changes to proceed.Oct 21 2020, 3:23 PM
swh/fuse/cache.py
178

Directly convert swhid to string since it is not used anymore for its SWHID method.

Fast graph sql query (union instead of union all).

Build is green

Patch application report for D4309 (id=15290)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit babcd40d99b2a265103faa14747eb32477f0be3e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit 8d2e1c203085300686ca9aaf287bcc293c778625
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/124/ for more details.

Add missing "if not exists" clause in sqlite index

Build is green

Patch application report for D4309 (id=15330)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit 427e5089fcd96eb3ddbd986bcee0d4cec0369675
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit 972399904864fab9153952657114a4c713613d52
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/125/ for more details.

This revision is now accepted and ready to land.Oct 22 2020, 5:32 PM
  • Update docstrings
  • Remove RevisionHistory history inner field

Build is green

Patch application report for D4309 (id=15344)

Rebasing onto 6a9b719655...

Current branch diff-target is up to date.
Changes applied before test
commit e8cb3c6d7456e9d85a9ba2db7160e436c87f7634
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 20 15:24:18 2020 +0200

    tests: add unit test for revision history
    
    Rework existing mock archive to account for a new graph backend.

commit a8abee511524058c0c92d0cda69ea24a351eac0d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 19 17:24:06 2020 +0200

    fuse: add flat commit view in a 'history' virtual directory
    
    Closes T2659.
    
    Rationale: ease batch commit workloads.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/126/ for more details.