Page MenuHomeSoftware Heritage
Differential D2076

gnu: Separate listing and parsing logic and add integration test
ClosedPublic
Actions

Authored by ardumont on Oct 5 2019, 6:05 PM.

Details

Reviewers
douardda
vlorentz
Group Reviewers
Reviewers
Commits
rDLS0f0b840178eb: gnu.tests: Checks lister output from scheduler
Summary

Also:

  • drop the unused package name loader side (also it collides between gnu/old-gnu listing).
  • Add more tests on the parsing side
  • clean up docstrings
  • Improve slightly the parsing code

Related T2023

Depends on D2077

Test Plan

tox

Diff Detail

Repository
rDLS Listers
Branch
master
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 8166
Build 11771: tox-on-jenkinsJenkins
Build 11770: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ardumont added inline comments.Oct 7 2019, 2:24 PM
swh/lister/gnu/tests/test_tree.py
95–101

Yes, i'm currently trying to finalize and propose a fixture for that in swh-core.
I'd like to refactor this part when that other cog lands though ;)

ardumont updated this revision to Diff 6979.Oct 7 2019, 2:25 PM

Add missing docstring code block formatting and some more dead code clean up

ardumont edited the summary of this revision. (Show Details)Oct 7 2019, 2:27 PM
swh-public-ci added a comment.Oct 7 2019, 2:28 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/390/ for more details.

Harbormaster completed remote builds in B8145: Diff 6979.Oct 7 2019, 2:28 PM
vlorentz requested changes to this revision.Oct 7 2019, 2:39 PM

You forgot to fix the false negative

swh/lister/gnu/lister.py
35–39

inside of code block should be indented

72–82

should be indented to be in the Returns block.

swh/lister/gnu/tree.py
110–123

should be indented to be in the Returns block.

This revision now requires changes to proceed.Oct 7 2019, 2:39 PM
ardumont mentioned this in rDSNIP7e370dd7a849: gnu.analysis: Bootstrap some code to analyze the gnu tree dataset.Oct 7 2019, 2:42 PM
ardumont added a comment.Oct 7 2019, 2:45 PM

You forgot to fix the false negative

Yes, i was not really set on fixing it... since that does not happen right now.

python ./analysis.py --dataset ./tree.json.gz | grep -c "\.tar.'"

Note: i have committed the snippets/ardumont/gnu/analysis.py script and referenced it for your benefit ;)

oh well.

vlorentz added a comment.Oct 7 2019, 2:46 PM
grep '\.tar"' /tmp/tree.json
            {"type":"file","name":"rms-french-interview-2001-11-20.tar","size":23418880,"time":"1017450803"},
        {"type":"file","name":"gzip-1.2.4.tar","size":798720,"time":"745830000"},
        {"type":"file","name":"gzip-1.2.4a.tar","size":798720,"time":"936571444"},
        {"type":"file","name":"gzip-1.3.12.tar","size":1822720,"time":"1176517974"},
        {"type":"file","name":"gzip-1.3.9.tar","size":1648640,"time":"1166172244"},
          {"type":"file","name":"sather-doc-000328-src.sgml.tar","size":289548,"time":"954345818"},
        {"type":"file","name":"tar-1.13.tar","size":3942400,"time":"936572499"},
            {"type":"file","name":".tar","size":327680,"time":"1011303620"},
            {"type":"file","name":"ctrl2cap.vxd.tar","size":30720,"time":"844973673"},
            {"type":"file","name":"keyswap.tar","size":30720,"time":"844973673"},
                {"type":"file","name":"ctrl2cap.vxd.tar","size":30720,"time":"974996877"},
                {"type":"file","name":"keyswap.tar","size":30720,"time":"974996877"},
ardumont added a comment.Oct 7 2019, 2:55 PM

heh righty right, I was on the .tar. use case not .tar indeed.

.tar. still returns false now, .tar is ok.

ardumont updated this revision to Diff 6983.Oct 7 2019, 3:14 PM

Fix docstring indentation block and fix ".tar" filtering bug

swh-public-ci added a comment.Oct 7 2019, 3:16 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/391/ for more details.

Harbormaster completed remote builds in B8149: Diff 6983.Oct 7 2019, 3:16 PM
ardumont mentioned this in rDLDBASE44b155cb302b: package.gnu: Drop the unused package parameter.Oct 7 2019, 5:48 PM
ardumont mentioned this in T2032: listers: Add proper integration tests to lister.Oct 8 2019, 9:06 AM
ardumont mentioned this in D2082: Add pytest fixture to allow mocking requests with data file.Oct 8 2019, 10:44 AM
vlorentz added inline comments.Oct 8 2019, 11:29 AM
swh/lister/gnu/lister.py
72–82

still not indented

swh/lister/gnu/tests/test_tree.py
107–110

consistency is still not done

swh/lister/gnu/tree.py
48

://

110–123

still not indented

157

bool: Whether filename is an archive or not

vlorentz requested changes to this revision.Oct 8 2019, 11:29 AM
This revision now requires changes to proceed.Oct 8 2019, 11:29 AM
ardumont added inline comments.Oct 8 2019, 11:39 AM
swh/lister/gnu/tree.py
157

I mentioned the returned type annotation so that's not needed.

ardumont updated this revision to Diff 6994.Oct 8 2019, 11:46 AM

Fix indentation and assertion consistency

swh-public-ci added a comment.Oct 8 2019, 11:49 AM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/392/ for more details.

Harbormaster completed remote builds in B8164: Diff 6994.Oct 8 2019, 11:49 AM
ardumont updated this revision to Diff 6995.Oct 8 2019, 12:10 PM

Add some more adaptations according to review

swh-public-ci added a comment.Oct 8 2019, 12:13 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/393/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/393/console

Harbormaster failed remote builds in B8165: Diff 6995!Oct 8 2019, 12:13 PM
ardumont updated this revision to Diff 6996.Oct 8 2019, 12:54 PM

Fix for python3.5 build

swh-public-ci added a comment.Oct 8 2019, 12:56 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/394/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/394/console

Harbormaster failed remote builds in B8166: Diff 6996!Oct 8 2019, 12:56 PM
ardumont added a comment.Oct 8 2019, 1:02 PM

Build has FAILED

/me grunts
investigating the issue...

ardumont updated this revision to Diff 6997.Oct 8 2019, 1:06 PM

Fix build for python3.5 (for real this time)

swh-public-ci added a comment.Oct 8 2019, 1:09 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/395/ for more details.

Harbormaster completed remote builds in B8167: Diff 6997.Oct 8 2019, 1:09 PM
ardumont added inline comments.Oct 8 2019, 1:10 PM
swh/lister/gnu/tests/__init__.py
25

D2082 defines the generic fixture for this btw.

swh/lister/gnu/tests/test_tree.py
95–101

D2082 defines a "datadir" fixture for that.

vlorentz requested changes to this revision.Oct 8 2019, 1:15 PM
This revision now requires changes to proceed.Oct 8 2019, 1:15 PM
vlorentz added inline comments.Oct 8 2019, 1:19 PM
swh/lister/gnu/tree.py
110–123

still not :/

ardumont updated this revision to Diff 6999.Oct 8 2019, 1:28 PM

Fix last mis-indentation block

vlorentz accepted this revision.Oct 8 2019, 1:31 PM
swh-public-ci added a comment.Oct 8 2019, 1:31 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/396/ for more details.

Harbormaster completed remote builds in B8169: Diff 6999.Oct 8 2019, 1:31 PM
ardumont updated this revision to Diff 7059.Oct 9 2019, 6:04 PM

Reuse swh.core.pytest_plugin.mock_requests_datadir* fixture

Harbormaster failed remote builds in B8252: Diff 7059!Oct 9 2019, 6:04 PM
swh-public-ci added a comment.Oct 9 2019, 6:04 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/401/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/401/console

ardumont retitled this revision from gnu: Separate listing and parsing logic to gnu: Separate listing and parsing logic and add integration test.Oct 9 2019, 6:06 PM
ardumont removed a parent revision: D2075: simple_lister: Extract common behavior in base class.
ardumont edited the summary of this revision. (Show Details)
ardumont added a parent revision: D2077: cgit.tests: Check the tasks from the scheduler.
swh-public-ci added a comment.Oct 9 2019, 6:08 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/402/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/402/console

ardumont added a comment.Oct 9 2019, 6:08 PM

Build has FAILED

Because the parent revision changed to D2077

ardumont updated this revision to Diff 7060.Oct 9 2019, 6:13 PM

Plug to master branch (really trigger a new ci build)

Harbormaster failed remote builds in B8253: Diff 7060!Oct 9 2019, 6:13 PM
swh-public-ci added a comment.Oct 9 2019, 6:13 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/403/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/403/console

ardumont added a comment.Oct 9 2019, 6:16 PM

BUILD has failed

mmm

[tox] $ arc patch --diff 7060 --nocommit --nobranch --force --conduit-uri=https://forge.softwareheritage.org/ ********
Checking patch tox.ini...
error: while searching for:
  .[testing]
  pytest-cov
commands =
  pytest --cov=swh --cov-branch {posargs}

well, yes, jenkins, please try and load the right tox.ini, it will work better after that...

ardumont added a comment.EditedOct 9 2019, 6:20 PM

well, yes, jenkins, please try and load the right tox.ini, it will work better after that...

I've gotten back to D2077 (which is now the parent task).
I've removed the parent revision from D2077 (since everything it depended on landed on master).
And since the diff dependencies changed... (rebase and all that)

I've restarted the diff's build after that...
Let's see if that works better now.

swh-public-ci added a comment.Oct 9 2019, 6:20 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/404/ for more details.

Harbormaster completed remote builds in B8253: Diff 7060.Oct 9 2019, 6:20 PM
ardumont added a comment.Oct 9 2019, 6:21 PM

Build is green

\m/

ardumont updated this revision to Diff 7061.Oct 9 2019, 6:24 PM

Remove dead code

swh/lister/gnu/tests/__init__.py
25

This needs to go away, on it.

swh-public-ci added a comment.Oct 9 2019, 6:27 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/405/ for more details.

Harbormaster completed remote builds in B8254: Diff 7061.Oct 9 2019, 6:27 PM
ardumont updated this revision to Diff 7066.Oct 10 2019, 9:48 AM

Plug to latest master

swh-public-ci added a comment.Oct 10 2019, 9:51 AM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/406/ for more details.

Harbormaster completed remote builds in B8259: Diff 7066.Oct 10 2019, 9:51 AM
douardda accepted this revision.Oct 10 2019, 10:15 AM
This revision is now accepted and ready to land.Oct 10 2019, 10:15 AM
Closed by commit rDLS0f0b840178eb: gnu.tests: Checks lister output from scheduler (authored by ardumont). · Explain WhyOct 10 2019, 10:32 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLS0f0b840178eb: gnu.tests: Checks lister output from scheduler.
ardumont mentioned this in rDLSef2c1847e4f5: gnu.lister: Move tests datadir into its dedicated folder.Oct 10 2019, 11:50 AM

Revision Contents
Changeset List

PathSize
swh/
lister/
gnu/
221 lines
tests/
25 lines
105 lines
147 lines
187 lines

Diff 6996

View Options

swh/lister/gnu/lister.py

Loading...
View Options

swh/lister/gnu/tests/__init__.py

Loading...
View Options

swh/lister/gnu/tests/test_lister.py

Loading...
View Options

swh/lister/gnu/tests/test_tree.py

Loading...
View Options

swh/lister/gnu/tree.py

Loading...
About · Developer information · Legal