Page MenuHomeSoftware Heritage
Differential D3069

scanner: json output format
ClosedPublic
Actions

Authored by DanSeraf on Apr 27 2020, 11:25 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDTSCN3f00bb004b4d: flat json output with known and swhid values
Summary

The json output now is flat and i've added the ndjson output format as @olasd suggested (Related T2363)

Diff Detail

Repository
rDTSCN Code scanner
Branch
json-output
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 12078
Build 18325: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 18324: arc lint + arc unit

Event Timeline

DanSeraf created this revision.Apr 27 2020, 11:25 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 27 2020, 11:25 AM
swh-public-ci added a comment.Apr 27 2020, 11:28 AM

Build is green

Patch application report for D3069 (id=10908)

Rebasing onto 3bca054d47...

Current branch diff-target is up to date.
Changes applied before test
commit e5f7cb3e9833e82725a0370481669adb8bd83f9b
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Mon Apr 27 11:13:36 2020 +0200

    new output format: ndjson

commit 015e87f66c25d19b83ea802488e838aac103fbcc
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Mon Apr 27 11:09:12 2020 +0200

    model: known attribute in Tree structure
    
    Now the scanner will save the "known" status while scanning
    a source code project. The swhid is saved also for unknown files.
    
    The view of the json format now is flat

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/10/ for more details.

Harbormaster completed remote builds in B12060: Diff 10908.Apr 27 2020, 11:28 AM
vlorentz added a subscriber: vlorentz.Apr 27 2020, 1:04 PM

Could you split this into two diffs?

DanSeraf added a parent revision: D3070: model: known attribute in Tree structure.Apr 27 2020, 4:06 PM
DanSeraf edited the summary of this revision. (Show Details)Apr 27 2020, 4:09 PM
DanSeraf added a comment.Apr 27 2020, 4:17 PM
In D3069#74580, @vlorentz wrote:

Could you split this into two diffs?

I've separated the diffs so i'll wait the other one (D3070) before updating this one.

DanSeraf updated this revision to Diff 10923.Apr 27 2020, 4:46 PM

parent

DanSeraf updated this revision to Diff 10925.Apr 27 2020, 4:54 PM

update

swh-public-ci added a comment.Apr 27 2020, 4:58 PM

Build is green

Patch application report for D3069 (id=10925)

Could not rebase; Attempt merge onto 3bca054d47...

Updating 3bca054..4f7fd3e
Fast-forward
 swh/scanner/model.py                               | 60 ++++++++++++++++++----
 swh/scanner/scanner.py                             | 10 ++--
 swh/scanner/tests/conftest.py                      |  6 +--
 swh/scanner/tests/data.py                          |  5 +-
 .../data/sample-folder-result-no-toexclude.json    | 12 -----
 swh/scanner/tests/data/sample-folder-result.json   | 13 -----
 swh/scanner/tests/flask_api.py                     | 14 ++---
 swh/scanner/tests/test_model.py                    | 40 ++++++++-------
 swh/scanner/tests/test_scanner.py                  | 30 +++++------
 9 files changed, 100 insertions(+), 90 deletions(-)
 delete mode 100644 swh/scanner/tests/data/sample-folder-result-no-toexclude.json
 delete mode 100644 swh/scanner/tests/data/sample-folder-result.json
Changes applied before test
commit 4f7fd3e0599c60e8d0adeb9e8f66e3c3f2bb60d7
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Mon Apr 27 16:00:10 2020 +0200

    scanner tests adapted to the new attribute on the model

commit 66d6394ba20e452771c47b99deea1b8f6f3e7352
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Mon Apr 27 15:55:01 2020 +0200

    model: known attribute in Tree structure
    
    The scanner now has the "known" attribute and the
    Software Heritage persistent identifier is saved
    also for unknown files.
    
    model tests adapted to the new attribute

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/17/ for more details.

Harbormaster completed remote builds in B12078: Diff 10925.Apr 27 2020, 4:58 PM
vlorentz added a comment.Apr 27 2020, 5:01 PM

you pushed the wrong commit to this diff

DanSeraf updated this revision to Diff 10926.Apr 27 2020, 5:18 PM

update

swh-public-ci added a comment.Apr 27 2020, 5:20 PM

Build is green

Patch application report for D3069 (id=10926)

Could not rebase; Attempt merge onto 3bca054d47...

Updating 3bca054..ca45546
Fast-forward
 swh/scanner/model.py                               | 67 ++++++++++++++--------
 swh/scanner/scanner.py                             | 10 ++--
 swh/scanner/tests/conftest.py                      |  6 +-
 swh/scanner/tests/data.py                          |  5 +-
 .../data/sample-folder-result-no-toexclude.json    | 12 ----
 swh/scanner/tests/data/sample-folder-result.json   | 13 -----
 swh/scanner/tests/flask_api.py                     | 14 ++---
 swh/scanner/tests/test_model.py                    | 40 +++++++------
 swh/scanner/tests/test_scanner.py                  | 30 ++++------
 9 files changed, 93 insertions(+), 104 deletions(-)
 delete mode 100644 swh/scanner/tests/data/sample-folder-result-no-toexclude.json
 delete mode 100644 swh/scanner/tests/data/sample-folder-result.json
Changes applied before test
commit ca455463e8bc81705637d1690b1efa5a460efae6
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Mon Apr 27 17:01:40 2020 +0200

    flat json output with known and swhid values

commit 80812af59952830df0462de14a0e90d86bde66d7
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Mon Apr 27 15:55:01 2020 +0200

    model: known attribute in Tree structure
    
    The scanner now has the "known" attribute and the
    Software Heritage persistent identifier is saved
    also for unknown files.
    
    tests adapted to the new attribute on the model

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/18/ for more details.

Harbormaster completed remote builds in B12079: Diff 10926.Apr 27 2020, 5:20 PM
vlorentz accepted this revision.Apr 28 2020, 4:55 PM
This revision is now accepted and ready to land.Apr 28 2020, 4:55 PM
DanSeraf updated this revision to Diff 10974.Apr 29 2020, 12:56 PM

rebase

This revision was landed with ongoing or failed builds.Apr 29 2020, 12:57 PM
Closed by commit rDTSCN3f00bb004b4d: flat json output with known and swhid values (authored by DanSeraf). · Explain Why
This revision was automatically updated to reflect the committed changes.
DanSeraf added a commit: rDTSCN3f00bb004b4d: flat json output with known and swhid values.
swh-public-ci added a comment.Apr 29 2020, 12:58 PM

Build is green

Patch application report for D3069 (id=10974)

Rebasing onto bbf296f752...

Current branch diff-target is up to date.
Changes applied before test
commit 3f00bb004b4db54659a8b60c7a14851fae2e1a18
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Wed Apr 29 12:53:26 2020 +0200

    flat json output with known and swhid values

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/20/ for more details.

Harbormaster completed remote builds in B12127: Diff 10974.Apr 29 2020, 12:58 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
scanner/
tests/
6 lines100%
Loading...
5 lines100%
Loading...
data/
--
--
14 lines46%
Loading...
30 lines100%
Loading...

Diff 10925

View Options

swh/scanner/tests/conftest.py

Loading...
View Options

swh/scanner/tests/data.py

Loading...
View Options

swh/scanner/tests/data/sample-folder-result-no-toexclude.json

Loading...
View Options

swh/scanner/tests/data/sample-folder-result.json

Loading...
View Options

swh/scanner/tests/flask_api.py

Loading...
View Options

swh/scanner/tests/test_scanner.py

Loading...
About · Developer information · Legal