Page MenuHomeSoftware Heritage

Indexer for Packagist(composer.json)
ClosedPublic

Authored by VickyMerzOwn on Jun 29 2022, 12:57 PM.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30248
Build 47284: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 47283: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8047 (id=29034)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/288/ for more details.

Adds test for ideal behavior of indexer
Removes type attribute in Packagist column of crosswalk.

Build has FAILED

Patch application report for D8047 (id=29059)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/293/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/293/console

Adds composer mappings to metadata dictionary

Build is green

Patch application report for D8047 (id=29094)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/300/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/indexer/data/codemeta/crosswalk.csv
68 ↗(On Diff #29094)

please don't change this file, it should not diverge from codemeta:

  1. it would make it hard to merge when codemeta updates its crosswalk
  2. it makes the citation inaccurate
swh/indexer/metadata_dictionary/composer.py
37–49

please do it without intermediary variables like other mappings, it's simpler

This revision now requires changes to proceed.Jun 30 2022, 7:43 PM
swh/indexer/data/codemeta/crosswalk.csv
68 ↗(On Diff #29094)

Should I just put the table in a separate file that goes to the data directory?
(I'll put only the relevant columns in this file if so)

Composer mappings use a separate table

VickyMerzOwn marked an inline comment as done.

Undo changes to crosswalk.csv

Build is green

Patch application report for D8047 (id=29096)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit d45e779327d2508f5ba637e09aa6ac66edfd8b4f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:00:29 2022 +0530

    Adds a crosswalk for composer mappings

commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/301/ for more details.

swh/indexer/data/codemeta/crosswalk.csv
68 ↗(On Diff #29094)

hmm, you could put it in swh/indexer/data/crosswalks/Composer.csv.

this way it still looks somewhat like https://github.com/codemeta/codemeta/tree/master/crosswalks and it will be easier to compare after they merge

Build is green

Patch application report for D8047 (id=29097)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit de51688050da73f9b05dbfe77faca427abb061d3
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:04:18 2022 +0530

    Undo changes to crosswalk.csv

commit d45e779327d2508f5ba637e09aa6ac66edfd8b4f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:00:29 2022 +0530

    Adds a crosswalk for composer mappings

commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/302/ for more details.

Undo changes to crosswalk.csv

Build is green

Patch application report for D8047 (id=29098)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit 0d8fd371bea5ca5e1715a2333af64585d510e325
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:09:01 2022 +0530

    Undo changes to crosswalk.csv

commit de51688050da73f9b05dbfe77faca427abb061d3
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:04:18 2022 +0530

    Undo changes to crosswalk.csv

commit d45e779327d2508f5ba637e09aa6ac66edfd8b4f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:00:29 2022 +0530

    Adds a crosswalk for composer mappings

commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/303/ for more details.

swh/indexer/data/composer.csv
1 ↗(On Diff #29098)

only keep the Property and Packagist columns (and rename Packagist to Composer)

swh/indexer/metadata_dictionary/composer.py
1

add copyright header (sorry, I forgot to mention it before)

swh/indexer/data/composer.csv
1 ↗(On Diff #29098)

Have to leave the codemeta-v1 column too as it is asserted to be present in _read_crosstable(fd)

swh/indexer/metadata_dictionary/composer.py
1

no problem. in a moment.

Adds license to composer.py
Excludes redundant columns from composer.csv

Build is green

Patch application report for D8047 (id=29099)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit 4a28b00ab6bdd443659efb51a64b95afa87c6fc1
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 02:12:52 2022 +0530

    Changes to composer.csv and Adds license to composer.py

commit 0d8fd371bea5ca5e1715a2333af64585d510e325
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:09:01 2022 +0530

    Undo changes to crosswalk.csv

commit de51688050da73f9b05dbfe77faca427abb061d3
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:04:18 2022 +0530

    Undo changes to crosswalk.csv

commit d45e779327d2508f5ba637e09aa6ac66edfd8b4f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:00:29 2022 +0530

    Adds a crosswalk for composer mappings

commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/304/ for more details.

Build is green

Patch application report for D8047 (id=29100)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit bcaf0b9ffe9a2bfd5208a0218c668be79a8beb2e
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 02:41:59 2022 +0530

    Edit Copyright year

commit 4a28b00ab6bdd443659efb51a64b95afa87c6fc1
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 02:12:52 2022 +0530

    Changes to composer.csv and Adds license to composer.py

commit 0d8fd371bea5ca5e1715a2333af64585d510e325
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:09:01 2022 +0530

    Undo changes to crosswalk.csv

commit de51688050da73f9b05dbfe77faca427abb061d3
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:04:18 2022 +0530

    Undo changes to crosswalk.csv

commit d45e779327d2508f5ba637e09aa6ac66edfd8b4f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:00:29 2022 +0530

    Adds a crosswalk for composer mappings

commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/305/ for more details.

swh/indexer/data/composer.csv
1 ↗(On Diff #29098)

I think you can remove the assertion. it was useful while I was prototyping the mappings, but now this is well covered by tests

swh/indexer/metadata_dictionary/composer.py
55

that property does not exist https://schema.org/role

Excludes codemeta-v1 column from composer.csv

Build is green

Patch application report for D8047 (id=29101)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit 5905866f3685b7a7ccccc5d1655b2b1142852527
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 03:14:29 2022 +0530

    Exclude codemeta-v1 column from composer.csv

commit bcaf0b9ffe9a2bfd5208a0218c668be79a8beb2e
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 02:41:59 2022 +0530

    Edit Copyright year

commit 4a28b00ab6bdd443659efb51a64b95afa87c6fc1
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 02:12:52 2022 +0530

    Changes to composer.csv and Adds license to composer.py

commit 0d8fd371bea5ca5e1715a2333af64585d510e325
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:09:01 2022 +0530

    Undo changes to crosswalk.csv

commit de51688050da73f9b05dbfe77faca427abb061d3
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:04:18 2022 +0530

    Undo changes to crosswalk.csv

commit d45e779327d2508f5ba637e09aa6ac66edfd8b4f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:00:29 2022 +0530

    Adds a crosswalk for composer mappings

commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/306/ for more details.

Excludes author.role from composer.py

Build was aborted

Patch application report for D8047 (id=29104)

Rebasing onto 1be4e184d4...

Current branch diff-target is up to date.
Changes applied before test
commit 6446cd46846e8e2254a231fe3bca165aa147531a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 15:55:22 2022 +0530

    Excludes author.role from composer.py

commit 5905866f3685b7a7ccccc5d1655b2b1142852527
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 03:14:29 2022 +0530

    Exclude codemeta-v1 column from composer.csv

commit bcaf0b9ffe9a2bfd5208a0218c668be79a8beb2e
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 02:41:59 2022 +0530

    Edit Copyright year

commit 4a28b00ab6bdd443659efb51a64b95afa87c6fc1
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 02:12:52 2022 +0530

    Changes to composer.csv and Adds license to composer.py

commit 0d8fd371bea5ca5e1715a2333af64585d510e325
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:09:01 2022 +0530

    Undo changes to crosswalk.csv

commit de51688050da73f9b05dbfe77faca427abb061d3
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:04:18 2022 +0530

    Undo changes to crosswalk.csv

commit d45e779327d2508f5ba637e09aa6ac66edfd8b4f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Fri Jul 1 01:00:29 2022 +0530

    Adds a crosswalk for composer mappings

commit 919ec49908cb5cbc24dcb2b379924903419fdd6a
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 22:35:42 2022 +0530

    Adds working indxer for composer

commit 3d27e70f07058e875aba4bbef76daaac6cfe0188
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Thu Jun 30 02:18:29 2022 +0530

    Changes to crosswalk and test_metadata

commit d5615ed2e79e093421d87b5d3f2ddc3edf95aa89
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    Update crosswalk and define ComposerMapping

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/307/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/307/console

Build has FAILED

Patch application report for D8047 (id=29142)

Rebasing onto 61b22345e7...

Current branch diff-target is up to date.
Changes applied before test
commit 4e1e2617eb90fcdb6394333a739e1fe6cacb9c43
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    metadata_dictionary: Add mappings for composer.json

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/326/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/326/console

Fix failed build: Corrects test in test_composer

This revision is now accepted and ready to land.Jul 4 2022, 2:28 PM

Build was aborted

Patch application report for D8047 (id=29143)

Rebasing onto 61b22345e7...

Current branch diff-target is up to date.
Changes applied before test
commit 79499ea596586386efb884eb07520cf39c513ad7
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    metadata_dictionary: Add mappings for composer.json

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/327/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/327/console

This revision was landed with ongoing or failed builds.Jul 4 2022, 2:46 PM
This revision was automatically updated to reflect the committed changes.
This revision is now accepted and ready to land.Jul 4 2022, 11:05 PM

Add SingleFileMapping as parent class to ComposerMapping

Build is green

Patch application report for D8047 (id=29157)

Rebasing onto 79499ea596...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for composer.json
Using index info to reconstruct a base tree...
M	swh/indexer/codemeta.py
M	swh/indexer/metadata_dictionary/__init__.py
M	swh/indexer/tests/test_cli.py
Falling back to patching base and 3-way merge...
Auto-merging swh/indexer/metadata_dictionary/composer.py
CONFLICT (add/add): Merge conflict in swh/indexer/metadata_dictionary/composer.py
Patch failed at 0001 metadata_dictionary: Add mappings for composer.json

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Rebase failed (ret=1)!

Could not rebase; Attempt merge onto 79499ea596...

Already up to date.
Changes applied before test
commit fcd48a45a6305af70baff1d66de07d591a270929
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jun 29 16:25:28 2022 +0530

    metadata_dictionary: Add mappings for composer.json

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/328/ for more details.

you already landed this diff, please open a new one

you already landed this diff, please open a new one

It is too little to open a new diff. Only one line is different between now and when it was landed.
Should I still do it?
Build is green now!

yes. one diff = one commit, and this diff's commit is already in master

I'm sorry. I'll close this diff first.
The diff got updated without me specifying --update

it's because your current commit is not on master but is in this diff, so Arcanist assumed you meant to update. Use --create to explicitly ask it to create a diff

it's because your current commit is not on master but is in this diff, so Arcanist assumed you meant to update. Use --create to explicitly ask it to create a diff

I'll keep this in mind, thank you.
The new diff is D8078

Build is green

Patch application report for D8047 (id=29158)

Rebasing onto 79499ea596...

Current branch diff-target is up to date.
Changes applied before test
commit ff2730c3f6e2514454ef2a8d17c077f860d34545
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Tue Jul 5 03:00:41 2022 +0530

    ComposerMapping: Adds SingleFileMapping as parent class

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/329/ for more details.