Page MenuHomeSoftware Heritage
Differential D1564

Manipulate origin URLs instead of origin ids.
ClosedPublic
Actions

Authored by vlorentz on Jun 7 2019, 4:23 PM.

Details

Reviewers
douardda
ardumont
Group Reviewers
Reviewers
Commits
rDCIDX8887a7d51d83: Manipulate origin URLs instead of origin ids.
Summary

Depends on D1559 (allows querying the storage without an origin id).

(This diff looks huge, but most of it is just updating all the test data)

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
origin-urls
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 6303
Build 8727: tox-on-jenkinsJenkins
Build 8726: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jun 7 2019, 4:23 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 7 2019, 4:23 PM
vlorentz edited the summary of this revision. (Show Details)Jun 7 2019, 4:24 PM
vlorentz added a parent revision: D1559: Add support for using URL instead of ID in snapshot_get_latest..
vlorentz removed a parent revision: D1559: Add support for using URL instead of ID in snapshot_get_latest..
swh-public-ci added a comment.Jun 7 2019, 4:26 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/542/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/542/console

Harbormaster failed remote builds in B6108: Diff 5150!Jun 7 2019, 4:26 PM
vlorentz updated this revision to Diff 5151.Jun 7 2019, 4:32 PM
vlorentz added a parent revision: D1559: Add support for using URL instead of ID in snapshot_get_latest..

bump dep on swh-storage.

swh-public-ci added a comment.Jun 7 2019, 4:33 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/543/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/543/console

Harbormaster failed remote builds in B6109: Diff 5151!Jun 7 2019, 4:33 PM
vlorentz removed a parent revision: D1559: Add support for using URL instead of ID in snapshot_get_latest..Jun 11 2019, 10:01 AM
vlorentz updated this revision to Diff 5165.Jun 11 2019, 10:20 AM
vlorentz added a parent revision: D1559: Add support for using URL instead of ID in snapshot_get_latest..

update schema version.

swh-public-ci added a comment.Jun 11 2019, 10:21 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/545/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/545/console

Harbormaster failed remote builds in B6126: Diff 5165!Jun 11 2019, 10:22 AM
douardda requested changes to this revision.Jun 13 2019, 2:26 PM
douardda added a subscriber: douardda.
douardda added inline comments.
swh/indexer/origin_head.py
46–49

why changing the behavior of head selection mechanism here?

why cannot we know at this point which type the processed origin is?

In all cases, if this try-based solution is now mandatory, it would be nice to have it encapsulated in a generic self.get_head() method IMHO.

This revision now requires changes to proceed.Jun 13 2019, 2:26 PM
vlorentz planned changes to this revision.Jun 13 2019, 2:41 PM
vlorentz added inline comments.
swh/indexer/origin_head.py
46–49

why cannot we know at this point which type the processed origin is?

That requires a new API endpoint in the storage, but indeed, we could (and should).

swh-public-ci added a comment.Jun 13 2019, 2:56 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/546/ for more details.

Harbormaster completed remote builds in B6126: Diff 5165.Jun 13 2019, 2:56 PM
vlorentz added inline comments.Jun 14 2019, 10:34 AM
swh/indexer/origin_head.py
46–49

D1581

vlorentz updated this revision to Diff 5247.Jun 14 2019, 3:05 PM
  • Drop origin ids from tests as well
  • Use new-style snapshot_add in the tests (long overdue!)
  • Use origin_visit_get_latest instead of snapshot_get_latest (which is deprecated), in order to know the visit type.
vlorentz updated this revision to Diff 5248.Jun 14 2019, 3:08 PM

add missing aliases.

swh-public-ci added a comment.Jun 14 2019, 3:10 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/548/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/548/console

Harbormaster failed remote builds in B6220: Diff 5247!Jun 14 2019, 3:10 PM
swh-public-ci added a comment.Jun 14 2019, 3:15 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/549/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/549/console

Harbormaster failed remote builds in B6221: Diff 5248!Jun 14 2019, 3:15 PM
vlorentz mentioned this in T1816: Make all clients of swh-storage use origin URL as identifier and use visit types instead of origin types.Jun 17 2019, 4:20 PM
vlorentz updated this revision to Diff 5320.Jun 18 2019, 5:34 PM

also patch the storage.

Harbormaster failed remote builds in B6303: Diff 5320!Jun 18 2019, 5:35 PM
swh-public-ci added a comment.Jun 18 2019, 5:35 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/551/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/551/console

vlorentz updated this revision to Diff 5321.Jun 18 2019, 5:36 PM

bump dependency version

Harbormaster failed remote builds in B6304: Diff 5321!Jun 18 2019, 5:36 PM
swh-public-ci added a comment.Jun 18 2019, 5:36 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/552/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/552/console

vlorentz updated this revision to Diff 5322.Jun 18 2019, 5:37 PM

rebase

vlorentz edited the summary of this revision. (Show Details)Jun 18 2019, 5:38 PM
swh-public-ci added a comment.Jun 18 2019, 5:38 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/553/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/553/console

Harbormaster failed remote builds in B6305: Diff 5322!Jun 18 2019, 5:38 PM
ardumont requested changes to this revision.Jun 19 2019, 10:32 AM
ardumont added a subscriber: ardumont.

Sounds good.

I'm missing a migration script for the actual data to backfill the new origin-url from the existing rows.
Or am i missing something?

Request changes for the sake of discussion on that point.

Cheers,

swh/indexer/sql/40-swh-func.sql
426

Did not read the rest yet... how are we backfilling the existing rows in the indexer db? Is there a script for that?

swh/indexer/storage/__init__.py
771

Unify with the other docstring one way (append : Url of the origin) or the other (drop the redundant definition)...

/me singing You've got the power! tududu du du du tududu du du du ;)

(~> lookup snap music if you don't grok that ;)

This revision now requires changes to proceed.Jun 19 2019, 10:32 AM
vlorentz updated this revision to Diff 5331.Jun 19 2019, 10:48 AM

unify docstrings

swh-public-ci added a comment.Jun 19 2019, 10:50 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/554/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/554/console

Harbormaster failed remote builds in B6314: Diff 5331!Jun 19 2019, 10:50 AM
vlorentz added inline comments.Jun 19 2019, 11:03 AM
swh/indexer/sql/40-swh-func.sql
426
  1. deploy indexers with that patch
  2. run a full pass on all origins (which I planned on doing anyway, since we had to drop the task queue)
  3. delete rows without an origin_url if any (these would be origins that had metadata but no longer do)
vlorentz marked an inline comment as done.Jun 19 2019, 11:04 AM
swh-public-ci added a comment.Jun 19 2019, 11:04 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/555/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/555/console

ardumont added a comment.Jun 19 2019, 11:11 AM

Don't know why the build now.

swh/indexer/sql/40-swh-func.sql
426

Sounds fine to me, thanks.

vlorentz added a comment.Jun 19 2019, 11:12 AM

Because Jenkins timeouted while compiling the package so it did not send it to PyPI the first time. I'm triggering a rebuild

swh-public-ci added a comment.Jun 19 2019, 11:18 AM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/556/ for more details.

Harbormaster completed remote builds in B6314: Diff 5331.Jun 19 2019, 11:18 AM
ardumont accepted this revision.Jun 19 2019, 11:43 AM
douardda accepted this revision.Jun 24 2019, 3:23 PM
This revision is now accepted and ready to land.Jun 24 2019, 3:23 PM
vlorentz updated this revision to Diff 5462.Jun 24 2019, 4:36 PM

rebase

This revision was landed with ongoing or failed builds.Jun 24 2019, 4:36 PM
Closed by commit rDCIDX8887a7d51d83: Manipulate origin URLs instead of origin ids. (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
Harbormaster failed remote builds in B6449: Diff 5462!Jun 24 2019, 4:36 PM
swh-public-ci added a comment.Jun 24 2019, 4:36 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/557/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/557/console

Revision Contents
Changeset List

PathSize
2 lines
sql/
upgrades/
11 lines
swh/
indexer/
50 lines
10 lines
27 lines
sql/
3 lines
8 lines
storage/
17 lines
2 lines
1 line
tests/
storage/
25 lines
1 line
83 lines
39 lines
202 lines

Diff 5320

View Options

requirements-swh.txt

Loading...
View Options

sql/upgrades/125.sql

Loading...
View Options

swh/indexer/indexer.py

Loading...
View Options

swh/indexer/metadata.py

Loading...
View Options

swh/indexer/origin_head.py

Loading...
View Options

swh/indexer/sql/30-swh-schema.sql

Loading...
View Options

swh/indexer/sql/40-swh-func.sql

Loading...
View Options

swh/indexer/storage/__init__.py

Loading...
View Options

swh/indexer/storage/db.py

Loading...
View Options

swh/indexer/storage/in_memory.py

Loading...
View Options

swh/indexer/tests/storage/test_storage.py

Loading...
View Options

swh/indexer/tests/test_cli.py

Loading...
View Options

swh/indexer/tests/test_origin_head.py

Loading...
View Options

swh/indexer/tests/test_origin_metadata.py

Loading...
View Options

swh/indexer/tests/utils.py

Loading...
About · Developer information · Legal