Page MenuHomeSoftware Heritage
Differential D5021

storage: Refactor OriginVisitStatus instantiation
ClosedPublic
Actions

Authored by ardumont on Feb 4 2021, 5:41 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDSTOe0e88b2f579e: storage: Refactor OriginVisitStatus instantiation
Summary

Depends on D5020

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 19017
Build 29472: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 29471: arc lint + arc unit

Event Timeline

ardumont created this revision.Feb 4 2021, 5:41 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 4 2021, 5:41 PM
swh-public-ci added a comment.Feb 4 2021, 5:55 PM

Build is green

Patch application report for D5021 (id=17898)

Could not rebase; Attempt merge onto 9a9f234e0a...

Updating 9a9f234e..abb7184d
Fast-forward
 sql/upgrades/167.sql              | 11 +++++++++++
 swh/storage/backfill.py           | 10 ++--------
 swh/storage/postgresql/db.py      | 18 ++++++------------
 swh/storage/postgresql/storage.py | 31 ++++++-------------------------
 swh/storage/sql/30-schema.sql     |  4 ++--
 5 files changed, 27 insertions(+), 47 deletions(-)
 create mode 100644 sql/upgrades/167.sql
Changes applied before test
commit abb7184dc49cdd35d57d7f357ea76357b5745f73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:16:57 2021 +0100

    storage: Refactor OriginVisitStatus instantiation

commit 4b4f423bf571ce3c625c5feb1733b189e1a25877
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:11:18 2021 +0100

    db: Unify sql joins on origin_visit_status using "USING"

commit 3ad3ac12d46285bb1befd91806f4986b74da7812
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:08:33 2021 +0100

    storage.postgresql: Use origin_visit_status.type value as source
    
    This stops using the origin_visit.type as fallback values as now, the database has been
    migrated.
    
    So this makes the origin_visit_status.type a not nullable column.
    
    This also drops now redundant join instructions on origin_visit table when reading.
    
    Related to T2968

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1128/ for more details.

Harbormaster completed remote builds in B19017: Diff 17898.Feb 4 2021, 5:55 PM
ardumont requested review of this revision.Feb 4 2021, 5:55 PM
vlorentz accepted this revision.Feb 4 2021, 6:12 PM
vlorentz added a subscriber: vlorentz.

nice

This revision is now accepted and ready to land.Feb 4 2021, 6:12 PM
ardumont updated this revision to Diff 17911.Feb 5 2021, 11:16 AM

Rebase

swh-public-ci added a comment.Feb 5 2021, 12:17 PM

Build was aborted

Patch application report for D5021 (id=17911)

Could not rebase; Attempt merge onto efd8815b89...

Updating efd8815b..89cf1e51
Fast-forward
 sql/upgrades/167.sql              | 11 +++++++++++
 swh/storage/backfill.py           | 10 ++--------
 swh/storage/postgresql/db.py      | 18 ++++++------------
 swh/storage/postgresql/storage.py | 31 ++++++-------------------------
 swh/storage/sql/30-schema.sql     |  4 ++--
 5 files changed, 27 insertions(+), 47 deletions(-)
 create mode 100644 sql/upgrades/167.sql
Changes applied before test
commit 89cf1e51cbbc01d61c159dcd24f98f4afd834953
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:16:57 2021 +0100

    storage: Refactor OriginVisitStatus instantiation

commit 37df1a5c75a5cd7528b9f155cb184e9b5a6d1cfa
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:11:18 2021 +0100

    db: Unify sql joins on origin_visit_status using "USING"

commit 67bc71ddd9d59a21fb4e5270f29d9cb67d0f711a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:08:33 2021 +0100

    storage.postgresql: Use origin_visit_status.type value as source
    
    This stops using the origin_visit.type as fallback values as now, the database has been
    migrated.
    
    So this makes the origin_visit_status.type a not nullable column.
    
    This also drops now redundant join instructions on origin_visit table when reading.
    
    Related to T2968

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1133/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1133/console

Harbormaster failed remote builds in B19030: Diff 17911!Feb 5 2021, 12:17 PM
ardumont updated this revision to Diff 18009.Feb 9 2021, 5:22 PM

Rebase

swh-public-ci added a comment.Feb 9 2021, 5:36 PM

Build is green

Patch application report for D5021 (id=18009)

Could not rebase; Attempt merge onto 046fe57c15...

Updating 046fe57c..e0e88b2f
Fast-forward
 swh/storage/postgresql/db.py      | 12 ++++--------
 swh/storage/postgresql/storage.py | 31 ++++++-------------------------
 2 files changed, 10 insertions(+), 33 deletions(-)
Changes applied before test
commit e0e88b2f579ea73132af04eff32ae32a95cb3a2f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:16:57 2021 +0100

    storage: Refactor OriginVisitStatus instantiation

commit d30ca93839655c1d1310a529dc8df3568afcab2f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 4 17:11:18 2021 +0100

    db: Unify sql joins on origin_visit_status using "USING"

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1153/ for more details.

Harbormaster completed remote builds in B19129: Diff 18009.Feb 9 2021, 5:36 PM
Closed by commit rDSTOe0e88b2f579e: storage: Refactor OriginVisitStatus instantiation (authored by ardumont). · Explain WhyFeb 9 2021, 5:37 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTOe0e88b2f579e: storage: Refactor OriginVisitStatus instantiation.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
storage/
postgresql/
31 lines95%
Loading...

Diff 17898

View Options

swh/storage/postgresql/storage.py

  • This file was added.
{
"name": "swh-search-query-language-parser",
"version": "1.0.0",
"description": "Parser for Software Heritage archive search query language",
"scripts": {
"generate": "cd query_language && tree-sitter generate --no-bindings",
"dev": "yarn generate && cd query_language && tree-sitter parse sample_query",
"test": "yarn generate && cd query_language && tree-sitter test",
anlambertUnsubmitted
Not Done
Inline Actions

You should use the --no-bindings option of tree-sitter generate, it will not generate node and rust bindings we do not need here.

Also can you add the generation of the wasm module for the parser in the generate script ?
It will enable to use the parser in swh-web using web-tree-sitter.

anlambert: You should use the `--no-bindings` option of `tree-sitter generate`, it will not generate node…
"build-so": "yarn generate && cd query_language && python3 build.py",
"build-wasm": "yarn generate && cd query_language && tree-sitter build-wasm && mv tree-sitter-swh_search_query_language.wasm swh_ql.wasm",
"build": "yarn build-so && yarn build-wasm",
"repl": "yarn generate && cd query_language && tree-sitter build-wasm && tree-sitter playground"
},
"repository": {
"type": "git",
"url": "https://forge.softwareheritage.org/source/swh-search.git"
},
"keywords": [
"swh",
"Software Heritage",
"treesitter",
"parser",
"custom",
"search",
"query",
vlorentzUnsubmitted
Done
Inline Actions
"language"
],
- "author": "Software Heritage",
+ "author": "The Software Heritage developers",
"license": "GPL-2.0-only",
vlorentz:
"language"
KShivenduAuthorUnsubmitted
Done
Inline Actions

Are the values of name, description, repository.url, keywords, author and license correct?

KShivendu: Are the values of `name, description, repository.url, keywords, author and license` correct?
vlorentzUnsubmitted
Done
Inline Actions

yes, as far as I can tell

vlorentz: yes, as far as I can tell
vlorentzUnsubmitted
Done
Inline Actions
"author": "Software Heritage",
- "license": "GPL-2.0-only",
+ "license": "GPL-3.0-only",
"dependencies": {
vlorentz:
],
"author": "The Software Heritage developers",
"license": "GPL-3.0-only",
"dependencies": {
"nan": "^2.14.2"
},
"devDependencies": {
"tree-sitter-cli": "^0.20.0"
}
}
No newline at end of file
About · Developer information · Legal