Page MenuHomeSoftware Heritage

Provenance: Docker for origin layer + logging improvement
ClosedPublic

Authored by vsellier on Jun 3 2022, 11:44 AM.

Details

Summary
  • Add the missing zmq dependency needed for the csv servers
  • Initiate a quick and dirty docker environment For now, it starts the provenance storage + the origin layer part It's full of shortcuts but it will help to reproduce the problem declared in the related issue
  • Exclude local .venv from git changes

Related to T4302

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D7956 (id=28646)

Rebasing onto 101a9a0aa2...

Current branch diff-target is up to date.
Changes applied before test
commit fa0c28f3be366c48c03c77e264010fb1d03f9a48
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:43:01 2022 +0200

    exclude local .venv from git changes

commit 9dcb4fa47cc6eeb33d54a0f8ab402e140e45fe0a
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:32:30 2022 +0200

    Initiate a quick and dirty docker environment
    
    For now, it starts the provenance storage + the origin layer part
    It's full of shortcuts but it will help to reproduce the problem declared
    in the related issue
    
    Related to T3402

commit d45d3628c831927f9ecbd7881524452584d7c2c1
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:39:01 2022 +0200

    Add the missing zmq dependency needed for the csv servers

commit 13d6089d6cda3a3fa8449af05f60f43ac27f9592
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:41:15 2022 +0200

    Improve origin layer logging
    
    Related to T4302

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/597/ for more details.

ardumont added a subscriber: ardumont.

lgtm

one remark inline.

docker-compose.yml
9 ↗(On Diff #28646)

you don't need it since you define it in the postgresql.conf override already.

docker/postgresql/postgresql.conf
2 ↗(On Diff #28646)

already configured here so -c max-connection can go away in docker-compose.yml

This revision is now accepted and ready to land.Jun 3 2022, 2:22 PM
docker-compose.yml
9 ↗(On Diff #28646)

right, good catch
I'm wondering if removing the config file and just keeping this should not be better

only use the postgresql paramater to configure the max connection

Build was aborted

Patch application report for D7956 (id=28686)

Rebasing onto 101a9a0aa2...

Current branch diff-target is up to date.
Changes applied before test
commit 5cbbb9a333b4c9ebc9b0305b7147a184a870b323
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:43:01 2022 +0200

    exclude local .venv from git changes

commit bfd7cac5c89e6b95b7a1b45d7ab6eb68249fb201
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:32:30 2022 +0200

    Initiate a quick and dirty docker environment
    
    For now, it starts the provenance storage + the origin layer part
    It's full of shortcuts but it will help to reproduce the problem declared
    in the related issue
    
    Related to T3402

commit d45d3628c831927f9ecbd7881524452584d7c2c1
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:39:01 2022 +0200

    Add the missing zmq dependency needed for the csv servers

commit 13d6089d6cda3a3fa8449af05f60f43ac27f9592
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:41:15 2022 +0200

    Improve origin layer logging
    
    Related to T4302

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/598/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/598/console