Differential D7956

Provenance: Docker for origin layer + logging improvement
ClosedPublic
Actions

Authored by vsellier on Jun 3 2022, 11:44 AM.

Details

Reviewers

ardumont

Group Reviewers

System administrators

Maniphest Tasks

T4302: provenance: origin processed several time by different clients

Commits

rDPROV5cbbb9a333b4: exclude local .venv from git changes
rDPROVd45d3628c831: Add the missing zmq dependency needed for the csv servers
rDPROVbfd7cac5c89e: Initiate a quick and dirty docker environment
rDPROV13d6089d6cda: Improve origin layer logging

Summary

Add the missing zmq dependency needed for the csv servers
Initiate a quick and dirty docker environment For now, it starts the provenance storage + the origin layer part It's full of shortcuts but it will help to reproduce the problem declared in the related issue
Exclude local .venv from git changes

Related to T4302

Diff Detail

Repository

rDPROV Provenance database

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vsellier created this revision.Jun 3 2022, 11:44 AM

Build is green

Patch application report for D7956 (id=28646)

Rebasing onto 101a9a0aa2...

Current branch diff-target is up to date.

Changes applied before test

commit fa0c28f3be366c48c03c77e264010fb1d03f9a48
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:43:01 2022 +0200

    exclude local .venv from git changes

commit 9dcb4fa47cc6eeb33d54a0f8ab402e140e45fe0a
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:32:30 2022 +0200

    Initiate a quick and dirty docker environment
    
    For now, it starts the provenance storage + the origin layer part
    It's full of shortcuts but it will help to reproduce the problem declared
    in the related issue
    
    Related to T3402

commit d45d3628c831927f9ecbd7881524452584d7c2c1
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:39:01 2022 +0200

    Add the missing zmq dependency needed for the csv servers

commit 13d6089d6cda3a3fa8449af05f60f43ac27f9592
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:41:15 2022 +0200

    Improve origin layer logging
    
    Related to T4302

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/597/ for more details.

Harbormaster completed remote builds in B29729: Diff 28646.Jun 3 2022, 11:55 AM

vsellier requested review of this revision.Jun 3 2022, 11:55 AM

lgtm

one remark inline.

docker-compose.yml
9 ↗	(On Diff #28646)	you don't need it since you define it in the postgresql.conf override already.
docker/postgresql/postgresql.conf
2 ↗	(On Diff #28646)	already configured here so -c max-connection can go away in docker-compose.yml

This revision is now accepted and ready to land.Jun 3 2022, 2:22 PM

vsellier added inline comments.Jun 3 2022, 5:57 PM

docker-compose.yml
9 ↗	(On Diff #28646)	right, good catch I'm wondering if removing the config file and just keeping this should not be better

only use the postgresql paramater to configure the max connection

This revision was landed with ongoing or failed builds.Jun 3 2022, 7:52 PM

Closed by commit rDPROV13d6089d6cda: Improve origin layer logging (authored by vsellier). · Explain Why

This revision was automatically updated to reflect the committed changes.

vsellier added a commit: rDPROV13d6089d6cda: Improve origin layer logging.

vsellier added a commit: rDPROVd45d3628c831: Add the missing zmq dependency needed for the csv servers.

vsellier added a commit: rDPROVbfd7cac5c89e: Initiate a quick and dirty docker environment.

vsellier added a commit: rDPROV5cbbb9a333b4: exclude local .venv from git changes.

Build was aborted

Patch application report for D7956 (id=28686)

Rebasing onto 101a9a0aa2...

Current branch diff-target is up to date.

Changes applied before test

commit 5cbbb9a333b4c9ebc9b0305b7147a184a870b323
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:43:01 2022 +0200

    exclude local .venv from git changes

commit bfd7cac5c89e6b95b7a1b45d7ab6eb68249fb201
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:32:30 2022 +0200

    Initiate a quick and dirty docker environment
    
    For now, it starts the provenance storage + the origin layer part
    It's full of shortcuts but it will help to reproduce the problem declared
    in the related issue
    
    Related to T3402

commit d45d3628c831927f9ecbd7881524452584d7c2c1
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:39:01 2022 +0200

    Add the missing zmq dependency needed for the csv servers

commit 13d6089d6cda3a3fa8449af05f60f43ac27f9592
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Fri Jun 3 11:41:15 2022 +0200

    Improve origin layer logging
    
    Related to T4302

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/598/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/598/console

Harbormaster failed remote builds in B29768: Diff 28686!Jun 3 2022, 8:04 PM