Page MenuHomeSoftware Heritage
Differential D8402

conftest: Refactor GraphServerProcess to be more flexible
ClosedPublic
Actions

Authored by vlorentz on Sep 6 2022, 5:03 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T4497: [sentry] Out of disk space
Commits
rDGRPH81d3deb2f1c1: conftest: Refactor GraphServerProcess to be more flexible
Summary
  1. queue is now encapsulated
  2. object in the queue is now a dict instead of a tuple (to allow adding more keys easily in the future + for readability)

This will be used in a future commit (which will address T4497),
to get the queue's object twice from different functions.

Diff Detail

Unit TestsFailed
View All

TimeTest
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_grpc::test_leaves
file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/test_grpc.py, line 38 def test_leaves(graph_grpc_stub): file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/conftest.py, line 59
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_grpc::test_neighbors
file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/test_grpc.py, line 56 def test_neighbors(graph_grpc_stub): file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/conftest.py, line 59
32 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_grpc::test_stats
file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/test_grpc.py, line 22 def test_stats(graph_grpc_stub): file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/conftest.py, line 59
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_grpc::test_visit_nodes
file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/test_grpc.py, line 75 def test_visit_nodes(graph_grpc_stub): file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/conftest.py, line 59
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_grpc::test_visit_nodes_filtered
file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/test_grpc.py, line 92 def test_visit_nodes_filtered(graph_grpc_stub): file /var/lib/jenkins/workspace/DGRPH/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/conftest.py, line 59
View Full Test Results (17 Failed · 13 Passed · 8 Skipped)

Event Timeline

vlorentz created this revision.Sep 6 2022, 5:03 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 6 2022, 5:03 PM
swh-public-ci added a comment.Sep 6 2022, 5:10 PM

Build has FAILED

Patch application report for D8402 (id=30322)

Rebasing onto dd88f177c3...

Current branch diff-target is up to date.
Changes applied before test
commit 2104133e41a924cf7d2678ac829b5bee44ba2138
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Sep 6 17:02:22 2022 +0200

    conftest: Refactor GraphServerProcess to be more flexible
    
    1. queue is now encapsulated
    2. object in the queue is now a dict instead of a tuple (to allow
       adding more keys easily in the future + for readability)
    
    This will be used in a future commit, to get the queue's object
    twice from different functions.

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/218/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/218/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 6 2022, 5:10 PM
Harbormaster failed remote builds in B31350: Diff 30322!
vlorentz added a child revision: D8403: Kill GraphServerProcess on test teardown.Sep 6 2022, 5:15 PM
vlorentz updated this revision to Diff 30325.Sep 6 2022, 5:19 PM

move hunks around

swh-public-ci added a comment.Sep 6 2022, 5:25 PM

Build has FAILED

Patch application report for D8402 (id=30325)

Rebasing onto dd88f177c3...

Current branch diff-target is up to date.
Changes applied before test
commit 46255335308446d7b077099fd9b1c68726faf140
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Sep 6 17:02:22 2022 +0200

    conftest: Refactor GraphServerProcess to be more flexible
    
    1. queue is now encapsulated
    2. object in the queue is now a dict instead of a tuple (to allow
       adding more keys easily in the future + for readability)
    
    This will be used in a future commit, to get the queue's object
    twice from different functions.

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/221/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/221/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 6 2022, 5:25 PM
Harbormaster failed remote builds in B31353: Diff 30325!
vlorentz updated this revision to Diff 30329.Sep 6 2022, 5:28 PM

meh

vlorentz added a task: T4497: [sentry] Out of disk space.Sep 6 2022, 5:31 PM
swh-public-ci added a comment.Sep 6 2022, 5:34 PM

Build is green

Patch application report for D8402 (id=30329)

Rebasing onto dd88f177c3...

Current branch diff-target is up to date.
Changes applied before test
commit 81d3deb2f1c1b584a076484250cb6f349009f032
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Sep 6 17:02:22 2022 +0200

    conftest: Refactor GraphServerProcess to be more flexible
    
    1. queue is now encapsulated
    2. object in the queue is now a dict instead of a tuple (to allow
       adding more keys easily in the future + for readability)
    
    This will be used in a future commit, to get the queue's object
    twice from different functions.

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/224/ for more details.

Harbormaster completed remote builds in B31357: Diff 30329.Sep 6 2022, 5:34 PM
vlorentz requested review of this revision.Sep 6 2022, 5:34 PM
olasd accepted this revision.Sep 6 2022, 5:36 PM
olasd added a subscriber: olasd.

Neat

This revision is now accepted and ready to land.Sep 6 2022, 5:36 PM
Closed by commit rDGRPH81d3deb2f1c1: conftest: Refactor GraphServerProcess to be more flexible (authored by vlorentz). · Explain WhySep 6 2022, 5:54 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDGRPH81d3deb2f1c1: conftest: Refactor GraphServerProcess to be more flexible.

Revision Contents
Changeset List

PathSize
swh/
graph/
tests/
34 lines

Diff 30322

View Options

swh/graph/tests/conftest.py

# Copyright (C) 2015 The Software Heritage developers # Copyright (C) 2015 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import configparser import configparser
import logging
import os import os
import yaml import yaml
SWH_CONFIG_DIRECTORIES = [ SWH_CONFIG_DIRECTORIES = [
'~/.config/swh', '~/.config/swh',
'~/.swh', '~/.swh',
'/etc/softwareheritage', '/etc/softwareheritage',
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Lines
def read_raw_config(base_config_path): def read_raw_config(base_config_path):
"""Read the raw config corresponding to base_config_path. """Read the raw config corresponding to base_config_path.
Can read yml or ini files. Can read yml or ini files.
""" """
yml_file = base_config_path + '.yml' yml_file = base_config_path + '.yml'
if exists_accessible(yml_file): if exists_accessible(yml_file):
logging.debug('Using config file %s', yml_file)
with open(yml_file) as f: with open(yml_file) as f:
return yaml.safe_load(f) return yaml.safe_load(f)
ini_file = base_config_path + '.ini' ini_file = base_config_path + '.ini'
if exists_accessible(ini_file): if exists_accessible(ini_file):
config = configparser.ConfigParser() config = configparser.ConfigParser()
config.read(ini_file) config.read(ini_file)
if 'main' in config._sections: if 'main' in config._sections:
logging.debug('Using config file %s', ini_file)
return config._sections['main'] return config._sections['main']
else:
logging.debug('Ignoring config file %s (no [main])', ini_file)
olasdUnsubmitted
Not Done
Inline Actions

That's an info at best, as it'd log an error message if a config file voluntarily doesn't exist (e.g. during tests).

exists_accessible already raises relevant errors when a file is present but inaccessible.

olasd: That's an info at best, as it'd log an error message if a config file voluntarily doesn't exist…
return {} return {}
def config_exists(config_path): def config_exists(config_path):
"""Check whether the given config exists""" """Check whether the given config exists"""
basepath = config_basepath(config_path) basepath = config_basepath(config_path)
return any(exists_accessible(basepath + extension) return any(exists_accessible(basepath + extension)
for extension in SWH_CONFIG_EXTENSIONS) for extension in SWH_CONFIG_EXTENSIONS)
▲ Show 20 Lines • Show All 172 Lines • Show Last 20 Lines
About · Developer information · Legal