Page MenuHomeSoftware Heritage
Differential D2249

swh-graph: add random walk endpoint
ClosedPublic
Actions

Authored by zack on Nov 11 2019, 7:05 PM.

Details

Reviewers
seirl
Maniphest Tasks
T2077: add random walk endpoint with limited retries
Summary

probably still a lot to be improved in terms of code cleanness, but the main
functionality seems to work and is both pretty robust and fast

Closes T2077

Diff Detail

Repository
rDGRPH Compressed graph representation
Branch
feature/random-walk
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8950
Build 13075: tox-on-jenkinsJenkins
Build 13074: arc lint + arc unit

Event Timeline

zack created this revision.Nov 11 2019, 7:05 PM
swh-public-ci added a comment.Nov 11 2019, 7:07 PM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/82/ for more details.

Harbormaster completed remote builds in B8934: Diff 7755.Nov 11 2019, 7:07 PM
seirl requested changes to this revision.Nov 12 2019, 1:12 AM
seirl added inline comments.
docs/api.rst
157

Not the correct format but this should be fixed in all the methods, so I think it's fine to have the wrong one until the rest of the doc is fixed.

java/src/main/java/org/softwareheritage/graph/algo/Traversal.java
308

I'm not quite sure this is the semantic we want here, as we do access the edges even when we don't pick them. (Feel free to disagree with my idea of what nbEdgesAccessed should represent.)

348

Asserts are disabled in release mode, maybe we want to throw an actual error here?

swh/graph/server/app.py
141

Maybe simpler:

if random:
    it = backend.random_walk(...)
else:
    it = backend.walk(...)

for res_node in it:
    ...
This revision now requires changes to proceed.Nov 12 2019, 1:12 AM
zack marked 4 inline comments as done.Nov 12 2019, 8:47 AM
zack added inline comments.
java/src/main/java/org/softwareheritage/graph/algo/Traversal.java
308

ah, good point, I'm updating to also count the skipped edges

(not that it matters much, as we are no longer returning the stats and we should probably remove that accounting anyway; but while it's in it should indeed be correct)

348

ah, that's why I didn't hit this while debugging :-)
fixed

zack updated this revision to Diff 7764.Nov 12 2019, 8:48 AM
zack marked 2 inline comments as done.
  • skeleton for random walk endpoint
  • REST client: add binding for /randomwalk
  • test_api_client: add test case for /randomwalk
  • REST doc: update /randomwalk doc to match what is returned
  • access edge stats: also count skipped edges for /randomwalk
  • /randomwalk: use proper runtime errors instead of assertions
  • app.py: add TODO about making RANDOM_RETRIES configurable
  • app.py: refactor random v. non-random walk iteration logic
swh-public-ci added a comment.Nov 12 2019, 8:50 AM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/87/ for more details.

Harbormaster completed remote builds in B8943: Diff 7764.Nov 12 2019, 8:50 AM
zack updated this revision to Diff 7766.Nov 12 2019, 10:58 AM
  • randomwalk: do not bother randomizing singleton sets
swh-public-ci added a comment.Nov 12 2019, 11:48 AM

Build was aborted

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/89/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/89/console

Harbormaster failed remote builds in B8945: Diff 7766!Nov 12 2019, 11:48 AM
zack updated this revision to Diff 7770.Nov 12 2019, 1:31 PM

swh-graph: add random walk endpoint

swh-public-ci added a comment.Nov 12 2019, 1:33 PM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/93/ for more details.

Harbormaster completed remote builds in B8950: Diff 7770.Nov 12 2019, 1:33 PM
zack updated this revision to Diff 7786.Nov 12 2019, 5:38 PM
  • random walk: include starting node into returned path
  • random walk: use reservoir sampling to pick random successor
swh-public-ci added a comment.Nov 12 2019, 5:40 PM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/94/ for more details.

Harbormaster completed remote builds in B8963: Diff 7786.Nov 12 2019, 5:40 PM
seirl accepted this revision.Nov 13 2019, 5:44 PM
This revision is now accepted and ready to land.Nov 13 2019, 5:44 PM
zack updated this revision to Diff 7823.Nov 13 2019, 6:01 PM
  • add random walk endpoint
  • random walk: include starting node into returned path
  • random walk: use reservoir sampling to pick random successor
swh-public-ci added a comment.Nov 13 2019, 6:03 PM

Build is green
See https://jenkins.softwareheritage.org/job/DGRPH/job/tox/96/ for more details.

Harbormaster completed remote builds in B9011: Diff 7823.Nov 13 2019, 6:03 PM
zack closed this revision.Nov 13 2019, 6:04 PM

merged in 97e79c289aff6625ebdd9068a006c10aca28fc12

Revision Contents
Changeset List

PathSize
docs/
34 lines
java/
src/
main/
java/
org/
softwareheritage/
graph/
21 lines
algo/
116 lines
swh/
graph/
10 lines
8 lines
server/
49 lines
tests/
13 lines

Diff 7770

View Options

docs/api.rst

Show All 22 Linesfrom swh.model.model import (
Content, Content,
Directory, Directory,
Person, Person,
SkippedContent, SkippedContent,
TimestampWithTimezone, TimestampWithTimezone,
) )
from . import converters, replay from . import converters, replay
from .utils import parse_external_definition from .utils import is_recursive_external, parse_external_definition
# When log message contains empty data # When log message contains empty data
DEFAULT_AUTHOR_MESSAGE = "" DEFAULT_AUTHOR_MESSAGE = ""
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
Show All 33 Lines ):
self.swhreplay = replay.Replay( self.swhreplay = replay.Replay(
conn=self.conn, conn=self.conn,
rootpath=self.local_url, rootpath=self.local_url,
svnrepo=self, svnrepo=self,
temp_dir=local_dirname, temp_dir=local_dirname,
) )
self.max_content_length = max_content_length self.max_content_length = max_content_length
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
self.replay_started = False self.replay_started = False
def __str__(self): def __str__(self):
return str( return str(
{ {
"swh-origin": self.origin_url, "swh-origin": self.origin_url,
"remote_url": self.remote_url, "remote_url": self.remote_url,
"local_url": self.local_url, "local_url": self.local_url,
▲ Show 20 Lines • Show All 127 Lines • ▼ Show 20 Lines def export_temporary(self, revision: int) -> Tuple[str, bytes]:
url = self.remote_url url = self.remote_url
# if some paths have external URLs relative to the repository URL but targeting # if some paths have external URLs relative to the repository URL but targeting
# paths oustide it, we need to export from the origin URL as the remote URL can # paths oustide it, we need to export from the origin URL as the remote URL can
# target a dump mounted on the local filesystem # target a dump mounted on the local filesystem
if self.replay_started and self.has_relative_externals: if self.replay_started and self.has_relative_externals:
# externals detected while replaying revisions # externals detected while replaying revisions
url = self.origin_url url = self.origin_url
elif not self.replay_started and self.remote_url.startswith("file://"): elif not self.replay_started:
# revisions replay has not started, we need to check if svn:externals # revisions replay has not started, we need to check if svn:externals
# properties are set from a checkout of the revision and if some # properties are set from a checkout of the revision and if some
# external URLs are relative to pick the right export URL # external URLs are relative to pick the right export URL,
# recursive externals are also checked
with tempfile.TemporaryDirectory( with tempfile.TemporaryDirectory(
dir=self.local_dirname, prefix=f"checkout-revision-{revision}." dir=self.local_dirname, prefix=f"checkout-revision-{revision}."
) as co_dirname: ) as co_dirname:
self.client.checkout( self.client.checkout(
self.remote_url, co_dirname, revision, ignore_externals=True self.remote_url, co_dirname, revision, ignore_externals=True
) )
# get all svn:externals properties recursively # get all svn:externals properties recursively
externals = self.client.propget( externals = self.client.propget(
"svn:externals", co_dirname, None, revision, True "svn:externals", co_dirname, None, revision, True
) )
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
for path, external_defs in externals.items(): for path, external_defs in externals.items():
if self.has_relative_externals: if self.has_relative_externals or self.has_recursive_externals:
break break
path = path.replace(self.remote_url.rstrip("/") + "/", "")
for external_def in os.fsdecode(external_defs).split("\n"): for external_def in os.fsdecode(external_defs).split("\n"):
# skip empty line or comment # skip empty line or comment
if not external_def or external_def.startswith("#"): if not external_def or external_def.startswith("#"):
continue continue
_, _, _, relative_url = parse_external_definition( (
external_path,
external_url,
_,
relative_url,
) = parse_external_definition(
external_def.rstrip("\r"), path, self.origin_url external_def.rstrip("\r"), path, self.origin_url
) )
if is_recursive_external(
self.origin_url, path, external_path, external_url,
):
self.has_recursive_externals = True
url = self.remote_url
break
if relative_url: if relative_url:
self.has_relative_externals = True self.has_relative_externals = True
url = self.origin_url url = self.origin_url
break break
self.client.export( self.client.export(
url.rstrip("/"), to=local_url, rev=revision, ignore_keywords=True, url.rstrip("/"),
to=local_url,
rev=revision,
ignore_keywords=True,
ignore_externals=self.has_recursive_externals,
) )
return local_dirname, os.fsencode(local_url) return local_dirname, os.fsencode(local_url)
def swh_hash_data_per_revision( def swh_hash_data_per_revision(
self, start_revision: int, end_revision: int self, start_revision: int, end_revision: int
) -> Iterator[ ) -> Iterator[
Tuple[ Tuple[
int, int,
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines
View Options

java/src/main/java/org/softwareheritage/graph/Entry.java

Show All 22 Linesfrom swh.model.model import (
Content, Content,
Directory, Directory,
Person, Person,
SkippedContent, SkippedContent,
TimestampWithTimezone, TimestampWithTimezone,
) )
from . import converters, replay from . import converters, replay
from .utils import parse_external_definition from .utils import is_recursive_external, parse_external_definition
# When log message contains empty data # When log message contains empty data
DEFAULT_AUTHOR_MESSAGE = "" DEFAULT_AUTHOR_MESSAGE = ""
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
Show All 33 Lines ):
self.swhreplay = replay.Replay( self.swhreplay = replay.Replay(
conn=self.conn, conn=self.conn,
rootpath=self.local_url, rootpath=self.local_url,
svnrepo=self, svnrepo=self,
temp_dir=local_dirname, temp_dir=local_dirname,
) )
self.max_content_length = max_content_length self.max_content_length = max_content_length
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
self.replay_started = False self.replay_started = False
def __str__(self): def __str__(self):
return str( return str(
{ {
"swh-origin": self.origin_url, "swh-origin": self.origin_url,
"remote_url": self.remote_url, "remote_url": self.remote_url,
"local_url": self.local_url, "local_url": self.local_url,
▲ Show 20 Lines • Show All 127 Lines • ▼ Show 20 Lines def export_temporary(self, revision: int) -> Tuple[str, bytes]:
url = self.remote_url url = self.remote_url
# if some paths have external URLs relative to the repository URL but targeting # if some paths have external URLs relative to the repository URL but targeting
# paths oustide it, we need to export from the origin URL as the remote URL can # paths oustide it, we need to export from the origin URL as the remote URL can
# target a dump mounted on the local filesystem # target a dump mounted on the local filesystem
if self.replay_started and self.has_relative_externals: if self.replay_started and self.has_relative_externals:
# externals detected while replaying revisions # externals detected while replaying revisions
url = self.origin_url url = self.origin_url
elif not self.replay_started and self.remote_url.startswith("file://"): elif not self.replay_started:
# revisions replay has not started, we need to check if svn:externals # revisions replay has not started, we need to check if svn:externals
# properties are set from a checkout of the revision and if some # properties are set from a checkout of the revision and if some
# external URLs are relative to pick the right export URL # external URLs are relative to pick the right export URL,
# recursive externals are also checked
with tempfile.TemporaryDirectory( with tempfile.TemporaryDirectory(
dir=self.local_dirname, prefix=f"checkout-revision-{revision}." dir=self.local_dirname, prefix=f"checkout-revision-{revision}."
) as co_dirname: ) as co_dirname:
self.client.checkout( self.client.checkout(
self.remote_url, co_dirname, revision, ignore_externals=True self.remote_url, co_dirname, revision, ignore_externals=True
) )
# get all svn:externals properties recursively # get all svn:externals properties recursively
externals = self.client.propget( externals = self.client.propget(
"svn:externals", co_dirname, None, revision, True "svn:externals", co_dirname, None, revision, True
) )
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
for path, external_defs in externals.items(): for path, external_defs in externals.items():
if self.has_relative_externals: if self.has_relative_externals or self.has_recursive_externals:
break break
path = path.replace(self.remote_url.rstrip("/") + "/", "")
for external_def in os.fsdecode(external_defs).split("\n"): for external_def in os.fsdecode(external_defs).split("\n"):
# skip empty line or comment # skip empty line or comment
if not external_def or external_def.startswith("#"): if not external_def or external_def.startswith("#"):
continue continue
_, _, _, relative_url = parse_external_definition( (
external_path,
external_url,
_,
relative_url,
) = parse_external_definition(
external_def.rstrip("\r"), path, self.origin_url external_def.rstrip("\r"), path, self.origin_url
) )
if is_recursive_external(
self.origin_url, path, external_path, external_url,
):
self.has_recursive_externals = True
url = self.remote_url
break
if relative_url: if relative_url:
self.has_relative_externals = True self.has_relative_externals = True
url = self.origin_url url = self.origin_url
break break
self.client.export( self.client.export(
url.rstrip("/"), to=local_url, rev=revision, ignore_keywords=True, url.rstrip("/"),
to=local_url,
rev=revision,
ignore_keywords=True,
ignore_externals=self.has_recursive_externals,
) )
return local_dirname, os.fsencode(local_url) return local_dirname, os.fsencode(local_url)
def swh_hash_data_per_revision( def swh_hash_data_per_revision(
self, start_revision: int, end_revision: int self, start_revision: int, end_revision: int
) -> Iterator[ ) -> Iterator[
Tuple[ Tuple[
int, int,
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines
View Options

java/src/main/java/org/softwareheritage/graph/algo/Traversal.java

Show All 22 Linesfrom swh.model.model import (
Content, Content,
Directory, Directory,
Person, Person,
SkippedContent, SkippedContent,
TimestampWithTimezone, TimestampWithTimezone,
) )
from . import converters, replay from . import converters, replay
from .utils import parse_external_definition from .utils import is_recursive_external, parse_external_definition
# When log message contains empty data # When log message contains empty data
DEFAULT_AUTHOR_MESSAGE = "" DEFAULT_AUTHOR_MESSAGE = ""
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
Show All 33 Lines ):
self.swhreplay = replay.Replay( self.swhreplay = replay.Replay(
conn=self.conn, conn=self.conn,
rootpath=self.local_url, rootpath=self.local_url,
svnrepo=self, svnrepo=self,
temp_dir=local_dirname, temp_dir=local_dirname,
) )
self.max_content_length = max_content_length self.max_content_length = max_content_length
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
self.replay_started = False self.replay_started = False
def __str__(self): def __str__(self):
return str( return str(
{ {
"swh-origin": self.origin_url, "swh-origin": self.origin_url,
"remote_url": self.remote_url, "remote_url": self.remote_url,
"local_url": self.local_url, "local_url": self.local_url,
▲ Show 20 Lines • Show All 127 Lines • ▼ Show 20 Lines def export_temporary(self, revision: int) -> Tuple[str, bytes]:
url = self.remote_url url = self.remote_url
# if some paths have external URLs relative to the repository URL but targeting # if some paths have external URLs relative to the repository URL but targeting
# paths oustide it, we need to export from the origin URL as the remote URL can # paths oustide it, we need to export from the origin URL as the remote URL can
# target a dump mounted on the local filesystem # target a dump mounted on the local filesystem
if self.replay_started and self.has_relative_externals: if self.replay_started and self.has_relative_externals:
# externals detected while replaying revisions # externals detected while replaying revisions
url = self.origin_url url = self.origin_url
elif not self.replay_started and self.remote_url.startswith("file://"): elif not self.replay_started:
# revisions replay has not started, we need to check if svn:externals # revisions replay has not started, we need to check if svn:externals
# properties are set from a checkout of the revision and if some # properties are set from a checkout of the revision and if some
# external URLs are relative to pick the right export URL # external URLs are relative to pick the right export URL,
# recursive externals are also checked
with tempfile.TemporaryDirectory( with tempfile.TemporaryDirectory(
dir=self.local_dirname, prefix=f"checkout-revision-{revision}." dir=self.local_dirname, prefix=f"checkout-revision-{revision}."
) as co_dirname: ) as co_dirname:
self.client.checkout( self.client.checkout(
self.remote_url, co_dirname, revision, ignore_externals=True self.remote_url, co_dirname, revision, ignore_externals=True
) )
# get all svn:externals properties recursively # get all svn:externals properties recursively
externals = self.client.propget( externals = self.client.propget(
"svn:externals", co_dirname, None, revision, True "svn:externals", co_dirname, None, revision, True
) )
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
for path, external_defs in externals.items(): for path, external_defs in externals.items():
if self.has_relative_externals: if self.has_relative_externals or self.has_recursive_externals:
break break
path = path.replace(self.remote_url.rstrip("/") + "/", "")
for external_def in os.fsdecode(external_defs).split("\n"): for external_def in os.fsdecode(external_defs).split("\n"):
# skip empty line or comment # skip empty line or comment
if not external_def or external_def.startswith("#"): if not external_def or external_def.startswith("#"):
continue continue
_, _, _, relative_url = parse_external_definition( (
external_path,
external_url,
_,
relative_url,
) = parse_external_definition(
external_def.rstrip("\r"), path, self.origin_url external_def.rstrip("\r"), path, self.origin_url
) )
if is_recursive_external(
self.origin_url, path, external_path, external_url,
):
self.has_recursive_externals = True
url = self.remote_url
break
if relative_url: if relative_url:
self.has_relative_externals = True self.has_relative_externals = True
url = self.origin_url url = self.origin_url
break break
self.client.export( self.client.export(
url.rstrip("/"), to=local_url, rev=revision, ignore_keywords=True, url.rstrip("/"),
to=local_url,
rev=revision,
ignore_keywords=True,
ignore_externals=self.has_recursive_externals,
) )
return local_dirname, os.fsencode(local_url) return local_dirname, os.fsencode(local_url)
def swh_hash_data_per_revision( def swh_hash_data_per_revision(
self, start_revision: int, end_revision: int self, start_revision: int, end_revision: int
) -> Iterator[ ) -> Iterator[
Tuple[ Tuple[
int, int,
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines
View Options

swh/graph/backend.py

Show All 22 Linesfrom swh.model.model import (
Content, Content,
Directory, Directory,
Person, Person,
SkippedContent, SkippedContent,
TimestampWithTimezone, TimestampWithTimezone,
) )
from . import converters, replay from . import converters, replay
from .utils import parse_external_definition from .utils import is_recursive_external, parse_external_definition
# When log message contains empty data # When log message contains empty data
DEFAULT_AUTHOR_MESSAGE = "" DEFAULT_AUTHOR_MESSAGE = ""
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
Show All 33 Lines ):
self.swhreplay = replay.Replay( self.swhreplay = replay.Replay(
conn=self.conn, conn=self.conn,
rootpath=self.local_url, rootpath=self.local_url,
svnrepo=self, svnrepo=self,
temp_dir=local_dirname, temp_dir=local_dirname,
) )
self.max_content_length = max_content_length self.max_content_length = max_content_length
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
self.replay_started = False self.replay_started = False
def __str__(self): def __str__(self):
return str( return str(
{ {
"swh-origin": self.origin_url, "swh-origin": self.origin_url,
"remote_url": self.remote_url, "remote_url": self.remote_url,
"local_url": self.local_url, "local_url": self.local_url,
▲ Show 20 Lines • Show All 127 Lines • ▼ Show 20 Lines def export_temporary(self, revision: int) -> Tuple[str, bytes]:
url = self.remote_url url = self.remote_url
# if some paths have external URLs relative to the repository URL but targeting # if some paths have external URLs relative to the repository URL but targeting
# paths oustide it, we need to export from the origin URL as the remote URL can # paths oustide it, we need to export from the origin URL as the remote URL can
# target a dump mounted on the local filesystem # target a dump mounted on the local filesystem
if self.replay_started and self.has_relative_externals: if self.replay_started and self.has_relative_externals:
# externals detected while replaying revisions # externals detected while replaying revisions
url = self.origin_url url = self.origin_url
elif not self.replay_started and self.remote_url.startswith("file://"): elif not self.replay_started:
# revisions replay has not started, we need to check if svn:externals # revisions replay has not started, we need to check if svn:externals
# properties are set from a checkout of the revision and if some # properties are set from a checkout of the revision and if some
# external URLs are relative to pick the right export URL # external URLs are relative to pick the right export URL,
# recursive externals are also checked
with tempfile.TemporaryDirectory( with tempfile.TemporaryDirectory(
dir=self.local_dirname, prefix=f"checkout-revision-{revision}." dir=self.local_dirname, prefix=f"checkout-revision-{revision}."
) as co_dirname: ) as co_dirname:
self.client.checkout( self.client.checkout(
self.remote_url, co_dirname, revision, ignore_externals=True self.remote_url, co_dirname, revision, ignore_externals=True
) )
# get all svn:externals properties recursively # get all svn:externals properties recursively
externals = self.client.propget( externals = self.client.propget(
"svn:externals", co_dirname, None, revision, True "svn:externals", co_dirname, None, revision, True
) )
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
for path, external_defs in externals.items(): for path, external_defs in externals.items():
if self.has_relative_externals: if self.has_relative_externals or self.has_recursive_externals:
break break
path = path.replace(self.remote_url.rstrip("/") + "/", "")
for external_def in os.fsdecode(external_defs).split("\n"): for external_def in os.fsdecode(external_defs).split("\n"):
# skip empty line or comment # skip empty line or comment
if not external_def or external_def.startswith("#"): if not external_def or external_def.startswith("#"):
continue continue
_, _, _, relative_url = parse_external_definition( (
external_path,
external_url,
_,
relative_url,
) = parse_external_definition(
external_def.rstrip("\r"), path, self.origin_url external_def.rstrip("\r"), path, self.origin_url
) )
if is_recursive_external(
self.origin_url, path, external_path, external_url,
):
self.has_recursive_externals = True
url = self.remote_url
break
if relative_url: if relative_url:
self.has_relative_externals = True self.has_relative_externals = True
url = self.origin_url url = self.origin_url
break break
self.client.export( self.client.export(
url.rstrip("/"), to=local_url, rev=revision, ignore_keywords=True, url.rstrip("/"),
to=local_url,
rev=revision,
ignore_keywords=True,
ignore_externals=self.has_recursive_externals,
) )
return local_dirname, os.fsencode(local_url) return local_dirname, os.fsencode(local_url)
def swh_hash_data_per_revision( def swh_hash_data_per_revision(
self, start_revision: int, end_revision: int self, start_revision: int, end_revision: int
) -> Iterator[ ) -> Iterator[
Tuple[ Tuple[
int, int,
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines
View Options

swh/graph/client.py

Show All 22 Linesfrom swh.model.model import (
Content, Content,
Directory, Directory,
Person, Person,
SkippedContent, SkippedContent,
TimestampWithTimezone, TimestampWithTimezone,
) )
from . import converters, replay from . import converters, replay
from .utils import parse_external_definition from .utils import is_recursive_external, parse_external_definition
# When log message contains empty data # When log message contains empty data
DEFAULT_AUTHOR_MESSAGE = "" DEFAULT_AUTHOR_MESSAGE = ""
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
Show All 33 Lines ):
self.swhreplay = replay.Replay( self.swhreplay = replay.Replay(
conn=self.conn, conn=self.conn,
rootpath=self.local_url, rootpath=self.local_url,
svnrepo=self, svnrepo=self,
temp_dir=local_dirname, temp_dir=local_dirname,
) )
self.max_content_length = max_content_length self.max_content_length = max_content_length
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
self.replay_started = False self.replay_started = False
def __str__(self): def __str__(self):
return str( return str(
{ {
"swh-origin": self.origin_url, "swh-origin": self.origin_url,
"remote_url": self.remote_url, "remote_url": self.remote_url,
"local_url": self.local_url, "local_url": self.local_url,
▲ Show 20 Lines • Show All 127 Lines • ▼ Show 20 Lines def export_temporary(self, revision: int) -> Tuple[str, bytes]:
url = self.remote_url url = self.remote_url
# if some paths have external URLs relative to the repository URL but targeting # if some paths have external URLs relative to the repository URL but targeting
# paths oustide it, we need to export from the origin URL as the remote URL can # paths oustide it, we need to export from the origin URL as the remote URL can
# target a dump mounted on the local filesystem # target a dump mounted on the local filesystem
if self.replay_started and self.has_relative_externals: if self.replay_started and self.has_relative_externals:
# externals detected while replaying revisions # externals detected while replaying revisions
url = self.origin_url url = self.origin_url
elif not self.replay_started and self.remote_url.startswith("file://"): elif not self.replay_started:
# revisions replay has not started, we need to check if svn:externals # revisions replay has not started, we need to check if svn:externals
# properties are set from a checkout of the revision and if some # properties are set from a checkout of the revision and if some
# external URLs are relative to pick the right export URL # external URLs are relative to pick the right export URL,
# recursive externals are also checked
with tempfile.TemporaryDirectory( with tempfile.TemporaryDirectory(
dir=self.local_dirname, prefix=f"checkout-revision-{revision}." dir=self.local_dirname, prefix=f"checkout-revision-{revision}."
) as co_dirname: ) as co_dirname:
self.client.checkout( self.client.checkout(
self.remote_url, co_dirname, revision, ignore_externals=True self.remote_url, co_dirname, revision, ignore_externals=True
) )
# get all svn:externals properties recursively # get all svn:externals properties recursively
externals = self.client.propget( externals = self.client.propget(
"svn:externals", co_dirname, None, revision, True "svn:externals", co_dirname, None, revision, True
) )
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
for path, external_defs in externals.items(): for path, external_defs in externals.items():
if self.has_relative_externals: if self.has_relative_externals or self.has_recursive_externals:
break break
path = path.replace(self.remote_url.rstrip("/") + "/", "")
for external_def in os.fsdecode(external_defs).split("\n"): for external_def in os.fsdecode(external_defs).split("\n"):
# skip empty line or comment # skip empty line or comment
if not external_def or external_def.startswith("#"): if not external_def or external_def.startswith("#"):
continue continue
_, _, _, relative_url = parse_external_definition( (
external_path,
external_url,
_,
relative_url,
) = parse_external_definition(
external_def.rstrip("\r"), path, self.origin_url external_def.rstrip("\r"), path, self.origin_url
) )
if is_recursive_external(
self.origin_url, path, external_path, external_url,
):
self.has_recursive_externals = True
url = self.remote_url
break
if relative_url: if relative_url:
self.has_relative_externals = True self.has_relative_externals = True
url = self.origin_url url = self.origin_url
break break
self.client.export( self.client.export(
url.rstrip("/"), to=local_url, rev=revision, ignore_keywords=True, url.rstrip("/"),
to=local_url,
rev=revision,
ignore_keywords=True,
ignore_externals=self.has_recursive_externals,
) )
return local_dirname, os.fsencode(local_url) return local_dirname, os.fsencode(local_url)
def swh_hash_data_per_revision( def swh_hash_data_per_revision(
self, start_revision: int, end_revision: int self, start_revision: int, end_revision: int
) -> Iterator[ ) -> Iterator[
Tuple[ Tuple[
int, int,
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines
View Options

swh/graph/server/app.py

Show All 22 Linesfrom swh.model.model import (
Content, Content,
Directory, Directory,
Person, Person,
SkippedContent, SkippedContent,
TimestampWithTimezone, TimestampWithTimezone,
) )
from . import converters, replay from . import converters, replay
from .utils import parse_external_definition from .utils import is_recursive_external, parse_external_definition
# When log message contains empty data # When log message contains empty data
DEFAULT_AUTHOR_MESSAGE = "" DEFAULT_AUTHOR_MESSAGE = ""
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
Show All 33 Lines ):
self.swhreplay = replay.Replay( self.swhreplay = replay.Replay(
conn=self.conn, conn=self.conn,
rootpath=self.local_url, rootpath=self.local_url,
svnrepo=self, svnrepo=self,
temp_dir=local_dirname, temp_dir=local_dirname,
) )
self.max_content_length = max_content_length self.max_content_length = max_content_length
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
self.replay_started = False self.replay_started = False
def __str__(self): def __str__(self):
return str( return str(
{ {
"swh-origin": self.origin_url, "swh-origin": self.origin_url,
"remote_url": self.remote_url, "remote_url": self.remote_url,
"local_url": self.local_url, "local_url": self.local_url,
▲ Show 20 Lines • Show All 127 Lines • ▼ Show 20 Lines def export_temporary(self, revision: int) -> Tuple[str, bytes]:
url = self.remote_url url = self.remote_url
# if some paths have external URLs relative to the repository URL but targeting # if some paths have external URLs relative to the repository URL but targeting
# paths oustide it, we need to export from the origin URL as the remote URL can # paths oustide it, we need to export from the origin URL as the remote URL can
# target a dump mounted on the local filesystem # target a dump mounted on the local filesystem
if self.replay_started and self.has_relative_externals: if self.replay_started and self.has_relative_externals:
# externals detected while replaying revisions # externals detected while replaying revisions
url = self.origin_url url = self.origin_url
elif not self.replay_started and self.remote_url.startswith("file://"): elif not self.replay_started:
# revisions replay has not started, we need to check if svn:externals # revisions replay has not started, we need to check if svn:externals
# properties are set from a checkout of the revision and if some # properties are set from a checkout of the revision and if some
# external URLs are relative to pick the right export URL # external URLs are relative to pick the right export URL,
# recursive externals are also checked
with tempfile.TemporaryDirectory( with tempfile.TemporaryDirectory(
dir=self.local_dirname, prefix=f"checkout-revision-{revision}." dir=self.local_dirname, prefix=f"checkout-revision-{revision}."
) as co_dirname: ) as co_dirname:
self.client.checkout( self.client.checkout(
self.remote_url, co_dirname, revision, ignore_externals=True self.remote_url, co_dirname, revision, ignore_externals=True
) )
# get all svn:externals properties recursively # get all svn:externals properties recursively
externals = self.client.propget( externals = self.client.propget(
"svn:externals", co_dirname, None, revision, True "svn:externals", co_dirname, None, revision, True
) )
self.has_relative_externals = False self.has_relative_externals = False
self.has_recursive_externals = False
for path, external_defs in externals.items(): for path, external_defs in externals.items():
if self.has_relative_externals: if self.has_relative_externals or self.has_recursive_externals:
break break
path = path.replace(self.remote_url.rstrip("/") + "/", "")
for external_def in os.fsdecode(external_defs).split("\n"): for external_def in os.fsdecode(external_defs).split("\n"):
# skip empty line or comment # skip empty line or comment
if not external_def or external_def.startswith("#"): if not external_def or external_def.startswith("#"):
continue continue
_, _, _, relative_url = parse_external_definition( (
external_path,
external_url,
_,
relative_url,
) = parse_external_definition(
external_def.rstrip("\r"), path, self.origin_url external_def.rstrip("\r"), path, self.origin_url
) )
if is_recursive_external(
self.origin_url, path, external_path, external_url,
):
self.has_recursive_externals = True
url = self.remote_url
break
if relative_url: if relative_url:
self.has_relative_externals = True self.has_relative_externals = True
url = self.origin_url url = self.origin_url
break break
self.client.export( self.client.export(
url.rstrip("/"), to=local_url, rev=revision, ignore_keywords=True, url.rstrip("/"),
to=local_url,
rev=revision,
ignore_keywords=True,
ignore_externals=self.has_recursive_externals,
) )
return local_dirname, os.fsencode(local_url) return local_dirname, os.fsencode(local_url)
def swh_hash_data_per_revision( def swh_hash_data_per_revision(
self, start_revision: int, end_revision: int self, start_revision: int, end_revision: int
) -> Iterator[ ) -> Iterator[
Tuple[ Tuple[
int, int,
▲ Show 20 Lines • Show All 81 Lines • Show Last 20 Lines
View Options

swh/graph/tests/test_api_client.py

Loading...
About · Developer information · Legal