Page MenuHomeSoftware Heritage
Differential D3085

scanner: ndjson output format
ClosedPublic
Actions

Authored by DanSeraf on Apr 29 2020, 1:16 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDTSCN623a9dbe6157: ndjson output format
Summary

Implementation of ndjson output

Related T2363

Diff Detail

Repository
rDTSCN Code scanner
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

DanSeraf created this revision.Apr 29 2020, 1:16 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 29 2020, 1:16 PM
DanSeraf edited the summary of this revision. (Show Details)Apr 29 2020, 1:17 PM
swh-public-ci added a comment.Apr 29 2020, 1:18 PM

Build is green

Patch application report for D3085 (id=10979)

Rebasing onto 3f00bb004b...

Current branch diff-target is up to date.
Changes applied before test
commit cd4c3d9cd06b9ddc28cb7b35d0f01bb5b79d4a5a
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Wed Apr 29 13:13:31 2020 +0200

    ndjson output format

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/21/ for more details.

Harbormaster completed remote builds in B12132: Diff 10979.Apr 29 2020, 1:18 PM
vlorentz requested changes to this revision.Apr 29 2020, 2:09 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/scanner/model.py
12

Why type: ignore? If mypy is complaining about a missing stub file, you should edit mypy.ini

This revision now requires changes to proceed.Apr 29 2020, 2:09 PM
DanSeraf updated this revision to Diff 10983.Apr 29 2020, 2:28 PM

missing import in mypy.ini

swh-public-ci added a comment.Apr 29 2020, 2:30 PM

Build is green

Patch application report for D3085 (id=10983)

Rebasing onto 3f00bb004b...

Current branch diff-target is up to date.
Changes applied before test
commit e61add138c814b2b170be5115cd6047ddf87745b
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Wed Apr 29 13:13:31 2020 +0200

    ndjson output format

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/22/ for more details.

Harbormaster completed remote builds in B12136: Diff 10983.Apr 29 2020, 2:30 PM
vlorentz accepted this revision.Apr 29 2020, 3:43 PM
This revision is now accepted and ready to land.Apr 29 2020, 3:43 PM
DanSeraf updated this revision to Diff 10997.Apr 29 2020, 4:40 PM

rebase

This revision was landed with ongoing or failed builds.Apr 29 2020, 4:40 PM
Closed by commit rDTSCN623a9dbe6157: ndjson output format (authored by DanSeraf). · Explain Why
This revision was automatically updated to reflect the committed changes.
DanSeraf added a commit: rDTSCN623a9dbe6157: ndjson output format.
swh-public-ci added a comment.Apr 29 2020, 4:42 PM

Build is green

Patch application report for D3085 (id=10997)

Rebasing onto dbf1becd26...

Current branch diff-target is up to date.
Changes applied before test
commit ab5fb88a7c8eddbea5fb9a8f775a3ae6ff7762b4
Author: Daniele Serafini <me@danieleserafini.eu>
Date:   Wed Apr 29 13:13:31 2020 +0200

    ndjson output format

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/23/ for more details.

Harbormaster completed remote builds in B12151: Diff 10997.Apr 29 2020, 4:42 PM

Revision Contents
Changeset List

PathSize
4 lines
1 line
swh/
scanner/
2 lines
6 lines

Diff 10998

View Options

mypy.ini

Show First 20 Lines • Show All 574 Lines • ▼ Show 20 Lines def close(self):
else: else:
# some external paths might have been removed in the current replayed # some external paths might have been removed in the current replayed
# revision by a delete operation on an overlapping versioned path so we # revision by a delete operation on an overlapping versioned path so we
# need to restore them # need to restore them
externals = prev_externals externals = prev_externals
# For each external, try to export it in reconstructed filesystem # For each external, try to export it in reconstructed filesystem
for path, (external_url, revision, relative_url) in externals.items(): for path, (external_url, revision, relative_url) in externals.items():
self.process_external(path, external_url, revision, relative_url)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
vlorentzUnsubmitted
Not Done
Inline Actions
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
- [relative_url for (_, relative_url) in self.editor.valid_externals.values()]
+ relative_url for (_, relative_url) in self.editor.valid_externals.values()
)
self.svnrepo.has_recursive_externals = any(

no need to materialize the list

vlorentz: no need to materialize the list
anlambertAuthorUnsubmitted
Done
Inline Actions

Ah right, will make the change in D7092 to avoid too much rebasing.

anlambert: Ah right, will make the change in D7092 to avoid too much rebasing.
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def process_external(
self, path: str, external_url: str, revision: Optional[int], relative_url: bool
) -> None:
external = (external_url, revision) external = (external_url, revision)
dest_path = os.fsencode(path) dest_path = os.fsencode(path)
dest_fullpath = os.path.join(self.path, dest_path) dest_fullpath = os.path.join(self.path, dest_path)
prev_externals = self.dir_states[self.path].externals
if ( if (
path in prev_externals path in prev_externals
and prev_externals[path] == external and prev_externals[path] == external
and dest_fullpath in self.directory and dest_fullpath in self.directory
): ):
# external already exported, nothing to do # external already exported, nothing to do
continue return
if is_recursive_external( if is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url
): ):
# recursive external, skip it # recursive external, skip it
continue return
if external not in self.editor.externals_cache: if external not in self.editor.externals_cache:
try: try:
# try to export external in a temporary path, destination path could # try to export external in a temporary path, destination path could
# be versioned and must be overridden only if the external URL is # be versioned and must be overridden only if the external URL is
# still valid # still valid
temp_dir = os.fsencode( temp_dir = os.fsencode(
tempfile.mkdtemp(dir=self.editor.externals_cache_dir) tempfile.mkdtemp(dir=self.editor.externals_cache_dir)
) )
temp_path = os.path.join(temp_dir, dest_path) temp_path = os.path.join(temp_dir, dest_path)
os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True) os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True)
if external_url not in self.editor.dead_externals: if external_url not in self.editor.dead_externals:
logger.debug( logger.debug("Exporting external %s to path %s", external_url, path)
"Exporting external %s to path %s", external_url, path
)
self.svnrepo.client.export( self.svnrepo.client.export(
external_url.rstrip("/"), external_url.rstrip("/"),
to=temp_path, to=temp_path,
peg_rev=revision, peg_rev=revision,
ignore_keywords=True, ignore_keywords=True,
) )
self.editor.externals_cache[external] = temp_path self.editor.externals_cache[external] = temp_path
self.editor.valid_externals[dest_fullpath] = ( self.editor.valid_externals[dest_fullpath] = (
external_url, external_url,
relative_url, relative_url,
) )
except SubversionException as se: except SubversionException as se:
# external no longer available (404) # external no longer available (404)
logger.debug(se) logger.debug(se)
self.editor.dead_externals.add(external_url) self.editor.dead_externals.add(external_url)
else: else:
temp_path = self.editor.externals_cache[external] temp_path = self.editor.externals_cache[external]
# subversion export will always create the subdirectories of the external # subversion export will always create the subdirectories of the external
# path regardless the validity of the remote URL # path regardless the validity of the remote URL
dest_path_split = dest_path.split(b"/") dest_path_split = dest_path.split(b"/")
current_path = self.path current_path = self.path
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
for subpath in dest_path_split[:-1]: for subpath in dest_path_split[:-1]:
current_path = os.path.join(current_path, subpath) current_path = os.path.join(current_path, subpath)
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
if os.path.exists(temp_path): if os.path.exists(temp_path):
# external successfully exported # external successfully exported
# remove previous path in from_disk model # remove previous path in from_disk model
self.remove_child(dest_fullpath) self.remove_child(dest_fullpath)
# copy exported path to reconstructed filesystem # copy exported path to reconstructed filesystem
fullpath = os.path.join(self.rootpath, dest_fullpath) fullpath = os.path.join(self.rootpath, dest_fullpath)
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
self.editor.modified_paths.add(dest_fullpath) self.editor.modified_paths.add(dest_fullpath)
# update from_disk model and store external paths # update from_disk model and store external paths
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
if os.path.isfile(temp_path): if os.path.isfile(temp_path):
shutil.copy(temp_path, fullpath) shutil.copy(os.fsdecode(temp_path), os.fsdecode(fullpath))
self.directory[dest_fullpath] = from_disk.Content.from_file( self.directory[dest_fullpath] = from_disk.Content.from_file(
path=fullpath path=fullpath
) )
else: else:
shutil.copytree(temp_path, fullpath, symlinks=True) shutil.copytree(
os.fsdecode(temp_path), os.fsdecode(fullpath), symlinks=True
)
self.directory[dest_fullpath] = from_disk.Directory.from_disk( self.directory[dest_fullpath] = from_disk.Directory.from_disk(
path=fullpath path=fullpath
) )
external_paths = set() external_paths = set()
for root, dirs, files in os.walk(fullpath): for root, dirs, files in os.walk(fullpath):
external_paths.update( external_paths.update(
[ [
os.path.join(root.replace(self.rootpath + b"/", b""), p) os.path.join(root.replace(self.rootpath + b"/", b""), p)
for p in chain(dirs, files) for p in chain(dirs, files)
] ]
) )
self.editor.external_paths.update(external_paths) self.editor.external_paths.update(external_paths)
self.editor.modified_paths.update(external_paths) self.editor.modified_paths.update(external_paths)
# ensure hash update for the directory with externals set # ensure hash update for the directory with externals set
self.directory[self.path].update_hash(force=True) self.directory[self.path].update_hash(force=True)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def remove_external_path(self, external_path: bytes) -> None: def remove_external_path(self, external_path: bytes) -> None:
"""Remove a previously exported SVN external path from """Remove a previously exported SVN external path from
the reconstruted filesystem. the reconstruted filesystem.
""" """
fullpath = os.path.join(self.path, external_path) fullpath = os.path.join(self.path, external_path)
self.remove_child(fullpath) self.remove_child(fullpath)
self.editor.external_paths.discard(fullpath) self.editor.external_paths.discard(fullpath)
self.editor.valid_externals.pop(fullpath, None) self.editor.valid_externals.pop(fullpath, None)
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines ):
self.rootpath = rootpath self.rootpath = rootpath
self.directory = directory self.directory = directory
self.file_states: Dict[bytes, FileState] = defaultdict(FileState) self.file_states: Dict[bytes, FileState] = defaultdict(FileState)
self.dir_states: Dict[bytes, DirState] = defaultdict(DirState) self.dir_states: Dict[bytes, DirState] = defaultdict(DirState)
self.external_paths: Set[bytes] = set() self.external_paths: Set[bytes] = set()
self.valid_externals: Dict[bytes, Tuple[str, bool]] = {} self.valid_externals: Dict[bytes, Tuple[str, bool]] = {}
self.dead_externals: Set[str] = set() self.dead_externals: Set[str] = set()
self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir) self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir)
self.externals_cache: Dict[Tuple[str, Optional[int]], str] = {} self.externals_cache: Dict[Tuple[str, Optional[int]], bytes] = {}
self.svnrepo = svnrepo self.svnrepo = svnrepo
self.revnum = None self.revnum = None
# to store the set of paths added or modified when replaying a revision # to store the set of paths added or modified when replaying a revision
self.modified_paths: Set[bytes] = set() self.modified_paths: Set[bytes] = set()
def set_target_revision(self, revnum) -> None: def set_target_revision(self, revnum) -> None:
self.revnum = revnum self.revnum = revnum
▲ Show 20 Lines • Show All 164 Lines • Show Last 20 Lines
View Options

requirements.txt

Show First 20 Lines • Show All 574 Lines • ▼ Show 20 Lines def close(self):
else: else:
# some external paths might have been removed in the current replayed # some external paths might have been removed in the current replayed
# revision by a delete operation on an overlapping versioned path so we # revision by a delete operation on an overlapping versioned path so we
# need to restore them # need to restore them
externals = prev_externals externals = prev_externals
# For each external, try to export it in reconstructed filesystem # For each external, try to export it in reconstructed filesystem
for path, (external_url, revision, relative_url) in externals.items(): for path, (external_url, revision, relative_url) in externals.items():
self.process_external(path, external_url, revision, relative_url)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
vlorentzUnsubmitted
Not Done
Inline Actions
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
- [relative_url for (_, relative_url) in self.editor.valid_externals.values()]
+ relative_url for (_, relative_url) in self.editor.valid_externals.values()
)
self.svnrepo.has_recursive_externals = any(

no need to materialize the list

vlorentz: no need to materialize the list
anlambertAuthorUnsubmitted
Done
Inline Actions

Ah right, will make the change in D7092 to avoid too much rebasing.

anlambert: Ah right, will make the change in D7092 to avoid too much rebasing.
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def process_external(
self, path: str, external_url: str, revision: Optional[int], relative_url: bool
) -> None:
external = (external_url, revision) external = (external_url, revision)
dest_path = os.fsencode(path) dest_path = os.fsencode(path)
dest_fullpath = os.path.join(self.path, dest_path) dest_fullpath = os.path.join(self.path, dest_path)
prev_externals = self.dir_states[self.path].externals
if ( if (
path in prev_externals path in prev_externals
and prev_externals[path] == external and prev_externals[path] == external
and dest_fullpath in self.directory and dest_fullpath in self.directory
): ):
# external already exported, nothing to do # external already exported, nothing to do
continue return
if is_recursive_external( if is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url
): ):
# recursive external, skip it # recursive external, skip it
continue return
if external not in self.editor.externals_cache: if external not in self.editor.externals_cache:
try: try:
# try to export external in a temporary path, destination path could # try to export external in a temporary path, destination path could
# be versioned and must be overridden only if the external URL is # be versioned and must be overridden only if the external URL is
# still valid # still valid
temp_dir = os.fsencode( temp_dir = os.fsencode(
tempfile.mkdtemp(dir=self.editor.externals_cache_dir) tempfile.mkdtemp(dir=self.editor.externals_cache_dir)
) )
temp_path = os.path.join(temp_dir, dest_path) temp_path = os.path.join(temp_dir, dest_path)
os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True) os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True)
if external_url not in self.editor.dead_externals: if external_url not in self.editor.dead_externals:
logger.debug( logger.debug("Exporting external %s to path %s", external_url, path)
"Exporting external %s to path %s", external_url, path
)
self.svnrepo.client.export( self.svnrepo.client.export(
external_url.rstrip("/"), external_url.rstrip("/"),
to=temp_path, to=temp_path,
peg_rev=revision, peg_rev=revision,
ignore_keywords=True, ignore_keywords=True,
) )
self.editor.externals_cache[external] = temp_path self.editor.externals_cache[external] = temp_path
self.editor.valid_externals[dest_fullpath] = ( self.editor.valid_externals[dest_fullpath] = (
external_url, external_url,
relative_url, relative_url,
) )
except SubversionException as se: except SubversionException as se:
# external no longer available (404) # external no longer available (404)
logger.debug(se) logger.debug(se)
self.editor.dead_externals.add(external_url) self.editor.dead_externals.add(external_url)
else: else:
temp_path = self.editor.externals_cache[external] temp_path = self.editor.externals_cache[external]
# subversion export will always create the subdirectories of the external # subversion export will always create the subdirectories of the external
# path regardless the validity of the remote URL # path regardless the validity of the remote URL
dest_path_split = dest_path.split(b"/") dest_path_split = dest_path.split(b"/")
current_path = self.path current_path = self.path
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
for subpath in dest_path_split[:-1]: for subpath in dest_path_split[:-1]:
current_path = os.path.join(current_path, subpath) current_path = os.path.join(current_path, subpath)
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
if os.path.exists(temp_path): if os.path.exists(temp_path):
# external successfully exported # external successfully exported
# remove previous path in from_disk model # remove previous path in from_disk model
self.remove_child(dest_fullpath) self.remove_child(dest_fullpath)
# copy exported path to reconstructed filesystem # copy exported path to reconstructed filesystem
fullpath = os.path.join(self.rootpath, dest_fullpath) fullpath = os.path.join(self.rootpath, dest_fullpath)
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
self.editor.modified_paths.add(dest_fullpath) self.editor.modified_paths.add(dest_fullpath)
# update from_disk model and store external paths # update from_disk model and store external paths
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
if os.path.isfile(temp_path): if os.path.isfile(temp_path):
shutil.copy(temp_path, fullpath) shutil.copy(os.fsdecode(temp_path), os.fsdecode(fullpath))
self.directory[dest_fullpath] = from_disk.Content.from_file( self.directory[dest_fullpath] = from_disk.Content.from_file(
path=fullpath path=fullpath
) )
else: else:
shutil.copytree(temp_path, fullpath, symlinks=True) shutil.copytree(
os.fsdecode(temp_path), os.fsdecode(fullpath), symlinks=True
)
self.directory[dest_fullpath] = from_disk.Directory.from_disk( self.directory[dest_fullpath] = from_disk.Directory.from_disk(
path=fullpath path=fullpath
) )
external_paths = set() external_paths = set()
for root, dirs, files in os.walk(fullpath): for root, dirs, files in os.walk(fullpath):
external_paths.update( external_paths.update(
[ [
os.path.join(root.replace(self.rootpath + b"/", b""), p) os.path.join(root.replace(self.rootpath + b"/", b""), p)
for p in chain(dirs, files) for p in chain(dirs, files)
] ]
) )
self.editor.external_paths.update(external_paths) self.editor.external_paths.update(external_paths)
self.editor.modified_paths.update(external_paths) self.editor.modified_paths.update(external_paths)
# ensure hash update for the directory with externals set # ensure hash update for the directory with externals set
self.directory[self.path].update_hash(force=True) self.directory[self.path].update_hash(force=True)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def remove_external_path(self, external_path: bytes) -> None: def remove_external_path(self, external_path: bytes) -> None:
"""Remove a previously exported SVN external path from """Remove a previously exported SVN external path from
the reconstruted filesystem. the reconstruted filesystem.
""" """
fullpath = os.path.join(self.path, external_path) fullpath = os.path.join(self.path, external_path)
self.remove_child(fullpath) self.remove_child(fullpath)
self.editor.external_paths.discard(fullpath) self.editor.external_paths.discard(fullpath)
self.editor.valid_externals.pop(fullpath, None) self.editor.valid_externals.pop(fullpath, None)
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines ):
self.rootpath = rootpath self.rootpath = rootpath
self.directory = directory self.directory = directory
self.file_states: Dict[bytes, FileState] = defaultdict(FileState) self.file_states: Dict[bytes, FileState] = defaultdict(FileState)
self.dir_states: Dict[bytes, DirState] = defaultdict(DirState) self.dir_states: Dict[bytes, DirState] = defaultdict(DirState)
self.external_paths: Set[bytes] = set() self.external_paths: Set[bytes] = set()
self.valid_externals: Dict[bytes, Tuple[str, bool]] = {} self.valid_externals: Dict[bytes, Tuple[str, bool]] = {}
self.dead_externals: Set[str] = set() self.dead_externals: Set[str] = set()
self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir) self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir)
self.externals_cache: Dict[Tuple[str, Optional[int]], str] = {} self.externals_cache: Dict[Tuple[str, Optional[int]], bytes] = {}
self.svnrepo = svnrepo self.svnrepo = svnrepo
self.revnum = None self.revnum = None
# to store the set of paths added or modified when replaying a revision # to store the set of paths added or modified when replaying a revision
self.modified_paths: Set[bytes] = set() self.modified_paths: Set[bytes] = set()
def set_target_revision(self, revnum) -> None: def set_target_revision(self, revnum) -> None:
self.revnum = revnum self.revnum = revnum
▲ Show 20 Lines • Show All 164 Lines • Show Last 20 Lines
View Options

swh/scanner/cli.py

Show First 20 Lines • Show All 574 Lines • ▼ Show 20 Lines def close(self):
else: else:
# some external paths might have been removed in the current replayed # some external paths might have been removed in the current replayed
# revision by a delete operation on an overlapping versioned path so we # revision by a delete operation on an overlapping versioned path so we
# need to restore them # need to restore them
externals = prev_externals externals = prev_externals
# For each external, try to export it in reconstructed filesystem # For each external, try to export it in reconstructed filesystem
for path, (external_url, revision, relative_url) in externals.items(): for path, (external_url, revision, relative_url) in externals.items():
self.process_external(path, external_url, revision, relative_url)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
vlorentzUnsubmitted
Not Done
Inline Actions
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
- [relative_url for (_, relative_url) in self.editor.valid_externals.values()]
+ relative_url for (_, relative_url) in self.editor.valid_externals.values()
)
self.svnrepo.has_recursive_externals = any(

no need to materialize the list

vlorentz: no need to materialize the list
anlambertAuthorUnsubmitted
Done
Inline Actions

Ah right, will make the change in D7092 to avoid too much rebasing.

anlambert: Ah right, will make the change in D7092 to avoid too much rebasing.
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def process_external(
self, path: str, external_url: str, revision: Optional[int], relative_url: bool
) -> None:
external = (external_url, revision) external = (external_url, revision)
dest_path = os.fsencode(path) dest_path = os.fsencode(path)
dest_fullpath = os.path.join(self.path, dest_path) dest_fullpath = os.path.join(self.path, dest_path)
prev_externals = self.dir_states[self.path].externals
if ( if (
path in prev_externals path in prev_externals
and prev_externals[path] == external and prev_externals[path] == external
and dest_fullpath in self.directory and dest_fullpath in self.directory
): ):
# external already exported, nothing to do # external already exported, nothing to do
continue return
if is_recursive_external( if is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url
): ):
# recursive external, skip it # recursive external, skip it
continue return
if external not in self.editor.externals_cache: if external not in self.editor.externals_cache:
try: try:
# try to export external in a temporary path, destination path could # try to export external in a temporary path, destination path could
# be versioned and must be overridden only if the external URL is # be versioned and must be overridden only if the external URL is
# still valid # still valid
temp_dir = os.fsencode( temp_dir = os.fsencode(
tempfile.mkdtemp(dir=self.editor.externals_cache_dir) tempfile.mkdtemp(dir=self.editor.externals_cache_dir)
) )
temp_path = os.path.join(temp_dir, dest_path) temp_path = os.path.join(temp_dir, dest_path)
os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True) os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True)
if external_url not in self.editor.dead_externals: if external_url not in self.editor.dead_externals:
logger.debug( logger.debug("Exporting external %s to path %s", external_url, path)
"Exporting external %s to path %s", external_url, path
)
self.svnrepo.client.export( self.svnrepo.client.export(
external_url.rstrip("/"), external_url.rstrip("/"),
to=temp_path, to=temp_path,
peg_rev=revision, peg_rev=revision,
ignore_keywords=True, ignore_keywords=True,
) )
self.editor.externals_cache[external] = temp_path self.editor.externals_cache[external] = temp_path
self.editor.valid_externals[dest_fullpath] = ( self.editor.valid_externals[dest_fullpath] = (
external_url, external_url,
relative_url, relative_url,
) )
except SubversionException as se: except SubversionException as se:
# external no longer available (404) # external no longer available (404)
logger.debug(se) logger.debug(se)
self.editor.dead_externals.add(external_url) self.editor.dead_externals.add(external_url)
else: else:
temp_path = self.editor.externals_cache[external] temp_path = self.editor.externals_cache[external]
# subversion export will always create the subdirectories of the external # subversion export will always create the subdirectories of the external
# path regardless the validity of the remote URL # path regardless the validity of the remote URL
dest_path_split = dest_path.split(b"/") dest_path_split = dest_path.split(b"/")
current_path = self.path current_path = self.path
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
for subpath in dest_path_split[:-1]: for subpath in dest_path_split[:-1]:
current_path = os.path.join(current_path, subpath) current_path = os.path.join(current_path, subpath)
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
if os.path.exists(temp_path): if os.path.exists(temp_path):
# external successfully exported # external successfully exported
# remove previous path in from_disk model # remove previous path in from_disk model
self.remove_child(dest_fullpath) self.remove_child(dest_fullpath)
# copy exported path to reconstructed filesystem # copy exported path to reconstructed filesystem
fullpath = os.path.join(self.rootpath, dest_fullpath) fullpath = os.path.join(self.rootpath, dest_fullpath)
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
self.editor.modified_paths.add(dest_fullpath) self.editor.modified_paths.add(dest_fullpath)
# update from_disk model and store external paths # update from_disk model and store external paths
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
if os.path.isfile(temp_path): if os.path.isfile(temp_path):
shutil.copy(temp_path, fullpath) shutil.copy(os.fsdecode(temp_path), os.fsdecode(fullpath))
self.directory[dest_fullpath] = from_disk.Content.from_file( self.directory[dest_fullpath] = from_disk.Content.from_file(
path=fullpath path=fullpath
) )
else: else:
shutil.copytree(temp_path, fullpath, symlinks=True) shutil.copytree(
os.fsdecode(temp_path), os.fsdecode(fullpath), symlinks=True
)
self.directory[dest_fullpath] = from_disk.Directory.from_disk( self.directory[dest_fullpath] = from_disk.Directory.from_disk(
path=fullpath path=fullpath
) )
external_paths = set() external_paths = set()
for root, dirs, files in os.walk(fullpath): for root, dirs, files in os.walk(fullpath):
external_paths.update( external_paths.update(
[ [
os.path.join(root.replace(self.rootpath + b"/", b""), p) os.path.join(root.replace(self.rootpath + b"/", b""), p)
for p in chain(dirs, files) for p in chain(dirs, files)
] ]
) )
self.editor.external_paths.update(external_paths) self.editor.external_paths.update(external_paths)
self.editor.modified_paths.update(external_paths) self.editor.modified_paths.update(external_paths)
# ensure hash update for the directory with externals set # ensure hash update for the directory with externals set
self.directory[self.path].update_hash(force=True) self.directory[self.path].update_hash(force=True)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def remove_external_path(self, external_path: bytes) -> None: def remove_external_path(self, external_path: bytes) -> None:
"""Remove a previously exported SVN external path from """Remove a previously exported SVN external path from
the reconstruted filesystem. the reconstruted filesystem.
""" """
fullpath = os.path.join(self.path, external_path) fullpath = os.path.join(self.path, external_path)
self.remove_child(fullpath) self.remove_child(fullpath)
self.editor.external_paths.discard(fullpath) self.editor.external_paths.discard(fullpath)
self.editor.valid_externals.pop(fullpath, None) self.editor.valid_externals.pop(fullpath, None)
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines ):
self.rootpath = rootpath self.rootpath = rootpath
self.directory = directory self.directory = directory
self.file_states: Dict[bytes, FileState] = defaultdict(FileState) self.file_states: Dict[bytes, FileState] = defaultdict(FileState)
self.dir_states: Dict[bytes, DirState] = defaultdict(DirState) self.dir_states: Dict[bytes, DirState] = defaultdict(DirState)
self.external_paths: Set[bytes] = set() self.external_paths: Set[bytes] = set()
self.valid_externals: Dict[bytes, Tuple[str, bool]] = {} self.valid_externals: Dict[bytes, Tuple[str, bool]] = {}
self.dead_externals: Set[str] = set() self.dead_externals: Set[str] = set()
self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir) self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir)
self.externals_cache: Dict[Tuple[str, Optional[int]], str] = {} self.externals_cache: Dict[Tuple[str, Optional[int]], bytes] = {}
self.svnrepo = svnrepo self.svnrepo = svnrepo
self.revnum = None self.revnum = None
# to store the set of paths added or modified when replaying a revision # to store the set of paths added or modified when replaying a revision
self.modified_paths: Set[bytes] = set() self.modified_paths: Set[bytes] = set()
def set_target_revision(self, revnum) -> None: def set_target_revision(self, revnum) -> None:
self.revnum = revnum self.revnum = revnum
▲ Show 20 Lines • Show All 164 Lines • Show Last 20 Lines
View Options

swh/scanner/model.py

Show First 20 Lines • Show All 574 Lines • ▼ Show 20 Lines def close(self):
else: else:
# some external paths might have been removed in the current replayed # some external paths might have been removed in the current replayed
# revision by a delete operation on an overlapping versioned path so we # revision by a delete operation on an overlapping versioned path so we
# need to restore them # need to restore them
externals = prev_externals externals = prev_externals
# For each external, try to export it in reconstructed filesystem # For each external, try to export it in reconstructed filesystem
for path, (external_url, revision, relative_url) in externals.items(): for path, (external_url, revision, relative_url) in externals.items():
self.process_external(path, external_url, revision, relative_url)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
vlorentzUnsubmitted
Not Done
Inline Actions
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
- [relative_url for (_, relative_url) in self.editor.valid_externals.values()]
+ relative_url for (_, relative_url) in self.editor.valid_externals.values()
)
self.svnrepo.has_recursive_externals = any(

no need to materialize the list

vlorentz: no need to materialize the list
anlambertAuthorUnsubmitted
Done
Inline Actions

Ah right, will make the change in D7092 to avoid too much rebasing.

anlambert: Ah right, will make the change in D7092 to avoid too much rebasing.
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def process_external(
self, path: str, external_url: str, revision: Optional[int], relative_url: bool
) -> None:
external = (external_url, revision) external = (external_url, revision)
dest_path = os.fsencode(path) dest_path = os.fsencode(path)
dest_fullpath = os.path.join(self.path, dest_path) dest_fullpath = os.path.join(self.path, dest_path)
prev_externals = self.dir_states[self.path].externals
if ( if (
path in prev_externals path in prev_externals
and prev_externals[path] == external and prev_externals[path] == external
and dest_fullpath in self.directory and dest_fullpath in self.directory
): ):
# external already exported, nothing to do # external already exported, nothing to do
continue return
if is_recursive_external( if is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url self.svnrepo.origin_url, os.fsdecode(self.path), path, external_url
): ):
# recursive external, skip it # recursive external, skip it
continue return
if external not in self.editor.externals_cache: if external not in self.editor.externals_cache:
try: try:
# try to export external in a temporary path, destination path could # try to export external in a temporary path, destination path could
# be versioned and must be overridden only if the external URL is # be versioned and must be overridden only if the external URL is
# still valid # still valid
temp_dir = os.fsencode( temp_dir = os.fsencode(
tempfile.mkdtemp(dir=self.editor.externals_cache_dir) tempfile.mkdtemp(dir=self.editor.externals_cache_dir)
) )
temp_path = os.path.join(temp_dir, dest_path) temp_path = os.path.join(temp_dir, dest_path)
os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True) os.makedirs(b"/".join(temp_path.split(b"/")[:-1]), exist_ok=True)
if external_url not in self.editor.dead_externals: if external_url not in self.editor.dead_externals:
logger.debug( logger.debug("Exporting external %s to path %s", external_url, path)
"Exporting external %s to path %s", external_url, path
)
self.svnrepo.client.export( self.svnrepo.client.export(
external_url.rstrip("/"), external_url.rstrip("/"),
to=temp_path, to=temp_path,
peg_rev=revision, peg_rev=revision,
ignore_keywords=True, ignore_keywords=True,
) )
self.editor.externals_cache[external] = temp_path self.editor.externals_cache[external] = temp_path
self.editor.valid_externals[dest_fullpath] = ( self.editor.valid_externals[dest_fullpath] = (
external_url, external_url,
relative_url, relative_url,
) )
except SubversionException as se: except SubversionException as se:
# external no longer available (404) # external no longer available (404)
logger.debug(se) logger.debug(se)
self.editor.dead_externals.add(external_url) self.editor.dead_externals.add(external_url)
else: else:
temp_path = self.editor.externals_cache[external] temp_path = self.editor.externals_cache[external]
# subversion export will always create the subdirectories of the external # subversion export will always create the subdirectories of the external
# path regardless the validity of the remote URL # path regardless the validity of the remote URL
dest_path_split = dest_path.split(b"/") dest_path_split = dest_path.split(b"/")
current_path = self.path current_path = self.path
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
for subpath in dest_path_split[:-1]: for subpath in dest_path_split[:-1]:
current_path = os.path.join(current_path, subpath) current_path = os.path.join(current_path, subpath)
self.add_directory(current_path) self.add_directory(os.fsdecode(current_path))
if os.path.exists(temp_path): if os.path.exists(temp_path):
# external successfully exported # external successfully exported
# remove previous path in from_disk model # remove previous path in from_disk model
self.remove_child(dest_fullpath) self.remove_child(dest_fullpath)
# copy exported path to reconstructed filesystem # copy exported path to reconstructed filesystem
fullpath = os.path.join(self.rootpath, dest_fullpath) fullpath = os.path.join(self.rootpath, dest_fullpath)
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
self.editor.modified_paths.add(dest_fullpath) self.editor.modified_paths.add(dest_fullpath)
# update from_disk model and store external paths # update from_disk model and store external paths
self.editor.external_paths.add(dest_fullpath) self.editor.external_paths.add(dest_fullpath)
if os.path.isfile(temp_path): if os.path.isfile(temp_path):
shutil.copy(temp_path, fullpath) shutil.copy(os.fsdecode(temp_path), os.fsdecode(fullpath))
self.directory[dest_fullpath] = from_disk.Content.from_file( self.directory[dest_fullpath] = from_disk.Content.from_file(
path=fullpath path=fullpath
) )
else: else:
shutil.copytree(temp_path, fullpath, symlinks=True) shutil.copytree(
os.fsdecode(temp_path), os.fsdecode(fullpath), symlinks=True
)
self.directory[dest_fullpath] = from_disk.Directory.from_disk( self.directory[dest_fullpath] = from_disk.Directory.from_disk(
path=fullpath path=fullpath
) )
external_paths = set() external_paths = set()
for root, dirs, files in os.walk(fullpath): for root, dirs, files in os.walk(fullpath):
external_paths.update( external_paths.update(
[ [
os.path.join(root.replace(self.rootpath + b"/", b""), p) os.path.join(root.replace(self.rootpath + b"/", b""), p)
for p in chain(dirs, files) for p in chain(dirs, files)
] ]
) )
self.editor.external_paths.update(external_paths) self.editor.external_paths.update(external_paths)
self.editor.modified_paths.update(external_paths) self.editor.modified_paths.update(external_paths)
# ensure hash update for the directory with externals set # ensure hash update for the directory with externals set
self.directory[self.path].update_hash(force=True) self.directory[self.path].update_hash(force=True)
# backup externals in directory state
if self.externals:
self.dir_states[self.path].externals = self.externals
self.svnrepo.has_relative_externals = any(
[relative_url for (_, relative_url) in self.editor.valid_externals.values()]
)
self.svnrepo.has_recursive_externals = any(
is_recursive_external(
self.svnrepo.origin_url, os.fsdecode(path), external_path, external_url
)
for path, dir_state in self.dir_states.items()
for external_path, (external_url, _, _) in dir_state.externals.items()
)
if self.svnrepo.has_recursive_externals:
# If the repository has recursive externals, we stop processing externals
# and remove those already exported,
# We will then ignore externals when exporting the revision to check for
# divergence with the reconstructed filesystem
for external_path in list(self.editor.external_paths):
self.remove_external_path(external_path)
def remove_external_path(self, external_path: bytes) -> None: def remove_external_path(self, external_path: bytes) -> None:
"""Remove a previously exported SVN external path from """Remove a previously exported SVN external path from
the reconstruted filesystem. the reconstruted filesystem.
""" """
fullpath = os.path.join(self.path, external_path) fullpath = os.path.join(self.path, external_path)
self.remove_child(fullpath) self.remove_child(fullpath)
self.editor.external_paths.discard(fullpath) self.editor.external_paths.discard(fullpath)
self.editor.valid_externals.pop(fullpath, None) self.editor.valid_externals.pop(fullpath, None)
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines ):
self.rootpath = rootpath self.rootpath = rootpath
self.directory = directory self.directory = directory
self.file_states: Dict[bytes, FileState] = defaultdict(FileState) self.file_states: Dict[bytes, FileState] = defaultdict(FileState)
self.dir_states: Dict[bytes, DirState] = defaultdict(DirState) self.dir_states: Dict[bytes, DirState] = defaultdict(DirState)
self.external_paths: Set[bytes] = set() self.external_paths: Set[bytes] = set()
self.valid_externals: Dict[bytes, Tuple[str, bool]] = {} self.valid_externals: Dict[bytes, Tuple[str, bool]] = {}
self.dead_externals: Set[str] = set() self.dead_externals: Set[str] = set()
self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir) self.externals_cache_dir = tempfile.mkdtemp(dir=temp_dir)
self.externals_cache: Dict[Tuple[str, Optional[int]], str] = {} self.externals_cache: Dict[Tuple[str, Optional[int]], bytes] = {}
self.svnrepo = svnrepo self.svnrepo = svnrepo
self.revnum = None self.revnum = None
# to store the set of paths added or modified when replaying a revision # to store the set of paths added or modified when replaying a revision
self.modified_paths: Set[bytes] = set() self.modified_paths: Set[bytes] = set()
def set_target_revision(self, revnum) -> None: def set_target_revision(self, revnum) -> None:
self.revnum = revnum self.revnum = revnum
▲ Show 20 Lines • Show All 164 Lines • Show Last 20 Lines
About · Developer information · Legal