Paths

Table of Contentst

Differential D6066

exporters/edges: Make swhid() format directly instead of instantiating ExtendedSWHID
ClosedPublic
Actions

Authored by vlorentz on Aug 6 2021, 12:40 PM.

Tags

None

Subscribers

Details

Reviewers

Group Reviewers

Commits

rDDATASET671e8861a574: exporters/edges: Make swhid() format directly instead of instantiating…

Summary

Before this commit, between 30 and 40% of the run time was spent in this
function (especially ExtendedSWHID.__init__).

Now, it is under 10%.

Diff Detail

Repository

rDDATASET Datasets

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz requested review of this revision.Aug 6 2021, 12:40 PM

vlorentz created this revision.

Harbormaster completed remote builds in B22920: Diff 21957.Aug 6 2021, 12:40 PM

vlorentz edited the summary of this revision. (Show Details)Aug 6 2021, 12:40 PM

ardumont added a reviewer: Reviewers.Aug 6 2021, 2:25 PM

Great. One suggestion inline.

This diff also adds type but it's not mentioned in the description ;)

Also what tool did you use to measure?

swh/dataset/exporters/edges.py
22–25	maybe it's worth a comment explaining the tradeoff here. So someone does not refactor this into the previous implementation by mistake.

ardumont accepted this revision.Aug 6 2021, 2:29 PM

This revision is now accepted and ready to land.Aug 6 2021, 2:29 PM

Also what tool did you use to measure?

$ pip3 install pyprof2calltree
$ sudo apt install kcachegrind
$ vim swh-dataset/swh/dataset/journalprocessor.py  # to make it single-process and single-thread
$ python3 -m cProfile -o ~/dataset_export.pyprof $(which swh) dataset -C graph.yml graph export /tmp/g --processes=8 --formats=edges
[...]
^C
$ pyprof2calltree -i ~/dataset_export.pyprof -k

explain rationale in a comment

Harbormaster completed remote builds in B22927: Diff 21964.Aug 6 2021, 3:44 PM

Closed by commit rDDATASET671e8861a574: exporters/edges: Make swhid() format directly instead of instantiating… (authored by vlorentz). · Explain WhyAug 6 2021, 3:44 PM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDDATASET671e8861a574: exporters/edges: Make swhid() format directly instead of instantiating….

Revision Contents
Changeset List

Path

Size

swh/

dataset/

exporters/

72 lines

Diff 21965

swh/dataset/exporters/edges.py

Loading...