Page MenuHomeSoftware Heritage
Differential D8527

model: avoid another extra creation of Model object
ClosedPublic
Actions

Authored by marmoute on Sep 23 2022, 11:31 AM.

Details

Reviewers
olasd
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T4595: Improve common tooling for loading
Commits
rDMODad3ecac9beae: model: avoid another extra creation of Model object
Summary
Do not create model object while sorting entry before creating model
object.

This is another case of "let us create object X to prepare the creation
of object X", slowing things down.

In practice, we will likely skip this code-path after the next
changeset, however this seems useful to get this performance footgun
out the way.

We tested this change on simple information of the Mercurial loader,
with a noop-loader stockage:

    swh loader run mercurial https://foss.heptapod.net/mercurial/mercurial-devel directory=/data/repos/mercurial-devel

= Median time of 3 run =
before  12 minutes 59 seconds
after:  11 minute  56 seconds

On a profile of the same run, the `to_model` call of the from_disk's `Directory` class took the following percentage:
before: 24%
after:  17%
Test Plan

ran tox, timing and profile

Diff Detail

Repository
rDMOD Data model
Branch
D8527
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 31690
Build 49579: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 49578: arc lint + arc unit

Event Timeline

marmoute created this revision.Sep 23 2022, 11:31 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 23 2022, 11:31 AM
swh-public-ci added a comment.Sep 23 2022, 11:34 AM

Build is green

Patch application report for D8527 (id=30710)

Could not rebase; Attempt merge onto 9ce6feb9d6...

Updating 9ce6feb..41500db
Fast-forward
 CONTRIBUTORS             |  1 +
 swh/model/from_disk.py   | 10 +++++++---
 swh/model/git_objects.py | 14 +++++++++-----
 3 files changed, 17 insertions(+), 8 deletions(-)
Changes applied before test
commit 41500db7ea8856f75812f4fe8d8a66c6fdc43a82
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 18:12:43 2022 +0200

    model: avoid another extra creation of Model object
    
    Do not create model object while sorting entry before creating model
    object.

commit 36185705f1805465d565b00b0df917b0e4f1c8e4
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 14:26:17 2022 +0200

    from_disk: only build a model object once
    
    Before this change, a Directory object was built to compute the `id` of
    we fed to the Directory object we built for `to_model`.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/513/ for more details.

Harbormaster completed remote builds in B31687: Diff 30710.Sep 23 2022, 11:34 AM
marmoute requested review of this revision.Sep 23 2022, 11:34 AM
marmoute updated this revision to Diff 30713.Sep 23 2022, 11:39 AM

testing update of diff

marmoute updated this revision to Diff 30721.Sep 23 2022, 11:42 AM

batch-update

swh-public-ci added a comment.Sep 23 2022, 11:43 AM

Build is green

Patch application report for D8527 (id=30713)

Could not rebase; Attempt merge onto 9ce6feb9d6...

Updating 9ce6feb..41500db
Fast-forward
 CONTRIBUTORS             |  1 +
 swh/model/from_disk.py   | 10 +++++++---
 swh/model/git_objects.py | 14 +++++++++-----
 3 files changed, 17 insertions(+), 8 deletions(-)
Changes applied before test
commit 41500db7ea8856f75812f4fe8d8a66c6fdc43a82
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 18:12:43 2022 +0200

    model: avoid another extra creation of Model object
    
    Do not create model object while sorting entry before creating model
    object.

commit 36185705f1805465d565b00b0df917b0e4f1c8e4
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 14:26:17 2022 +0200

    from_disk: only build a model object once
    
    Before this change, a Directory object was built to compute the `id` of
    we fed to the Directory object we built for `to_model`.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/515/ for more details.

Harbormaster completed remote builds in B31690: Diff 30713.Sep 23 2022, 11:43 AM
swh-public-ci added a comment.Sep 23 2022, 11:45 AM

Build is green

Patch application report for D8527 (id=30721)

Could not rebase; Attempt merge onto 9ce6feb9d6...

Updating 9ce6feb..41500db
Fast-forward
 CONTRIBUTORS             |  1 +
 swh/model/from_disk.py   | 10 +++++++---
 swh/model/git_objects.py | 14 +++++++++-----
 3 files changed, 17 insertions(+), 8 deletions(-)
Changes applied before test
commit 41500db7ea8856f75812f4fe8d8a66c6fdc43a82
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 18:12:43 2022 +0200

    model: avoid another extra creation of Model object
    
    Do not create model object while sorting entry before creating model
    object.

commit 36185705f1805465d565b00b0df917b0e4f1c8e4
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 14:26:17 2022 +0200

    from_disk: only build a model object once
    
    Before this change, a Directory object was built to compute the `id` of
    we fed to the Directory object we built for `to_model`.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/519/ for more details.

Harbormaster completed remote builds in B31698: Diff 30721.Sep 23 2022, 11:45 AM
marmoute planned changes to this revision.Sep 23 2022, 12:35 PM

I am in the process to adding more timing/profile data to the commit message (then to fight to get phabricator to display them)

marmoute updated this revision to Diff 30726.Sep 23 2022, 4:47 PM

batch-update

swh-public-ci added a comment.Sep 23 2022, 4:50 PM

Build is green

Patch application report for D8527 (id=30726)

Could not rebase; Attempt merge onto 9ce6feb9d6...

Updating 9ce6feb..ad3ecac
Fast-forward
 CONTRIBUTORS             |  1 +
 swh/model/from_disk.py   | 10 +++++++---
 swh/model/git_objects.py | 14 +++++++++-----
 3 files changed, 17 insertions(+), 8 deletions(-)
Changes applied before test
commit ad3ecac9beae015b9e9144c1da180097958b2827
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 18:12:43 2022 +0200

    model: avoid another extra creation of Model object
    
    Do not create model object while sorting entry before creating model
    object.
    
    This is another case of "let us create object X to prepare the creation
    of object X", slowing things down.
    
    In practice, we will likely skip this code-path after the next
    changeset, however this seems useful to get this performance footgun
    out the way.
    
    We tested this change on simple information of the Mercurial loader,
    with a noop-loader stockage:
    
        swh loader run mercurial https://foss.heptapod.net/mercurial/mercurial-devel directory=/data/repos/mercurial-devel
    
    = Median time of 3 run =
    before  12 minutes 59 seconds
    after:  11 minute  56 seconds
    
    On a profile of the same run, the `to_model` call of the from_disk's `Directory` class took the following percentage:
    before: 24%
    after:  17%

commit 814a6c8416d56f5f8b3e590d419d5aea7a888ab2
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 14:26:17 2022 +0200

    from_disk: only build a model object once
    
    Before this change, a Directory object was built to compute the `id` of
    we fed to the Directory object we built for `to_model`.
    
    We tested this change on simple information of the Mercurial loader,
    with a noop-loader stockage:
    
        swh loader run mercurial https://foss.heptapod.net/mercurial/mercurial-devel directory=/data/repos/mercurial-devel
    
    = Median time of 3 run =
    before: 17 minutes 48 seconds
    after:  12 minutes 59 seconds
    
    On a profile of the same run, the `to_model` call of the from_disk's `Directory` class took the following percentage:
    before: 43%
    after:  24%

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/523/ for more details.

Harbormaster completed remote builds in B31703: Diff 30726.Sep 23 2022, 4:50 PM
marmoute retitled this revision from avoid another extra creation of Model object to model: avoid another extra creation of Model object.Sep 23 2022, 4:51 PM
marmoute edited the summary of this revision. (Show Details)
marmoute added reviewers: olasd, vlorentz.
marmoute edited the summary of this revision. (Show Details)
marmoute edited the test plan for this revision. (Show Details)
marmoute updated this revision to Diff 30733.Sep 23 2022, 7:49 PM
marmoute retitled this revision from model: avoid another extra creation of Model object to model: avoid another extra creation of Model object.
marmoute edited the summary of this revision. (Show Details)

batch-update

swh-public-ci added a comment.Sep 23 2022, 7:53 PM

Build is green

Patch application report for D8527 (id=30733)

Could not rebase; Attempt merge onto 9ce6feb9d6...

Updating 9ce6feb..ad3ecac
Fast-forward
 CONTRIBUTORS             |  1 +
 swh/model/from_disk.py   | 10 +++++++---
 swh/model/git_objects.py | 14 +++++++++-----
 3 files changed, 17 insertions(+), 8 deletions(-)
Changes applied before test
commit ad3ecac9beae015b9e9144c1da180097958b2827
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 18:12:43 2022 +0200

    model: avoid another extra creation of Model object
    
    Do not create model object while sorting entry before creating model
    object.
    
    This is another case of "let us create object X to prepare the creation
    of object X", slowing things down.
    
    In practice, we will likely skip this code-path after the next
    changeset, however this seems useful to get this performance footgun
    out the way.
    
    We tested this change on simple information of the Mercurial loader,
    with a noop-loader stockage:
    
        swh loader run mercurial https://foss.heptapod.net/mercurial/mercurial-devel directory=/data/repos/mercurial-devel
    
    = Median time of 3 run =
    before  12 minutes 59 seconds
    after:  11 minute  56 seconds
    
    On a profile of the same run, the `to_model` call of the from_disk's `Directory` class took the following percentage:
    before: 24%
    after:  17%

commit 814a6c8416d56f5f8b3e590d419d5aea7a888ab2
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 14:26:17 2022 +0200

    from_disk: only build a model object once
    
    Before this change, a Directory object was built to compute the `id` of
    we fed to the Directory object we built for `to_model`.
    
    We tested this change on simple information of the Mercurial loader,
    with a noop-loader stockage:
    
        swh loader run mercurial https://foss.heptapod.net/mercurial/mercurial-devel directory=/data/repos/mercurial-devel
    
    = Median time of 3 run =
    before: 17 minutes 48 seconds
    after:  12 minutes 59 seconds
    
    On a profile of the same run, the `to_model` call of the from_disk's `Directory` class took the following percentage:
    before: 43%
    after:  24%

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/527/ for more details.

Harbormaster completed remote builds in B31710: Diff 30733.Sep 23 2022, 7:53 PM
olasd accepted this revision.Sep 26 2022, 1:35 PM
This revision is now accepted and ready to land.Sep 26 2022, 1:35 PM
vlorentz accepted this revision.Sep 26 2022, 1:48 PM
Closed by commit rDMODad3ecac9beae: model: avoid another extra creation of Model object (authored by marmoute). · Explain WhySep 26 2022, 2:28 PM
This revision was automatically updated to reflect the committed changes.
marmoute added a commit: rDMODad3ecac9beae: model: avoid another extra creation of Model object.
bchauvet added a task: T4595: Improve common tooling for loading.Sep 30 2022, 9:57 AM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
model/
14 lines91%
Loading...

Diff 30713

View Options

swh/model/git_objects.py

Loading...
About · Developer information · Legal