Page MenuHomeSoftware Heritage
Differential D8541

Make HTTP requests more resilient
ClosedPublic
Actions

Authored by marmoute on Sep 27 2022, 12:25 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T4592: swh-scanner - generic bug fixing, robustness and performance improvements
Commits
rDTSCN11b13ce99119: client: add some basic error detection and rate limiting
rDTSCN5f3140ab75da: client: add an option to display http debug information
rDTSCN1b05e7f3ee5d: client: prepare for retry
rDTSCNa17846cda5b5: client: transforme the swhid to string earlier
rDTSCN038e7b9454e7: client: compute the client endpoin once and ealier
rDTSCNc44df1c8ecfb: client: move `make_request` to be a method
rDTSCN549faee3fcbf: client: reorder some conditionnal
rDTSCN5293bbe477f7: client: extract the `get_chunk` closure
Summary

Series on change to clean the http client code a bit and add a first layer of retry and rate limiting capabilities

This have no reason to be a closure, so we extract it as a proper
function.

client: reorder some conditionnal

Let us put the simple case first and the longer one after it.

client: move make_request to be a method

This has not reason to be a closure, so let it not be a closure.

client: transforme the swhid to string earlier

This simplify the function we call in the loop, cannot hurt.

client: compute the client endpoin once and ealier

This make the user function simpler.

fixup! output: pick "summary" as the default

client: prepare for retry

client: add some basic error detection and rate limiting

We retry on error and we adjust to the X-RateLimit header the server
send us. This is quite basic, and has to struggle against the async
implementation of the request.

So further works should probably refactor the concurrency of the request
implementation before going further.

See inline documentation for details.

Better handling of ClientConnectionError are also needed.

Test Plan

ran tox and successully did not got banned from sending http request to SWH

Diff Detail

Repository
rDTSCN Code scanner
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Sep 27 2022, 12:25 AM
swh-public-ci added a comment.Sep 27 2022, 12:27 AM

Build has FAILED

Patch application report for D8541 (id=30798)

Could not rebase; Attempt merge onto 90534ecae4...

Updating 90534ec..9f262b3
Fast-forward
 swh/scanner/cli.py    |  21 +++++++--
 swh/scanner/client.py | 124 +++++++++++++++++++++++++++++++++++++++++++-------
 swh/scanner/output.py |  53 ++++++++++++++++++++-
 3 files changed, 177 insertions(+), 21 deletions(-)
Changes applied before test
commit 9f262b332db30fa90933ee9135560f16b2ff1d3e
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 23:07:59 2022 +0200

    client: add some basic error detection and rate limiting
    
    We retry on error and we adjust to the `X-RateLimit` header the server
    send us. This is quite basic, and has to struggle against the async
    implementation of the request.
    
    So further works should probably refactor the concurrency of the request
    implementation before going further.
    
    See inline documentation for details.
    
    Better handling of ClientConnectionError are also needed.

commit a64576a5bf9c2530699f67deb3963b3f45226f42
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:58:21 2022 +0200

    client: prepare for retry

commit 69762dac8f7995b9514c2b026b3215f6a6dffbd9
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:51:01 2022 +0200

    fixup! output: pick "summary" as the default

commit 32baf4dd99bd27b8c6cc50433fe59e186086757c
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:44:54 2022 +0200

    client: compute the client endpoin once and ealier
    
    This make the user function simpler.

commit 1a0d29d3c0ebe55ea7360c71cb40f2bd28575326
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:43:49 2022 +0200

    client: transforme the swhid to string earlier
    
    This simplify the function we call in the loop, cannot hurt.

commit 765da31c4292b2613a440823c9ea6a06239b1faf
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:31:01 2022 +0200

    client: move `make_request` to be a method
    
    This has not reason to be a closure, so let it not be a closure.

commit 6610f1aeacbb263f6c897d36ee4955a43526497a
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:27:13 2022 +0200

    client: reorder some conditionnal
    
    Let us put the simple case first and the longer one after it.

commit d0ff46eed15aed17fdeadd5bda04a9574f7201c4
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:25:38 2022 +0200

    client: extract the `get_chunk` closure
    
    This have no reason to be a closure, so we extract it as a proper
    function.

commit 0fcecd6c4db648ec9e8a7ae876bdf3ca6518ec28
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:23:40 2022 +0200

    output: pick "summary" as the default
    
    This should be less scary than "text" for a first contact.

commit eb62246d6ca347aeb9239ab6c0ef978284c4f593
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:07:18 2022 +0200

    output: add a "summary" output
    
    This is a small and minimal output to display some information without
    overwhelming the user with a wall of text.
    
    Example output:
Files:                     89
            known:         40   ( 44%)
directories:               11
      fully-known:          3   ( 27%)
  partially-known:          0   (  0%)
```

It can most probably be improved, but this can come later.

commit 821873a035417ceb73a7973687d4745c774a3aee
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:10:38 2022 +0200

scan: start documenting the --output-format option better

This is still quite short, but already better. This leave room for
expanding the format list and the associated description.

commit b3b8ca5f0f5715f9ab470a0993d61d2cebdfb074
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:09:30 2022 +0200

scan: use the current directory as the default the target directory

This seems like a good default and smooth the first contact with the
tool.
Link to build: https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/166/
See console output for more information: https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/166/console
Harbormaster returned this revision to the author for changes because remote builds failed.Sep 27 2022, 12:27 AM
Harbormaster failed remote builds in B31775: Diff 30798!
marmoute edited the summary of this revision. (Show Details)Sep 27 2022, 12:27 AM
marmoute updated this revision to Diff 30799.Sep 27 2022, 12:28 AM
marmoute edited the summary of this revision. (Show Details)

folded the fixup commit

swh-public-ci added a comment.Sep 27 2022, 12:29 AM

Build has FAILED

Patch application report for D8541 (id=30799)

Could not rebase; Attempt merge onto 90534ecae4...

Updating 90534ec..9f262b3
Fast-forward
 swh/scanner/cli.py    |  21 +++++++--
 swh/scanner/client.py | 124 +++++++++++++++++++++++++++++++++++++++++++-------
 swh/scanner/output.py |  53 ++++++++++++++++++++-
 3 files changed, 177 insertions(+), 21 deletions(-)
Changes applied before test
commit 9f262b332db30fa90933ee9135560f16b2ff1d3e
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 23:07:59 2022 +0200

    client: add some basic error detection and rate limiting
    
    We retry on error and we adjust to the `X-RateLimit` header the server
    send us. This is quite basic, and has to struggle against the async
    implementation of the request.
    
    So further works should probably refactor the concurrency of the request
    implementation before going further.
    
    See inline documentation for details.
    
    Better handling of ClientConnectionError are also needed.

commit a64576a5bf9c2530699f67deb3963b3f45226f42
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:58:21 2022 +0200

    client: prepare for retry

commit 69762dac8f7995b9514c2b026b3215f6a6dffbd9
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:51:01 2022 +0200

    fixup! output: pick "summary" as the default

commit 32baf4dd99bd27b8c6cc50433fe59e186086757c
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:44:54 2022 +0200

    client: compute the client endpoin once and ealier
    
    This make the user function simpler.

commit 1a0d29d3c0ebe55ea7360c71cb40f2bd28575326
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:43:49 2022 +0200

    client: transforme the swhid to string earlier
    
    This simplify the function we call in the loop, cannot hurt.

commit 765da31c4292b2613a440823c9ea6a06239b1faf
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:31:01 2022 +0200

    client: move `make_request` to be a method
    
    This has not reason to be a closure, so let it not be a closure.

commit 6610f1aeacbb263f6c897d36ee4955a43526497a
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:27:13 2022 +0200

    client: reorder some conditionnal
    
    Let us put the simple case first and the longer one after it.

commit d0ff46eed15aed17fdeadd5bda04a9574f7201c4
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:25:38 2022 +0200

    client: extract the `get_chunk` closure
    
    This have no reason to be a closure, so we extract it as a proper
    function.

commit 0fcecd6c4db648ec9e8a7ae876bdf3ca6518ec28
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:23:40 2022 +0200

    output: pick "summary" as the default
    
    This should be less scary than "text" for a first contact.

commit eb62246d6ca347aeb9239ab6c0ef978284c4f593
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:07:18 2022 +0200

    output: add a "summary" output
    
    This is a small and minimal output to display some information without
    overwhelming the user with a wall of text.
    
    Example output:
Files:                     89
            known:         40   ( 44%)
directories:               11
      fully-known:          3   ( 27%)
  partially-known:          0   (  0%)
```

It can most probably be improved, but this can come later.

commit 821873a035417ceb73a7973687d4745c774a3aee
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:10:38 2022 +0200

scan: start documenting the --output-format option better

This is still quite short, but already better. This leave room for
expanding the format list and the associated description.

commit b3b8ca5f0f5715f9ab470a0993d61d2cebdfb074
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:09:30 2022 +0200

scan: use the current directory as the default the target directory

This seems like a good default and smooth the first contact with the
tool.
Link to build: https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/167/
See console output for more information: https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/167/console
Harbormaster returned this revision to the author for changes because remote builds failed.Sep 27 2022, 12:29 AM
Harbormaster failed remote builds in B31776: Diff 30799!
marmoute updated this revision to Diff 30801.Sep 27 2022, 12:42 AM

Fix mypy complains

swh-public-ci added a comment.Sep 27 2022, 12:45 AM

Build is green

Patch application report for D8541 (id=30801)

Could not rebase; Attempt merge onto 90534ecae4...

Updating 90534ec..cfdea72
Fast-forward
 swh/scanner/cli.py    |  21 +++++++--
 swh/scanner/client.py | 124 +++++++++++++++++++++++++++++++++++++++++++-------
 swh/scanner/output.py |  53 ++++++++++++++++++++-
 3 files changed, 177 insertions(+), 21 deletions(-)
Changes applied before test
commit cfdea7228ac81d84505dfe6d4432cb24683fb68b
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 23:07:59 2022 +0200

    client: add some basic error detection and rate limiting
    
    We retry on error and we adjust to the `X-RateLimit` header the server
    send us. This is quite basic, and has to struggle against the async
    implementation of the request.
    
    So further works should probably refactor the concurrency of the request
    implementation before going further.
    
    See inline documentation for details.
    
    Better handling of ClientConnectionError are also needed.

commit 48a20c68177bda452837af01dd640e28fa3838f8
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:58:21 2022 +0200

    client: prepare for retry

commit 5050a20f6993bcdf2b566eeee4a7af61db796ee9
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:44:54 2022 +0200

    client: compute the client endpoin once and ealier
    
    This make the user function simpler.

commit 89497f290b7534c19f22f79daacfd078b41f90e1
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:43:49 2022 +0200

    client: transforme the swhid to string earlier
    
    This simplify the function we call in the loop, cannot hurt.

commit 56f61ce8e1749512301b5473ef28605735a923f9
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:31:01 2022 +0200

    client: move `make_request` to be a method
    
    This has not reason to be a closure, so let it not be a closure.

commit d327f68a7206bcafafc8699496799c4e9c36daa6
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:27:13 2022 +0200

    client: reorder some conditionnal
    
    Let us put the simple case first and the longer one after it.

commit ac46a201af3e4480439fb64f8dfe111894766da0
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:25:38 2022 +0200

    client: extract the `get_chunk` closure
    
    This have no reason to be a closure, so we extract it as a proper
    function.

commit 1a417494dff6c76ec03c4dc2c849f8b35cdc9aff
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:23:40 2022 +0200

    output: pick "summary" as the default
    
    This should be less scary than "text" for a first contact.

commit 31ba53e242327bb06bbdbebcf35337254774b82e
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:07:18 2022 +0200

    output: add a "summary" output
    
    This is a small and minimal output to display some information without
    overwhelming the user with a wall of text.
    
    Example output:
Files:                     89
            known:         40   ( 44%)
directories:               11
      fully-known:          3   ( 27%)
  partially-known:          0   (  0%)
```

It can most probably be improved, but this can come later.

commit 821873a035417ceb73a7973687d4745c774a3aee
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:10:38 2022 +0200

scan: start documenting the --output-format option better

This is still quite short, but already better. This leave room for
expanding the format list and the associated description.

commit b3b8ca5f0f5715f9ab470a0993d61d2cebdfb074
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:09:30 2022 +0200

scan: use the current directory as the default the target directory

This seems like a good default and smooth the first contact with the
tool.
See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/169/ for more details.
Harbormaster completed remote builds in B31778: Diff 30801.Sep 27 2022, 12:45 AM
marmoute requested review of this revision.Sep 27 2022, 12:45 AM
marmoute removed a reviewer: bchauvet.Sep 27 2022, 5:42 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 27 2022, 5:42 PM
vlorentz added a subscriber: vlorentz.Sep 27 2022, 6:54 PM

I feel all this retry logic should be in swh.web.client, because all clients need it, not just swh.scanner

swh/scanner/client.py
30–33

use this instead: https://docs.softwareheritage.org/devel/apidoc/swh.core.utils.html#swh.core.utils.grouper

marmoute added a comment.Sep 27 2022, 7:09 PM
In D8541#222294, @vlorentz wrote:

I feel all this retry logic should be in swh.web.client, because all clients need it, not just swh.scanner

The current code is not using swh.web.client and use async/await logic. So it looks like a good idea to merge the two code eventually, however this seems out of scope for that diff that focus on making the swh-scanner more usable.

Before we can envision rate-limit management in swh.web.client, we need to (at least):

  • decide what to do about this async/await version (In my current understanding of the scanner code it is more a burden than an improvement and we should probably drop it)
  • debug the inconsistent X-RateLimit header sent by https://archive.software-heritage.org/
swh/scanner/client.py
30–33

"That will teach me to merely move code around" :-)

This looks like a good idea. I'll update the diff.

vlorentz accepted this revision.Sep 28 2022, 10:00 AM
vlorentz added inline comments.
swh/scanner/client.py
30–33

Oh I didn't notice you merely moved it.

Yeah that code didn't really go through proper reviews so far, so that's bound to happen

This revision is now accepted and ready to land.Sep 28 2022, 10:00 AM
vlorentz retitled this revision from More resilient http requesting. to Make HTTP requests more resilient.Sep 28 2022, 10:03 AM
marmoute updated this revision to Diff 30891.Sep 28 2022, 3:12 PM
  • client: add an option to display http debug information
swh-public-ci added a comment.Sep 28 2022, 3:15 PM

Build is green

Patch application report for D8541 (id=30891)

Could not rebase; Attempt merge onto 90534ecae4...

Updating 90534ec..838b70a
Fast-forward
 swh/scanner/cli.py     |  49 +++++++++++++--
 swh/scanner/client.py  | 159 +++++++++++++++++++++++++++++++++++++++++++------
 swh/scanner/output.py  |  51 +++++++++++++++-
 swh/scanner/scanner.py |  15 ++++-
 4 files changed, 248 insertions(+), 26 deletions(-)
Changes applied before test
commit 838b70a07444f5aa9920f216f4748d54b3f0459d
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Wed Sep 28 14:57:04 2022 +0200

    client: add an option to display http debug information
    
    This should help use to validate the rate limit approach and understand
    the inconsistency in what the server is sending us.

commit c9f735d56b323fa43afece8164403b4affd103aa
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 23:07:59 2022 +0200

    client: add some basic error detection and rate limiting
    
    We retry on error and we adjust to the `X-RateLimit` header the server
    send us. This is quite basic, and has to struggle against the async
    implementation of the request.
    
    So further works should probably refactor the concurrency of the request
    implementation before going further.
    
    See inline documentation for details.
    
    Better handling of ClientConnectionError are also needed.

commit 081f16d886fae54ad540dd6e97e0dcb3eeaeae76
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:58:21 2022 +0200

    client: prepare for retry

commit 854ec22c1e8c592c35ca4a3f627b884f999e2b4b
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:44:54 2022 +0200

    client: compute the client endpoin once and ealier
    
    This make the user function simpler.

commit 890f6f785af56cb0451812d44e5fb1b78d1a8367
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:43:49 2022 +0200

    client: transforme the swhid to string earlier
    
    This simplify the function we call in the loop, cannot hurt.

commit f1c96a31a5b49f61c904f04558269f56d24d452b
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:31:01 2022 +0200

    client: move `make_request` to be a method
    
    This has not reason to be a closure, so let it not be a closure.

commit f4f77020a5042a943540c1010ce33f54d9f16a3a
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:27:13 2022 +0200

    client: reorder some conditionnal
    
    Let us put the simple case first and the longer one after it.

commit de9ff6ae8affac7c02e3de251218ad8c72a4ae40
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:25:38 2022 +0200

    client: extract the `get_chunk` closure
    
    This have no reason to be a closure, so we extract it as a proper
    function.

commit f4f091776951fbbbcdb46d6593715b5883a88f41
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:23:40 2022 +0200

    output: pick "summary" as the default
    
    This should be less scary than "text" for a first contact.

commit 6e3a0bbee9732f530fdfa529b98275b37a7cfa43
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:07:18 2022 +0200

    output: add a "summary" output
    
    This is a small and minimal output to display some information without
    overwhelming the user with a wall of text.
    
    Example output:
Files:                     89
            known:         40   ( 44%)
directories:               11
      fully-known:          3   ( 27%)
  partially-known:          0   (  0%)
```

It can most probably be improved, but this can come later.

commit 821873a035417ceb73a7973687d4745c774a3aee
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:10:38 2022 +0200

scan: start documenting the --output-format option better

This is still quite short, but already better. This leave room for
expanding the format list and the associated description.

commit b3b8ca5f0f5715f9ab470a0993d61d2cebdfb074
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date: Fri Sep 23 22:09:30 2022 +0200

scan: use the current directory as the default the target directory

This seems like a good default and smooth the first contact with the
tool.
See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/176/ for more details.
Harbormaster completed remote builds in B31862: Diff 30891.Sep 28 2022, 3:15 PM
marmoute updated this revision to Diff 30918.Sep 28 2022, 6:18 PM
use the logging module instead of a print
swh-public-ci added a comment.Sep 28 2022, 6:20 PM

Build is green

Patch application report for D8541 (id=30918)

Rebasing onto f4f0917769...

Current branch diff-target is up to date.
Changes applied before test
commit 4a416183ebe96efd8fbbf6d9d012e874a0f1d17c
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Wed Sep 28 14:57:04 2022 +0200

    client: add an option to display http debug information
    
    This should help use to validate the rate limit approach and understand
    the inconsistency in what the server is sending us.

commit c9f735d56b323fa43afece8164403b4affd103aa
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 23:07:59 2022 +0200

    client: add some basic error detection and rate limiting
    
    We retry on error and we adjust to the `X-RateLimit` header the server
    send us. This is quite basic, and has to struggle against the async
    implementation of the request.
    
    So further works should probably refactor the concurrency of the request
    implementation before going further.
    
    See inline documentation for details.
    
    Better handling of ClientConnectionError are also needed.

commit 081f16d886fae54ad540dd6e97e0dcb3eeaeae76
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:58:21 2022 +0200

    client: prepare for retry

commit 854ec22c1e8c592c35ca4a3f627b884f999e2b4b
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:44:54 2022 +0200

    client: compute the client endpoin once and ealier
    
    This make the user function simpler.

commit 890f6f785af56cb0451812d44e5fb1b78d1a8367
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:43:49 2022 +0200

    client: transforme the swhid to string earlier
    
    This simplify the function we call in the loop, cannot hurt.

commit f1c96a31a5b49f61c904f04558269f56d24d452b
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:31:01 2022 +0200

    client: move `make_request` to be a method
    
    This has not reason to be a closure, so let it not be a closure.

commit f4f77020a5042a943540c1010ce33f54d9f16a3a
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:27:13 2022 +0200

    client: reorder some conditionnal
    
    Let us put the simple case first and the longer one after it.

commit de9ff6ae8affac7c02e3de251218ad8c72a4ae40
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:25:38 2022 +0200

    client: extract the `get_chunk` closure
    
    This have no reason to be a closure, so we extract it as a proper
    function.

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/183/ for more details.

Harbormaster completed remote builds in B31891: Diff 30918.Sep 28 2022, 6:20 PM
marmoute updated this revision to Diff 30959.Sep 29 2022, 5:28 PM
rebase on master
swh-public-ci added a comment.Sep 29 2022, 5:30 PM

Build is green

Patch application report for D8541 (id=30959)

Rebasing onto 4363c9a9e3...

Current branch diff-target is up to date.
Changes applied before test
commit 5f3140ab75dace2248c890c63fdabab11b8f9368
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Wed Sep 28 14:57:04 2022 +0200

    client: add an option to display http debug information
    
    This should help use to validate the rate limit approach and understand
    the inconsistency in what the server is sending us.

commit 11b13ce99119e230161e7e4505c121fd9808df45
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 23:07:59 2022 +0200

    client: add some basic error detection and rate limiting
    
    We retry on error and we adjust to the `X-RateLimit` header the server
    send us. This is quite basic, and has to struggle against the async
    implementation of the request.
    
    So further works should probably refactor the concurrency of the request
    implementation before going further.
    
    See inline documentation for details.
    
    Better handling of ClientConnectionError are also needed.

commit 1b05e7f3ee5daf0528e5c5bbe7714126c4795b30
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 19:58:21 2022 +0200

    client: prepare for retry

commit 038e7b9454e70bd7dfc05816b66c052a30ad21cb
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:44:54 2022 +0200

    client: compute the client endpoin once and ealier
    
    This make the user function simpler.

commit a17846cda5b57e449abbf6aa374ca042d1a30ee3
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:43:49 2022 +0200

    client: transforme the swhid to string earlier
    
    This simplify the function we call in the loop, cannot hurt.

commit c44df1c8ecfbe870c9ced6f4b9c46316c94c612e
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:31:01 2022 +0200

    client: move `make_request` to be a method
    
    This has not reason to be a closure, so let it not be a closure.

commit 549faee3fcbf4008f2b12f35688faf8fc9b4dc72
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:27:13 2022 +0200

    client: reorder some conditionnal
    
    Let us put the simple case first and the longer one after it.

commit 5293bbe477f71527b7aa43c6b4d5e75fd5ccddde
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Mon Sep 26 17:25:38 2022 +0200

    client: extract the `get_chunk` closure
    
    This have no reason to be a closure, so we extract it as a proper
    function.

See https://jenkins.softwareheritage.org/job/DTSCN/job/tests-on-diff/188/ for more details.

Harbormaster completed remote builds in B31931: Diff 30959.Sep 29 2022, 5:31 PM
Closed by commit rDTSCN5293bbe477f7: client: extract the `get_chunk` closure (authored by marmoute). · Explain WhySep 29 2022, 5:35 PM
This revision was automatically updated to reflect the committed changes.
marmoute added a commit: rDTSCN5293bbe477f7: client: extract the `get_chunk` closure.
marmoute added a commit: rDTSCNc44df1c8ecfb: client: move `make_request` to be a method.
marmoute added a commit: rDTSCN549faee3fcbf: client: reorder some conditionnal.
marmoute added a commit: rDTSCN038e7b9454e7: client: compute the client endpoin once and ealier.
marmoute added a commit: rDTSCNa17846cda5b5: client: transforme the swhid to string earlier.
marmoute added a commit: rDTSCN1b05e7f3ee5d: client: prepare for retry.
marmoute added a commit: rDTSCN5f3140ab75da: client: add an option to display http debug information.
marmoute added a commit: rDTSCN11b13ce99119: client: add some basic error detection and rate limiting.
bchauvet added a task: T4592: swh-scanner - generic bug fixing, robustness and performance improvements.Sep 30 2022, 9:24 AM

Revision Contents
Changeset List

PathSize
swh/
scanner/
13 lines

Diff 30963

View Options

swh/scanner/client.py

Loading...
About · Developer information · Legal