Differential D6487

Return 0 slot if no more slots available in the queues
ClosedPublic
Actions

Authored by ardumont on Oct 15 2021, 3:24 PM.

Details

Reviewers

None

Group Reviewers

Reviewers

Commits

rDSCH3aed7bf1cc11: Return 0 slot if no more slots available in the queues

Summary

This scenario happens with the loader oneshot for example. This loader deals with more
than 1 type of origins to ingest in the same queue. So the computation of that function
returned negative value [1]. Which is ultimately not possible to execute in sql [1].

This commits fixes that behavior. This also explicits that the function must return
positive values in its docstring.

[1]

...
psycopg2.errors.InvalidRowCountInLimitClause: LIMIT must not be negative

[2]

Traceback (most recent call last):
  File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/scheduler/cli/origin.py", line 205, in send_to_celery
    lister_uuid=lister_uuid,
  File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
    return meth(self, *args, db=db, cur=cur, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/scheduler/backend.py", line 525, in grab_next_visits                                                                                                                                                    cur.execute(query, tuple(query_args))
  File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 243, in execute
    return super(RealDictCursor, self).execute(query, vars)
psycopg2.errors.InvalidRowCountInLimitClause: LIMIT must not be negative

Test Plan

tox

patched in production and it no longer crashes with [2]

Diff Detail

Repository

rDSCH Scheduling utilities

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Oct 15 2021, 3:24 PM

ardumont edited the summary of this revision. (Show Details)Oct 15 2021, 3:26 PM

ardumont edited the test plan for this revision. (Show Details)

Build is green

Patch application report for D6487 (id=23565)

Rebasing onto ecc14007aa...

Current branch diff-target is up to date.

Changes applied before test

commit 3aed7bf1cc114a874ecc86e87a2fb9df961b6773
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Oct 15 15:14:12 2021 +0200

    Return 0 slot if no more slots available in the queues
    
    This scenario happens with the loader oneshot for example. This loader deals with more
    than 1 type of origins to ingest in the same queue. So the computation of that function
    returned negative value [1]. Which is ultimately not possible to execute in sql [1].
    
    This commits fixes that behavior. This also explicits that the function must return
    positive values in its docstring.
    
    [1]

...
psycopg2.errors.InvalidRowCountInLimitClause: LIMIT must not be negative
```

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/454/ for more details.

Harbormaster completed remote builds in B24466: Diff 23565.Oct 15 2021, 3:27 PM

ardumont requested review of this revision.Oct 15 2021, 3:27 PM

This revision was not accepted when it landed; it landed in state Needs Review.Oct 15 2021, 3:49 PM

Closed by commit rDSCH3aed7bf1cc11: Return 0 slot if no more slots available in the queues (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSCH3aed7bf1cc11: Return 0 slot if no more slots available in the queues.

Revision Contents
Changeset List

Path

Size

swh/

scheduler/

celery_backend/

config.py

9 lines

tests/

test_config.py

10 lines

Diff 23568

View Options

swh/scheduler/celery_backend/config.py