Page MenuHomeSoftware Heritage

models: Keep scheduler task ids reference on deposit model
ClosedPublic

Authored by ardumont on Tue, May 7, 1:53 PM.

Details

Summary

First step to actually ease the rescheduling of new deposits, keeping the
scheduler task identifiers reference on the deposit side.

Related T1703

Test Plan

tox

Then using docker-dev:

doco up -d
  • Trigger a deposit
swh deposit upload --url http://localhost:5006/1 --username test \
                   --password test --collection test \
                   --archive ../swh-docker-dev.tgz  --author mg \
                   --name 'swh-docker-dev'
  • check in db the check_task_id and load_task_id are referenced within the deposit record
$ doco exec swh-deposit bash -c 'psql swh-deposit -c "select id, status, swh_id, check_task_id, load_task_id from deposit"'
 id | status |                       swh_id                       | check_task_id | load_task_id
----+--------+----------------------------------------------------+---------------+--------------
  1 | done   | swh:1:dir:3b0919ddd42be1ba0405d33f383b6e0ee8dedcba | 1             | 2
  (1 row)
  • check those corresponds to the scheduling task:
$ swh scheduler task list
Found 2 tasks

Task 1
  Next run: 19 minutes ago (2019-05-07 11:25:32+00:00)
  Interval: 1 day, 0:00:00
    Type: swh-deposit-archive-checks
    Policy: oneshot
    Status: completed
    Priority:
    Args:
    Keyword args:
      deposit_check_url: '/1/private/test/1/check/'

Task 2
  Next run: 3 minutes ago (2019-05-07 11:41:27+00:00)
    Interval: 1 day, 0:00:00
    Type: swh-deposit-archive-loading
    Policy: oneshot
    Status: next_run_scheduled  # <- strange as the scheduling took place, issue unrelated to the deposit's code though
    Priority:
    Args:
    Keyword args:
      archive_url: '/1/private/test/1/raw/'
      deposit_meta_url: '/1/private/test/1/meta/'
      deposit_update_url: '/1/private/test/1/update/'
  • Empty the record and change status to 'verified'
swh-deposit=# update deposit
swh-deposit-# set status='verified', swh_id=null, swh_anchor_id=null, swh_id_context=null, swh_anchor_id_context=null
swh-deposit-# where id=1;
UPDATE 1
swh-deposit=# select id, status, swh_id, check_task_id, load_task_id from deposit;
 id |  status  | swh_id | check_task_id | load_task_id
----+----------+--------+---------------+--------------
  1 | verified |        | 1             | 2
  (1 row)
  • Respawn manually the loading task using the associated task id
$ swh scheduler task respawn 2
  • Wait for the loading to keep up
  • Check the deposit's status is 'done' again with the right ids (same as initial)
swh-deposit=# select id, status, swh_id, check_task_id, load_task_id from deposit;
 id | status |                       swh_id                       | check_task_id | load_task_id
----+--------+----------------------------------------------------+---------------+--------------
  1 | done   | swh:1:dir:3b0919ddd42be1ba0405d33f383b6e0ee8dedcba | 1             | 2
  (1 row)

Diff Detail

Repository
rDDEP swh-deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Tue, May 7, 1:53 PM
ardumont edited the test plan for this revision. (Show Details)Tue, May 7, 1:54 PM
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)Tue, May 7, 1:56 PM
ardumont added a project: SWORD deposit.
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)Tue, May 7, 2:01 PM
ardumont added inline comments.Tue, May 7, 2:04 PM
swh/deposit/signals.py
77

This is because this function call will also be triggered by the instance.save() below...

douardda added inline comments.Tue, May 7, 2:26 PM
swh/deposit/signals.py
77

So put this in a comment in the code instead of here (IMHO)

ardumont added inline comments.Tue, May 7, 2:45 PM
swh/deposit/signals.py
77

Right, did not really know where to put it ;)

ardumont updated this revision to Diff 4728.Tue, May 7, 2:51 PM

Add explanatory comment

douardda added inline comments.Tue, May 7, 3:55 PM
swh/deposit/models.py
125–126

I'm gonna make my annoying nitpicker, but there should be something in there (comment or description string) giving a clue on what these fields are for.

douardda requested changes to this revision.Tue, May 7, 3:55 PM
This revision now requires changes to proceed.Tue, May 7, 3:55 PM
ardumont added inline comments.Tue, May 7, 4:01 PM
swh/deposit/models.py
125–126

That ain't nitpicking ;)

ardumont added inline comments.Tue, May 7, 4:11 PM
swh/deposit/models.py
125–126

TIL: https://docs.djangoproject.com/en/2.2/topics/db/models/#verbose-field-names

There is a verbose_name parameter for that ;)

ardumont updated this revision to Diff 4732.Tue, May 7, 4:20 PM

Add verbose_name to new fields

douardda accepted this revision.Tue, May 7, 5:35 PM
This revision is now accepted and ready to land.Tue, May 7, 5:35 PM
This revision was automatically updated to reflect the committed changes.