Page MenuHomeSoftware Heritage

save-code-now: Rename origin type 'bundle' to 'archives'
ClosedPublic

Authored by ardumont on May 10 2021, 4:44 PM.

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D5724 (id=20449)

Could not rebase; Attempt merge onto 1e97586914...

Updating 1e975869..3ea98ab3
Fast-forward
 Makefile.local                              |  18 ++--
 assets/src/bundles/save/index.js            |  40 +++++++--
 cypress/integration/origin-save.spec.js     |  79 +++++++++++++++++
 cypress/support/index.js                    |   4 +
 swh/web/api/views/origin_save.py            |  12 +--
 swh/web/auth/utils.py                       |  13 ++-
 swh/web/common/origin_save.py               | 133 +++++++++++++++++++++-------
 swh/web/common/swh_templatetags.py          |   7 +-
 swh/web/common/typing.py                    |   2 +-
 swh/web/misc/origin_save.py                 |   8 +-
 swh/web/templates/misc/origin-save.html     |  22 ++++-
 swh/web/tests/api/test_throttling.py        |  15 +---
 swh/web/tests/api/views/test_origin_save.py |  72 ++++++++++++++-
 swh/web/tests/common/test_origin_save.py    |  61 ++++++++++++-
 swh/web/tests/create_test_users.py          |  23 +++--
 swh/web/tests/misc/test_origin_save.py      |  45 ++++++++--
 swh/web/tests/test_create_users.py          |  16 ++++
 swh/web/tests/utils.py                      |  24 ++++-
 18 files changed, 506 insertions(+), 88 deletions(-)
 create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 3ea98ab3cb00d901743b3e272491cb861ee68c5b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon May 10 15:53:32 2021 +0200

    save-code-now: Rename origin type 'bundle' to 'archives'
    
    Related to T3213

commit bd095c6a4b161576a9680847e92aebd9559e74c1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:41:34 2021 +0200

    Allow privileged user to trigger save code now bundle visit type
    
    Related to T3213

commit 1c8708c1fdf393c36ad7179fbd7bc7596b64f6ee
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu May 6 14:05:15 2021 +0200

    tests: Add new ambassador user for frontend tests
    
    This refactor existing permission code to allow creation of user with some permissions.
    
    This reworks the Makefile.local to create the users accordingly in different targets.
    
    Related to T3213

commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:25:33 2021 +0200

    Add new bundle type to list only when connected as privileged user
    
    This is only doing listing for now. It's in a dedicated commit so review is easier.
    
    Some more work is required ui and server side to actually schedule those new visit type.
    It will be done in a future commit.
    
    Related to T3213

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/817/ for more details.

Build has FAILED

Patch application report for D5724 (id=20452)

Could not rebase; Attempt merge onto 1e97586914...

Updating 1e975869..6a9fcdd3
Fast-forward
 assets/src/bundles/save/index.js            |  40 +++++++--
 cypress/integration/origin-save.spec.js     |  79 +++++++++++++++++
 cypress/support/index.js                    |   4 +
 swh/web/api/views/origin_save.py            |  12 +--
 swh/web/auth/utils.py                       |  13 ++-
 swh/web/common/origin_save.py               | 133 +++++++++++++++++++++-------
 swh/web/common/swh_templatetags.py          |   7 +-
 swh/web/common/typing.py                    |   2 +-
 swh/web/misc/origin_save.py                 |   8 +-
 swh/web/templates/misc/origin-save.html     |  22 ++++-
 swh/web/tests/api/test_throttling.py        |  15 +---
 swh/web/tests/api/views/test_origin_save.py |  72 ++++++++++++++-
 swh/web/tests/common/test_origin_save.py    |  61 ++++++++++++-
 swh/web/tests/create_test_users.py          |  23 +++--
 swh/web/tests/misc/test_origin_save.py      |  45 ++++++++--
 swh/web/tests/test_create_users.py          |  16 ++++
 swh/web/tests/utils.py                      |  24 ++++-
 17 files changed, 496 insertions(+), 80 deletions(-)
 create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 6a9fcdd3b9d79d180407ee7f5fc9a8b36a54e30e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon May 10 15:53:32 2021 +0200

    save-code-now: Rename origin type 'bundle' to 'archives'
    
    Related to T3213

commit 34bbf06e4e29c95d92c3c458b35c715432662e17
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:41:34 2021 +0200

    Allow privileged user to trigger save code now bundle visit type
    
    Related to T3213

commit ee8ece2e6b2c2820a2ee493f9f7f7ff987eaa6f4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu May 6 14:05:15 2021 +0200

    tests: Add new ambassador user for frontend tests
    
    This refactor existing permission code to allow creation of user with some permissions.
    
    This reworks the Makefile.local to create the users accordingly in different targets.
    
    Related to T3213

commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:25:33 2021 +0200

    Add new bundle type to list only when connected as privileged user
    
    This is only doing listing for now. It's in a dedicated commit so review is easier.
    
    Some more work is required ui and server side to actually schedule those new visit type.
    It will be done in a future commit.
    
    Related to T3213

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/820/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/820/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 10 2021, 5:21 PM
Harbormaster failed remote builds in B21429: Diff 20452!

Build has FAILED

Patch application report for D5724 (id=20452)

Could not rebase; Attempt merge onto 1e97586914...

Updating 1e975869..6a9fcdd3
Fast-forward
 assets/src/bundles/save/index.js            |  40 +++++++--
 cypress/integration/origin-save.spec.js     |  79 +++++++++++++++++
 cypress/support/index.js                    |   4 +
 swh/web/api/views/origin_save.py            |  12 +--
 swh/web/auth/utils.py                       |  13 ++-
 swh/web/common/origin_save.py               | 133 +++++++++++++++++++++-------
 swh/web/common/swh_templatetags.py          |   7 +-
 swh/web/common/typing.py                    |   2 +-
 swh/web/misc/origin_save.py                 |   8 +-
 swh/web/templates/misc/origin-save.html     |  22 ++++-
 swh/web/tests/api/test_throttling.py        |  15 +---
 swh/web/tests/api/views/test_origin_save.py |  72 ++++++++++++++-
 swh/web/tests/common/test_origin_save.py    |  61 ++++++++++++-
 swh/web/tests/create_test_users.py          |  23 +++--
 swh/web/tests/misc/test_origin_save.py      |  45 ++++++++--
 swh/web/tests/test_create_users.py          |  16 ++++
 swh/web/tests/utils.py                      |  24 ++++-
 17 files changed, 496 insertions(+), 80 deletions(-)
 create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 6a9fcdd3b9d79d180407ee7f5fc9a8b36a54e30e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon May 10 15:53:32 2021 +0200

    save-code-now: Rename origin type 'bundle' to 'archives'
    
    Related to T3213

commit 34bbf06e4e29c95d92c3c458b35c715432662e17
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:41:34 2021 +0200

    Allow privileged user to trigger save code now bundle visit type
    
    Related to T3213

commit ee8ece2e6b2c2820a2ee493f9f7f7ff987eaa6f4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu May 6 14:05:15 2021 +0200

    tests: Add new ambassador user for frontend tests
    
    This refactor existing permission code to allow creation of user with some permissions.
    
    This reworks the Makefile.local to create the users accordingly in different targets.
    
    Related to T3213

commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:25:33 2021 +0200

    Add new bundle type to list only when connected as privileged user
    
    This is only doing listing for now. It's in a dedicated commit so review is easier.
    
    Some more work is required ui and server side to actually schedule those new visit type.
    It will be done in a future commit.
    
    Related to T3213

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/822/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/822/console

why plural?

That's what @anlambert suggested in irc as this will be about adding multiple archives per origin.
And i saw no harm to use the plural form nor anyone suggesting otherwise.

I don't mind either way.

Build is green

Patch application report for D5724 (id=20452)

Could not rebase; Attempt merge onto 1e97586914...

Updating 1e975869..6a9fcdd3
Fast-forward
 assets/src/bundles/save/index.js            |  40 +++++++--
 cypress/integration/origin-save.spec.js     |  79 +++++++++++++++++
 cypress/support/index.js                    |   4 +
 swh/web/api/views/origin_save.py            |  12 +--
 swh/web/auth/utils.py                       |  13 ++-
 swh/web/common/origin_save.py               | 133 +++++++++++++++++++++-------
 swh/web/common/swh_templatetags.py          |   7 +-
 swh/web/common/typing.py                    |   2 +-
 swh/web/misc/origin_save.py                 |   8 +-
 swh/web/templates/misc/origin-save.html     |  22 ++++-
 swh/web/tests/api/test_throttling.py        |  15 +---
 swh/web/tests/api/views/test_origin_save.py |  72 ++++++++++++++-
 swh/web/tests/common/test_origin_save.py    |  61 ++++++++++++-
 swh/web/tests/create_test_users.py          |  23 +++--
 swh/web/tests/misc/test_origin_save.py      |  45 ++++++++--
 swh/web/tests/test_create_users.py          |  16 ++++
 swh/web/tests/utils.py                      |  24 ++++-
 17 files changed, 496 insertions(+), 80 deletions(-)
 create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 6a9fcdd3b9d79d180407ee7f5fc9a8b36a54e30e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon May 10 15:53:32 2021 +0200

    save-code-now: Rename origin type 'bundle' to 'archives'
    
    Related to T3213

commit 34bbf06e4e29c95d92c3c458b35c715432662e17
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:41:34 2021 +0200

    Allow privileged user to trigger save code now bundle visit type
    
    Related to T3213

commit ee8ece2e6b2c2820a2ee493f9f7f7ff987eaa6f4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu May 6 14:05:15 2021 +0200

    tests: Add new ambassador user for frontend tests
    
    This refactor existing permission code to allow creation of user with some permissions.
    
    This reworks the Makefile.local to create the users accordingly in different targets.
    
    Related to T3213

commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed May 5 15:25:33 2021 +0200

    Add new bundle type to list only when connected as privileged user
    
    This is only doing listing for now. It's in a dedicated commit so review is easier.
    
    Some more work is required ui and server side to actually schedule those new visit type.
    It will be done in a future commit.
    
    Related to T3213

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/824/ for more details.

This revision is now accepted and ready to land.May 10 2021, 6:54 PM

After having hacked the docker environment to add the archive loader task (currently load-tar), I got that error in swh-scheduler service:

swh-scheduler_1                 | ERROR:root:null value in column "retries_left" violates not-null constraint
swh-scheduler_1                 | DETAIL:  Failing row contains (1, load-tar, {"args": [], "kwargs": {"url": "https://www.kermitproject.org/ar..., 2021-05-10 16:55:16.171508+00, null, next_run_not_scheduled, oneshot, null, high).
swh-scheduler_1                 | CONTEXT:  SQL statement "insert into task (type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1                 |                     retries_left, priority)
swh-scheduler_1                 |     select type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1                 |            retries_left, priority
swh-scheduler_1                 |     from tmp_task t
swh-scheduler_1                 |     where not exists(select 1
swh-scheduler_1                 |                      from task
swh-scheduler_1                 |                      where type = t.type and
swh-scheduler_1                 |                            md5(arguments::text) = md5(t.arguments::text) and
swh-scheduler_1                 |                            arguments = t.arguments and
swh-scheduler_1                 |                            policy = t.policy and
swh-scheduler_1                 |                            priority is not distinct from t.priority and
swh-scheduler_1                 |                            status = t.status)"
swh-scheduler_1                 | PL/pgSQL function swh_scheduler_create_tasks_from_temp() line 12 at SQL statement
swh-scheduler_1                 | Traceback (most recent call last):
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
swh-scheduler_1                 |     rv = self.dispatch_request()
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
swh-scheduler_1                 |     return self.view_functions[rule.endpoint](**req.view_args)
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 153, in newf
swh-scheduler_1                 |     return f.negotiator(*args, **kwargs)
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__
swh-scheduler_1                 |     result = self.func(*args, **kwargs)
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 460, in _f
swh-scheduler_1                 |     return obj_meth(**kw)
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/db/common.py", line 62, in _meth
swh-scheduler_1                 |     return meth(self, *args, db=db, cur=cur, **kwargs)
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/backend.py", line 492, in create_tasks
swh-scheduler_1                 |     cur.execute(query)
swh-scheduler_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/psycopg2/extras.py", line 251, in execute
swh-scheduler_1                 |     return super(RealDictCursor, self).execute(query, vars)
swh-scheduler_1                 | psycopg2.errors.NotNullViolation: null value in column "retries_left" violates not-null constraint
swh-scheduler_1                 | DETAIL:  Failing row contains (1, load-tar, {"args": [], "kwargs": {"url": "https://www.kermitproject.org/ar..., 2021-05-10 16:55:16.171508+00, null, next_run_not_scheduled, oneshot, null, high).
swh-scheduler_1                 | CONTEXT:  SQL statement "insert into task (type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1                 |                     retries_left, priority)
swh-scheduler_1                 |     select type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1                 |            retries_left, priority
swh-scheduler_1                 |     from tmp_task t
swh-scheduler_1                 |     where not exists(select 1
swh-scheduler_1                 |                      from task
swh-scheduler_1                 |                      where type = t.type and
swh-scheduler_1                 |                            md5(arguments::text) = md5(t.arguments::text) and
swh-scheduler_1                 |                            arguments = t.arguments and
swh-scheduler_1                 |                            policy = t.policy and
swh-scheduler_1                 |                            priority is not distinct from t.priority and
swh-scheduler_1                 |                            status = t.status)"
swh-scheduler_1                 | PL/pgSQL function swh_scheduler_create_tasks_from_temp() line 12 at SQL statement
swh/web/common/origin_save.py
117

The loader swh.loader.package.archive.ArchiveLoader is currently named tar so I guess it should be renamed.

swh/web/common/origin_save.py
117

Do you mean we should change the type 'tar' to type 'archive'?

That type 'tar' is the original one fom the first 'tar' loader we had. So we kept it for
retro-compatibility, I guess.

Depends on what you mean by renaming.

If moving from 'tar' to 'archive' from now on for new values, it should be simple
enough. If we plan to also migrate the data, that could be painful (migrate values in
backend + kafka topics).

Relatedly, i forgot to open D5732 for the new save code now task.