Details
Details
- Reviewers
anlambert vlorentz - Group Reviewers
Reviewers - Maniphest Tasks
- T3213: Enable save code now of software source code archives for specific users
- Commits
- rDWAPPS6a9fcdd3b9d7: save-code-now: Rename origin type 'bundle' to 'archives'
tox
Diff Detail
Diff Detail
- Repository
- rDWAPPS Web applications
- Lint
Automatic diff as part of commit; lint not applicable. - Unit
Automatic diff as part of commit; unit tests not applicable.
Event Timeline
Comment Actions
Build is green
Patch application report for D5724 (id=20449)
Could not rebase; Attempt merge onto 1e97586914...
Updating 1e975869..3ea98ab3 Fast-forward Makefile.local | 18 ++-- assets/src/bundles/save/index.js | 40 +++++++-- cypress/integration/origin-save.spec.js | 79 +++++++++++++++++ cypress/support/index.js | 4 + swh/web/api/views/origin_save.py | 12 +-- swh/web/auth/utils.py | 13 ++- swh/web/common/origin_save.py | 133 +++++++++++++++++++++------- swh/web/common/swh_templatetags.py | 7 +- swh/web/common/typing.py | 2 +- swh/web/misc/origin_save.py | 8 +- swh/web/templates/misc/origin-save.html | 22 ++++- swh/web/tests/api/test_throttling.py | 15 +--- swh/web/tests/api/views/test_origin_save.py | 72 ++++++++++++++- swh/web/tests/common/test_origin_save.py | 61 ++++++++++++- swh/web/tests/create_test_users.py | 23 +++-- swh/web/tests/misc/test_origin_save.py | 45 ++++++++-- swh/web/tests/test_create_users.py | 16 ++++ swh/web/tests/utils.py | 24 ++++- 18 files changed, 506 insertions(+), 88 deletions(-) create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 3ea98ab3cb00d901743b3e272491cb861ee68c5b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Mon May 10 15:53:32 2021 +0200
save-code-now: Rename origin type 'bundle' to 'archives'
Related to T3213
commit bd095c6a4b161576a9680847e92aebd9559e74c1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:41:34 2021 +0200
Allow privileged user to trigger save code now bundle visit type
Related to T3213
commit 1c8708c1fdf393c36ad7179fbd7bc7596b64f6ee
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Thu May 6 14:05:15 2021 +0200
tests: Add new ambassador user for frontend tests
This refactor existing permission code to allow creation of user with some permissions.
This reworks the Makefile.local to create the users accordingly in different targets.
Related to T3213
commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:25:33 2021 +0200
Add new bundle type to list only when connected as privileged user
This is only doing listing for now. It's in a dedicated commit so review is easier.
Some more work is required ui and server side to actually schedule those new visit type.
It will be done in a future commit.
Related to T3213See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/817/ for more details.
Comment Actions
Build has FAILED
Patch application report for D5724 (id=20452)
Could not rebase; Attempt merge onto 1e97586914...
Updating 1e975869..6a9fcdd3 Fast-forward assets/src/bundles/save/index.js | 40 +++++++-- cypress/integration/origin-save.spec.js | 79 +++++++++++++++++ cypress/support/index.js | 4 + swh/web/api/views/origin_save.py | 12 +-- swh/web/auth/utils.py | 13 ++- swh/web/common/origin_save.py | 133 +++++++++++++++++++++------- swh/web/common/swh_templatetags.py | 7 +- swh/web/common/typing.py | 2 +- swh/web/misc/origin_save.py | 8 +- swh/web/templates/misc/origin-save.html | 22 ++++- swh/web/tests/api/test_throttling.py | 15 +--- swh/web/tests/api/views/test_origin_save.py | 72 ++++++++++++++- swh/web/tests/common/test_origin_save.py | 61 ++++++++++++- swh/web/tests/create_test_users.py | 23 +++-- swh/web/tests/misc/test_origin_save.py | 45 ++++++++-- swh/web/tests/test_create_users.py | 16 ++++ swh/web/tests/utils.py | 24 ++++- 17 files changed, 496 insertions(+), 80 deletions(-) create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 6a9fcdd3b9d79d180407ee7f5fc9a8b36a54e30e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Mon May 10 15:53:32 2021 +0200
save-code-now: Rename origin type 'bundle' to 'archives'
Related to T3213
commit 34bbf06e4e29c95d92c3c458b35c715432662e17
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:41:34 2021 +0200
Allow privileged user to trigger save code now bundle visit type
Related to T3213
commit ee8ece2e6b2c2820a2ee493f9f7f7ff987eaa6f4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Thu May 6 14:05:15 2021 +0200
tests: Add new ambassador user for frontend tests
This refactor existing permission code to allow creation of user with some permissions.
This reworks the Makefile.local to create the users accordingly in different targets.
Related to T3213
commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:25:33 2021 +0200
Add new bundle type to list only when connected as privileged user
This is only doing listing for now. It's in a dedicated commit so review is easier.
Some more work is required ui and server side to actually schedule those new visit type.
It will be done in a future commit.
Related to T3213Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/820/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/820/console
Comment Actions
Build has FAILED
Patch application report for D5724 (id=20452)
Could not rebase; Attempt merge onto 1e97586914...
Updating 1e975869..6a9fcdd3 Fast-forward assets/src/bundles/save/index.js | 40 +++++++-- cypress/integration/origin-save.spec.js | 79 +++++++++++++++++ cypress/support/index.js | 4 + swh/web/api/views/origin_save.py | 12 +-- swh/web/auth/utils.py | 13 ++- swh/web/common/origin_save.py | 133 +++++++++++++++++++++------- swh/web/common/swh_templatetags.py | 7 +- swh/web/common/typing.py | 2 +- swh/web/misc/origin_save.py | 8 +- swh/web/templates/misc/origin-save.html | 22 ++++- swh/web/tests/api/test_throttling.py | 15 +--- swh/web/tests/api/views/test_origin_save.py | 72 ++++++++++++++- swh/web/tests/common/test_origin_save.py | 61 ++++++++++++- swh/web/tests/create_test_users.py | 23 +++-- swh/web/tests/misc/test_origin_save.py | 45 ++++++++-- swh/web/tests/test_create_users.py | 16 ++++ swh/web/tests/utils.py | 24 ++++- 17 files changed, 496 insertions(+), 80 deletions(-) create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 6a9fcdd3b9d79d180407ee7f5fc9a8b36a54e30e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Mon May 10 15:53:32 2021 +0200
save-code-now: Rename origin type 'bundle' to 'archives'
Related to T3213
commit 34bbf06e4e29c95d92c3c458b35c715432662e17
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:41:34 2021 +0200
Allow privileged user to trigger save code now bundle visit type
Related to T3213
commit ee8ece2e6b2c2820a2ee493f9f7f7ff987eaa6f4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Thu May 6 14:05:15 2021 +0200
tests: Add new ambassador user for frontend tests
This refactor existing permission code to allow creation of user with some permissions.
This reworks the Makefile.local to create the users accordingly in different targets.
Related to T3213
commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:25:33 2021 +0200
Add new bundle type to list only when connected as privileged user
This is only doing listing for now. It's in a dedicated commit so review is easier.
Some more work is required ui and server side to actually schedule those new visit type.
It will be done in a future commit.
Related to T3213Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/822/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/822/console
Comment Actions
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/823/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/823/console
Comment Actions
why plural?
That's what @anlambert suggested in irc as this will be about adding multiple archives per origin.
And i saw no harm to use the plural form nor anyone suggesting otherwise.
I don't mind either way.
Comment Actions
Build is green
Patch application report for D5724 (id=20452)
Could not rebase; Attempt merge onto 1e97586914...
Updating 1e975869..6a9fcdd3 Fast-forward assets/src/bundles/save/index.js | 40 +++++++-- cypress/integration/origin-save.spec.js | 79 +++++++++++++++++ cypress/support/index.js | 4 + swh/web/api/views/origin_save.py | 12 +-- swh/web/auth/utils.py | 13 ++- swh/web/common/origin_save.py | 133 +++++++++++++++++++++------- swh/web/common/swh_templatetags.py | 7 +- swh/web/common/typing.py | 2 +- swh/web/misc/origin_save.py | 8 +- swh/web/templates/misc/origin-save.html | 22 ++++- swh/web/tests/api/test_throttling.py | 15 +--- swh/web/tests/api/views/test_origin_save.py | 72 ++++++++++++++- swh/web/tests/common/test_origin_save.py | 61 ++++++++++++- swh/web/tests/create_test_users.py | 23 +++-- swh/web/tests/misc/test_origin_save.py | 45 ++++++++-- swh/web/tests/test_create_users.py | 16 ++++ swh/web/tests/utils.py | 24 ++++- 17 files changed, 496 insertions(+), 80 deletions(-) create mode 100644 swh/web/tests/test_create_users.py
Changes applied before test
commit 6a9fcdd3b9d79d180407ee7f5fc9a8b36a54e30e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Mon May 10 15:53:32 2021 +0200
save-code-now: Rename origin type 'bundle' to 'archives'
Related to T3213
commit 34bbf06e4e29c95d92c3c458b35c715432662e17
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:41:34 2021 +0200
Allow privileged user to trigger save code now bundle visit type
Related to T3213
commit ee8ece2e6b2c2820a2ee493f9f7f7ff987eaa6f4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Thu May 6 14:05:15 2021 +0200
tests: Add new ambassador user for frontend tests
This refactor existing permission code to allow creation of user with some permissions.
This reworks the Makefile.local to create the users accordingly in different targets.
Related to T3213
commit 2a4f2e966a034e034a736e7e10b2b572ef32c89b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed May 5 15:25:33 2021 +0200
Add new bundle type to list only when connected as privileged user
This is only doing listing for now. It's in a dedicated commit so review is easier.
Some more work is required ui and server side to actually schedule those new visit type.
It will be done in a future commit.
Related to T3213See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/824/ for more details.
Comment Actions
After having hacked the docker environment to add the archive loader task (currently load-tar), I got that error in swh-scheduler service:
swh-scheduler_1 | ERROR:root:null value in column "retries_left" violates not-null constraint
swh-scheduler_1 | DETAIL: Failing row contains (1, load-tar, {"args": [], "kwargs": {"url": "https://www.kermitproject.org/ar..., 2021-05-10 16:55:16.171508+00, null, next_run_not_scheduled, oneshot, null, high).
swh-scheduler_1 | CONTEXT: SQL statement "insert into task (type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1 | retries_left, priority)
swh-scheduler_1 | select type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1 | retries_left, priority
swh-scheduler_1 | from tmp_task t
swh-scheduler_1 | where not exists(select 1
swh-scheduler_1 | from task
swh-scheduler_1 | where type = t.type and
swh-scheduler_1 | md5(arguments::text) = md5(t.arguments::text) and
swh-scheduler_1 | arguments = t.arguments and
swh-scheduler_1 | policy = t.policy and
swh-scheduler_1 | priority is not distinct from t.priority and
swh-scheduler_1 | status = t.status)"
swh-scheduler_1 | PL/pgSQL function swh_scheduler_create_tasks_from_temp() line 12 at SQL statement
swh-scheduler_1 | Traceback (most recent call last):
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
swh-scheduler_1 | rv = self.dispatch_request()
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
swh-scheduler_1 | return self.view_functions[rule.endpoint](**req.view_args)
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 153, in newf
swh-scheduler_1 | return f.negotiator(*args, **kwargs)
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__
swh-scheduler_1 | result = self.func(*args, **kwargs)
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 460, in _f
swh-scheduler_1 | return obj_meth(**kw)
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/db/common.py", line 62, in _meth
swh-scheduler_1 | return meth(self, *args, db=db, cur=cur, **kwargs)
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/backend.py", line 492, in create_tasks
swh-scheduler_1 | cur.execute(query)
swh-scheduler_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/psycopg2/extras.py", line 251, in execute
swh-scheduler_1 | return super(RealDictCursor, self).execute(query, vars)
swh-scheduler_1 | psycopg2.errors.NotNullViolation: null value in column "retries_left" violates not-null constraint
swh-scheduler_1 | DETAIL: Failing row contains (1, load-tar, {"args": [], "kwargs": {"url": "https://www.kermitproject.org/ar..., 2021-05-10 16:55:16.171508+00, null, next_run_not_scheduled, oneshot, null, high).
swh-scheduler_1 | CONTEXT: SQL statement "insert into task (type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1 | retries_left, priority)
swh-scheduler_1 | select type, arguments, next_run, status, current_interval, policy,
swh-scheduler_1 | retries_left, priority
swh-scheduler_1 | from tmp_task t
swh-scheduler_1 | where not exists(select 1
swh-scheduler_1 | from task
swh-scheduler_1 | where type = t.type and
swh-scheduler_1 | md5(arguments::text) = md5(t.arguments::text) and
swh-scheduler_1 | arguments = t.arguments and
swh-scheduler_1 | policy = t.policy and
swh-scheduler_1 | priority is not distinct from t.priority and
swh-scheduler_1 | status = t.status)"
swh-scheduler_1 | PL/pgSQL function swh_scheduler_create_tasks_from_temp() line 12 at SQL statement| swh/web/common/origin_save.py | ||
|---|---|---|
| 117 | The loader swh.loader.package.archive.ArchiveLoader is currently named tar so I guess it should be renamed. | |
| swh/web/common/origin_save.py | ||
|---|---|---|
| 117 | Do you mean we should change the type 'tar' to type 'archive'? That type 'tar' is the original one fom the first 'tar' loader we had. So we kept it for Depends on what you mean by renaming. If moving from 'tar' to 'archive' from now on for new values, it should be simple Relatedly, i forgot to open D5732 for the new save code now task. | |