Jan 8 2023
Jan 8 2023
gitlab-migration changed the status of T2972: Port npm lister to the new Lister API from Resolved to Migrated.
gitlab-migration changed the status of T1634: Create database for npm lister from Resolved to Migrated.
Oct 19 2022
Oct 19 2022
gitlab-migration changed the status of T3773: Deploy swh-scheduler v0.22.0, a subtask of T3769: NPM lister is failing with a database update conflict, from Resolved to Migrated.
gitlab-migration changed the status of T3773: Deploy swh-scheduler v0.22.0 from Resolved to Migrated.
gitlab-migration changed the status of T3769: NPM lister is failing with a database update conflict from Resolved to Migrated.
Dec 8 2021
Dec 8 2021
The lister was fixed with the deployment of the swh-scheduler v0.22.0.
vsellier closed T3773: Deploy swh-scheduler v0.22.0, a subtask of T3769: NPM lister is failing with a database update conflict, as Resolved.
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.
deployment of version v0.22.0 in production
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.
Deployment of the version v0.22.0 in staging
Dec 7 2021
Dec 7 2021
vsellier renamed T3773: Deploy swh-scheduler v0.22.0 from Deploy swh-scheduler v0.21.0 to Deploy swh-scheduler v0.22.0.
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.
Version v0.21.0 deployed in staging:
root@scheduler0:~# apt list --upgradable 2>/dev/null | grep swh | cut -f1 -d'/' | xargs -t apt install apt install python3-swh.core python3-swh.counters python3-swh.journal python3-swh.lister python3-swh.loader.core python3-swh.model python3-swh.objstorage python3-swh.scheduler python3-swh.storage ... root@scheduler0:~# systemctl reload gunicorn-swh-scheduler.service
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.
The problem is reproduced in staging before the deployment
swhworker@worker1:~$ swh lister -C /etc/softwareheritage/lister.yml run -l npm Traceback (most recent call last): File "/usr/bin/swh", line 11, in <module> load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')() File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main return swh(auto_envvar_prefix="SWH") File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/lib/python3/dist-packages/swh/lister/cli.py", line 65, in run get_lister(lister, **config).run() File "/usr/lib/python3/dist-packages/swh/lister/pattern.py", line 130, in run full_stats.origins += self.send_origins(origins) File "/usr/lib/python3/dist-packages/swh/lister/pattern.py", line 234, in send_origins ret = self.scheduler.record_listed_origins(batch_origins) File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_ return self.post(meth._endpoint_path, post_data) File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 278, in post return self._decode_response(response) File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 354, in _decode_response self.raise_for_status(response) File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 344, in raise_for_status raise exception from None swh.core.api.RemoteException: <RemoteException 500 CardinalityViolation: ['ON CONFLICT DO UPDATE command cannot affect row a second time\nHINT: Ensure that no rows proposed for insertion within the same command have duplicate constrained values.\n']>
Dec 6 2021
Dec 6 2021
ardumont added a comment to T3769: NPM lister is failing with a database update conflict.
We should probably update the scheduler function to just coalesce multiple ListedOrigin objects with the same url before sending them to the database backend, so that this issue is taken care of once and for all.
olasd added a comment to T3769: NPM lister is failing with a database update conflict.
This happens when a given listed page returns two origins with the same URL.
ardumont updated the task description for T3769: NPM lister is failing with a database update conflict.
vsellier renamed T3769: NPM lister is failing with a database update conflict from Unstuck npm lister in production to NPM lister is failing with a database update conflict.
vsellier added a comment to T3769: NPM lister is failing with a database update conflict.
The last run failed with this error:
vsellier added a comment to T3769: NPM lister is failing with a database update conflict.
Unstuck the task scheduling:
softwareheritage-scheduler=> begin; update task set next_run=now(), status='next_run_not_scheduled' where id=153874548; BEGIN UPDATE 1 softwareheritage-scheduler=*> commit; COMMIT
vsellier changed the status of T3769: NPM lister is failing with a database update conflict from Open to Work in Progress.
Jan 22 2021
Jan 22 2021
ardumont moved T2972: Port npm lister to the new Lister API from code review to done on the Sprint 2021 01 board.
anlambert closed T2972: Port npm lister to the new Lister API as Resolved by committing rDLSff232f0d9180: npm: Reimplement lister using new Lister API.
Jan 19 2021
Jan 19 2021
anlambert moved T2972: Port npm lister to the new Lister API from in-progress to code review on the Sprint 2021 01 board.
Jan 18 2021
Jan 18 2021
Jan 15 2021
Jan 15 2021
anlambert changed the status of T2972: Port npm lister to the new Lister API from Open to Work in Progress.
May 18 2020
May 18 2020
Apr 10 2019
Apr 10 2019
ardumont added a comment to T1634: Create database for npm lister.
- Create role
Dec 3 2018
Dec 3 2018
anlambert closed T1398: npm incremental lister as Resolved by committing rDLSffe4ac9a3c4d: swh.lister.npm: Add an incremental npm lister.
Nov 30 2018
Nov 30 2018
anlambert created Npm Lister.