Page MenuHomeSoftware Heritage

Npm ListerFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Recent Activity

Jan 8 2023

gitlab-migration changed the status of T2972: Port npm lister to the new Lister API from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:33 PM · Npm Lister, Lister, Sprint 2021 01
gitlab-migration changed the status of T1634: Create database for npm lister from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:26 PM · Npm Lister, Origin-npm
gitlab-migration changed the status of T1398: npm incremental lister from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:26 PM · Npm Lister, Origin-npm
gitlab-migration changed the status of T1380: npm lister from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:26 PM · Npm Lister, Origin-npm

Oct 19 2022

gitlab-migration changed the status of T3773: Deploy swh-scheduler v0.22.0, a subtask of T3769: NPM lister is failing with a database update conflict, from Resolved to Migrated.
Oct 19 2022, 6:05 PM · System administration, Npm Lister
gitlab-migration changed the status of T3773: Deploy swh-scheduler v0.22.0 from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:05 PM · System administration, Npm Lister
gitlab-migration changed the status of T3769: NPM lister is failing with a database update conflict from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:05 PM · System administration, Npm Lister

Dec 8 2021

vsellier closed T3769: NPM lister is failing with a database update conflict as Resolved.

The lister was fixed with the deployment of the swh-scheduler v0.22.0.

Dec 8 2021, 3:05 PM · System administration, Npm Lister
vsellier closed T3773: Deploy swh-scheduler v0.22.0, a subtask of T3769: NPM lister is failing with a database update conflict, as Resolved.
Dec 8 2021, 2:17 PM · System administration, Npm Lister
vsellier closed T3773: Deploy swh-scheduler v0.22.0 as Resolved.
Dec 8 2021, 2:17 PM · System administration, Npm Lister
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.

deployment of version v0.22.0 in production

Dec 8 2021, 2:17 PM · System administration, Npm Lister
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.

Deployment of the version v0.22.0 in staging

Dec 8 2021, 11:44 AM · System administration, Npm Lister

Dec 7 2021

vsellier renamed T3773: Deploy swh-scheduler v0.22.0 from Deploy swh-scheduler v0.21.0 to Deploy swh-scheduler v0.22.0.
Dec 7 2021, 6:43 PM · System administration, Npm Lister
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.

Version v0.21.0 deployed in staging:

root@scheduler0:~# apt list --upgradable 2>/dev/null | grep swh | cut -f1 -d'/' | xargs -t apt install
apt install python3-swh.core python3-swh.counters python3-swh.journal python3-swh.lister python3-swh.loader.core python3-swh.model python3-swh.objstorage python3-swh.scheduler python3-swh.storage 
...
root@scheduler0:~# systemctl reload gunicorn-swh-scheduler.service
Dec 7 2021, 9:48 AM · System administration, Npm Lister
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.

The problem is reproduced in staging before the deployment

swhworker@worker1:~$ swh lister -C /etc/softwareheritage/lister.yml run -l npm 
Traceback (most recent call last):
  File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/lister/cli.py", line 65, in run
    get_lister(lister, **config).run()
  File "/usr/lib/python3/dist-packages/swh/lister/pattern.py", line 130, in run
    full_stats.origins += self.send_origins(origins)
  File "/usr/lib/python3/dist-packages/swh/lister/pattern.py", line 234, in send_origins
    ret = self.scheduler.record_listed_origins(batch_origins)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_
    return self.post(meth._endpoint_path, post_data)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 278, in post
    return self._decode_response(response)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 354, in _decode_response
    self.raise_for_status(response)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 344, in raise_for_status
    raise exception from None
swh.core.api.RemoteException: <RemoteException 500 CardinalityViolation: ['ON CONFLICT DO UPDATE command cannot affect row a second time\nHINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.\n']>
Dec 7 2021, 9:02 AM · System administration, Npm Lister
vsellier changed the status of T3773: Deploy swh-scheduler v0.22.0 from Open to Work in Progress.
Dec 7 2021, 8:59 AM · System administration, Npm Lister

Dec 6 2021

vsellier added a revision to T3769: NPM lister is failing with a database update conflict: D6753: Ensure there is no duplicated origins in the insertion batches.
Dec 6 2021, 4:35 PM · System administration, Npm Lister
ardumont added a comment to T3769: NPM lister is failing with a database update conflict.

We should probably update the scheduler function to just coalesce multiple ListedOrigin objects with the same url before sending them to the database backend, so that this issue is taken care of once and for all.

Dec 6 2021, 2:16 PM · System administration, Npm Lister
olasd added a comment to T3769: NPM lister is failing with a database update conflict.

This happens when a given listed page returns two origins with the same URL.

Dec 6 2021, 2:07 PM · System administration, Npm Lister
ardumont updated the task description for T3769: NPM lister is failing with a database update conflict.
Dec 6 2021, 2:00 PM · System administration, Npm Lister
vsellier renamed T3769: NPM lister is failing with a database update conflict from Unstuck npm lister in production to NPM lister is failing with a database update conflict.
Dec 6 2021, 12:34 PM · System administration, Npm Lister
vsellier added a comment to T3769: NPM lister is failing with a database update conflict.

The last run failed with this error:

Dec 6 2021, 12:32 PM · System administration, Npm Lister
vsellier added a comment to T3769: NPM lister is failing with a database update conflict.

Unstuck the task scheduling:

softwareheritage-scheduler=> begin; update task set next_run=now(), status='next_run_not_scheduled' where id=153874548;
BEGIN
UPDATE 1
softwareheritage-scheduler=*> commit;
COMMIT
Dec 6 2021, 12:30 PM · System administration, Npm Lister
vsellier changed the status of T3769: NPM lister is failing with a database update conflict from Open to Work in Progress.
Dec 6 2021, 12:29 PM · System administration, Npm Lister

Jan 22 2021

ardumont moved T2972: Port npm lister to the new Lister API from code review to done on the Sprint 2021 01 board.
Jan 22 2021, 5:13 PM · Npm Lister, Lister, Sprint 2021 01
anlambert closed T2972: Port npm lister to the new Lister API as Resolved by committing rDLSff232f0d9180: npm: Reimplement lister using new Lister API.
Jan 22 2021, 11:11 AM · Npm Lister, Lister, Sprint 2021 01

Jan 19 2021

anlambert moved T2972: Port npm lister to the new Lister API from in-progress to code review on the Sprint 2021 01 board.
Jan 19 2021, 2:14 PM · Npm Lister, Lister, Sprint 2021 01

Jan 18 2021

anlambert added a revision to T2972: Port npm lister to the new Lister API: D4877: npm: Reimplement lister using new Lister API.
Jan 18 2021, 6:34 PM · Npm Lister, Lister, Sprint 2021 01

Jan 15 2021

anlambert changed the status of T2972: Port npm lister to the new Lister API from Open to Work in Progress.
Jan 15 2021, 4:23 PM · Npm Lister, Lister, Sprint 2021 01
anlambert triaged T2972: Port npm lister to the new Lister API as Normal priority.
Jan 15 2021, 4:23 PM · Npm Lister, Lister, Sprint 2021 01

May 18 2020

ardumont removed a revision from T1398: npm incremental lister: D3162: admin/deposit: Extract origin from swh_anchor_id according to latest change.
May 18 2020, 2:07 PM · Npm Lister, Origin-npm
ardumont added a revision to T1398: npm incremental lister: D3162: admin/deposit: Extract origin from swh_anchor_id according to latest change.
May 18 2020, 1:57 PM · Npm Lister, Origin-npm

Apr 10 2019

ardumont closed T1634: Create database for npm lister as Resolved.
Apr 10 2019, 3:57 PM · Npm Lister, Origin-npm
ardumont added a comment to T1634: Create database for npm lister.
  1. Create role
Apr 10 2019, 3:57 PM · Npm Lister, Origin-npm
anlambert added a project to T1634: Create database for npm lister: Npm Lister.
Apr 10 2019, 10:53 AM · Npm Lister, Origin-npm

Dec 3 2018

anlambert closed T1398: npm incremental lister as Resolved by committing rDLSffe4ac9a3c4d: swh.lister.npm: Add an incremental npm lister.
Dec 3 2018, 6:02 PM · Npm Lister, Origin-npm

Nov 30 2018

anlambert added a project to T1380: npm lister: Npm Lister.
Nov 30 2018, 7:37 PM · Npm Lister, Origin-npm
anlambert added a project to T1398: npm incremental lister: Npm Lister.
Nov 30 2018, 7:36 PM · Npm Lister, Origin-npm
anlambert created Npm Lister.
Nov 30 2018, 7:36 PM