functional loader through cli
ActivePublic
Actions

Authored by ardumont on Mar 18 2020, 2:31 PM.

Event Timeline

ardumont created this paste.Mar 18 2020, 2:31 PM

so sentry see the issues...

some error not reported in sentry:

DEBUG:swh.loader.package.utils:extrinsic_metadata                                                                                                                                                                                     [84/8086]
ERROR:swh.loader.package.loader:Fail to uncompress https://github.com/lihaoyi/Ammonite/releases/download/1.6.9/2.12-1.6.9
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/core/tarball.py", line 72, in uncompress
    shutil.unpack_archive(tarpath, extract_dir=dest)
  File "/usr/lib/python3.7/shutil.py", line 995, in unpack_archive
    raise ReadError("Unknown archive format '{0}'".format(filename))
shutil.ReadError: Unknown archive format '/tmp/tmpmnd54wj6/2.12-1.6.9'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 368, in _load_revision
    uncompressed_path = self.uncompress(dl_artifacts, dest=tmpdir)
  File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 211, in uncompress
    uncompress(a_path, dest=uncompressed_path)
  File "/usr/lib/python3/dist-packages/swh/core/tarball.py", line 74, in uncompress
    raise ValueError(f'Problem during unpacking {tarpath}. Reason: {e}')
ValueError: Problem during unpacking /tmp/tmpmnd54wj6/2.12-1.6.9. Reason: Unknown archive format '/tmp/tmpmnd54wj6/2.12-1.6.9'

ardumont edited the content of this paste. (Show Details)Mar 18 2020, 2:47 PM

well the export instructions were wrong. Fixed!

well, still no message in sentry.
I'll stop that script and use the proper scheduler stanza on staging.

swhscheduler@scheduler0:~$ swh scheduler -C /etc/softwareheritage/scheduler.yml task-type register --plugins loader.functional
INFO:swh.core.config:Loading config file /etc/softwareheritage/global.ini
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
[INFO] swh.core.config -- Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.functional
[INFO] swh.scheduler.cli.task_type -- Loading entrypoint for plugin loader.functional
INFO:swh.scheduler.cli.task_type:Create task type load-functional in scheduler
[INFO] swh.scheduler.cli.task_type -- Create task type load-functional in scheduler
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add load-functional url=https://nix-community.github.io/nixpkgs-swh/sources.json
INFO:swh.core.config:Loading config file /etc/softwareheritage/global.ini
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
[INFO] swh.core.config -- Loading config file /etc/softwareheritage/scheduler.yml
Created 1 tasks

Task 1186662
  Next run: just now (2020-03-18 15:19:03+00:00)
  Interval: 1 day, 0:00:00
  Type: load-functional
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://nix-community.github.io/nixpkgs-swh/sources.json'

Related to T1991

	#!/usr/bin/env bash

	# to run as swhworker

	export SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_functional.yml
	export SWH_SENTRY_DSN=https://redacted@sentry.softwareheritage.org/9
	export SWH_SENTRY_ENVIRONMENT=staging
	export SWH_MAIN_PACKAGE=swh.loader.core

	swh --log-level=DEBUG loader run functional https://nix-community.github.io/nixpkgs-swh/sources.json

functional loader through cliActivePublicActions

Event Timeline

functional loader through cli
ActivePublic
Actions