Page MenuHomeSoftware Heritage

Fix loading process for deposits
Closed, ResolvedPublic

Description

The deposit is blocked on status verified on the deposit-admin page- https://archive.softwareheritage.org/admin/deposit/
But the status of the deposit is green http://status.softwareheritage.org/

The deposits were made on the 10.10.

IRC chat about that:

olasd (@freenode_olasd:matrix.org)
moranegg: looks like the deposit loader tasks don't have the right shape and the workers are rejecting them
moranegg
olasd : Thanks for the info!

HAL- preprod

When testing with hal-preprod, I get the following error on the interface page:

SWORD SWH response: Forbidden

HAL-Prod

when testing with prod, the deposit is transfered to SWH correctly.

Course of action:

  • have deposit 334 and 335 push correctly to loading worker
  • view and fix deposit tasks that are pushed to loading worker
  • [Opened T2058] review green status (open task or link to existing task)
  • [Opened T2057] check why HAL-preprod has a forbidden error
  • test again

Event Timeline

moranegg triaged this task as Unbreak Now! priority.Oct 28 2019, 11:04 AM
moranegg created this task.

Looks like swh.deposit v0.0.74 has been deployed, but it generates tasks according to T2024 which is still WIP.

see task arguments of recent tasks like:

Task 217689078
  Next run: il y a 2 jours (2019-10-28 09:44:51+00:00)
  Interval: 1 day, 0:00:00
  Type: load-deposit
  Policy: oneshot
  Status: next_run_scheduled
  Priority: 
  Args:
  Keyword args:
    deposit_id: 336
    url: 'https://hal.archives-ouvertes.fr/hal-02132469'

vs.

Task 215063600
  Next run: il y a un mois (2019-09-20 13:24:34+00:00)
  Interval: 1 day, 0:00:00
  Type: load-deposit
  Policy: oneshot
  Status: completed
  Priority: 
  Args:
  Keyword args:
    archive_url: '/1/private/hal/333/raw/'
    deposit_meta_url: '/1/private/hal/333/meta/'
    deposit_update_url: '/1/private/hal/333/update/'

So as far as I understand it, we should revert deposit back to 0.0.73 and recreate these tasks with proper arguments.

I've created 'old-style' tasks, using swh scheduler

$ swh scheduler task add load-deposit -p oneshot  -P high \
    archive_url=/1/private/hal/336/raw/ \
    deposit_meta_url=/1/private/hal/336/meta/ \
    deposit_update_url=/1/private/hal/336/update/ 

@olasd can you (if it's not too much painful) revert python3-swh.deposit to 0.0.73?

olasd added a comment.Oct 30 2019, 2:15 PM

@olasd can you (if it's not too much painful) revert python3-swh.deposit to 0.0.73?

We don't really archive old Debian package versions, so it's not exactly convenient (plus using "not the last version" breaks some assumptions on deployments).

It'd be much easier to revert rDDEP12e3966b168e7be7b19aeb8c0a8d664667c81b3d and reintroduce it when the loader is ready to accept it.

moranegg updated the task description. (Show Details)Nov 5 2019, 10:56 AM

what's the current status of this task?

A full deposit from HAL doesn't work.
It seems to be on HAL side, the deposits stay in partial status and aren't digested.
On HAL interface after moderation there is "Application Error".
I signalled this to Bruno and he said he's fixing it.

moranegg closed this task as Resolved.Fri, Nov 15, 3:27 PM
moranegg updated the task description. (Show Details)