staging: Deploy metadata loader
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	ardumont
	Apr 22 2022, 3:50 PM

Description

Plan to be defined further:

Prepare repository with the correct flag (has-debian-branches, ...) [1]
Prepare ci [2] (already done)
Prepare git repository hosting server (tate) so debian package build happens out of a tag
rDLDMD7350b5bdf6943f30cda72339307757fbc517b7a6: Prepare debian package
debian builds ok? yes [4]
~~Create new sentry issue for the loader~~
~~Prepare puppet manifest for new service swh-worker@loader_metadata~~
D7687: Adapt swh-worker@loader_git manifest to optionally install the dependency on loader metadata
Deploy and restart swh-worker@loader_git

[1] https://docs.softwareheritage.org/devel/tutorials/add-new-package.html

[2] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDMD/

[3]

root@tate:/srv/phabricator# phabricator-setup-hook /srv/phabricator/repos/258 post-receive-swh-modules
Hook post-receive-swh-modules successfully installed on /srv/phabricator/repos/258:
lrwxrwxrwx 1 phabricator phabricator 39 Apr 26 12:57 post-receive -> ../../../hooks/post-receive-swh-modules
root@tate:/srv/phabricator# ls -lah repos/258/hooks/post-receive
lrwxrwxrwx 1 phabricator phabricator 39 Apr 26 12:57 repos/258/hooks/post-receive -> ../../../hooks/post-receive-swh-modules

[4] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDMD/job/gbp-buildpackage/

Revisions and Commits

rDLDBASE Generic VCS/Package Loader
	D7701	rDLDBASEabc5f0889f39 cli: Pass metadata_fetcher_credentials from the config to the loader
rSPSITE puppet-swh-site
	D7687	rSPSITE7d6db46bef09 staging: Activate metadata loader for git loader

Related Objects
Search...

Status	Assigned	Task
Migrated	gitlab-migration	T2201 Indexing / mining
Migrated	gitlab-migration	T2202 Collect extrinsic metadata
Migrated	gitlab-migration	T833 When listing an origin, add origin level metadata to RMD storage
Migrated	gitlab-migration	T4283 Load https://github.com/chromium/chromium with a higher packfile size limit
Migrated	gitlab-migration	T3273 Use "fork" relationships to speed-up initial load of large repositories
Migrated	gitlab-migration	T1102 Handle all GitHub elements
Migrated	gitlab-migration	T1740 fetch extrinsic origin metadata from GitHub
Migrated	gitlab-migration	T4193 staging: Deploy metadata loader
Migrated	gitlab-migration	T4194 staging: Deploy swh-scheduler 1.1.0
Migrated	gitlab-migration	T4195 staging: Deploy swh-loader-core 3.1.0
Migrated	gitlab-migration	T4189 Deploy swh.loader.core v3.0 (and other impacted loaders)

Event Timeline

ardumont triaged this task as Normal priority.Apr 22 2022, 3:50 PM

ardumont created this task.

ardumont updated the task description. (Show Details)

vlorentz added a parent task: T1740: fetch extrinsic origin metadata from GitHub.Apr 22 2022, 3:52 PM

ardumont renamed this task from staging: Deploy metadata fetcher to staging: Deploy metadata loader.Apr 22 2022, 3:53 PM

ardumont added a project: System administration.Apr 22 2022, 3:59 PM

ardumont moved this task from Backlog to Weekly backlog on the System administration board.Apr 25 2022, 10:56 AM

ardumont changed the status of subtask T4194: staging: Deploy swh-scheduler 1.1.0 from Open to Work in Progress.Apr 26 2022, 2:44 PM

ardumont updated the task description. (Show Details)Apr 26 2022, 2:53 PM

ardumont updated the task description. (Show Details)

ardumont updated the task description. (Show Details)Apr 26 2022, 2:58 PM

ardumont mentioned this in rDLDMD7350b5bdf694: Prepare debian package for version 0.0.1.Apr 26 2022, 3:32 PM

ardumont changed the task status from Open to Work in Progress.Apr 26 2022, 3:32 PM

ardumont updated the task description. (Show Details)

ardumont moved this task from Weekly backlog to in-progress on the System administration board.

ardumont updated the task description. (Show Details)Apr 26 2022, 3:36 PM

ardumont updated the task description. (Show Details)Apr 26 2022, 3:42 PM

ardumont mentioned this in rSPPRIVC704471d34724: Reference new sentry project token.Apr 26 2022, 3:45 PM

ardumont updated the task description. (Show Details)Apr 26 2022, 3:51 PM

ardumont updated the task description. (Show Details)Apr 26 2022, 3:55 PM

ardumont updated the task description. (Show Details)

ardumont added a revision: D7687: staging: Activate metadata loader for git loader.Apr 26 2022, 4:28 PM

ardumont updated the task description. (Show Details)Apr 26 2022, 4:29 PM

ardumont mentioned this in rDLDMDbe3a0e24dd19: Drop extra swh/loader/__init__.py.Apr 26 2022, 4:43 PM

ardumont added a commit: rSPSITE7d6db46bef09: staging: Activate metadata loader for git loader.Apr 26 2022, 5:14 PM

ardumont updated the task description. (Show Details)Apr 26 2022, 5:19 PM

ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.

ardumont mentioned this in rSPSITE0b617eed6b76: Properly rename expected metadata_fetcher_credentials config key.Apr 26 2022, 5:54 PM

So status on this, the git loader queue is quite filled in.
So triggering specific loading for github origins through the cli [1] was in order.
We do obtain results after that [2]

[1]

swhworker@worker3:~$ url=https://github.com/progval/irctest ; swh loader run git $url lister_name=github lister_instance_name="github"
WARNING:swh.core.cli:Could not load subcommand graph: DistributionNotFound(Requirement.parse('py4j'), None)
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/progval/irctest' with type 'git'
WARNING:swh.lister.github.utils:No tokens set in configuration, using anonymous mode
Enumerating objects: 6351, done.
Counting objects: 100% (1686/1686), done.
Compressing objects: 100% (294/294), done.
Total 6351 (delta 1456), reused 1572 (delta 1391), pack-reused 4665
INFO:swh.loader.git.loader:Listed 172 refs for repo https://github.com/progval/irctest
/usr/lib/python3/dist-packages/swh/storage/api/client.py:45: DeprecationWarning: Call to deprecated method post. (Use _post instead) -- Deprecated since version 2.1.0.
  return self.post("content/add", {"content": content})
{'status': 'eventful'} for origin 'https://github.com/progval/irctest'

[2]

16:08:22 swh@db1:5432=> select target, format from raw_extrinsic_metadata where target > 'swh:1:ori:' and format='application/vnd.github.v3+json' limit 10;
+----------------------------------------------------+--------------------------------+
|                       target                       |             format             |
+----------------------------------------------------+--------------------------------+
| swh:1:ori:fbc59d47d4cc806cfdae45afc0a77a9a3dd482a6 | application/vnd.github.v3+json |
| swh:1:ori:8d179abca2faf8d58ed7a16c7b619e873b278285 | application/vnd.github.v3+json |
| swh:1:ori:a0016c9d9fc19f7dcb6a76f42bd66c76de3c12eb | application/vnd.github.v3+json |
| swh:1:ori:a0016c9d9fc19f7dcb6a76f42bd66c76de3c12eb | application/vnd.github.v3+json |
+----------------------------------------------------+--------------------------------+
(4 rows)

Time: 113.204 ms

ardumont added a revision: D7701: cli: Pass metadata_fetcher_credentials from the config to the loader.Apr 27 2022, 4:19 PM

vlorentz added a commit: rDLDBASEabc5f0889f39: cli: Pass metadata_fetcher_credentials from the config to the loader.Apr 27 2022, 4:26 PM

ardumont closed this task as Resolved.Apr 27 2022, 4:56 PM

ardumont claimed this task.

ardumont closed subtask T4194: staging: Deploy swh-scheduler 1.1.0 as Resolved.

ardumont moved this task from deployed/landed/monitoring to done on the System administration board.