Page MenuHomeSoftware Heritage
Feed Advanced Search

May 23 2022

ardumont merged task T4146: Test gitlab migrations in the gitlab staging instance into T4064: Test GitLab migration scripts.
May 23 2022, 2:26 PM · System administration, GitLab migration
ardumont merged T4146: Test gitlab migrations in the gitlab staging instance into T4064: Test GitLab migration scripts.
May 23 2022, 2:26 PM · System administration, GitLab migration, Roadmap 2020
ardumont changed the status of T4146: Test gitlab migrations in the gitlab staging instance from Open to Work in Progress.
May 23 2022, 2:24 PM · System administration, GitLab migration
ardumont moved T4266: prod: Deploy swh-loader-core 3.5.0 on staging from Backlog to Weekly backlog on the System administration board.
May 23 2022, 2:24 PM · System administration, Core Loader
ardumont added a comment to T4266: prod: Deploy swh-loader-core 3.5.0 on staging.

3.4.1 is deployed everywhere:

root@pergamon:~# clush -b -w @swh-workers -w @swh-workers-large -w @staging-workers 'dpkg -l python3-swh.loader.core'
---------------
worker[01-18],worker[0-3].internal.staging.swh.network (22)
---------------
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                    Version              Architecture Description
+++-=======================-====================-============-=================================
ii  python3-swh.loader.core 3.4.1-1~swh1~bpo10+1 all          Software Heritage Loader Core
May 23 2022, 2:24 PM · System administration, Core Loader
ardumont added a comment to D7879: Adapt maven lister to list canonical gh urls if any.

So now let's fix the unit tests.
I've managed to break the tests with the recent fixes.

May 23 2022, 2:23 PM
ardumont added inline comments to D7879: Adapt maven lister to list canonical gh urls if any.
May 23 2022, 12:48 PM
ardumont updated the diff for D7879: Adapt maven lister to list canonical gh urls if any.

Adapt according to review

May 23 2022, 11:54 AM
ardumont added inline comments to D7879: Adapt maven lister to list canonical gh urls if any.
May 23 2022, 11:48 AM
ardumont added a comment to D7812: [WIP] Add arch lister module..

I've made several experiments in order to find a better way to list arch linux
package.

The most efficient way I've found is to download tar.gz files which contains package
name as directory and a "desc" file with easy to parse metadata. It works fine but
retrieve only the latest version of a package.

Here are some time execution metrics for downloading archive and parse desc files.

Found 266 packages from https://archive.archlinux.org/repos/last/core/os/x86_64/core.files.tar.gz in 1.4924319160054438 seconds

Found 3035 packages from https://archive.archlinux.org/repos/last/extra/os/x86_64/extra.files.tar.gz in 5.644616681995103 seconds

Found 9161 packages from https://archive.archlinux.org/repos/last/community/os/x86_64/community.files.tar.gz in 16.14458583202213 seconds

Example of retrieved package data after parsing:

{'arch': 'x86_64',
 'repo': 'core',
 'base': 'acl',
 'builddate': '1643730617',
 'conflicts': 'xfsacl',
 'csize': '138970',
 'desc': 'Access control list utilities, libraries and headers',
 'filename': 'acl-2.3.1-2-x86_64.pkg.tar.zst',
 'isize': '325349',
 'license': 'LGPL',
 'md5sum': '718c93159ce4dfc6f789ffe27ce276e8',
 'name': 'acl',
 'packager': 'Christian Hesse <eworm@archlinux.org>',
 'pgpsig': 'iHUEABYIAB0WIQQEKYl95fO9rFN6MGltQr3RFuAGjwUCYflW2QAKCRBtQr3RFuAGj/waAP9U7gJZ0YRfftuGdc4shJdSIfspuWb3nZK+fj7My5z4zQD/SBpepSM3Cxr8Pw2LU5adq4UI0HWFZFsHrg3179XJqgI=',
 'project_url': 'https://savannah.nongnu.org/projects/acl',
 'replaces': 'xfsacl',
 'sha256sum': '20873a994a0728de5b05857129c290e9a8c9bba2236cc30bcffa7b746ffe9218',
 'url': 'https://archive.archlinux.org/packages/.all/acl-2.3.1-2-x86_64.pkg.tar.zst',
 'version': '2.3.1-2'}

If we are ok to get only latest version, we can go this way.

May 23 2022, 10:52 AM
ardumont added a comment to D7812: [WIP] Add arch lister module..

Note:
List the contents of pacman databases as JSON for web applications snippet [1]
[1] https://bbs.archlinux.org/viewtopic.php?pid=1969414#p1969414

May 23 2022, 9:56 AM
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 23 2022, 9:47 AM · Lister
ardumont updated the diff for D7879: Adapt maven lister to list canonical gh urls if any.
  • Improve commit message
  • Reword docstrings
  • Add comments
May 23 2022, 9:24 AM
ardumont updated the summary of D7879: Adapt maven lister to list canonical gh urls if any.
May 23 2022, 9:21 AM

May 20 2022

ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 6:48 PM · Lister
ardumont updated the diff for D7879: Adapt maven lister to list canonical gh urls if any.

What i said but using the right repository this time ¯\_(ツ)_/¯

May 20 2022, 6:29 PM
ardumont updated the test plan for D7879: Adapt maven lister to list canonical gh urls if any.
May 20 2022, 6:29 PM
ardumont updated the diff for D7879: Adapt maven lister to list canonical gh urls if any.

Bump requirements, tests should pass now.

May 20 2022, 6:28 PM
ardumont closed D7880: Deal with git protocol url to canonicalize to https.
May 20 2022, 6:17 PM
ardumont committed rDCORE817c9e099c3b: Deal with git protocol url to canonicalize to https (authored by ardumont).
Deal with git protocol url to canonicalize to https
May 20 2022, 6:17 PM
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 6:16 PM · Lister
ardumont retitled D7879: Adapt maven lister to list canonical gh urls if any from wip: Adapt maven lister to list canonical gh urls if any to Adapt maven lister to list canonical gh urls if any.
May 20 2022, 6:03 PM
ardumont requested review of D7879: Adapt maven lister to list canonical gh urls if any.
May 20 2022, 6:03 PM
ardumont accepted D7881: Add method process_data(), run between fetch_data() and store_data().
May 20 2022, 5:51 PM
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 5:48 PM · Lister
ardumont added inline comments to D7880: Deal with git protocol url to canonicalize to https.
May 20 2022, 5:30 PM
ardumont accepted D7878: svn: Wraps commit info retrieval in a retryable SvnRepo method.

lgtm, one suggestion inline.

May 20 2022, 5:29 PM
ardumont requested review of D7880: Deal with git protocol url to canonicalize to https.
May 20 2022, 5:28 PM
ardumont added a revision to T4232: Listers: Canonicalize listed github origins: D7880: Deal with git protocol url to canonicalize to https.
May 20 2022, 5:26 PM · Lister
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 4:38 PM · Lister
ardumont added a revision to T4232: Listers: Canonicalize listed github origins: D7879: Adapt maven lister to list canonical gh urls if any.
May 20 2022, 4:38 PM · Lister
ardumont closed D7877: Use swh.core.github.pytest_plugin in github tests.
May 20 2022, 4:28 PM
ardumont committed rDLS2ffe9c2aea23: Use swh.core.github.pytest_plugin in github tests (authored by ardumont).
Use swh.core.github.pytest_plugin in github tests
May 20 2022, 4:27 PM
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 4:27 PM · Lister
ardumont requested review of D7877: Use swh.core.github.pytest_plugin in github tests.
May 20 2022, 4:10 PM
ardumont added a revision to T4232: Listers: Canonicalize listed github origins: D7877: Use swh.core.github.pytest_plugin in github tests.
May 20 2022, 4:06 PM · Lister
ardumont accepted D7868: Move store_data from DVCSLoader to a new BaseGitLoader.
May 20 2022, 3:49 PM
ardumont accepted D7875: svn_retry: Retry operation when server sent a truncated HTTP response.
May 20 2022, 3:47 PM
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 3:35 PM · Lister
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 3:35 PM · Lister
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 3:34 PM · Lister
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 3:34 PM · Lister
ardumont accepted D7872: cvs2gitdump: Fix local variable 'expkw' referenced before assignment.
May 20 2022, 3:30 PM
ardumont closed D7870: Use GitHubSession to make canonical computation deal with rate limit.
May 20 2022, 3:26 PM
ardumont committed rDCOREcbcbf0102f30: Use GitHubSession to make canonical computation deal with rate limit (authored by ardumont).
Use GitHubSession to make canonical computation deal with rate limit
May 20 2022, 3:26 PM
ardumont accepted D7864: icinga: Don't monitor kubernetes local mounts.
May 20 2022, 1:14 PM
ardumont accepted D7867: django/models: Map keycloak groups to django ones in OIDCUser model.
May 20 2022, 1:14 PM
ardumont updated the diff for D7870: Use GitHubSession to make canonical computation deal with rate limit.

Add tests around github session as well

May 20 2022, 12:53 PM
ardumont requested review of D7870: Use GitHubSession to make canonical computation deal with rate limit.
May 20 2022, 12:50 PM
ardumont updated the task description for T4232: Listers: Canonicalize listed github origins.
May 20 2022, 12:49 PM · Lister
ardumont updated the summary of D7840: Refactor swh.lister.github.utils to swh.core.github.utils.
May 20 2022, 12:49 PM
ardumont added a revision to T4232: Listers: Canonicalize listed github origins: D7870: Use GitHubSession to make canonical computation deal with rate limit.
May 20 2022, 12:48 PM · Lister
ardumont added a comment to T4064: Test GitLab migration scripts.

Clean up gitlab instance

May 20 2022, 12:30 PM · System administration, GitLab migration, Roadmap 2020
ardumont added a comment to T4064: Test GitLab migration scripts.

Gist of the actions are currently:

  • Mirror the forgerie repository to add some docker commands to allow sandboxed execution (see diffs ^)
  • Adaptations in the forgerie code source to allow migration runs with our current gitlab/phabricator instances (see diffs ^)
  • Current runs are working a bit but do not succeed entirely
May 20 2022, 12:17 PM · System administration, GitLab migration, Roadmap 2020
ardumont updated the task description for T4064: Test GitLab migration scripts.
May 20 2022, 12:15 PM · System administration, GitLab migration, Roadmap 2020
ardumont added a reviewer for D7866: gitlab: Allow gitlab rail command to run without ssh: System administration.
May 20 2022, 9:42 AM
ardumont updated the diff for D7866: gitlab: Allow gitlab rail command to run without ssh.

update

May 20 2022, 9:39 AM
ardumont updated the diff for D7866: gitlab: Allow gitlab rail command to run without ssh.

Add new variable to generated template

May 20 2022, 9:38 AM
ardumont retitled D7866: gitlab: Allow gitlab rail command to run without ssh from wip: gitlab: Allow gitlab rail command to run without ssh to gitlab: Allow gitlab rail command to run without ssh.
May 20 2022, 9:33 AM
ardumont updated the diff for D7866: gitlab: Allow gitlab rail command to run without ssh.

Update

May 20 2022, 9:32 AM
ardumont updated the diff for D7866: gitlab: Allow gitlab rail command to run without ssh.

Drop unnecessary changes

May 20 2022, 9:29 AM
ardumont retitled D7866: gitlab: Allow gitlab rail command to run without ssh from wip: gitlab: Allow grail command to run without ssh to wip: gitlab: Allow gitlab rail command to run without ssh.
May 20 2022, 9:28 AM
ardumont updated the diff for D7866: gitlab: Allow gitlab rail command to run without ssh.

Update docstring and generate template to match docstrings

May 20 2022, 9:28 AM
ardumont added a revision to T4064: Test GitLab migration scripts: D7866: gitlab: Allow gitlab rail command to run without ssh.
May 20 2022, 9:07 AM · System administration, GitLab migration, Roadmap 2020
ardumont requested review of D7866: gitlab: Allow gitlab rail command to run without ssh.
May 20 2022, 9:07 AM
ardumont added a revision to T4064: Test GitLab migration scripts: D7865: Allow remote mysql connection.
May 20 2022, 9:05 AM · System administration, GitLab migration, Roadmap 2020
ardumont requested review of D7865: Allow remote mysql connection.
May 20 2022, 9:05 AM
ardumont updated the diff for D7858: Allow main script executions within container.

Rebase

May 20 2022, 9:03 AM
ardumont retitled D7858: Allow main script executions within container from Allow execution of script within the container to Allow main script executions within container.
May 20 2022, 9:02 AM
ardumont updated the summary of D7858: Allow main script executions within container.
May 20 2022, 9:02 AM
ardumont retitled D7857: Install Dockerfile to allow containerized script execution from Allow main script executions within container to Install Dockerfile to allow containerized script execution.
May 20 2022, 9:02 AM
ardumont retitled D7857: Install Dockerfile to allow containerized script execution from Install Dockerfile to allow containerized script execution to Allow main script executions within container.
May 20 2022, 9:01 AM
ardumont retitled D7857: Install Dockerfile to allow containerized script execution from Bootstrap Dockerfile to ease execution to Install Dockerfile to allow containerized script execution.
May 20 2022, 9:01 AM
ardumont updated the diff for D7857: Install Dockerfile to allow containerized script execution.

Use workdir keyword

May 20 2022, 9:01 AM
ardumont added a revision to T4064: Test GitLab migration scripts: D7858: Allow main script executions within container.
May 20 2022, 9:00 AM · System administration, GitLab migration, Roadmap 2020
ardumont updated the summary of D7858: Allow main script executions within container.
May 20 2022, 9:00 AM
ardumont added a revision to T4064: Test GitLab migration scripts: D7856: docs: Fix typos and clean up whitespace.
May 20 2022, 9:00 AM · System administration, GitLab migration, Roadmap 2020
ardumont updated the summary of D7856: docs: Fix typos and clean up whitespace.
May 20 2022, 9:00 AM
ardumont added a revision to T4064: Test GitLab migration scripts: D7857: Install Dockerfile to allow containerized script execution.
May 20 2022, 8:59 AM · System administration, GitLab migration, Roadmap 2020
ardumont updated the summary of D7857: Install Dockerfile to allow containerized script execution.
May 20 2022, 8:59 AM

May 19 2022

ardumont changed the status of T4064: Test GitLab migration scripts, a subtask of T2225: Migrate to GitLab, from Open to Work in Progress.
May 19 2022, 4:55 PM · meta-task, Roadmap 2022, GitLab migration, Roadmap 2020
ardumont changed the status of T4064: Test GitLab migration scripts from Open to Work in Progress.
May 19 2022, 4:55 PM · System administration, GitLab migration, Roadmap 2020
ardumont added a project to T4064: Test GitLab migration scripts: System administration.
May 19 2022, 4:55 PM · System administration, GitLab migration, Roadmap 2020
ardumont closed D7856: docs: Fix typos and clean up whitespace.
May 19 2022, 4:40 PM
ardumont committed R263:47e142c81e72: docs: Fix typos and clean up whitespace (authored by ardumont).
docs: Fix typos and clean up whitespace
May 19 2022, 4:40 PM
ardumont closed D7850: Add script to mirror an upstream repository to our forge.
May 19 2022, 4:40 PM
ardumont committed rSENV542fa523bf86: Add script to mirror an upstream repository to our forge (authored by ardumont).
Add script to mirror an upstream repository to our forge
May 19 2022, 4:40 PM
ardumont added a comment to D7859: fix mailto encoding for inbound email.

*thumbs up*

May 19 2022, 10:55 AM
ardumont updated the diff for D7858: Allow main script executions within container.

Rebase and improve commit message

May 19 2022, 10:52 AM
ardumont updated the diff for D7857: Install Dockerfile to allow containerized script execution.

Improve readme

May 19 2022, 10:51 AM
ardumont updated the summary of D7858: Allow main script executions within container.
May 19 2022, 10:17 AM
ardumont updated the diff for D7857: Install Dockerfile to allow containerized script execution.

Integrate forgerie deps computation within the docker image.

May 19 2022, 10:16 AM
ardumont updated the diff for D7858: Allow main script executions within container.

Rebase

May 19 2022, 10:15 AM
ardumont updated the test plan for D7858: Allow main script executions within container.
May 19 2022, 9:34 AM
ardumont accepted D7854: base-buster/Dockerfile: Bump nodejs to 14.x.
May 19 2022, 9:31 AM
ardumont accepted D7851: Include counters in the the main docker-compose file.
May 19 2022, 9:31 AM

May 18 2022

ardumont added a reviewer for D7858: Allow main script executions within container: System administrators.
May 18 2022, 6:03 PM
ardumont added a reviewer for D7857: Install Dockerfile to allow containerized script execution: System administrators.
May 18 2022, 6:03 PM
ardumont added a reviewer for D7856: docs: Fix typos and clean up whitespace: System administrators.
May 18 2022, 6:02 PM
ardumont updated the diff for D7858: Allow main script executions within container.

Rebase

May 18 2022, 6:02 PM