Page MenuHomeSoftware Heritage

Jenkins jobs for swh-graph time out while cloning repository
Closed, MigratedEdits Locked

Description

Every Jenkins job for swh-graph (eg. https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/269/console ) fails with this error:

17:54:41   > git init /var/lib/jenkins/workspace/DGRPH/tests-on-diff # timeout=10
17:54:41  Fetching upstream changes from https://forge.softwareheritage.org/source/swh-graph.git
17:54:41   > git --version # timeout=10
17:54:41   > git --version # 'git version 2.30.2'
17:54:41   > git fetch --tags --force --progress -- https://forge.softwareheritage.org/source/swh-graph.git +refs/heads/*:refs/remotes/origin/* # timeout=10
17:55:42  ERROR: Error cloning remote repo 'origin'
17:55:42  hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://forge.softwareheritage.org/source/swh-graph.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
17:55:42  stdout: 
17:55:42  stderr: error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
17:55:42  fatal: the remote end hung up unexpectedly
17:55:42  
17:55:42  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2675)
17:55:42  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2099)
17:55:42  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:85)
17:55:42  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:619)
17:55:42  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:848)
17:55:42  	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1226)
17:55:42  	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
17:55:42  	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
17:55:42  	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
17:55:42  	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
17:55:42  	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
17:55:42  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
17:55:42  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
17:55:42  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
17:55:42  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
17:55:42  	at java.base/java.lang.Thread.run(Thread.java:829)

Event Timeline

vlorentz created this task.

Passing --single-branch option to git clone do not result in a timeout:

$ git clone https://forge.softwareheritage.org/source/swh-graph.git
Cloning into 'swh-graph'...
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
fatal: the remote end hung up unexpectedly

$ git clone --single-branch https://forge.softwareheritage.org/source/swh-graph.git
Cloning into 'swh-graph'...
remote: Enumerating objects: 859, done.
remote: Counting objects: 100% (859/859), done.
remote: Compressing objects: 100% (154/154), done.
remote: Total 7975 (delta 761), reused 722 (delta 692), pack-reused 7116
Receiving objects: 100% (7975/7975), 4.73 MiB | 7.00 MiB/s, done.
Resolving deltas: 100% (3792/3792), done.

This would not resolve the issue of cloning the whole repo through HTTPS but could at least unblock the CI.

This translates the following way in Jenkins pipeline:

(swh) ✔ ~/swh/swh-jenkins-jobs [master|✚ 1⚑ 6] 
15:34 $ git diff
diff --git a/jobs/templates/swh-pipeline.groovy.j2 b/jobs/templates/swh-pipeline.groovy.j2
index ff20db4..d0e0149 100644
--- a/jobs/templates/swh-pipeline.groovy.j2
+++ b/jobs/templates/swh-pipeline.groovy.j2
@@ -30,11 +30,12 @@ pipeline {
           $class: 'GitSCM',
           branches: [[name: "${params.REVISION}"]],
           doGenerateSubmoduleConfigurations: false,
-          extensions: [],
+          extensions: [[$class: 'CloneOption', honorRefspec: true]],
           gitTool: 'Default',
           submoduleCfg: [],
           userRemoteConfigs: [
-            [url: 'https://forge.softwareheritage.org/source/{{repo_name}}.git'],
+            [url: 'https://forge.softwareheritage.org/source/{{repo_name}}.git',
+             refspec: '+refs/heads/master:refs/remotes/origin/master'],
           ],
           browser: [$class: 'Phabricator', repoUrl: 'https://forge.softwareheritage.org', repo: '{{name}}']
         ])

I locally hacked the swh-graph jobs configuration and it effectively removes the timeout issue.

anlambert claimed this task.

Jenkins jobs for swh packages have been updated to only fetch the master branch of repositories, closing this.