Jenkins 并行流水线构建

时间：2021-01-02 00:21:23

标签： jenkins groovy build parallel-processing jenkins-pipeline

并行管道似乎正在跨并行分支重用构建。

我一直在玩并行管道以了解它们的工作原理。假设“FailRarely”是另一项有 10% 的时间失败但有 90% 的时间报告成功的作业。

我的管道脚本如下：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
            }
        }
    }
}

如您所见，它只是尝试并行运行作业六次。 不幸的是，日志似乎表明它正在为所有并行作业使用 相同 作业：

[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

请注意，它对所有管道构建使用了“作业 57”。

我做错了什么？

*** 编辑：添加附加信息 ***

做同样的工作，我嵌入了在 FailRarely 中执行的任务，以便不再执行从管道中调用另一个构建：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 2') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 3') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 4') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 5') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 6') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
            }
        }
    }
}

这符合我的预期：

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\parallel-pipeline-2
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] // node
[Pipeline] // node
[Pipeline] {
[Pipeline] {
[Pipeline] }
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] {
[Pipeline] // stage
[Pipeline] }
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

所以这似乎与主作业如何调用“FailRarely”作业有关。

1 个答案:

答案 0 :(得分：0)

在主要并行作业和工作线程 ('FailRarely') 项目中尝试了多种组合后，我不断得出一个结论：

如果对工作器有相同的调用，主作业实际上只会启动一个工作器，并为所有实例重用结果。

在我看来，这是在启动构建时管道中的某种优化，尽管我找不到这方面的文档。

解决方法很简单。一旦您向工作程序项目添加一个参数，并为该参数提供不同的值，主作业就会意识到它需要多次启动作业并正确执行。

因此，我向 FailRarely 作业（名为 IGNORED）添加了一个实际上并未使用的字符串参数。然后，我为每个 build 命令的参数添加了唯一值以使其唯一。

在主作业中更改 build 的示例：

build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]

完整修改的主脚本：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '1')]
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '2')]
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '3')]
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '5')]
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '6')]
                    }
                }
            }
        }
    }
}

以及输出，显示每个实例确实已创建：

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
Starting building: FailRarely #149Starting building: FailRarely #148
Starting building: FailRarely #147

Starting building: FailRarely #150
Starting building: FailRarely #151
Starting building: FailRarely #152
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

问题已通过变通方法解决。如果有人可以在几天/几周内给我一个更好的答案，我会保持开放状态。

相关问题
最新问题