snakemake-执行多项工作后，工作流失败

时间：2018-09-07 11:49:48

标签： python memory multiprocessing workflow snakemake

我已经准备了一个Snakemake管道（Snakemake v5.2.2，Python v3.6.5）来分析RRBS数据（full pipeline viewable on GitHub），并将其放入奇点（v2.5.2，由Ubuntu 18.04-{{3} }。我已经用5个缩短的FASTQ文件测试了此工作流程，并且工作良好。

当我将简短的测试FASTQ文件的数量增加到20（该实验的分析单位）时，问题就开始了。运行多个步骤后，管道失败，出现Error 127: command not found。此错误并非特定于任何文件或规则-有时在trim galore规则时崩溃，有时在bismark时崩溃。

经过一番调查，我设法找到了解决方案：为每个规则（例如resources: 32000）添加内存使用限制，并使用标志--resources mem_mb=100000运行Snakemake。可悲的是，当我使用原尺寸的文件开始实际分析时，返回了错误。我什至已经尝试过最绝望的事情：

  1. 删除蛇文件中的所有mv命令，并仅用一行shell命令保留每个规则

  2. 运行带有各种标志（--wait-for-files，--latency-wait 30 --nolock --force-use-threads --restart-times 2）的snakemake

  3. 将管道中使用的所有程序放入奇异路径

  4. 增加每个规则最多50GB RAM的资源限制

  5. 在一个工作流实例中运行较少数量的样本

在我的PC和工作中的服务器上使用Singularity容器时均出现错误，但没有出现。通过命令行单独运行时，失败的相同命令可以正常工作。我不使用Snakemake的群集模式。

我已经尝试并检查了所有我能想到的一切，并且我已经（愚蠢地）告诉我的主管分析正在运行（这是T__T）。我将非常感谢您的帮助。有任何想法怎么回事以及我该如何解决？

这是一个完整的示例蛇形输出，带有错误：

also viewable on GitHub

这是问题的症结所在：

[Thu Sep  6 22:05:40 2018]
rule Bismark:
    input: Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz, Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz
    output: Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam, Reports/Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt
    jobid: 5
    wildcards: b=B_Control-NM_86_S1
    threads: 10
    resources: mem_mb=32000

/Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/
/bin/bash: error_exit: command not found
/bin/bash: error_exit: command not found
Full Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 1313, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell)
  File "/Analysis/Snakefile", line 218, in __rule_Bismark
  File "/usr/local/lib/python3.6/dist-packages/snakemake/shell.py", line 133, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.

    [Thu Sep  6 22:05:41 2018]
    Error in rule Bismark:
        jobid: 5
        output: Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam, Reports/Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt

Full Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 1313, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell)
  File "/Analysis/Snakefile", line 218, in __rule_Bismark
  File "/usr/local/lib/python3.6/dist-packages/snakemake/shell.py", line 133, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 346, in _callback
    raise ex
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 1325, in run_wrapper
    show_traceback=True))
snakemake.exceptions.RuleException: CalledProcessError in line 119 of /Analysis/Snakefile:
Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.
  File "/Analysis/Snakefile", line 119, in __rule_Bismark

RuleException:
CalledProcessError in line 119 of /Analysis/Snakefile:
Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.
  File "/Analysis/Snakefile", line 119, in __rule_Bismark
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[Thu Sep  6 22:08:09 2018]
Finished job 6.
8 of 47 steps (17%) done
[Thu Sep  6 22:58:49 2018]
Finished job 42.
9 of 47 steps (19%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /Analysis/.snakemake/log/2018-09-06T191518.240813.snakemake.log
unlocking
removing lock
removing lock
removed all locks

编辑

我认为我已经找到了解决此问题的方法。不幸的是，我仍然不知道是什么引起了原始错误。不过，请不要在这个问题上浪费您的时间，除非其他人在下面报告了该问题...

0 个答案:

没有答案
相关问题
最新问题