Question

主要修改：通过注释解决了几个问题，并写了一个最小的可复制示例来帮助我的助手，我将问题缩小到本地执行和使用DRMAA之间的差异。

Here是一种可重现的最小管道，不需要下载任何外部文件，可以直接在git存储库中执行或克隆以下代码：

git clone git@github.com:kevinrue/snakemake-issue-all.git

使用DRMAA运行管道时，出现以下错误：

Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 100
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3
InputFunctionException in line 22 of /ifs/research-groups/sims/kevin/snakemake-issue-all/workflow/Snakefile:
SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=A

但是，如果我在本地运行管道（--cores 1），则可以正常运行：

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3

[Sat Jun 13 08:49:46 2020]
rule cat:
    input: data/A1, data/A2
    output: results/A/cat
    jobid: 1
    wildcards: sample=A

[Sat Jun 13 08:49:46 2020]
Finished job 1.
1 of 3 steps (33%) done

[Sat Jun 13 08:49:46 2020]
rule cat:
    input: data/B1, data/B2
    output: results/B/cat
    jobid: 2
    wildcards: sample=B

[Sat Jun 13 08:49:46 2020]
Finished job 2.
2 of 3 steps (67%) done

[Sat Jun 13 08:49:46 2020]
localrule all:
    input: results/A/cat, results/B/cat
    jobid: 0

[Sat Jun 13 08:49:46 2020]
Finished job 0.
3 of 3 steps (100%) done
Complete log: /ifs/research-groups/sims/kevin/snakemake-issue-all/.snakemake/log/2020-06-13T084945.632545.snakemake.log

我的DRMAA个人资料如下：

jobs: 100
default-resources: 'mem_free=4G'
drmaa: "-V -notify -p -10 -l mem_free={resources.mem_free} -pe dedicated {threads} -v MKL_NUM_THREADS={threads} -v OPENBLAS_NUM_THREADS={threads} -v OMP_NUM_THREADS={threads} -R y -q all.q"
drmaa-log-dir: /ifs/scratch/kevin
use-conda: true
conda-prefix: /ifs/home/kevin/devel/snakemake/envs
printshellcmds: true
reason: true

简而言之，Snakefile看起来像这样

# The main entry point of your workflow.
# After configuring, running snakemake -n in a clone of this repository should successfully execute a dry-run of the workflow.


report: "report/workflow.rst"

# Allow users to fix the underlying OS via singularity.
singularity: "docker://continuumio/miniconda3"

include: "rules/common.smk"
include: "rules/other.smk"

rule all:
    input:
        # The first rule should define the default target files
        # Subsequent target rules can be specified below. They should start with all_*.
        expand("results/{sample}/cat", sample=samples['sample'])


rule cat:
    input:
        file1="data/{sample}1",
        file2="data/{sample}2"
    output:
        "results/{sample}/cat"
    shell:
        "cat {input.file1} {input.file2} > {output}"

跑步snakemake -np给了我我期望的东西：

$ snakemake -np
          sample  condition
sample_id                  
A              A  untreated
B              B    treated
Building DAG of jobs...
Job counts:
    count   jobs
    1   all
    2   cat
    3

[Sat Jun 13 08:51:19 2020]
rule cat:
    input: data/B1, data/B2
    output: results/B/cat
    jobid: 2
    wildcards: sample=B

cat data/B1 data/B2 > results/B/cat

[Sat Jun 13 08:51:19 2020]
rule cat:
    input: data/A1, data/A2
    output: results/A/cat
    jobid: 1
    wildcards: sample=A

cat data/A1 data/A2 > results/A/cat

[Sat Jun 13 08:51:19 2020]
localrule all:
    input: results/A/cat, results/B/cat
    jobid: 0

Job counts:
    count   jobs
    1   all
    2   cat
    3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

我不确定如何进一步调试它。很高兴根据需要提供更多信息。

注意：我使用的是蛇形版本5.19.2

谢谢！

编辑使用--verbose选项，Snakemake似乎在我的'drmaa'个人资料中定义的'default-resources: 'mem_free=4G'和/或drmaa: "-l mem_free={resources.mem_free}上跳闸（见上文）。

$ snakemake --profile drmaa --verbose
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 100
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 100}
Ready jobs (2):
    cat
    cat
Full Traceback (most recent call last):
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 941, in apply
    res, _ = self.apply_input_function(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 684, in apply_input_function
    raise e
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 678, in apply_input_function
    value = func(Wildcards(fromdict=wildcards), **_aux_params)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/resources.py", line 10, in callable
    value = eval(
  File "<string>", line 1
    4G
     ^
SyntaxError: unexpected EOF while parsing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/__init__.py", line 626, in snakemake
    success = workflow.execute(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/workflow.py", line 951, in execute
    success = scheduler.schedule()
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 394, in schedule
    run = self.job_selector(needrun)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 540, in job_selector
    a = list(map(self.job_weight, jobs))  # resource usage of jobs
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 613, in job_weight
    res = job.resources
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/jobs.py", line 267, in resources
    self._resources = self.rule.expand_resources(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 977, in expand_resources
    resources[name] = apply(name, res, threads=threads)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 960, in apply
    raise InputFunctionException(e, rule=self, wildcards=wildcards)
snakemake.exceptions.InputFunctionException: SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=B

InputFunctionException in line 20 of /ifs/research-groups/sims/kevin/snakemake-issue-all/workflow/Snakefile:
SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=B
unlocking
removing lock
removing lock
removed all locks

Answer 1

感谢@JohannesKöster，我意识到我的个人资料设置有误。

--default-resources [NAME=INT [NAME=INT ...]]表示当我提供字符串（即mem_free=4G）时，仅支持整数值，天真地希望这些整数值也受支持。

我已经更新了个人资料中的以下设置，并成功运行了snakemake --cores 1和snakemake --profile drmaa。

default-resources: 'mem_free=4'
drmaa: "-V -notify -p -10 -l mem_free={resources.mem_free}G -pe dedicated {threads} -v MKL_NUM_THREADS={threads} -v OPENBLAS_NUM_THREADS={threads} -v OMP_NUM_THREADS={threads} -R y -q all.q"

请注意将整数值4设置为默认资源，以及如何将G移至drmaa: ... -l mem_free=...G设置。

非常感谢大家的帮助！

InputFunctionException：解析时发生意外的EOF

1 个答案: