在子子目标中使用仅订单先决条件时,在第二次运行时生成-jN块

时间:2014-09-10 15:42:04

标签: parallel-processing makefile

我使用仅限订单的先决条件和并行模式的Makefile遇到了一个非常奇怪的问题:

  • 在顺序模式下运行时,Makefile第一次完成它的工作,下一次,它什么都不做,并按预期返回。
  • 以并行模式运行时(例如-j4),它第一次完成它的工作,但下次它只是挂起。该过程被阻止无所事事,甚至不会退出。但是它仍然是响应的,如果我使用Ctrl-C终止它,它会清除它可能已经创建并最终停止的中间文件。

在并行模式下,如果我make clean,它会再次运行而不会出现问题。同样,如果我只是touch 所有源文件,它的工作原理(再次,并行模式)。但是,如果我touch 某些源文件,它运行正常,直到完成它们,然后它就会挂起。

所以我创建了一个最小的工作示例,主文件夹只包含一个名为'txt'的文件夹,里面有一些* .txt文件(可以是任何文件)和Makefile

TXT_FILES = $(shell ls txt/*.txt)
A_FILES = $(patsubst txt/%.txt, a/%.txt, $(TXT_FILES))

all: a

a: $(A_FILES)

dir:
    @if [ ! -d "a" ]; then echo "create directory a" && mkdir -p a; fi

a/tmp_%: txt/%.txt | dir
    @echo "creating tmp file"
    @cp $< $@

a/%.txt: a/tmp_%
    @echo creating $@
    @cp $< $@

clean: clean-dir

clean-dir:
    rm -rf a

.PHONY: all a
.PRECIOUS: a/%.txt

它为此测试做的是:

  1. 列出txt文件(file1.txt,...,fileN.txt)
  2. 创建目录'a'(如果它不存在)
  3. 将'txt'中的每个文件复制到临时文件'a / tmp_fileN'
  4. 将每个临时文件从'a / tmp_fileN'复制到其最终目的地'a / fileN.txt'
  5. 为了避免每次调用make时都将'txt'中的所有文件重新复制到'a'中,我使用了a/tmp_%目标的'dir'先决条件的仅限订单的先决条件。

    请注意,原始的Makefile做了更复杂的事情,这使得必须使用临时文件,我在这个例子中保留它来重现有问题的行为(没有a/tmp_%目标,并且放置| dir到a/%.txt目标,没有问题发生。)

    这是带有--debug的日志(我翻译自法语,所以这些术语可能是近似的):

    Reading makefiles...
    Update targets....
     File « all » doesn't exist.
       File « a » doesn't exist.
            File « dir » doesn't exist.
           Must rebuild target « dir ».
     File « all » doesn't exist.
       File « a » doesn't exist.
     File « all » doesn't exist.
       File « a » doesn't exist.
     File « all » doesn't exist.
       File « a » doesn't exist.
     File « all » doesn't exist.
       File « a » doesn't exist.
     ...
    

    并且它一遍又一遍地遍历这两个消息。但如果这是某种循环依赖,我就看不到它(除了make的第一次运行不起作用,是吗?)。

    这在两个不同版本的Linux(CentOS 6.3和Ubuntu 12.04)的两个不同硬件上100%可重现,两者都使用GNU Make 3.81

    更新:正如Etan(以及相关评论)的回答所指出的那样,问题实际上是双重的:

    1. 即使无事可做,也要做一些目标。这可以通过适当重命名目标和目录来解决
    2. 如果没有第1点的修复,当上面的示例中没有任何操作时,以并行模式挂起。这可能是make 3.81中的某种bug或限制,因为3.82(或4.0)的更新解决了它(请注意,如果应用了针对第1点的解决方案,则问题也不会发生,因此无需更新)

1 个答案:

答案 0 :(得分:1)

让我们从线索开始,让你觉得你的makefile中的某些内容不太正确并且按照我们的方式工作,对吗?

添加:

donothing: static
        touch '$@'

到makefile(我们将在一分钟内使用它。)

这是我们的起始目录结构:

$ ls -R
.:
Makefile  static  txt

./txt:
1.txt

让我们一次运行make

$ make
create directory a
creating tmp file
creating a/1.txt
rm a/tmp_1

现在运行make donothing一次:

$ make donothing
touch 'donothing'

现在再次'make':

$ make

现在再次'做好事':

$ make donothing
make: `donothing' is up to date

看到那里的输出差异?这是线索。 make认为即使实际上没有做任何工作(我们可以看到),您的默认目标仍有一些工作要做。

我们当前的目录结构(仅为完整性):

$ ls -R
.:
Makefile  a  donothing  static  txt

./a:
1.txt

./txt:
1.txt

那么make运行make donothing时会做什么?

$ make -rRd donothing
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-redhat-linux-gnu
Reading makefiles...
Reading makefile `Makefile'...
Updating makefiles....
 Considering target file `Makefile'.
  Looking for an implicit rule for `Makefile'.
  No implicit rule found for `Makefile'.
  Finished prerequisites of target file `Makefile'.
 No need to remake target `Makefile'.
Updating goal targets....
Considering target file `donothing'.
  Considering target file `static'.
   Looking for an implicit rule for `static'.
   No implicit rule found for `static'.
   Finished prerequisites of target file `static'.
  No need to remake target `static'.
 Finished prerequisites of target file `donothing'.
 Prerequisite `static' is older than target `donothing'.
No need to remake target `donothing'.
make: `donothing' is up to date.

运行默认目标时make会做什么?

$ make
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-redhat-linux-gnu
Reading makefiles...
Reading makefile `Makefile'...
Updating makefiles....
 Considering target file `Makefile'.
  Looking for an implicit rule for `Makefile'.
  No implicit rule found for `Makefile'.
  Finished prerequisites of target file `Makefile'.
 No need to remake target `Makefile'.
Updating goal targets....
Considering target file `all'.
 File `all' does not exist.
  Considering target file `a'.
   File `a' does not exist.
    Considering target file `a/1.txt'.
     Looking for an implicit rule for `a/1.txt'.
     Trying pattern rule with stem `1'.
     Trying implicit prerequisite `a/tmp_1'.
     Trying pattern rule with stem `1'.
     Trying implicit prerequisite `a/tmp_1'.
     Looking for a rule with intermediate file `a/tmp_1'.
      Avoiding implicit rule recursion.
      Trying pattern rule with stem `1'.
      Trying implicit prerequisite `txt/1.txt'.
      Trying rule prerequisite `dir'.
     Found an implicit rule for `a/1.txt'.
       Considering target file `txt/1.txt'.
        Looking for an implicit rule for `txt/1.txt'.
        No implicit rule found for `txt/1.txt'.
        Finished prerequisites of target file `txt/1.txt'.
       No need to remake target `txt/1.txt'.
       Considering target file `dir'.
        File `dir' does not exist.
        Finished prerequisites of target file `dir'.
       Must remake target `dir'.
Putting child 0x0af5df90 (dir) PID 15558 on the chain.
Live child 0x0af5df90 (dir) PID 15558
Reaping winning child 0x0af5df90 PID 15558
Removing child 0x0af5df90 PID 15558 from chain.
       Successfully remade target file `dir'.
     Finished prerequisites of target file `a/1.txt'.
     Prerequisite `a/tmp_1' of target `a/1.txt' does not exist.
    No need to remake target `a/1.txt'.
   Finished prerequisites of target file `a'.
  Must remake target `a'.
  Successfully remade target file `a'.
 Finished prerequisites of target file `all'.
Must remake target `all'.
Successfully remade target file `all'.

啊哈哈。

make认为它需要重建dir目标(即使它没有做任何事情)但是没有一个文件告诉make否则所以它必须尝试。所以这就是为什么它不打印“无所事事”,因为有事可做。

因此,让我们向make提供所需的信息,以便每次都需要运行dir目标。最简单的方法是触摸具有该名称的文件。 (请注意,这实际上不是正确,因为使用此make 永远不会再次运行dir目标,因为它没有任何先决条件,但我们'我会稍稍谈谈。)

变化:

dir:
        @if [ ! -d "a" ]; then echo "create directory a" && mkdir -p a; fi

dir:
        @if [ ! -d "a" ]; then echo "create directory a" && mkdir -p a; fi
        touch dir

同时将dir添加到clean规则:

clean: clean-dir
        rm -f dir

现在让我们再次运行make

$ make
touch dir

好。这就是我们想要的文件。

让我们看看当我们再次运行make时会发生什么:

$ make
make: Nothing to be done for `all'.

好。这就是我们想要的。

我们现在试试-j4吗?

$ make clean
$ make -j4
create directory a
touch dir
creating tmp file
creating a/1.txt
rm a/tmp_1
$ make -j4
make: Nothing to be done for `all'.

大。这看起来很有效。

我们应该检查-d输出以确定:

$ make -j4 -rRd
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-redhat-linux-gnu
Reading makefiles...
Reading makefile `Makefile'...
Updating makefiles....
 Considering target file `Makefile'.
  Looking for an implicit rule for `Makefile'.
  No implicit rule found for `Makefile'.
  Finished prerequisites of target file `Makefile'.
 No need to remake target `Makefile'.
Updating goal targets....
Considering target file `all'.
 File `all' does not exist.
  Considering target file `a'.
   File `a' does not exist.
    Considering target file `a/1.txt'.
     Looking for an implicit rule for `a/1.txt'.
     Trying pattern rule with stem `1'.
     Trying implicit prerequisite `a/tmp_1'.
     Trying pattern rule with stem `1'.
     Trying implicit prerequisite `a/tmp_1'.
     Looking for a rule with intermediate file `a/tmp_1'.
      Avoiding implicit rule recursion.
      Trying pattern rule with stem `1'.
      Trying implicit prerequisite `txt/1.txt'.
      Trying rule prerequisite `dir'.
     Found an implicit rule for `a/1.txt'.
       Considering target file `txt/1.txt'.
        Looking for an implicit rule for `txt/1.txt'.
        No implicit rule found for `txt/1.txt'.
        Finished prerequisites of target file `txt/1.txt'.
       No need to remake target `txt/1.txt'.
       Considering target file `dir'.
        Finished prerequisites of target file `dir'.
       No need to remake target `dir'.
     Finished prerequisites of target file `a/1.txt'.
     Prerequisite `a/tmp_1' of target `a/1.txt' does not exist.
    No need to remake target `a/1.txt'.
   Finished prerequisites of target file `a'.
  Must remake target `a'.
  Successfully remade target file `a'.
 Finished prerequisites of target file `all'.
Must remake target `all'.
Successfully remade target file `all'.
make: Nothing to be done for `all'.

所以这似乎是问题所在。确定。

现在,还记得我说过这不是一个好的解决方案吗?原因如下:

$ rm -rf a
$ .:
Makefile  dir  donothing  static  txt

./txt:
1.txt
$ make
creating tmp file
cp: cannot create regular file `a/tmp_1': No such file or directory
$ ls
Makefile  dir  donothing  static  txt

糟糕。 a目录未创建,因为make没有意识到需要再次运行dir目标。

make -rRd输出的相关摘录,因为这有点长:

$ make -rRd
....
       Considering target file `dir'.
        Finished prerequisites of target file `dir'.
       No need to remake target `dir'.
....

那我们该如何解决这个问题呢?好吧,我们可以停止使用这些间接规则,让它们了解真正的先决条件信息。

所以我们首先放弃只会混淆事情的.PHONY a目标。

all: $(A_FILES)

.PHONY: all

并删除a: $(A_FILES)行。

但这还不够,因为我们仍然有dir先决条件令人困惑并且表现得很糟糕。

但是,由于我们不再拥有与我们的目录名匹配的目标,并且我们知道如何使用仅订单的先决条件来获取目录创建行为,我们可以使用它。

dir:替换为a:,将| dir替换为| a(并从rm -f dir规则中删除clean,因为我们不需要它我们最终得到了:

TXT_FILES = $(shell ls txt/*.txt)
A_FILES = $(patsubst txt/%.txt, a/%.txt, $(TXT_FILES))

all: $(A_FILES)

a:
        @if [ ! -d "a" ]; then echo "create directory a" && mkdir -p a; fi

a/tmp_%: txt/%.txt | a
        @echo "creating tmp file"
        @cp $< $@

a/%.txt: a/tmp_%
        @echo creating $@
        @cp $< $@

clean: clean-dir

clean-dir:
        rm -rf a

.PHONY: all
.PRECIOUS: a/%.txt

donothing: static
        touch '$@'

这有用吗?

$ ls -R
.:
Makefile  donothing  static  txt

./txt:
1.txt
$ make
create directory a
creating tmp file
creating a/1.txt
rm a/tmp_1
$ make
make: Nothing to be done for `all'.
$ make -j4
make: Nothing to be done for `all'.
$ make clean
rm -rf a
$ make -j4
create directory a
creating tmp file
creating a/1.txt
rm a/tmp_1
$ make -j4
make: Nothing to be done for `all'.

看起来像。

好的衡量标准:

$ make -j4 -rRd
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-redhat-linux-gnu
Reading makefiles...
Reading makefile `Makefile'...
Updating makefiles....
 Considering target file `Makefile'.
  Looking for an implicit rule for `Makefile'.
  No implicit rule found for `Makefile'.
  Finished prerequisites of target file `Makefile'.
 No need to remake target `Makefile'.
Updating goal targets....
Considering target file `all'.
 File `all' does not exist.
  Considering target file `a/1.txt'.
   Looking for an implicit rule for `a/1.txt'.
   Trying pattern rule with stem `1'.
   Trying implicit prerequisite `a/tmp_1'.
   Trying pattern rule with stem `1'.
   Trying implicit prerequisite `a/tmp_1'.
   Looking for a rule with intermediate file `a/tmp_1'.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `1'.
    Trying implicit prerequisite `txt/1.txt'.
    Trying rule prerequisite `a'.
   Found an implicit rule for `a/1.txt'.
     Considering target file `txt/1.txt'.
      Looking for an implicit rule for `txt/1.txt'.
      No implicit rule found for `txt/1.txt'.
      Finished prerequisites of target file `txt/1.txt'.
     No need to remake target `txt/1.txt'.
     Considering target file `a'.
      Finished prerequisites of target file `a'.
     No need to remake target `a'.
   Finished prerequisites of target file `a/1.txt'.
   Prerequisite `a/tmp_1' of target `a/1.txt' does not exist.
  No need to remake target `a/1.txt'.
 Finished prerequisites of target file `all'.
Must remake target `all'.
Successfully remade target file `all'.
make: Nothing to be done for `all'.

呼!我希望你能遵循这一切。我们在那里报道了很多。

最后一个分手评论:你不需要向ls发送shell来获取该文件列表。您可以改为使用$(wildcard txt/*.txt)