parallel processing - make -jN blocks on second run when using an order-only prerequisite in sub-sub-target -

i've been having strange problem makefile using order-only prerequisite , parallel mode:

• when running in sequential mode, first time makefile job, , next times, nothing , returns, expected.
• when running in parallel mode (-j4 instance), first time job, next times hangs. process blocked doing nothing, doesn't exit. still responsive in sense if terminate using ctrl-c, cleans intermediate files may have created , stops.

in parallel mode, if make clean, runs again without problem. likewise, if touch all source files, works (again, in parallel mode). if touch some of source files, runs fine until it's done them, hangs.

so created minimal working example, main folder contains folder named 'txt' *.txt file in (could file) , makefile

txt_files = $(shell ls txt/*.txt) a_files = $(patsubst txt/%.txt, a/%.txt, $(txt_files))  all:  a: $(a_files)  dir:     @if [ ! -d "a" ]; echo "create directory a" && mkdir -p a; fi  a/tmp_%: txt/%.txt | dir     @echo "creating tmp file"     @cp $< $@  a/%.txt: a/tmp_%     @echo creating $@     @cp $< $@  clean: clean-dir  clean-dir:     rm -rf  .phony: .precious: a/%.txt 

what test is:

1. list files in txt (file1.txt, ..., filen.txt)
2. create directory 'a' if doesn't exist
3. copy each file 'txt' in temporary file 'a/tmp_filen'
4. copy each temporary file 'a/tmp_filen' final destination 'a/filen.txt'

in order avoid recopying files 'txt' 'a' every time call make, use order-only prerequisite 'dir' prerequisite of a/tmp_% target.

please note original makefile more complicated things makes use of temporary file necessary, kept in example reproduce problematic behavior (without a/tmp_% target, , putting | dir a/%.txt target, no problem occurs).

here log --debug (i translated french, terms might approximate):

reading makefiles... update targets....  file « » doesn't exist.    file « » doesn't exist.         file « dir » doesn't exist.        must rebuild target « dir ».  file « » doesn't exist.    file « » doesn't exist.  file « » doesn't exist.    file « » doesn't exist.  file « » doesn't exist.    file « » doesn't exist.  file « » doesn't exist.    file « » doesn't exist.  ... 

and loops on these 2 messages on , on again. if kind of circular dependency, fail see (besides first run of make wouldn't work, ?).

this 100% reproducible on 2 different hardwares 2 different versions of linux (centos 6.3 , ubuntu 12.04), both using gnu make 3.81

update: pointed out in answer etan (and related comments), problem twofold:

1. even when there nothing do, make runs target. can solved proper renaming of targets , directories
2. without fix point 1, make hangs in parallel mode when there nothing in example above. sort of bug or limitation in make 3.81 since update 3.82 (or 4.0) solves (note if solution point 1 applied, problem doesn't occur either, no need update)

let start clue make giving in makefile isn't quite right , work our way up, shall we?

add:

donothing: static         touch '$@' 

to makefile (we'll use in minute).

here's our starting directory structure:

$ ls -r .: makefile  static  txt  ./txt: 1.txt 

let run make once:

$ make create directory creating tmp file creating a/1.txt rm a/tmp_1 

now run make donothing once:

$ make donothing touch 'donothing' 

now 'make' again:

$ make 

now 'make donothing' again:

$ make donothing make: `donothing' date 

see difference in output there? that's clue. make believes there default target when doesn't work (that can see).

our current directory structure (just completeness):

$ ls -r .: makefile   donothing  static  txt  ./a: 1.txt  ./txt: 1.txt 

so make when run make donothing?

$ make -rrd donothing gnu make 3.81 copyright (c) 2006  free software foundation, inc. free software; see source copying conditions. there no warranty; not merchantability or fitness particular purpose.  program built x86_64-redhat-linux-gnu reading makefiles... reading makefile `makefile'... updating makefiles....  considering target file `makefile'.   looking implicit rule `makefile'.   no implicit rule found `makefile'.   finished prerequisites of target file `makefile'.  no need remake target `makefile'. updating goal targets.... considering target file `donothing'.   considering target file `static'.    looking implicit rule `static'.    no implicit rule found `static'.    finished prerequisites of target file `static'.   no need remake target `static'.  finished prerequisites of target file `donothing'.  prerequisite `static' older target `donothing'. no need remake target `donothing'. make: `donothing' date. 

and make when default target run?

$ make gnu make 3.81 copyright (c) 2006  free software foundation, inc. free software; see source copying conditions. there no warranty; not merchantability or fitness particular purpose.  program built x86_64-redhat-linux-gnu reading makefiles... reading makefile `makefile'... updating makefiles....  considering target file `makefile'.   looking implicit rule `makefile'.   no implicit rule found `makefile'.   finished prerequisites of target file `makefile'.  no need remake target `makefile'. updating goal targets.... considering target file `all'.  file `all' not exist.   considering target file `a'.    file `a' not exist.     considering target file `a/1.txt'.      looking implicit rule `a/1.txt'.      trying pattern rule stem `1'.      trying implicit prerequisite `a/tmp_1'.      trying pattern rule stem `1'.      trying implicit prerequisite `a/tmp_1'.      looking rule intermediate file `a/tmp_1'.       avoiding implicit rule recursion.       trying pattern rule stem `1'.       trying implicit prerequisite `txt/1.txt'.       trying rule prerequisite `dir'.      found implicit rule `a/1.txt'.        considering target file `txt/1.txt'.         looking implicit rule `txt/1.txt'.         no implicit rule found `txt/1.txt'.         finished prerequisites of target file `txt/1.txt'.        no need remake target `txt/1.txt'.        considering target file `dir'.         file `dir' not exist.         finished prerequisites of target file `dir'.        must remake target `dir'. putting child 0x0af5df90 (dir) pid 15558 on chain. live child 0x0af5df90 (dir) pid 15558 reaping winning child 0x0af5df90 pid 15558 removing child 0x0af5df90 pid 15558 chain.        remade target file `dir'.      finished prerequisites of target file `a/1.txt'.      prerequisite `a/tmp_1' of target `a/1.txt' not exist.     no need remake target `a/1.txt'.    finished prerequisites of target file `a'.   must remake target `a'.   remade target file `a'.  finished prerequisites of target file `all'. must remake target `all'. remade target file `all'. 

ah hah.

make believes needs rebuild dir target (even though doesn't anything) there isn't file there tell make otherwise has try. that's why doesn't print "nothing do" because there do.

so let give make information needs know doesn't need run dir target every time. simplest way touch file name. (note isn't correct since make never run dir target again has no prerequisites we'll in bit.)

change:

dir:         @if [ ! -d "a" ]; echo "create directory a" && mkdir -p a; fi 

to

dir:         @if [ ! -d "a" ]; echo "create directory a" && mkdir -p a; fi         touch dir 

also add dir clean rule:

clean: clean-dir         rm -f dir 

now let run make again:

$ make touch dir 

good. made file wanted.

let's see happens when run make again:

$ make make: nothing done `all'. 

good. that's wanted.

shall try -j4 now?

$ make clean $ make -j4 create directory touch dir creating tmp file creating a/1.txt rm a/tmp_1 $ make -j4 make: nothing done `all'. 

great. looks worked.

we should check -d output sure though:

$ make -j4 -rrd gnu make 3.81 copyright (c) 2006  free software foundation, inc. free software; see source copying conditions. there no warranty; not merchantability or fitness particular purpose.  program built x86_64-redhat-linux-gnu reading makefiles... reading makefile `makefile'... updating makefiles....  considering target file `makefile'.   looking implicit rule `makefile'.   no implicit rule found `makefile'.   finished prerequisites of target file `makefile'.  no need remake target `makefile'. updating goal targets.... considering target file `all'.  file `all' not exist.   considering target file `a'.    file `a' not exist.     considering target file `a/1.txt'.      looking implicit rule `a/1.txt'.      trying pattern rule stem `1'.      trying implicit prerequisite `a/tmp_1'.      trying pattern rule stem `1'.      trying implicit prerequisite `a/tmp_1'.      looking rule intermediate file `a/tmp_1'.       avoiding implicit rule recursion.       trying pattern rule stem `1'.       trying implicit prerequisite `txt/1.txt'.       trying rule prerequisite `dir'.      found implicit rule `a/1.txt'.        considering target file `txt/1.txt'.         looking implicit rule `txt/1.txt'.         no implicit rule found `txt/1.txt'.         finished prerequisites of target file `txt/1.txt'.        no need remake target `txt/1.txt'.        considering target file `dir'.         finished prerequisites of target file `dir'.        no need remake target `dir'.      finished prerequisites of target file `a/1.txt'.      prerequisite `a/tmp_1' of target `a/1.txt' not exist.     no need remake target `a/1.txt'.    finished prerequisites of target file `a'.   must remake target `a'.   remade target file `a'.  finished prerequisites of target file `all'. must remake target `all'. remade target file `all'. make: nothing done `all'. 

so seems have been problem. ok.

now, remember how said wasn't solution? here's why:

$ rm -rf $ .: makefile  dir  donothing  static  txt  ./txt: 1.txt $ make creating tmp file cp: cannot create regular file `a/tmp_1': no such file or directory $ ls makefile  dir  donothing  static  txt 

oops. a directory didn't created because make didn't realize needed run dir target again.

relevant snippet of make -rrd output since getting bit long:

$ make -rrd ....        considering target file `dir'.         finished prerequisites of target file `dir'.        no need remake target `dir'. .... 

so how should fix this? well, can stop using such indirect rules , let make know real prerequisite information wants it.

so start dropping .phony a target confuses things.

all: $(a_files)  .phony: 

and remove a: $(a_files) line.

but that's not enough since still have dir prerequisite confusing things , behaving badly.

but since no longer have target matches our directory name , know how use order-only prerequisites directory creation behaviour can use that.

replace dir: a: , | dir | a (and drop rm -f dir clean rule since don't need anymore) , end with:

txt_files = $(shell ls txt/*.txt) a_files = $(patsubst txt/%.txt, a/%.txt, $(txt_files))  all: $(a_files)  a:         @if [ ! -d "a" ]; echo "create directory a" && mkdir -p a; fi  a/tmp_%: txt/%.txt |         @echo "creating tmp file"         @cp $< $@  a/%.txt: a/tmp_%         @echo creating $@         @cp $< $@  clean: clean-dir  clean-dir:         rm -rf  .phony: .precious: a/%.txt  donothing: static         touch '$@' 

does work?

$ ls -r .: makefile  donothing  static  txt  ./txt: 1.txt $ make create directory creating tmp file creating a/1.txt rm a/tmp_1 $ make make: nothing done `all'. $ make -j4 make: nothing done `all'. $ make clean rm -rf $ make -j4 create directory creating tmp file creating a/1.txt rm a/tmp_1 $ make -j4 make: nothing done `all'. 

looks it.

and measure:

$ make -j4 -rrd gnu make 3.81 copyright (c) 2006  free software foundation, inc. free software; see source copying conditions. there no warranty; not merchantability or fitness particular purpose.  program built x86_64-redhat-linux-gnu reading makefiles... reading makefile `makefile'... updating makefiles....  considering target file `makefile'.   looking implicit rule `makefile'.   no implicit rule found `makefile'.   finished prerequisites of target file `makefile'.  no need remake target `makefile'. updating goal targets.... considering target file `all'.  file `all' not exist.   considering target file `a/1.txt'.    looking implicit rule `a/1.txt'.    trying pattern rule stem `1'.    trying implicit prerequisite `a/tmp_1'.    trying pattern rule stem `1'.    trying implicit prerequisite `a/tmp_1'.    looking rule intermediate file `a/tmp_1'.     avoiding implicit rule recursion.     trying pattern rule stem `1'.     trying implicit prerequisite `txt/1.txt'.     trying rule prerequisite `a'.    found implicit rule `a/1.txt'.      considering target file `txt/1.txt'.       looking implicit rule `txt/1.txt'.       no implicit rule found `txt/1.txt'.       finished prerequisites of target file `txt/1.txt'.      no need remake target `txt/1.txt'.      considering target file `a'.       finished prerequisites of target file `a'.      no need remake target `a'.    finished prerequisites of target file `a/1.txt'.    prerequisite `a/tmp_1' of target `a/1.txt' not exist.   no need remake target `a/1.txt'.  finished prerequisites of target file `all'. must remake target `all'. remade target file `all'. make: nothing done `all'. 

whew! hope followed of that. covered bit of ground there.

one last parting comment: don't need shell out ls list of files. can use $(wildcard txt/*.txt) instead.