Problem :
I’m having problems with grep filtering of make output. In particular,
make target 2>&1 | grep -E --color=never "^make.*"
works as expected, but the following will print no output to the console:
make target 2>&1 | grep -E --color=never "^make.*" | cat
Am I missing something obvious? Why does the first command output but not the second? Is it to do with some sort of IO buffering? Or am I just being stupid?
[EDIT]: cat is just a minimum test case placeholder for the actual command I want to use.
[EDIT]: This does not seem to be a problem with grep, as replacing it with ack leads to the same behaviour.
[EDIT]: Script that cat is a placeholder for:
#!/bin/bash
cat -
| grep -E --color=never "^.*warning:.*|^.*error:.*|^make.*[Ee]rror.*|^make.*"
| hilite.sh -r "^.*warning:.*" -f yellow -B
| hilite.sh -r "^.*error:.*" -f red -B
| hilite.sh -r "^make.*[Ee]rror.*" -f red -B
| hilite.sh -r "^make.*" -f magenta
[EDIT]: I think it’s a buffering/IO issue. I’m leaving the build running over the w/e and will see if it eventually gets the output where it needs to!
Solution :
Likely grep is detecting that it’s not writing to a TTY, so it buffers more output, rather than operating in its usual line buffered mode (i.e. you see each line as it’s found). This behavior is more efficient as it results in fewer write() system calls, but if you’re a user who’s waiting for pipelined output, then it may be a bit misleading.
If you’re using GNU grep (and based on your linux tag I’m assuming you are), check out the --line-buffered option, which will force grep to work in the more familiar line buffered mode. This could technically decrease the performance of grep (as noted in the man page), but since you’re looking at the output of a live build, I doubt it will make any difference in that regard.
Not sure what your command is supposed to do, cat any files that have make.* in their file name?
Try changing the cat for xargs cat?
You might try experimenting with unbuffer, e.g.,
make target 2>&1 | unbuffer -p grep -E --color=never "^make.*" | cat