Bash snippets & functions : Make a variable readable/reachable from outside a pipe + loop (AKA process substitution)

Introduction

This post is an answer to an issue I often face when I am writing scripts : How can I set a flag (a variable) inside a loop (for / while etc …) to which I am sending data using a pipe and get its value (of the flag / variable) outside this loop ?

The answer is quite simple actually, but given the number of keywords needed for searching this on the web it is somehow difficult to find a quick and clear answer, unless you know that you need to use a : Process substitution.

 

More “bash snippets & functions” posts

 

 

1) Description

As explained in the excellent Advanced Bash-Scripting Guide : Process substitution feeds the output of a process (or processes) into the stdin of another process.

This make it possible, for example, to feed a while loop with the output of as many as pipelined commands as you need, and still be able to get the value of a variable which was set inside this while loop.

Let’s take the following code :

ls /home |
  sort |
    while read line ; do
      <some_command>
    done

echo "Value of \$line outside the loop : '$line'"

Here is how I understand it :

ls /home |               # This part fork a subshell, and send the output to the next command
sort |                   # here is another subshell (we could say sub-subshell !)
while read line ; do     # and finally we get ourselves inside the loop ...
done                     # let's go back to the first shell now

 

As you can see, we are, at the while loop level, in the second layer shell. Given that no process is capable of sending any data to its parent shell, the value of $line variable (which is set during the while loop execution) is definitely not reachable at the end of the loop (remember that at this loop end we have to come back to the top layer shell (the one in which we launched the script / ls command).

When using the process substitution, we are replacing the ls /home/ | sort process by the parent process (the script or function or current shell process), this allow us to get the $line value : no subshell were forked here.

You can use $BASH_SUBSHELL built-in variable (with bash 3+ only) to check and understand what is happening inside your script, e.g :

#!/bin/bash
echo "Subshell level OUTSIDE = $BASH_SUBSHELL"
ls /home |
  sort |
    while read line ; do

echo "Subshell level INSIDE = $BASH_SUBSHELL"

      echo "line is : $line"
      ligne=$line

    done

echo "Value of \$ligne outside the loop : '$ligne'"

sample output :

[pier@server tmp]$ ./test-no.bash
Subshell level OUTSIDE = 0
Subshell level INSIDE = 1
line is : pier
Subshell level INSIDE = 1
line is : test
Subshell level INSIDE = 1
line is : wide
Value of $ligne outside the loop : ''

I let you run the same test with the process substitution

Thank you bash developers ! Please refer to the Resources section at the bottom of this page for a comprehensive page about how process substitution works.

 

 

2) The code

In the following example I am trying to get the value of a variable which is set inside a while loop. To make everything more readable I just echo the value of the aforementioned variable.

2.1 Without the process substitution

#!/bin/bash

ls /home |
  sort |
    while read line ; do

      echo "line is : $line"

    done

echo "Value of \$line outside the loop : '$line'"

 2.1.1 Sample output

[pier@server tmp]$ ./ProcessSubst.bash
line is : pier
line is : test
line is : wide
Value of $line outside the loop : ''

We can see in the above example the UN-expected value of $line variable : empty !

 

2.2 With the process substitution

#!/bin/bash

while read line ; do

  echo "line is : $line"

done < <(ls /home |sort )

echo "Value of \$line outside the loop : '$line'"

 2.2.1 Sample output

[pier@server tmp]$ ./ProcessSubst.bash
line is : pier
line is : test
line is : wide
Value of $line outside the loop : 'wide'

We can see in the above example the expected value of $line variable : wide.

 

 

Conclusion

I hope this will help someone, feel free to ask anything in the comment section.

 

 

Resources

Leave a Reply

Your email address will not be published. Required fields are marked *

This site supports SyntaxHighlighter via WP SyntaxHighlighter. It can highlight your code.
How to highlight your code: Paste your code in the comment form, select it and then click the language link button below. This will wrap your code in a <pre> tag and format it when submitted.