3/2/2020 python - How to run multiple commands synchronously from one subprocess.Popen command?

- Stack Overflow

How to run multiple commands synchronously from one

subprocess.Popen command?
Asked 3 years, 5 months ago Active 4 months ago Viewed 30k times

Is it possible to execute an arbitrary number of commands in sequence using the same subprocess
command?
15 I need each command to wait for the previous one to complete before executing and I need them all to
be executed in the same session/shell. I also need this to work in Python 2.6, Python 3.5. I also need
the subprocess command to work in Linux, Windows and macOS (which is why I'm just using echo
commands as examples here).
6
See non-working code below:

import sys
import subprocess

cmds = ['echo start', 'echo mid', 'echo end']

p = subprocess.Popen(cmd=tuple([item for item in cmds]),

stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

for line in iter(p.stdout.readline, b''):

sys.stdout.flush()
print(">>> " + line.rstrip())

If this is not possible, which approach should I take in order to execute my commands in synchronous
sequence within the same session/shell?

python subprocess python-3.5 python-2.6

edited Sep 27 '16 at 10:19 asked Sep 27 '16 at 10:10

fredrik
6,164 7 47 87

Why not using three Popen in sequence? – Riccardo Petraglia Sep 27 '16 at 10:18

How about running cmd1, cmd2, cmd3 in separate execution of Popen? – AlokThakur Sep 27 '16 at 10:19

@RiccardoPetraglia I edited my question since it involves an arbitrary number of commands. – fredrik Sep 27
'16 at 10:19

@AlokThakur if I execute one Popen per cmd, they will be executed in their own sessions. I need them all to
be executed in the same session. I'm editing my question to include that info now. – fredrik Sep 27 '16 at
10:20

Possible duplicate of running multiple bash commands with subprocess – sdikby Aug 22 '18 at 7:55

5 Answers

https://stackoverflow.com/questions/39721924/how-to-run-multiple-commands-synchronously-from-one-subprocess-popen-command 1/7
3/2/2020 python - How to run multiple commands synchronously from one subprocess.Popen command? - Stack Overflow

If you want to execute many commands one after the other in the same session/shell, you must start a
shell and feed it with all the commands, one at a time followed by a new line, and close the pipe at the
8 end. It makes sense if some commands are not true processes but shell commands that could for
example change the shell environment.

Example using Python 2.7 under Windows:

encoding = 'latin1'
p = subprocess.Popen('cmd.exe', stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for cmd in cmds:
p.stdin.write(cmd + "\n")
p.stdin.close()
print p.stdout.read()

To have this code run under Linux, you would have to replace cmd.exe with /bin/bash and probably
change the encoding to utf8.

For Python 3, you would have to encode the commands and probably decode their output, and to use
parentheses with print.

Beware: this can only work for little output. If there was enough output to fill the pipe buffer before
closing the stdin pipe, this code would deadlock. A more robust way would be to have a second thread
to read the output of the commands to avoid that problem.

answered Sep 27 '16 at 14:37

Serge Ballesta
95k 9 73 154

This is similar to the answer posted by Serge Ballesta, but not quite. Use his for asynchronous
execution, where you don't care about the results. Use mine for synchronous processing and result
4 gathering. Like his answer, I'm showing the Windows solution here - run a bash process in Linux rather
than cmd in Windows.

from subprocess import Popen, PIPE

process = Popen( "cmd.exe", shell=False, universal_newlines=True,
stdin=PIPE, stdout=PIPE, stderr=PIPE )
out, err = process.communicate( commands )

USAGE DETAILS: The commands argument being passed here to the process.communicate method is
a newline delimited string. If, for example you just read a batch file contents into a string, you could
run it this way because it would already have the newlines.

https://stackoverflow.com/questions/39721924/how-to-run-multiple-commands-synchronously-from-one-subprocess-popen-command 2/7
3/2/2020 python - How to run multiple commands synchronously from one subprocess.Popen command? - Stack Overflow

Important: your string must end in a newline "\n" . If it does not, that final command will fail to execute.
Just like if you typed it into your command prompt but didn't hit enter at the end. You will however see
a mysterious More? line in the end of the stdout returned. (that's the cause if you encounter this).

process.communicate runs synchronously by definition, and returns the stdout and stderr messages (if
you directed them to subprocess.PIPE in your Popen constructor).

When you create a cmd.exe process in this way, and pass a string to it, the results will be exactly like if
you were to open a command prompt window and entered commands into. And I mean that quite
literally. If you test this, you will see that the stdout which is returned contains your commands. (It does
NOT matter if you include an @echo off like if executing a batch file).

Tips for those who care about "clean" stdout results:

@echo off will not suppress your commands from appearing in this returned string, but it does
remove extra newlines that find their way in there otherwise. (universal_newlines=True strips an
another set of those)
Including an @ symbol prefix to your commands allows them to still execute. In a "normal" batch
process that's the line-by-line way to "hide" your commands. In this context, it's a safe an easy
marker by which you can find stdout lines you want to remove. (if one were so inclined)
The cmd.exe "header" will appear in your output (which says the version of Windows etc.). Since
you probably want to start your set of commands with @echo off , to cut out the extra newlines,
that is also a great way to find where the header lines stopped and your commands/results began.

Finally, to address concerns about "large" output filling the pipes and causing you problems - first I think
you need a HUGE amount of data coming back for that to be an issue - more than most people will
encounter in their use cases. Second, if it really is a concern just open a file for writing and pass that file
handle (the reference to the file object) to stdout/err instead of PIPE . Then, do whatever you want with
the file you've created.

edited Oct 11 '19 at 21:05 answered Apr 29 '17 at 16:10

Cartucho BuvinJ
2,593 1 24 47 5,933 3 44 62

raise ValueError("Cannot send input after starting communication") do not work – hopieman Jun 28 '18 at 22:04

Please clarify what you are saying or asking. If you encountered this exception, and would like help, please
say what OS you are using, the version of Python, and ideally share your exact code. – BuvinJ Jun 29 '18 at
12:27

One possible solution, looks like its running in same shell:

3 subprocess.Popen('echo start;echo mid;echo end', shell=True)

Note - If you pass your command as a string then shell has to be True Note - This is working on linux
only, you may have to find something similar way out on windows.

Hope it will help.

https://stackoverflow.com/questions/39721924/how-to-run-multiple-commands-synchronously-from-one-subprocess-popen-command 3/7
3/2/2020 python - How to run multiple commands synchronously from one subprocess.Popen command? - Stack Overflow

From python doc -

On Unix with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the
command to execute through the shell. This means that the string must be formatted exactly as it
would be when typed at the shell prompt.

edited Sep 27 '16 at 10:39 answered Sep 27 '16 at 10:35

AlokThakur
2,625 1 13 26

Yes, using shell=True is not a preferable way – AlokThakur Sep 27 '16 at 10:40

Apparently, at least in python 2.7 this one works also on windows: >On Windows with shell=True, the
COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on
Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need
shell=True to run a batch file or console-based executable.< – Riccardo Petraglia Sep 27 '16 at 10:52

@AlokThakur will all the above 3 commands dump their output on STDOUT in serialized fashion or there is a
possibility of getting a jumbled output. – Krishna Oza May 8 '18 at 12:13

1 Although shell=True is not preferable on documentation, in some special cases (e.g. if you want to execute git
push from jupyter notebook with a non-default key), this turns out to be the only solution. So one big THANK
YOU, @AlokThakur. – Borislav Aymaliev Oct 10 '18 at 15:10

it is not preferable when there is a risk of injection through unsanitized data. For the given question it is best
possible solution – Serge Oct 18 '18 at 14:29

This one works in python 2.7 and should work also in windows. Probably some small refinement is
needed for python >3.
1 The produced output is (using date and sleep it is easy to see that the commands are executed in row):

>>>Die Sep 27 12:47:52 CEST 2016

>>>
>>>Die Sep 27 12:47:54 CEST 2016

As you see the commands are executed in a row.

import sys
import subprocess
import shlex

cmds = ['date', 'sleep 2', 'date']

cmds = [shlex.split(x) for x in cmds]

outputs =[]
for cmd in cmds:
outputs.append(subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT).communicate())

for line in outputs:

print ">>>" + line[0].strip()
https://stackoverflow.com/questions/39721924/how-to-run-multiple-commands-synchronously-from-one-subprocess-popen-command 4/7
3/2/2020 python - How to run multiple commands synchronously from one subprocess.Popen command? - Stack Overflow

This is what I obtain merging with @Marichyasana answer:

import sys
import os

def run_win_cmds(cmds):

@Marichyasana code (+/-)

def run_unix_cmds(cmds):

import subprocess
import shlex

cmds = [shlex.split(x) for x in cmds]

outputs =[]
for cmd in cmds:
outputs.append(subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT).communicate())

rc = ''
for line in outputs:
rc += line[0].strip()+'\n'

return rc

cmds = ['date', 'sleep 2', 'date']

if os.name == 'nt':
run_win_cmds(cmds)
elif os.name == 'posix':
run_unix_cmds(cmds)

Ask is this one do not fit your needs! ;)

edited Sep 27 '16 at 14:19 answered Sep 27 '16 at 10:49

Riccardo Petraglia
1,226 7 19

This looks like it could work! But on Windows, I'm getting Traceback (most recent call last): File
"seq_test.py", line 12, in <module> stdout=subprocess.PIPE,
stderr=subprocess.STDOUT).communicate()) File "c:\python27\lib\subprocess.py", line 710, in
__init__ errread, errwrite) File "c:\python27\lib\subprocess.py", line 958, in
_execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file
specified - any idea why? (I replaced the commands with echo commands as per my initial question so
those commands can run in Windows) – fredrik Sep 27 '16 at 11:50

Did you try your commands with a simple Popen/communicate on windows? It could be that cmds must be a
string (reading the last part of the subprocess documentation) – Riccardo Petraglia Sep 27 '16 at 12:23

1 This doesn't look it is executing the commands within the same process, which is a fundamental component
to this question. – BuvinJ Apr 27 '17 at 23:29

https://stackoverflow.com/questions/39721924/how-to-run-multiple-commands-synchronously-from-one-subprocess-popen-command 5/7
3/2/2020 python - How to run multiple commands synchronously from one subprocess.Popen command? - Stack Overflow

The question is very clear about this point. It's what the question is all about it's core. Both of your code
samples execute commands in separate sub processes, not the same one. Look at Serge Ballesta's answer,
where he pipes in multiple commands to the same shell session. The results of that would be radically
different, and answers the question. As an example, if you have a command which changes the directory, or
assigns a value to a variable, before then executing some other command the result of having a second
process is to throw away the execution of the prior command! – BuvinJ Apr 28 '17 at 14:22

At least your answer avoids the potential lock ups that Serge warns about in his answer. I'm going to dig into
the popen details at some point to figure that out. So this question still does not have a perfect answer yet... –
BuvinJ Apr 28 '17 at 15:12

Here is a function (and main to run it) that I use. I would say that you can use it for your problem. And it
is flexible.
1
# processJobsInAList.py
# 2016-09-27 7:00:00 AM Central Daylight Time

import win32process, win32event

def CreateMyProcess2(cmd):
''' create process width no window that runs a command with arguments
and returns the process handle'''
si = win32process.STARTUPINFO()
info = win32process.CreateProcess(
None, # AppName
cmd, # Command line
None, # Process Security
None, # Thread Security
0, # inherit Handles?
win32process.NORMAL_PRIORITY_CLASS,
None, # New environment
None, # Current directory
si) # startup info
# info is tuple (hProcess, hThread, processId, threadId)
return info[0]

if __name__ == '__main__' :
''' create/run a process for each list element in "cmds"
output may be out of order because processes run concurrently '''

cmds=["echo my","echo heart","echo belongs","echo to","echo daddy"]

handles = []
for i in range(len(cmds)):
cmd = 'cmd /c ' + cmds[i]
handle = CreateMyProcess2(cmd)
handles.append(handle)

rc = win32event.WaitForMultipleObjects( handles, 1, -1) # 1 wait for all, -1

wait infinite
print 'return code ',rc

output:
heart
my
belongs
to
daddy
return code 0
https://stackoverflow.com/questions/39721924/how-to-run-multiple-commands-synchronously-from-one-subprocess-popen-command 6/7
3/2/2020 python - How to run multiple commands synchronously from one subprocess.Popen command? - Stack Overflow

UPDATE: If you want to run the same process, which will serialize things for you:
1) Remove line: handles.append(handle)
2) Substitute the variable "handle" in place of the list "handles" on the "WaitFor" line
3) Substitute WaitForSingleObject in place of WaitForMultipleObjects

edited Sep 27 '16 at 15:01 answered Sep 27 '16 at 13:21

Marichyasana
2,512 1 14 19

https://stackoverflow.com/questions/39721924/how-to-run-multiple-commands-synchronously-from-one-subprocess-popen-command 7/7