`runners`¶

class invoke.runners.Runner(context)¶

Partially-abstract core command-running API.

This class is not usable by itself and must be subclassed, implementing a number of methods such as start, wait and returncode. For a subclass implementation example, see the source code for Local.

__init__(context)¶

Create a new runner with a handle on some Context.

Parameters:

context –

a Context instance, used to transmit default options and provide access to other contextualized information (e.g. a remote-oriented Runner might want a Context subclass holding info about hostnames and ports.)

Note

The Context given to Runner instances must contain default config values for the Runner class in question. At a minimum, this means values for each of the default Runner.run keyword arguments such as echo and warn.

Raises: exceptions.ValueError – if not all expected default values are found in context.

context = None¶: The Context given to the same-named argument of __init__.

program_finished = None¶

A threading.Event signaling program completion.

Typically set after wait returns. Some IO mechanisms rely on this to know when to exit an infinite read loop.

read_chunk_size = 1000¶: How many bytes (at maximum) to read per iteration of stream reads.

input_sleep = 0.01¶: How many seconds to sleep on each iteration of the stdin read loop and other otherwise-fast loops.

warned_about_pty_fallback = None¶: Whether pty fallback warning has been emitted.

watchers = None¶: A list of StreamWatcher instances for use by respond. Is filled in at runtime by run.

run(command, **kwargs)¶

Execute command, returning an instance of Result.

Note

All kwargs will default to the values found in this instance’s context attribute, specifically in its configuration’s run subtree (e.g. run.echo provides the default value for the echo keyword, etc). The base default values are described in the parameter list below.

Parameters:	command (str) – The shell command to execute. shell (str) – Which shell binary to use. Default: `/bin/bash`. warn (bool) – Whether to warn and continue, instead of raising `UnexpectedExit`, when the executed command exits with a nonzero status. Default: `False`. Note This setting has no effect on exceptions, which will still be raised, typically bundled in `ThreadException` objects if they were raised by the IO worker threads. Similarly, `WatcherError` exceptions raised by `StreamWatcher` instances will also ignore this setting, and will usually be bundled inside `Failure` objects (in order to preserve the execution context). hide – Allows the caller to disable `run`’s default behavior of copying the subprocess’ stdout and stderr to the controlling terminal. Specify `hide='out'` (or `'stdout'`) to hide only the stdout stream, `hide='err'` (or `'stderr'`) to hide only stderr, or `hide='both'` (or `True`) to hide both streams. The default value is `None`, meaning to print everything; `False` will also disable hiding. Note Stdout and stderr are always captured and stored in the `Result` object, regardless of `hide`’s value. Note `hide=True` will also override `echo=True` if both are given (either as kwargs or via config/CLI). pty (bool) – By default, `run` connects directly to the invoked process and reads its stdout/stderr streams. Some programs will buffer (or even behave) differently in this situation compared to using an actual terminal or pseudoterminal (pty). To use a pty instead of the default behavior, specify `pty=True`. Warning Due to their nature, ptys have a single output stream, so the ability to tell stdout apart from stderr is not possible when `pty=True`. As such, all output will appear on `out_stream` (see below) and be captured into the `stdout` result attribute. `err_stream` and `stderr` will always be empty when `pty=True`. fallback (bool) – Controls auto-fallback behavior re: problems offering a pty when `pty=True`. Whether this has any effect depends on the specific `Runner` subclass being invoked. Default: `True`. echo (bool) – Controls whether `run` prints the command string to local stdout prior to executing it. Default: `False`. Note `hide=True` will override `echo=True` if both are given. env (dict) – By default, subprocesses recieve a copy of Invoke’s own environment (i.e. `os.environ`). Supply a dict here to update that child environment. For example, `run('command', env={'PYTHONPATH': '/some/virtual/env/maybe'})` would modify the `PYTHONPATH` env var, with the rest of the child’s env looking identical to the parent. See also `replace_env` for changing ‘update’ to ‘replace’. replace_env (bool) – When `True`, causes the subprocess to receive the dictionary given to `env` as its entire shell environment, instead of updating a copy of `os.environ` (which is the default behavior). Default: `False`. encoding (str) – Override auto-detection of which encoding the subprocess is using for its stdout/stderr streams (which defaults to the return value of `default_encoding`). out_stream – A file-like stream object to which the subprocess’ standard output should be written. If `None` (the default), `sys.stdout` will be used. err_stream – Same as `out_stream`, except for standard error, and defaulting to `sys.stderr`. in_stream – A file-like stream object to used as the subprocess’ standard input. If `None` (the default), `sys.stdin` will be used. If `False`, will disable stdin mirroring entirely (though other functionality which writes to the subprocess’ stdin, such as autoresponding, will still function.) Disabling stdin mirroring can help when `sys.stdin` is a misbehaving non-stream object, such as under test harnesses or headless command runners. watchers (list) – A list of `StreamWatcher` instances which will be used to scan the program’s `stdout` or `stderr` and may write into its `stdin` (typically `str` or `bytes` objects depending on Python version) in response to patterns or other heuristics. See Automatically responding to program output for details on this functionality. Default: `[]`. echo_stdin (bool) – Whether to write data from `in_stream` back to `out_stream`. In other words, in normal interactive usage, this parameter controls whether Invoke mirrors what you type back to your terminal. By default (when `None`), this behavior is triggered by the following: Not using a pty to run the subcommand (i.e. `pty=False`), as ptys natively echo stdin to stdout on their own; And when the controlling terminal of Invoke itself (as per `in_stream`) appears to be a valid terminal device or TTY. (Specifically, when `isatty` yields a `True` result when given `in_stream`.) Note This property tends to be `False` when piping another program’s output into an Invoke session, or when running Invoke within another program (e.g. running Invoke from itself). If both of those properties are true, echoing will occur; if either is false, no echoing will be performed. When not `None`, this parameter will override that auto-detection and force, or disable, echoing.
Returns:	`Result`, or a subclass thereof.
Raises:	`UnexpectedExit`, if the command exited nonzero and `warn` was `False`.
Raises:	`Failure`, if the command didn’t even exit cleanly, e.g. if a `StreamWatcher` raised `WatcherError`.
Raises:	`ThreadException` (if the background I/O threads encountered exceptions other than `WatcherError`).

generate_result(**kwargs)¶

Create & return a suitable Result instance from the given kwargs.

Subclasses may wish to override this in order to manipulate things or generate a Result subclass (e.g. ones containing additional metadata besides the default).

read_proc_output(reader)¶

Iteratively read & decode bytes from a subprocess’ out/err stream.

Parameters:

reader –

A literal reader function/partial, wrapping the actual stream object in question, which takes a number of bytes to read, and returns that many bytes (or None).

reader should be a reference to either read_proc_stdout or read_proc_stderr, which perform the actual, platform/library specific read calls.

Returns:

A generator yielding Unicode strings (unicode on Python 2; str on Python 3).

Specifically, each resulting string is the result of decoding read_chunk_size bytes read from the subprocess’ out/err stream.

write_our_output(stream, string)¶

Write string to stream.

Also calls .flush() on stream to ensure that real terminal streams don’t buffer.

Parameters:	stream – A file-like stream object, mapping to the `out_stream` or `err_stream` parameters of `run`. string – A Unicode string object.
Returns:	`None`.

handle_stdout(buffer_, hide, output)¶

Read process’ stdout, storing into a buffer & printing/parsing.

Intended for use as a thread target. Only terminates when all stdout from the subprocess has been read.

Parameters:	buffer (list) – The capture buffer shared with the main thread. hide (bool) – Whether or not to replay data into `output`. output – Output stream (file-like object) to write data into when not hiding.
Returns:	`None`.

handle_stderr(buffer_, hide, output)¶

Read process’ stderr, storing into a buffer & printing/parsing.

Identical to handle_stdout except for the stream read from; see its docstring for API details.

read_our_stdin(input_)¶

Read & decode bytes from a local stdin stream.

Parameters:	input – Actual stream object to read from. Maps to `in_stream` in `run`, so will often be `sys.stdin`, but might be any stream-like object.
Returns:	A Unicode string, the result of decoding the read bytes (this might be the empty string if the pipe has closed/reached EOF); or `None` if stdin wasn’t ready for reading yet.

handle_stdin(input_, output, echo)¶

Read local stdin, copying into process’ stdin as necessary.

Intended for use as a thread target.

Note

Because real terminal stdin streams have no well-defined “end”, if such a stream is detected (based on existence of a callable .fileno()) this method will wait until program_finished is set, before terminating.

When the stream doesn’t appear to be from a terminal, the same semantics as handle_stdout are used - the stream is simply read() from until it returns an empty value.

Parameters:	input – Stream (file-like object) from which to read. output – Stream (file-like object) to which echoing may occur. echo (bool) – User override option for stdin-stdout echoing.
Returns:	`None`.

should_echo_stdin(input_, output)¶

Determine whether data read from input_ should echo to output.

Used by handle_stdin; tests attributes of input_ and output.

Parameters:	input – Input stream (file-like object). output – Output stream (file-like object).
Returns:	A `bool`.

respond(buffer_)¶

Write to the program’s stdin in response to patterns in buffer_.

The patterns and responses are driven by the StreamWatcher instances from the watchers kwarg of run - see Automatically responding to program output for a conceptual overview.

Parameters:	buffer (list) – The capture buffer for this thread’s particular IO stream.
Returns:	`None`.

generate_env(env, replace_env)¶

Return a suitable environment dict based on user input & behavior.

Parameters:	env (dict) – Dict supplying overrides or full env, depending. replace_env (bool) – Whether `env` updates, or is used in place of, the value of `os.environ`.
Returns:	A dictionary of shell environment vars.

should_use_pty(pty, fallback)¶

Should execution attempt to use a pseudo-terminal?

Parameters:	pty (bool) – Whether the user explicitly asked for a pty. fallback (bool) – Whether falling back to non-pty execution should be allowed, in situations where `pty=True` but a pty could not be allocated.

has_dead_threads¶

Detect whether any IO threads appear to have terminated unexpectedly.

Used during process-completion waiting (in wait) to ensure we don’t deadlock our child process if our IO processing threads have errored/died.

Returns:	`True` if any threads appear to have terminated with an exception, `False` otherwise.

wait()¶

Block until the running command appears to have exited.

Returns:	`None`.

write_proc_stdin(data)¶

Write encoded data to the running process’ stdin.

Parameters:	data – A Unicode string.
Returns:	`None`.

decode(data)¶: Decode some data bytes, returning Unicode.

process_is_finished¶

Determine whether our subprocess has terminated.

Note

The implementation of this method should be nonblocking, as it is used within a query/poll loop.

Returns:	`True` if the subprocess has finished running, `False` otherwise.

start(command, shell, env)¶

Initiate execution of command (via shell, with env).

Typically this means use of a forked subprocess or requesting start of execution on a remote system.

In most cases, this method will also set subclass-specific member variables used in other methods such as wait and/or returncode.

read_proc_stdout(num_bytes)¶

Read num_bytes from the running process’ stdout stream.

Parameters:	num_bytes (int) – Number of bytes to read at maximum.
Returns:	A string/bytes object.

read_proc_stderr(num_bytes)¶

Read num_bytes from the running process’ stderr stream.

Parameters:	num_bytes (int) – Number of bytes to read at maximum.
Returns:	A string/bytes object.

default_encoding()¶

Return a string naming the expected encoding of subprocess streams.

This return value should be suitable for use by encode/decode methods.

send_interrupt(interrupt)¶

Submit an interrupt signal to the running subprocess.

In almost all implementations, the default behavior is what will be desired: submit to the subprocess’ stdin pipe. However, we leave this as a public method in case this default needs to be augmented or replaced.

Parameters:	interrupt – The locally-sourced `KeyboardInterrupt` causing the method call.
Returns:	`None`.

returncode()¶

Return the numeric return/exit code resulting from command execution.

Returns:	`int`

stop()¶

Perform final cleanup, if necessary.

This method is called within a finally clause inside the main run method. Depending on the subclass, it may be a no-op, or it may do things such as close network connections or open files.

Returns:	`None`

__weakref__¶: list of weak references to the object (if defined)

class invoke.runners.Local(context)¶: Execute a command on the local system in a subprocess.

Note

When Invoke itself is executed without a controlling terminal (e.g. when sys.stdin lacks a useful fileno), it’s not possible to present a handle on our PTY to local subprocesses. In such situations, Local will fallback to behaving as if pty=False (on the theory that degraded execution is better than none at all) as well as printing a warning to stderr.

To disable this behavior, say fallback=False.

class invoke.runners.Result(stdout='', stderr='', encoding=None, command='', shell='', env=None, exited=0, pty=False, hide=())¶

A container for information about the result of a command execution.

All params are exposed as attributes of the same name and type.

Parameters:

stdout (str) – The subprocess’ standard output.
stderr (str) – Same as stdout but containing standard error (unless the process was invoked via a pty, in which case it will be empty; see Runner.run.)
encoding (str) – The string encoding used by the local shell environment.
command (str) – The command which was executed.
shell (str) – The shell binary used for execution.
env (dict) – The shell environment used for execution. (Default is the empty dict, {}, not None as displayed in the signature.)
exited (int) – An integer representing the subprocess’ exit/return code.
pty (bool) – A boolean describing whether the subprocess was invoked with a pty or not; see Runner.run.
hide (tuple) –
A tuple of stream names (none, one or both of ('stdout', 'stderr')) which were hidden from the user when the generating command executed; this is a normalized value derived from the hide parameter of Runner.run.

For example, run('command', hide='stdout') will yield a Result where result.hide == ('stdout',); hide=True or hide='both' results in result.hide == ('stdout', 'stderr'); and hide=False (the default) generates result.hide == () (the empty tuple.)

Note

Result objects’ truth evaluation is equivalent to their ok attribute’s value. Therefore, quick-and-dirty expressions like the following are possible:

if run("some shell command"):
    do_something()
else:
    handle_problem()

However, remember Zen of Python #2.

return_code¶: An alias for .exited.

ok¶: A boolean equivalent to exited == 0.

__weakref__¶: list of weak references to the object (if defined)

failed¶

The inverse of ok.

I.e., True if the program exited with a nonzero return code, and False otherwise.

runners¶

`runners`¶