...
 
Commits (6)
INFERIOR-SHELL
INFERIOR-SHELL: spawning shell commands and pipes from Common Lisp
==================================================================
What is inferior-shell?
-----------------------
This CL library allows you to spawn local or remote processes and shell pipes.
It lets me use CL in many cases where I would previously write shell scripts.
......@@ -23,19 +27,21 @@ The name inferior-shell was suggested by Michael Livshin,
as inspired by the equivalent notion in GNU Emacs.
Example use of inferior-shell, from the rpm system:
(defun rpms-installed (&key (packagenames t) host)
(run/lines
`(pipe (rpm -qa)
,@(unless (eq packagenames t)
`((egrep ("^(" ,@(loop :for (name . more) :on packagenames
:collect name :when more :collect "|")
")-[^-]+(-[^-]+)?$")))))
:host host))
(defun rpms-installed (&key (packagenames t) host)
(run/lines
`(pipe (rpm -qa)
,@(unless (eq packagenames t)
`((egrep ("^(" ,@(loop :for (name . more) :on packagenames
:collect name :when more :collect "|")
")-[^-]+(-[^-]+)?$")))))
:host host))
==== Limitations ====
Limitations
-----------
By default, inferior-shell uses asdf-driver:run-program
By default, inferior-shell uses uiop:run-program
as its universal execution backend, and has its limitations,
which are as follows.
......@@ -46,31 +52,28 @@ IOlib requires C compilation and linking, and may or may not support Windows.
executor only supports select implementations.
A future extension to inferior-shell may use IOlib as a backend.
Second, there is currently limited support for input redirection.
The only possible input redirection is from /dev/null
or by inheriting the parent's standard input
when running in :interactive mode.
However, using shell redirection, you can also redirect input from a file,
or from a numbered file descriptor (except 0, 1, 2).
Finally, supported platforms at this time include:
Second, supported platforms at this time include:
ABCL, Allegro, CLISP, ClozureCL, CMUCL, ECL, LispWorks, RMCL, SBCL, SCL, XCL.
Platforms NOT (yet) supported include:
CormanLisp (untested), GCL (untested), Genera (unimplemented), MKCL (untested).
On supported platforms, inferior-shell works on both Unix and Windows.
==== Exported Functionality ====
Exported Functionality
----------------------
The inferior-shell library creates a package INFERIOR-SHELL,
that exports the following macros and functions:
PARSE-PROCESS-SPEC SPEC
* `PARSE-PROCESS-SPEC SPEC`
parse an expression in the process-spec mini-language into
objects specifying a pipeline of processes to be executed.
See the PROCESS-SPEC mini-language below.
PRINT-PROCESS-SPEC SPEC &OPTIONAL OUTPUT
* `PRINT-PROCESS-SPEC SPEC &OPTIONAL OUTPUT`
print a process specification to given OUTPUT
into a portable form usable by a Unix shell.
OUTPUT is as per FORMAT's stream output argument,
......@@ -79,83 +82,105 @@ PRINT-PROCESS-SPEC SPEC &OPTIONAL OUTPUT
a CONS to be parsed by PARSE-PROCESS-SPEC,
or a string for a process-spec that has already been formatted.
*CURRENT-HOST-NAMES*
* `*CURRENT-HOST-NAMES*`
a variable, a list of strings, the aliases for the localhost.
You may need to initialize it if the defaults don't suffice.
CURRENT-HOST-NAME-P X
* `CURRENT-HOST-NAME-P X`
a function, returns true if X is a string member of *CURRENT-HOST-NAMES*
INITIALIZE-CURRENT-HOST-NAMES
* `INITIALIZE-CURRENT-HOST-NAMES`
function that initializes the *CURRENT-HOST-NAMES*
with "localhost" and the results from $(hostname -s) and $(hostname -f).
RUN CMD &KEY ON-ERROR TIME SHOW HOST OUTPUT
RUN will execute the given command CMD, which can be
a CONS to be parsed by PARSE-PROCESS-SPEC,
a PROCESS-SPEC object already parsed,
* `RUN CMD &KEY ON-ERROR TIME SHOW HOST OUTPUT`
`RUN` will execute the given command `CMD`, which can be
a `CONS` to be parsed by `PARSE-PROCESS-SPEC`,
a `PROCESS-SPEC` object already parsed,
or a string to be passed to a Unix shell.
ON-ERROR specifies behavior in case the command doesn't successfully exit
with exit code 0. NIL means continue and return the error code;
T (default) means signal the error (same as :ON-ERROR 'SIGNAL);
any other value is called as by UIOP:CALL-FUNCTION with the condition as argument,
and if it returns, its return values are returned by RUN.
TIME is a boolean which if true causes the execution to be timed as per TIME.
SHOW is a boolean which if true causes a message to be sent
to the *TRACE-OUTPUT* before execution.
HOST is either NIL (execute on localhost) or a string specifying a host
on which to run the command using ssh if
it's not an alias for localhost as recognized by CURRENT-HOST-NAME-P
(be sure to have passphraseless login using ssh-agent).
The INPUT, OUTPUT and ERROR-OUTPUT arguments are as for UIOP:RUN-PROGRAM,
except that they default to NIL, T, T respectively instead of NIL, NIL, NIL.
In particular, OUTPUT is as per UIOP:SLURP-OUTPUT-STREAM one of
NIL for no output (redirect to /dev/null),
`ON-ERROR` specifies behavior in case the command doesn't successfully exit
with exit code 0. `NIL` means continue and return the error code;
`T` (default) means signal the error (same as `:ON-ERROR 'SIGNAL`);
any other value is called as by `UIOP:CALL-FUNCTION` with the condition as argument,
and if it returns, its return values are returned by `RUN`.
`TIME` is a boolean which if true causes the execution to be timed as per TIME.
`SHOW` is a boolean which if true causes a message to be sent
to the `*TRACE-OUTPUT*` before execution.
`HOST` is either `NIL` (execute on localhost) or a string specifying a host
on which to run the command using `ssh` if
it's not an alias for localhost as recognized by `CURRENT-HOST-NAME-P`
(be sure to have passphraseless login using `ssh-agent`).
The `INPUT`, `OUTPUT` and `ERROR-OUTPUT` arguments are as for `UIOP:RUN-PROGRAM`,
except that they default to `NIL`, `T`, `T` respectively instead of `NIL`, `NIL`, `NIL`.
In particular, `OUTPUT` is as per `UIOP:SLURP-OUTPUT-STREAM` one of
`NIL` for no output (redirect to `/dev/null`),
a stream for itself,
T (default) for the current *standard-output*,
:INTERACTIVE for inheriting the parent process's stdout,
:LINES for returning one result per line,
:STRING for returning the output as one big string,
:STRING/STRIPPED is like :STRING
`T` (default) for the current `*standard-output*`,
`:INTERACTIVE` for inheriting the parent process's stdout,
`:LINES` for returning one result per line,
`:STRING` for returning the output as one big string,
`:STRING/STRIPPED` is like `:STRING`
but strips any line-ending at the end of the results,
just like a shell's `cmd` or $(cmd) would, and
just like a shell's ``cmd`` or `$(cmd)` would, and
more options are accepted and you can define your own, as per
asdf-driver's slurp-input-stream protocol.
On Windows, RUN will not succeed for pipes, only for simple commands.
On Windows, `RUN` will not succeed for pipes, only for simple commands.
On Unix, simple commands on localhost are executed directly, but
remote commands and pipes are executed by spawning a shell.
RUN/NIL CMD &KEY ON-ERROR TIME SHOW HOST
RUN/NIL is a shorthand for RUN with :INPUT :OUTPUT :ERROR-OUTPUT bound to NIL.
* `RUN/NIL CMD &KEY ON-ERROR TIME SHOW HOST`
`RUN/NIL` is a shorthand for `RUN` with `:INPUT` `:OUTPUT` `:ERROR-OUTPUT` bound to `NIL`.
RUN/S CMD &KEY ON-ERROR TIME SHOW HOST
RUN/S is a shorthand for RUN with :OUTPUT bound to :STRING,
* `RUN/S CMD &KEY ON-ERROR TIME SHOW HOST`
`RUN/S` is a shorthand for `RUN` with `:OUTPUT` bound to `:STRING`,
returning as a string what the inferior command sent to its standard output.
RUN/SS CMD &KEY ON-ERROR TIME SHOW HOST
RUN/S is a shorthand for RUN :OUTPUT :STRING/STRIPPED,
just like a shell's `cmd` or $(cmd) would do.
* `RUN/SS CMD &KEY ON-ERROR TIME SHOW HOST`
`RUN/S` is a shorthand for `RUN :OUTPUT :STRING/STRIPPED`,
just like a shell's ``cmd`` or `$(cmd)` would do.
RUN/INTERACTIVE CMD &KEY ON-ERROR TIME SHOW HOST
RUN/INTERACTIVE is a shorthand for RUN with :INPUT :OUTPUT :ERROR-OUTPUT
all boud to :INTERACTIVE, so you may run commands that interact with users,
* `RUN/INTERACTIVE CMD &KEY ON-ERROR TIME SHOW HOST`
`RUN/INTERACTIVE` is a shorthand for `RUN` with `:INPUT` `:OUTPUT` `:ERROR-OUTPUT`
all bound to `:INTERACTIVE`, so you may run commands that interact with users,
inheritting the stdin, stdout and stderr of the current process.
RUN/LINES CMD &KEY ON-ERROR TIME SHOW HOST
run/lines is a shorthand for RUN :OUTPUT :LINES,
* `RUN/I CMD &KEY KEYS`
`RUN/I` is a shorthand alias for `RUN/INTERACTIVE`
* `RUN/LINES CMD &KEY ON-ERROR TIME SHOW HOST`
`RUN/LINES` is a shorthand for `RUN :OUTPUT :LINES`,
returning as a list of one string per line (stripped of line-ending)
what the inferior command sent to its standard output.
*BACKEND*
* `*BACKEND*`
a variable to choose between backends. Currently, only supported are
:AUTO (the default, using asdf-driver:run-program, and
`:AUTO` (the default, using asdf-driver:run-program, and
spawning a shell unless it's a simple process), and
:SBCL (only available on #+(and sbcl sb-thread unix),
`:SBCL` (only available on `#+(and sbcl sb-thread unix)`,
doesn't need a shell but has some limitations such as
only supporting redirection of stdin, stdout, stderr).
only supporting redirection of stdin, stdout, stderr),
and is not recommended (and not actively supported at the moment).
==== THE PROCESS-SPEC MINI-LANGUAGE ====
The PROCESS-SPEC mini-language
------------------------------
This library offers a SEXP syntax to specify processes
and pipelines of processes in the manner of Unix shells,
......@@ -167,6 +192,7 @@ depending on its capabilities and on the complexity of the pipeline.
SEXP mini-language
```
;; A process is a pipe or a command
process := pipe | or | and | progn | fork | command
......@@ -207,6 +233,7 @@ redirection := (
<& fd fd | >& fd fd | ;; redirect fds: the left one is the new number, the right one the old number.
>& pn | >&! | ;; redirect both fd 1 and 2 to pathname (respectively, clobbering)
>>& pn | >>&! ) ;; redirect both fd 1 and 2 to append to pathname (respectively, clobbering)
```
Note that these are all exported symbols from the INFERIOR-SHELL package,
except that a few of them are also inherited from COMMON-LISP: < > -
......@@ -221,9 +248,10 @@ in this context, a string represents itself (assuming it's already a printed pro
and a cons is a specification in the minilanguage to be parsed with parse-process-spec first.
==== TO DO ====
TO DO
-----
Document it.
Better document it.
Have a complementary inferior-shell-watcher library that uses iolib to spawn
pipes locally, and watch the subprocesses as part of the iolib event loop.
;;; -*- Lisp -*-
#-asdf3 (error "inferior-shell requires ASDF 3.0.3 or later")
(defsystem "inferior-shell"
:version "2.0.3.1"
:version "2.0.3.3"
:description "spawn local or remote processes and shell pipes"
:author "Francois-Rene Rideau"
:license "MIT"
:defsystem-depends-on (:asdf #-asdf3 "uiop")
:depends-on ((:version #+asdf3 "asdf" #-asdf3 "uiop" "3.0.3") ; input and error-output redirection
#+sbcl "sb-posix"
:depends-on ((:version "asdf" "3.0.3") ; input and error-output redirection
#+sbcl (:require "sb-posix")
"alexandria" "optima"
"fare-utils" "fare-quasiquote-extras" "fare-mop")
:around-compile "uiop:call-with-safe-io-syntax" ;; ensures that quasiquote syntax doesn't escape
......@@ -19,11 +19,10 @@
(:file "host" :depends-on ("pkgdcl"))
(:file "run" :depends-on ("process-spec" "macros"))
(:file "run-generic" :depends-on ("process-spec" "macros"))
#+asdf3
(:file "run-sbcl" :depends-on ("process-spec" "macros" "run-generic")
:if-feature (:and :sbcl :sb-thread :unix)))
:in-order-to ((test-op (load-op "inferior-shell/test")))
:perform (test-op :after (o s) ;; symbol-call will only work if loaded with ASDF3
:perform (test-op (o s) ;; symbol-call will only work if loaded with ASDF3
(symbol-call :inferior-shell-test :test-suite)))
(defsystem "inferior-shell/test"
......
......@@ -4,7 +4,7 @@
(:mix :fare-utils :uiop :alexandria)
(:use :cl :optima :named-readtables :fare-mop)
(:export
#:run #:run/nil #:run/s #:run/ss #:run/interactive #:run/lines
#:run #:run/nil #:run/s #:run/ss #:run/interactive #:run/i #:run/lines
#:simple-command-line-token #:token-string
#:process-spec #:command-spec #:pipe-spec
#:or-spec #:and-spec #:progn-spec #:fork-spec
......
......@@ -54,7 +54,8 @@
(defun run/nil (cmd &rest keys
&key time show host (on-error t)
&allow-other-keys)
"run command CMD"
"run command CMD. Unless otherwise specified, discard the subprocess's output and error-output.
See the documentation for INFERIOR-SHELL:RUN for other keyword arguments."
(when (eq on-error t) (setf on-error 'error))
(when show
(format *trace-output* "; ~A~%" (print-process-spec cmd)))
......@@ -68,37 +69,87 @@
(defun run (cmd &rest keys
&key on-error time show host (output t op) (error-output t eop) &allow-other-keys)
"run command CMD"
(declare (ignore on-error time show host))
(apply 'run/nil cmd `(,@(unless op `(:output ,output))
,@(unless eop `(:error-output ,error-output))
"run command CMD. Unless otherwise specified, copy the subprocess's output to *standard-output*
and its error-output to *error-output*. Return values for the output, error-output and exit-code.
The command specified by COMMAND can be a list as parsed by PARSE-PROCESS-SPEC, or a string
to execute with a shell (/bin/sh on Unix, CMD.EXE on Windows).
Signal a continuable SUBPROCESS-ERROR if the process wasn't successful (exit-code 0),
unless ON-ERROR is specified, which if NIL means ignore the error (and return the three values).
If OUTPUT is a pathname, a string designating a pathname, or NIL designating the null device,
the file at that path is used as output.
If it's :INTERACTIVE, output is inherited from the current process;
beware that this may be different from your *STANDARD-OUTPUT*,
and under SLIME will be on your *inferior-lisp* buffer.
If it's T, output goes to your current *STANDARD-OUTPUT* stream.
Otherwise, OUTPUT should be a value that is a suitable first argument to
SLURP-INPUT-STREAM (qv.), or a list of such a value and keyword arguments.
In this case, RUN-PROGRAM will create a temporary stream for the program output;
the program output, in that stream, will be processed by a call to SLURP-INPUT-STREAM,
using OUTPUT as the first argument (or the first element of OUTPUT, and the rest as keywords).
The primary value resulting from that call (or NIL if no call was needed)
will be the first value returned by RUN-PROGRAM.
E.g., using :OUTPUT :STRING will have it return the entire output stream as a string.
And using :OUTPUT '(:STRING :STRIPPED T) will have it return the same string
stripped of any ending newline.
ERROR-OUTPUT is similar to OUTPUT, except that the resulting value is returned
as the second value of RUN-PROGRAM. T designates the *ERROR-OUTPUT*.
Also :OUTPUT means redirecting the error output to the output stream,
in which case NIL is returned.
INPUT is similar to OUTPUT, except that VOMIT-OUTPUT-STREAM is used,
no value is returned, and T designates the *STANDARD-INPUT*.
Use ELEMENT-TYPE and EXTERNAL-FORMAT are passed on
to your Lisp implementation, when applicable, for creation of the output stream.
One and only one of the stream slurping or vomiting may or may not happen
in parallel with the subprocess, depending on options and implementation,
and with priority being given to output processing.
Other streams are completely produced or consumed
before or after the subprocess is spawned, using temporary files.
RUN returns 3 values:
0- the result of the OUTPUT slurping if any, or NIL
1- the result of the ERROR-OUTPUT slurping if any, or NIL
2- either 0 if the subprocess exited with success status,
or an indication of failure via the EXIT-CODE of the process"
(declare (ignore on-error time show host output error-output))
(apply 'run/nil cmd `(,@(unless op `(:output t))
,@(unless eop `(:error-output t))
,@keys)))
(defun run/s (cmd &rest keys &key on-error time show host)
"run command CMD, return its standard output results as a string."
"run command CMD, return its standard output results as a string.
Unless otherwise specified, discard its error-output.
See the documentation for INFERIOR-SHELL:RUN for other keyword arguments."
(declare (ignore on-error time show host))
(apply 'run/nil cmd :output 'string keys))
(defun run/ss (cmd &rest keys &key on-error time show host)
"Like run/s, but strips the line ending off the result string;
very much like `cmd` or $(cmd) at the shell"
"run command CMD, return its standard output results as a string like run/s,
but strips the line ending off the result string very much like `cmd` or $(cmd) at the shell
See the documentation for INFERIOR-SHELL:RUN for other keyword arguments."
(declare (ignore on-error time show host))
(apply 'run/nil cmd :output :string/stripped keys))
(apply 'run/nil cmd :output '(:string :stripped t) keys))
(defun run/interactive (cmd &rest keys &key on-error time show host)
"run command CMD interactively."
"run command CMD interactively, connecting the subprocess's input, output and error-output
to the same file descriptors as the current process
See the documentation for INFERIOR-SHELL:RUN for other keyword arguments."
(declare (ignore on-error time show host))
(apply 'run/nil cmd :input :interactive :output :interactive :error-output :interactive keys))
(defun slurp-stream-string/stripped (input-stream)
(stripln (slurp-stream-string input-stream)))
(defmethod slurp-input-stream ((x (eql :string/stripped)) input-stream
&key &allow-other-keys)
(slurp-stream-string/stripped input-stream))
(defun run/i (cmd &rest keys)
"alias for run/interactive"
(apply 'run/interactive cmd keys))
(defun run/lines (cmd &rest keys &key on-error time show host)
"Like run/s, but return a list of lines rather than one string"
"run command CMD, return its standard output results as a list of strings, one per line,
discarding line terminators. Unless otherwise specified, discard error-output.
See the documentation for INFERIOR-SHELL:RUN for other keyword arguments."
(declare (ignore on-error time show host))
(apply 'run/nil cmd :output :lines keys))
......@@ -23,6 +23,6 @@
(deftest test-inferior-shell ()
(let ((*backend* :auto))
(do-test-inferior-shell))
#+(and sbcl sb-thread unix)
#|#+(and sbcl sb-thread unix)
(let ((*backend* :sbcl))
(do-test-inferior-shell)))
(do-test-inferior-shell))|#)