------------------------------------------------------------------------------- For reasons on why this is so difficult (buffering problems) look at the file.hints document Shell scripts sometimes have to control a program that normally requires or enforces direct user interaction (terminal-wise not GUI-wise). This can be tricky as not only do you need to get the programs results, you also need to input information to the program basied on the programs output. that is you need to wait for prompts, or other returned results, to decide on what input you should next send to the program. typical examples of programs that you want ------------------------------------------------------------------------------- Problems with interactive commands. A wrapper program (usally involving a PTY) is used around a command for quite a number of reasons, when you are programmically handling an interactive. 1/ If the command reads directly from the TTY device rather than standard input, a PTY is the only want to allow you to pipe in your own automatied input. The "ptybandage" of the PTY package encloses a program completely in its own TTY, not just the stdin and stdout as other problems do. That of course means stderr will be merged with stdout as a side-effect. 2/ You may need to avoid the system STDIO library from 'block' buffering the input/output, and keep it buffered line by line (both input and output). One way to do this is to pty the command so it looks like it is in an interactive terminal. Note that some modern versions of 'cat' and 'grep' have a 'line-by-line' buffering option added for this very reason, causing the program to forcefully flush after every end-of-line character, rather than leaving it up to the stdio library buffer handling. This last problem can be worse for network commands where lines are also network packets. This can be solved by using "condom" (or "pty -0") of the PTY package. The "nobuf" script of the replacement "ptyget" package. The "unbuffer" of the "expect" package (standard on most linuxes). The "script" typescripting program is also suposed to do this. 3/ Some commands will flush (junk) all old input just before prompting for new input as a security precaution. "Telnet" does this just after it connects to a remote system. while "passwd" does it just before prompting for a new password. In that case you either need to delay, or wait for the prompt (which may not have a final newline!) 4/ What 'end-of-data' marker should you use? How do you know your resquested information is complete? Basically when has the program finished returning data? Many unix commands for example don't given any final 'end-of-data' indicator. For remote shell programs, this is not so bad, as the end-of-data may be a shell prompt, However be careful as this will not usally have a final newline character, which can cause buffering problems with nont PTY wrapped programs. That is the prompt gets buffered somewhere waiting for the newline that never comes. Caution and testing is recomended. 5/ Handling out-of-band data and errors... Some programs may output data, at any time, which could mix with the output you requested. For example a status display on a terminal. It can also output unexpected errors and problem reports. That in turn can complicate the whole situation as you may have to continuiously monitor the interactive command, for these updates, before, during, and even after you have read the data you specifically requested. It gets worse if this out-of-band data looks simular to specifically requested data! Which confuses the issue with the last point. Worse still the unexpected data cause a erroneous 'end-of-data' sequence, or worse still no 'end-of-data' indicator at all (and causing a lockup). Believe me this is the bane of interactive program control, and the reason such programs are often flaky and have a bad reputation. The only solution is to not only look for your 'end-of-data' but you also have to look for and identify these status and error data as seperate to the data you was expecting. Unfortunatally unless you are intimately familiar with a program (and all varients of that program!) you are unlikely to be able to predict all such 'exceptions'. The common solution is generally a timeout of some type. The problem with that is "how long do you wait?". Too long and your program can become very slow when lots of such 'problems' happen. Too short, and you can fail to automatically handle problems caused by slow network connections. Another solution, if that application allows it is to program in some sort of 'heartbeat'. That is you get the program to output an out-of-band indicator that it is still alive and working. Or have it respond to asycronious input "are you ok?" type signal. From users this is typically a key sequence that produces a small response of no real importance. With shell scripts, you have no simple way to just read all data that is currently waiting, without somehow inserting some extra markers, or slowing your whole application with read timeouts. 6/ Avoiding Lockups.... Programmically the worse problem with interact command programming is a lockup. Basically the program has finished outputing its data, and is witing for new input, while your controling program is still wait for some response from the interactive program that will never come. Typically this is caused by the interactive program aborting unexpectally or a unexpected error causing the program to never produce the desired 'end-of-data' response you are waiting for. ------------------------------------------------------------------------------- Programs/Wrappers expect This is the traditional TCL scripting for running interactive commands. However it is not usally thought of as 'simple' to use and does not interact well with more complex scripting (except TCL) pty A general run program under PTY's that is suposed to 'just work'. Last release pty-4.0 (1992) ptyget A complete re-write of the pty package, by the same author Release 0.50 empty A shell based TCL/Expect replacement using a C program developed from pty-4.0. Basically it automates much of the more complex aspects of generating interactive controls. However it does not seem to allow for 'multiple' string tests. http://empty.sourceforge.net/ unbuffer Another TCL/Expect script to remove buffering from command output ------------------------------------------------------------------------------- Interactive programs from shell script (eg mail, ftp, telnet, tip, passwd) The shell itself cannot interact with interactive tty-based programs like these. Fortunately some programs have been written to manage the connection to a pseudo-tty so that you can run these sorts of programs in a script. 'expect' is a one such program, which you can ftp pub/expect.shar.Z from durer.cme.nist.gov. # username is passed as 1st arg, password as 2nd set password [index $argv 2] spawn passwd [index $argv 1] expect "*password:" send "$password\r" expect "*password:" send "$password\r" expect eof Another solution is provided by the 'pty' program, which runs a program under a pty session and was posted to comp.sources.unix, volume 23, issue 31. You can also ftp pub/flat/pty-* from stealth.acf.nyu.edu . #!/bin/sh ( sleep 5; echo "$2"; sleep 5; echo "$2") | pty passwd "$1" This is simple pty passwd solution, but is dependant on the sleep timing as it has not syncronization. If you could 'waitfor' specific output before supplying input, you can improve the stability of the whole system... #!/bin/sh /etc/mknod out.$$ p; exec 2>&1 ( exec 4/dev/null ) | ( pty passwd "$1" >out.$$ ) rm -f out.$$ The 'waitfor' is a simple C program that searches for its argument in the input, character by character. You can ftp pub/flat/misc-waitfor.c from stealth.acf.nyu.edu (See a shell equivelent below) This uses a named pipe to pipe the output of the program back into the input of the above pipeline. Looks complex but it isn't really. Here is a simplier form of the above... =======8<-------- #!/bin/sh mkfifo out.fifo in.fifo telnet -K localhost 1> out.fifo 0< in.fifo & cat > in.fifo & # send all sendout to program input cat out.fifo > out.fifo & # ????? # Do the 'expect' login. pid=`jobid` waitfor "ogin" echo "luser" waitfor "word" echo "TopSecret" # Now run the remote command sleep 1 echo 'who am i > /tmp/test.txt' > in.fifo sleep 1 echo "exit" > in.fifo rm out.fifo in.fifo kill $pid =======8<-------- However the last part is still unstable as it relays on sleeps. ------------------------------------------------------------------------------- WaitFor Alturnative You can also do the waitfor in shells too! See the example "interactive.shell" for one such line based a method. Here is a 'shell expect' type program... That uses "out.fifo" and "in.fifo" named pipes. =======8<-------- #!/bin/sh # # expect.sh "search" "response" # while :; do dd if=out.fifo bs=1b count=1 2>/dev/null | grep "$1" if [ $? -eq 0 ]; then # Match found, send response echo "$2" > in.fifo exit 0 fi # Match not found, continue to search. done =======8<-------- The above is untested - I am not certain about its full value. ------------------------------------------------------------------------------- Using expect to switch a commands output to line buffering Command 'unbuffer' =======8<-------- #!/bin/sh # # Description: unbuffer stdout/stderr of a program # Author: Don Libes, NIST # # Option: -p allow run program to read from stdin (for simplification) # # Note that expect can 'continue' a comment line, so the follow re-runs this # as a expect command regardless of its PATH location. # \ exec expect -- "$0" ${1+"$@"} if {[string compare [lindex $argv 0] "-p"] == 0} { # pipeline set stty_init "-echo" eval spawn -noecho [lrange $argv 1 end] interact } else { set stty_init "-opost" eval spawn -noecho $argv set timeout -1 expect } =======8<-------- Note that this command is now often instealled as part of the expect package. ------------------------------------------------------------------------------- For ftp here is a simple solution # download file from a anonymous ftp server ftp -n -i marlin.jcu.edu.au <<-EOF # 2>/dev/null user ftp your_email_address cd /direct/file/is/stored binary get filename dest_filename bye EOF ------------------------------------------------------------------------------- For Telnet WARNING: Some versions of telnet has two problems associated with it. First it does not read 'piped' input (especially of Sun Micosystems Computers). Second some versions flushes its input after it has connected, meaning you have to delay input until after that point. The "mconnect" command (undet solaris) or "netcat" (often installed as "nc") or even a perl equivelent (like "tcp_client") should be used instead. However you may still require the use of the ``waitfor'' program... -------------------------------------------------------------------------------