|
Softpanorama |
May the source be with you, but remember the KISS principle ;-)
|
| Recommended Links | Reference | ||||
| VNC | Screen | Perl Expect.pm module | Python Module | Humor | Etc |
Expect is a great tool in a system administrators arsenal and can be used to easily automate tasks that require periodic user input. This can allow the administrator to make better use of their time than watching the application or utility to spot the next time it requires input. For example expect is a uniquely powerful tool for automating interactive applications such as telnet and ftp. For the same reason expect is also useful for testing interactive applications. And by adding Tk, you can also wrap interactive applications in X11 GUIs. Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily. Expect - Expect - Home Page -- an extremely interesting page of Don Libes the famous author of Expect. Here is a short history of Expect:
Expect was conceived of in September, 1987. The bulk of version 2 was designed and written between January and April, 1990. Minor evolution occurred after that until Tcl 6.0 was released. At that time (October, 1991) approximately half of Expect was rewritten for version 3. See the HISTORY file for more information. The HISTORY file is included with the Expect distribution.
Around January 1993, an alpha version of Expect 4 was introduced. This included Tk support as well as a large number of enhancements. A few changes were made to the user interface itself, which is why the major version number was changed. A production version of Expect 4 was released in August 1993.
In October 1993, an alpha version of Expect 5 was released to match Tcl 7.0. A large number of enhancements were made, including some changes to the user interface itself, which is why the major version number was changed (again). The production version of Expect 5 was released in March '94.
In the summer of 1999, substantial rewriting of Expect was done in order to support Tcl 8.2. (Expect was never ported to 8.1 as it contained fundamental deficiencies.) This included the creation of an exp-channel driver and object support in order to take advantage of the new regexp engine and UTF/Unicode. The user interface is highly but not entirely backward compatible. See the NEWS file in the distribution for more detail.
There are important differences between Expect 3, 4, and 5. See the CHANGES.* files in the distribution if you want to read about the differences. Expect 5.30 and earlier versions have ceased development and are not supported. However, the old code is available from http://expect.nist.gov/old.
The Expect book became available in January '95. It describes Expect 5 as it is today, rather than how Expect 5 was when it was originally released. Thus, if you have not upgraded Expect since before getting the book, you should upgrade now.
Historical notes on Tcl and Tk according to John Ousterhout
I got the idea for Tcl while on sabbatical leave at DEC's Western Research Laboratory in the fall of 1987. I started actually implementing it when I got back to Berkeley in the spring of 1988; by summer of that year it was in use in some internal applications of ours, but there was no Tk. The first external releases of Tcl were in 1989, I believe. I started implementing Tk in 1989, and the first release of Tk was in 1991.
Among other things it contains rscript 1.0 which is based upon Expect/Tcl and allows you to automate remote logins and execute remote commands, somewhat like rsh.
The site also contains many valuable examples. Among them are multixterm, kibitz, rftp (recursive ftp), passmass, autoexpect. Here are man pages for some of the examples:
There are many interesting papers about expect. Among them:
|
Expect is a venerable tool for scripting interactive command-line tools. One normally sees expect coupled with the TCL programming language -- for example, in the DejaGNU test environment. Expect-lite is a wrapper for expect designed to allow you to capture an interactive session more directly mapped into a script. The expect-lite language also includes simple conditionals and other programming language elements, and can be readily mixed with bash programming.Expect-lite is itself an expect script, so to use it you will need to install expect from your distribution's package repository. Once you have it installed, expand the expect-lite tarball and copy the expect-lite.proj/expect-lite file to somewhere in your $PATH and give it appropriate permissions.# tar xzvf /.../expect-lite_3.0.5.tar.gz # cp expect-lite.proj/expect-lite /usr/local/bin # chown root.root /usr/local/bin/expect-lite # chmod 555 /usr/local/bin/expect-liteThe main goal of expect and expect-lite is to automate an interactive session with a command. I'll refer to the command that expect(-lite) is talking to as the spawned command. A basic expect(-lite) script might spawn a command, wait for the command to ask for some input, and send some data in response to that request.
In expect, the
sendcommand sends information to the spawned command, and theexpectcommand is used to wait for the spawned command to send a particular piece of information. Normally there is a timeout set to handle the case where the spawned command does not provide the expected output within a given time; in other words, when the spawned program does not behave as planned by the expect(-lite) script.I'll use gdb as an example to demonstrate the difference between using expect and expect-lite to automate a simple interaction. Shown below is the code for a trivial C++ application and the start of a normal gdb debug session for this application.
$ cat main.cpp #include <iostream> using namespace std; int main( int, char** ) { int x = 2; x *= 7; cout << x << endl; return 0; } $ g++ -g main.cpp -o main $ gdb ./main GNU gdb Red Hat Linux (6.6-35.fc8rh) ... (gdb) br main Breakpoint 1 at 0x400843: file main.cpp, line 7. (gdb) r Breakpoint 1, main () at main.cpp:7 7 int x = 2; (gdb)The expect program below will automate the execution of the above program with gdb printing out the value of
xat a point in the program execution.$ cat ./gdb-main-expect #!/usr/bin/expect spawn gdb ./main; send "br main\n"; expect "Breakpoint 1 at"; send "r\n"; expect "Breakpoint 1, main"; expect "(gdb)"; send "step\n"; expect "(gdb)"; send "step\n"; expect "(gdb)"; send "print x\n"; send "quit\n"; expect "Exit anyway"; send "y\n"; $ ./gdb-main-expect spawn gdb ./main br main GNU gdb Red Hat Linux (6.6-35.fc8rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... Using host libthread_db library "/lib64/libthread_db.so.1". (gdb) br main Breakpoint 1 at 0x400843: file main.cpp, line 7. (gdb) r ... Breakpoint 1, main () at main.cpp:7 7 int x = 2; (gdb) step 8 x *= 7; (gdb) step 9 cout << x << endl; (gdb) print x $1 = 14 (gdb) quit The program is running. Exit anyway? (y or n) $The same automated interaction is expressed in expect-lite below. Notice that the quoting and other syntax is now missing and the expect-lite program is much closer to what a user would actually see and type. Lines starting with
>>are things that expect-lite will send to the spawned command. Lines starting with<are things that expect-lite will wait to see in the output of the spawned command.$ cat gdb-main-expect-lite.elt > gdb /home/ben/expect-lite-examples/main >>br main <Breakpoint 1 at >>r <Breakpoint 1, main >>step <(gdb) >>step <(gdb) >>print x >>quit <Exit anyway >>yExpect-lite was created primarily for running automated software tests. A side effect of this heritage is that any script execution requires the specification of the host where the test is to be executed. In the example below I have set up SSH to allow me to log in to the same user on localhost without a password, as detailed after the example. There should be no security implications with this. At the end of the expect-lite interaction you can see an overall result as passed, which is also due to expect-lite's primary purpose being software testing.
$ expect-lite remote_host=localhost cmd_file=gdb-main-expect-lite.elt spawn ssh localhost Last login: ... 2008 from localhost.localdomain $ bash ->Check Colour Prompt Timed Out! $ gdb /home/ben/expect-lite-examples/main br main ... (gdb) br main Breakpoint 1 at 0x400843: file main.cpp, line 7. (gdb) r Breakpoint 1, main () at main.cpp:7 7 int x = 2; (gdb) step 8 x *= 7; (gdb) step 9 cout << x << endl; (gdb) print x $1 = 14 (gdb) quit The program is running. Exit anyway? (y or n) y ##Overall Result: PASSThe below commands set up SSH to allow you to connect to the localhost without a passphrase. In this case the "elo" hostname is also available as a shortcut to connect using the correct SSH identity file. In the above session I could have used
expect-lite remote_host=elo cmd_file=...to execute the command on localhost.$ mkdir -p ~/.ssh $ chmod 700 ~/.ssh $ cd ~/.ssh $ ssh-keygen -f expect-lite-key $ cat expect-lite-key.pub >> authorized_keys2 $ vi ~/.ssh/config Host localhost IdentityFile ~/.ssh/expect-lite-key Host elo HostName localhost IdentityFile ~/.ssh/expect-lite-key $ chmod 600 config $ chmod 600 authorized_keys2 $ ssh eloFor the gdb example I could not use the more normal
>to describe what expect-lite should send, because the>expect-lite command has some built-in logic to first detect a prompt from the spawned command.>>on the other hand just sends what you have specified right now. The prompt detection logic in expect-lite did not detect the gdb prompt in the gdb-main-expect-lite.elt example above and so would time out waiting for a prompt from the spawned command.Expect-lite did not detect the gdb prompt because the regular expressions that it uses in
wait_for_promptdid not handle that prompt format. One way to change that situation would be to customize those regular expressions in the expect-lite file if you are using expect-lite against gdb often. Because I was forced to use>>to send data to gdb, I had to also include lines at some places which explicitly waited for gdb to send a prompt. If the prompt detection regular expressions can detect your prompts, then using the>command to send data will make a less cluttered script, as you will not need to explicitly wait for prompts from the spawned program.You can assign values to variables by capturing part of the output of the spawned command. When your expect-lite script starts executing, the spawned command will be bash, so you can directly run programs by sending the command line to bash with the
>expect-lite command. As seen above, if you execute an interactive program such as gdb, then the>expect-lite command talks with gdb.The below script captures the output of the
idcommand and throws away everything except the user name. The conditional expression in expect-lite has a similar syntax to that of a similar expression in C; the difference is that in expect-lite the IF portion is prefixed and terminated with a question mark and the separator between the THEN and ELSE statement is a double colon instead of a single colon. The reason that I wait for the final "..." at the last line of the script is so expect-lite will not close the session after issuing the echo command and before the results of echo are shown.$ id uid=777(ben) gid=777(ben) $ cat branches.elt >id +$user=[a-z]+=[0-9]+\(([a-z]+) ? $user == ben ? >echo "howdee ben..." :: >echo "A stranger eh?..." <...Expect-lite supports loops by using labels and jump to label together with conditionals. Because expect-lite starts a bash shell on the remote host, you can also use the shell's conditionals and looping support. Shown below is an expect-lite script that uses bash to perform a loop:
$ cat shell-conditionals.elt >for if in `seq 1 10`; do > echo $if >done >echo ... <...A caveat is that to get at shell variables from expect-lite you have to first
>>echo $bashvarand then read it into an expect-lite variable with something like+$expectvar=.*\n. For instance:$ cat pwd-test.elt >echo $PWD +$expectpwd=\n(.*)\n >echo $expectpwd... <...You can also directly embed expect code into an expect-lite script. This might come in handy if you already have some familiarity with expect and need to perform a more advanced interaction at some stage in the script. Expect-lite lines starting with
!are embedded expect.Needing to specify the host name to expect-lite all the time requires a level of preparation work before one can start using expect-lite. The usage message for expect-lite informs you that you can set
EL_REMOTE_HOSTto define a default value for the host, but if you try this then you will discover thatexpect-lite/read_args{}only works when passed two command-line arguments (version 3.0.5), so if you define a default host with the environment variable, you will need to pass a dummy argument to expect-lite in order for its argument parsing to allow script execution.Ben Martin has been working on filesystems for more than 10 years. He completed his Ph.D. and now offers consulting services focused on libferris, filesystems, and search solutions
About:
Expect-lite is a wrapper for expect, created to make expect programming even easier. The wrapper permits the creation of expect script command files by using special character(s) at the beginning of each line to indicate the expect-lite action. Basic expect-lite scripts can be created by simply cutting and pasting text from a terminal window into a script, and adding '>' 'Release focus: Major feature enhancements
Changes:
The entire command script read subsystem has changed. The previous system read directly from the script file. The new system reads the script file into a buffer, which can be randomly accessed. This permits looping (realistically only repeat loops). Infinite loop protection has been added. Variable increment and decrement have been added to support looping.Author:
Craig Miller [contact developer]
31 Jul 2007 | www.ibm.com/developerworks
If you manage systems and networks, you need Expect.
More precisely, why would you want to be without Expect? It saves hours common tasks otherwise demand. Even if you already depend on Expect, though, you might not be aware of the capabilities described below.
Expect automates command-line interactions
You don't have to understand all of Expect to begin profiting from the tool; let's start with a concrete example of how Expect can simplify your work on AIX® or other operating systems:
Suppose you have logins on several UNIX® or UNIX-like hosts and you need to change the passwords of these accounts, but the accounts are not synchronized by Network Information Service (NIS), Lightweight Directory Access Protocol (LDAP), or some other mechanism that recognizes you're the same person logging in on each machine. Logging in to a specific host and running the appropriate
passwdcommand doesn't take long—probably only a minute, in most cases. And you must log in "by hand," right, because there's no way to script your password?Wrong. In fact, the standard Expect distribution (full distribution) includes a command-line tool (and a manual page describing its use!) that precisely takes over this chore.
passmass(see Resources) is a short script written in Expect that makes it as easy to change passwords on twenty machines as on one. Rather than retyping the same password over and over, you can launchpassmassonce and let your desktop computer take care of updating each individual host. You save yourself enough time to get a bit of fresh air, and multiple opportunities for the frustration of mistyping something you've already entered.This
passmassapplication is an excellent model—it illustrates many of Expect's general properties:
- It's a great return on investment: The utility is already written, freely downloadable, easy to install and use, and saves time and effort.
- Its contribution is "superficial," in some sense. If everything were "by the book"—if you had NIS or some other domain authentication or single sign-on system in place—or even if login could be scripted, there'd be no need for
passmass. The world isn't polished that way, though, and Expect is very handy for grabbing on to all sorts of sharp edges that remain. Maybe Expect will help you create enough free time to rationalize your configuration so that you no longer need Expect. In the meantime, take advantage of it.- As distributed,
passmassonly logs in by way oftelnet,rlogin, orslogin. I hope all current developerWorks readers have abandoned these protocols forssh, whichpassmasssdoes not fully support.- On the other hand, almost everything having to do with Expect is clearly written and freely available. It only takes three simple lines (at most) to enhance
passmassto respectsshand other options.You probably know enough already to begin to write or modify your own Expect tools. As it turns out, the
passmassdistribution actually includes code to log in by means ofssh, but omits the command-line parsing to reach that code. Here's one way you might modify the distribution source to putsshon the same footing astelnetand the other protocols:
Listing 1. Modified passmass fragment that accepts the -ssh argument
...
} "-rlogin" {
set login "rlogin"
continue
} "-slogin" {
set login "slogin"
continue
} "-ssh" {
set login "ssh"
continue
} "-telnet" {
set login "telnet"
continue
...
In my own code, I actually factor out more of this "boilerplate." For now, though, this cascade of tests, in the vicinity of line #100 of
passmass, gives a good idea of Expect's readability. There's no deep programming here—no need for object-orientation, monadic application, co-routines, or other subtleties. You just ask the computer to take over typing you usually do for yourself. As it happens, this small step represents many minutes or hours of human effort saved.
An unreliable program can be controlled from a perl program using the Expect.pm module. A description of the unreliable program and the use of the Expect module is presented.I have a program that I need to run a large number of times. This program has a nasty bug in it. When you feed it bad data, it just sits there forever instead of providing a helpful error message. Bad Program!
I can't change the program, but I need to call this program inside a loop in my code. So I am using the perl Expect module to skip over the problem cases and continue with the rest of the runs of the program.
The Expect.pm module is capable of managing this process, so I wrote a few little test programs to help me understand how to accomplish this task. This document includes the tests along with a few words of explanation. For more documentation about the Expect module, search CPAN.
Automate your repetitive interactionsThe Expect language (an extension of Tcl/Tk, but other variations are also available) is used to write scripts that run sessions with interactive programs, as if the script were a user interacting directly with the program.
Expect scripts can save you a great deal of time, particularly when you find yourself engaging in repetitive tasks. Expect can interact with multiple programs including shells and text-based Web browsers, start remote sessions, and run over the network.
For example, if you frequently connect to a system on your local intranet to run particular program -- the
test-serverscommand, for instance -- you can automate it with an Expect script namedservmaint, whose contents appear in Listing 6.
Listing 6. Sample Expect script to automate remote system program execution
#!/usr/bin/expect -f spawn telnet webserv4 expect "login:" send "joe\r" expect "Password:" send "secret\r" expect "webserv4>$" send "test-servers\r" expect "webserv4>$" send "bye\r" expect eof
Now, instead of going through the entire process of having to runtelnetto connect to the remote system, log in with your username and password, run the command(s) on that system, and then log out. You just run theservmaintscript as given in Listing 6; everything else is done for you. Of course, if you give a password or other proprietary information in such a script, there is a security consideration; at minimum, you should change the file's permissions so that you're the only user (besides the superuser) who can read it.Any repetitive task involving system interaction can be programmed in Expect -- it's capable of branching, conditionals, and all other features of a high-level language so that the response and direction of the interaction with the program(s) can be completely automated.
Multixterm creates multiple xterms that can be driven together or separately. It can be used to login via SSH to multiple hosts and control them simultaneously, or for ad hoc things where you want to see the results as you type. Each xterm may also be driven separately. Multixterm is scriptable so that you can easily fire up, for example, a dozen xterms with a single command, tiled nicely on your screen. In addition to SSH, multixterm can drive rlogin, telnet, passwd, or any program that runs in an xterm.
Expect is a great tool in a system administrators arsenal and can be used to easily automate tasks that require periodic user input. This can allow the administrator to make better use of their time than watching the application or utility to spot the next time it requires input.
In the following example expect is used to automate the inputing of a password for a series of rsync commands tunneled through ssh.
The script automates a series of rsync operations using only the password for access to the remote host so that the security of the two machines is not reduced by making the source machine trust the destination machine in any way (for example .rhosts or a ssh key with an empty pass phrase).
The script reads a password from the user and then holds that password in a variable for use each time the ssh application that rsync is using as a tunnel asks for it.
The "stty -echo" prevents the password from being echoed to the screen when it is typed in and the "stty echo" turns it back on.
#!/usr/bin/expect -f spawn date expect "#" send_user "The password for HOSTNAME: " stty -echo expect_user -re "(.*)\n" {set PASSPH $expect_out(1,string)} send_user "\n" stty echo set timeout -1 spawn date expect "#" spawn rsync -ave ssh --numeric-ids HOSTNAME:/etc /sdc/ expect "password:" { send "$PASSPH\n"} expect "#" spawn date expect "#" spawn rsync -ave ssh --numeric-ids HOSTNAME:/admin /sdc/ expect "password:" { send "$PASSPH\n"} expect "#" spawn date expect "#" spawn rsync -ave ssh --numeric-ids HOSTNAME:/home /sdd expect "password:" { send "$PASSPH\n"} expect "#" spawn date expect "#" spawn rsync -ave ssh --numeric-ids HOSTNAME:/mail /sdd expect "password:" { send "$PASSPH\n"} expect "#" spawn date expect "#" spawn rsync -ave ssh --numeric-ids HOSTNAME:/work /sdc/ expect "password:" { send "$PASSPH\n"} expect "#" spawn date expect "#"(Submitted by Noel Mon Nov 17, 2003 )
Copyright 1999-2003 Noel Davis
All trademarks are the property of their owners.
All articles are owned by their author
Expect plays a crucial role in network management by Cameron Laird
passmass , along with
all the rest of the
example tools and documentation standard for the latest Expect, is available
online. autoexpect, exp_internal,
and interact, mostly—that are specific to Expect. Don Libes -- an extremely interesting page of the famous author of Expect
Rscript is based upon Expect/Tcl and allows you to automate remote logins and execute remote commands, somewhat like rsh.
Copyright © 1996-2007 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: April 01, 2008