Tag: expect

Expect script suppress output

You might write your program like this:

#!/bin/sh
output=$(expect -c '
# suppress the display of the process interaction
log_user 0

spawn telnet '"$HOST $PORT"'
sleep 1
send "\r"
send "\r"
# after a prompt, send the interesting command
expect Prompt> { send "dir\r"  }
# eat the \n the remote end sent after we sent our \r
expect "\n"
# wait for the next prompt, saving its position in expect_out(buffer)
expect -indices Prompt>

# output what came after the command and before the next prompt
# (i.e. the output of the "dir" command)
puts [string range $expect_out(buffer) \
                   0 [expr $expect_out(0,start) - 1]]
')
echo "======="
echo "$output"
echo "======="

Remote ls output is not redirecting to file

When I run the below code, I’getting this error

bash: /var/out.txt: No such file or directory

#!/usr/bin/expect

set timeout -1    
spawn ssh user@10.103.234.1 'ls -t /var/backups/archives/' > /var/outp.log    
expect "user@10.103.234.1's password:"    
send "Password\n"    
expect eof

if [catch wait] {    
    puts "failed"    
    exit 1    
}

exit 0

Solution:

Expect/Tcl does not understant the redirection (>) char. Try this:

spawn bash -c "ssh user@10.103.234.1 ls -t /var/backups/archives/ > /var/outp.log"

SCP getting hang

While executing the below code. the script gets hanged. /var/backups/archives/ contains about 3 files with around 5GB in size. It copies the first file and scripts hangs.

spawn scp -rp user@10.10.234.1:/var/backups/archives/ /var/backup
expect "user@10.10.234.1's password:"
send "password\n"
expect eof
if [catch wait] {
    puts "failed"
    exit 1
}
exit 0

Solution:

I assume you’ll see expect: timed out when you run your script with expect -d.

Insert set timeout 60 in a new line after send "password\n". Replace 60 with a number greater than the transfer time (in seconds) of all files.

SFTP Manual command succeed but script is failing at 1 or 2%

I wrote a script to transfer a folder from VPS to Synology NAS Server. The script fails at 1 or 2% transfer without any indication why. When I run the command manually it works fine without any issues and it transfer the folder flawlessly. I’m not sure what is causing this and would appreciate if i can get some guidance on where to look.

SCRIPT:

#!/usr/bin/expect
set dir [timestamp -format "%Y-%m-%d"]
spawn sftp user@host.synology.me
expect "Password:"
send "passw0rd\n"
expect "sftp>"
send "cd /directory/\n"
expect "sftp>"
send "put -r /home/backup/$dir\n"
expect "sftp>"
send "bye\n"

RESULT:

root@host [/scripts]# ./sftp.sh
spawn sftp user@host.synology.me
user@host.synology.me's password:
Connected to host.synology.me
sftp> cd /directory/for/backup
sftp> put -r /home/backup/2017-12-26
Uploading /home/backup/2017-12-26/ to /backup/2017-12-26
Entering /home/backup/2017-12-26/
Entering /home/backup/2017-12-26/accounts
/home/backup/2017-12-26/file1.tar.gz       2%   26MB   1.8MB/s   08:41 ETAroot@host [/scripts]#
root@host [/scripts]#

MANUAL COMMAND:

root@host [/scripts]# sftp user@host.synology.me
backup@host.synology.me's password:
Connected to host.synology.me.
sftp> cd directory
sftp> put -r /home/backup/2017-12-26
Uploading /home/backup/2017-12-26/ to /backups/2017-12-26
Entering /home/backup/2017-12-26/
Entering /home/backup/2017-12-26/accounts
/home/backup/2017-12-26/file.tar.gz             100%  959MB 820.2KB/s   19:57
/home/backup/2017-12-26/file2.tar.gz            100%  147MB   1.4MB/s   01:45
sftp> bye
root@host [/bwbScripts]#

Solution:

I guess, in your case, timeout happened with respect to Expect, since the file transfer takes some extra time than the default timeout (10seconds). You can either set the timeout to -1 or you can increase it to some higher value and the they are in the units of seconds. 

set timeout -1; # Infinite amount of waiting time

or 

set timeout 300; # 5 minutes of waiting time

You can keep the above statement in your code.

send "put -r /home/backup/$dir\n"
expect {
    timeout {puts "timeout happened"}
    "sftp>"
}
send "bye\r"
expect eof

Always use \r character to send ‘Enter’ key instead of \n. The last statement expect eof will wait for the graceful closure of sftp program.

Hashicorp Consul Installation on Centos 7

Consul must first be installed on your machine. Consul is distributed as a binary package for all supported platforms and architectures. This page will not cover how to compile Consul from source, but compiling from source is covered in the documentation for those who want to be sure they’re compiling source they trust into the final binary.

Installing Consul

To install Consul, find the appropriate package for your system and download it. Consul is packaged as a zip archive.

After downloading Consul, unzip the package. Consul runs as a single binary named consul. Any other files in the package can be safely removed and Consul will still function.

The final step is to make sure that the consul binary is available on the PATH. See this page for instructions on setting the PATH on Linux and Mac. This page contains instructions for setting the PATH on Windows.

Verifying the Installation

After installing Consul, verify the installation worked by opening a new terminal session and checking that consul is available. By executing consul you should see help output similar to this:

$ consul
usage: consul [--version] [--help]  []

Available commands are:
    agent          Runs a Consul agent
    event          Fire a new event

# ...

If you get an error that consul could not be found, your PATH environment variable was not set up properly. Please go back and ensure that your PATH variable contains the directory where Consul was installed.

Run the Consul Agent

After Consul is installed, the agent must be run. The agent can run either in server or client mode. Each datacenter must have at least one server, though a cluster of 3 or 5 servers is recommended. A single server deployment is highlydiscouraged as data loss is inevitable in a failure scenario.

All other agents run in client mode. A client is a very lightweight process that registers services, runs health checks, and forwards queries to servers. The agent must be running on every node that is part of the cluster.

For more detail on bootstrapping a datacenter, see this guide.

Starting the Agent

For simplicity, we’ll start the Consul agent in development mode for now. This mode is useful for bringing up a single-node Consul environment quickly and easily. It is not intended to be used in production as it does not persist any state.

-$ consul agent -dev
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
           Version: 'v0.7.0'
         Node name: 'Armons-MacBook-Air'
        Datacenter: 'dc1'
            Server: true (bootstrap: false)
       Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas: 

==> Log data will now stream in as it occurs:

    2016/09/15 10:21:10 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:127.0.0.1:8300 Address:127.0.0.1:8300}]
    2016/09/15 10:21:10 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2016/09/15 10:21:10 [INFO] serf: EventMemberJoin: Armons-MacBook-Air 127.0.0.1
    2016/09/15 10:21:10 [INFO] serf: EventMemberJoin: Armons-MacBook-Air.dc1 127.0.0.1
    2016/09/15 10:21:10 [INFO] consul: Adding LAN server Armons-MacBook-Air (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2016/09/15 10:21:10 [INFO] consul: Adding WAN server Armons-MacBook-Air.dc1 (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2016/09/15 10:21:13 [DEBUG] http: Request GET /v1/agent/services (180.708µs) from=127.0.0.1:52369
    2016/09/15 10:21:13 [DEBUG] http: Request GET /v1/agent/services (15.548µs) from=127.0.0.1:52369
    2016/09/15 10:21:17 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2016/09/15 10:21:17 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2016/09/15 10:21:17 [DEBUG] raft: Votes needed: 1
    2016/09/15 10:21:17 [DEBUG] raft: Vote granted from 127.0.0.1:8300 in term 2. Tally: 1
    2016/09/15 10:21:17 [INFO] raft: Election won. Tally: 1
    2016/09/15 10:21:17 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2016/09/15 10:21:17 [INFO] consul: cluster leadership acquired
    2016/09/15 10:21:17 [DEBUG] consul: reset tombstone GC to index 3
    2016/09/15 10:21:17 [INFO] consul: New leader elected: Armons-MacBook-Air
    2016/09/15 10:21:17 [INFO] consul: member 'Armons-MacBook-Air' joined, marking health alive
    2016/09/15 10:21:17 [INFO] agent: Synced service 'consul'

As you can see, the Consul agent has started and has output some log data. From the log data, you can see that our agent is running in server mode and has claimed leadership of the cluster. Additionally, the local member has been marked as a healthy member of the cluster.

You can start the consul in the server mode by executing the following command:
consul agent -data-dir=/opt/consul/data -bind=192.168.1.15 -server -bootstrap-expect 1 -ui -client 0.0.0.0 &

-data-dir – This flag provides a data directory for the agent to store state. This is required for all agents. The directory should be durable across reboots. This is especially critical for agents that are running in server mode as they must be able to persist cluster state.

-bind – The address that should be bound to for internal cluster communications. This is an IP address that should be reachable by all other nodes in the cluster. By default, this is “0.0.0.0”, meaning Consul will bind to all addresses on the local machine and will advertise the first available private IPv4 address to the rest of the cluster. If there are multiple private IPv4 addresses available, Consul will exit with an error at startup.

-server – This flag is used to control if an agent is in server or client mode. When provided, an agent will act as a Consul server.

-bootstrap-expect – This flag provides the number of expected servers in the datacenter. Either this value should not be provided or the value must agree with other servers in the cluster. When provided, Consul waits until the specified number of servers are available and then bootstraps the cluster.

-ui – Enables the built-in web UI server and the required HTTP routes. This eliminates the need to maintain the Consul web UI files separately from the binary.

-client – The address to which Consul will bind client interfaces, including the HTTP and DNS servers. By default, this is “127.0.0.1”, allowing only loopback connections.

Now you can access the ui by hitting the following in the browser:
http://{IP_of_consul_server}:8500/ui

You can create a configuration file as follows if you do not want to use the command line args to start consul:
Create a file /etc/consul/consul.json with the following content:

{
  "data_dir": "/opt/consul/data",
  "server": true,
  "bind_addr": "192.168.1.15",
  "bootstrap_expect": 1,
  "ui": true,
  "client_addr": "0.0.0.0"
}

Then start consul with:

consul agent -config-file=/etc/consul/consul.json &

Thus we have installed and configured consul.

If you want to configure vault with consul then you can use the following hcl:

storage “consul” {
address = “192.168.1.15:8500”
path = “vault”
}
listener “tcp” {
address = “0.0.0.0:8200”
tls_disable = 1
}