Archive for the ‘ssh’ Tag

SSHing efficiently   6 comments

I personally have a numerous number of hosts which I sometimes have to SSH to. It can get rather confusing and inefficient if you get lost among them.

I’m going to show you here how you can get your SSHing to be heaps more efficient with just 5 minutes of your time.

.ssh/config

In $HOME/.ssh/config I usually store all my hosts in such a way:

Host host1
    Port 1234
    User root
    HostName host1.potentially.very.long.domain.name.com

Host host2
    Port 5678
    User root
    HostName host2.potentially.very.long.domain.name.com

Host host3
    Port 9012
    User root
    HostName host3.potentially.very.long.domain.name.com

You obviously got the idea. So if I’d like to ssh to host2, all I have to do is:

ssh host2

That will ssh to root@host2.potentially.very.long.domain.name.com:5678 – saves a bit of time.

I usually manage all of my hosts in that file. Makes life simpler, even use git if you feel like it…

Auto complete

I’ve added to my .bashrc the following:

_ssh_hosts() {
    local cur="${COMP_WORDS[COMP_CWORD]}"
    COMPREPLY=()
    local ssh_hosts=`grep ^Host ~/.ssh/config | cut -d' ' -f2 | xargs`
    [[ ! ${cur} == -* ]] && COMPREPLY=( $(compgen -W "${ssh_hosts}" -- ${cur}) )
}

complete -o bashdefault -o default -o nospace -F _ssh_hosts ssh 2>/dev/null \
    || complete -o default -o nospace -F _ssh_hosts ssh
complete -o bashdefault -o default -o nospace -F _ssh_hosts scp 2>/dev/null \
    || complete -o default -o nospace -F _ssh_hosts scp

Sweet. All that you have to do now is:

$ ssh TAB TAB
host1 host2 host3

We are a bit more efficient today.

Posted March 31, 2013 by malkodan in Bash, Linux, System Administration

Tagged with , , ,

NetworkManager-ssh   3 comments

SSH is amazing

Show me one unix machine today without SSH. It’s everywhere, for a reason.
OpenSSH specifically allows you to do so much with it. What would we have done without SSH?

OpenSSH Tunnelling and full VPN

Tunnelling with SSH is really cool, utilizing the secure SSH connection you can virtually secure any TCP/IP connection using port forwarding (-R and -L):
http://www.openssh.org/faq.html#2.11

However for full VPN support, you can use -w which opens a tun/tap device on both ends of connection, allowing you potentially to have all of your network passing via your SSH connection. In other words – full VPN support for free!!!

Server configuration

On the server, the configuration would be minimal:

  • Allow tunnelling in sshd configuration
  • echo 'PermitTunnel=yes' >> /etc/ssh/sshd_config
    service sshd reload
    
  • Allow forwarding
  • -I FORWARD -i tun+ -j ACCEPT
    -I FORWARD -o tun+ -j ACCEPT
    -I INPUT -i tun+ -j ACCEPT
    -I POSTROUTING -o EXTERNAL_INTERFACE -j MASQUERADE
    echo 1 > /proc/sys/net/ipv4/ip_forward
    

That’s all!! Congratulations on your new VPN server!!

Client configuration (your personal linux machine)

These 2 commands will configure you with a very simple VPN (run as root!!!):

ssh -f -v -o Tunnel=point-to-point \
  -o ServerAliveInterval=10 \
  -o TCPKeepAlive=yes \
  -w 100:100 root@YOUR_SSH_SERVER \
  '/sbin/ifconfig tun100 172.16.40.1 netmask 255.255.255.252 pointopoint 172.16.40.2' && \
/sbin/ifconfig tun100 172.16.40.2 netmask 255.255.255.252 pointopoint 172.16.40.1

The only downside of this awesome VPN is that you have to be root on both ends.
But this whole setup is rather clumsy, lets use some UI for that, no?

NetworkManager-ssh

Somewhere in time, after intensively working in a company dealing with VPNs (but no SSH VPNs at all) I was looking at my taskbar at NetworkManager and thinking “Hey! There’s an OpenVPN, PPTP and IPSEC plugin for NetworkManager, why not build a SSH VPN plugin?”
And hell, why not?
I started searching the Internet frantically, believing that someone already implemented that ingenious idea (like most good ideas), but except for one mailing list post from a few years ago where someone suggested to implement it – nada.

Guess it’s my prime time. Within a week of forking the code of NetworkManager-openvpn (the NetworkManager OpenVPN plugin) I managed to get something that actually works (ssh-agent authentication only). I was surprised because I’ve never dealt with glib/gtk infrastructure not to mention UI programming (I’m a pure backend/infrastructure developer for the most of it).

And today?

I’m writing this post perhaps 2 months after I started development and committed my first alpha release. While writing this post I’m trying to submit NetworkManager-ssh to fedora (fedora-extras to be precise).

Getting into the bits and bytes behind it is redundant, all that you have to know is that the source is available here:
https://github.com/danfruehauf/NetworkManager-ssh
It compiles easily into a RPM or DEB for your convenience. I urge you to give it a shot and please open me issues on github if you find any.

Posted March 23, 2013 by malkodan in C++, Linux

Tagged with , , , , , , , , ,

Creating a puppet ready image (CentOS/Fedora)   10 comments

Cloud computing and being lazy

The need to create template images in our cloud environment is obvious. Especially with Amazon EC2 offering an amazing API and spot instances in ridiculously low prices.
In the following post I’ll show what I am doing in order to prepare a “puppet-ready” image.

Puppet for the rescue

In my environment I have puppet configured and provisioning any of my machines. With puppet I can deploy anything I need – “if it’s not in puppet – it doesn’t exist”.
Coupled with Puppet dashboard the interface is rather simple for manually adding nodes. But doing stuff manually is slow. I assume that given the right base image I (and you) can deploy and configure that machine with puppet.
In other words, the ability to convert a bare machine to a usable machine is taken for granted (although it is heaps of work on its own).

Handling the “bare” image

Most cloud computing providers today provide you (usually) with an interface for starting/stopping/provisioning machines on its cloud.
The images the cloud providers are usually supplying are bare, such as CentOS 6.3 with nothing. Configuring an image like that will require some manual labour as you can’t even auto-login to it without some random password or something similar.

Create a “puppet ready” image

So if I boot up a simple CentOS 6.x image, these are the steps I’m taking in order to configure it to be “puppet ready” (and I’ll do it only once per cloud computing provider):

# install EPEL, because it's really useful
rpm -q epel-release-6-8 || rpm -Uvh http://download.fedoraproject.org/pub/epel/6/`uname -i`/epel-release-6-8.noarch.rpm

# install puppet labs repository
rpm -q puppetlabs-release-6-6 || rpm -ivh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-6.noarch.rpm

# i usually disable selinux, because it's mostly a pain
setenforce 0
sed -i -e 's!^SELINUX=.*!SELINUX=disabled!' /etc/selinux/config

# install puppet
yum -y install puppet

# basic puppet configuration
echo '[agent]' > /etc/puppet/puppet.conf
echo '  pluginsync = true' >> /etc/puppet/puppet.conf
echo '  report = true' >> /etc/puppet/puppet.conf
echo '  server = YOUR_PUPPETMASTER_ADDRESS' >> /etc/puppet/puppet.conf
echo '  rundir = /var/run/puppet' >> /etc/puppet/puppet.conf

# run an update
yum update -y

# highly recommended is to install any package you might deploy later on
# the reason behind it is that it will save a lot of precious time if you
# install 'httpd' just once, instead of 300 times, if you deploy 300 machines
# also recommended is to run any 'baseline' configuration you have for your nodes here
# such as changing SSH port or applying common firewall configuration for instance
yum install -y MANY_PACKAGES_YOU_MIGHT_USE

# and now comes the cleanup phase, where we actually make the machine "bare", removing
# any identity it could have

# set machine hostname to 'changeme'
hostname changeme
sed -i -e "s/^HOSTNAME=.*/HOSTNAME=changeme" /etc/sysconfig/network

# remove puppet generated certificates (they should be recreated)
rm -rf /etc/puppet/ssl

# stop puppet, as you should change the hostname before it will be permitted to run again
service puppet stop; chkconfig puppet off

# remove SSH keys - they should be recreated with the new machine identity
rm -f /etc/ssh/ssh_host_*

# finally add your key to authorized_keys
mkdir -p /root/.ssh; echo "YOUR_SSH_PUBLIC_KEY" > /root/.ssh/authorized_keys

Power off the machine and create an image. This is your “puppet-ready” image.

Using the image

Now you’re good to go, create a new image from that machine and any machine you’re going to create in the future should be based on that image.

When creating a new machine the steps you should follow are:

  • Start the machine with the “puppet-ready” image
  • Set the machine’s hostname
  • hostname=uga.bait.com
    hostname $hostname
    sed -i -e "s/^HOSTNAME=.*/HOSTNAME=$hostname/" /etc/sysconfig/network
    
  • Run ‘puppet agent –test’ to generate a new certificate request
  • Add the puppet configuration for the machine, for puppet dashboard it’ll be something similar to:
  • hostname=uga.bait.com
    sudo -u puppet-dashboard RAILS_ENV=production rake -f /usr/share/puppet-dashboard/Rakefile node:add name=$hostname
    sudo -u puppet-dashboard RAILS_ENV=production rake -f /usr/share/puppet-dashboard/Rakefile node:groups name=$hostname groups=group1,group2
    sudo -u puppet-dashboard RAILS_ENV=production rake -f /usr/share/puppet-dashboard/Rakefile node:parameters name=$hostname parameters=parameter1=value1,parameter2=value2
    
  • Authorize the machine in puppetmaster (if autosign is disabled)
  • Run puppet:
    # initial run, might actually change stuff
    puppet agent --test
    service puppet start; chkconfig puppet on
    

This is 90% of the work if you want to quickly create usable machines on the fly, it shortens the process significantly and can be easily implemented to support virtually any cloud computing provider!

I personally have it all scripted and a new instance on EC2 takes me 2-3 minutes to load + configure. It even notifies me politely via email when it’s done.

I’m such a lazy bastard.

Posted March 23, 2013 by malkodan in Bash, Linux, System Administration

Tagged with , , , , , , , , , , ,

Bash RPC   2 comments

Auto configuring complex cluster architectures is a task many of you might probably skip, using the excuse that it’s a one time task and it’ll never repeat itself. WRONG!
Being lazy as I could and the need to quickly deploy systems for hungry customers, I started out with a small little infrastructure that helped me along the way to auto configure clusters of two nodes or more.
My use cases were:
1. RedHat cluster
2. LVS
3. Heartbeat
4. Oracle RAC

Auto configuring an Oracle RAC DB is not an easy task at all. However, with the proper infrastructure, it can become noticeably easier.
The common denominators for all cluster configurations I had to carry were:
1. They had to run on more than one node
2. Except from entering a password once for all nodes, I didn’t want any interaction
3. They all consisted from steps that should either run on all nodes, or on just one node
4. Sometimes you could logically split the task into a few phases of configuration, making it easier to comprehend the tasks you have to achieve

Even though it is not the most tidy piece of Bash code I’m going to post here, I’m very proud of it as it saved me countless hours. I give you the skeleton of code, which is the essence of what I nicknamed Bash RPC. On top of this you should be able to easily auto configure various configurations involving more than one computer.

Using it

The sample attached is a simple script that should be able to bake /home/cake on all relevant nodes.
In order to use the script properly, edit it using your favorite editor and stroll through the configuration_vars() function. Populate the array HOST_IP with your relevant hosts.
Now you can simply run the script.
I’m aware to the slight disadvantage that you can’t have your configuration come from command line, on the other hand – when dealing with big, complex, do you really think your configuration can be defined in a single line of arguments?

Taming it

OK, this is the interesting part. Obviously no one needs to bake cakes on his nodes, it is truly pointless and was merely given as an example.
So how would you go about customizing this skeleton to your needs?
First and foremost, we must plan. Planning and designing is the key for every tech related activity you carry. Be it a SysAdmin task or a pure development task. While configuring your cluster for the first time, keep notes of the steps (and commands) you have to go through. Later on, try to logically separate the steps into phases. When you have them all, we can start hacking Bash RPC.
Start drawing your phases and steps, using functions in the form of PHASEx_STEPx.
Fill up your functions with your ideas and start testing! and that’s it!

How does it work?

Simplicity is the key for everything.
Bash RPC can be ran in 2 ways – either running all phases (full run) or running just one step.
If you give Bash RPC just one argument, it assumes it is a function you have to run. If no arguments are given it will run the whole script.
Have a look at run_function_on_node(). This function receives a node and functions it should run on. It will copy the script to the destination node and initiate it with the arguments it received.
And this is more or less the essence of Bash RPC. REALLY!

Oh, there’s one small thing. Sometimes you have to run things on just one host, in that case you can add a suffix of ___SINGLE_HOST for your steps. This will make sure the step will run on just one host (the first one you defined).

I’m more than aware that this skeleton of Bash RPC can be polished some more and indeed I have a list of TODOs for this skeleton. But all in all – I really think this one is a big time saver.

Real world use cases

I consider this script a success mainly because of 2 real world use cases.
The first one is the act of configuring from A to Z Oracle RAC. Those of you who had to go through this nightmare can testify that configuring Oracle RAC takes 2-3 days (modest estimation) of both a DBA and SysAdmin working closely together. How about an unattended script running in the background, notifying you 45 minutes later you have an Oracle RAC ready for service?

The second use case is my good friend and former colleague, Oren Held. Oren could easily take this skeleton and use it for auto configuring a totally different cluster using LVS over Heartbeat. He was even satisfied while using it. Oren never consulted me while performing this task – and this is the great achievement.

I hope you could use this code snippet and customize it for your own needs, continuing with the YOU CAN CREATE A SCRIPT FOR EVERYTHING attitude!!

Have a look at cluster-config-cake.sh to get an idea about how it’s being done.

Posted October 10, 2009 by malkodan in Bash, Linux, System Administration

Tagged with , , , , , , , , , ,

Buying time   1 comment

Wearing both hats of developer and SysAdmin, I believe you shouldn’t work hard as a SysAdmin. It is for a reason that sometimes developers look at SysAdmins as inferior. It’s not because SysAdmins are really inferior, or has an easier job. I think it is mainly because a large portion of their workload could be automated. And if it can’t be automated (very little things), you should at least be able to do it quickly.

Obviously, we cannot buy time, but we can at least be efficient. In the following post I’ll introduce a few of my most useful aliases (or functions) I’d written and used in the last few years. Some of them are development related while the others could make you a happy SysAdmin.
You may find them childish – but trust me, sometimes the most childish alias is your best friend.

Jumping on the tree

Our subversion trunk really reminds me of a tree sometimes. The trunk is very thick, but it has many branches and eventually many leaves. While dealing with the leaves is rare, jumping on the branches is very common. Many times i have found myself typing a lot of ‘cd’ commands, where some are longer than others, repeatedly, just to get to a certain place. Here my stupid aliases come to help me.

lib takes me straight away to our libraries sub directory, where sim takes me to the simulators sub directory. Not to mention tr (shortcut for trunk), which takes me exactly to the sub directory where the sources are checked out. Oh, and pup which takes me to my puppet root, on my puppet master. Yes, I’m aware to the fact that you probably wonder now “Hey, is this guy going to teach me something new today? – Aliases are for babies!!”, I can identify. I didn’t come to teach you how to write aliases, I’m here to preach you to start using aliases. Ask yourself how many useful day-to-day aliases you really have defined. Do you have any at all? – Don’t be shy to answer no.

*nix jobs are diverse and heterogeneous, but let’s see if I can encourage you to write some useful aliases after all.
In case you need some ideas for aliases, run the following:

$ history | tr -s ' ' | cut -d' ' -f3- | sort | uniq -c | sort -n

Yes, this will show the count of the most recent commands you have used. Still not getting it? – OK, I’ll give you a hint, it should be similar to this:

$ alias recently_used_commands="history | tr -s ' ' | cut -d' ' -f3- | sort | uniq -c | sort -n"

If you did it – you’ve just kick-started your way to liberation. Enjoy.
As a dessert – my last childish alias:

$ alias rsrc='source ~/.bashrc'

Always useful if you want to re-source your .bashrc while working on some new aliases.

Two more things I must mention though:

  1. Enlarge your history size, I guess you can figure out alone how to do it.
  2. If you’re feeling generous – periodically collect the history files from your fellow team members (automatically of course, with another alias) and create aliases that will suit them too.

The serial SSHer

Our network is on 192.168.8.0/24. Many times I’ve found myself issuing commands like:

$ ssh root@192.168.8.1
$ ssh root@192.168.8.2
$ ssh root@192.168.8.3

Dozens of these in a single day. It was really frustrating. One day I decided to make an end to it:

# function for easier ssh
# $1 - network
# $2 - subnet
# $3 - host in subnet
_ssh_specific_network() {
	local network=$1; shift
	local subnet=$1; shift
	local host=$1; shift
	ssh root@$network.$subnet.$host
}

# easy ssh to 192.168.8.0/24
# $1 - host
_ssh_net_192_168_8() {
	local host=$1; shift
	_ssh_specific_network 192.168 8 $host
}
alias ssh8='_ssh_net_192_168_8'

Splendid, now I can run the following:

$ ssh8 1

Which is equal to:

$ ssh root@192.168.8.1

Childish, but extremely efficient. Do it also for other commands you would like to use, such as ping, telnet, rdesktop and many others.

The cop

Using KDE’s konsole? – I really like DCOP, let’s add some spice to the above function, we’ll rename the session name to the host we’ll ssh to, and then restore the session name back after logging out:

# returns session name
_konsole_get_session_name() {
	# using dcop - obtain session name
	dcop $KONSOLE_DCOP_SESSION sessionName
}

# renames a konsole session
# $1 - session name
_konsole_rename_session_name() {
	local session_name=$1; shift
	_konsole_store_session_name `_konsole_get_session_name`
	dcop $KONSOLE_DCOP_SESSION renameSession "$session_name"
}

# store the current session name
_konsole_store_session_name() {
	STORED_SESSION_NAME=`_konsole_get_session_name`
}

# restores session name
_konsole_restore_session_name() {
	if [ x"$STORED_SESSION_NAME" != x ]; then
		_konsole_rename_session_name "$STORED_SESSION_NAME"
	fi
}

# function for easier ssh
# $1 - network
# $2 - subnet
# $3 - host in subnet
_ssh_specific_network() {
	local network=$1; shift
	local subnet=$1; shift
	local host=$1; shift
	# rename the konsole session name
	_konsole_rename_session_name .$subnet.$host
	ssh root@$network.$subnet.$host
	# restore the konsole session name
	_konsole_restore_session_name
}

Extend it as needed, this is only the tip of the iceberg! I can assure you that my aliases are much more complex than these.

Finders keepers

For the next one I’m not taking full credit – this one belongs to Uri Sivan – obviously one of the better developers I’ve met along the way.
Grepping cpp files is essential, many times I’ve found myself looking for a function reference on all of our cpp files.
The following usually does it:

$ find . -name "*.cpp" | xargs grep -H -r 'HomeCake'

But seriously, do I look like someone that likes to work hard?

# $* - string to grep
grepcpp() {
	local grep_string="$*";
	local filename=""
	find . -name "*.cpp" -exec grep -l "$grep_string" "{}" ";" | while read filename; do
		echo "=== $filename"
		grep -C 3 --color=AUTO "$grep_string" "$filename"
		echo ""
	done
}

OK, let’s generalize it:

# grepping made easy, taken from the suri
# grepext - grep by extension
# $1 - extension of file
# $* - string to grep
_grepext() {
	local extension=$1; shift
	local grep_string="$*"
	local filename=""
	find . -name "*.${extension}" -exec grep -l "$grep_string" "{}" ";" | while read filename; do
		echo "=== $filename"
		grep -C 3 --color=AUTO "$grep_string" "$filename"
		echo ""
	done
}

# meta generate the grepext functions
declare -r GREPEXT_EXTENSIONS="h c cpp spec vpj sh php html js"
_meta_generate_grepext_functions() {
	local tmp_grepext_functions=`mktemp`
	local extension=$1; shift
	for extension in $GREPEXT_EXTENSIONS; do
		echo "grep$extension() {" >> $tmp_grepext_functions
		echo '  local grep_string=$*' >> $tmp_grepext_functions
		echo '  _grepext '"$extension"' "$grep_string"' >> $tmp_grepext_functions
		echo "}" >> $tmp_grepext_functions
	done
	source $tmp_grepext_functions
	rm -f $tmp_grepext_functions
}

After this, you have all of your C++/Bash/PHP/etc developers happy!

Time to showoff

My development environment is my theme park, here is my proof:

$ (env; declare -f; alias) | wc -l
2791

I encourage you to run this as well, if my line count is small and I’m bragging about something I shouldn’t – let me know!

Posted August 21, 2009 by malkodan in Bash, Linux, System Administration

Tagged with , , , , , , , , ,

Design a site like this with WordPress.com
Get started