Archive for the ‘Bash’ Tag
Imagine you are in Tasmania and need to move 35TB (1 million files) to S3 in the Sydney region. The link between Tasmania and continental Australia will undergo maintenance in the next month, which means either one or both:
- You cannot use network links to transfer the data
- Tasmania might be drifting further away from the mainland now that it is untethered
In short, I’m going to be presented with a bunch of HDs and I need to copy the data on them, fly to Sydney and upload the data to S3. If the HD given would be 35TB I could just copy the data and be done with it – no dramas. Likely though, the HDs will be smaller than 35TB, so I need to look at a few options of doing that.
Things to consider are:
- Files should be present on the HDs in their original form – so they can be uploaded to S3 directly without needing a staging space for unzipping etc
- HDs should be accessible independently, in case a HD is faulty I can easily identify what files need copying again
- Copy operation should be reproducible, so previous point could be satisfied if anything goes wrong in the copying process
- Copying should be done in parallel (it’s 35TB, it’ll take a while)
- It has to be simple to debug if things go wrong
LVM/ZFS over a few HDs
Building a larger volume over a few HDs require me to connect all HDs at the same time to a machine and if any of them fail I will lose all the data. I decide to not do that – too risky. It’ll also be difficult to debug if anything goes wrong.
tar | split
Not a bad option on its own. An archive can be built and split into parts, then the parts could be copied onto the detination HDs. But the lose of a single HD will prevent me from copying the files on the next HD.
tar also supports -L (tape length) and can potentially split the backup on its own without the use of split. Still, it’ll take a very long time to spool it to multiple HDs as it wouldn’t be able to do it in parallel. In addition, I’ll have to improvise something for untarring and uploading to S3 as I will have no staging area to untar those 35TB. I’ll need something along the lines of tar -O -xf ... | s3cmd.
tar also has an interesting of -L (tape length), which will split a volume to a few tapes. Can’t say I am super keen using it. It has to work the first time.
Span Files
I decided to write a utility that’ll do what I need since there’s only one chance of getting it right – it’s called span-files.sh. It operates in three phases:
- index – lists all files to be copied and their sizes
- span – given a maximum size of a HD, iterate on the index and generate a list of files to be copied per HD
- copy – produces
rsync --files-from=list.X commands to run per HD. They can all be run in parallel if needed
The utility is available here:
https://github.com/danfruehauf/Scripts/tree/master/span-files
I’ll let you know how it all went after I do the actual copy. I still wonder whether I forgot some things…
Every so often I come across Bash scripts which are written as if Bash is a pile of rubbish and you just have to mould something ugly with it.
True, Bash is supposedly not the most “powerful” scripting language out there, but on the other hand if you’re using traditional methods then you can avoid installing gazillion ruby gems or perl/python modules (probably not even using RPM or DEB!!) just to configure your system. Bash is simple and can be elegant. But that’s not the point.
The point is that too often Bash scripts which people write have zero maintainability and readability. Why is that??
I’m not going to point at any bad examples because that’s not a very nice thing to do, although I can and easily.
Please do follow these three simple guidelines and you’ll get 90% of the job done in terms of maintainability and readability:
- Functions – Write code in functions. Break your code into manageable pieces, like any other programming language, ey?
- Avoid global variables – Global variables just make it all too complicated to follow what’s going on where. Sometimes they are needed but you can minimize the use of them.
- INDENTATION – INDENT YOUR BLOODY CODE. If you have an if or for or what not, please just indent the block under it. It’s that simple and makes your code so much more readable.
That was my daily rant.
My Bash coding (or scripting) conventions cover a bit more and can be found here:
https://github.com/danfruehauf/Scripts/tree/master/bash_scripting_conventions
The Assignment
You have a directory with gazillion files. Since most filesystems are not very efficient with many files in one directory, it is advisable to spread them among a hierarchy of directories. Write a program (or script) which handles a directory with many files and spreads them in an efficient hierarchy.
Does that sounds like a University assignment or something? Yes, it does.
Well apparently such a situation just happened to me in real life. Searching across the internet I couldn’t find anything too useful. And I will stand corrected if there is something which already deals with that problem. Post ahead if so.
And yes, thank god I’m using Unix (Linux), don’t even want to think what one would do on Windows.
The Situation
An application was spooling many files to the same directory, generating up to a million files in the same directory. I’m sorry I cannot disclose any more information about it, but lets just say it is a well known open source application.
Access to these files was obviously fast having ext4 and dir_index, but the directory index is too big to actually list files or do anything else without clogging everything in the system. And we need these files.
So we’ve decided to model the files in a way that’ll be more efficient for browsing and we can then handle it from there.
The Solution
After implementing something pretty quick and dirty for the situation, to mitigate the pain, I’ve sat down and wrote something a bit more generic. I’m happy to introduce the spread_files.sh utility.
What does it take care of:
- Reading the directory index just once
- Hierarchy depth as parameter
- Stacking up to X files per mv command
- Has recursion in Bash!!
Obviously the best solution would be to never get to that situation, however if you do, feel free to use spread_files.sh.
I personally have a numerous number of hosts which I sometimes have to SSH to. It can get rather confusing and inefficient if you get lost among them.
I’m going to show you here how you can get your SSHing to be heaps more efficient with just 5 minutes of your time.
.ssh/config
In $HOME/.ssh/config I usually store all my hosts in such a way:
Host host1
Port 1234
User root
HostName host1.potentially.very.long.domain.name.com
Host host2
Port 5678
User root
HostName host2.potentially.very.long.domain.name.com
Host host3
Port 9012
User root
HostName host3.potentially.very.long.domain.name.com
You obviously got the idea. So if I’d like to ssh to host2, all I have to do is:
ssh host2
That will ssh to root@host2.potentially.very.long.domain.name.com:5678 – saves a bit of time.
I usually manage all of my hosts in that file. Makes life simpler, even use git if you feel like it…
Auto complete
I’ve added to my .bashrc the following:
_ssh_hosts() {
local cur="${COMP_WORDS[COMP_CWORD]}"
COMPREPLY=()
local ssh_hosts=`grep ^Host ~/.ssh/config | cut -d' ' -f2 | xargs`
[[ ! ${cur} == -* ]] && COMPREPLY=( $(compgen -W "${ssh_hosts}" -- ${cur}) )
}
complete -o bashdefault -o default -o nospace -F _ssh_hosts ssh 2>/dev/null \
|| complete -o default -o nospace -F _ssh_hosts ssh
complete -o bashdefault -o default -o nospace -F _ssh_hosts scp 2>/dev/null \
|| complete -o default -o nospace -F _ssh_hosts scp
Sweet. All that you have to do now is:
$ ssh TAB TAB
host1 host2 host3
We are a bit more efficient today.
Have decided to publish the infamous Bash scripting conventions.
Here they are:
https://github.com/danfruehauf/Scripts/tree/master/bash_scripting_conventions
Please, comment, challenge and help me modify it. I’m very open for feedback.
Auto configuring complex cluster architectures is a task many of you might probably skip, using the excuse that it’s a one time task and it’ll never repeat itself. WRONG!
Being lazy as I could and the need to quickly deploy systems for hungry customers, I started out with a small little infrastructure that helped me along the way to auto configure clusters of two nodes or more.
My use cases were:
1. RedHat cluster
2. LVS
3. Heartbeat
4. Oracle RAC
Auto configuring an Oracle RAC DB is not an easy task at all. However, with the proper infrastructure, it can become noticeably easier.
The common denominators for all cluster configurations I had to carry were:
1. They had to run on more than one node
2. Except from entering a password once for all nodes, I didn’t want any interaction
3. They all consisted from steps that should either run on all nodes, or on just one node
4. Sometimes you could logically split the task into a few phases of configuration, making it easier to comprehend the tasks you have to achieve
Even though it is not the most tidy piece of Bash code I’m going to post here, I’m very proud of it as it saved me countless hours. I give you the skeleton of code, which is the essence of what I nicknamed Bash RPC. On top of this you should be able to easily auto configure various configurations involving more than one computer.
Using it
The sample attached is a simple script that should be able to bake /home/cake on all relevant nodes.
In order to use the script properly, edit it using your favorite editor and stroll through the configuration_vars() function. Populate the array HOST_IP with your relevant hosts.
Now you can simply run the script.
I’m aware to the slight disadvantage that you can’t have your configuration come from command line, on the other hand – when dealing with big, complex, do you really think your configuration can be defined in a single line of arguments?
Taming it
OK, this is the interesting part. Obviously no one needs to bake cakes on his nodes, it is truly pointless and was merely given as an example.
So how would you go about customizing this skeleton to your needs?
First and foremost, we must plan. Planning and designing is the key for every tech related activity you carry. Be it a SysAdmin task or a pure development task. While configuring your cluster for the first time, keep notes of the steps (and commands) you have to go through. Later on, try to logically separate the steps into phases. When you have them all, we can start hacking Bash RPC.
Start drawing your phases and steps, using functions in the form of PHASEx_STEPx.
Fill up your functions with your ideas and start testing! and that’s it!
How does it work?
Simplicity is the key for everything.
Bash RPC can be ran in 2 ways – either running all phases (full run) or running just one step.
If you give Bash RPC just one argument, it assumes it is a function you have to run. If no arguments are given it will run the whole script.
Have a look at run_function_on_node(). This function receives a node and functions it should run on. It will copy the script to the destination node and initiate it with the arguments it received.
And this is more or less the essence of Bash RPC. REALLY!
Oh, there’s one small thing. Sometimes you have to run things on just one host, in that case you can add a suffix of ___SINGLE_HOST for your steps. This will make sure the step will run on just one host (the first one you defined).
I’m more than aware that this skeleton of Bash RPC can be polished some more and indeed I have a list of TODOs for this skeleton. But all in all – I really think this one is a big time saver.
Real world use cases
I consider this script a success mainly because of 2 real world use cases.
The first one is the act of configuring from A to Z Oracle RAC. Those of you who had to go through this nightmare can testify that configuring Oracle RAC takes 2-3 days (modest estimation) of both a DBA and SysAdmin working closely together. How about an unattended script running in the background, notifying you 45 minutes later you have an Oracle RAC ready for service?
The second use case is my good friend and former colleague, Oren Held. Oren could easily take this skeleton and use it for auto configuring a totally different cluster using LVS over Heartbeat. He was even satisfied while using it. Oren never consulted me while performing this task – and this is the great achievement.
I hope you could use this code snippet and customize it for your own needs, continuing with the YOU CAN CREATE A SCRIPT FOR EVERYTHING attitude!!
Have a look at cluster-config-cake.sh to get an idea about how it’s being done.
Wearing both hats of developer and SysAdmin, I believe you shouldn’t work hard as a SysAdmin. It is for a reason that sometimes developers look at SysAdmins as inferior. It’s not because SysAdmins are really inferior, or has an easier job. I think it is mainly because a large portion of their workload could be automated. And if it can’t be automated (very little things), you should at least be able to do it quickly.
Obviously, we cannot buy time, but we can at least be efficient. In the following post I’ll introduce a few of my most useful aliases (or functions) I’d written and used in the last few years. Some of them are development related while the others could make you a happy SysAdmin.
You may find them childish – but trust me, sometimes the most childish alias is your best friend.
Jumping on the tree
Our subversion trunk really reminds me of a tree sometimes. The trunk is very thick, but it has many branches and eventually many leaves. While dealing with the leaves is rare, jumping on the branches is very common. Many times i have found myself typing a lot of ‘cd’ commands, where some are longer than others, repeatedly, just to get to a certain place. Here my stupid aliases come to help me.
lib takes me straight away to our libraries sub directory, where sim takes me to the simulators sub directory. Not to mention tr (shortcut for trunk), which takes me exactly to the sub directory where the sources are checked out. Oh, and pup which takes me to my puppet root, on my puppet master. Yes, I’m aware to the fact that you probably wonder now “Hey, is this guy going to teach me something new today? – Aliases are for babies!!”, I can identify. I didn’t come to teach you how to write aliases, I’m here to preach you to start using aliases. Ask yourself how many useful day-to-day aliases you really have defined. Do you have any at all? – Don’t be shy to answer no.
*nix jobs are diverse and heterogeneous, but let’s see if I can encourage you to write some useful aliases after all.
In case you need some ideas for aliases, run the following:
$ history | tr -s ' ' | cut -d' ' -f3- | sort | uniq -c | sort -n
Yes, this will show the count of the most recent commands you have used. Still not getting it? – OK, I’ll give you a hint, it should be similar to this:
$ alias recently_used_commands="history | tr -s ' ' | cut -d' ' -f3- | sort | uniq -c | sort -n"
If you did it – you’ve just kick-started your way to liberation. Enjoy.
As a dessert – my last childish alias:
$ alias rsrc='source ~/.bashrc'
Always useful if you want to re-source your .bashrc while working on some new aliases.
Two more things I must mention though:
- Enlarge your history size, I guess you can figure out alone how to do it.
- If you’re feeling generous – periodically collect the history files from your fellow team members (automatically of course, with another alias) and create aliases that will suit them too.
The serial SSHer
Our network is on 192.168.8.0/24. Many times I’ve found myself issuing commands like:
$ ssh root@192.168.8.1
$ ssh root@192.168.8.2
$ ssh root@192.168.8.3
Dozens of these in a single day. It was really frustrating. One day I decided to make an end to it:
# function for easier ssh
# $1 - network
# $2 - subnet
# $3 - host in subnet
_ssh_specific_network() {
local network=$1; shift
local subnet=$1; shift
local host=$1; shift
ssh root@$network.$subnet.$host
}
# easy ssh to 192.168.8.0/24
# $1 - host
_ssh_net_192_168_8() {
local host=$1; shift
_ssh_specific_network 192.168 8 $host
}
alias ssh8='_ssh_net_192_168_8'
Splendid, now I can run the following:
$ ssh8 1
Which is equal to:
$ ssh root@192.168.8.1
Childish, but extremely efficient. Do it also for other commands you would like to use, such as ping, telnet, rdesktop and many others.
The cop
Using KDE’s konsole? – I really like DCOP, let’s add some spice to the above function, we’ll rename the session name to the host we’ll ssh to, and then restore the session name back after logging out:
# returns session name
_konsole_get_session_name() {
# using dcop - obtain session name
dcop $KONSOLE_DCOP_SESSION sessionName
}
# renames a konsole session
# $1 - session name
_konsole_rename_session_name() {
local session_name=$1; shift
_konsole_store_session_name `_konsole_get_session_name`
dcop $KONSOLE_DCOP_SESSION renameSession "$session_name"
}
# store the current session name
_konsole_store_session_name() {
STORED_SESSION_NAME=`_konsole_get_session_name`
}
# restores session name
_konsole_restore_session_name() {
if [ x"$STORED_SESSION_NAME" != x ]; then
_konsole_rename_session_name "$STORED_SESSION_NAME"
fi
}
# function for easier ssh
# $1 - network
# $2 - subnet
# $3 - host in subnet
_ssh_specific_network() {
local network=$1; shift
local subnet=$1; shift
local host=$1; shift
# rename the konsole session name
_konsole_rename_session_name .$subnet.$host
ssh root@$network.$subnet.$host
# restore the konsole session name
_konsole_restore_session_name
}
Extend it as needed, this is only the tip of the iceberg! I can assure you that my aliases are much more complex than these.
Finders keepers
For the next one I’m not taking full credit – this one belongs to Uri Sivan – obviously one of the better developers I’ve met along the way.
Grepping cpp files is essential, many times I’ve found myself looking for a function reference on all of our cpp files.
The following usually does it:
$ find . -name "*.cpp" | xargs grep -H -r 'HomeCake'
But seriously, do I look like someone that likes to work hard?
# $* - string to grep
grepcpp() {
local grep_string="$*";
local filename=""
find . -name "*.cpp" -exec grep -l "$grep_string" "{}" ";" | while read filename; do
echo "=== $filename"
grep -C 3 --color=AUTO "$grep_string" "$filename"
echo ""
done
}
OK, let’s generalize it:
# grepping made easy, taken from the suri
# grepext - grep by extension
# $1 - extension of file
# $* - string to grep
_grepext() {
local extension=$1; shift
local grep_string="$*"
local filename=""
find . -name "*.${extension}" -exec grep -l "$grep_string" "{}" ";" | while read filename; do
echo "=== $filename"
grep -C 3 --color=AUTO "$grep_string" "$filename"
echo ""
done
}
# meta generate the grepext functions
declare -r GREPEXT_EXTENSIONS="h c cpp spec vpj sh php html js"
_meta_generate_grepext_functions() {
local tmp_grepext_functions=`mktemp`
local extension=$1; shift
for extension in $GREPEXT_EXTENSIONS; do
echo "grep$extension() {" >> $tmp_grepext_functions
echo ' local grep_string=$*' >> $tmp_grepext_functions
echo ' _grepext '"$extension"' "$grep_string"' >> $tmp_grepext_functions
echo "}" >> $tmp_grepext_functions
done
source $tmp_grepext_functions
rm -f $tmp_grepext_functions
}
After this, you have all of your C++/Bash/PHP/etc developers happy!
Time to showoff
My development environment is my theme park, here is my proof:
$ (env; declare -f; alias) | wc -l
2791
I encourage you to run this as well, if my line count is small and I’m bragging about something I shouldn’t – let me know!
I like Bash, I really like Bash. It’s simple, it’s neat, it’s quick, it’s many things. No one will convince me otherwise. It’s not for big things though, I know, but for most system administration tasks it will suffice.
The problem with Bash begins where incompetent SysAdmins start to use it. They will most likely treat it poorly and lamely.
Conventions are truly needed when writing in Bash. In our subversion we have ~30,000 lines of bash. And HELL NO! our product is definitely not a lame conglomerate of Bash scripts gluing many pieces of binaries together. Bash is there for everything that C++ (in our case) shouldn’t do, things like:
- Packaging and anything related to packaging (post scripts of packages for instance)
- SysV service infrastructure
- Backups
- Customization of the development environment
- Deployment infrastructure
Yes, so where have we been? – Oh, how do we manage ~30,000 lines of Bash without getting everything messed out. For this, we need 3 main things:
- Version control system (CVS, SVN, git, pick your kick)
- Competent people
- Coding conventions
Configuring a version control system is easy, espcially if you are a competent developer, I’ll skip to the 3rd one.
Conventions. Show me just one competent C/C++/Java/C# developer who would skip on using coding conventions and program like an ape. OK, I know, there might be some, but we both think the same thing about them. Scripting in Bash shouldn’t be an exception in that case. We should use strict scripting conventions on Bash in particular and on any other scripting language in general.
There’s nothing uglier and messier than a Bash script that is written without conventions (well, maybe Perl without conventions).
I’ll try in the following post to introduce my Bash scripting conventions and if you find them neat – feel free to adopt them. Up until today, sadly, I havn’t seen anyone suggesting any Bash scripting conventions.
Before starting any Bash script, I have the following skeleton :
#!/bin/bash
main() {
}
main "$@"
Call me crazy, but without my main() function I ain’t going anywhere. Now we can start writing some bash. Once you have your main() it means you’re going to write code ONLY inside functions. I don’t care it’s a scripting language – let’s be strict.
#!/bin/bash
# $1 - hostname
# $2 - destination directory
main() {
local hostname=$1; shift
local destination_directory=$1; shift
}
main "$@"
Need to receive any arguments in a function? – Do the following:
- Document the variables above the function
- Use shift after receiving each variable, and always use $1, it’s easier to later move the order of the variables
Notice that i’m using the local keyword to make sure the scope of the variable is only within its function. Be strict with variables. If for some reason you decide you need a global variable, it’s OK, but make sure it’s in capital letters and declared as needed:
# a read only variable
declare -r CONFIGURATION_FILE="/etc/named.conf"
# a read only integer variable
declare -i -r TIMEOUT=600
# inside a function
sample_function() {
local -i retries=0
}
You are getting the point – I’m not going to make it any easier for you. Adopt whatever you like and remember that being strict in Bash yields code in higher quality, not to mention better readability. Feeling like writing a sloppy script today? – Go home and come back tomorrow.
Following is a script written by the conventions I’m suggesting. This is a very simple script that simply displays a dialog and asks the user which host he would like to ping, then displays the result in a tailbox dialog. This is more or less how many of my scripts look like. I admit it took me a while to find a script which is not too long (under 100 lines) and can still represent some of the ideas I was mentioning.
I urge you to question my way and conventions and suggest some of your own, anyway, here it is:
#!/bin/bash
# dialog options (notice it's read only)
# notice as well it's in capital letters
declare -r DIALOG_OPTS="0 0 0"
declare -i -r OUTPUT_DIALOG_WIDTH=60
# ping timeout in seconds (using declare -i because it's an integer and -r because it's read only)
declare -i -r PING_TIMEOUT=5
# size of pings
declare -i -r DEFAULT_SIZE_OF_PINGS=56
# number of pings to perform
declare -i -r DEFAULT_NUMBER_OF_PINGS=1
# neatly pipes a command to a tailbox dialog
# here we can specify the parameters the function expects
# this is how i like to do it, perhaps there are smarter ways that may integrate better
# with doxygen or some other tools
# $1 - tailbox dialog height
# $2 - title
# $3 - command
pipe_command_to_tailbox_dialog() {
# parameters extraction, always using $1, with `shift` right afterwards
# in case you want to play with the order of parameters, just move the lines around
local -i height=5+$1; shift
local title="$1"; shift
local command="$1"; shift
# need a temporary file? - always use `mktemp`, please spare me the `/tmp/$$` stuff
# or other insecure alternative to temporary file creations, use only `mktemp`
local output_file=`mktemp`
# run in a subshell and with eval, so we can pass pipes and stuff...
# eval is a favorite of mine, it means you truely understand the Bash shell
# nevertheless - it is really needed in this case
(eval $command) >& $output_file &
# need to obtain a pid? - it's surely an integer, so use 'local -i'
local -i bg_job=$!
# ok, show the user the dialog
dialog --title "$title" --tailbox $output_file $height $OUTPUT_DIALOG_WIDTH
# TODO my lame way of checking if a process is running on linux, anyone has a better way??
# my way of specifying a 'TODO' is in the above line
if [ -d /proc/$bg_job ]; then
# if the process is stubborn, use 'kill -9'
kill $bg_job || kill -9 $bg_job >& /dev/null
fi
# wait for process to end itself
wait $bg_job
# not cleaning up your temporary files is similar to a memory leak in C++
rm -f $output_file
}
# pings a host with a nice dialog
ping_host_dialog() {
local ping_params_tmp=`mktemp`
# slice lines nicely, i have long commands, i'm not going even to mention
# line indentation - that goes without saying
# i like to use dialogs, it makes more people use your scripts
if dialog --ok-label "Submit" \
--form "Ping host" \
$DIALOG_OPTS \
"Address:" 1 1 "" 1 30 40 0 \
"Size of pings:" 2 1 "$DEFAULT_SIZE_OF_PINGS" 2 30 40 0 \
"Number of pings:" 3 1 "$DEFAULT_NUMBER_OF_PINGS" 3 30 40 0 2> $ping_params_tmp; then
# ping_params_tmp will be empty if the user aborted the dialog...
local address=`head -1 $ping_params_tmp | tail -1`
# yet again if you expect an integer, use 'local -i'
local -i size_of_pings=`head -2 $ping_params_tmp | tail -1`
local -i number_of_pings=`head -3 $ping_params_tmp | tail -1`
fi
rm -f $ping_params_tmp
# this is my standard way of checking if a variable is empty
# may not be the prettiest way, but it surely catches the eye...
if [ x"$address" != x ]; then
pipe_command_to_tailbox_dialog 15 "Pinging host \"$address\"" "ping -c $number_of_pings -s $size_of_pings -W $PING_TIMEOUT \"$address\""
fi
}
# main function, can't live without it
main() {
ping_host_dialog
}
# although there are no parameters passed in this script, i still pass $* to main, as a habit
#main $*
# after Uri's comment, I'm fixing the following and calling "$@" instead
main "$@"
I don’t want this post to be too long (it is already quite long), but I think you got the idea. I still have some conventions I did not introduce here. In case you liked what you’ve seen – do not hesitate to contact me so I can provide you with a document describing all of my Bash scripting conventions.
After several conversations with my good friend and now also boss I decided to open a small little Blog which will describe my Linux/SysAdmin/C++/Bash adventures.
I’m Dan, you can learn a lot about me from my website at http://nevela.com . My expertise is Unix/Linux System administration and C++ development. Linux is my home and Bash is my mother’s tongue. In this Blog I hope to share interesting topics, stories and decisions in the mentioned fields.
I’ll start with something fresh and new for me – Puppet (http://reductivelabs.com/trac/puppet).
I believe System administration is an art, just like programming. SysAdmins divide into 2 types, from what I’ve seen:
- The fireman – most likely the mediocre SysAdmin type you are very familiar with. This type of SysAdmin knows how to troubleshoot and apply the specific fix where’s it’s needed. They get things done, but not more than that. Gradually their workload grows (especially if they do a bad job) and consume all of their time. This is the bad type of SysAdmin. Words like planning are usually not among their jargon (although they might argue with you).
- The developer – this type of SysAdmin plans before doing anything. I hope I can belong to this group. He thinks as a developer – whenever a problem arises in a system he manages – he treats it as a bug rather then a problem. This type of SysAdmin will always have his configuration stored in a source control repository. He will always solve bugs he encounter by fixing the script or program that caused the so called problem.
If you’re a fireman – then this post is not for you. However, if you’re the second type, then puppet is exactly for you.
As a SysAdmin in a small startup company, I got to a position where I have to manage 2 types of machines:
- Production machines
- In house machines for internal company usage
Luckily managing dozens of production machines is easy for us (for me :)) as the configuration is most likely to be the same on all of them. On the other hand, ironically, managing the in house machines became a more time consuming task, as each machine has many different and diverse jobs. This is where puppet kicks in.
Up until today, we had a few main servers doing most of the simple tasks, among them:
- LDAP
- Samba/NFS file sharing
- SVN
- Apache
- In house DNS and DNS caching
- Buildbot master
- Nagios
- And many more other things (you got the point)
Obviously I have all of my configuration stored in our SVN and backed up. Given that one of these servers would die, I could generate a new one within hours. But yet, most of the configuration there was done manually. Who is the crazy SysAdmin that would write a script to meta-generate named.conf? Who would auto create SVN repositories (especially if you have just one repository to manage)?
Yes, this is where puppet goes in. Puppet forces you to work correctly. By working with puppet, you realize that the way you’re going to work is only by writing puppet recipes and automating everything that makes your servers what they are. Puppet is also revolutionary in many other ways, but I’ll focus on the way that it forces the SysAdmin to work in a clean manner.
One thing I have to warn from, puppet has a slightly steep learning curve, but if you’re the serious Linux SysAdmin type – you wouldn’t find that one steep. You’ll struggle with it. You must.
So I said that no one (most likely) would write a script to generate its DNS configuration, but I saved mine in SVN and wrote a recipe for DNS servers to use:
$dns_username = 'named'
$dns_group = 'named'
$dns_configuration_file = '/etc/named.conf'
class dns-server {
# puppet takes care to install the latest package bind and it's cross platform!
package { bind:
ensure => latest
}
# configuring named. subscribing it on /etc/named.conf, which means that every time this
# file would change, named will refresh
service { named:
ensure => running,
enable => true,
subscribe => File[$dns_configuration_file],
require => Package["bind"]
}
# this is the /etc/named.conf file. in subclasses we'll specify the source of the file
file { $dns_configuration_file:
owner => $dns_username,
group => $dns_group,
replace => true,
mode => 640
}
}
Alright, this puppet code snippet only configures /etc/named.conf for usage with bind – but we never told puppet where to take the configuration from. I’ll share with you my DNS master server configuration:
class dns-master inherits dns-server {
# if it's a DNS master - lets take the named-master.conf file (obviously stored in SVN as well)
File[$dns_configuration_file] {
source => "$puppet_fileserver_base/etc/named/named-master.conf",
}
# copy all the zone files
file { "/var/named":
source => "$puppet_fileserver_base/etc/named/zones",
ignore => ".svn*",
checksum => "md5",
sourceselect => all,
recurse => true,
purge => false,
owner => $dns_username,
group => $dns_group,
replace => true,
mode => 640
}
# subscribe all the zone files in named
Service[named] {
subscribe +> File["/var/named"]
}
}
Great, so now every time I’ll change a zone in my puppet master, named will automatically restart. I believe this is great. Needless to say I do not use dynamic DNS updates, so I care not about zone files being updated.
Yes, so I did it once for DNS. If I ever needed now to change the configuration, I’d do it via puppet and let the configuration propagate. Maybe a small little phase of checking the new configuration, but right afterwards would come the svn commit – and it’s inside. Now I know I’m tidy and clean forever.
Needless to say that if I’ll ever need to configure another DNS server, I’ll use this recipe, or extend it as needed – puppet lets you inherit from base classes and add properties and behavior as needed (I have also a class for a dns-slave as you could have imagined).
OK, I should sum it up. To be honest – I’m glad I’m writing this part of the post after several discussions with a few of my wise colleagues.
Why should I use puppet?!
The answer to this one if not easy – and I’m not the one who should answer it. It is you who should answer it.
Adopting puppet reflects one main thing about you as a SysAdmin:
You are a lazy, strict and tidy SysAdmin, an expert.