cli

How to monitoring/show the progress of CLI tools in Linux (using cv and pv)

cv/coreutils viewer@github supports all basic utilities in coreutils package. It is written in C and shows the progress as percentage. Works by scanning /proc for commands that it supports, checks directories fd and fdinfo for opened files and seek positions and finally reports the progress for the largest file.

$ apt-get|yum|yaourt cv

'-m,--monitor' loop while monitored processes are still running
'-w,--wait' estimate I/O throughput and ETA
'-c,--command command' monitor only this command

# see all current and upcoming coreutils commands
$ watch cv -q
# see downloads progress
$ watch cv -wc firefox
# launch and monitor any heavy command using '$!'
$ cp bigfile newfile & cv -mp $!

from cv: progress bar for cp, mv, rm, dd…

pv/pipe viewer/pv@man is a terminal-based tool for monitoring the progress of data through a pipeline.

$ apt-get|yum|pacman install pv

command1 | pv | command2
pv input.file | command1 | pv > output.file

# display options
'-p,--progress/-t,--timer/-e,--eta/-b,--byte' show progress/timer/ETA/bytes
# output modifiers
'-N,--name NAME' prefix the output information
'-c,--cursor' use cursor positioning escape sequences
'-l,--line-mode' instead of counting bytes, count lines
'-s,--size SIZE' assume the total amount of data to be transferred is SIZE
# data transfer modifiers
'-L,--rate-limit RATE' limite transfer rate by bytes/sec
'-B,--buffer-size BYTES'

# watch how quickly a file is transferred using nc
$ pv file | nc -w 1 host 3000
# see progress of both pipes
$ pv -cN rawlogfile file.log | gzip | pv -cN gziplogfile > file.log.gz
# with ncurses's dialog
$ (pv -n backup.tar.gz | tar xzf - -C path/to/data ) 2>&1 | dialog --gauge "Running tar, please wait..." 10 70 0
# rsync and pv
$ rsync options source dest | pv -lpes Number-Of-Files

from pv@nixcraft and rsync and pv@nixcraft

How to non-interactive/unattended password login to ssh (using sshpass and ssh_askpass)
  • sshpass@man is a noninteractive ssh password provider. Its an less secure alternative to public key authentication. sshpass only works if the password prompt ends in assword:.
## install
$ apt-get install sshpass | yum install sshpass (EPEL) | pacman -S sshpass

'-p PASSWORD' take password from cli
'-e' take password from SSHPASS env var

# need to disable host key checking
$ sshpass -p PASSWORD ssh -o StrictHostKeyChecking=no user@host

# rsync (and scp) over password ssh
$ SSHPASS=PASSWORD rsync --rsh='sshpass -e ssh -l user' host:path .
$ sshpass -p PASSWORD scp user@host:path .

from sshpass@cyberciti

  • ssh@man using SSH_ASKPASS. If ssh does not have a terminal associated with it but DISPLAY and SSH_ASKPASS are set, it will execute the program specified by SSH_ASKPASS and open an X11 window to read the passphrase.
$ vi mypass.sh
#!/bin/sh
echo PASSWORD
$ export DISPLAY=:0 ; export SSH_ASKPASS=`pwd`/mypass.sh
$ setsid ssh -o StrictHostKeyChecking=no user@host

from ssh password from stdin

How to mass rename/copy/link files in Linux (using mmv and rename)
  • mmv moves (or copies, appends, or links, as specified) each source file matching a from pattern to the target name specified by the to pattern.
$(el) yum install mmv (NUX)
$(deb) apt-get install mmv
$(arch) yaourt -S mmv

# '*.jpeg' -> '*.jpg'
$ mmv '*.jpeg' '#1.jpg'
# '.html.en' -> 'en.html'
$ mmv '*.html.??' '#1.#2#3.html'
  • rename@man/rename@ubuntu is a perl script which can be used to mass rename files according to a regular expression.
$(el) yum install util-linux-ng
$(deb) apt-get install perl
$(arch) pacman -S perl-rename

# '*.php' -> '*.html'
$ rename -n 's/.php$/.html/' *.php
# upper case -> lower case
$ rename 'y/A-Z/a-z/' *
# strip '.bak' extension
$ rename 's/.bak$//' *.bak

from rename@cyberciti and rename@tecmint

Using youtube-dl/ffmpeg to download Youtube videos/playlists and convert to mp3

youtube-dl is a small command-line python program to download videos from YouTube.com and a few more sites.

## install
$(el) sudo yum install youtube-dl ffmpeg(RPMforge)
$(deb) sudo apt-get install youtube-dl ffmpeg(>=15.04)|libav-tools(<=14.10)
$(osx) brew install youtube-dl ffmpeg
$(arch) sudo pacman -S youtube-dl ffmpeg
$(pip) sudo pip install -U youtube-dl
$(bin) sudo wget https://yt-dl.org/latest/youtube-dl -O /usr/local/bin/youtube-dl ; sudo chmod a+x /usr/local/bin/youtube-dl

youtube-dl [OPTIONS] URL [URL...] (where URL can also be a playlist)

# video formats
'-F,--list-formats' list available formats
'-f,--format FORMAT' download selected format
'--max-quality FORMAT' highest quality format to download
$ youtube-dl -f 17 URL

# filesystem/download options
'-a,--batch-file FILE' file containing URLs to download, - to stdin
'-o,--output TEMPLATE' output filename template, e.g.: 'downloads/%(uploader)s/%(title)s-%(id)s.%(ext)s'
'-c,--continue' force resume of partially downloaded files
'-i,--ignore-errors' continue on download errors, e.g.: skip unavailable videos in a playlist
'-w,--no-overwrites' do not overwrite files
$ youtube-dl -a FILE

# post-processing (e.g: extract audio)
'-x,--extract-audio' convert video files to audio-only files (requires ffmpeg or avconv and ffprobe or avprobe)
'--audio-format FORMAT' best(default), aac, vorbis, mp3,m4a, opus or wav
'--recode-video FORMAT' encode the video to another format (mp4,flv,ogg,webm,mkv)
$ youtube-dl PlaylistURL -cit --max-quality FORMAT --extract-audio --audio-format mp3

from save youtube videos, install ffmpeg and download youtube playlist

clipgrab is an alternative to youtube-dl. Its a GUI app that grabs URLs from clipboard and adds them to a download list.

Using terminal pager in Linux (“less is more”, multitail and most)

Terminal pager is used to view (but not modify) the contents of a text file moving down the file one line or one screen at a time. Some, but not all, pagers allow movement up a file.

  • less@wiki/less@man is similar to more@wki, but has the extended capability of allowing forward/backward navigation and following a file.
## install
$ sudo apt-get|yum install util-linux

# navigation
'space,pgdown' next page
'b,pgup' previous page
'g,<' first line
'G,>' last line
'<n> G' line n
'F' enter follow mode, exit CTRL+C
$ less +F FILE

# search
'/ <text>' forward regexp search
'? <text>' backward regexp search
'n' next match
'N' previous match
$ less -I -p PATTERN FILE
  • multitail@wiki/multitail@man splits the terminal window or console (using ncurses) of a into two or more subwindows into which it can merge log files and command outputs.
## install
$ sudo apt-get|yum install multitail (EPEL) | sudo pacman -S multitail

# navigation
'b/B' scroll back selected/all windows
  'q' quit, 'g/G' goto first/last line, 'pgup/pgdown' page up/down
'Y' toggle line wrap
'F' simulate 'tail -f' mode

# window
'a/d' add/delete new window (or file to existing window)
'v' toggle vertical/horizontal window

# search
'/' regexp search
'I' toggle case sensitivity

'-i/I FILE' open file in new/previous window
'--mergeall' merge all windows
$ multitail /var/log/dmesg -I /var/log/syslog

'-l,L CMD' open command output in new/previous window
'-r/R INTERVAL' restarts command periodically, '-R' only shows differences
$ multitail -R 2 -l "netstat -tap"

'-s X' splits vertically 'x' columns
'-sn Y' how many windows per colum
$(2x2) multitail -s 2 -sn 2,2 FILE1..4

'-q INTERVAL PATH' periodicaly checks path for new files, '-Q' merges all in one window
$ multitail -Q 5 /var/log/*.log

from multitail@howtoforge

  • most@wiki/most@ is similar to more@wki, but has the extended capability of allowing both forward/backward and left/right navigation through the file, and supports multiple windows.
## install
$ sudo apt-get|yum install most | sudo pacman -S most

# navigation
'space,D,pgdown' page down
'U,delete,pgup' page up
'T/B' scroll top/bottom
'</>' scroll left/right
'J,G' goto line
':n' skip to next file

# window
'ctrl-w,x 2' horizontal split window in half
'ctrl-w,x 1' delete all other windows
'ctrl-x 0' delete this window
'o,ctrl-x 0' move to other window

# search
'S,f,/' forward search
'?' backward search
'n' next match

Useful CLI tools for Linux system devops/admins and developers

Here is a list of commands used by sysadmins and developers to do just about anything from the CLI. From backup, to network configuration, compression, compilation, debugging, package management, process management, text editing, …

Backup/Copy Tools
* dd
* duplicity
* rsync/scp
* rdiff-backup
* rsnapshot
* unison

Command Interpreters (Shells)
* bash
* csh
* dash/ash
* mc
* zsh

Compiler Tools
* autoconf/automake
* clang
* gcc/ld/as/gdb
* make

Compression and Archiving Tools
* 7z
* bzip2
* gzip
* pax
* tar
* zip
* xz

Daemon Tools
* service
* systemctl/systemd doc/systemd arch/systemd debian/systemd cheatsheet
* upstart

Download Tools
* axel
* curl
* lftp
* wget

File Managers
* mc
* ranger
* tree
* vifm

Hardware Tools
* inxi
* lspci
* lshw

Logging
* ccze
* logrotate
* rsyslog

Network active monitoring
* arp-scan
* iperf
* netcat/socat
* ping/tcpping
* sprobe
* tracepath/traceroute

Network configuration
* dig/nslookup
* ip/route/ifconfig
* ipcalc/sipcalc/whatmask
* tc

Network packet sniffing
* dhcpdump
* dsniff
* iptraf
* httpry
* ngrep
* p0f
* pktstat
* snort
* tcpdump
* tshark

Network passive flow/stats
* bmon
* iftop
* lsof
* nethogs
* netstat/ss
* speedometer
* speedtest-cli
* tcptrack
* vnstat

Online Resources
* commandlinefu
* free programming books

Package Management Tools
* apt-get/aptitude/dpkg
* yum/rpm
* pacman
* zypper
* pkg(BSD)

Performance Monitoring Tools
* dstat
* iotop
* iostat
* httpry
* nethogs
* ngxtop
* ps
* sar
* smem
* top/htop/atop

Processor Management Tools
* kill/killall/pkill
* nice/renice
* pgrep
* taskset

Productivity Tools
* byobu
* cal
* cheat
* cmus
* fortune
* mutt
* pv
* screen
* screenfetch
* ssh
* tmux
* weather/WMO/ICAO
* weechat/irssi

Security Tools
* getfacl/setfacl
* lynis
* nmap
* iptables
* passwd/apg

Source Control
* git
* hg
* svn

Storage Tools
* lvm
* mount

Text Processing Tools
* awk
* diff/patch
* grep/ack
* tail/multitail/ztail
* sed

Text Editors
* emacs
* nano
* vim

from Useful CLI tools for Linux system admins

How to stream/pipe a file to the browser from CLI (using eventsource or websockets)

It may be useful to stream/pipe a file content (a “tail -f”) to the browser, maybe a log.

Using Server-sent events and netcat@wiki

# on the server, install netcat
$ sudo apt-get install netcat | sudo apt-get install netcat-openbsd

# and stream a file
$ (echo -e "HTTP/1.1 200 OK\nAccess-Control-Allow-Origin: *\nContent-type: text/event-stream\n" && tail -f file | sed -u -e 's/^/data: /;s/$/\n/') | nc -l 1234

# on the browser, run js snippet
new EventSource("http://localhost:1234/").onmessage = function(e) { console.log(e.data); };

See sse@caniuse and sse@html5rocks

Using WebSocket and websocketd (static binary written in go)

# on the server, install websocketd, see https://github.com/joewalnes/websocketd/wiki/Download-and-install

# and stream a file
$ websocketd --port 1234 tail -f /path/to/file

# on the browser, run js snippet
new WebSocket("ws://localhost:1234/").onmessage = function(e) { console.log(e.data); };
# or from the cli, using http://einaros.github.io/ws/
$ npm install -g ws ; wscat -c ws://localhost -p 1234

See ws@caniuse and ws@html5rocks

How to configure network in Linux (using initscripts/ifcfg/ifupdown, NetworkManager/nmcli, netctl and systemd-networkd)

You can use network managers (nmcli,…), initscripts/systemd or call user-land network command line tools directly (iproute2/net-tools) to configure your network devices either static or dynamic/dhcp.

  • ifup/ifdown@man network interface configuration files used by ifup/ifdown, called by initscripts (used prior to systemd). Works all distros (but configuration file location and syntax changes). 
## fedora/rhel
$ cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=myhostname
#static
#GATEWAY=192.168.1.1
$ cat /etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE=eth0
ONBOOT=yes
#dynamic
BOOTPROTO=dhcp
#static
#BOOTPROTO=static
#IPADDR=192.168.1.2
#NETMASK=255.255.255.0
$ sudo /etc/init.d/network restart

## debian/ubuntu
$ sudo apt-get install ifupdown
$ man interfaces
$ cat /etc/network/interfaces
auto lo
iface lo inet loopback
#dynamic
#auto eth0
#    allow-hotplug eth0
#    iface eth0 inet dhcp
#static
auto eth0
iface eth0 inet static
    address 192.168.1.2
    netmask 255.255.255.0
    gateway 192.168.1.1
$ sudo networking restart

## dns (static)
$ cat /etc/resolv.conf
search mydomain.net
nameserver 10.254.0.121

## manual, non-persistent static using iproute2/net-tools
$ ip link set eth0 up 
$ ip addr add 192.168.1.2/24 broadcast 192.168.1.255 dev eth0
$ ip route add default via 192.168.1.1
$ ifconfig eth0 up
$ ifconfig eth0 192.168.1.2/24 broadcast 192.168.1.255
$ route add default gw 192.168.1.1 eth0

from NetworkConfiguration@debian, networkscripts@rhel and static ip@nixcraft

$ sudo yum install NetworkManager | sudo apt-get install network-manager | sudo pacman -Sy networkmanager

# nmcli manages all interfaces/devices except 'NM_CONTROLLED=no' in '/etc/sysconfig/network-scripts/ifcfg-*'
$ nmcli dev show ; nmcli con show

## unattened add/modify
nmcli con add type ethernet con-name <con-name> ifname <interface-name> ?ip4 <address/netmask> gw4 <defaultgw>?
nmcli con mod <con-name> +|-<setting>.<property> "<value>"
## interactive edit
nmcli con edit <con-name>
> goto ipv4 ; set addresses ip/mask defgw ; set dns ip1 ip2 ; set dns-search domain ; verify ; save persistent ; quit

## static
$ nmcli con mod ens192 ipv4.addresses "10.4.2.108/8 10.254.0.2"
$ nmcli con mod ens192 ipv4.dns "10.254.0.121 10.254.0.122" ipv4.dns-search "mydomain"

## dynamic 
$ nmcli con mod ens192 ipv4.method auto

## hostname
$ nmcli general hostname <myhostname>

$ systemctl restart NetworkManager | service network-manager restart
or
$ systemctl restart network.service | service networking restart

from configure static ip in centos7@xmodulo

  • netctl@arch is a CLI-based tool used to configure and manage network connections via profiles. Arch only.
## install
$ sudo pacman -Sy netctl

## static
$ cp /etc/netctl/examples/ethernet-static /etc/netctl/profile1
$ cat /etc/netctl/profile1
Description='A basic static ethernet connection'
Interface=eth0
Connection=ethernet
IP=static
Address=('192.168.1.23/24' '192.168.1.87/24')
#Routes=('192.168.0.0/24 via 192.168.1.2')
Gateway='192.168.1.1'
DNS=('192.168.1.1')

## dynamic
$ cp /etc/netctl/examples/ethernet-dhcp /etc/netctl/profile1
Description='A basic dhcp ethernet connection'
Interface=eth0
Connection=ethernet
IP=dhcp

$ netctl start profile1

## enable on boot
# either enable by profile; this creates and enables a systemd service on boot
$ netctl enable profile1
# or enable all profiles with eth0, needs 'pacmman -Sy ifplugd'
$ systemctl enable netctl-ifplugd@eth0.service
  • systemd.network as of version 210, systemd supports basic network configuration through udev and networkd.
$ systemctl enable systemd-networkd ; systemctl restart systemd-networkd

## static
$ cat /etc/systemd/network/10-static.network
[Match]
Name=ens32
[Network]
Address=10.4.2.111/8
Gateway=10.254.0.2
DNS=10.254.0.121
DNS=10.254.0.122
Domains=mydomain

## dynamic
$ cat /etc/systemd/network/20-dhcp.network
[Match]
Name=en*
[Network]
DHCP=v4

## hostname
$ hostnamectl set-hostname myhostname

from systemd-networkd@arch and systemd-networkd@coreos

How to record/playback terminal in Linux (using scriptreplay, ttyrec, TermRecord, asciinema, showterm)

It is useful to be able to rapidly share computer activity with a screencast, by recording and playback TTY output of a text-mode program.

$ script -t 2> timingfile
# script started, file is typescript
$ ls
<etc, etc>
$ exit
# script done, file is typescript
$ scriptreplay timingfile
  • ttyrec/ttyplay is a program and/or its file format capable of recording the TTY output of a text-mode program together with timestamps and then replaying it. similar to the script command, but also allows for pausing, slowing down or speeding up playback. ttycast broadcast your ttyrec in real time.
$ ttyrec
# in the executed shell, do whatever you want and exit
$ ttyplay ttyrecord

$ npm install -g ttycast
$ ttyreccast outfile.tty
# open browser to 'http://localhost:13377/'
$ reset && ttyrec /tmp/ttycast
  • TermRecord@github saves your recording locally as a sel-contained html file and you can replay it in your browser.
$ pip install TermRecord
$ TermRecord -o /path/to/output_html
  • asciinema.org is a free and open source solution for recording the terminal sessions and sharing them on the web.
$ curl -sL https://asciinema.org/install | sh
$ asciinema rec
  • showterm.io variation of asciinema for replaying terminal sessions but this one is in pure text. It is both lightweight and the result is copy and pasteable. 
$ gem install showterm
$ showterm 
# or if you have not installed showterm, you can run the standalone
$ bash <(curl record.showterm.io)
  • monitor recordings are uploaded to commands.com. You can also download the scripts (check the installation step).
$ curl commands.io/install-monitor-ubuntu
$ monitor -u myusername

How to detach a Linux process from shell (using bg/disown/nohup)

Linux will automatically a child process by sending SIGINT when parent (a shell exit) ends. To avoid this you must detach the child from the parent.

  • disown: Prevents a closing shell from sending SIGUP signals.
$ cmd & 
# or hit CTRL-Z to interrupt and bg
$ gb
[1] cmd &

# disown is supported in bash and zsh, but shells like csh, tcsh, dash do not support it
$ disown %1

see fg@cyberciti and disown@cyberciti

$ nohup cmd &

# nohupping backgrounded jobs is typically used to avoid terminating them when logging off from a remote SSH session
$ cmd </dev/null &>/dev/null &
or
$ nohup cmd > foo.out 2> foo.err < /dev/null &

see nohup@cyberciti

  • tmux@wiki/gnu screen@wiki: can also detach a process from the current shell. It allows one to reattach to the process later on.
# run cmd detached from SIGHUP and in the background
$ screen -A -m -d -S somename cmd &

from How to detach a Linux process from shell