Nduli's World - computer

GUIX OS with GeneNetwork

2024-07-22T00:00:00+03:00

Guix OS install involves downloading the iso; setting it on a flash disk with: dd if=guix.iso of=/dev/sdX status=progress; and using the graphical guide. Some quirks I got were:

I needed an internet connection but wifi didn't work out of the box. I used USB tethering though.
I forgot to edit the final configuration to UEFI. To boot, I used grub by:

# ref: https://superuser.com/a/1512531
set pager=1
ls
# find the drive that has guix_os by searching for /etc/config.scm file
cat drive_from_ls/etc/config.scm # e.g. cat (hd0,1)/etc/config.scm
set root=drive_from_ls # e.g. set root=(hd0,1)
cat /boot/grub/grub.cfg
# run the commands in the menuentry manually
linux /gnu/store/......
initrd /gnu/store/.....
# boot the OS
boot

I changed my bootloader config in /etc/config.scm to fix the efi issue

;; Ref: https://guix.gnu.org/manual/en/html_node/Bootloader-Configuration.html#index-bootloader_002dconfiguration
(bootloader (bootloader-configuration
  (bootloader grub-efi-bootloader)
  (targets '("/boot/efi"))
  (keyboard-layout keyboard-layout)))

and rebuild the OS with sudo guix system reconfigure --allow-downgrades /etc/config.scm - I experienced screen flickering in my terminal and modified my boot parameters to fix it:

;; ref: https://wiki.archlinux.org/title/Intel_graphics#Screen_flickering
(bootloader (bootloader-configuration
              (bootloader grub-efi-bootloader)
              (targets '("/boot/efi"))
              (keyboard-layout keyboard-layout)))
(kernel-arguments (list "i915.enable_psr=0"))

Quality of Life Changes

Zsh as default shell

Change /etc/config.scm to have:

;; ref: https://lists.gnu.org/archive/html/help-guix/2017-07/msg00024.html
;; added this
(use-package-modules shells)

(users (cons* (user-account
                (name "rookie")
                (comment "rookie")
                (group "users")
                (home-directory "/home/rookie")
                ;; added this line
                (shell #~(string-append #$zsh "/bin/zsh))
                (supplementary-groups '("wheel" "netdev" "audio" "vidoe")))
            %base-user-accounts))

When installing oh-my-zsh I also got an error: ZSH not installed and I had to do an extra guix install zsh.

Non Guix

Non Guix Reference

For wifi to work, I modified /etc/config.scm to have:

;; non free linux module
(use-modules (nongnu packages linux)
             (nongnu system linux-initrd))

(operating-system
  (kernel linux)
  (firmware (list linux-firmware))

The README did most of the heavy lifting, but I couldn't figure out how to add the substitutes.

;; what ended up in my services section
  (services
   (append (list (service xfce-desktop-service-type)

                 ;; To configure OpenSSH, pass an 'openssh-configuration'
                 ;; record as a second argument to 'service' below.
                 (service openssh-service-type)
                 (service cups-service-type)
                 (set-xorg-configuration
                  (xorg-configuration (keyboard-layout keyboard-layout))))

           ;; This is the default list of services we
           ;; are appending to.
           ;; %desktop-services)
                (modify-services %desktop-services
                     (guix-service-type config => (guix-configuration
                       (inherit config)
                       (substitute-urls
                        (append (list "https://substitutes.nonguix.org")
                          %default-substitute-urls))
                       (authorized-keys
                        (append (list (plain-file "non-guix.pub"
                                                ;; output from https://substitutes.nonguix.org/signing-key.pub
                                                ;; merged into one line
                                                  "(public-key (ecc (curve Ed25519) (q #C1FD53E5D4CE971933EC50C9F307AE2171A2D3B52C804642A7A35F84F3A4EA98#)))"
                                                  ))

                          %default-authorized-guix-keys))))))
   )

Locale Bug

I couldn't get tmux to start, getting tmux: invalid LC_ALL, LC_CTYPE or LANG and running locale -a failed too. The root cause was that my applications were built on a different version of glibc and running guix update fixed this.

GeneNetwork Setup

Follow the instructions here to set up genenetwork

Small changes:

# pick the channels file from 
curl https://ci.genenetwork.org/channels.scm > channels.scm
# gn2 setup
guix pull -C channels.scm -p ~/.guix-extra-channels/gn2
GUIX_PROFILE=$HOME/.guix-extra-profiles/gn2
. $GUIX_PROFILE/etc/profile
guix install genenetwork2 -p ~/.guix-extra-channels/gn2
guix install genenetwork3 -p ~/.guix-extra-channels/gn2 # setups an older version

# gn3 setup
guix pull -C channels.scm -p ~/.guix-extra-channels/gn3
GUIX_PROFILE=$HOME/.guix-extra-profiles/gn3
. $GUIX_PROFILE/etc/profile
guix install genenetwork3 -p ~/.guix-extra-channels/gn3
guix install genenetwork2 -p ~/.guix-extra-channels/gn3
guix package -i python-mysqlclient -p ~/.guix-extra-channels/gn3
guix package -i python-pyld -p ~/.guix-extra-channels/gn3
pytest -k unit_tests # succeeds after this
pytest # still fails
FLASK_DEBUG=1 FLASK_APP="main.py" flask run --port=8080 # works

# gn-auth set up is a bit different from the above
# the README is clear in this case
# just make sure you have genenetwork2 and genetwork3 installed in the profile

Generate HTML when Using Markdown in Vimwiki

2023-09-02T00:00:00+03:00

I moved to markdown in vimwiki but couldn't generate html. VimwikiAll2HTML only supported vimwiki syntax. I explored static site generators looking for one that:

didn't require moving my files and folders.
didn't need a lot of configuration to set up.

I tested out mkdocs, zola, docusaurus and vitepress and only vitepress met both conditions. It also worked out the box and I liked the default theme.

npm install -D vitepress
npx vitepress init

Note:

This works with "vitepress": "^1.0.0-beta.3" in my package.json.
This overrides index.md, so have a back up. I used git to restore it.

I generated the hmtl using:

npx run docs:build

The sidebars didn't look great though, and I used vitepress-plugin-auto-sidebar to set this to by my folders. It's config looks like:

sidebar: getSidebar({ contentRoot: '/', contentDirs: ['team'], collapsible: true, collapsed: true })

and this forced me to set up a long list of contentDirs. I instead used an ignore list and generated my contentDirs from code with:

const EXCLUDE_DIRS: string[] = ['node_modules', 'diary', 'scripts'];
const CONTENT_ROOT = '/';

function getSideBarFolders(): string[] {
  var content_root = path.join(process.cwd(), CONTENT_ROOT)
  return fs.readdirSync(content_root, {withFileTypes: true})
           .filter(f => f.isDirectory() && !f.name.startsWith('.') && !EXCLUDE_DIRS.includes(f.name))
           .map(f => f.name)
           .sort();
}

changing the plugin's config to:

sidebar: getSidebar({ contentRoot: CONTENT_ROOT, contentDirs: getSideBarFolders(), collapsible: true, collapsed: true }),

I added a post-commit hook that would update my vimwiki site any time I changed my wiki:

set -euo pipefail

cd "$(git rev-parse --show-toplevel)"
npm run docs:build
rsync --recursive --archive --update --verbose --compress -P .vitepress/dist/ user@ip:/path/to/folder

Migrate From Vimwiki To Markdown Syntax

2023-07-29T00:00:00+03:00

Vimwiki supports markdown and I switched to this because:

I didn't use the default syntax anywhere else.
I could convert to html using only VimwikiAll2HTML.

Fixing My Config

Vimwiki expects some options set before it loads (ref). With lazy, I use init like:

{ 'vimwiki/vimwiki',
  branch = 'dev',
  dependencies = {'mattn/calendar-vim'},
  init = function()
    vim.g.vimwiki_list = {
      {path = "~/vimwiki", path_html= "~/vimwiki/public_html", auto_tags = 1, auto_diary_index = 1, syntax = 'markdown', ext = '.md'},
      -- experimental folder for markdown conversion
      {path = "~/projects/vimwiki", path_html= "~/projects/vimwiki_markdown/public_html", syntax = 'markdown', ext = '.md'}, }
    vim.g.vimwiki_markdown_link_ext = 1
    vim.g.vimwiki_stripsym = ' '
    vim.g.vimwiki_global_ext = 0
  end
},

Pandoc Conversion

I tested pandoc for conversion with the file:

== Title ==

- [o] list item i:
  - [.] sublist
    - [X] sub sub
    - [ ] sub usb
    - [ ] sub sub1
  - [X] final

TODO: task 1

%% a comment I've left to address later

I'm a normal paragraph.

[[link to file]]

{{{bash
  ls -lah
  find . -iname '*.wiki'
}}}

Another code block

{{{
  ls -lah
  find . -iname '*.wiki'
}}}

Running pandoc --from vimwiki --to commonmark_x trial.wiki gave me:

╰─$ pandoc --from vimwiki --to commonmark_x trial.wiki
## Title {#Title}

- []{.done2}list item i:
  - []{.done1}sublist
    - []{.done4}sub sub
    - []{.done0}sub usb
    - []{.done0}sub sub1
  - []{.done4}final

[TODO:]{.todo} task 1

I\'m a normal paragraph.

[link to file](link to file "wikilink")

``` bash
  ls -lah
  find . -iname '*.wiki'
```

Another code block

      ls -lah
      find . -iname '*.wiki'

I had these problems with the conversion:

the checklist generated wouldn't work with vimwiki
I didn't want the escaped ' character in my files
The link to a file had the string "wikilink" which wouldn't work in vimwiki
I lost my comment
The last code block wasn't wrapped around ` characters

I'd combine sed with pandoc to get what I wanted.

Preprocessing

I replaced the comment identifier %% with TODO: comment to the prevent losing them during conversion. I also replaced code blocks without a language i.e. {{{ to have bash i.e. {{{bash and I'd replace this when editting later.

╰─$ echo -e "%% save me\nNormal line %% with ending\n{{{" \
      | sed -r -e 's/\{\{\{$/\{\{\{bash/g' \
               -e 's/%%/TODO: comment/g'
TODO: comment save me
Normal line TODO: comment with ending
{{{bash

Post processing

I changed the vimwikie-converted linked i.e. [link_to_file](link_to_file "wikilink") to use the file name i.e. [link_to_file](link_to_file.md).

echo -e "[link_to_file](link_to_file \"wikilink\")" \
  | sed -r -e 's/(\[.*\])\(([^#]*)((.*) "wikilink")\)/\1\(\2.md\4\)/g'

I converted all checklists to github's checklist syntax with:

echo -e "- []{.done4}done\n- []{.done0} not_done" \
  | sed -r -e "s/\[\]\{\.done[0-3]\}/\[ \] /g" \
           -e "s/\[\]\{\.done4\}/\[X\] /g"

and also unquoted all ' characters with:

echo -e "\\'" | sed -e "s/\\\'/\'/g"

Here's the final post processing step:

  sed -r -i -e 's/(\[.*\])\(([^#]*)((.*) "wikilink")\)/\1\(\2.md\4\)/g' \
    -e "s/\\\'/\'/g" \
    -e "s/\[\]\{\.done[0-3]\}/\[ \] /g" \
    -e "s/\[\]\{\.done4\}/\[X\] /g" \
    "$md_file"

Putting It All Together

#!/bin/bash

set -euo pipefail

# files to change extension from to md format
readarray -d '' mv_files < <( \
  find . \( -iwholename '*diary/*.wiki' \
    -or -iwholename '*unorganized_things/*.wiki' \
    -or -iwholename '*design_docs/*.wiki' \
    -and -not -iname 'diary.wiki' \
    -and -not -iname 'index.wiki' \) \
    -print0 
)

for file in "${mv_files[@]}"; do
  md_file="${file%%.wiki}.md"
  mv "$file" "$md_file"
done

readarray -d '' files < <(find . -name "*.wiki" -print0)
for file in "${files[@]}"; do
  md_file="${file%%.wiki}.md"
  sed -r -e 's/\{\{\{$/\{\{\{bash/g' -e 's/%%/TODO: comment/g' "$file" | pandoc --from vimwiki --to commonmark_x -o "$md_file"
  sed -r -i -e 's/(\[.*\])\(([^#]*)((.*) "wikilink")\)/\1\(\2.md\4\)/g' \
    -e "s/\\\'/\'/g" \
    -e "s/\[\]\{\.done[0-3]\}/\[ \] /g" \
    -e "s/\[\]\{\.done4\}/\[X\] /g" \
    "$md_file"
  rm "$file"
done

SC-IM Color Configuration

2023-07-01T00:00:00+03:00

I couldn't read cells in sc-im because there was little contrast between the background and foreground colors. I temporarily fixed this by changing my terminal's color scheme, but I'd prefer to keep it. A permanent solution was to change sc-im's colors, for example:

color "type=INPUT fg=WHITE bg=BLACK"

and changed the colors of the help documentation with:

color "type=NORMAL fg=WHITE bg=BLACK"

The help menu provides more details for the colors to use. You type :help and search for color with \color, getting the explanations for all the types supported. I experimented with a couple of options and saved my final colors in ~/.scimrc. Here's what I had:

color "type=INPUT fg=WHITE bg=DEFAULT_COLOR"
color "type=NORMAL fg=WHITE bg=DEFAULT_COLOR"
color "type=ERROR_MSG fg=RED bg=BLACK"
color "type=CELL_SELECTION fg=BLACK bg=WHITE"
color "type=HEADINGS fg=WHITE bg=DEFAULT_COLOR"
color "type=WELCOME fg=WHITE bg=DEFAULT_COLOR"
color "type=MODE fg=WHITE bg=DEFAULT_COLOR"

I also found a ticket on the problem and presented my solution here.

Dev Experience For Personal Projects

2023-04-29T00:00:00+03:00

My personal projects have poor developer experience. Any time I want to fix a bug or improve a service, I experience a lot of frustration.

For example, I wanted to upgrade the UI of my pomodoro script to have colors for completed and cancelled tasks, better prompts and mark the task status. I couldn't get how the original script worked and any change I made broke something. I added the features but the experience was painful. I didn't have another way to tell that I broke something except from running the script end to end.

I set up my first server manually, without documentation. There were a lot of experiments I did, which means it's impossible to create a similar setup in future or upgrade the same server. A better way would have been to document the set up, or put everything in code using ansible, with comments that explain non-obvious configurations.

I'll try and have at least these in my personal projects to help with dev experience:

easy set up of project
easy and fast testing
clear documentation on manual and non-obvious steps
documentation to run and build the project
documentation on how I deployed the project
explanation for non-obvious debugging steps

Testing GUIX

2023-03-31T00:00:00+03:00

I installed guix with:

# Install nscd (name service cache daemon) to help pick up guix libs
sudo apt install nscd
sudo systemctl enable nscd

# install guix
# Ref: https://guix.gnu.org/manual/en/html_node/Binary-Installation.html
cd /tmp
wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
chmod +x guix-install.sh
sudo ./guix-install.sh

# update guix
guix pull

# log out and in again
# confirm guix-daemon is running with:
ps aux | grep -i guix

#  installing guix locales, used by libc installed via guix
guix install glibc-locales

# ensures guix libs and apps can do https
guix install nss-certs

# Add the following to .profile
GUIX_PROFILE="$HOME/.guix-profile"
GUIX_LOCPATH=$GUIX_PROFILE/lib/locale
. "$GUIX_PROFILE/etc/profile"

I like i3 window manager and the ubuntu provided version doesn't yet support gaps. I installed i3 using the ubuntu package manager (apt), which configured the login manager for i3 in /usr/share/xsessions/. I then installed i3-gaps using guix and it had preference to the i3 installed via apt.

sudo apt-install i3
guix install i3-gaps

I set up fonts using:

guix install fontconfig
fc-cache -rv

and installed other useful packages:

# installing my editor
guix install python neovim python-pynvim git

I update my guix system using:

guix pull
guix package -u

I needed to install neovim that was build with a later version of the gcc-tool chain. I solved this with:

guix package -i neovim --with-c-toolchain=tree-sitter=gcc-tool-chain@12.2

Stack Static Missing Library With Guix

2023-02-25T00:00:00+03:00

I installed stack_static by downloading the binary from github and moving it to .local/bin. I was able to compile haskell programs, but running them failed with the error:

$ stack run
/home/rookie/projects/random/helloworld/.stack-work/install/x86_64-linux-tinfo6/48795ad824c5ab2c527b42cc08a43bf5cd587c042c536072710c2302b997bb41/9.2.5/bin/helloworld-exe: error while loading shared libraries: libgmp.so.10: cannot open shared object file: No such file or directory

ldd showed that the library was missing:

$ ldd .stack-work/install/x86_64-linux-tinfo6/48795ad824c5ab2c527b42cc08a43bf5cd587c042c536072710c2302b997bb41/9.2.5/bin/helloworld-exe
        linux-vdso.so.1 (0x00007ffd0958e000)
        libgmp.so.10 => not found
        libc.so.6 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libc.so.6 (0x00007f5b9d530000)
        libm.so.6 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libm.so.6 (0x00007f5b9d3ef000)
        librt.so.1 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/librt.so.1 (0x00007f5b9d3e5000)
        libdl.so.2 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libdl.so.2 (0x00007f5b9d3e0000)
        libpthread.so.0 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libpthread.so.0 (0x00007f5b9d3be000)
        libgcc_s.so.1 => /gnu/store/4zvswpr2h3b7dvqpvjcdam8vfhyjrmgl-gcc-12.2.0-lib/lib/libgcc_s.so.1 (0x00007f5b9d39d000)
        /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/ld-linux-x86-64.so.2 => /gnu/store/ayc9r7162rphy4zjw8ch01pmyh214h82-glibc-2.33/lib/ld-linux-x86-64.so.2 (0x00007f5b9d6f4000)

I noticed the search path for most libraries were guix related, so I ran the OS provided ldd and it found the libgmp.so.10 library.

$ /usr/bin/ldd .stack-work/install/x86_64-linux-tinfo6/48795ad824c5ab2c527b42cc08a43bf5cd587c042c536072710c2302b997bb41/9.2.5/bin/helloworld-exe
        linux-vdso.so.1 (0x00007ffc41cc0000)
        libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f4148ad2000)
        libc.so.6 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libc.so.6 (0x00007f4148910000)
        libm.so.6 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libm.so.6 (0x00007f41487cf000)
        librt.so.1 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/librt.so.1 (0x00007f41487c5000)
        libdl.so.2 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libdl.so.2 (0x00007f41487c0000)
        libpthread.so.0 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libpthread.so.0 (0x00007f414879e000)
        libgcc_s.so.1 => /gnu/store/4zvswpr2h3b7dvqpvjcdam8vfhyjrmgl-gcc-12.2.0-lib/lib/libgcc_s.so.1 (0x00007f414877d000)
        /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f4148b68000)

I didn't know how to add the library to guix (guix install gmp didn't fix this). The solution was to compile the program using native OS level utilities.

# opens bash without guix loaded utilities
env -i /bin/bash --login --noprofile --norc  
export PATH=$PATH:~/.local/bin
stack build

The binary worked without an errors.

Comic Site Migration to SQLite

2023-01-07T00:00:00+03:00

Postgres was the database for my comic site but I chose to migrate to sqlite because it would:

be easier to back up
use less resources on my server
be one less service to manage

I also don't get a lot of traffic so it wouldn't have an impact.

The process followed this general pattern:

Change the code to use sqlite instead of postgres. I'd maintain the same paths to the static resources so that I wouldn't have to move these (I stored static resources on the host).
Change the docker-compose yml file to:
- Use a different local port for the sqlite instance
- Use a different project name
Create a json dump from the production server using: python manage.py dumpdata --natural-foreign --natural-primary > /var/backups/comic_server/natural_dump.json
Change ansible to deploy the project to a different folder and run it.

Perform a migration and load the data with:

python manage.py migrate
python manage.py loaddata /var/backups/comic_server/natural_dump.json

Verify the new instance works.
Change nginx configuration to point to the new instance.
Turn down the old instance with docker-compose down. I'll clear out the volumes once I'm sure everything is ok.

Note: python commands run in the docker-compose container by first running: docker compose exec comic_server /bin/bash

How To Code Review

2022-10-29T00:00:00+03:00

Code reviews improve the health of a system while spreading ownership of code and knowledge. A good review process is fast and we can handle low hanging fruits by automating trivial checks (e.g. running unit tests, style choices) and choosing style guidelines. We can have high level discussions on the changes before coding to prevent rewrites.

Once the changes are ready, the reviewer:

responds quickly to the review request (if not in the middle of deep work). At most 1 day shouldn't pass before they respond.
checks the code and tests for:
- correctness, test coverage, functionality and best practices.
- hard to understand code, unclear names, commented out code, untested code,
- unhandled edge cases, duplication, bad test structure
- too many changes (if these are a lot, ask dev to split the PR).
- comments. They should explain why not what the code is doing (comments != documentation).
- consistency
- documentation updates.
is kind when commenting on code. They:
- comment on what they like in the PR (positive feedback)
- comment on code and not the coder e.g. instead of you're creating many db connections here have this code creates many db connections.
- Use questions instead of statements. Questions feel less like criticism and can trigger creative solutions while also assuming the author knows more than you do e.g:
  - I find this confusing becomes Could you elaborate on how this works?
  - This is unmaintainable becomes What was your vision on maintaining this?
- Use I-messages instead of you-messages i.e. I suggest/think/believe/would, It's hard for me, For me it seems e.g. you are writing cryptic code becomes It's hard for me to grasp what's going on in this code. We see You-messages absolute statements or attacks, and get defensive.
- Use we instead of I/me/you which makes comments less judgemental e.g. could we change this function to do it this way?
ensures the dev implements explanations in code (i.e. rewrite or a comment). Explanations in the review tool won't help future devs.
accepts that there are different solutions and distinguish between common best practices and personal taste. They are okay with compromises.
Use the "Yes, and ... " technique to keep an innovative atmosphere. Don't dismiss fresh and fragile ideas in a draft PR.
Have discussions in public, and ensure code documents these.
is explicit about the action they want. Prefix your comments with emojis or the intent e.g. nit, FYI.
Reach out to dev if they have too many comments or comments that are too long, solving misunderstanding early. Talk these out instead of using comments.
Have a bias for action e.g. approve a PR with minor comments.

References

My Firefox Configuration

2022-07-02T00:00:00+03:00

I like firefox, and I've tweaked it to be ideal for me. The extensions I have are:

Dark reader: I prefer dark themes, so this applies one to websites. This works great except for my bank's website, which I've excluded in the plugin.

Tridactyl: provides vim keybindings. It's intuitive and I'm fond of how it implemented bookmarks. I changed the theme to shydactyl-dark using:

colors --url https://raw.githubusercontent.com/eeshugerman/shydactyl-variations/master/dist/shydactyl-dark.css shydactyl-dark
colourscheme shydactyl-dark

When opening a new tab, there'd be a white canvas that immediately switched to dark-mode, a flash bang, which was disorienting. I found a solution on this github comment and applied it by opening $HOME/.mozilla/firefox/profile-dir/chrome/userContent.css and adding:

/*
   dark background in new tabs without a white flash (with tridactyl newtab)
   @see: https://github.com/tridactyl/tridactyl/issues/2510
*/
:root {
  --tridactyl-bg: #1d1b19 !important;
  --tridactyl-fg: white !important;
}

/*
    set the background color of the new tab page (without tridactyl or with tridactyl without newtab)
*/
@-moz-document url-prefix(about:home), url-prefix(about:newtab) {
  body {
    background: #1d1b19;
  }
}

uBlock origin: ad blocker
Leechblock NG: blocks sites that I don't want to visit. I have to add these sites to a block list though, but once done, it's difficult to open them. I've added youtube, twitter and more, and this helps me concentrate on important tasks.
Remove youtube suggestions: blocks distracting features on youtube. I've removed the Explore link, Shorts link and End of Video suggestions.
Tab Counter: shows how many windows and tabs I have open, and the number of tabs in the current window. This gives me a rough idea of whether my tabs are becoming unmanageable.
SponsorBlock: skips sponsor sections in youtube videos.

I also change the following configs:

dom.event.clipboardevents.enabled = False
layout.css.devPixelsPerPx = 1.25
ui.systemUsesDarkTheme = true
toolkit.legacyUserProfileCustomizations.stylesheets = true
browser.display.background_color = #000000
browser.display.foreground_color = #ffffff

I previously had a problem where the first menu item on right click would be chosen in firefox. I haven't experienced this in a while, but here's a solution just in case: soln for mouse clicking

Moving home directory to another partition

2022-04-30T00:00:00+03:00

I'd need to set up ubuntu, but I didn't want to lose my home directory. Everything was in one partition, so I'd lose my home folder with a re-install. I didn't want this, so I moved it into another partition.

The new partition would be the size of my home folder at least, and I found this by running:

cd /home
du -sch $(ls -A)

I didn't have enough space, so I cleared up some space using these instructions. I made sure the free space was equal to m home directory + 50GB. I don't want to be unable to store new files.

I resized (shrunk) the current partition, then created the new partition from the unallocated space. I used gparted live usb for this. It's easy to use the GUI tool to shrink and create new partitions.

Once done, I logged into my original OS as the root user, mounted the new partition and copied over my home folder with:

mount /dev/sdc2 /mnt
cp -rp /home/* /mnt # cp -r --preserve=all /home/* /mnt might be better
mv /home /home.old

I got the uuid for the new partition with blkid, and modified my /etc/fstab to automatically mount this partition on boot as my home directory with:

# home directory mount
UUID=uuid_from_blkid_output         /home           ext4            rw,relatime     0 2

I rebooted my laptop and everything seemed ok.

Lastly, I cleaned out the home.old folder with:

rm -rf /home.old

And now I could safely change my OS without losing my home directory.

Kubernetes Basics

2022-02-26T00:00:00+03:00

"We've got this project on kubernetes that you'll be helping maintain"

This started my journey with kubernetes. I found a great tutorial from freecodecamp and this blog is an attempt to set up some personal projects using it. It's best to read the original though as it's more in-depth.

I wanted to serve an index.html for a first project. A pod would be used. A pod is the smallest deployable unit in kubernetes, and it can contain one or more containers (e.g. docker containers). Its an isolated environment to run a docker image, providing storage and networking.

To create a pod, I can create one manually or use a workload resource, which provide extra features like replication, recreating pods when one stops working and more. The workload resources have a pod template, which provides a description of how to create the pods we want.

A deployment is a workload resource that creates replicas of the pod. I wanted 3 pods serving the index.html file, so I'd use a deployment.

To start off, I installed packages to help me play with kubernetes locally:

sudo pacman -S minikube kubectl docker
sudo systemctl start docker
minikube config set driver docker
minikube start

I also created a dockerfile that served the index.html file using nginx.

# tagged static:0.1.0
FROM nginx:1.21.1
COPY index.html /usr/share/nginx/html/index.html

I created a yml file that defined the deployment resource that would be created in minikube.

# static_deployment.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-website-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: static-website-pod
  template:
    metadata:
      labels:
        app: static-website-pod
    # pod template section
    spec:
      containers:
      - name: static-website-container
        image: static:0.1.0
        ports:
        - containerPort: 80

This creates a deployment work resource with the name static-website-deployment. The deployment ensures that there are 3 pods running the docker image at any one time (defined in replicas). The selector.matchLabels.app is used to define what pods are being managed, and are the same as the metadata.labels.app found in the template section. This metadata is applied to each pod that is created. The pod template defines how the pod are created, so in each of the 3 pods there will be a running container named static-website-container, and the pod would expose port 80.

To run the above:

minikube start
eval $(minikube -p minikube docker-env) # ensure docker images are built in minikube context
docker build -t static:0.1.0 -f Dockerfile_static_content .
kubectl apply -f k8s/deployment.yml

To check that things are running as expected:

╰─$ kubectl get deployment
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
static-website-deployment   3/3     3            3           17s
╰─$ kubectl get pods
NAME                                         READY   STATUS    RESTARTS   AGE
static-website-deployment-57bdbf7d94-7ngwt   1/1     Running   0          4s
static-website-deployment-57bdbf7d94-9l5cv   1/1     Running   0          4s
static-website-deployment-57bdbf7d94-gj59k   1/1     Running   0          4s

I wanted to use curl to verify the pods are running correctly, but the kubernetes environment is isolated. To deal with this, kubernetes has services which provide a means of exposing a set of pods. I set up a LoadBalancer service, which provides an ip address and a port that can be used to access the pods.

# static_load_balancer.yml
apiVersion: v1
kind: Service
metadata:
  name: static-load-balancer
spec:
  selector:
    app: static-website-pod
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer

and ran the following:

╰─$ kubectl apply -f static_load_balancer.yml
service/static-load-balancer unchanged
╰─$ kubectl get services
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes             ClusterIP      10.96.0.1       <none>        443/TCP        6d5h
static-load-balancer   LoadBalancer   10.105.222.19   <pending>     80:30133/TCP   116s
╰─$ curl $(minikube ip):30133
<!DOCTYPE html>
<html lang="en">
    <head>

minikube ip provides the ip address of minikube, and the port is the second part of the PORTS section of the static-load-balancer service.

Since what we're exposing is http traffic, I could also use an ingress object, which is a type of controller that can expose http and https traffic. Other advantages include ssl termination and name-based virtual hosting. I first needed to enable ingress in minikube with:

minikube addons enable ingress

The ingress controller links up with a service, so we could use the LoadBalancer service previously created.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: static-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: static-load-balancer
                port:
                  number: 80

However, it doesn't make much sense to have both an ingress object and a load balancer pointing to the same thing. Another service I could use is the ClusterIP which provides an ip internal to the cluster. This way we only have one entry point into minikube.

# static_ingress.yml
---
apiVersion: v1
kind: Service
metadata:
  name: static-clusterip
spec:
  selector:
    app: static-website-pod
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: static-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: static-clusterip
                port:
                  number: 80

And now running:

╰─$ kubectl get services
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes             ClusterIP      10.96.0.1       <none>        443/TCP        6d6h
static-clusterip       ClusterIP      10.109.227.9    <none>        80/TCP         108s
static-load-balancer   LoadBalancer   10.105.222.19   <pending>     80:30133/TCP   81m
╰─$ kubectl get ingress
NAME             CLASS    HOSTS   ADDRESS     PORTS   AGE
static-ingress   <none>   *       localhost   80      25m
╰─$ curl $(minikube ip)
<!DOCTYPE html>
<html lang="en">
    <head>
    .
    .

Having the basics of kubernetes i.e. controllers, services and ingress down, I tried to set up a django project. I'll add a link to this when it's ready.

Sending Mail from Cron in Ubuntu 20.04 LTS

2022-01-15T00:00:00+03:00

I created a cronjob to back up my server content and wanted to test out emailing of errors when they occurred. To set this up, I installed msmtp and msmtp-mta to help send emails using an smtp server and bsd-mailx to test sending emails from terminal. The cronjob entry was:

MAILTO="email@example.org"
*/1 * * * * echo "using default crontab mail functionality variable"

However, I got the following errors (check out /var/log/syslog):

Dec  2 06:14:02 ubuntu-docker-comic-misc-server cron[7901]: sendmail: the server did not accept the mail
Dec  2 06:14:02 ubuntu-docker-comic-misc-server cron[7901]: sendmail: server message: 551 5.7.1 Not authorised to send from this header address
Dec  2 06:14:02 ubuntu-docker-comic-misc-server cron[7901]: sendmail: could not send mail (account default from /home/rookie/.config/msmtp/config)
Dec  2 06:14:02 ubuntu-docker-comic-misc-server CRON[7899]: (rookie) MAIL (mailed 50 bytes of output but got status 0x0045 from MTA#012)

which confused me since msmtp should have had the from field set. Here's the config:

# Set default values for all following accounts.
defaults
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth           on
logfile        ~/.msmtp.log

account fastmail
host smtp.fastmail.com
port 465
from email@replace_me.com
user replace_me@fastmail.com
password some_password
tls_starttls off

# Set a default account
account default : fastmail

and msmtp logs showed that the appropriate from address was being used.

Dec 02 06:15:02 host=smtp.fastmail.com tls=on auth=on user=replace_me@fastmail.com from=email@replace_me.com recipients=email@example.org smtpstatus=551 smtpmsg='551 5.7.1 Not authorised to send from this header address' errormsg='the server did not accept the mail' exitcode=EX_UNAVAILABLE

I had made an incorrect assumption. I read msmtp docs and found that the set from = someemail configuration is used only when the original email is lacking the from field. A later release would support set_from_header on flag that would replace the from field with the config value. The version I was on is:

rookie@ubuntu-docker-comic-misc-server:~$ apt policy msmtp
msmtp:
  Installed: 1.8.6-1
  Candidate: 1.8.6-1
  Version table:
 *** 1.8.6-1 500
        500 http://mirrors.digitalocean.com/ubuntu focal/universe amd64 Packages
        100 /var/lib/dpkg/status

Next option was to find out if ubuntu's cron has support for setting a custom from address. Ubuntu uses a patched vixie-cron, but looking through the source code on vixie-cron's github page, the from field was either set up to the cron user or root.

There was a future patched version from ubuntu that provided support for a MAILFROM field, found in this bug discussion, but this was not available yet in my ubuntu release.

rookie@ubuntu-docker-comic-misc-server:~$ apt policy cron
cron:
  Installed: 3.0pl1-136ubuntu1
  Candidate: 3.0pl1-136ubuntu1
  Version table:
 *** 3.0pl1-136ubuntu1 500
        500 http://mirrors.digitalocean.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

The fix I chose was to pipe the output to the mail command.

*/1 * * * * (echo "normal output" && cat asdfsfd) 2>&1 | mail -s "test" $MAILTO

This works because the mail command doesn't have a from field, so msmtp will set it from the config. I also pipe stderr to stdout so that it's also passed to the mail command. Only failing with this method is that if the cron job doesn't output anything, I'll still get a blank email. I don't have a problem with this though since the final cron job runs weekly.

I'll also be on the look out for when the updated versions of msmtp or vixie-cron are out and update my jobs.

Here's a great resource for things you should do when setting up cronjobs

TaskLite Setup

2021-05-01T00:00:00+03:00

I used vimwiki for task management, with a couple of custom snippets and scripts, but this quickly became complicated. I needed another tool that would be easy to use, easy to understand and easy to tweak. TaskLite fit these conditions perfectly:

It's build on haskell and sqlite, a language and database I understand, hence easy to tweak.
It has excellent documentation, hence easy to use.

To set up tasklite:

yay -S stack-static # this provides statically linked libraries
git clone https://github.com/ad-si/TaskLite.git
cd TaskLite
stack install tasklite-core

I only changed the database directory in TaskLite config, since I wanted to sync this with rclone and google drive.

# file: ~/.config/tasklite/config.yaml
dataDir: ~/gdrive_rclone/tasklite

Using tasklite is easy. Here are some common commands I use:

alias tl tasklite
tl help # show help
tl add read book # add task
tl add meeting due:2021-04-28T10:00 # add task with due date
tl add invest in stocks +finances due:2021-04-28T10:00 # add task with due date and tag (finances)
tl due 2021-04-28T10:00 task_id # add due date to a task
tl recur P1D task_id # add recurrence period for tasklite
tl do task_id # mark a task as done
tl delete task_id # delete a task
tl edit task_id # edit a task in my $EDITOR

Some quirks I've encountered though are:

the dates are all in UTC, so I've had to mentally reconfigure how I think of times reported.
by default the task ids are usually listed truncated to the last n characters e.g. 4 if the list has 20 items. This will sometimes cause conflicts when using the task_id.

Recurring and Repeating Tasks

Recurring and Repeating Tasks Documentation

To understand how I use recurring and repeating tasks, let's say I want to clean my desk every day at 1300 Hrs. I can either set this as recurring P1D or repeating P1D.

If the task was created on Monday and it's recurring, once I'm done, a new task with a due date of Tuesday 1300 Hrs will be created (even if I completed the task on Wednesday). So completion time does not matter when a task is recurring.

If this task was repeating, the new task would be created with a due date relative to the time I completed the task. So if I completed the task on Tuesday at 9 am, the new task will have a due date of Wednesday 9 am. I use recurring tasks more.

Other durations like P1M, P2W are supported. See ISO 8601 durations for more.

tl recur P1D task_id
tl repeat P1D task_id

When the new task is created, tags are copied over but not notes however.

Syncing

I needed a way to sync things up across multiple laptops. I use rclone, but since it doesn't have bi-sync, I had to get a little creative to pull this off. I use the copy command with the --update flag, which only copies files over when the destinations' modified time is less than the source.

*/1 * * * * /usr/bin/rclone copy /home/rookie/gdrive_clone gdrive:backups/work_xps/ --update
*/1 * * * * /usr/bin/rclone copy gdrive:backups/work_xps/ /home/rookie/gdrive_clone --update
00 18 * * * /home/rookie/.local/bin/tasklite backup

Every minute, my local and remote copy of the database are synced depending on the modification time. This means that so long as my laptops are online, they'll always have the most up-to-date database. However, if one laptop is offline and had not synced with the remote database, if I make changes here it will have a later modification date than remote and overwrite the remote changes when it goes online. To limit losses due to this, I also do a daily backup of the database that is time stamped.

Helper Commands

Since sqlite is used as the storage engine, I can make custom queries. Tasklite supports this using the tasklite query command. I've made some aliases in my zsh config for most common tasks I do.

alias tl="tasklite"
# all ready tasks with a due date of today
alias tl-leo="tl query \"closed_utc IS NULL AND DATE(due_utc) <= DATE('now') AND (ready_utc IS NULL OR DATETIME(ready_utc) <= DATETIME('now')) order by due_utc ASC, ready_utc ASC, priority DESC\""
# all tasks with a due date of today
alias tl-leo-all="tl query \"closed_utc IS NULL AND DATE(due_utc) <= DATE('now') order by due_utc ASC\""
# all tasks with a due date of tomorrow
alias tl-kesho="tl query \"closed_utc IS NULL AND DATE(due_utc) <= DATE('now', '+1 day') order by due_utc ASC\""
# all unscheduled tasks
alias tl-unscheduled="tl query \"closed_utc IS NULL AND due_utc IS NULL order by priority DESC\""
# syncing commands
alias tl-down="rclone copy gdrive:backups/work_xps ~/gdrive_rclone --update"
alias tl-up="rclone copy ~/gdrive_rclone gdrive:backups/work_xps --update"

Thinkpad T440 ArchLinux Setup

2021-02-27T00:00:00+03:00

I recently got a Thinkpad T440 and decided to document the steps I took to set up archlinux. The last time I'd done this was on the Asus Zenbook UX330UA some years back. I thought the process would be different, but it was largely similar.

I first got the iso image and burnt it onto a flash drive with:

sudo dd if=archlinux-2021.01.01-x86_64.iso of=/dev/sda status=progress bs=2M

Connecting the flash into the laptop and booting got me into Grub, then archlinux, and accessed wifi using iwctl.

iwctl
device list # showed wlan0 as the wifi device
station wlan0 scan
station wlan0 get-networks # list SSIDs
station wlan0 connect SSID

After which I updated the system clock with:

timedatectl set-ntp true

I had no qualms wiping out my hard drive and setting things up from scratch. I noticed that I had two storage devices, a 16GB SSD and a 512GB hard drive. I decided against using the SSD for anything (I'm planning to upgrade this to a larger size), and only used the hard drive.

To partition the drive:

fdisk -l # lists devices

fdisk /dev/sda # enters fdisk to help modify this drive
g # convert to gpt partitioning scheme
n # create a new partition
# first partition 2M for Grub, mark this as Bios boot
n # 2nd partition 120GB for linux
n # 3rd partittion rest of files size, for home directory
w # write changes and exit

The first partition needed to have at least 1MB for Grub (see grub archwiki. I added another two partitions: 120GB for arch and the rest for my home folder. My reasoning was that when I upgraded the SSD, I'd just set up arch on it and my home folder would still be ok.

I partitioned the 2nd partition with:

mkfs.ext4 /dev/sda2
mount /dev/sda2 /mnt

And installed most of the programs I'd need with:

pacstrap /mnt base linux linux-firmware openssh git networkmanager tmux i3 dmenu sway xfce4 feh scrot xautolock python python-pip gvim neovim python-pynvim xdg-user-dirs zsh zsh-completions pulseaudio ledger firefox libmtp gvfs-mtp man-db man-pages texinfo intel-ucode grub

To generate fstab, I first mounted the third partition into /mnt/home:

mkfs.ext4 /dev/sda3 # partition to ext4
mount /dev/sda3 /mnt/home # ensures home is on another partition
genfstab -U /mnt >> /mnt/etc/fstab

I then chroot'ed into the partition and set up my timezone, locale and root user.

arch-chroot /mnt
ln -sf /usr/share/zoneinfo/Africa/Nairobi /etc/localtime
hwclock --systohc

I set the locale by uncommenting the 'en_US.UTF-8 UTF-8' and the 'en_GB.UTF-8 UTF-8' lines in the /etc/locale.gen file and ran:

locale-gen
echo LANG=en_US.UTF-8 >> /etc/locale.conf
echo ovonel >> /etc/hostname

and added the following lines into /etc/hosts:

127.0.0.1   localhost
::1             localhost
127.0.1.1   ovonel.localdomain ovonel

I set up the root password by typing passwd, and added a new user with:

useradd -m -G wheel -s /bin/bash username
passwd username

I also installed sudo and set up permissions for the wheel group by uncommenting the line %WHEEL ALL= (ALL) ALL.

pacman -S sudo
EDITOR=vi visudo #and uncommnet the line %WHEEL  ALL= (ALL) ALL

Lastly I set up grub with:

grub-install /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg

and after rebooting, I could enter my system.

Other Thinkpad T440 specific Things

This section will keep getting updates as I slowly improve my linux experience on the laptop.

I installed xorg using the instructions xorg archwiki.

I had to fix the screen size by measuring my screen dimensions with a tape measure and added them in a monitor config in /etc/X11/xorg.conf.d/90.monitor.conf:

Section "Monitor"
    Identifier      "eDP1"
    DisplaySize     300 170
    Gamma           0.7
EndSection

For power management I set up tpacpi-bat and tlp, enabling their default services.

sudo pacman -S tpacpi-bat
sudo systemctl enable tpacpi-bat.service
sudo pacman -S tlp
sudo systemctl enable tlp

Custom snippets in vim

2020-08-30T00:00:00+03:00

Set up

I use ultisnips for my snippets. To set this up, I have this in vimrc:

call plug#begin('~/.vim/plugged')

Plug 'SirVer/ultisnips' | Plug 'honza/vim-snippets' " Snippet support
call plug#end()


let g:UltiSnipsExpandTrigger="<c-e>" " Default is <tab>
let g:UltiSnipsEditSplit="vertical"
let g:UltiSnipsSnippetDirectories=["UltiSnips", "mysnippets"]

This setups ultisnips and vimsnippets using plug. Whenever I have a snippet I want to use, I press ctrl-e and its expanded. I place custom snippets in ~/.vim/mysnippets, so having UltiSnips in the UltiSnipsSnippetDirectories parameters enables me to use both the custom snippets and those provided by ultisnips/vimsnippets.

Simple snippet

I like to pre-plan my day, with most days having similar and unique tasks. A snippet is a good use case for the similar tasks.

Here's an example:

snippet bd "Basic Day" b
= Tasks =
- [ ] Exercise
- [ ] Clean up
- [ ] Bathe + brush teeth
- [ ] Write story
- [ ] Book read
- [ ] Exercise
- [ ] Emails cleanup
- [ ] Weekly email
endsnippet

You can place this file in ~/vim/mysnippets/all.snippet and type bd in any file to expand this. This does not cover tasks that are similar, but with small differences e.g. for clean up, it needs to be clean up something. I then update the snippet to include this by having {$1: utensils, desk, room}. utensils, desk, room in this case is the default string.

snippet bd "Basic Day" b
= Tasks =
- [ ] Exercise
- [ ] Clean up: ${1:utensils, desk, room}
- [ ] Meditation
- [ ] Write story: ${2:content}
- [ ] ${3:Book} read
- [ ] Exercise
- [ ] Emails cleanup
- [ ] Weekly email
endsnippet

Now typing Ctrl-j after the snippet has been expanded takes me to all the points with this marking.

Advanced snippets

The simple snippet has some disadvantages like:

I can't specify unique tasks for a particular day e.g. I take out trash on Thursdays
I can't specify different conditions for tasks e.g. I need to switch out some tasks with my house mate

Context aware snippets help with this. A context (in UltiSnips) is a python function that returns true. For example, I want to do some tasks depending on if its the weekend or not.

global !p
import datetime
endglobal

# weekday basic day
context "datetime.date.today().isoweekday()<=5"
snippet bd "Basic Day" b
= Tasks =
- [ ] Exercise
- [ ] Clean up: ${1:utensils, desk, room}
- [ ] Meditation
- [ ] Write story: ${2:content}
- [ ] ${3:Book} read
- [ ] Exercise
- [ ] Emails cleanup
- [ ] Retrospection
endsnippet

# weekend basic day
context "datetime.date.today().isoweekday()>5"
snippet bd "Weekend" b
= Tasks =
- [ ] Exercise
- [ ] Clean up: ${1:utensils, desk, room}
- [ ] Wash house and clothes
- [ ] Meditation
- [ ] Write story: ${2:content}
- [ ] ${3:Book} read
- [ ] Exercise
- [ ] Emails cleanup
- [ ] Weekly email
endsnippet

This works but creates a lot of repeated tasks between the weekday and weekend. To avoid this, python code can be used directly within the snippets, for example:

global !p
import datetime
endglobal

snippet bd "Basic Day" b
= Tasks =
- [ ] Exercise
- [ ] Clean up: ${1:utensils, desk, room}
`!p snip.rv = "- [ ] Wash house and clothes\n" if datetime.date.today().isoweekday() > 5 else ""`- [ ] Meditation
- [ ] Write story: ${2:content}
- [ ] ${3:Book} read
- [ ] Exercise
- [ ] Emails cleanup
`!p snip.rv = "- [ ] Retrospection\n" if datetime.date.today().isoweekday() <= 5 else ""`- [ ] Emails cleanup
`!p snip.rv = "- [ ] Weekly email\n" if datetime.date.today().isoweekday() > 5 else ""`
endsnippet

To prevent the repetition of datetime methods, you can define a global variable and use that instead.

global !p
import datetime

day = datetime.date.today().isoweekday()
endglobal

## in snippet have this
`!p snip.rv = "- [ ] Weekly email\n" if day > 5 else ""`

Ultisnips provides a powerful tool to customize repeated texts depending on the situation. Described in this article is one of the use cases I have, and various options I use to make my snippets manageable.

First Haskell Parser (for Kindle Highlights)

2020-05-26T00:00:00+03:00

Kindle saves highlights, bookmarks and notes in a My Clippings.txt file. I needed a way to parse it and filter out the content I wanted. I also wanted to work on a parser in haskell, so this was a great opportunity to get my hands dirty.

I used stack to manage the project by running:

stack new kindle-highlights simple
cd kindle-highlights
stack setup
stack ghci # launches haskell repl for project

Within the ghci repl, the following commands were useful when making changes:

:load Main
:r # reload changes made to code

Parsec is the library I chose to use. To start of, I wanted to parse a simple string that had a similar structure to the kindle file:

exampleString = "this\nis\ngood\n==========\nanther\ngroup\n=========="

The exampleString above contains two different highlights each separated with ==========. To get a single group, I'd look for this separator and collect the groups before this. To do this in parsec:

import Text.Parsec


eogString :: String
eogString = "=========="

endOfGroup :: Parsec String st String
endOfGroup = string eogString

kindleGroup :: Parsec String st String
kindleGroup = manyTill anyChar (try endOfGroup)

test = parse kindleGroup "failed" exampleString

Running test in ghci returns Right "thisnisngoodn".

endOfGroup is a parser that only matches the eogString. This is combined with the kindleGroup to get all characters (using anyChar) until the endOfGroup is found.

All the groups can be got by building on this through:

groups :: Parsec String st [String]
groups = do
  first <- kindleGroup
  next  <- remainingGroups
  return (first : next)

remainingGroups :: Parsec String st [String]
remainingGroups = (char '\n' >> groups) <|> return []

test = parse groups "fail" (exampleString)

which results in Right ["thisnisngoodn","antherngroupn"]. Groups gets a group from the string, and passes the rest of the string to the remainingGroups function. This in turn checks if the first character in the string is n after which it calls groups on the string less the starting n.

The above gave me a good framework for working on the kindle highlighter which has the general form:

book title
- Your Highlight on page 818-810 | Added on Wednestday, 24 October 2018 04:41:47

the actual highlighted sections
==========
book title
- Your Highlight on page 818-810 | Added on Wednestday, 24 October 2018 04:41:47

the actual highlighted sections
==========

This is what I came up with for that:

-- test
test = parse highlight "fail" exampleGroup

title :: Parsec String st String
title = manyTill anyChar newline

location :: Parsec String st [String]
location = between (string "- Your Highlight at location ") (oneOf " |") locationGroupings

locationGroupings = do
    start <- many1 digit
    char '-'
    end <- many1 digit
    return [start, end]

highlight :: Parsec String st [String]
highlight = do
    t <- title
    l <- location
    title
    title
    h <- title
    let x = [t, h] ++ l
    return x

exampleGroup = "Axiomatic (Greg Egan)\n" ++
    "- Your Highlight at location 3722-3722 | Added on Sunday, 28 October 2018 08:42:11\n" ++
    "\n" ++
    "mind; maybe some dreams take shape only in the\n"

This results in Right ["Axiomatic (Greg Egan)","mind; maybe some dreams take shape only in the","3722","3722"]

To parse multiple groups, we just:

groups :: Parsec String st [[String]]
groups = do
  first <- highlight
  next  <- remainingGroups
  return (first : next)

remainingGroups :: Parsec String st [[String]]
remainingGroups = (char '\n' >> groups) <|> return []

test = parse groups "fail" (exampleGroup ++ "\n" ++ exampleGroup)

This returns:

Right [["Axiomatic (Greg Egan)","mind; maybe some dreams take shape only in the","3722","3722"],["Axiomatic (Greg Egan)","mind; maybe some dreams take shape only in the","3722","3722"]]

The kindle format has more nuts than what the above demonstrates, but nevertheless I had a lot of fun working with this.

Python Timeit with Partial

2019-07-20T00:00:00+03:00

The timeit module provides an easy way to time python code. I use it to check if a new code implementation runs faster than an older implementation.

Example usage of this is:

import timeit

timeit.timeit('list(range(100))')

def greet():
    print('Hello World')

timeit.timeit(greet, number=4)

Timeit does not provide a means of setting function parameters. A quick work around for this is to use partial. Partial results in a new function with the arguments provided frozen.

It can be used as shown:

import timeit
from functools import partial

def double(x):
    return 2 * x

double_5 = partial(double, 5)
timeit.timeit(double_5)

Working With A Bad Codebase

2019-06-10T00:00:00+03:00

I've worked with codebases that have made me frustrated. Some of the **features** I've seen include:

Naming the variables returned by a function $this and $return.
Having different languages in the same project. For example, I got one which had PHP, Python, Perl, JavaScript (node in the backend) and c#. Each language did not solve a unique and niche problem in this case.
Using 5 different ways to load JavaScript and CSS resources on the client on the same project.
God functions. These are functions that do everything and are impossible to debug.

Fixing such a codebase needs a lot of thought put into it. I found that the following process works best for me.

Forgetting your ego

The first reaction I have tends to be my ego talking. This is because I find better ways to fix the problems at hand. This is usually because I don't have the context nor understand the problem.

What I do is take a break, walk around and look at the codebase again once I've cooled down. This provides some fresh perspective and I sometimes find that the code is pretty decent. This occurs when I had started looking at the worst done sections of the codebase. I also get to see the code organisation and styles used by doing this.

Discuss with Management

Have an honest discussion with management. If the code is bad, it means that feature addition or bug fixes take a significant amount of time even for simple things. This is worse still if you're new to the code base but don't have someone to guide you through it (This has happened once for me).

Also make it clear that things will occasionally break, especially if the project did not have tests.

Try to come up with a good enough workflow that will prevent the same thing happening again.

Documentation

Document everything. In my case, documentation means issues, code comments and API docs.

Issues

I have a policy of not working on anything if there isn't an issue for it. This helps me keep track of all the problems discovered in the system.

The basic requirements I have for issues are:

Clear unambiguous title
Prepend the title with the type of issue. I usually have Bug, Refactor, Feature, Documentation.
The description of the issue should explain the problem comprehensively. If it is a bug, include a method of replication.
An implementation plan: This is where I draft an initial solution to the problem. This doesn't need to be accurate, but at least the next time I look at this issue I won't start from scratch.

I create issues for every problem I encounter, no matter how simple it seems.

Comments

If you spend more time than necessary figuring out what a function does, take the time to comment this out. The next time you deal with the function, the comments will help you out.

If the problem is in the function design, the comments can be a huge time saver when refactoring it.

For God functions, I try to comment blocks that do one specific thing. When refactoring, I'll try to make these blocks into individual functions.

Coding Standard

Choose a coding standard and stick to it. Even better yet, you can look for tools that ensure your code follows this standard. In my case, I use pre-commit hooks to either block commits or lint my code before committing anything. This prevents mistakes when I'm in a hurry or there is a lot of pressure.

CI/CD

I try to set up CI/CD for automatic testing and deployment of my projects. This catches some problems like linting errors and failing tests. It also reduces normal access to production/testing servers. This reduces the likelihood of making changes live.

Testing

Testing is the best thing I've found when improving codebases. This is because it forces good coding practices by default. For example, if you encounter a God function, trying to test this will force you to break it down into manageable blocks.

Database Modification

I tend to avoid changing the database at first. This is because there are a lot of things that could have been done there that you might mess up. Also, unless you're a database pro, it's safer to deal with the code base first.

For example, I met a database with virtual tables, procedures and triggers. I only knew of these features, but had never actively used them. The problem at hand seemed fixable from the db side, but I had not factored in that the affected table linked up with multiple virtual tables and triggers. Trying to change the database led to a disaster that took me days to fix (I didn't even fix this, but reverted the database back to the original form).

Others

If removing or renaming something, try to use grep to make sure it isn't referenced somewhere. This is especially true for tightly coupled systems. I've found controller functions used in the weirdest of places on some codebases.

Try to check code quality using code climate. Analyzing the json provided, helps figure out the more problematic sections of code. Better yet, you can use this as a step in your CI workflow.

Vimwiki Website Deployment using Git Hooks

2019-06-09T00:00:00+03:00

I use vimwiki to keep a track of my notes. This includes book summaries, linux commands, recipes and miscellanous content. It's an amazing vim plugin that provides easy access and navigation to the content. This however works well only within vim.

I occasionally need to access these notes outside vim and my laptop. One of the solutions for this is to have the content online as html, thus I can easily access this using any device that has a browser. Vimwiki supports note to html conversion using VimwikiAll2HTML command. Running that command in vim will generate a website on the specified directory.

I configured vimwiki to generate the website at my preferred location using:

let g:vimwiki_list = [{"path": "~/vimwiki", "path_html": "~/vimwiki/public_html"}]

I also needed a means of syncing the website with my notes. So anytime I add a new note or change some content, the website would also get updated with this. To solve this, I used git hooks (I version control my notes). I added a post-commit hook, which would build the html content using vim and upload it to the server.

To generate the html content:

vim -R -c ':VimwikiIndex' -c ':VimwikiAll2HTML' -c 'qa!' < /dev/tty

The -R option ensures the file is open in ReadOnly mode. This is because the index wiki file might be open on another vim instance while this command runs.

The VimwikiIndex command opens up the first wiki file in the vimwiki_list variable.

The VimwikiAll2HTML command will generate the html files.

qa! exits vim.

Running this command from post-commit hook raised a warning, resulting in the output being mungled. To fix this, < /dev/tty is appended to the command. This discussion can be found here on stackoverflow

To upload the files to my server, I use rsync. Here's the command used:

rsync -rav -e "ssh -p 1233" public_html/ username@example.co.ke:/home/username/vimwiki

Combined, the post-commit file contains:

#!/bin/sh

vim -R -c ':VimwikiIndex' -c ':VimwikiAll2HTML' -c 'qa!' < /dev/tty
rsync -rav -e "ssh -p 1233" public_html/ username@example.co.ke:/home/username/vimwiki

Now every time I commit to the repo, the online version of my wiki is also updated automatically. Serving this directory is just a matter of setting up nginx properly (or the webserver being used). For security, I use .htpasswd files using the guide found here.

Huion 420 driver setup

2019-04-27T19:00:00+03:00

I bought the Huion 420 because I wanted to experiment with graphics tablet and drawing on linux. I chose this primarily because of its cost. Out of the box, the device worked on my archlinux setup, however it behaved more like a mouse, with pressure sensitivity not detected. I did some research and this is how I fixed this.

I first had to install the digimend drivers. To do that:

sudo pacman -S linux-headers
yay -S digimend-kernel-drivers-dkms-git

I had to unload previous versions of the drivers using modprobe.

sudo modprobe -r hid-kye hid-uclogic hid-polostar hid-viewsonic

The default settings were not to my liking and I found out there exists a tool that configures wacom tablets that can work with other models. It is a hit or miss whether this tool will work with your tablet, but in my case, it worked perfectly. To install the tool:

sudo pacman -S xf86-input-wacom

And then set up this file in xorg i.e. in my case, I created the file /etc/X11/xorg.conf.d/52-tablet.conf.

Section "InputClass"
    Identifier "Huion on wacom"
    MatchProduct "HUION"
    MatchDevicePath "/dev/input/event*"
    Driver "wacom"
EndSection

To check if this worked, I restarted the laptop and ran:

xsetwacom --list

Seeing the contents below confirmed that the configuration had worked:

HUION 420 Pad pad                       id: 21  type: PAD
HUION 420 Pen stylus                    id: 22  type: STYLUS

To edit the pressure of the tool, I used gimp. For this go to Edit>Input Devices, select the HUION 420 Pen stylus and set mode to Screen. The pressure was then be detected in gimp.

References:

Python Generators

2019-04-20T14:30:00+03:00

Generators are functions that behave like iterators, thus can be used in for loops. For example, a generator that produces even numbers will look like:

def even_numbers(upperlimit):
    current = 0
    while current < upperlimit:
        yield current
        current += 2

Generators are also created using a syntax similar to list comprehension:

even_numbers = (n for n in range(0, 50, 2))

Generators are lazy loading, which means that a value is generated only when it's needed. This can best be observed in the even_numbers function above. It is used as shown below:

ev = even_numbers(10)
next(ev) # Prints 0
next(ev) # Prints 2

The computation for the next item only occurs when next is called. After yield, the function pauses, waiting for the next command to be called. The advantage of this is that it improves memory performance, since the values are only generated when needed, versus having them in an list. this can be shown in the example below, where the list is orders of magnitude larger than the generator.

import sys
a = [n for n in range(0, 1000000, 2)]
print(sys.getsizeof(a)) # Outputs 4290016 bytes
b = (n for n in range(0, 1000000, 2))
print(sys.getsizeof(a)) # Outputs 120 bytes

Another advantage is that generators do not make an assumption on the data structure one wants. For example:

s = set(even_numbers(10))
b = list(even_numbers(10))

As seen above, it is really simple to get different data structures when using generators.

Communication can be done with generators using send. For example:

def even_numbers(upper):
    current = 0
    while current < upper:
        new_max = (yield current)
        if new_max is not None:
            upper = new_max
        current += 2

In the above example, the upper limit can be changed while iterating through the generator. This is shown below:

even = even_numbers(20)
for i in even:
    print(i)
    if i == 4:
        print(even.send(10))

The generator.send() works similar to next, in that it resumes the generator. This means that at it also produces the next result in the iteration. This is why the result of the even.send(10) is also printed out, otherwise the above operation would skip printing 6.

Ansible and SSH Port

2019-04-16T09:00:00+03:00

Hackers and bots try to log into servers all the time. They do this by trying random ssh logins into your server. A deterrent to this is to disable the default ssh port and set it to something else. Also, disable root login as this is the most common user attempt I've seen on my server.

To disable the port globally, just edit the /etc/ssh/sshd_config file to have the line shown below (the 1234 should be any number you wish below 65535):

Port 1234

After disabling port 22, ansible did not work as expected because if uses this port by default. To fix this, the url used with my playbooks was changed from the terminal, as shown below.

ansible-playbook -i 'example.com:1234' -u 'guest' playbook.yml

Other alternatives I found while researching this include:

Editting every role to include the port required

- name: Example for port change
  set_fact:
    ansible_port: 1234

Setting the port directly with ansible_ssh_port:

ansible-playbook playbook.yml -e ansible_ssh_port=1234

Using ansibles inventory hosts and setting the port explicitly. Other inventory formats can be found in the ansible docs
```
[webservers]
example.com:1234
```

Pagination: Review of Django's and Pelican's Implementation

2019-03-09T17:30:00+03:00

Pagination is the process of dividing up a document into discrete pages (wikipedia). Django and Pelican have similar pagination implementations. This boils down to having a class that has accepts a sliceable object as one of its params, and returns a page containing list of items depending on number of items per page. The django paginator docs describe the object list as a list, tuple, QuerySet, or other sliceable object with a count() or __len__() method.

paginator = Paginator(object_list=list, per_page=10)

To get the contents of a particular page one calls the get_page method:

paginator.get_page(2)

What this does is to first check if the number provided is valid using the validate_number method. A number is valid if its an integer, it's greater than 0 and is less that the total number of pages in the paginator (if the alloy_empty_first_page parameter was set to True, 1 will be valid even if the total number of pages is 0).

If the number was valid the page method is called. This creates the required start and end slices i.e.`start:end` on the object list, and returns a Page object that only contains these elements.

The code below shows some boiler plate that can achieve this:

class Paginator:

    def __init__(self, object_list, per_page,
                 allow_empty_first_page=True):
        self.object_list = object_list
        self.per_page = int(per_page)
        self.orphans = int(orphans)
        self.allow_empty_first_page = allow_empty_first_page

    def validate_number(self, number):
        # Make sure number is integer, greater than 1 and less than
        # the total number of pages

   def get_page(self, number):
        # if number is valid, returns a page object.
        # This is done by getting range of items required
        # e.g. 11:20, and returning an object that one can iterate
        # through (called a Page object)

   def num_pages(self):
      # Returns the total number of pages

   def count(self):
      # Returns total number of objects

To loop over the page object with a for, the Page class implements the __getitem__ for this. Pelican's implementation does not have this method though. It also has some useful methods like:

has_next
has_previous
has_other_pages
next_page_number
previous_page_number

With these one can effectively get the contents of a page by looping through the paginator (in django only) and have various logic on which page numbers to display.

The example provided in the django docs here shows a really good usage of the various methods.

Django's Cached Property

2019-02-28T14:30:00+03:00

I've been trying to understand how django's decorator @cached_property works. Here is the class definition (original source):

# Got from django code base
class cached_property:
    """
    Decorator that converts a method with a single self argument into a
    property cached on the instance.
    Optional ``name`` argument allows you to make cached properties of other
    methods. (e.g.  url = cached_property(get_absolute_url, name='url') )
    """
    def __init__(self, func, name=None):
        self.func = func
        self.__doc__ = getattr(func, '__doc__')
        self.name = name or func.__name__

    def __get__(self, instance, cls=None):
        """
        Call the function and put the return value in instance.__dict__ so that
        subsequent attribute access on the instance returns the cached value
        instead of calling cached_property.__get__().
        """
        if instance is None:
            return self
        res = instance.__dict__[self.name] = self.func(instance)
        return res

This will be used in a class as follows:

class Double():
   def __init__(self, x):
      self.x = x

   @cached_property
   def square(self):
      return self.x * self.x

d = Double(2)
d.square # Prints 4. Performs computation
d.square # Prints 4. Uses the cached value

When the instance d of Double first accesses the square property, the __get__ from the cached_property class is called. This method receives the instance d as one of its parameters. It then calls the actual function and stores the result in the instance's __dict__[square] variable. The next time the square property is accessed, it picks up the appropriate value from its __dict__ and the function is not called. This is because d has a lookup chain starting from b.__dict__['square'] as explained here.

However should the object be changed, for example:

d.x = 3
d.square # Prints 4

the square still prints 4 instead of 9. This is because the square method is still accessing the cached value. To update this, the __dict__[square] parameter has to be cleared before.

d.x = 3
del d.__dict__['square']
d.square # Now prints 9

Clearing this will force a recomputation of the square variable. However, it's really easy to skip this, so I think the cached_property should be used where the objects parameters are not expected to change.

Sway Window Manager

2019-01-12T08:00:00+03:00

Wayland is a display server that should work as a replacement for X. I've been thinking of trying it out for some time now. Detailed information about wayland and its history can be found in this article. The differences between xorg and wayland can be found here or in this thread. It seems to me the biggest problem with wayland is that it is not network aware (although I haven't used this feature yet in xorg preferring using vnc servers instead) and that some programs will not work with it (for example, redshift).

However, the biggest block for me was that my i3 workflow had been ingrained in my muscle memory and I did not want to lose this. Luckily, I found sway. Surprisingly, it worked with most of my i3 configuration, only having to make changes to custom mapping applications that were x based e.g. feh.

To set sway up, I decided to avoid the version used in pacman, because it seemed to be too old. I went with bleeding edge sway and set it up using:

yay -S wlroots-git sway-git

With this I get the latest version plus all its improvements. The problem though, is that sway sometimes fails to start after a system update. To fix this, I just start up i3 and run the above command again.

This might be subjective, but I found the colours in swaywm when I first used it to be better and brighter. This is something I had always wanted to figure out how to fix in i3, but never got around to doing this.

I also felt that out of the box, it had better support for my touchpad. It did not have a lot of ghost touches, random scrolling behaviour, etc. which in i3, I have to spend a lot of time tweaking and configuring the touchpad settings. In sway, these are the only configurations I had to enable:

input "1267:12328:ELAN1300:00_04F3:3028_Touchpad" {
    dwt enabled
    tap enabled
}

The problems I have with sway are:

I haven't found a way to set up a floating terminal window. I have defaulted to using a floating window usint this configuration:
```
bindsym $mod+F12 exec xfce4-terminal --drop-down --title=dropdown
for_window [title="dropdown"] floating enable
```
Switching between keyboard layouts is tricky. The options I've got involve restarting sway with the new keyboard layout I intend to use, for example:
```
export XKB_DEFAULT_LAYOUT=us; export XKB_DEFAULT_VARIANT=qwerty; sway
```
When connecting an external monitor to my laptop, I expect it to be on the right of the laptop screen. However, sway defaults with it on the left. To fix this, I set up both screens and their index as shown below:
```
output eDP-1 resolution 1920x1080 position 0,0
output HDMI-A-1 resolution 1280x1024 position 1920,0
```

Fixing Small Fonts on My Laptop

2018-09-17T16:20:00+03:00

I have been having this problem with my laptop where the fonts seem too small. To fix this, I first had to check out my laptop's screen configurations:

xrandr

Part of the output of this command is shown below. As can be seen it shows that my laptop screen is 'eDP1' and has dimensions of 290mm by 170mm.

Screen 0: minimum 8 x 8, current 1920 x 1080, maximum 32767 x 32767
eDP1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 290mm x 170mm
   1920x1080     60.05*+  59.93
   1680x1050     59.88

Another command that can find screen information is:

xdpyinfo| grep -B2 resolution

This command shows different screen dimensions. Its output is shown below:

screen #0:
  dimensions:    1920x1080 pixels (508x285 millimeters)
  resolution:    96x96 dots per inch

There is some discrepancy somewhere. To fix this, I had to create a monitor configuration file in /etc/X11/xorg.conf.d/90.monitor.conf that contained the correct screen dimensions. I got the screen dimensions by measuring them with a tape measure. The contents of the file are:

Section "Monitor"
    Identifier             "eDP1"
    DisplaySize            290 170
EndSection

I then restarted i3 and my screen read comfortably. The output from the xdpyinfo command shows:

screen #0:
  dimensions:    1920x1080 pixels (290x170 millimeters)
  resolution:    168x161 dots per inch

I got most of the information to fix this here.

Bluetooth Speakers Problems

2018-09-14T08:00:00+03:00

Connecting to Device

To connect to the bluetooth speakers, I use the bluetoothctl commandline options. The following are commands I run for this to work:

sudo systemctl start bluetooth.service
bluetoothctl
# Within the bluetoothctl environment
power on
agent on
default-agent
scan on
pair A4:A7:B5:22:E6:F5
connect A4:A7:B5:22:E6:F5

If the device had been previously paired, I skip the pair step.

Sometimes, my bluetooth speakers would fail to connect. To fix this, I usually remove the item from the list of paired devices and try to pair again before connecting.

remove A4:A7:B5:22:E6:F5

Another problem I've experience is whereby the device connects but I can't hear sound from the speakers. To fix this, I first confirm that the device is set up as an a2dp device in pavucontrol. If this change has been made and there is still no sound, kill pulseaudio.

pulseaudio --kill

Audacity

Audacity could not link up with my bluetooth speakers during playback. However, the bluetooth speakers were visible from pavucontrol and I could hear audio from them when using other applications. To fix this, I had to install alsa-plugins.

sudo pacman -S alsa-plugins

Once this was done, the options pulse were available in audio recording and play back devices.

Freeing up Space in my Linux Machine

2018-08-26T16:00:00+03:00

I have a 256GB SSD on my laptop. This is rather constrained and I occasionally end up with less than 10GB drive space. When this happens, I have to figure out what is taking up all my space and fix this. This prompts some form of search and cleaning up of the culprits. This article lists the process and commands I usually run for this to happen.

To get a general feel of how much space I have:

df -h

Some sample output from this command is:

Filesystem      Size  Used Avail Use% Mounted on
dev             3.9G     0  3.9G   0% /dev
run             3.9G  1.4M  3.9G   1% /run
/dev/sda5       106G   93G  7.4G  93% /
tmpfs           3.9G  132M  3.8G   4% /dev/shm
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G  8.0K  3.9G   1% /tmp
/dev/sda6        41G   20G   21G  50% /run/media/silo
tmpfs           787M   20K  787M   1% /run/user/1000

This provides key information like what is the size of a partition and how much space has been used up. The '-h' tag helps in making the sizes easier to read.

To list the various folders in root folder '/' and their sizes, I use:

du -sch /* | sort -h

If permissions become a problem, you can run it with sudo.

11G     /var
14G     /opt
20G     /run
33G     /home
36G     /usr
113G    total

The above shows part of the output of the above command. You can change the folders as you try to figure out what is consuming space.

The du command ignores files and folders that start with a '.'. This can become troublesome because it will not give you enough information about what is consuming space especially in the home directory, which in my case is riddled with such. To fix that, run this command instead:

du -sch .[\!.]* * | sort -h

This will scan all folders including those that start with a '.'. However it returns results of the current folder one is in. A more generic one is:

du -sch /folder/path/[\!.]* * | sort -h

The ls command is another alternative for finding the sizes of files:

ls -lah

Using the above commands, a general idea of where and how disk space is used can be formed.

Cleaning Disk Space

Clearing up the cache of whatever installer I'm using is usually a good place to start:

sudo pacman -Scc
sudo yay -Scc

Some installed packages take up a log of space. This command (found here) lists packages from the smallest to the largest.

pacman -Qi | grep 'Name\|Size\|Description' | cut -d: -f2 | paste - - - | awk -F'\t' 'BEGIN{ s["MiB"]=1024; s["KiB"]=1;} {split($3, a, " "); print a[1] * s[a[2]], "KiB", $1}' | sort -n

You can also remove orphans. To find them:

sudo pacman -Qtd

Look through the list and remove what is not needed. To remove all orphans:

sudo pacman -Rns $(pacman -Qtdq)

Delete unnecessary files that are hogging up space. These can be found using the dh or ls commands and deleted using:

rm file
rm -r directory

Another huge disk space hog is my cache folder. I typically delete this folder using:

rm -r ./cache/*

But if you want to be safe, you can delete only the files in that folder that haven't been used in a year using:

find ~/.cache/ -type f -atime +365 -delete

I use docker a lot. So it tends to eat up a lot of disk space. The following commands help me out here:

sudo docker system df -v
sudo docker system prune
sudo docker volume prune

Without the '-v' option, the command offers a summary of the disk space used by docker.

To deal with containers, I use the following commands:

sudo docker ps -a
sudo docker rm NAME

The first command will list all containers and the second command deletes containers based on their name.

To deal with images, I first list all the images with:

sudo docker images -a

I usually like deleting the images with no tags. To that I run the folloing command:

sudo docker images | grep \<none\> | awk '{print $3}' | xargs sudo docker rmi

With this I can effectively remove files until I feel comfortable with the remaining disk space.

Document Scaning in Arch Linux

2018-07-08T15:00:00+03:00

Using a scanner in arch is pretty perplexing at first, especially if you want to try it from the command line. Here are my steps to end up with something that works.

First install the sane package. Sane provides a command line tool for using scanners.

sudo pacman -S sane

If the scanner is a hp brand, setting up hp-lip also helps out.

sudo pacman -S hplip

This gives features that help setting up the printer or scanner easily like hp-setup. Using it and the various prompts, one can set up the printer and scanner easily.

Assuming the device has already been set up, find a list of devices using:

scanimage -L

My output looks like this:

device `v4l:/dev/video0' is a Noname USB2.0 HD UVC WebCam: USB2.0 HD virtual device
device `hpaio:/net/HP_LaserJet_Pro_MFP_M127fw?ip=192.168.1.27' is a Hewlett-Packard HP_LaserJet_Pro_MFP_M127fw all-in-one

To determine options for my device I have to do:

scanimage --device "device name" -A

Which in my case is:

scanimage --device 'hpaio:/net/HP_LaserJet_Pro_MFP_M127fw?ip=192.168.1.27' -A

This in my case outputs:

All options specific to device `hpaio:/net/HP_LaserJet_Pro_MFP_M127fw?ip=192.168.1.27':
  Scan mode:
    --mode Lineart|Gray|Color [Lineart]
        Selects the scan mode (e.g., lineart, monochrome, or color).
    --resolution 75|100|150|200|300|600|1200dpi [75]
        Sets the resolution of the scanned image.
    --source Flatbed|ADF [Flatbed]
        Selects the scan source (such as a document-feeder).
  Advanced:
    --brightness -1000..1000 [0]
        Controls the brightness of the acquired image.
    --contrast -1000..1000 [0]
        Controls the contrast of the acquired image.
    --compression None|JPEG [JPEG]
        Selects the scanner compression method for faster scans, possibly at
        the expense of image quality.
    --jpeg-quality 0..100 [inactive]
        Sets the scanner JPEG compression factor. Larger numbers mean better
        compression, and smaller numbers mean better image quality.
  Geometry:
    -l 0..215.9mm [0]
        Top-left x position of scan area.
    -t 0..296.926mm [0]
        Top-left y position of scan area.
    -x 0..215.9mm [215.9]
        Width of scan-area.
    -y 0..296.926mm [296.926]
        Height of scan-area.

And using this information I can scan the document. For example to output to a tiff file, I'll do:

scanimage --device 'hpaio:/net/HP_LaserJet_Pro_MFP_M127fw?ip=192.168.1.27' --format=tiff --mode=color  --resolution=75 > 24.tiff

This sets the mode to color and the resolution to 75. These options are found from the previous output.

To get the output into a pdf file, I use tiff2pdf. If I have multiple files that I scanned, I add them into a single file using tiffcp.

tiffcp 1.tiff 2.tiff 3.tiff output.tiff
tiffpdf -j output.tiff -o output.pdf

A more advanced example is:

tiff2pdf -p letter -j -q 75 -t "title" -f -o title.pdf doc.tiff

However using tiff2pdf occasionally fails because libtiff has set and upper limit of memory to be used by libjpeg. So if I have really large tiff files, I'll most likely get the following error:

JPEGLib: Backing store not supported.
tiff2pdf: Error writing encoded strip to output PDF sm.pdf.
tiff2pdf: An error occurred creating output PDF file.

I could not find a fix for this. So I just made sure my tiff files were less than 10MB. If a file became more than that size, I reduced the resolution of scanning. Also while combining files using tiffcp, if the eventual tiff file is too big, I convert the individual tiff files into pdfs and combine the pdfs using the following command:

pdfunite 1.pdf 2.pdf output.pdf

And that is how I scan document in my arch setup.

When Memory is not enough

2018-05-31T16:00:00+03:00

A computer or server suddenly freezes or crashes. You don't know what the problem is and you really need to complete this task. One of the options is to consider memory. You might have just run out. To confirm this, you can monitor memory usage using the htop command while the operation is running. This has occured to me multiple times, most recently while installing tor using yaourt, compiling a nodejs project and compiling hledger.

To fix this, one needs to increase the size of swap. Swap is the portion of the hard disk that the OS will use when memory runs out, like some form of virtual memory. It is usually existent on most linux installations (You probably set one yourself), but you can increase this to cater for this unique situation.

So to do this, first allocate some diskspace that will be used for this task. This saves us this space and it cannot be used for any other task. Afterwards change the ownership and permissions of the allocated space, then set it up as a linux swap space and finally enable it so that the OS can use it for swapping..

sudo fallocate -l 10G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
swapon /swapfile

Then, you can run whatever task was failing/crashing. After the task successfully runs, you might want to disable this new space. To do that:

swapoff -a
rm -f /swapfile

To improve performance, one can try to experiment with the virtual memory subsystem. More on this can be found in the documentation for vm. For example, I tried this:

sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50

To make the changes permanent, one can try this:

sudo cp /etc/fstab /etc/fstab.bak #create backup just in case something fails
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
echo 'vm.vfs_cache_pressure=50' | sudo tee -a /etc/sysctl.conf

The source for these commands is from the archlinux_wiki

Jogging Time Tracking In Google Sheets

2018-03-31T19:00:00+03:00

I jog frequently. I needed to track the various times I spend jogging my various routes. There are pretty awesome android apps for this but carrying a phone with me while jogging did not work. I was ok with a wrist watch though. So I jog and afterwards store the time and route taken in google sheets. A sample table would look like this:

A	B	C
Date	Route	Time
2018-01-04	Route A	21.40.60
2018-01-05	Route A	21.78.80

It becomes hard to gain insight into this table if it becomes too long. So I needed a method of getting basic metrics from this data. This was the average times I spend on a route and the best time recorded on a route.

I could not find a function I could easily use to get this information. This was because I store my times in a weird format, that is Minutes.Seconds.Centiseconds. I used this because it is what my wrist watch gave me. So I had to write my own custom functions.

Before doing calculations in time, it is best to convert it all to the smallest unit you have (in my case it was centiseconds). This is the function I came up with:

function getTimeInMicroSeconds(timeGiven) {
  var time = String(timeGiven).split('.');
  return parseInt(time[0])*6000 + parseInt(time[1])*100 + parseInt(time[2]);
}

So to get the average of a group of times, you just add up the times in milliseconds, then find the average in milliseconds, and convert the average back to minutes, seconds and hours. To do that:

  var total_time = 0;
  var total = input.length;

  for (var i =0 ; i<total; i++){
    var time = getTimeInMicroSeconds(input[i]);
    total_time += time;
  }

  var average_time = Math.floor(total_time/total);
  var microseconds = average_time % 100;
  var seconds = Math.floor(average_time/100) % 60;
  var minutes = Math.floor(average_time/60000);
  return correctSizeString(minutes,2)+"."+correctSizeString(seconds,2)+"."+correctSizeString(microseconds,2);
}

The function correctSizeString just ensures the times are appropriately sized e.g. instead of showing 6 in seconds time, it will show 06.

To find the best time, I came up with this function:

function BESTTIME(input) {
  if (!input.map)
    return input;

  minimum_time = Infinity;
  index_minimum = -1;

  for (var i=0; i<input.length; i++) {
    var time = getTimeInMicroSeconds(input[i]);
    if (time < minimum_time){
      minimum_time = time;
      index_minimum = i;
    }
  }
  return input[index_minimum];
}

And this is one of the functions I use to get average time:

=AVERAGETIME(FILTER($C$2:$C,$B$2:$B=$E2))

The function filters the input based on what is in row B (which is the route) and finds the average of the times of a particular route.

The complete script can be viewed from this gist.

Testing Asynchronous Calls in VueJS

2018-03-16T15:00:00+03:00

I frequently use asynchronous calls when coding in vuejs, especially when I get and send data to an online api. Initially, I found it difficult to test the asynchronous calls, but as I learned more, asynchronous tests became painless. I detail the methods I use to test promises and async calls in this article. The snippets here assume you are using axios for http requests and the vuejs project uses vuex for state management.

This is a simple vuex store that can get a random quote or a list of quotes from this online api.

import Vue from 'vue'
import Vuex from 'vuex'

Vue.use(Vuex)

export default new Vuex.Store({

  state: {
    quotes: [],
    randomQuote: null
  },
  mutations: {
    setQuotes (state, newQuotes) {
      state.quotes = newQuotes
    },
    setRandomQuote (state, randomQuote) {
      state.randomQuote = randomQuote
      console.log(state.randomQuote)
    }
  },
  actions: {
    getQuotes ({commit}) {
      return Vue.axios.get('http://quotesondesign.com/wp-json/posts')
        .then((response) => {
          commit('setQuotes', response.data)
        })
    },
    getRandomQuote ({ commit }) {
      return Vue.axios.get('http://quotesondesign.com/wp-json/posts?filter[orderby]=rand&filter[posts_per_page]=1')
        .then((response) => {
          commit('setRandomQuote', response.data)
        })
    }
  }
})

This is vue component that will display the list of random quotes got:

<template>
  <div>
    <h2>Random Quote</h2>
    <div id='random-quote' v-for="item in randomQuote" :key="item.ID">
      <h2>{{ item.title }}</h2>
      {{ item.content }}
    </div>
    <div v-if="randomQuote === null">It is null</div>
    <button id='get-random' v-on:click="getQuote">GetRandomQuote</button>
  </div>
</template>
<script>
import { mapState } from 'vuex'

export default {
  name: 'GetQuotes',
  computed: mapState({
    randomQuote: state => state.randomQuote
  }),
  methods: {
    getQuote: function () {
      this.$store.dispatch('getRandomQuote')
    }
  }
}
</script>

If I want to test the getQuote method, I'll have to figure out how to make the asyc ajax call to the server. Since the call happens in another thread, it becomes tricky, because I have to wait for that thread to finish before testing the result. Luckily, there is a library called moxios. To set it up on the project, do:

npm install moxios --save-dev

With moxios, I can mock the ajax calls (from axios). With this, I get dummy content that the test can treat similarly to how the component will treat server results. Moxios also has a wait function, which will pause execution of test until the promise completes. To test the above component with moxios:

import Vue from 'vue'
import Vuex from 'vuex'
import { mount } from 'vue-test-utils'
import RandomQuote from '@/components/RandomQuote'
import store from '@/store'
import axios from 'axios'
import VueAxios from 'vue-axios'
import moxios from 'moxios'

Vue.use(Vuex)
Vue.use(VueAxios, axios)

describe('Login.vue', () => {
  let wrapper

  beforeEach(() => {
    moxios.install()

    moxios.stubRequest('http://quotesondesign.com/wp-json/posts?filter[orderby]=rand&filter[posts_per_page]=1', {
      status: 200,
      response: [
        {
          ID: 1622,
          title: 'Stephen Anderson',
          content: '<p>At this point in experience design evolution, satisfaction ought to be the norm, and delight out to be the goal.</p>\n'
        }
      ]
    })

    wrapper = mount(RandomQuote, {
      store
    })
  })

  afterEach(() => {
    moxios.uninstall()
  })

  it('should show random quote when clicked', (done) => {
    wrapper.find('#get-random').trigger('click')
    moxios.wait(() => {
      let randomQuote = wrapper.find('#random-quote')
      expect(randomQuote.text()).toContain('At this point in experience design evolution, satisfaction ought to be the norm, and delight out to be the goal.')
      done()
    })
  })
})

From the above, I use stubRequest to mock the http request. In the actual test, I use moxios.wait(), which will wait until the api call completes before testing the contents of the random quote. I like this method because it means I get to test the store indirectly while testing my components.

Instead of moxios.call(), I could use vue's nextTick(), which also waits for the promise to complete.

it('should show random quote when clicked', (done) => {
  wrapper.find('#get-random').trigger('click')
  wrapper.vm.$nextTick(() => {
    let randomQuote = wrapper.find('#random-quote')
    expect(randomQuote.text()).toContain('At this point in experience design evolution, satisfaction ought to be the norm, and delight out to be the goal.')
    done()
  })
})

Another cleaner method of testing promises is using the library flush-promises. With flush-promises, the test would look like:

it('should show random quote when clicked', async () => {
  wrapper.find('#get-random').trigger('click')
  await flushPromises()
  let randomQuote = wrapper.find('#random-quote')
  expect(randomQuote.text()).toContain('At this point in experience design evolution, satisfaction ought to be the norm, and delight out to be the goal.')
})

The advantage flushPromises has is that if the component has multiple asynchronous calls at the same time e.g. call to user api to confirm if logged in, get list of quotes, get comments, and for some reason the test does not work, we can just add multiple flushPromises in the test.

it('should show random quote when clicked', async () => {
  wrapper.find('#get-random').trigger('click')
  await flushPromises()
  await flushPromises()
  let randomQuote = wrapper.find('#random-quote')
  expect(randomQuote.text()).toContain('At this point in experience design evolution, satisfaction ought to be the norm, and delight out to be the goal.')
})

Jupyter Notebook Setup on a VPS

2018-03-02T15:00:00+03:00

These steps have been tested on Digital Ocean server and a scaleway ARM64 server. I used the cheapest provided options for my setups.

First, set up ubuntu on the server. Both Digital Ocean and Scaleway provide an easy way to do this. Just follow their prompts, and you'll have your server up and running. After the server is ready, update ubuntu:

sudo apt update
sudo apt upgrade

The server only has the root user. It is not advisable to use root for normal work. So, I create a new user and switch to it:

useradd -m -s /bin/bash user
usermod -aG sudo username
sudo passwd username
su - user

Pip will be used to install various python packages. To install it:

sudo apt install python3-pip

I then install various ml libraries I'll use later:

python3 -m pip install --user numpy scipy matplotlib pandas sympy nose seaborn pillow keras tensorflow scikit-learn scikit-image kaggle-cli
pythom3 -m pip install ipython[notebook] jupyter

When setting up scaleway, some of the libraries failed to install. Here are the dependenices I had to install for pillow:

sudo apt-get install libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk

And here are the dependencies I had to install for scikit:

sudo apt-get install gcc gfortran python-dev libopenblas-dev liblapack-dev

And for kaggle-cli too, I had to do this:

sudo apt-get install libxml2-dev libxslt1-dev python3-lxml

Tensorflow cannot be installed in the arm64 scaleway instance using pip. So remove it from the packages.

The packages are installed, but cannot be got from the terminal. This is because the bash environment does not have it in its PATH variable. To fix this, add this to the ~/.profile file:

if [ -d "$HOME/.local/bin" ] ; then
        PATH="$HOME/.local/bin:$PATH"
fi

Having set up most of the libraries I'll need, I have to configure jupyter so that I can access it from my browser.

jupyter notebook --generate-config

A file is generated in ~/.jupyter/jupyter_notebook_config.py. This file contains various configuration options for jupyter.

I change the following:

c.NotebookApp.open_browser = False
c.NotebookApp.ip = '0.0.0.0'

To set up the password to be used on logging in do:

jupyter notebook password

I then start jupyter using:

nohup jupyter notebook

The jupyter can be accessed from : ip_address:8888 using any browser.

To install Tensorflow on scaleway's arm processors, I found instructions for a custom build here.

The instructions are to:

curl -L https://github.com/lherman-cs/tensorflow-aarch64/releases/download/r1.4/tensorflow-1.4.0rc0-cp35-cp35m-linux_aarch64.whl > /tmp/tensorflow-1.4.0rc0-cp35-cp35m-linux_aarch64.whl
python3 -m pip install /tmp/tensorflow-1.4.0rc0-cp35-cp35m-linux_aarch64.whl

SSH AND SERVERS

2018-02-16T15:00:00+03:00

SSH provides a secure means of remote login from one computer to another computer. When setting up linux servers, the OS usually provides ssh by default. To login:

ssh user@ipaddress

This prompts for a password, after which you have access to a remote terminal. Sometimes, no password is requested, and you are required to set one up after your first login.

The password login method is not really secure. This is because someone can try to guess it. A more secure method is to disable the password login and set up ssh-keys on the server. Therefore, someone will need to have the private ssh-key to be able to access your server.

To generate a key:

ssh-keygen -t rsa -b 4096 -C 'comment'

There will be some prompts that guide you through the process. If you choose to save the key in the default location, this key will be used wheneve a custom key is not provided. You should take care however, since you can easily overwrite a key that was in use. The command generates two files, the private key and a public key ( has a .pub suffix).

To set up the key, copy the public key to the server by:

ssh-copy-id -i file.pub user@ipaddress

If you are copying the keys found in the default location i.e. ~/.ssh/id_rsa.pub, the -i option can be ignored.

Having set the key up on the server, you can log in using:

ssh -i /path/to/privatekey user@ipaddress

If using the default file location, the -i option can be ignored.

To disable the password login option, log into the server and edit the /etc/ssh/sshd_config file (to disable password access for the server) or ~/.ssh/config file (to disable password access for a specific user). The following line should be added to the file:

PasswordAuthentication no

After this, restart ssh using:

reload ssh

SSH Forwarding

SSH forwarding is where the private key used to login to the server, can be used by the server to perform some tasks. This is useful in CI/CD, or when working with git on a server(Need to push to a repository on github).

To enable this, edit the ~/.ssh/config file from the local machine and add:

Host example.com
    ForwardAgent yes

To set up the key for use by the server:

eval $(ssh-agent)
ssh-add ~/.ssh/key

Eval sets environment variables to ouput of ssh-agent. To confirm the key has been added, you can do:

ssh-add -L

When you connect to the server example.com, it will now have you private sshkey in its environment and can be used to access repositories, other servers, etc.

Frozen ssh session

Sometimes after connecting to the server, you might lose your internet connection or hibernate/sleep you computer. The ssh prompt after this will be frozen, with no commands running. To fix this, you have to leave the prompt and log in again to the server. To do this, press the following keys in order from the prompt:

Enter
~
.

SciKit-Learn GraphSearch CV

2018-01-24T19:00:00+03:00

Scikit learn provides a lot of algorithms to help in machine learning. However, sometimes those algorithms have too many options. It is not usually evident which options will produce the best results. One method of solving this is to loop through all possible options and check the score.

crit = ['gini', 'entropy']
max_depth = [9,10,11,12]
min_samples_split= [2,3,4,5]
best_score = 0
best_tree = None

for criterion in crit:
    for depth in max_depth:
        for min_samples in min_samples_split:
            tr = tree(criterion=criterion, max_depth=depth,
            min_samples_split=min_samples)
            tr.fit(X_train, y_train)
            cur_score = tr.score(X_test, y_test)
            if (cur_score > best_score):
                best_core = cur_score
                best_tree = tr

This is rather tedious and with more options to consider, the code can become unreadable. Luckily, scikit-learn provides a method of automatically doing this. All you do is provide a matrix of options and scikit learn will loop through all possible combinations to get the best fit for your data. It even provides the options that worked best.

from sklearn.model_selection import cross_val_score
from sklearn import tree
from sklearn.model_selection import GridSearchCV

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=2)

parameters = {'criterion':('gini', 'entropy'),
              'min_samples_split':[2,3,4,5],
              'max_depth':[9,10,11,12],
              'class_weight':('balanced', None),
              'presort':(False,True),
             }


tr = tree.DecisionTreeClassifier()
gsearch = GridSearchCV(tr, parameters)
gsearch.fit(X_train, y_train)
model = gsearch.best_estimator_

Ansible

2018-01-06T18:00:00+03:00

Ansible is an automation tool that easens work a lot. I just set it up once and from then on I can forget about configurations and settings for my projects. It's really great for doing repeatable tasks like setting up nginx for subdomains, redeploying django projects, updating the OS, etc.

For example, to set up a script that updates the server:

---

- hosts: my_server
   vars_files:
     - vars.yml
   remote_user: "{{ user }}"
   become: true
   tasks:
       - name: Running update and safe-upgrade
          apt:
              update_cache=yes
              upgrade=dist
          register: result
       - debug: var=result.stdout_lines

I store my variables in the vars.yml file. In the example above, the user variable is found here. The file looks like this:

---

user: thisuser

The my_server variable is set in /etc/ansible/hosts file and looks like this:

[my_server]
random_url.co.ke

Since updating the system requires admin priviledges, make sure the user has those permissions. This is why the script has remote_user and become in its body. Also ansible will need to run with these priviledges, so to do so:

ansible-playbook --ask-become-pass updates.yml

And just like that the system gets updated. This asks for the server password for the elevated user.

Let's say I want to update my nginx config files to enable another subdomain. Ansible should upload my config file to my server and also restart nginx so that these changes are accepted.

---
- hosts: my_server
   vars_files:
     - vars.yml

   tasks:
   # copy project nginx set up to server
     - name: set up nginx config on server
        remote_user: "{{ user }}"
        become: true
        template:
          src=config/nginx.j2
          dest=/etc/nginx/sites-enabled/{{ project_name }}
        notify:
          - restart nginx

   handlers:
     - name: restart nginx
        remote_user: "{{ user }}"
        become: true
        service: name=nginx state=restarted

Since ansible supports jinja templating, the nginx config file is written in this format. Also the variables project_name and user are found from the vars.yml file. What the above does is copy the nginx config file to /etc/nginx/sites-enabled/project_name restarts nginx.

Here is the template used. You can also use variables defined in the vars.yml in the template too, which is really helpful.

server {
#listen 443;
    listen 80;
    server_name {{ server_name }};
    charset utf-8;
    client_max_body_size 2M;

    location / {
        # django
        include proxy_params;
        proxy_pass http://unix:{{ sock_file }};
    }

    location /static {
        autoindex on;
        alias {{ static_root }};
    }

    location /media {
        autoindex on;
        alias {{ media_root }};
    }
}

Some commands can run with ansible-playbook file.yml but others need elevated permissions. To help me deal with this I usually have a Makefile in my ansible playbooks directory. For example to deal with the two configs above, I'd have:

ANSIBLE_SUDO = ansible-playbook --ask-become-pass

update:
    $(ANSIBLE_SUDO) updates.yml

nginx:
    #(ANSIBLE_SUDO) nginx_setup.yml

You can find moer about ansible here

Project Management in Google Sheets

2017-12-16T08:00:00+03:00

Outline of Project

I wanted a google sheet file that could manage simple projects. This meant that it could track tasks, send notifications when tasks were late and differentiate done, undone and late tasks when the file was opened and editted. To do this, I decided to research into Google Sheets Scripts, which ended up being a powerful tool that I could use to achieve my means.

Sending notifications for Late Tasks

For this I first had to set a trigger. There are 2 methods of doing this that I currently know of.

Using the menu bar by going to: Edit > Current project's triggers > Add a new trigger
Programmatically by creating a function and running it once.

I went with the programmatic way because this would enable me to easier set it up on other sheets.

So assuming there is a function called mainFunction, I write the following trigger:

function createTimeDrivenTriggers() {
  ScriptApp.newTrigger('mainFunction')
      .timeBased()
      .everyDays(1)
      .create();
}

This will run the function 'mainFunction' once per day. To activate the trigger, click on Run > Run Function > createTimeDrivenTriggers. The trigger can now be seen in the triggers list menu. When running the trigger for the first time, a page will pop up requesting for some permissions to be allowed.

To send a mail using the script is also fairly easy. Since the project sheet might have may participants, I set up a function that sends emails to an array provided to it with a particular message.

function sendEmail(emails, message, subject){
  for (var i=0; i<emails.length; i++){
    MailApp.sendEmail(emails[i], subject, message);
  }
}

The emails are set in a sheet called 'lookup'. So the script will find this sheet and read the emails from it. The emails are stored in a column named 'Emails'.

function getEmailsToNotify(data) {
  var emailCol = getColumnByName(PropertiesService.getScriptProperties().getProperty('EMAILS_COLUMN'), data);
  var emails = [];
  if (emailCol < 0)
    return emails;
  for (var i=1; i<data.length; i++) {
    var email = data[i][emailCol];
    if (email.trim() !== ''){
      emails.push(email);
    }
  }
  return emails;
}

For the script, the preconditions I set for it to be able to run was that the first row should have the following names in the various cells:

Features
Status
DateDue

This means that each sheet has to follow this pattern. To enforce this I wrote the following script:

function getColumnByName(colName, data) {
  var col = data[0].indexOf(colName);
  return col;
}

function getNotificationsFromSheet(range) {
  var data = range.getValues();
  var dateDueCol = getColumnByName('DateDue', data);
  var featuresCol = getColumnByName('Features', data);
  var statusCol = getColumnByName('Status', data);
  if (dateDueCol < 0 || featuresCol < 0 || statusCol < 0 ){
    return ["error : Sheet does not follow expected format"];
  } else {
    messages = getLateDateDues(dateDueCol, statusCol, featuresCol, range);
    return messages;
  }
  // checkLateDateDues(dateDueCol, statusCol, featuresCol, data);
  // return ['this is cool', 'this is another one', 'I love this'];
}

The function getColumnByName checks the first row for any columnName provided to it. If a columnName is not found it returns -1. So in the function getNotificationsFromSheet, we check for the 3 cells, and if there is any that is missing and error is returned.

The getLateDateDues function just loops through all the dates comparing them to the current date. If any date is late and its status is not 'done', it adds this to the messages for alerting and also colors that particular row red.

function getLateDateDues(dateDueCol, statusCol, featuresCol, range) {
  // loop through the date due column only
  var data = range.getValues();
  var dateNow = new Date();
  var messages = [];
  for (var i =0; i<data.length; i++) {
    var dateDue = new Date(data[i][dateDueCol]);
    var rowRange = range.offset(i, 0, 1);
    if (dateNow > dateDue && data[i][statusCol] !== 'done'){
      messages.push(data[i][featuresCol] + ' was due on ' + dateDue);
      rowRange.setBackground('#ffcdd2');
    }
  }
  return messages;
}

You can find a copy of the complete gist here:late

Changing Colors while Editting

While editing the file, you also need to get some visual feedback on deadlines and errors. To do this, google sheets provides simple triggers that help out. For example, you can try this out:

function onEdit(e) {
  Logger.log('The sheet has been editted');
}

If you save this, and edit any cell on the sheet, you will see that message on the logger. To access the logger : View > Logs

The first thing to check for is if the editted cell belongs to the DateDue or status column. To do this, you need to get the activeSheet and the DataRange.

function onEdit(e) {
    var activeSheet = e.source.getActiveSheet();
    var sheetName = activeSheet.getName();
    var range = activeSheet.getDataRange();
}

Then we can get the column editted and row editted. Note that e.range.getColumn() and e.range.getRow(), start their indexes at 1, so to work properly with arrays we need to subtract 1 from the values we get.

function editTasksSheet(range, e){
  var data = range.getValues();
  var columnEditted = e.range.getColumn()-1;
  var rowEditted = e.range.getRow()-1;
  var valueInput = data[rowEditted][columnEditted];
  var dateDueCol = getColumnByName('DateDue', data);
  var statusCol = getColumnByName('Status', data);
  if (dateDueCol < 0 || statusCol < 0){
    return;
  } else if (columnEditted === dateDueCol) {
    var rowRange = range.offset(rowEditted, 0, 1);
    // var dateInput = data[rowEditted][columnEditted];
    formatBasedOnDate(rowRange, valueInput);
  } else if (columnEditted === statusCol) {
    if (valueInput === 'done') {
      var rowRange = range.offset(rowEditted, 0, 1);
      rowRange.setBackground('#c8e6c9');
    } else {
      formatBasedOnDate(rowRange, data[rowEditted][dateDueCol]);
    }
  }
}

From the above code snippet, we first look for the columns named 'DateDue' and 'Status'. If they are not existent, the script stops working because the sheet is not properly formatted. Afterwards we just format the sheet appropriately based off these rules:

If the date input is older than the current date, color the row red since it is a late task
If date input is later than current task, reset the background color to white.
If status input is done, color the row green
If status is something other than done, do the date formatting.

The completed file can be found in the following gist here:edit

ArchLinux Signature Error

2017-11-30T20:00:00+03:00

When updating archlinux using:

sudo pacman -Syu

I sometimes get an error message that looks something like this:

error: zstd: signature from "Andrzej Giniewicz (giniu) <randomemail@gmail.com>" is marginal trust
:: File /var/cache/pacman/pkg/zstd-1.3.2-1-x86_64.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] Y
error: failed to commit transaction (invalid or corrupted package (PGP signature))
Errors occurred, no packages were upgraded.

Pacman uses a collection of PGP keys which check signed packages and databases. If a key is corrupted or invalid, a package cannot be installed or updated. In the example presented above, the package zstd prevented my whole system from getting upgraded.

To fix this, you try to update the keys using the following:

sudo pacman -Sy archlinux-keyring
sudo pacman-key --populate archlinux
sudo pacman-key --refresh-keys

The source for this fix is from here:source

Using Pass - A Password Manager/Generator

2017-11-16T14:00:00+03:00

Pass is a password manager that stores passwords locally on the host machine. Since the passwords are stored locally there is minimal risk on your passwords being exposed after some data breach occurring. Also, you can use git to version control the passwords, and set them up in a repo on any machine.

To install:

sudo pacman -S pass

Then to initialize the password store:

pass init email

To insert a password into the store:

pass insert /email/email@dns.com

If you get the following error:

gpg: email@dns.com: skipped: No public key
gpg: [stdin]: encryption failed: No public key
Password encryption aborted.

Then you have to first generate the gpg keys:

gpg --full-gen-key

This will take you through the process step by step. All someone has to do is follow the prompts.

To generate a password, do:

pass generate name/of/service 10

which will generate a password 10 characters long

pass -n generate name/of/service 10

will generate password without special symbols i.e. only alphanumeric characters.

To edit or overwrite password

password edit name/of/service or pass inserte

Other commands include:

pass name/of/service # displays the password
pass -c name/of/service # copies the password to xclip

pass rm name/of/sercie # removes password
pass rm -r name # removes full directory

pass git init # sets up git in pass folder

To find out more about pass, you can check here

Keeping a Diary in Vim

2017-11-05T09:00:00+03:00

A diary is a good way to write down one's thoughts and memories of the day.

Being a vim fan, I had to find a way to easily help me write down my diary. This is where vim wiki comes in. It is basically a means of writing down wikis in vim, and has an awesome diary mode.

I use vundle as my vim plugin manager. So to install vimwiki, I added to the following to my vimrc file:

Plugin 'vimwiki/vimwiki'

And then ran the following from withing vim (while the vimrc file is the current buffer):

:source %
:PluginInstall

To enter diary mode, I just type <leader>wi in any vim session and it will direct me into the diary entries.

To add a diary on the day's happenings, I just type <leader>w<leader>w and it will open a blank buffer. When this is saved it will be a new day's entry.

To update the diary with the new entry, <leader>wi will take you to the page containing the various entries. Then type <leader>w<leader>i and this will update the list with the new entry.

Since I use vim a lot, I find this really useful when collecting my thoughts during a day.

So in summary these are the commands helpful for vimwiki diary mode:

<leader>w<leader>w " create new diary note for the day
<leader>wi " show all diary entries
<leader>w<leader>i " update entries

I3 Window Manager

2017-10-21T08:00:00+03:00

I've been using xfce4 as my primary window manager for some time now. Its pretty light weight and works really well on my laptop. I sometimes use an Acer D270 with 2GB Ram and 1.6Ghz processor, which is weak.

However, I decided I wanted to try out something new. I'd never actually used another window manager in linux apart from the defaults provided by various distros and xfce4 ( which I always install). So I did some digging around and found i3.

i3 is a tiling window manager, and my current default window manager too. I don't know if its the greatest one or even a good one. It was just the first other window manager I tried after xfce4 and I liked it a lot. In fact, I liked it so much that I haven't tried tinkering with another window manager yet.

To install i3 in archlinux:

sudo pacman -S i3wm i3lock i3status dmenu

The default i3 config file works but I had to make some of my changes.

First off, I really like the xfce4-terminal, so I wanted to still use it as my default terminal. This terminal also has a drop-down feature which I use a lot, so I also had to figure out a way to do this. So this was my change to support this:

bindsym $mod+Return exec xfce4-terminal
# xfce4 dropdown terminal
bindsym F12 exec xfce4-terminal --drop-down
for_window [class="xfce4-terminal"] floating enable

Since I'm a dedicated vim user, I also had to find a way to customize i3 to work similarly to vim.

# change focus
bindsym $mod+h focus left
bindsym $mod+j focus down
bindsym $mod+k focus up
bindsym $mod+l focus right

# move focused window
bindsym $mod+Shift+h move left
bindsym $mod+Shift+j move down
bindsym $mod+Shift+k move up
bindsym $mod+Shift+l move right

There are also some custom applications I use, and that I wanted to start up with i3. An example is redshift. To do that I just added this to the config file.

# redshift
exec --no-startup-id redshift-gtk

To take a screenshot, I decided to use scrot. Here is my addition to i3 for that.

# scrot screen shot
bindsym $mod+p exec "scrot ~/Downloads/Screenshot%Y-%m-%d%H:%M:%S.png"

To change the desktop wallpater, I decided to use feh.

# background image
exec feh --bg-scale ~/Pictures/i3_solarized.png

To use i3lock, add the following to the config: .. code-block:: bash

#setting up i3lock exec --no-startup-id xautolock -time 1 -locker 'i3lock -n'

I'll be making a lot of more changes to my i3 config file. To checkout what I've implemented or changed you can view the latest config file here

GRUB RESCUE FiX

2017-09-21T15:00:00+03:00

So I found myself in windows some time this week, and it started updating. The process had one major consequence, grub could not discover my linux partition, and so brought me to a prompt containing 'grub_rescue'.

To fix this, I first did:

ls

This will show a list of the partitions of my hard drive, something like:

(hd0) (hd0,gpt2) (hd0,gpt3) (hd0,gpt4)

I needed to get the partition with my linux on it. So to do this I did:

ls (hd0,gpt2)/
ls (hd0,gpt3)/

While doing this, I was looking for a partition that has folders similar to those in root. So when I got a partition that has bin, boot, home, lib, etc in it, I knew I had got the correct one.

Assuming its (hd0,gpt3), I then did:

set root=(hd0,gpt3)
set prefix=(hd0,gpt3)/boot/grub
insmod normal
normal

With this, I was able to load the normal grub menu and boot into archlinux.

Tox this permanently:

mount /dev/sda1 /boot/efi
sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=grub
sudo grub-mkconfig -o /boot/grub/grub.cfg

This is just from my steps on installation of archlinux on my asus laptop found here

Asus Zenbook Keyboard & Screen Brightness Fix On i3

2017-09-01T14:00:00+03:00

Keyboard backlight fix

Source for this information can be found here

The link has a file called kb-light.py. I set up this file in /usr/local/bin.

I then opened my i3config file and added the following:

bindsym XF86KbdBrightnessUp exec python /usr/local/bin/kb-light.py +
bindsym XF86KbdBrightnessDown exec python /usr/local/bin/kb-light.py -

And now when I press the keyboard brightness up and down key, the keyboard lights change as expected.

Screen brightness fix

For this I wanted to use xbacklight. The problem I got was that when I tried

xbacklight -inc 10

I got the error "No outputs have backlight propery".

To fix this, I installed the intel driver

sudo pacman -S xf86-video-intel

And after rebooting it worked well. So to set up the keyboard keys for screen brightness, I added this to my i3 config file:

bindsym XF86MonBrightnessUp exec xbacklight -inc 10 # increase screen brightness
bindsym XF86MonBrightnessDown exec xbacklight -dec 10 # decrease screen brightness

Media Keys Usage Fix

For this, I first had to install pulseaudio.

sudo pacman -S pulseaudio

Then I added this to my i3 config files:

bindsym XF86AudioRaiseVolume exec --no-startup-id pactl set-sink-volume 0 +5% #increase sound volume
bindsym XF86AudioLowerVolume exec --no-startup-id pactl set-sink-volume 0 -5% #decrease sound volume
bindsym XF86AudioMute exec --no-startup-id pactl set-sink-mute 0 toggle # mute sound

Now audio can be increased / decreased using my media keys.

My i3 config file can be found here:i3_config_file

My First Vim Function

2017-08-09T08:20:00+03:00

I finally got down to writing my first vim function. It was a gruelling journey but I finally got through it. So I'll write down the problem I had and the fix I finally came up with as my solution.

Problem Statement

I usually keep a to do list of things I should do during the day and week. Over time, tasks get lost as I remove some lines and I can't keep a track of my progress in terms of planning and implementation. So basically I wanted a means for later on in the year or in my life, I could have a file that showed how much I planned and how much I had achieved.

So I created the file. It had the following syntax. I use vimwiki, so to generate the table I just type:

:VimwikiTable 5 3

The file looks like this:

| Week | Total | Achieved | %    | Comments                        |
|------|-------|----------|------|---------------------------------|
| 1    | 13    | 1        | 7.69 | setting up lap  took up time    |
| 2    | 20    | 9        |      |                                 |

This table has a percentage column that I wanted to be automatically calculated. This would provide a means of checking whether I'm becoming better at goal setting and goal achievement over time.

Implementation

To get the percentage, I had to basically get the row value that corresponded to total, and the row value that corresponded to achieved.

To get the once corresponding to totaL, I did this in normal mode:: ^3wviw

What this does is this:

^ goes to beginning of line 3w goes 3 words ahead after going to beginning of line viw selects the current word, which happens to be the total

With this I had the number in total column selected, so I had to find out how to store the number for later use. Thats when registers came into play. So adding:

"py

Will yank the selected word and store it in the named register p. With this I can get the variable from the register. So to get the number 13 in the first row, this was the function:

function! TodoPercentage()
    normal ^3viw"py
    let total = str2float(getreg("p"))
endfunction

The getreg() function just gets the values stored in a register, and the str2float function converts a string to a float variable.

To get the achieved value, I also did the same.

function! TodoPercentage()
    normal ^3wviw"py
    let total = str2float(getreg("p"))
    normal ^5wviw"py
    let achieved = str2float(getreg("p"))
endfunction

Since I had these two variables, I could now do the arithmetic.

let percentage = (achieved/total) * 100

And since I just wanted to display the number in maximum of 2 decimal places, I used printf command, which sets the variable to 2 decimal places and also saves it in the per variable as a string.

let per = printf("%.2f", percentage)

To display the number in the appropriate section, I decided to use the execute command. This would execute the string, converting special characters to vim commands e.g. <esc> to actually pressing escape. The "normal!" executes the commands in normal mode, the ! eliminates the custom mappings made.

execute "normal! ^6wa ".per."\<esc>"

After the "normal!" the following happens:: ^6w : moves to the start of the line, then 6 words after a : appends characters, enter insert mode .per. : concatenates the percentage string. Thus it will be typed character by character "<esc>" : initiates the esc sequence leaving insert mode. The is used to signify that the key ESC is what is meant and not the actual characters.

So at the end of it all this was my function:

function! TodoPercentage()
    normal ^3wviw"py
    let total = str2float(getreg("p"))
    normal ^5wviw"py
    let achieved = str2float(getreg("p"))
    let percentage = (achieved / total) * 100
    let per = printf("%.2f", percentage)
    execute "normal! ^6wa ".per."\<esc>"
endfunction

Now to run this function I decided to map it to <leader>cp meaning calculate percentage.

autocmd FileType vimwiki nnoremap <leader>cp :call TodoPercentage() <Cr>

You can find this function implemented in my .vimrc file here

ASUS ZenBook UX330UA ArchLinux Setup

2017-06-28T15:00:00+03:00

I recently got the ASUS ZenBook UX330UA-AH54, which is a really awesome machine. I've been using it for one month now, and its beautiful.

So first thing I did with the laptop was install archlinux (dual boot with windows). These are the steps I took to set up archlinux. Most of these steps can be found in one form or another on the archlinux wiki

Hard Drive Preparation

I partitioned the hard drive using windows tools.

I then downloaded archlinux iso file in another machine that had archlinux, and attached a flash drive. To setup the iso file, I did this from the terminal:

dd if=archlinux.iso of=/dev/sdb bs=4M status=progress

I then connected this flash drive to the Asus laptop and did a reboot. When the Asus logo shows, I pressed <Esc> so that I could choose which bootable media to use. I selected the flash drive and unfortunately got the error:

Invalid signature detected. Check secure boot policy in setup.

I then went to set up and disabled secure boot policy.

I successfully managed to enter the archlinux environment. I verified uefi mode by using:

ls /sys/firmware/efi/efivars

This gave me some content, thus it was fine.

Internet Connection

I connected to the internet via wifi. To do this:

iw dev #to get name of wireless interface. I got wlp2s0
ip link set wlp2s0 up #I had to activate the kernel interface
ip link show wlp2s0 #To check if activated. There was an UP between the <> info
iw dev wlp2s0 scan | less #list wifi Access Points in area
wpa_supplicant -i wlp2s0 -c <(wpa_passphrase "SSID" "keytoaccesswifi") -B #connect to wifi
iw dev wlp2s0 link #to check if associated
dhcpd wlp2s0 #to get up address from wifi

System Clock

To update the system clock, I did:

timedatectl set-ntp true
timedatectl status # checks the time

Initial Setup

To install arch, I had to view the partitions and the sizes so that I could choose the correct one.

fdisk -l

After getting the correct one:

mkfs.ext4 /dev/sda4 #format partition as ext4
mount /dev/sda4 /mnt #mount partition to start working with it

I then set up some basic packages for arch:

pacstrap /mnt base base-devel

I generated the fstab:

genfstab -U /mnt >> /mnt/etc/fstab

Now change into the new file system root:

arch-chroot /mnt

I set the time zone for my system:

ln -sf /usr/share/zoneinfo/Africa/Nairobi /etc/localtime
hwclock --systohc

For the time zone once '/user/share/zoneinfo' has been type, clicking tab will autocomplete.

I set the locale by uncommenting the 'en_US.UTF-8 UTF-8' and the 'en_GB.UTF-8 UTF-8' lines in the /etc/locale.gen file and then ran:

locale-gen

To set the default locale, I create a file /etc/locale.conf and added LANG=en_US.UTF-8 to it.

echo LANG=en_US.UTF-8 >> /etc/locale.conf

The Hostnames file /etc/hosts was already existing so I did not edit it. I then set the root password by running:

passwd

Bootloader installation

I first installed grub and efibootmgr.

pacman -S grub efibootmgr

I needed parted so that I could know my ESP partition ( EFI system partition). The actual partition will have the name shown. So to do this:

sudo pacman -S parted
parted /dev/sda print # to know which was my ESP partition

After getting the ESP partition, I mount /boot/efi onto it and install grub.

mount /dev/sda1 /boot/efi
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=grub
pacman -S intel-ucode #enabling microcode updates
grub-mkconfig -o /boot/grub/grub.cfg

I then reboot the machine. I get an error at the login prompt but it can be ignored.

User Creation

To create a user:

useradd -m -G wheel -s /bin/bash username
passwd username

I then set up sudo on the laptop:

pacman -S sudo
EDITOR=vi visudo #and uncommnet the line %WHEEL  ALL= (ALL) ALL

The line containing "%WHEEL ALL= (ALL) ALL" is uncommented.

Window Managers and Other Applications

I first set up xdg-dirs, and the run xdg-user-dirs-update. I then open the file .config/user-dirs.dirs and set the values to how I want my home directory to be organised.

I then copy the /etc/xdg/user-dirs.code file to the .config folder and set enabled to false.

I then create all my directories as pwer user-dirs.dirs manually using mkdir.

sudo pacman -S xdg-user-dirs
run xdg-user-dirs-update
vi .config/user-dirs.dirs
cp /etc/xdg/user-dirs.conf ~/.config/
vi .config/user-dirs.conf #change enabled=False

I then install vim and tmux and configure it as per the link .

sudo pacman -S gvim tmux

I also set up powerline fonts as found here to help in vim_airline setup.

I then install zsh:

sudo pacman -S zsh zsh-completions

And then setup oh-my-zsh as per the instructions here: zshlink

I then install xorg group, sfce4 and xfce4-goodies for one of my desktop environments.

I also install i3 as my main window manager.

Other Installs

sudo pacman -S python python2 python-pip python2-pip
sudo pacman -S kicad kicad-library kicad-library-3d
sudo pacman -S openssh
sudo pacman -S mtp gvfs-mtp

I also use ledger to manage my finances so I have to install it. Ledger is only found in AUR so I first installed yaourt from instructions found here :yaourt

After that I install ledger with:

yaourt -S ledger

The following installs are requirement for my i3 config to work properly:

sudo pacman -S redshift feh scrot dmenu
sudo pacman -S python-gobject python-xdg librsvg #required for redshift-gtk

For xfce4, I added the following shortcuts:

exo-open --launch TerminalEmulator  Super+Return
xfce4-terminal --drop-down F12

To switch capslock and escape, I added this to my .xinitrc:

setxkbmap -option caps:swapescape

The audio by default was muted so to fix this I did the following:

sudo pacman -S alsa-utils
alsamixer
#went to master and unmuted by pressing m

And with that I was comfortable enough with my archlinux setup.

Vim Airline Setup

2017-06-12T08:20:00+03:00

I was watching this video on how to use vim+tmux effectively (Video Link), and this guy had some really awesome looking vim. I decided to try it out and see if I could end up with something just that great.

I did my research and found two alternatives, powerline and vim-airline. Powerline is pretty heavy and vim-airline states itself to be lightweight, so I went ahead and installed vim-airline.

"Added so that vim airline can get git info
Plugin 'tpope/vim-fugitive'
Plugin 'vim-airline/vim-airline'

After this I just run :PluginInstall.

However, after installation, the statusline did not look that great. I needed to do something more. I foundout that vim-airline needs some special fonts so that it can display some things properly, like the > and <. To do this, I had to install powerline fonts. I followed the instruction found here: powerine fonts and ended up installing the powerline fonts.

I then switched to a powerline font in Xfce4-terminal. I use DejaVu Sans Mono for Powerline Book.

But still this was not enough. I then had to go to my .vimrc and add some few more sections.

"To allow colors in the status line
set t_Co=256
"To allow status bar to be seen even before splitting
set laststatus=2
"To allow powerline symbols to be seen
let g:airline_powerline_fonts=1

With all this I had almost reached my goal. However, I have a 10.1" screen laptop, and the status bar seemed to be hiding some details because of the little screen landscape. So I decided to change some sections. After some digging I added this to my .vimrc

"Reduces the space occupied by section z
let g:airline_section_z = "%3p%% %l:%c"

And now I have a pretty decent status bar.

Git Branches

2017-06-04T15:00:00+03:00

Lets say we have have a project A that works really well and serves our purpose. Then there is this new killer feature that we want to add to the project. Adding it directly to project A is not a really good thing as it might break whats working.

So what happens. We create a branch on which we start implementing this new feature. The branch is a copy of the code, that we can work with safely. The branch can be experimental features or just things we want to test to see if they are stable enough.

Branches are an important feature in git ahd here are the basic commands I use:

git branch #list all branches
git branch <name> #creates branch called name
git branch -m <name> #change branch name to name
git branch -d <name> #deletes the branch. If changes havent been
                    #merged an error message is shown.
git branch -D <name> #force delete branch even with unmerged conflicts
git checkout <existing-branch> #switch to this branch
git checkout -b <name> #create branch name and switch to it

Sometimes you create a branch but it cannot push to the server. To fix this, you do:

git push --set-upstream origin <branch_name>

And when you clone from a server, the git branch command will only show the master branch. To view all the branches you do:

git branch -a

These are the branch commands I interact with most of the time. I'll add more over time as I get more familiar with branching.

Adb Usage at Work

2017-04-09T08:20:00+03:00

To use adb in archlinux, first install android-tools.

sudo pacman -S android-tools

I then add the adb resources to the android rules files. To do this, first do lsusb, you'll get something like this:

lsusb
Bus 001 Device 004: ID 12d1:256b Huawei Technologies Co., Ltd.

To add this to the rules, open up /etc/udev/rules.d/51-android.rules And add this:

SUBSYSTEM=="usb", ATTR{idVendor}=="12d1", MODE="0660", GROUP="plugdev"

To list all the packages installed:

adb shell pm list packages -f

To get the main activity of a package

adb shell pm dump com.randompackage.android.application | grep -A 1 MAIN

To launch activities

#launches the launcher activity
adb shell monkey -p app.package.name -c android.intent.category.LAUNCHER 1

or:

adb shell am start -n com.package.name/com.package.name.ActivityName

To start a specific action

am start -a com.example.ACTION_NAME -n com.package.name/com.package.name.ActivityName

To install apk

#if not previously installed
adb install filename.apk
#if it had been installed eg. when upgrading
adb install -r filename.apk

Why To Use Github

2017-03-19T10:20:00+03:00

Github is a really awesome tool. It's where most open source project live. In my case, I use github as a means of control and tracking. I've been doing so many projects in my free time, but however I do not have a record or something that can show what I've been doing. For example, I made a game a couple of years ago, but somehow lost the code. So I can't actually state that I've made a game.

Github too works really well with Kicad. Kicad is an electronic schematic capture and pcb design tool. It works well because kicad stores all of its files in text format. Thus github is a good way to manage my circuits.

Github, I believe will also help me keep a track of where I'm coming from and where I'm going. I've discovered recently that using github make me use it even more.

Postgresql Weird Error - Failed To Start

2017-03-12T19:44:00+03:00

After updating my Archlinux, I tried starting up postgreSQL with:

sudo systmctl start postgresql.service

The service was unable to start. I then checked on what systemctl was saying the error was.

(env) [rookie@ArchRookie eventmanagement]$ sudo systemctl status postgresql
[sudo] password for rookie:
● postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2017-02-19 17:43:52 EAT; 3min 24s ago
Process: 4210 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGROOT}/data (code=exited, status=1/FAILURE)
Feb 19 17:43:52 ArchRookie systemd[1]: Starting PostgreSQL database server...
Feb 19 17:43:52 ArchRookie systemd[1]: postgresql.service: Control process exited, code=exited status=1
Feb 19 17:43:52 ArchRookie systemd[1]: Failed to start PostgreSQL database server.
Feb 19 17:43:52 ArchRookie systemd[1]: postgresql.service: Unit entered failed state.
Feb 19 17:43:52 ArchRookie systemd[1]: postgresql.service: Failed with result 'exit-code'.

I then decided to check via journalctl.

(env) [rookie@ArchRookie eventmanagement]$ journalctl -xe
-- The result is failed.
Feb 19 17:43:52 ArchRookie polkitd[321]: Unregistered
Authentication Agent for unix-process:4201:1104590 (system
Feb 19 17:43:52 ArchRookie systemd[1]: postgresql.service: Unit
entered failed state.
Feb 19 17:43:52 ArchRookie systemd[1]: postgresql.service: Failed
with result 'exit-code'.

This too was not descriptive enough. I then went to the archlinux wiki docs and found a solution. Apparently something had messed up my '/var/lib/postgres/data' and so this was my fix:

su root
su postgres
initdb --locale $LANG -E UTF8 -D '/var/lib/postgres/data'

This is the Source Link .. Source: https://bbs.archlinux.org/viewtopic.php?id=194040

Solarized Theme

2017-02-26T08:20:00+03:00

When using a tool or anything for a really long time, one has to really like how it looks like. I use the terminal, vim and tmux a lot. So I had to look for a theme that would be easy on my eyes. A quick internet search revealed solarized, which I then went on to set up. I have never looked back since then.

Xfce4 terminal setup

Go to Preferences>Colors

Then change the terminal colors to "solarized-dark"

Vim setup

I use vundle as my plugin manager in vim. So setting up solarized was as easy as adding the following line:

Plugin 'altercation/vim-colors-solarized'

And then running:

:PluginInstall

After these I added the following lines to my .vimrc:

syntax enable
set background=dark
colorscheme solarized

And vim looked great. More configuration settings can be found here: solarized-github-page

Tmux setup

I use the tmux plugin manager (tpm-link). So to setup solarized on tmux, it's as simple as:

set -g @plugin 'seebi/tmux-colors-solarized'

Then I run prefix + I , whicn in my case it Ctrl+b and then I press I.

This will download and install the tmux solarized theme.

After this I add the following line to my tmux config:

set -g @colors-solarized 'dark'

For more information of tmux solarized config, check here: tmux-solarized

Know Thy Editor Well

2017-02-19T08:20:00+03:00

Someone once told me that I should really know my editor, and know it well. I did not take this advice into consideration but lately, as I get better and better at using vim, I think I'm beginning to understand the fella well.

If you know your editor well enough, it becomes the single point of reference for every task you will try to do. And the more you know your editor, the better it works for you. Also, your performance improves significantly as you become more comfortable with your editor.

If you become comfortable with the editor, you might end up using it for the most importabt aspects of your life. It can end up integrating into a lot of really cool stuff you use.

Here is a list of some of the things I use vim for: * Managing my finances, works together with ledger * Writing a diary/journal of my life * Note taking using vimwiki * Todo lists using vimwiki and/or vim orgmode * Coding -- obviously * Blogging * Server setup and management * Documentation * Make presentations

As can be seen, knowing vim well will greatly optimize my time and also effectively manage my life well. It should not only be vim, but know that whatever editor you choose to use, know it well, know its community and learn every day how to optimize your work.

Changing Menu Icon MinCss

2017-02-12T08:55:00+03:00

I wanted to make this pelican theme that is really light weight. I did some research and came upon mincss. Now this is a really small CSS framework that supports a wide range of browsers. For more info, check out here

I encountered a small problem however. I wanted to change my menu background colour. By default, micsss menu has a black backgroun and a menu icon that is white in colour.

So I thought they used an icon or image, only to find out mincss has this really neat treat of using borders. The following line shows its implementation:

.nav div:before {
    background: #000;
    border-bottom: 10px double;
    border-top: 3px solid;
    content: '';
    float: right;
    height: 4px;
    position: relative;
    right: 3px;
    top: 14px;
    width: 20px;
}

This code creates three lines with the border command thus the menu icon.

To make it appear well in my theme, ie. in a white background with black lines, I just changed the code to this.

.nav div:before {
    background: #fff;
    <!--everything else remains the same-->
}

I then added a bit of snippets on my own to make the background white.

.nav{
    background: #ffffff;
}

And thus I had a menu I pretty much liked.

Tmux Usage

2017-02-05T08:20:00+03:00

I use xfce4-terminal which is really awesome. It light on my computer and yet heavy on features. It has tabs, a drop down terminal option, and many more.

However, I soon came to realize I needed something more. Something that I could use with together with the features of xfce4-terminal. That's when I got to find tmux.

Tmux put simple is a terminal multiplexer, meaning it gives you multiple terminals within the same window i.e kinda like tabs but more powerful. You can have multiple windows on the same screen(it works like vsplit and hsplit of vim) and multiple windows in the same session.

To make you understand, I just have to show you how to use it:

sudo pacman -S tmux

To start a tmux session:

tmux

After than you can create mutliple windows on screen or in buffers

C-b c " creates a new buffer
C-b , ' renames a new buffer
C-b number "moves to the numbered buffer
C-b " 'Creates a vsplit
C-b % "creates a hsplit

To move through sections I wanted to use vim directional buttons h,j,k,l so I had to configure this in tmux config file. So I created a .tmux.conf in my home folder and added the following:

bind -r k select-pane -U 
bind -r j select-pane -D 
bind -r h select-pane -L 
bind -r l select-pane -R

So if I have multiplw window splits on screen, I just do:

C-b h/j/k/l " move to window

I also set the resizing of split to use

bind -r C-k resize-pane -U
bind -r C-j resize-pane -D
bind -r C-h resize-pane -L
bind -r C-l resize-pane -R

So to resize a window split I just do:

C-b C-h/j/k/l

If you have multiple windows and have exited from one, the numbering is usually flawed ie. it can be 1,2,4,5. To renumber the windows, I do

C-n : "command mode
movew -r "renumber the windows

Digital Ocean Server Setup

2017-01-22T07:20:00+03:00

Setting up a digital ocean server is pretty straight forward. First you have to pay some amount via paypal or credit card ( I paid $5). Then there is an easy html page set up tool that helps set up the account. In my case, I did not choose to use ipv6 or to use ssh key on login. However most of the options can be changed later on as I learnt. I chose Ubuntu 16.04 as my preffered OS of choice. Everything wasthen setup automatically and I just had to connect to my server.

The first thing I did was add public key authentification for my root user. I did that by:

ssh root@server-ip
answer yes to question that appears
input password

This just confirms that I can log into the system. I then exit ssh and do the following:

ssh-keygen
ssh-copy-id root@server-ip

After this process I can now login to my server without the need for my password.

I then add another user to the system who will have root privileges:

ssh root@server-ip
adduser username
usermod -aG sudo username

Add public key authentification:

ssh-keygen
ssh-copy-id username@server_ip

I then diabled password access via ssh. This makes it more secure as it prevents people guessing my server password.ie. You either have the key to access my server or not.

sudo vi /etc/ssh/sshd_config
set
    PasswordAuthentification no
sudo system reload sshd

set up firewall:

sudo ufw app list 
sudo ufw allow OpenSSH
sudo ufw status ;should see OpenSSH

I then installed nginx as my server of choice:

sudo apt update
sudo apt install nginx

And then added it to my firewall:

sudo ufw app list
sudo ufw enable 'Nginx HTTP'
sudo ufw status
systemctl status nginx

I only enable Nginx HTTP because I have not yet got an SSL cerificate, so port 43 for HTTPS is not currently in use.

This is a neat trick I got about how to get your ip address from the terminal

ip addr show eth0 | grep inet | awk '{ print $2; }' | sed
's/\/.*$//'

sudo apt-get install curl
curl -4 icanhazip.com

For a database I decided to install mariadb

sudo apt install mariadb-server mariadb-client
sudo mysql_secure_installation

I then installed php:

sudo apt-get install php-fpm php-mysql

sudo vim /etc/php/7.0/fpm/php.ini
set: cgi.fix_pathinfo =0; secure better the system

Org Mode in Vim

2017-01-09T20:20:00+03:00

So emacs has this awesome tool called orgmode that I recently saw in use. It really impressed me, especially the versatility of it. Problem is, its emacs and I'm already a vim fan boy.

So what's the next best option? Easy, look for a vim plugin, and oh boy did I get one... vim-orgmode.

It isn't perfect, heck I'd even suggest that you use the original orgmode if you really want it. But here's how I go about using orgmode:

First install it. When using vundle( I use vundle as my plugin manager), just place the line below in the .vimrc file.

Plugin 'jceb/vim-orgmode'

Then here are a few commands that will make life a breeze:

help vim-ormode "tutorial doc. You should really start here.
/hh "create a title
/cn "create a check list
/sa "create a date
/cc "mark check list as done
tab "unfold/ fold trees

So here's how I work: I create a daily document at ~/ that has the date of the current day. Something like plan_sun_05_01_2016.org Within that folder I create a tree of various plans I'd like to do Something like

* Plan for day 
** Morning tasks [/]
** Server setup [/]
** Research [/]
** Night tasks [/]

The [/] is just a way to keep a track of the subtasks on the major tasks. I then write out the subtasks. This is accomplished by \cn just below the subtask. The [/] will also be autofilled by vim. I can end up with a document that looks like this:

* Task for the day
** Morning Tasks [2/3]
   - [X] Exercise
   - [X] Eat breakfast
   - [ ] Read emails
** Server setup [0/3]
   - [ ] Buy account
   - [ ] Set up apache
   - [ ] Set up mysql
** Research [0/2]
   - [ ] How to use org mode
   - [ ] Wrtiing vim scripts
** Night tasks [0/2]
   - [ ] Read book
   - [ ] Write blog post

There are a couple of things happening above here.

The [2/3] is auto generated, and each time to add an item or mark an item with /cc it is updated.

Once I got the hang of it all, I read the docs a bit more. I got a means of adding orgmodes into the global something so that I can access their details wherever I want.

let g:org_agenda_files = ['~/org/index.org','~/org/projects.org']

Once that is set in my .vimrc, whenever I am in any org file, I can just type "\caa" to list all items for the current week and "\cat" to view all todos for the week. It just scans through the files in the list shown above.

Remapping Keyboard Keys(Esc and Caps)

2016-12-06T20:20:00+03:00

I use vim as an editor, and this comes with some curse. I press "ESC" a lot. And yet this key is located at an inconvenient location, at the top row of my keyboard. For a key I often use, I would rather it occurs somewhere closer to the home row.

The "CapsLock" key is one that I rarely use, yet its located at the home row. Furthermore, its use is redundant because I can always use the "SHIFT" key to do the same work.

I found a nifty trick on how to remap these keys on my keyboard in linux.

Open the terminal and type:

xmodmap -e "clear Lock" -e "keycode 9 = Caps_Lock NoSymbol Caps_Lock" -e "keycode 66 = Escape NoSymbol Escape"

If this does not work replace: clear Lock with remove Lock = Caps_Lock

xmodmap -e "remove Lock = Caps_Lock" -e "keycode 9 = Caps_Lock NoSymbol Caps_Lock" -e "keycode 66 = Escape NoSymbol Escape"

However, this change is temporary. On system restart the keyboard will be the old same.

To make the changes permanent within the system:

xmodmap -pke >~/.xmodmap
# Add this line to ~/.xinitrc
xmodmap .xmodmap

Another alternative is to use setxkbmap:

# replaces caps with escape functionality
setxkbmap -option caps:escape
# swaps capslock and escape
setxkbmap -option caps:swapescape

Add whatever command to the .xinitrc file to work.

This switch is not all hymns and praises because:

My laptop just became more unusable to others
My muscle memory has already got this new configuration. This makes it difficult to work with other machines.