# Build an iRedMail fail-over Cluster with KeepAlived, HAProxy, GlusterFS, OpenLDAP, Mariadb

Attention

Check out the lightweight on-premises email archiving software developed by iRedMail team: Spider Email Archiver.

Warning

This tutorial was contributed by Setyo Prayitno `<jrsetyo _at_ gmail.com>` (forum user name `t10`) on March 13, 2016. Thanks Setyo. :) iRedMail Team doesn't offer tech support for this setup.

## TODO

- Use clear server hostnames and IP addresses for all involved servers.
- Install adminer from http://adminer.org

## Goal

Build a fail-over cluster with 4 servers (2 backend servers behind HAProxy + KeepAlived).

## Requirements

- A valid mail domain name. We use `example.com` as mail domain name in this document.

- 4 servers, all are CentOS 7.

  - 2 servers run HAProxy + KeepAlived as a frontend for load-balance (HAProxy) and fail-over (KeepAlived).
  - 2 servers run the actual mail services. We will install the latest iRedMail release for this.

The big picture:

## Summary

Hostnames and IP addresses:

- We use hostname `ha1.example.com` and `ha2.example.com` for our 2 servers which run HAProxy and KeepAlived, use `ha1` and `ha2` for short.

- We use hostname `mail1.example.com` and `mail2.example.com` for our 2 servers which run iRedMail for mail services, use `mail1` and `mail2` for short.

- IP addresses:

    192.168.1.1 ha1
    192.168.1.2 ha2
    192.168.1.3 mail1
    192.168.1.4 mail2

The procedure:

1. Install and configure KeepAlived
2. Install and configure HAProxy
3. Install and configure GlusterFS as glusterserver & glusterclient (you can use separate machine for glusterserver) it's better to use a new hard drive with the same capacity
4. Install and configure iRedMail
5. Setup OpenLDAP replication (Master-Slave)
6. Setup MariaDB replication (Master-Master)

## Install and configure KeepAlived

Install on 2 servers (ha1 & ha2)

- on both servers, update `/etc/hosts`:

    192.168.1.1 ha1
    192.168.1.2 ha2
    192.168.1.3 mail1
    192.168.1.4 mail2

- Install KeepAlived and backup default config file:

    yum install -y keepalived
    mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf_DEFAULT

- on ha1:

    nano /etc/keepalived/keepalived.conf

- change eth0 to your existing interface

    vrrp_script chk_haproxy {
        script "killall -0 haproxy" # check the haproxy process
        interval 2 # every 2 seconds
        weight 2 # add 2 points if OK
    }

    vrrp_instance VI_1 {
        interface eth0 # interface to monitor
        state MASTER # MASTER on ha1, BACKUP on ha2
        virtual_router_id 51
        priority 101 # 101 on ha1, 100 on ha2
        virtual_ipaddress {
        192.168.1.10 # virtual ip address
        }
        track_script {
            chk_haproxy
        }
    }

- on ha2, update `/etc/keepalived/keepalived.conf`

change `eth0` to your existing interface

    vrrp_script chk_haproxy {
        script "killall -0 haproxy" # check the haproxy process
        interval 2 # every 2 seconds
        weight 2 # add 2 points if OK
    }
    vrrp_instance VI_1 {
        interface eth0 # interface to monitor
        state BACKUP # MASTER on ha1, BACKUP on ha2
        virtual_router_id 51
        priority 101 # 101 on ha1, 100 on ha2
        virtual_ipaddress {
        192.168.1.10 # virtual ip address
        }
        track_script {
            chk_haproxy
        }
    }

- activate KeepAlived service on both servers:

    systemctl enable keepalived
    systemctl start keepalived

- Check status of virtual IP (192.168.1.10) with command below:

    ip a

## Install and configure HAProxy

- Install on both servers (ha1 & ha2)

    yum install -y haproxy
    mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_DEFAULT

- on ha1: update `/etc/haproxy/haproxy.cfg`

    global
            log 127.0.0.1   local0
            log 127.0.0.1   local1 debug
            maxconn   45000 # Total Max Connections.
            daemon
            nbproc      1 # Number of processing cores.

    defaults
            timeout server 86400000
            timeout connect 86400000
            timeout client 86400000
            timeout queue   1000s

    # [HTTP Site Configuration]
    listen  http_web 192.168.1.10:80
            bind *:80
            bind *:443 ssl crt /etc/ssl/iredmail.org/iredmail.org.pem
            redirect scheme https if !{ ssl_fc }
            mode http
            balance roundrobin  # Load Balancing algorithm
            option httpchk
            option forwardfor
            server mail1 192.168.1.3:443 weight 1 maxconn 512 check
            server mail2 192.168.1.4:443 weight 1 maxconn 512 check

    # [HTTPS Site Configuration]
    listen  https_web 192.168.1.10:443
            mode tcp
            balance source# Load Balancing algorithm
            reqadd X-Forwarded-Proto:\ http
            server mail1 192.168.1.3:443 weight 1 maxconn 512 check
            server mail2 192.168.1.4:443 weight 1 maxconn 512 check

    # Reporting
    listen stats
        bind :9000
        mode http

        # Enable statistics
        stats enable

        # Hide HAPRoxy version, a necessity for any public-facing site
        stats hide-version

        # Show text in authentication popup
        stats realm Authorization

        # URI of the stats page: localhost:9000/haproxy_stats
        stats uri /haproxy_stats

        # Set a username and password
        stats auth yourUsername:yourPassword

- on ha2, update `/etc/haproxy/haproxy.cfg`

    global
            log 127.0.0.1   local0
            log 127.0.0.1   local1 debug
            maxconn   45000 # Total Max Connections.
            daemon
            nbproc      1 # Number of processing cores.

    defaults
            timeout server 86400000
            timeout connect 86400000
            timeout client 86400000
            timeout queue   1000s

    # [HTTP Site Configuration]
    listen  http_web 192.168.1.10:80
            bind *:80
            bind *:443 ssl crt /etc/ssl/iredmail.org/iredmail.org.pem
            redirect scheme https if !{ ssl_fc }
            mode http
            balance roundrobin  # Load Balancing algorithm
            option httpchk
            option forwardfor
            server mail1 192.168.1.3:80 weight 1 maxconn 512 check
            server mail2 192.168.1.4:80 weight 1 maxconn 512 check

    # [HTTPS Site Configuration]
    listen  https_web 192.168.1.10:443
            mode tcp
            balance source# Load Balancing algorithm
            reqadd X-Forwarded-Proto:\ http
            server mail1 192.168.1.3:443 weight 1 maxconn 512 check
            server mail2 192.168.1.4:443 weight 1 maxconn 512 check

    # Reporting
    listen stats
        bind :9000
        mode http

        # Enable statistics
        stats enable

        # Hide HAPRoxy version, a necessity for any public-facing site
        stats hide-version

        # Show text in authentication popup
        stats realm Authorization

        # URI of the stats page: localhost:9000/haproxy_stats
        stats uri /haproxy_stats

        # Set a username and password
        stats auth yourUsername:yourPassword

- on both servers:

create cert for ssl redirect (to iRedMail Servers)

    mkdir /etc/ssl/iredmail.org/
    openssl genrsa -out /etc/ssl/iredmail.org/iredmail.org.key 2048
    openssl req -new -key /etc/ssl/iredmail.org/iredmail.org.key -out /etc/ssl/iredmail.org/iredmail.org.csr
    openssl x509 -req -days 365 -in /etc/ssl/iredmail.org/iredmail.org.csr -signkey /etc/ssl/iredmail.org/iredmail.org.key -out /etc/ssl/iredmail.org/iredmail.org.crt
    cat /etc/ssl/iredmail.org/iredmail.org.crt /etc/ssl/iredmail.org/iredmail.org.key > /etc/ssl/iredmail.org/iredmail.org.pem

activate HAProxy service

    systemctl enable haproxy
    systemctl start haproxy

check log if any errors

    tail -f /var/log/messages

allow http, https, haproxystat ports

    firewall-cmd --zone=public --permanent --add-port=80/tcp
    firewall-cmd --zone=public --permanent --add-port=443/tcp
    firewall-cmd --zone=public --permanent --add-port=9000/tcp
    firewall-cmd --complete-reload

## GlusterFS

### Add new hard disk and format with preferred file system

first, add new hard drive with the same capacity

- on both servers, update `/etc/hosts`:

    192.168.1.3 mail1
    192.168.1.4 mail2

- add new disk on `mail1`:

type 'n', and hit enter for next question, (don't forget to write) hit 'w'

    fdisk /dev/sdb
    /sbin/mkfs.ext4 /dev/sdb1
    mkdir /glusterfs1

Update `/etc/fstab`:

    /dev/sdb1 /glusterfs1      ext4    defaults        1 2

remount all:

    mount -a

- add new disk on mail2:

type 'n', and hit enter for next question, (don't forget to write) hit 'w'

    fdisk /dev/sdb
    /sbin/mkfs.ext4 /dev/sdb1
    mkdir /glusterfs2

Update /etc/fstab:

    /dev/sdb1 /glusterfs2      ext4    defaults        1 2

remount all

    mount -a

### Install and Configure GulsterFS

- on both servers (mail1 & mail2):

    yum -y install epel-release
    yum -y install centos-release-gluster38.noarch
    yum -y install glusterfs glusterfs-fuse glusterfs-server

activate the service

    systemctl enable glusterd.service
    systemctl start glusterd.service

disabling firewall

    systemctl stop firewalld.service
    systemctl disable firewalld.service

- on mail1:

    gluster peer probe mail2

- on mail2:

    gluster peer probe mail1

you can check status with command below:

    gluster peer status

- ONLY on mail1:

    gluster volume create mailrep-volume replica 2  mail1:/glusterfs1/vmail  mail2:/glusterfs2/vmail force
    gluster volume start mailrep-volume

check it

    gluster volume info mailrep-volume

- create folder for vmail and mount glusterfs to vmail folder

on mail1:

    mkdir  /var/vmail
    mount.glusterfs mail1:/mailrep-volume /var/vmail/

Update /etc/fstab

    mail1:/mailrep-volume /var/vmail glusterfs defaults,_netdev 0 0

remount all

    mount -a

check it

    df -h

- on mail2:

    mkdir  /var/vmail
    mount.glusterfs mail2:/mailrep-volume /var/vmail/

Update /etc/fstab:

    mail2:/mailrep-volume /var/vmail glusterfs defaults,_netdev 0 0

remount all

    mount -a

check it

    df -h

you can test it by creating some files on one of your mail servers

    cd /var/vmail; touch R1 R2 R3 R4 R5 R6

make sure it syncs, by checking files on both servers

    ls -la /var/vmail

## Install and configure iRedMail

- Install the latest iRedMail on 2 servers (mail1 & mail2)

- For installing iRedMail on CentOS, please check its installation guide: Install iRedMail on Red Hat Enterprise Linux, CentOS

Note

- install iRedMail on `mail1` first, after mail1 finish you can install it to mail2 (it's better to not reboot after installing iRedMail, wait until it finishes installation/configuration)

- Don't forget to choose LDAP and using default mail folder: `/var/vmail`

- Choose Nginx as web server

## Configure LDAP replication (Master-Slave)

- on mail1 (MASTER), update `/etc/openldap/slapd.conf`:

    moduleload syncprov
    index entryCSN,entryUUID eq
    overlay syncprov
    syncprov-checkpoint 100 10
    syncprov-sessionlog 200

- on mail2 (SLAVE), update `/etc/openldap/slapd.conf`:

Attention

You can find password of bind dn `cn=vmail,dc=xx,dc=xx` in Postfix LDAP query files under `/etc/postfix/ldap/`.

    syncrepl rid=001
             provider=ldap://mail1:389
             searchbase="dc=iredmail,dc=kom"
             bindmethod=simple
             binddn="cn=vmail,dc=iredmail,dc=kom"
             credentials=erec3xiThBUW9QnnU9Bnifp3434
             schemachecking=on
             type=refreshOnly
             retry="60 +"
             scope=sub
             interval=00:00:01:00
             attrs="*,+"

on both servers set firewalld to accept gluster port, ldap port, and database to each servers, or you can set your own rules:

    firewall-cmd --permanent \
         --zone=iredmail \
         --add-rich-rule='rule family="ipv4" source address="192.168.1.3/24" port protocol="tcp" port="389" accept'

    firewall-cmd --permanent \
        --zone=iredmail \
        --add-rich-rule='rule family="ipv4" source address="192.168.1.4/24" port protocol="tcp" port="3306" accept'

    firewall-cmd --zone=iredmail --permanent --add-port=111/udp
    firewall-cmd --zone=iredmail --permanent --add-port=24007/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=24008/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=24009/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=139/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=445/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=965/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=2049/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=38465-38469/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=631/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=963/tcp
    firewall-cmd --zone=iredmail --permanent --add-port=49152-49251/tcp

reload firewall rules:

    firewall-cmd --complete-reload

Restart OpenLDAP service:

    systemctl restart slapd

## Configure MariaDB replication (Master-Master)

- on mail1, update `/etc/my.cnf`:

    server-id                   = 1
        log_bin                 = /var/log/mariadb/mariadb-bin.log
        log-slave-updates
        log-bin-index           = /var/log/mariadb/log-bin.index
        log-error               = /var/log/mariadb/error.log
        relay-log               = /var/log/mariadb/relay.log
        relay-log-info-file     = /var/log/mariadb/relay-log.info
        relay-log-index         = /var/log/mariadb/relay-log.index
        auto_increment_increment = 10
        auto_increment_offset   = 1
        binlog_do_db            = amavisd
        binlog_do_db            = iredadmin
        binlog_do_db            = roundcubemail
        binlog_do_db            = sogo
        binlog-ignore-db=test
        binlog-ignore-db=information_schema
        binlog-ignore-db=mysql
        binlog-ignore-db=iredapd
        log-slave-updates
        replicate-ignore-db=test
        replicate-ignore-db=information_schema
        replicate-ignore-db=mysql
        replicate-ignore-db=iredapd

Restart MariaDB service:

    systemctl restart mariadb

*on mail2, update `/etc/my.cnf`:

    server-id                   = 2
        log_bin                 = /var/log/mariadb/mariadb-bin.log
        log-slave-updates
        log-bin-index           = /var/log/mariadb/log-bin.index
        log-error               = /var/log/mariadb/error.log
        relay-log               = /var/log/mariadb/relay.log
        relay-log-info-file     = /var/log/mariadb/relay-log.info
        relay-log-index         = /var/log/mariadb/relay-log.index
        auto_increment_increment = 10
        auto_increment_offset = 1
        binlog_do_db            = amavisd
        binlog_do_db            = iredadmin
        binlog_do_db            = roundcubemail
        binlog_do_db            = sogo
        binlog-ignore-db=test
        binlog-ignore-db=information_schema
        binlog-ignore-db=mysql
        binlog-ignore-db=iredapd
        log-slave-updates
        replicate-ignore-db=test
        replicate-ignore-db=information_schema
        replicate-ignore-db=mysql
        replicate-ignore-db=iredapd

Restart MariaDB service:

    systemctl restart mariadb

### create replicator dbuser on both servers

- on mail1, login as MariaDB root user, then execute sql commands below:

    create user 'replicator'@'%' identified by '12345678';
    grant replication slave on *.* to 'replicator'@'%';
    SHOW MASTER STATUS;
    +--------------------+----------+----------------------------------------------+-------------------------------+
    | File               | Position | Binlog_Do_DB                                 | Binlog_Ignore_DB              |
    +--------------------+----------+----------------------------------------------+-------------------------------+
    | mariadb-bin.000001 |      245 | amavisd,iredadmin,iredapd,roundcubemail,sogo | test,information_schema,mysql |
    +--------------------+----------+----------------------------------------------+-------------------------------+

check master status in column `File` and `Position`:

- on mail2:

    create user 'replicator'@'%' identified by '12345678';
    grant replication slave on *.* to 'replicator'@'%';
    slave stop;
    CHANGE MASTER TO MASTER_HOST = '192.168.1.3', MASTER_USER = 'replicator', MASTER_PASSWORD = '12345678', MASTER_LOG_FILE = 'mariadb-bin.000001', MASTER_LOG_POS = 245;
    slave start;
    SHOW MASTER STATUS;
    +--------------------+----------+----------------------------------------------+-------------------------------+
    | File               | Position | Binlog_Do_DB                                 | Binlog_Ignore_DB              |
    +--------------------+----------+----------------------------------------------+-------------------------------+
    | mariadb-bin.000001 |     289 | amavisd,iredadmin,iredapd,roundcubemail,sogo | test,information_schema,mysql |
    +--------------------+----------+----------------------------------------------+-------------------------------+

    show slave status\G;

- change to your own master status MASTER_LOG_FILE is from `File`, MASTER_LOG_POS is from `Position` of master mail1
- check master status for `File` and `Position`

Restart MariaDB service:

    systemctl restart mariadb

- on mail1, login as MariaDB root user:

    slave stop;
    CHANGE MASTER TO MASTER_HOST = '192.168.1.4', MASTER_USER = 'replicator', MASTER_PASSWORD = '12345678', MASTER_LOG_FILE = 'mariadb-bin.000001', MASTER_LOG_POS = 289;
    slave start;
    show slave status\G;
    exit;

- change to your own master status MASTER_LOG_FILE is from `File`, MASTER_LOG_POS is from `Position` of master mail2*.

Restart MariaDB service:

    systemctl restart mariadb

- reboot one of mailserver and wait until up, then reboot the other mailserver

## Testing

- For HA Testing, you can try to shutdown one of your server to test it (ha1 or ha2 --/OR-- mail1 or mail2)
- you can create users using iredadmin on mail1, then check users from mail2 and you should see it's already synced
- try to login using roundcubemail from any mailserver, then you can check users on database 'roundcubemail->users', and it should already have synced
- only mail1 'can add and modify' users
- this mailservers acts as Glusterserver & Glusterclient, if you want to reboot the servers, please reboot first server until this up, then reboot the second server.
- if all servers are rebooted at the same time it will not mount '/var/vmail' folder. you must force mount manually using this command 'gluster volume start mailrep-volume force'

To view the DB easily, you may want to install adminer from http://adminer.org/ (it's a web-based SQL management tool, just a single PHP file):