Tutorials

Cloud Failover & High Availability Solution

Here at ProfitBricks, our SLA guarantees a 99.95% availability for hardware, networks and data for our customers on our hosted systems. Customers can enhance their availability by ensuring they have two machines, one in each fire zone. This post is part of a series that demonstrates how to leverage the power of a load balancing to create a failover cloud solution.

For systems that are directly connected to the internet, this can (at first glance) look pretty difficult. But don’t worry - this is what the Virtual Router Redundancy Protocol (VRRP) is for – giving you a lot of bang for your buck by enabling you to share only one IP address among numerous machines using a round-robin approach to assigning the incoming IP addresses.

We’re going to build an example of this simple solution in this blog post. We’ll use the Keepalived (http://www.keepalived.org) tool, as a failover and load balancing solution. We’re going to install a web server (Apache) directly on the VMs. This gives you a solution (with Keepalived) that can easily be created using minimal resources. A more complex scenario will be described in a following blog post.

This type of setup is recommended for productive usability for all your public IP addresses (as it is described here below).

Our basic DCD setup:

DCD Diagram

For the purpose of this test environment, we will use two Debian 7 machines, installed and configured identically.

All steps must be carried out on both machines (unless it’s stated otherwise), so please make sure to adjust your IP addresses and the other modifications mentioned below to each configuration.

We will need additional IP addresses for this particular scenario. We can easily reserve new IP Addresses in the DCD (look in the “IP Manager” section):

Reserve IPs

We add additional IP Addresses (162.254.25.182) to the VRRP master.

Add IPs

I have the following network configuration in my test setup.

VRRP Master

Public IP: eth0 162.254.26.80

Private IP: eth1 10.10.26.12

VRRP Slave

Public IP: eth0 162.254.26.81

Private IP: eth1 10.10.26.11

Virtual IP(VRRP Master)

Public IP: eth0 162.254.25.182

Now we come to your operating system configuration. We install Keepalived, a routing software for load balancing and high availability which the VRR protocol will use.

> apt-get update    
> apt-get upgrade    
> apt-get install apache2    
> apt-get install keepalived    
> apt-get install ipvsadm

We will require unique hostnames so that the failover will work properly.

I named the hosts “vrrpmaster” and “vrrpslave”.

> vi /etc/hostname

Then performed the following script:

> /etc/init.d/hostname.sh

The following image is used and given as an option in our standard default images, but if you want to use your own image now you can. Here’s configuring your network interface with DHCP.

> vi /etc/network/interfaces
   auto eth1
   iface eth1 inet dhcp

> dhclient eth1

We just created the config data for Keepalived.

** Remember to adjust the IP Addresses of your environment.

> vi /etc/keepalived/keepalived.conf

# Configuration File for Keepalived

global_defs {
  router_id LVS_MASTER # string identifying the machine
}

# describe virtual service ip
# VRRP Configuration
vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 1
  priority 100
  authentication {
    auth_type PASS
    auth_pass secretpassphrase
  }
  virtual_ipaddress {
    162.254.25.182/32
  }
  # Invoked to master transition
  notify_master "/etc/keepalived/bypass_ipvs.sh del 162.254.25.182"
  # Invoked to slave transition
  notify_backup "/etc/keepalived/bypass_ipvs.sh add 162.254.25.182"
  # Invoked to fault transition
  notify_fault "/etc/keepalived/bypass_ipvs.sh add 162.254.25.182"
  }

virtual_server 162.254.25.182 80 {
  delay_loop 30
  lb_algo rr
  lb_kind DR
  persistence_timeout 50
  protocol TCP

  real_server 10.10.26.12 80 {
    HTTP_GET {
      url {
        path /
        digest 21dde95d9d269cbb2fa6560309dca40c
      }
      connect_timeout 3
      nb_get_retry 3
      delay_before_retry 2
    }
  }
  real_server 10.10.26.11 80 {
    HTTP_GET {
      url {
        path /
        digest 21dde95d9d269cbb2fa6560309dca40c
      }
      connect_timeout 3
      nb_get_retry 3
      delay_before_retry 2
      }
    }
  }

Replace the following information with your IP Addresses.

162.254.25.182 -> Virtual IP Address 10.10.26.12 -> VRRP Master Private IP(eth1 IP) 10.10.26.11 -> VRRP Slave Private IP(eth1 IP)

On the VRRP Slave conf file we have to make the following changes:

router_id LVS_MASTER -> router_id LVS_SLAVE state MASTER -> state BACKUP priority 100 -> priority 50

Under auth_pass you can also enter a password. This needs to be identical on all your additional VMs.

Depending on these values, Keepalived checks whether the host is available. In our case, this is the path to the index page. These are the standard hash-values from Apache for this standard page. When the index.html data is changed, the Hash-values will change too. You can check the values with this command:

> cd /var/www/
> genhash -s 10.10.26.12 -p 80 -u /index.html
MD5SUM = 21dde95d9d269cbb2fa6560309dca40c

MDSUM

At this point your keepalived.conf is complete.

In this minimal configuration it’s vital that the machine where the “SLAVE” mode is located has an active NAT rule. This is necessary to ensure that the active Web server (with the virtual IP active - on the master node) is in fact reachable. For this, we use the following script “/etc/keepalived/bypass_ipvs.sh” from Gael Charrière.

#! /bin/sh
#
# Gael Charriere <gael.charriere@gmail.com>
# 10.11.2008
#
# Invoked by keepalived from master/slave
# to slave/master transition to add or remove
# a PREROUTING rule
#
# Essential for slave to redirect incoming
# service packet to localhost. Otherwise a
# loop can appear between master and slave.
#
# The routing table is consulted when a packet
# that creates a new connection is encountered.
# PREROUTING rule alters packets as soon as they come in.
# REDIRECT statement redirects the packet to the machine
# itself by changing the destination IP to the primary
# address of the incoming interface (locally-generated
# packets are mapped to the 127.0.0.1 address).

# Check number of command line args
EXPECTED_ARGS=2
if [ $# -ne $EXPECTED_ARGS ]; then
  echo "Usage: $0 {add|del} ipaddress"
  exit 1
fi

# Check if second arg is a valid ip address
VIP=$2
OLD_IFS=$IFS
IFS="."
VIP=( $VIP )
IFS=$OLD_IFS
# Check that ip has 4 parts
  if [ ${#VIP[@]} -ne 4 ]; then
    echo "IP address must have 4 parts"
    echo "Usage: $0 {add|del} ipaddress"
    exit 1
fi

# Check that each parts is a number which
# varies between 0 and 255
for oct in ${VIP[@]} ; do
    echo $oct | egrep "^[0-9]+$" >/dev/null 2>&1
  if [ $? -ne 0 ]; then
    echo "$oct: Not numeric"
    echo "Usage: $0 {add|del} ipaddress"
    exit 1
else
  if [ $oct -lt 0 -o $oct -gt 255 ]; then
    echo "$oct: Out of range"
    echo "Usage: $0 {add|del} ipaddress"
    exit 1
  fi
fi
done

# If we are here, ip address is validated
VIP="${VIP[0]}.${VIP[1]}.${VIP[2]}.${VIP[3]}"

# Add or remove the prerouting rule
case "$1" in
add)
        # check if the rule was already specified
        n=$(iptables -t nat -L| grep $VIP | wc -l)
  if [[ $n == 0 ]]; then
        # the rule was not found, add it
        iptables -A PREROUTING -t nat -d $VIP -p tcp -j REDIRECT
    fi
    ;;
    del)
        # check if the rule was already specified
      n=$(iptables -t nat -L| grep $VIP | wc -l)
      while [[ $n > 0 ]]; do
        # remove the rule
        iptables -D PREROUTING -t nat -d $VIP -p tcp -j REDIRECT
        n=$(($n-1))
      done
    ;;
    *)
    echo "Usage: $0 {add|del} ipaddress"
    exit 1
    esac
    exit 0

We set permissions to make sure the file can be executed:

> chmod 755 /etc/keepalived/bypass_ipvs.sh

IP forwarding needs to be on and IP reverse path filtering turned off. For this we have to make the following changes:

# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.eth0.rp_filter=0
net.ipv4.conf.eth1.rp_filter=0

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

Fill out the following fields:

> sysctl –p

These are the settings we need for this scenario.

We can start Keepalived using the following command (If you get an error notice it means that Keepalived is already running):

> service keepalived start

In Debian, Keepalived places messages under /var/log/Messages. If you want to have more text in your debugging messages, you can also start Keepalived manually (stating your wished-for options):

> keepalived -d –log-detail

We can now see our virtual IP Addresses at eth0 on VRRP Master. The address should respond to ping, and the Apache Webserver is also available at this IP address. In order to get information of our network interfaces we’ll use the following command:

> ip addr list

enter image description here

If the VRRP Master fails, the VRRP Slave will take over the IP address; after 1 or 2 pings the IP address will be available again.

enter image description here

You can run a similar test by stopping Keepalived.

> service keepalived stop

When you start Keepalived again, the IP address will automatically be given to the VRRP Master. This happens because of the higher priority for keepalived.conf files.

The load balancing function is based on a “Linux Virtual Server”, and is implemented as a Round Robin. Every new IP address is routed to a different host. If the VRRP Master fails, the VRRP Slave then picks up the slack.

It’s now possible to have an additional test on our Apache2 server. If we create an HTML file on both web servers.

Please note: When you adjust the “index.html” content, the hash value also changes, so make sure you generate and modify the conf file with the appropriate MD5SUM value.

> vi /var/www/index.html

Insert the following:

<HTML>
 <BODY>
   <H2>
   Welcome to WebServer 1
   </H2>
 </BODY>
</HTML>

Similar to the above, we change the html file to Test WebServer2 on the VRRP Slave.

When we submit an http request using the virtual IP address (http://162.254.25.182/index.html), we can see which server we have landed on. A second connection (another IP address) can be set up on another web server. You can then check the status of the load balancer component using the following command:

> ipvsadm -l

We have now set up a fail-safe system. If your VRRP Master (located in the first fire zone area) crashes, the VRRP Slave (located in the second fire zone area) is there to pick up the slack.

For a more complex load balancing solution, please check out this tutorial: [Cloud Failover & High Availability Solution II][1]