Tutorials

Set up a Three-Node Crate Cluster

Table of Contents

Introduction

Crate.IO (Crate) is an elastic distributed SQL data store. We have covered installation on a few different Linux distributions as separate tutorials. This tutorial is designed to demonstrate setting up a basic Crate cluster.

Requirements

  • A ProfitBricks account and access to the DCD (Data Center Designer)
  • Three servers with Crate installed.

Installation

If you need to set up three Crate servers quickly, follow one of the existing installation tutorials:

For this tutorial, we have Crate running on one node of each operating system. All three nodes are set up in a single data center. Each node has NIC 0 connected to LAN 1 and has a public IP address assigned by DHCP. It looks like this in the DCD:

Crate Cluster DCD starting point

We will add a new internal (or private) network and connect NIC 1 on all three servers to it:

Crate Cluster DCD add network

After provisioning those changes, take a look at a server's Network tab in the DCD to see what IP addresses were assigned to NIC 1:

Crate Cluster DCD NIC detail

Depending on the operating system, we could also use ifconfig -a or ip addr show to see which IP addresses have been assigned.

For the tutorial, we have the following DHCP-assigned IP addresses to work with:

| Server    | IP Address  |
| --------- | ----------  |
| Ubuntu 15 | 10.12.98.11 |
| Debian 8  | 10.12.98.12 |
| CentOS 7  | 10.12.98.13 |

Configuration

Configuration information for Crate is stored in the /etc/crate/crate.yml text file. We will be making several changes to that file in order to configure the cluster. By default, Crate will make use of multicast to find other nodes. Since multicast is not available on our network we will need to use unicast and specify the individual nodes by IP address.

Inside the /etc/crate/crate.yml file we will accomplish the following tasks:

  • Rename the cluster
  • Rename the nodes
  • Change the interface/IP that Crate is listening on
  • Disable multicast
  • Enable unicast and add the IP addresses of the other nodes and hosts

First off, we will rename the cluster. Around line 20 in the crate.yml file, we find this section:

20 ################################### Cluster ###################################
21
22 # Cluster name identifies your cluster for auto-discovery. If you're running
23 # multiple clusters on the same network, make sure you're using unique names.
24 #
25 cluster.name: tutorial

We will uncomment line 25 and change the name to: "tutorial".

Next we will change the name of the nodes. Crate assigns a random "fun" name to each node unless we specify one. Around line 60 we find this section:

60 #################################### Node #####################################
61
62 # Node names are generated dynamically on startup, so you're relieved
63 # from configuring them manually. You can tie this node to a specific name:
64 #
65 # node.name: "Franz Kafka"
66 node.name: "Node 1"

In this case, we copy line 65 and paste it in as line 66, then uncomment it and rename it to "Node 1" for the first Crate node in our cluster. The others will be named "Node 2" and "Node 3" respectively.

Now we want to change the IP address that Crate is listening on by specifying the internal IP. This is so the cluster will communicate with its nodes over the internal/private network, rather than the external/public one. For this tutorial, the IP would be 10.12.98.11 for "Node 1". The change is made in the "Network And HTTP" section which starts around line 272.

272 ############################## Network And HTTP ###############################
273
274 # Crate, by default, binds itself to the 0.0.0.0 address, and listens
275 # on port [4200-4300] for HTTP traffic and on port [4300-4400] for node-to-node
276 # communication. (the range means that if the port is busy, it will automatically
277 # try the next port).
278
279 # Set the bind address specifically (IPv4 or IPv6):
280 #
281 # network.bind_host: 192.168.0.1
282
283 # Set the address other nodes will use to communicate with this node. If not
284 # set, it is automatically derived. It must point to an actual IP address.
285 #
286 # network.publish_host: 192.168.0.1
287 network.publish_host: 10.12.98.11

We can copy line 286 and paste it as 287, then uncomment it and change the 192.168.0.1 entry to 10.12.98.11 for "Node 1". I will use 10.12.98.12 for "Node 2", and 10.12.98.13 for "Node 3". You will need to substitute in the appropriate IP addresses for your setup.

Since we are already editing the crate.yml file, we will update two items in the Gateway section starting around line 314. As the comments in crate.yml indicate, these options are related to how a Crate cluster handles recovery.

314 ################################### Gateway ###################################
315
316 # The gateway persists cluster meta data on disk every time the meta data
317 # changes. This data is stored persistently across full cluster restarts
318 # and recovered after nodes are started again.
319
320 # Defines the number of nodes that need to be started before any cluster
321 # state recovery will start.
322 #
323 gateway.recover_after_nodes: 2
324
325 # Defines the time to wait before starting the recovery once the number
326 # of nodes defined in gateway.recover_after_nodes are started.
327 #
328 #gateway.recover_after_time: 5m
329
330 # Defines how many nodes should be waited for until the cluster state is
331 # recovered immediately. The value should be equal to the number of nodes
332 # in the cluster.
333 #
334 gateway.expected_nodes: 3

Uncomment lines 323 and 334 making sure they are set to 2 and 3 respectively. These values will need to be adjusted if more nodes are added to the cluster in the future.

In the "Discovery" section starting around line 430:

430 ################################## Discovery ##################################
431
432 # Discovery infrastructure ensures nodes can be found within a cluster
433 # and master node is elected. Multicast discovery is the default.
434
435 # Set to ensure a node sees M other master eligible nodes to be considered
436 # operational within the cluster. Its recommended to set it to a higher value
437 # than 1 when running more than 2 nodes in the cluster.
438 #
439 # We highly recommend to set the minimum master nodes as follows:
440 #   minimum_master_nodes: (N / 2) + 1 where N is the cluster size
441 # That will ensure a full recovery of the cluster state.
442 #
443 discovery.zen.minimum_master_nodes: 2

Uncomment line 443 and increase the value from 1 to 2.

Multicast discovery is disabled around line 466.

464 # 1. Disable multicast discovery (enabled by default):
465 #
466 discovery.zen.ping.multicast.enabled: false
467 #
468 # 2. Configure an initial list of master nodes in the cluster
469 #    to perform discovery when new nodes (master or data) are started:
470 #
471 # If you want to debug the discovery process, you can set a logger in
472 # 'config/logging.yml' to help you doing so.
473 #
474 discovery.zen.ping.unicast.hosts:
475    - 10.12.98.11:4300
476    - 10.12.98.12:4300
477    - 10.12.98.13:4300

We have uncommented lines 466 and 474, and added the host ip addresses for each node on lines 475 through 477. Make sure and save the changes to crate.yml.

Finally we can restart Crate as appropriate for your specific OS. (sudo service crate restart on Ubuntu/Debian, or sudo systemctl restart crate.service on CentOS)

Test the Cluster

We should be able to access the Crate admin web interface by connecting to the public IP address of one of the three nodes on port 4200. The URL is http://YOUR_IP:4200/admin.

Crate Cluster Admin Interface

Selecting the bottom icon on the left navigation section of the admin interface takes us to a cluster details page. All three of our nodes are showing up here:

Crate Cluster Admin Interface Detail

Troubleshooting

Logs are kept in /var/log/crate/ by default. If you have problems, this is a good place to start trying to figure out what is happening with the cluster.

A firewall is a likely source of trouble when getting a cluster set up for the first time. The nodes need to be able to communicate with each other on port 4300. Port 4200 is used for for Crate's REST API and accessing the admin web interface.

How you accomplish this is going to vary depending on which OS you are running in your cluster. The tutorial cluster involved three different Linux distributions and three different firewalls. The changes that were made to "Node 1" which is running ufw on Ubuntu 15 looked like this:

sudo ufw allow proto tcp from 10.12.98.12 to 10.12.98.11 port 4300
sudo ufw allow proto tcp from 10.12.98.13 to 10.12.98.11 port 4300

Those two rules allow traffic from "Node 2" and "Node 3" to "Node 1" on port 4300. The status of the firewall on "Node 1" is:

sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    ADMIN_IP
4200/tcp                   ALLOW IN    ADMIN_IP
10.12.98.11 4300/tcp       ALLOW IN    10.12.98.12
10.12.98.11 4300/tcp       ALLOW IN    10.12.98.13

"Node 2" was configured with Debian 8. No firewall was enabled by default and I chose to use the ProfitBricks firewall configured in the DCD to control external access to NIC 0. No firewall is active on the internal interface, NIC 1.

"Node 3" is running CentOS 7 with firewalld active. I made use of the built in "internal" zone. To allow the three nodes to properly communicate, eth1 was placed into the "internal" zone.

sudo firewall-cmd --zone=internal --change-interface=eth1

Then rules were added to "internal" as follows:

sudo firewall-cmd --zone=internal --add-port=4300/tcp --permanent
sudo firewall-cmd --zone=internal --add-port=4200/tcp --permanent
sudo firewall-cmd --zone=internal --add-source=10.12.98.11/32 --permanent
sudo firewall-cmd --zone=internal --add-source=10.12.98.12/32 --permanent

The result is:

sudo firewall-cmd --zone=internal --list-all
internal (active)
  interfaces: eth1
  sources: 10.12.98.11/32 10.12.98.12/32
  services: dhcpv6-client ipp-client mdns samba-client ssh
  ports: 4200/tcp 4300/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

Those were the changes used to make this example setup work. The tutorial cluster was deliberately built in this non-standard way so that we would have a chance to look at a few different firewall configurations. Most clusters will be built using the same OS across all nodes to simplify deployment and administration.

Placing the entire cluster on private IP addresses behind a dedicated firewall, perhaps running something like IPFire, could simplify the configuration. Using static IP addresses is also recommended, as DHCP-assigned addresses can change when a node reboots.

Summary

We have our three-node cluster up and running. After importing some tweets as sample data, we can see everything is healthy:

Crate Cluster Admin Interface Detail

If you would like to explore Crate further, take a look at the excellent Crate.IO Documentation. If you have any feedback, leave a comment below or in the ProfitBricks Community section of this site.