{"id":3200,"date":"2018-03-27T18:06:41","date_gmt":"2018-03-27T14:06:41","guid":{"rendered":"https:\/\/www.howtoforge.com\/tutorial\/ubuntu-drbd-heartbeat-high-availability\/"},"modified":"2018-03-27T18:06:41","modified_gmt":"2018-03-27T14:06:41","slug":"how-to-setup-mariadb-high-availability-with-heartbeat-and-drbd-on-ubuntu-16-04-lts","status":"publish","type":"post","link":"https:\/\/afaghhosting.net\/blog\/how-to-setup-mariadb-high-availability-with-heartbeat-and-drbd-on-ubuntu-16-04-lts\/","title":{"rendered":"How to Setup MariaDB High Availability with Heartbeat and DRBD on Ubuntu 16.04 LTS"},"content":{"rendered":"<div><img decoding=\"async\" src=\"https:\/\/afaghhosting.net\/blog\/wp-content\/uploads\/2018\/03\/how-to-setup-mariadb-high-availability-with-heartbeat-and-drbd-on-ubuntu-16-04-lts.gif\" class=\"ff-og-image-inserted\" alt=\"\" title=\"\"><\/div>\n<p>Heartbeat and DRBD both are used for a cluster solution for any application using two servers. Both servers are work in active and passive mode, one server will work at the same time and another server as a backup server. DRBD (Distributed Replicated Block Device) is a kernel-level service that synchronizes data between two servers in real-time. Heartbeat is an open source program that allows a primary and a backup Linux server to determine if the other is &#8220;alive&#8221; and if the primary isn&#8217;t, failover resources to the backup. It will also manage the IP high availability and other services in your servers.<\/p>\n<p>In this tutorial, we will learn how to achieve high availability of MariaDB using Heartbeat and DRBD on Ubuntu 16.04 server.<\/p>\n<h2 id=\"requirements\">Requirements<\/h2>\n<ul>\n<li>Two nodes with Ubuntu 16.04 server installed.<\/li>\n<li>Two network cards installed on each node.<\/li>\n<li>Additional unpartitioned hard drive installed on each node.<\/li>\n<li>Non root user with sudo privileges setup on each node.<\/li>\n<\/ul>\n<h2 id=\"getting-started\">Getting Started<\/h2>\n<p>Before starting, you will need to setup IP address on each node. Use the following IP address on each node:<\/p>\n<p class=\"highlight\">Node1 :<\/p>\n<p>172.16.0.1 on eth0 and 192.168.0.101 on eth1<\/p>\n<p class=\"highlight\">Node2 :<\/p>\n<p>172.16.0.2 on eth0 and 192.168.0.102 on eth1<\/p>\n<p>IP 192.168.0.103 will be the high availability IP.<\/p>\n<p>Next, you will also need to setup hostname and hostname resolution on each node. So each node can communicate with each other. On the first Node, open the <span class=\"system\">\/etc\/hosts<\/span> file and <span class=\"system\">\/etc\/hostname<\/span> file:<\/p>\n<p class=\"command\">sudo nano \/etc\/hosts<\/p>\n<p>Add the following lines at the end of the file:<\/p>\n<pre>172.16.0.1 Node1&#13;\n172.16.0.2 Node2&#13;\n<\/pre>\n<p class=\"command\">sudo nano \/etc\/hostname<\/p>\n<p>Change the file as shown below:<\/p>\n<pre>Node1&#13;\n<\/pre>\n<p>Save and close the file when you are finished.<\/p>\n<p>On the second node, open the <span class=\"system\">\/etc\/hosts<\/span> file and <span class=\"system\">\/etc\/hostname<\/span> file:<\/p>\n<p class=\"command\">sudo nano \/etc\/hosts<\/p>\n<p>Add the following lines at the end of the file:<\/p>\n<pre>172.16.0.1 Node1&#13;\n172.16.0.2 Node2&#13;\n<\/pre>\n<p class=\"command\">sudo nano \/etc\/hostname<\/p>\n<p>Change the file as shown below:<\/p>\n<pre>Node2&#13;\n<\/pre>\n<p>Save and close the file when you are finished.<\/p>\n<p>Next, update each node with the latest version with the following command:<\/p>\n<p class=\"command\">sudo apt-get update -y<br \/>sudo apt-get upgrade -y<\/p>\n<p>Once your system is updated, restart the system to apply these changes.<\/p>\n<h2 id=\"install-drbd-and-heartbeat\">Install DRBD and Heartbeat<\/h2>\n<p>Next, you will need to install DRBD and Heartbeat on both nodes. By default, both are available in Ubuntu 16.04 default repository. You can install them by just running the following command on both Nodes:<\/p>\n<p class=\"command\">sudo apt-get install drbd8-utils heartbeat -y<\/p>\n<p>Next, start DRBD and Heartbeat service and enable them to start on boot time:<\/p>\n<p class=\"command\">sudo systemctl start drbd<br \/>sudo systemctl start heartbeat<br \/>systemctl enable drbd<br \/>systemctl enable heartbeat<\/p>\n<h2 id=\"configure-drbd-and-heartbeat\">Configure DRBD and Heartbeat<\/h2>\n<p>Next, you will need to setup DRBD device on each Node. Create a single partition on second unpartitioned drive <span class=\"system\">\/dev\/sdb<\/span> on each Node.<\/p>\n<p>You can do this by just running the following command on each Node:<\/p>\n<p class=\"command\">sudo echo -e &#8216;n\\np\\n1\\n\\n\\nw&#8217; | fdisk \/dev\/sdb<\/p>\n<p>Next, you will need to configure DRBD on both nodes. You can do this by creating <span class=\"system\">\/etc\/drbd.d\/r0.res<\/span> file on each Node.<\/p>\n<p class=\"command\">sudo nano \/etc\/drbd.d\/r0.res<\/p>\n<p>Add the following lines:<\/p>\n<pre>global {&#13;\nusage-count no;&#13;\n}&#13;\nresource r0 {&#13;\nprotocol C;&#13;\nstartup {&#13;\ndegr-wfc-timeout 60;&#13;\n}&#13;\ndisk {&#13;\n}&#13;\nsyncer {&#13;\nrate 100M;&#13;\n}&#13;\nnet {&#13;\ncram-hmac-alg sha1;&#13;\nshared-secret \"aBcDeF\";&#13;\n}&#13;\non Node1 {&#13;\ndevice \/dev\/drbd0;&#13;\ndisk \/dev\/sdb1;&#13;\naddress 172.16.0.1:7789;&#13;\nmeta-disk internal;&#13;\n}&#13;\non Node2 {&#13;\ndevice \/dev\/drbd0;&#13;\ndisk \/dev\/sdb1;&#13;\naddress 172.16.0.2:7789;&#13;\nmeta-disk internal;&#13;\n}&#13;\n}&#13;\n<\/pre>\n<p>Save and close the file when you are finished, then open another configuration file on each Node:<\/p>\n<p class=\"command\">sudo nano \/etc\/ha.d\/ha.cf<\/p>\n<p>Add the following lines:<\/p>\n<pre># Check Interval&#13;\nkeepalive 1&#13;\n# Time before server declared dead&#13;\ndeadtime 10&#13;\n# Secondary wait delay at boot&#13;\ninitdead 60&#13;\n# Auto-failback&#13;\nauto_failback off&#13;\n# Heartbeat Interface&#13;\nbcast eth1&#13;\n# Nodes to monitor&#13;\nnode Node1&#13;\nnode Node2&#13;\n<\/pre>\n<p>Save and close the file.<\/p>\n<p>Next, open the resource file <span class=\"system\">\/etc\/ha.d\/haresources<\/span> on each Node:<\/p>\n<p class=\"command\">sudo nano \/etc\/ha.d\/haresources<\/p>\n<p>Add the following lines:<\/p>\n<pre>Node1 192.168.0.103\/24 drbddisk::r0 Filesystem::\/dev\/drbd0::\/var\/lib\/mysql::ext4::noatime&#13;\n<\/pre>\n<p>Here, <span class=\"system\">Node1<\/span> is the hostname of your main active node, <span class=\"system\">192.168.0.103<\/span> is the floating point IP address, <span class=\"system\">\/var\/lib\/mysql<\/span> is a mount point and <span class=\"system\">\/dev\/drbd0<\/span> is DRBD device.<\/p>\n<p>Next, you will need to need to define and store identical authorization keys on both nodes. You can do this by <span class=\"system\">\/etc\/ha.d\/authkeys<\/span> file on each Node:<\/p>\n<p class=\"command\">sudo nano \/etc\/ha.d\/authkeys<\/p>\n<p>Add the following lines:<\/p>\n<pre>auth1&#13;\n1 sha1 your-secure-password&#13;\n<\/pre>\n<p>Here, your-secure-password is your secure password. Use the same password on both nodes.<\/p>\n<p>Next, create and start DRBD by running the following command on Node1:<\/p>\n<p class=\"command\">sudo drbdadm create-md r0<br \/>sudo systemctl restart drbd<br \/>sudo drbdadm outdate r0<br \/>sudo drbdadm &#8212; &#8211;overwrite-data-of-peer primary all<br \/>sudo drbdadm primary r0<br \/>sudo mkfs.ext4 \/dev\/drbd0<br \/>sudo chmod 600 \/etc\/ha.d\/authkeys<br \/>sudo mkdir \/var\/lib\/mysql<\/p>\n<p>Once the DRBD disk is created on Node1, create the DRBD disk on Node2 with the following command:<\/p>\n<p class=\"command\">sudo drbdadm create-md r0<br \/>sudo systemctl restart drbd<br \/>sudo chmod 600 \/etc\/ha.d\/authkeys<br \/>sudo mkdir \/var\/lib\/mysql<\/p>\n<p>Now, you can verify the DRBD disk is connected and is properly syncing by running the following command:<\/p>\n<p class=\"command\">sudo cat \/proc\/drbd<\/p>\n<p>If everything is fine, you should see the following output:<\/p>\n<pre>version: 8.4.5 (api:1\/proto:86-101)&#13;\nsrcversion: F446E16BFEBS8B115AJB14H&#13;\n0: cs:SyncSource ro:Primary\/Secondary ds:UpToDate\/Inconsistent C r-----&#13;\nns:210413 nr:0 dw:126413 dr:815311 al:35 bm:0 lo:0 pe:11 ua:0 ap:0 ep:1 wo:f oos:16233752&#13;\n[&gt;....................] sync'ed: 3.3% (14752\/14350)M&#13;\nfinish: 0:12:23 speed: 12,156 (16,932) K\/sec&#13;\n<\/pre>\n<p>Next, start heartbeat on both Nodes to enable the failover portion of your setup.<\/p>\n<p class=\"command\">sudo systemctl start heartbeat<\/p>\n<p>Next, verify the mounted DRBD partition with the following command on Node1:<\/p>\n<p class=\"command\">sudo mount | grep drbd<\/p>\n<p>You should see the following output:<\/p>\n<pre>\/dev\/drbd0 on \/var\/lib\/mysql type ext4 (rw,noatime,data=ordered)&#13;\n<\/pre>\n<p>Next, verify the floating IP is only bound to Node1 with the following command:<\/p>\n<p class=\"command\">sudo ip addr show | grep 192.168.0.103<\/p>\n<p>You should see the following output:<\/p>\n<pre>inet 192.168.0.103\/24 brd 192.168.0.255 scope global secondary eth1:0&#13;\n<\/pre>\n<h2 id=\"install-and-configure-mariadb\">Install and Configure MariaDB<\/h2>\n<p>Once everything is configured properly on both Nodes, it&#8217;s time to install MariaDB server on both Nodes.<\/p>\n<p>Run the following command on both Nodes to install MariaDB server:<\/p>\n<p class=\"command\">sudo apt-get install mariadb-server -y<\/p>\n<p>Next, you will need to disable MariaDB service on both Nodes:<\/p>\n<p class=\"command\">sudo systemctl disable mysql<\/p>\n<p>Here, we will use Node1 as primary and databases on Node2 should be created and populated through synchronization with Node1. So you will need to stop MariaDB service and remove content inside <span class=\"system\">\/var\/lib\/mysql<\/span> on Node2. You can do this with the following command:<\/p>\n<p class=\"command\">sudo systemctl stop mysql<br \/>sudo rm -rf \/var\/lib\/mysql\/*<\/p>\n<p>Next, you will need to copy the MySQL Maintenance configuration file from Node1 to Node2. You can do this by running the following command:<\/p>\n<p class=\"command\">sudo scp \/etc\/mysql\/debian.cnf <a href=\"https:\/\/www.howtoforge.com\/cdn-cgi\/l\/email-protection\" class=\"__cf_email__\" data-cfemail=\"98eaf7f7ecd8a9afaab6a9aeb6a8b6aa\" target=\"_blank\" rel=\"noopener\">[email\u00a0protected]<\/a>:\/etc\/mysql\/debian.cnf<\/p>\n<p>Next, you will need to create a root user for remote management of and access to the databases on the highly available MySQL instance.<\/p>\n<p>You can do this by running the following command on Node1:<\/p>\n<p class=\"command\">mysql -u root -p<\/p>\n<p>Enter your root password, then create a root user with the following command:<\/p>\n<p class=\"command\">MariaDB [(none)]&gt; CREATE USER &#8216;root&#8217;@&#8217;192.168.0.%&#8217; IDENTIFIED BY &#8216;password&#8217;;<br \/>MariaDB [(none)]&gt; GRANT ALL PRIVILEGES ON *.* TO &#8216;root&#8217;@&#8217;192.168.0..%&#8217; WITH GRANT OPTION;<br \/>MariaDB [(none)]&gt; FLUSH PRIVILEGES;<br \/>MariaDB [(none)]&gt; QUIT;<\/p>\n<p>Next, Set the bind address for MySQL on both Nodes with the following command:<\/p>\n<p class=\"command\">sudo sed -i &#8216;s\/127.0.0.1\/0.0.0.0\/g&#8217; \/etc\/mysql\/mariadb.conf.d\/*.cnf<\/p>\n<h2 id=\"initiate-heartbeat-for-mariadb-service\">Initiate Heartbeat for MariaDB Service<\/h2>\n<p>Next, you will need to add the MariaDB service in your heartbeat instances on both Nodes. You can do this by editing \/etc\/ha.d\/haresources file:<\/p>\n<p class=\"command\">sudo nano \/etc\/ha.d\/haresources<\/p>\n<p>Modify the following lines:<\/p>\n<pre>Node1 192.168.0.103\/24 drbddisk::r0 Filesystem::\/dev\/drbd0::\/var\/lib\/mysql::ext4::noatime mysql&#13;\n<\/pre>\n<p>Save and close the file, when you are finished.<\/p>\n<p>Once the heartbeat is configured, you will need to restart it on both Nodes.<\/p>\n<p>First, restart heartbea on Node1:<\/p>\n<p class=\"command\">sudo systemctl restart heartbeat<\/p>\n<p>Next, wait for 50 seconds, then restart heartbeat service on Node2:<\/p>\n<p class=\"command\">sudo systemctl restart heartbeat<\/p>\n<h2 id=\"test-heartbeat-and-drbd\">Test Heartbeat and DRBD<\/h2>\n<p>Now, everything is configured properly, it&#8217;s time to perform a series of tests to verify that heartbeat will actually trigger a transfer from the active server to the passive server when the active server fails in some way.<\/p>\n<p>First, verify that Node1 is the primary drbd node with the following command on Node1:<\/p>\n<p class=\"command\">sudo cat \/proc\/drbd<\/p>\n<p>You should see the following output:<\/p>\n<pre>version: 8.4.5 (api:1\/proto:86-101)&#13;\nsrcversion: F446E16BFEBS8B115AJB14H&#13;\nO cs:Connected ro:Primary\/Secondary ds:UpToDate\/UpToDate C r-----&#13;\nns:22764644 nr:256 dw:529232 dr:22248299 al:111 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0&#13;\n<\/pre>\n<p>Next, we will verify that the DRBD disk is mounted with the following command:<\/p>\n<p class=\"command\">sudo mount | grep drbd<\/p>\n<pre>\/dev\/drbd0 on \/var\/lib\/mysql type ext4 (rw,noatime,data=ordered)&#13;\n<\/pre>\n<p>Next, verify the MariaDB service with the following command:<\/p>\n<p class=\"command\">sudo systemctl status mysql<\/p>\n<p>Next, access MariaDB server from the remote machine using floating IP and create a test database:<\/p>\n<p class=\"command\">mysql -h 192.168.0.103 -u root -p<\/p>\n<p class=\"command\">MariaDB [(none)]&gt; create database test;<br \/>MariaDB [(none)]&gt; quit<\/p>\n<p>Next, restart heartbeat on Node1:<\/p>\n<p class=\"command\">sudo systemctl restart heartbeat<\/p>\n<p>Now, heartbeat will interpret this restart as a failure of MariaDB on Node1 and should trigger failover to make Node2 the primary server.<\/p>\n<p>You can check that DRBD is now treating Node1 as the secondary server with the following command on Node1:<\/p>\n<p class=\"command\">sudo cat \/proc\/drbd<\/p>\n<p>You should see the following output:<\/p>\n<pre>version: 8.4.5 (api:1\/proto:86-101)&#13;\nsrcversion: F446E16BFEBS8B115AJB14H&#13;\n0: cs:Connected ro:Secondary\/Primary ds:UpToDate\/UpToDate C r-----&#13;\nns:22764856 nr:388 dw:529576 dr:22248303 al:112 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0&#13;\n<\/pre>\n<p>Now, verify that Node2 is the primary drbd node by running the following command on Node2:<\/p>\n<p class=\"command\">sudo cat \/proc\/drbd<\/p>\n<p>You should see the following output:<\/p>\n<pre>version: 8.4.5 (api:1\/proto:86-101)&#13;\nsrcversion: F446E16BFEBS8B115AJB14H&#13;\n0: cs:Connected ro:Primary\/Secondary ds:UpToDate\/UpToDate C r-----&#13;\nns:412 nr:20880892 dw:20881304 dr:11463 al:7 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0&#13;\n<\/pre>\n<p>Next, Check to make sure that MariaDB is running on Node2:<\/p>\n<p class=\"command\">sudo systemctl status mysql<\/p>\n<p>Now, connect to MariaDB server using floating IP on Node2 from remote user.<\/p>\n<p class=\"command\">mysql -h 192.168.0.103 -u root -p<\/p>\n<p>Next, view the test database we created earlier while Node1 was the primary server.<\/p>\n<p class=\"command\">MariaDB [(none)]&gt; show databases;<\/p>\n<p>You should see the following output:<\/p>\n<pre> +--------------------+&#13;\n| Database |&#13;\n+--------------------+&#13;\n| test |&#13;\n| information_schema |&#13;\n| lost+found |&#13;\n| mysql |&#13;\n| performance_schema |&#13;\n+--------------------+&#13;\n5 rows in set (0.04 sec)<\/pre>\n<div>\n<p><b>Share this page:<\/b><\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Heartbeat and DRBD both are used for a cluster solution for any application using two servers. Both servers are work in active and passive mode, one server will work at the same time and another server as a backup server. DRBD (Distributed Replicated Block Device) is a kernel-level service that synchronizes data between two servers [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[36],"tags":[],"class_list":["post-3200","post","type-post","status-publish","format-standard","hentry","category-36"],"_links":{"self":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts\/3200","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/comments?post=3200"}],"version-history":[{"count":0,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts\/3200\/revisions"}],"wp:attachment":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/media?parent=3200"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/categories?post=3200"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/tags?post=3200"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}