{"id":22782,"date":"2024-09-24T07:03:43","date_gmt":"2024-09-24T07:03:43","guid":{"rendered":"https:\/\/www.insentragroup.com\/au\/insights\/uncategorized\/a-guide-to-running-multiple-instances-of-boothd-process-on-a-single-arbitrator-node\/"},"modified":"2024-12-13T02:22:20","modified_gmt":"2024-12-13T02:22:20","slug":"a-guide-to-running-multiple-instances-of-boothd-process-on-a-single-arbitrator-node","status":"publish","type":"post","link":"https:\/\/www.insentragroup.com\/au\/insights\/geek-speak\/modern-workplace\/a-guide-to-running-multiple-instances-of-boothd-process-on-a-single-arbitrator-node\/","title":{"rendered":"A Guide to Running Multiple Instances of Boothd Process on a Single Arbitrator Node"},"content":{"rendered":"\n<p>So, you\u2019ve got booth clusters\u2014a lot of them. And you\u2019ve got one trusty arbitrator node. Now, you&#8217;re wondering, &#8220;Can this one arbitrator manage multiple booth processes, like a bouncer handling entry at multiple nightclub doors?&#8221; The answer: absolutely, but it&#8217;s going to require some fancy footwork.&nbsp;<\/p>\n\n\n\n<p>Imagine your arbitrator node as a skilled bartender in a bustling pub, serving drinks (read: tickets) to multiple patrons (read: clusters) at the same time. Just like a bartender who can mix a cocktail while keeping an eye on the crowd, your arbitrator node can run multiple instances of the boothd process, ensuring that each set of clusters gets their tickets handled with precision and care. The question isn\u2019t whether it can be done\u2014it\u2019s whether your arbitrator is ready for the happy hour rush!&nbsp;<\/p>\n\n\n\n<p>A few things to remember: In a default configuration, the boothd process manages all the clusters as a single cluster that connect to it. In order for an arbitrator node to manage only a subset of all clusters, the arbitrator node should be running multiple instances of booth.&nbsp;<\/p>\n\n\n\n<p>Let\u2019s get into this.&nbsp;<\/p>\n\n\n\n<p>Firstly, what we need to do is to create a booth configuration on the existing clusters. Let\u2019s assume you have two sets of clusters in multi-site topology:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Two, two-node clusters in multi-site topology providing HA\/DR for the application <strong>mq<\/strong>&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Two, two-node clusters in multi-site topology providing HA\/DR for the application <strong>web<\/strong>&nbsp;<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img fetchpriority=\"high\" decoding=\"async\" width=\"642\" height=\"436\" src=\"https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/one-booth-process.jpg\" alt=\"\" class=\"wp-image-22783\" title=\"\" srcset=\"https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/one-booth-process.jpg 642w, https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/one-booth-process-300x204.jpg 300w\" sizes=\"(max-width: 642px) 100vw, 642px\" \/><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\"><strong>Booth Server Configuration<\/strong>&nbsp;<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Install required packages:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# dnf install pcs booth-core booth-arbitrator&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you are running the firewalld, add the service:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# firewall-cmd --add-service=high-availability --permanent\n<br>\n&#91;arbitrator]# firewall-cmd --reload&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Or update the service to only include the booth related ports&nbsp;<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"790\" height=\"240\" src=\"https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/image.png\" alt=\"update the service to only include the booth related ports\" class=\"wp-image-22785\" title=\"\" srcset=\"https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/image.png 790w, https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/image-300x91.png 300w, https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/image-768x233.png 768w\" sizes=\"(max-width: 790px) 100vw, 790px\" \/><\/figure>\n<\/div>\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# firewall-cmd --reload\n<br>\n&#91;arbitrator]# firewall-cmd --add-service=booth --permanent\n<br>\n&#91;arbitrator]# firewall-cmd --reload<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Booth Configuration on Cluster Nodes<\/strong>&nbsp;<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Set up a booth configuration on a single node within the cluster. Ensure that the addresses provided for each cluster and the arbitrator are IP addresses. For each cluster, use a floating IP address. Run the following command on cluster1.node1 of the cluster1 cluster. Executing this command will generate the configuration files \/etc\/booth\/booth.conf and \/etc\/booth\/booth.key on the node where the command is executed.&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;cluster1.node1]# pcs booth setup sites &lt;Floating IP cluster1&gt; &lt;Floating IP of cluster2&gt; arbitrators &lt;Arbitrator IP&gt;&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create a ticket for the Booth configuration on the first node of the first cluster (cluster1.node1). This is the ticket that you will use to define the resource constraint that will allow resources to run only when this ticket has been granted to the cluster. This basic failover configuration procedure uses only one ticket, but you can create additional tickets for more complicated scenarios where each ticket is associated with a different resource or resources.&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;cluster1.node1]# pcs booth ticket add mqticket&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Synchronise the Booth configuration to all nodes in cluster1. Note that this command will copy the booth.conf and booth.key to \/etc\/booth\/ directory on all the nodes of the cluster1.&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;cluster1.node1]# pcs booth sync&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You have two options to copy the configuration files to another cluster (cluster2). Either authenticate the cluster2.node1 to the cluster1.node1 and pull the configuration using the following commands:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;cluster2.node1]# pcs host auth cluster1.node1\n<br>\n&#91;cluster2.node1]# pcs booth pull cluster1.node1&nbsp;\n<br>&nbsp;\n&#91;cluster2.node1]# pcs booth sync&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You can also copy the \/etc\/booth\/booth.conf and \/etc\/booth\/booth.key to cluster2.node1 (ensure to keep the same permissions) and the following command to copy the config to all the nodes in cluster2.&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;cluster2.node1]# pcs booth sync&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Continue with the boot configuration, configuring the booth as a cluster resource. This process requires the Floating IP assigned to each cluster. The following commands will create a resource group with booth-ip and booth-service as member of the group:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;cluster1.node1] # pcs booth create ip &lt;Floating IP cluster1&gt;&nbsp;\n<br>\n&#91;cluster2.node1] # pcs booth create ip &lt;Floating IP cluster2&gt;&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Add a ticket constraint to the resource groups defined for each cluster&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;cluster1.node1] # pcs constraint ticket add mqticket mq_app_rg&nbsp;\n<br>\n&#91;cluster2.node1] # pcs constraint ticket add mqticket mq_app_rg&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Repeat all the above steps for cluster3 and cluster4, remembering that we are creating webticket and adding the ticket to the web_app_rg.&nbsp;<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Update the configuration on the booth server<\/strong>&nbsp;<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Copy the \/etc\/booth\/booth.conf and \/etc\/booth\/booth.key from cluster1 and cluster 3 to the arbitrator server to \/etc\/booth&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Save the booth.conf from the mq cluster as mqcluster.conf&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Save the booth.key from the mq cluster as mqbooth.key&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Save the booth.conf from the web cluster as webcluster.conf&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Save the booth.key from the web cluster as webbooth.key&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Update the configuration for the mqcluster.conf as follows:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>port = 9929 # Default booth port&nbsp;\n<br>\ntransport = UDP # You can use either UDP or TCP ports&nbsp;\n<br>\nsite = 192.168.201.100&nbsp;\n<br>\nsite = 192.168.202.100&nbsp;\n<br>\narbitrator = 192.168.50.151&nbsp;\n<br>\nauthfile = \/etc\/booth\/mqbooth.key&nbsp;\n<br>\nticket = \"mqticket\"&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Update the configuration for the webcluster.conf as follows:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>port = 9939&nbsp;\n<br>\ntransport = UDP&nbsp;\n<br>\nsite = 192.168.203.100&nbsp;\n<br>\nsite = 192.168.204.100&nbsp;\n<br>\narbitrator = 192.168.50.151&nbsp;\n<br>\nauthfile = \/etc\/booth\/webbooth.key&nbsp;\n<br>\nticket = \"webticket\"&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you are running SELinux on the arbitrator (as you should), you need to add a new port to the configuration:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# semanage port -a -t boothd_port_t -p tcp 9939&nbsp;\n<br>\n&#91;arbitrator]# semanage port -a -t boothd_port_t -p udp 9939&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Verify the SELinux configuration:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# semanage port -l | grep boothd \n<br>\nboothd_port_t&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tcp&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 9939, 9929\n<br>\n&nbsp;\nboothd_port_t&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; udp&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 9939, 9929&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Start the boothd services, where the part following \u2018@\u2019 should be the name of the configuration file without \u2018.conf\u2019&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# systemctl enable \u2013-now booth@mqcluster.service&nbsp;\n<br>\n&#91;arbitrator]# systemctl enable \u2013-now booth@webcluster.service&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Verify if the booth service is running:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# systemctl status booth@mqcluster.service&nbsp;\n<br>\n&#91;arbitrator]# systemctl status booth@webcluster.service&nbsp;<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>List the tickets for the given cluster:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;arbitrator]# booth list -c \/etc\/booth\/mqcluster.conf&nbsp;\n<br>\nticket: mqticket, leader: NONE&nbsp;\n<br>\n&#91;arbitrator]# booth list -c \/etc\/booth\/webcluster.conf&nbsp;\n<br>\nticket: webticket, leader: NONE&nbsp;<\/code><\/pre>\n\n\n\n<p>The following diagram is our final configuration:&nbsp;<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"642\" height=\"436\" src=\"https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/multiple-booth-process.jpg\" alt=\"\" class=\"wp-image-22787\" title=\"\" srcset=\"https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/multiple-booth-process.jpg 642w, https:\/\/www.insentragroup.com\/au\/wp-content\/uploads\/sites\/22\/2024\/09\/multiple-booth-process-300x204.jpg 300w\" sizes=\"(max-width: 642px) 100vw, 642px\" \/><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\"><strong>Final Notes<\/strong>&nbsp;<\/h2>\n\n\n\n<p>The above Booth configuration sets up automatic site failover, where the Booth arbitrator monitors the clusters. If communication with one of the clusters is lost, the arbitrator allows another cluster to take over the ticket and enable resources.&nbsp;<\/p>\n\n\n\n<p>For manual failover, the Booth arbitrator is not required, as the clusters continuously exchange information with each other. To configure manual failover, modify the booth.conf configuration file on each cluster as follows:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>port = 9929&nbsp;\n<br>\ntransport = UDP&nbsp;\n<br>\nsite = 192.168.201.100&nbsp;\n<br>\nsite = 192.168.202.100\n<br>\narbitrator = 192.168.50.151\n<br>\nauthfile = \/etc\/booth\/mqbooth.key&nbsp;&nbsp;\n<br>&nbsp;\nticket = \"mqticket\"&nbsp;\n<br>\nmode = MANUAL&nbsp;<\/code><\/pre>\n\n\n\n<p>Refer to the following links for more information: &nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/documentation.suse.com\/sle-ha\/15-SP6\/html\/SLE-HA-all\/cha-ha-geo-booth.html\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">https:\/\/documentation.suse.com\/sle-ha\/15-SP6\/html\/SLE-HA-all\/cha-ha-geo-booth.html<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/github.com\/ClusterLabs\/booth\/blob\/main\/docs\/boothd.8.txt\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">https:\/\/github.com\/ClusterLabs\/booth\/blob\/main\/docs\/boothd.8.txt<\/a>&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>If you have questions or need assistance with your booth configuration, feel free to browse our <a href=\"https:\/\/www.insentragroup.com\/au\/insights\/\" target=\"_blank\" rel=\"noreferrer noopener\">Insentra Insights<\/a> or <a href=\"https:\/\/www.insentragroup.com\/au\/contact\/\" target=\"_blank\" rel=\"noreferrer noopener\">reach out to us<\/a> for a consultation.\u00a0<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to configure your arbitrator to manage multiple booth processes across clusters. Master the setup with our step-by-step guide. Read more now! <\/p>\n","protected":false},"author":67,"featured_media":22789,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[19],"tags":[],"class_list":["post-22782","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-modern-workplace","entry"],"_links":{"self":[{"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/posts\/22782","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/comments?post=22782"}],"version-history":[{"count":4,"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/posts\/22782\/revisions"}],"predecessor-version":[{"id":22791,"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/posts\/22782\/revisions\/22791"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/media\/22789"}],"wp:attachment":[{"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/media?parent=22782"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/categories?post=22782"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.insentragroup.com\/au\/wp-json\/wp\/v2\/tags?post=22782"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}