{"id":10787,"date":"2022-08-05T07:46:40","date_gmt":"2022-08-05T07:46:40","guid":{"rendered":"https:\/\/www.insentragroup.com\/us\/?p=10787"},"modified":"2024-12-13T02:01:29","modified_gmt":"2024-12-13T02:01:29","slug":"ansible-disaster-recovery-guide-aws","status":"publish","type":"post","link":"https:\/\/www.insentragroup.com\/us\/insights\/geek-speak\/modern-workplace\/ansible-disaster-recovery-guide-aws\/","title":{"rendered":"Ansible Disaster Recovery Guide AWS"},"content":{"rendered":"\n

Step-by-Step Guide to Disaster Recovery for the Ansible Automation Platform installed in AWS<\/strong><\/h2>\n\n\n\n

Not to sound negative, but organisations should always try to prepare for the worst and hope for the best.<\/p>\n\n\n\n

Disaster Recovery (DR) is critical for every organisation. Ensuring business remains uninterrupted is key, whether you need to prepare for unforeseen incidents like a data centre outage or reside somewhere susceptible to natural disasters. But how can you guarantee the changes don\u2019t impact the end user?<\/p>\n\n\n\n

There are several ways to provide Disaster Recovery for the Red Hat Ansible Automation Platform installed in AWS (you might have seen my \u2018How-to guide on Ansible Tower Backup and Restore on Azure<\/a>\u2019). AAP provides a built-in backup method which can be executed by using the same installation script with \u2018-b\u2019 switch: setup.sh -b. This approach backs up the entire AAP configuration including; the Postgres DB, all the controller, execution, and hub nodes. As a result, we have a backup which can be used to recover the entire environment. <\/p>\n\n\n\n

Often this approach might be too difficult to implement in a cloud environment as it requires a prolonged outage for the entire AAP environment and preferably the recovery is done in a freshly provisioned environment.  This step-by-step guide describes the steps required to restore Ansible’s Automation Platform DB in AWS when RDS is used without the need for the AAP backup file.<\/p>\n\n\n\n

PREREQUISITES:<\/strong><\/h3>\n\n\n\n
  1. AAP installed with AWS RDS Database<\/li>
  2. RDS DB is being protected with AWS Snapshots<\/li>
  3. RDS DB snapshot is available<\/li>
  4. Access to relevant sections of AWS console<\/li>
  5. Access to controller and hub nodes<\/li><\/ol>\n\n\n\n

    HOW TO RESTORE DB FOR AAP:<\/strong><\/h3>\n\n\n\n
    1. Log into AWS<\/li>
    2. Navigate to RDS > Snapshots<\/li>
    3. Select the Snapshot to restore:<\/li><\/ol>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      1. Select Actions –> Restore Snapshot:<\/li><\/ol>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        1. In the configuration of Restore snapshot, specify a new DB instance identifier, for example —> aapdb02<\/li><\/ol>\n\n\n\n
          \"\"<\/figure>\n\n\n\n
          1. Ensure all other settings are the same as for the original instance (especially VPC security groups)<\/li>
          2. Click Restore DB Instance and wait patiently until it is restored and online<\/li>
          3. Log into the controller node, preferably as a root<\/li>
          4. Verify connection to a new DB using psql or podman container. The syntax is like the below, with the exception of the username (-U) and the database we are connecting to (the name can be found in the inventory file)<\/li><\/ol>\n\n\n\n
            [root@aapcontroller01 ~]# psql -h aapdb02.cbr8auhjg1vp.us-west-1.rds.amazonaws.com -U awx -d aapdb\nPassword for user awx: \npsql (13.7, server 12.10)\nSSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)\nType \"help\" for help.\n \naapdb=> \\l\n                                  List of databases\n   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   \n-----------+----------+----------+-------------+-------------+-----------------------\n aapdb     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc\/postgres         +\n           |          |          |             |             | postgres=CTc\/postgres+\n           |          |          |             |             | awx=CTc\/postgres\n aaphubdb  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc\/postgres         +\n           |          |          |             |             | postgres=CTc\/postgres+\n           |          |          |             |             | awx=CTc\/postgres\n postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | \n rdsadmin  | rdsadmin | UTF8     | en_US.UTF-8 | en_US.UTF-8 | rdsadmin=CTc\/rdsadmin\n template0 | rdsadmin | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c\/rdsadmin          +\n           |          |          |             |             | rdsadmin=CTc\/rdsadmin\n template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c\/postgres          +\n           |          |          |             |             | postgres=CTc\/postgres\n(6 rows)\naapdb=> \n<\/code><\/pre>\n\n\n\n
            1. Once connectivity has been confirmed, stop the services on all servers (controllers and hubs):<\/li><\/ol>\n\n\n\n
              Run the following command on controllers:<\/p>\n\n\n\n
              automation-controller-service stop<\/code><\/pre>\n\n\n\n
              Run the following command on the hubs:<\/p>\n\n\n\n
              systemctl stop pulp* nginx redis<\/code><\/pre>\n\n\n\n
              1. On controller nodes, change directory to: \/etc\/tower\/conf.d\/<\/li><\/ol>\n\n\n\n
                cd \/etc\/tower\/conf.d<\/code><\/pre>\n\n\n\n
                1. Edit the postgres.py file and update the DB Host name on all controller nodes<\/li><\/ol>\n\n\n\n
                  # Ansible Automation Platform controller database settings.\n \nDATABASES = {\n   'default': {\n       'ATOMIC_REQUESTS': True,\n       'ENGINE': 'awx.main.db.profiled_pg',\n       'NAME': 'aapdb',\n       'USER': 'awx',\n       'PASSWORD': \"\"\"Password\"\"\",\n       'HOST': 'aapdb02.cbr8auhjg1vp.us-west-1.rds.amazonaws.com',\n       'PORT': '5432',\n       'OPTIONS': { 'sslmode': 'prefer',\n                    'sslrootcert': '\/etc\/pki\/tls\/certs\/ca-bundle.crt',\n       },\n   }\n}\n \n<\/code><\/pre>\n\n\n\n
                  1. On the hub nodes, change directory to \/etc\/pulp:<\/li><\/ol>\n\n\n\n
                    cd \/etc\/pulp<\/code><\/pre>\n\n\n\n
                    1. Edit the settings.py file and update the database name in DATABASES section:<\/li><\/ol>\n\n\n\n
                      DATABASES = {'default': {'HOST': 'aapdb02.cbr8auhjg1vp.us-west-1.rds.amazonaws.com', 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'aaphubdb', 'USER': 'awx', 'PASSWORD': 'reducted', 'PORT': 5432, 'OPTIONS': {'sslmode': 'prefer', 'sslrootcert': '\/etc\/pki\/tls\/certs\/ca-bundle.crt'}}}\nREDIS_HOST = 'localhost'\nREDIS_PORT = 6379\nCACHE_ENABLED = True\nGALAXY_COLLECTION_SIGNING_SERVICE = 'ansible-default'\nPRIVATE_KEY_PATH = '\/etc\/pulp\/certs\/token_private_key.pem'\nPUBLIC_KEY_PATH = '\/etc\/pulp\/certs\/token_public_key.pem'\nTOKEN_SERVER = 'https:\/\/aaphub01.example.net\/token'\nTOKEN_SIGNATURE_ALGORITHM = 'ES256'\nALLOWED_CONTENT_CHECKSUMS = ['sha224', 'sha256', 'sha384', 'sha512']\nSECRET_KEY = 'reducted'\nCONTENT_ORIGIN = 'https:\/\/aaphub01.example.net'\nX_PULP_API_PROTO = 'https'\nX_PULP_API_HOST = 'aaphub01.example.net'\nX_PULP_API_PORT = '443'\nX_PULP_API_PREFIX = 'pulp_ansible\/galaxy\/automation-hub\/api'\nGALAXY_API_DEFAULT_DISTRIBUTION_BASE_PATH = 'published'\nGALAXY_ENABLE_API_ACCESS_LOG = False\nGALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_ACCESS = False\nGALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_DOWNLOAD = False\nGALAXY_REQUIRE_CONTENT_APPROVAL = True\nGALAXY_AUTO_SIGN_COLLECTIONS = False\nREDIS_URL = 'unix:\/\/\/var\/run\/redis\/redis.sock'\nANSIBLE_API_HOSTNAME = 'https:\/\/aaphub01.example.net'\nANSIBLE_CONTENT_HOSTNAME = 'https:\/\/aaphub01.example.net'\nCONTENT_BIND = 'unix:\/var\/run\/pulpcore-content\/pulpcore-content.sock'\nCONNECTED_ANSIBLE_CONTROLLERS = ['https:\/\/aapcontroller01.example.net', 'https:\/\/aapcontroller02.example.net']\nDEPLOY_ROOT = \"\/var\/lib\/pulp\"\nMEDIA_ROOT = \"\/var\/lib\/pulp\/media\"\nSTATIC_ROOT = \"\/var\/lib\/pulp\/assets\"\nWORKING_DIRECTORY = \"\/var\/lib\/pulp\/tmp\"\nFILE_UPLOAD_TEMP_DIR = \"\/var\/lib\/pulp\/tmp\"\nDB_ENCRYPTION_KEY = \"\/etc\/pulp\/certs\/database_fields.symmetric.key\"\n<\/code><\/pre>\n\n\n\n
                      1. Reboot the hub nodes<\/li>
                      2. Start the service on all controller nodes:<\/li><\/ol>\n\n\n\n
                        automation-controller-service start<\/code><\/pre>\n\n\n\n
                        1. On the controller node run the following command as root. The command will verify the connection to the DB and connection between the controller nodes. All nodes should be visible and have the recent heartbeat timestamp:<\/li><\/ol>\n\n\n\n
                          [root@aapcontroller01 ~]# awx-manage list_instances\n[controlplane capacity=270 policy=100%]\naapcontroller01.example.net capacity=135 node_type=hybrid version=4.2.0 heartbeat=\"2022-06-23 05:44:45\"\naapcontroller02.example.net capacity=135 node_type=hybrid version=4.2.0 heartbeat=\"2022-06-23 05:44:44\"\n \n[default capacity=270 policy=100%]\naapcontroller01.example.net capacity=135 node_type=hybrid version=4.2.0 heartbeat=\"2022-06-23 05:44:45\"\naapcontroller02.example.net capacity=135 node_type=hybrid version=4.2.0 heartbeat=\"2022-06-23 05:44:44\"\n<\/code><\/pre>\n\n\n\n
                          1. Connect to each controller node using web browser and verify the communication and configuration<\/li>
                          2. Connect to each hub node using web browser and verify the communication and configuration<\/li><\/ol>\n\n\n\n
                            Don\u2019t let an outage catch you off guard! Reach out to Insentra<\/a> if you would like to explore a joint Disaster Recovery solution tailored to your needs.<\/p>\n\n\n\n