{"id":688,"date":"2018-11-20T01:00:00","date_gmt":"2018-11-20T01:00:00","guid":{"rendered":"http:\/\/inswwdev.azurewebsites.net\/au\/insights\/uncategorized\/protecting-the-automation-engine-backup-for-ansible-awx-project\/"},"modified":"2024-10-10T07:41:12","modified_gmt":"2024-10-10T07:41:12","slug":"protecting-the-automation-engine-backup-for-ansible-awx-project","status":"publish","type":"post","link":"https:\/\/www.insentragroup.com\/us\/insights\/geek-speak\/modern-workplace\/protecting-the-automation-engine-backup-for-ansible-awx-project\/","title":{"rendered":"Protecting the automation engine \u2013 Backup for Ansible AWX Project"},"content":{"rendered":"<p style=\"text-align: justify;\">As for any system, AWX (upstream for Ansible Tower) must be protected by means of a consistent and reliable backup. I should not have to stress the importance of this action.<\/p>\n<p style=\"text-align: justify;\">So lets get to the how:<\/p>\n<p style=\"text-align: justify;\">Let\u2019s backup AWX in preparation for migration or upgrade (re-running an ansible-playbook for AWX).<\/p>\n<p style=\"text-align: justify;\">There are two methods which could be used to achieve this and both are\u00a0 easy and work every time (most of the time) ?<\/p>\n<p style=\"text-align: justify;\">The methods:<\/p>\n<h3 style=\"padding-bottom: 15px; margin-bottom: 30px; margin-top: 40px; border-bottom: 1px solid #f16020;\"><span>1. First method:<\/span><\/h3>\n<p style=\"text-align: justify;\"><strong>a.<\/strong><span>\u00a0<\/span>on the server running AWX, install tower-cli<\/p>\n<p style=\"text-align: justify;\">pip install ansible-tower-cli<\/p>\n<p style=\"text-align: justify;\"><strong>b.\u00a0<\/strong>Given ansible-tower-cli is using tower\u2019s API, first we need to configure the tool by providing the URL, username and password for the AWX (tower). To find the URL, simply run the following command on the host running AWX:<\/p>\n<p style=\"text-align: justify;\">docker ps -a | grep awx_web<\/p>\n<p style=\"text-align: justify;\">74cc80a22e7b\u00a0 ansible\/awx_web:latest \u201c\/tini \u2014 \/bin\/sh \u2026\u201d\u00a0 18 minutes ago\u00a0 Up 18 minutes 0.0.0.0:8080-&gt;8052\/tcp awx_web<\/p>\n<p style=\"text-align: justify;\"><strong>c.\u00a0<\/strong>Note the following: \u201c0.0.0.0:8080\u201d \u2013 this excerpt shows the mapping of the external port (8080) to the internal port used in the container (8052). This implies that the URL is: http:\/\/localhost:8080 or http:\/\/server_name_running_awx:8080<\/p>\n<p style=\"text-align: justify;\"><strong>d.\u00a0<\/strong>Use the following commands to configure tower-cli. The commands provided below are used to allow tower-cli to communicate via API with the AWX. Note that AWX does not support TLS at this moment (if you are interested how to configure TLS for AWX without proxies and forwarders, check my blog on this \u2013 https:\/\/inswwdev.azurewebsites.net\/au\/making-awx-ssl-compliant\/).<\/p>\n<p style=\"text-align: justify;\">tower-cli config host http:\/\/127.0.0.1:8080<\/p>\n<p style=\"text-align: justify;\">tower-cli config username admin<\/p>\n<p style=\"text-align: justify;\">tower-cli config password<\/p>\n<p style=\"text-align: justify;\">tower-cli config verify_ssl \u2018false\u2019<\/p>\n<p style=\"text-align: justify;\"><strong>e.\u00a0<\/strong>It is just enough to \u2018export\u2019 the entire configuration to the json file. The resulting file will have the entire configuration (except the passwords which might need to be re-entered).<\/p>\n<p style=\"text-align: justify;\">tower-cli receive \u2013all &gt; backup.json<\/p>\n<p style=\"text-align: justify;\"><strong>f.<span>\u00a0<\/span><\/strong>Performing a restore of the configuration from the backup file is also very easy:<\/p>\n<p style=\"text-align: justify;\">tower-cli send backup.json<\/p>\n<p style=\"text-align: justify;\"><strong>g.<span>\u00a0<\/span><\/strong>If you would like to find out more about the tower-cli \u2013 check the following URL:<\/p>\n<p style=\"text-align: justify;\">http:\/\/tower-cli.readthedocs.io\/en\/latest\/<\/p>\n<h3 style=\"padding-bottom: 15px; margin-bottom: 30px; margin-top: 40px; border-bottom: 1px solid #f16020;\"><span>2. Second method:<\/span><\/h3>\n<p style=\"text-align: justify;\"><strong>a.\u00a0<\/strong>Before the installation of AWX, ensure the inventory file has been configured properly and the postgresql datafiles and configuration is stored on the persistent file system. For example:<\/p>\n<p style=\"text-align: justify;\"># Common Docker parameters<\/p>\n<p style=\"text-align: justify;\">postgres_data_dir=\/var\/lib\/awx\/pgdocker<\/p>\n<p style=\"text-align: justify;\"><strong>b.<span>\u00a0<\/span><\/strong>Once AWX is installed and configured (including credentials, projects, templates etc), ensure no jobs are running and stop the postgres container:<\/p>\n<p style=\"text-align: justify;\">docker ps -a | grep postgres<\/p>\n<p style=\"text-align: justify;\">7154cfa15e3f\u00a0 postgres:9.6 \u201cdocker-entrypoint\u2026\u201d 33 minutes ago\u00a0\u00a0 Up 21 minutes 5432\/tcp postgres<\/p>\n<p>\u00a0<\/p>\n<p style=\"text-align: justify;\">docker stop 7154cfa15e3f<\/p>\n<p style=\"text-align: justify;\"><strong>c.<span>\u00a0<\/span><\/strong>If your postgres_data_dir is located in \/var\/lib\/awx\/pgdocker, change directory to \/var\/lib\/awx\/pgdocker and take a backup of the pgdata directory using your preferred method, for example tar:<\/p>\n<p style=\"text-align: justify;\">tar cvpf pgdata.tar pgdata<\/p>\n<p style=\"text-align: justify;\"><strong>d.<span>\u00a0<\/span><\/strong>Start the postgres container:<\/p>\n<p style=\"text-align: justify;\">docker start 7154cfa15e3f<\/p>\n<p style=\"text-align: justify;\"><strong>e.<span>\u00a0<\/span><\/strong>Verify if the database is working and accepting connections:<\/p>\n<p style=\"text-align: justify;\">docker logs -f postgres<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 received smart shutdown request<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 autovacuum launcher shutting down<\/p>\n<p style=\"text-align: justify;\">FATAL:\u00a0 the database system is shutting down<\/p>\n<p style=\"text-align: justify;\">FATAL:\u00a0 the database system is shutting down<\/p>\n<p style=\"text-align: justify;\">FATAL:\u00a0 the database system is shutting down<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 shutting down<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 database system is shut down<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 database system was shut down at 2018-08-15 03:27:16 UTC<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 MultiXact member wraparound protections are now enabled<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 database system is ready to accept connections<\/p>\n<p style=\"text-align: justify;\">LOG:\u00a0 autovacuum launcher started<\/p>\n<p style=\"text-align: justify;\"><strong>f.<span>\u00a0<\/span><\/strong>Log in to awx UI to verify if everything is working as expected.<\/p>\n<p style=\"text-align: justify;\"><strong>g.<span>\u00a0<\/span><\/strong>To restore from the tar backup, follow the steps to stop the postgres container, copy the pgdata directory (or move) as a precaution and untar the file. Once it is done \u2013 start the container and enjoy the configuration.<\/p>\n<h3 style=\"padding-bottom: 15px; margin-bottom: 30px; margin-top: 40px; border-bottom: 1px solid #f16020;\"><span>Recommendations:<\/span><\/h3>\n<p style=\"text-align: justify;\">I recommend using both methods in case the db is corrupted and you cannot recover from the backup. Alternatively use the ansible playbook to automate the process, for example:<\/p>\n<p style=\"text-align: justify;\">\u2014<\/p>\n<p style=\"text-align: justify;\">\u2013 name: Backup awx<\/p>\n<p style=\"text-align: justify;\">hosts: all<\/p>\n<p style=\"text-align: justify;\">tasks:<\/p>\n<p style=\"text-align: justify;\">\u2013 name: Stop the postgres container on awx server<\/p>\n<p style=\"text-align: justify;\">docker_container:<\/p>\n<p style=\"text-align: justify;\">name: postgres<\/p>\n<p style=\"text-align: justify;\">state: stopped<\/p>\n<p style=\"text-align: justify;\">\u2013 name: Take the backup of the database<\/p>\n<p style=\"text-align: justify;\">archive:<\/p>\n<p style=\"text-align: justify;\">path: \/var\/lib\/awx\/pgdocker\/pgdata<\/p>\n<p style=\"text-align: justify;\">dest: \/var\/lib\/awx\/pgdocker\/pgdata-{{ ansible_date_time.iso8601_basic_short }}.gz<\/p>\n<p style=\"text-align: justify;\">\u2013 name: Start the postgres container on awx server<\/p>\n<p style=\"text-align: justify;\">docker_container:<\/p>\n<p style=\"text-align: justify;\">name: postgres<\/p>\n<p style=\"text-align: justify;\">state: started<\/p>\n<p style=\"text-align: justify;\">\u2013 name: Take the tower-cli backup<\/p>\n<p style=\"text-align: justify;\">shell: \u2018tower-cli receive \u2013all &gt; \/var\/lib\/awx\/pgdocker\/backup-{{ ansible_date_time.iso8601_basic_short }}.json\u2019<\/p>\n<p style=\"text-align: justify;\">register: backup<\/p>\n<p style=\"text-align: justify;\">failed_when: backup.rc == \u201c1\u201d<\/p>\n<p style=\"text-align: justify;\"><strong>Conclusion :\u00a0<span>\u00a0<\/span><\/strong>So\u2026 now you know how to protect and restore AWX, you should get a plan in place as soon as possible to ensure continuous protection. For more information reach out to info@insentragroup.com<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As for any system, AWX (upstream for Ansible Tower) must be protected by means of a consistent and reliable backup. I should not have to stress the importance of this action. So lets get to the how: Let\u2019s backup AWX in preparation for migration or upgrade (re-running an ansible-playbook for AWX). There are two methods&hellip; <a class=\"more-link\" href=\"https:\/\/www.insentragroup.com\/us\/insights\/geek-speak\/modern-workplace\/protecting-the-automation-engine-backup-for-ansible-awx-project\/\">Continue reading <span class=\"screen-reader-text\">Protecting the automation engine \u2013 Backup for Ansible AWX Project<\/span><\/a><\/p>\n","protected":false},"author":65,"featured_media":689,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[19],"tags":[],"class_list":["post-688","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-modern-workplace","entry"],"_links":{"self":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts\/688","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/users\/65"}],"replies":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/comments?post=688"}],"version-history":[{"count":1,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts\/688\/revisions"}],"predecessor-version":[{"id":22111,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts\/688\/revisions\/22111"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/media\/689"}],"wp:attachment":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/media?parent=688"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/categories?post=688"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/tags?post=688"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}