Protecting the automation engine – Backup for Ansible AWX Project

As for any system, AWX (upstream for Ansible Tower) must be protected by means of a consistent and reliable backup. I should not have to stress the importance of this action.

So lets get to the how:

Let’s backup AWX in preparation for migration or upgrade (re-running an ansible-playbook for AWX).

There are two methods which could be used to achieve this and both are  easy and work every time (most of the time) 🙂

The methods:

1. First method:

a. on the server running AWX, install tower-cli

pip install ansible-tower-cli

b. Given ansible-tower-cli is using tower’s API, first we need to configure the tool by providing the URL, username and password for the AWX (tower). To find the URL, simply run the following command on the host running AWX:

docker ps -a | grep awx_web

74cc80a22e7b  ansible/awx_web:latest “/tini — /bin/sh …”  18 minutes ago  Up 18 minutes 0.0.0.0:8080->8052/tcp awx_web

c. Note the following: “0.0.0.0:8080” – this excerpt shows the mapping of the external port (8080) to the internal port used in the container (8052). This implies that the URL is: http://localhost:8080 or http://server_name_running_awx:8080

d. Use the following commands to configure tower-cli. The commands provided below are used to allow tower-cli to communicate via API with the AWX. Note that AWX does not support TLS at this moment (if you are interested how to configure TLS for AWX without proxies and forwarders, check my blog on this – https://www.insentra.com.au/making-awx-ssl-compliant/).

tower-cli config host http://127.0.0.1:8080

tower-cli config username admin

tower-cli config password

tower-cli config verify_ssl ‘false’

e. It is just enough to ‘export’ the entire configuration to the json file. The resulting file will have the entire configuration (except the passwords which might need to be re-entered).

tower-cli receive –all > backup.json

f. Performing a restore of the configuration from the backup file is also very easy:

tower-cli send backup.json

g. If you would like to find out more about the tower-cli – check the following URL:

http://tower-cli.readthedocs.io/en/latest/

2. Second method:

a. Before the installation of AWX, ensure the inventory file has been configured properly and the postgresql datafiles and configuration is stored on the persistent file system. For example:

# Common Docker parameters

postgres_data_dir=/var/lib/awx/pgdocker

b. Once AWX is installed and configured (including credentials, projects, templates etc), ensure no jobs are running and stop the postgres container:

docker ps -a | grep postgres

7154cfa15e3f  postgres:9.6 “docker-entrypoint…” 33 minutes ago   Up 21 minutes 5432/tcp postgres

 

docker stop 7154cfa15e3f

c. If your postgres_data_dir is located in /var/lib/awx/pgdocker, change directory to /var/lib/awx/pgdocker and take a backup of the pgdata directory using your preferred method, for example tar:

tar cvpf pgdata.tar pgdata

d. Start the postgres container:

docker start 7154cfa15e3f

e. Verify if the database is working and accepting connections:

docker logs -f postgres

LOG:  received smart shutdown request

LOG:  autovacuum launcher shutting down

FATAL:  the database system is shutting down

FATAL:  the database system is shutting down

FATAL:  the database system is shutting down

LOG:  shutting down

LOG:  database system is shut down

LOG:  database system was shut down at 2018-08-15 03:27:16 UTC

LOG:  MultiXact member wraparound protections are now enabled

LOG:  database system is ready to accept connections

LOG:  autovacuum launcher started

f. Log in to awx UI to verify if everything is working as expected.

g. To restore from the tar backup, follow the steps to stop the postgres container, copy the pgdata directory (or move) as a precaution and untar the file. Once it is done – start the container and enjoy the configuration.

Recommendations:

I recommend using both methods in case the db is corrupted and you cannot recover from the backup. Alternatively use the ansible playbook to automate the process, for example:

– name: Backup awx

hosts: all

tasks:

– name: Stop the postgres container on awx server

docker_container:

name: postgres

state: stopped

– name: Take the backup of the database

archive:

path: /var/lib/awx/pgdocker/pgdata

dest: /var/lib/awx/pgdocker/pgdata-{{ ansible_date_time.iso8601_basic_short }}.gz

– name: Start the postgres container on awx server

docker_container:

name: postgres

state: started

– name: Take the tower-cli backup

shell: ‘tower-cli receive –all > /var/lib/awx/pgdocker/backup-{{ ansible_date_time.iso8601_basic_short }}.json’

register: backup

failed_when: backup.rc == “1”

Conclusion :  So… now you know how to protect and restore AWX, you should get a plan in place as soon as possible to ensure continuous protection. For more information reach out to info@insentragroup.com

Join the Insentra Community with the Insentragram Newsletter

Hungry for more?

[Modern Workplace]

Project Management and Change Management – How Insentra ensures projects run smoothly

By [Marni Noble]

I am going to say something that will really blow your mind… are you ready? It seems in business today that change is the only constant in this crazy fast-paced world of variables.

[Modern Workplace]

Farewell Smart Scale, Hello Autoscale…

In some cases, ‘farewells’ can carry a little sadness and it’s no different in the land of technology when a product or service that brought value, gets sent to the chopping block via a decision made by the vendor for the greater good (in most cases 

[Modern Workplace]

Earned Value Analysis in Project Cost Management

By [Michael Chen]

Most have heard the terms “on budget”, “over budget” or “under budget” used in project status reporting and in particular for fixed-price projects which have an set budget to deliver an agreed project scope.