{"id":19336,"date":"2023-10-05T03:14:04","date_gmt":"2023-10-05T03:14:04","guid":{"rendered":"https:\/\/www.insentragroup.com\/us\/insights\/uncategorized\/ansible-automation-platform-troubleshooting\/"},"modified":"2024-12-13T02:00:09","modified_gmt":"2024-12-13T02:00:09","slug":"ansible-automation-platform-troubleshooting","status":"publish","type":"post","link":"https:\/\/www.insentragroup.com\/us\/insights\/geek-speak\/modern-workplace\/ansible-automation-platform-troubleshooting\/","title":{"rendered":"Ansible Automation Platform Troubleshooting\u00a0"},"content":{"rendered":"\n<p>To troubleshoot communication between the controller nodes and execution nodes in Ansible Automation Platform, you can follow these steps:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\" start=\"1\">\n<li>Check the network connectivity and firewall settings between the nodes. You can use tools like ping, traceroute, telnet, nc, etc. to test the network reachability and latency. You can also use the ansible -m ping command to test the Ansible connectivity between the nodes.<\/li>\n\n\n\n<li>Check the receptor configuration and status on each node. You can use the receptorctl command to view and manage the receptor mesh network. You can also use the receptorctl status command to see the list of nodes, connections, and work types in the mesh<\/li>\n\n\n\n<li>Check the logs and metrics of the receptor service on each node. You can use tools like journalctl, tail, grep, etc. to view and filter the logs.<\/li>\n\n\n\n<li>Check the Ansible Automation Platform web UI and API for any errors or warnings related to the node registration, grouping, or health.&nbsp;&nbsp;<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Verify the Receptor&nbsp;<\/h3>\n\n\n\n<p>Receptor is a networking layer that provides a mechanism for the Ansible Platform to communicate with execution nodes (formerly known as managed nodes). When working with the newer versions of Ansible Automation Platform, Receptor serves as the underlying communication backbone.&nbsp;<\/p>\n\n\n\n<p>Here are some steps and commands you can use to troubleshoot Receptor communication between an Ansible Controller and execution nodes:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Check Receptor Status. On the Ansible Controller, you can check the status of the Receptor using:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>receptorctl --socket \/var\/run\/awx-receptor\/receptor.sock status <\/code><\/pre>\n\n\n\n<p>Example output:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Node ID: aap01.example.net \n<br>\nVersion: 1.4.1 \n<br>\nSystem CPU Count: 2 \n<br>\nSystem Memory MiB: 7761 \n<br>\n <br>\n<br>\nConnection                          Cost \n<br>\naap02.example.net 1 \n<br>\naap03.example.net 1 \n<br>\naap01gcp.example.net        1 \n<br>\n <br>\n<br>\nKnown Node                          Known Connections \n<br>\naap01.example.net aap02.example.net: 1 aap03.example.net: 1 aap01gcp.example.net: 1 \n<br>\naap02.example.net aap01.example.net: 1 aap03.example.net: 1 aap01gcp.example.net: 1 \n<br>\naap03.example.net aap01.example.net: 1 aap02.example.net: 1 aap01gcp.example.net: 1 \n<br>\naap01gcp.example.net        aap01.example.net: 1 aap02.example.net: 1 aap03.example.net: 1 \n<br>\n <br>\n<br>\nRoute                               Via \n<br>\naap02.example.net aap02.example.net \n<br>\naap03.example.net aap03.example.net \n<br>\naap01gcp.example.net        aap01gcp.example.net \n<br>\n <br>\n<br>\nNode                                Service   Type       Last Seen             Tags \n<br>\naap01.example.net control   StreamTLS  2023-09-13 14:26:18   {'type': 'Control Service'} \n<br>\naap01gcp.example.net        control   StreamTLS  2023-09-13 14:25:54   {'type': 'Control Service'} \n<br>\naap03.example.net control   StreamTLS  2023-09-13 14:25:24   {'type': 'Control Service'} \n<br>\naap02.example.net control   StreamTLS  2023-09-13 14:25:54   {'type': 'Control Service'} \n<br>\n <br>\n<br>\nNode                                Secure Work Types \n<br>\naap01.example.net local, kubernetes-runtime-auth, kubernetes-incluster-auth \n<br>\naap01gcp.example.net        ansible-runner \n<br>\naap03.example.net local, kubernetes-runtime-auth, kubernetes-incluster-auth \n<br>\naap02.example.net local, kubernetes-runtime-auth, kubernetes-incluster-auth <\/code><\/pre>\n\n\n\n<p>Review Receptor Logs. Logs can be found in the system&#8217;s journal. You can view them using:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>journalctl -u receptor <\/code><\/pre>\n\n\n\n<p>Verify mesh level communication. From the Ansible Controller, you can try pinging the execution node using the receptor command to see if it&#8217;s reachable:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@aap01 receptor]# receptorctl --socket \/var\/run\/awx-receptor\/receptor.sock ping aap02.example.net \n<br>\nReply from aap02.example.net in 811.135\u00b5s \n<br>\nReply from aap02.example.net in 814.871\u00b5s \n<br>\nReply from aap02.example.net in 852.096\u00b5s \n<br>\nReply from aap02.example.net in 848.816\u00b5s \n<br>\n&#91;root@aap01 receptor]# receptorctl --socket \/var\/run\/awx-receptor\/receptor.sock ping aap01gcp.example.net \n<br>\nReply from aap01gcp.example.net in 4.143774ms \n<br>\nReply from aap01gcp.example.net in 4.049415ms \n<br>\nReply from aap01gcp.example.net in 7.643543ms \n<br>\nReply from aap01gcp.example.net in 4.131193ms <\/code><\/pre>\n\n\n\n<p>Repeat the test from the execution node to the controller nodes:&nbsp;&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@aap01gcp ~]# receptorctl --socket \/var\/run\/awx-receptor\/receptor.sock ping aap01.example.net \n<br>\nReply from aap01.example.net in 3.998636ms \n<br>\nReply from aap01.example.net in 4.07025ms \n<br>\nReply from aap01.example.net in 4.053869ms \n<br>\nReply from aap01.example.net in 4.43546ms \n<br>\n <br>\n<br>\n&#91;root@aap01gcp ~]# receptorctl --socket \/var\/run\/awx-receptor\/receptor.sock ping aap02.example.net \n<br>\nReply from aap02.example.net in 3.131143ms \n<br>\nReply from aap02.example.net in 3.118211ms \n<br>\nReply from aap02.example.net in 3.35466ms \n<br>\nReply from aap02.example.net in 3.120776ms <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The receptor\u2019s configuration should not change. If you suspect that the configuration has changed, review the receptor configuration. Ensure that the receptor configuration files on both the Controllers and the execution node(s) are correctly configured. The configuration is in \/etc\/receptor\/receptor.conf. Review this file for any misconfigurations. The following configuration file has been configured during the installation:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@aap01 receptor]# cat receptor.conf \n<br>\n--- \n<br>\n- node: \n<br>\n    id: aap01.example.net \n<br>\n    firewallrules: \n<br>\n      - action: \"reject\" \n<br>\n        tonode: \"aap01.example.net\" \n<br>\n        toservice: \"control\" \n<br>\n <br>\n<br>\n- work-signing: \n<br>\n    privatekey: \/etc\/receptor\/work_private_key.pem \n<br>\n    tokenexpiration: 1m \n<br>\n <br>\n<br>\n- work-verification: \n<br>\n    publickey: \/etc\/receptor\/work_public_key.pem \n<br>\n <br>\n<br>\n <br>\n<br>\n# Log Level \n<br>\n- log-level: info \n<br>\n <br>\n<br>\n# Control Service \n<br>\n- control-service: \n<br>\n    service: control \n<br>\n    filename: \/var\/run\/awx-receptor\/receptor.sock \n<br>\n    permissions: 0660 \n<br>\n    tls: tls_server \n<br>\n <br>\n<br>\n# TLS \n<br>\n- tls-server: \n<br>\n    name: tls_server \n<br>\n    cert: \/etc\/receptor\/tls\/aap01.example.net.crt \n<br>\n    key: \/etc\/receptor\/tls\/aap01.example.net.key \n<br>\n    clientcas: \/etc\/receptor\/tls\/ca\/mesh-CA.crt \n<br>\n    requireclientcert: true \n<br>\n <br>\n<br>\n- tls-client: \n<br>\n    name: tls_client \n<br>\n    cert: \/etc\/receptor\/tls\/aap01.example.net.crt \n<br>\n    key: \/etc\/receptor\/tls\/aap01.example.net.key \n<br>\n    rootcas: \/etc\/receptor\/tls\/ca\/mesh-CA.crt \n<br>\n    insecureskipverify: false \n<br>\n <br>\n<br>\n <br>\n<br>\n# Peers \n<br>\n- tcp-peer: \n<br>\n    address: aap02.example.net:27199 \n<br>\n    redial: true \n<br>\n    tls: tls_client \n<br>\n- tcp-peer: \n<br>\n    address: aap03.example.net:27199 \n<br>\n    redial: true \n<br>\n    tls: tls_client \n<br>\n- tcp-peer: \n<br>\n    address: aap01gcp.example.net:27199 \n<br>\n    redial: true \n<br>\n    tls: tls_client \n<br>\n <br>\n<br>\n# Work-commands \n<br>\n- work-command: \n<br>\n    worktype: local \n<br>\n    command: \/var\/lib\/awx\/venv\/awx\/bin\/ansible-runner \n<br>\n    params: worker \n<br>\n    allowruntimeparams: true \n<br>\n    verifysignature: true \n<br>\n <br>\n<br>\n- work-kubernetes: \n<br>\n    worktype: kubernetes-runtime-auth \n<br>\n    authmethod: runtime \n<br>\n    allowruntimeauth: true \n<br>\n    allowruntimepod: true \n<br>\n    allowruntimeparams: true \n<br>\n    verifysignature: true \n<br>\n <br>\n<br>\n- work-kubernetes: \n<br>\n    worktype: kubernetes-incluster-auth \n<br>\n    authmethod: incluster \n<br>\n    allowruntimeauth: true \n<br>\n    allowruntimepod: true \n<br>\n    allowruntimeparams: true \n<br>\n    verifysignature: true <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Restart Receptor. Sometimes simply restarting the Receptor can help resolve minor issues:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>  systemctl restart receptor <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensure there aren\u2019t any firewall rules or networking issues preventing communication. Check firewall settings on both the Controller and execution nodes to ensure the required ports for Receptor are open. The receptor is using port 27199. You can use the receptor status or ping commands to verify the communication. If the receptor ping does not work, that might indicate routing or firewall issues. Things to check:&nbsp;&nbsp;\n<ul class=\"wp-block-list\">\n<li>firewall-cmd &#8211;list-all (if the firewalld is used on the host)<\/li>\n\n\n\n<li>firewall configuration at the network level<\/li>\n\n\n\n<li>routing &#8211; use system level trouceroute command or receptor level traceroute.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@aap01 receptor]# receptorctl --socket \/var\/run\/awx-receptor\/receptor.sock traceroute aap01gcp.example.net \n<br>\n0: aap01.example.net in 249.524\u00b5s \n<br>\n1: aap01gcp.example.net in 4.00961ms <\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Receptor TLS\/SSL Issues<\/h3>\n\n\n\n<p>&nbsp;When verifying TLS\/SSL configurations, especially in the context of Receptor communications in Ansible Automation Platform, you can follow the steps below to ensure everything is in order:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Check Certificate Expiry. You can use the openssl tool to inspect a certificate&#8217;s details, including its expiration date. Check if \u2018Not Before\u2019 and \u2018Not After\u2019 are correct and the certificate is still valid:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@aap01 receptor]# openssl x509 -in \/etc\/receptor\/tls\/aap01.example.net.crt -noout -text \n<br>\nCertificate: \n<br>\n    Data: \n<br>\n        Version: 3 (0x2) \n<br>\n        Serial Number: 1676526891 (0x63edc52b) \n<br>\n        Signature Algorithm: sha256WithRSAEncryption \n<br>\n        Issuer: CN = Ansible Automation Controller Nodes Mesh ROOT CA \n<br>\n        Validity \n<br>\n            Not Before: Feb 16 05:54:51 2023 GMT \n<br>\n            Not After : Feb  6 05:54:32 2033 GMT \n<br>\n        Subject: CN = aap01.example.net \n<br>\n        Subject Public Key Info: \n<br>\n            Public Key Algorithm: rsaEncryption \n<br>\n                RSA Public-Key: (4096 bit) \n<br>\n                Modulus: \n<br>\n                    87:82:34:3d:3d:3b:7a:c7:bd:7f:0d:4f:b6:cf:ea: \n<br>\n                    26:36:01:94:b5:87:02:b4:4c:00:98:ba:6b:4c:6f: \n<br>\n                    7f:2a:4b:f7:6f:b9:50:af:43:80:ea:f7:4b:b5:68: \n<br>\n                    e2:75:de:93:e0:df:dd:90:72:5e:45:8d:5a:4e:35: \n<br>\n                    b7:12:3c:2f:f2:c4:22:1f:87:d8:ca:6f:ae:84:1e: \n<br>\n                    2e:f8:01:4c:a2:22:fd:fd:4c:2b:ea:31:b8:a7:5b: \n<br>\n                    d0:8d:08:4f:a7:58:25:b3:6d:15:11:67:b7:b1:51: \n<br>\n                    da:39:ed:61:3a:77:15:9a:cd:e2:4e:4c:ee:97:17: \n<br>\n                    31:cf:13:df:e8:5a:ee:8e:35:3e:3c:60:dc:7e:10: \n<br>\n                    c2:23:2f:37:c8:72:75:aa:79:26:c1:c0:83:76:33: \n<br>\n                    a2:a8:63:de:e8:cd:07:46:3d:66:3b:3e:63:71:ed: \n<br>\n                    a9:d9:7e:ba:79:db:ab:dd:66:a0:6f:27:88:79:7a: \n<br>\n                    51:cc:fe:76:1e:94:d4:ac:dc:8c:d6:70:56:67:cc: \n<br>\n                    47:4c:ba:58:e3:e9:50:c3:69:73:b6:a0:5e:e0:1a: \n<br>\n                    ef:6e:91:15:08:41:b5:9c:d4:e5:2b:97:cf:db:22: \n<br>\n                    53:48:fa:50:28:a8:6e:17:3f:dd:0b:4e:b1:0e:6a: \n<br>\n                    dc:28:6d:ec:eb:5f:16:f0:eb:33:ac:d2:f9:60:2a: \n<br>\n                    ba:02:44:89:b5:80:3e:d9:0f:21:08:cd:3e:e2:f4: \n<br>\n                    4d:04:11:8f:f6:d2:af:23:ed:9f:5c:a2:87:2a:52: \n<br>\n                    81:c0:f0:81:64:7f:47:13:2c:18:40:9b:88:25:47: \n<br>\n                    3a:d4:a8:5c:43:26:27:7f:7f:1f:40:4f:7f:1d:38: \n<br>\n                    00:fa:de:47:c6:16:58:a5:54:a7:86:cc:e3:df:43: \n<br>\n                    72:40:d2:09:4b:47:77:05:4b:9f:23:d9:62:ce:70: \n<br>\n                    35:0c:05:09:1a:79:d2:9b:0d:6f:d4:6e:db:97:89: \n<br>\n                    1a:0b:fb:ed:ae:c8:2d:fb:7c:8d:b3:47:38:78:36: \n<br>\n                    5a:0b:b5:37:9d:f8:de:d0:81:6f:76:bf:75:30:40: \n<br>\n                    b1:6c:71 \n<br>\n                Exponent: 65537 (0x10001) \n<br>\n        X509v3 extensions: \n<br>\n            X509v3 Key Usage: critical \n<br>\n                Digital Signature \n<br>\n            X509v3 Extended Key Usage: \n<br>\n                TLS Web Client Authentication, TLS Web Server Authentication \n<br>\n            X509v3 Authority Key Identifier: \n<br>\n <br>               keyid:B3:59:0F:79:B2:41:78:5C:7D:31:9F:95:DD:98:0F:6E:B6:7B:C8:FC \n<br>\n <br>\n<br>\n            X509v3 Subject Alternative Name: \n<br>\n                DNS:aap01.example.net, IP Address:10.29.32.222, othername:&lt;unsupported&gt; \n<br>\n    Signature Algorithm: sha256WithRSAEncryption <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensure private key matches the certificate. You can check that the private key corresponds to the certificate:&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>openssl x509 -noout -modulus -in \/path\/to\/certificate.crt | openssl md5  \n<br>\nopenssl rsa -noout -modulus -in \/path\/to\/private.key | openssl md5 <\/code><\/pre>\n\n\n\n<p>The MD5 hashes from both commands should match.&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@aap01 receptor]# openssl x509 -noout -modulus -in \/etc\/receptor\/tls\/aap01.example.net.crt | openssl md5 \n<br>\n(stdin)= 8baa6271452a553a492cb79f70586100 \n<br>\n&#91;root@aap01 receptor]# openssl rsa -noout -modulus -in \/etc\/receptor\/tls\/aap01.example.net.key | openssl md5 \n<br>\n(stdin)= 8baa6271452a553a492cb79f70586100 <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Test TLS Connection. You can use openssl to manually establish a connection to the Receptor service to check the TLS handshake&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>openssl s_client -connect receptor-node-ip:port -CAfile \/path\/to\/ca.crt \n<br>\n&#91;root@aap01 receptor]# openssl s_client -connect aap02.example.net:27199 -CAfile \/etc\/receptor\/tls\/ca\/mesh-CA.crt \n<br>\nCONNECTED(00000003) \n<br>\ndepth=1 CN = Ansible Automation Controller Nodes Mesh ROOT CA \n<br>\nverify return:1 \n<br>\ndepth=0 CN = aap02.example.net \n<br>\nverify return:1 \n<br>\n--- \n<br>\nCertificate chain \n<br>\n 0 s:CN = aap02.example.net \n<br>\n   i:CN = Ansible Automation Controller Nodes Mesh ROOT CA \n<br>\n--- \n<br>\nServer certificate \n<br>\n <\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Troubleshooting Execution Environments<\/h3>\n\n\n\n<p>Troubleshooting execution environments (EE) on Ansible Controller involves several steps, as execution environments play a vital role in encapsulating resources needed to run playbooks. Here&#8217;s a structured approach:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\" start=\"1\">\n<li><strong>Verify EE Image<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure that the EE image exists and is correctly specified. Use podman to list images:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>podman images <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\">\n<li><strong>Run EE Image Manually<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Try to run the image manually to see if there are any issues:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>podman run -it --rm &lt;image_name_or_id&gt; \/bin\/bash <\/code><\/pre>\n\n\n\n<p>This will allow you to enter the EE container. You can inspect its content and check if all required tools, libraries, and Ansible collections or roles are present.&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\" start=\"3\">\n<li><strong>Verify Resources<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Ensure that the host running the Ansible Controller has sufficient resources (CPU, memory, disk space). Running out of resources can cause unexpected issues with execution environments. For example, verify if there is free space on \/var\/lib\/awx file system. Note that the controller keeps its container images under \/var\/lib\/awx\/.local<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>df -h <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"4\">\n<li><strong>Ansible Configuration<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Check the Ansible configuration inside the EE. The ansible.cfg file should have the right parameters. If you&#8217;re inside the EE container, you can display it using:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>cat \/etc\/ansible\/ansible.cfg <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"5\">\n<li><strong>EE Specific Errors<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>If your playbooks refer to custom modules, roles, or collections, ensure they are available within the EE. Remember, EEs should encapsulate all the required dependencies to run the playbook.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"6\">\n<li><strong>Network Access<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure the EE has appropriate network access to reach target nodes, any required repositories, or other resources.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"7\">\n<li><strong>Security Contexts and Privileges<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>If the Controller runs on a system with SELinux, ensure that your EE is granted the necessary contexts or privileges to operate.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"8\">\n<li><strong>Dependencies and Pipelines<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>If your EE has dependencies on external systems or services, ensure they are operational. This could include SCM repositories, credential stores, or third-party services.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Validate Playbooks<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>It might be an issue with the playbook rather than the EE. Try running the playbook with verbosity:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>ansible-playbook -vvv your_playbook.yml <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>This might give more insights into where and why the playbook is failing&nbsp;<\/li>\n<\/ul>\n\n\n\n<ol class=\"wp-block-list\" start=\"11\">\n<li><strong>Custom EE Builds<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>If you&#8217;re building your own EE images, ensure that the build process completes without errors. Check the Containerfile for any issues.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"12\">\n<li><strong>Controller Configuration<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure that Ansible Controller itself is correctly configured to use EEs. This includes verifying paths, image names, or any other settings specific to EEs.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p>By following these steps, you can systematically identify and resolve issues with execution environments on the Ansible Controller. If you&#8217;re still encountering problems, consult the official Ansible documentation or reach out to Red Hat support.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Troubleshooting Communication between AAP Controller and Hub<\/h2>\n\n\n\n<ol class=\"wp-block-list\" start=\"1\">\n<li><strong>Basic Connectivity<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Check if the Controller node can reach Automation Hub:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>ping &lt;automation_hub_host&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\">\n<li><strong>Check Ports<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Automation Hub typically listens on port 443 for SSL traffic. Ensure this port is open and reachable:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>ssh -vp 443 &lt;automation_hub_host&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"3\">\n<li><strong>View Logs on Controller Node<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>For the Controller services:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>journalctl -u automation-controller <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Also, refer to the logs located in \/var\/log\/tower\/.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ol class=\"wp-block-list\" start=\"4\">\n<li><strong>View Logs on Automation Hub<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>For the Pulp services, which back Automation Hub:&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>journalctl -u pulpcore-worker@*.service <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Check the Pulp logs typically located in \/var\/log\/pulp\/.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ol class=\"wp-block-list\" start=\"5\">\n<li><strong>Verify SSL\/TLS<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure certificates are correctly configured, valid, and trusted on both ends.&nbsp;<\/li>\n\n\n\n<li>If self-signed certificates are in use, ensure they&#8217;re added to the trust store on the Controller nodes.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"6\">\n<li><strong>API Authentication<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>The controller communicates with Automation Hub using token-based authentication. Ensure tokens are valid and not expired.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"7\">\n<li><strong>Proxy Issues<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>If a proxy is in use, ensure its correctly configured. Check proxy logs for denied requests.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"8\">\n<li><strong>Firewall Rules<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Check if there are firewall rules blocking communication between the Controller and Automation Hub.&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"9\">\n<li><strong>DNS Issues<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure DNS resolution works correctly from Controller to Automation Hub:<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>nslookup &lt;automation_hub_host&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"10\">\n<li><strong>Network Configuration<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Check for changes in network configurations that might have affected communication.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Automation Hub Health<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Check that Automation Hub&#8217;s services and processes are running and healthy.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Database Connectivity for Automation Hub<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure the database backend for Automation Hub (Pulp) can connect without issues.&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Controller Version Compatibility<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure Controller and Automation Hub versions are compatible.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Updates\/Patches<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Check for any updates or patches that might address communication issues.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">SAML authentication issues between Ansible Controller and Microsoft Azure AD&nbsp;<\/h3>\n\n\n\n<ol class=\"wp-block-list\" start=\"1\">\n<li><strong>Configuration Check<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Ensure that the configuration on both Ansible Controller and Azure AD side matches. This includes Entity IDs, Assertion Consumer Service URLs, and other key SAML attributes.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Certificate Validation<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure the certificate used for signing the SAML assertion in Azure AD is still valid and has not expired. Note that the on AAP side you are using the certificate used for tower.cert; tower.key&nbsp;<\/li>\n\n\n\n<li>The same certificate should be configured in Ansible Controller&#8217;s SAML settings.&nbsp;<\/li>\n\n\n\n<li>If you&#8217;re using encrypted assertions, make sure you&#8217;ve provided the correct<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Assertion Content<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Using tools like SAML Tracer for Firefox or the SAML Chrome Panel for Chrome can help you capture the SAML assertion sent from Azure AD to Ansible Controller.&nbsp;<\/li>\n\n\n\n<li>Check if the attributes in the assertion match what&#8217;s expected by Ansible Controller.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Azure AD Configuration<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure that the user trying to authenticate is part of the Azure AD user group that&#8217;s allowed to log into Ansible Controller.&nbsp;<\/li>\n\n\n\n<li>Confirm that the SAML configuration on Azure AD side, especially the claim rules, are correctly set up to provide the necessary claims to Ansible Controller.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Endpoint URLs<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Ensure that the SSO URL and Entity ID on both Azure AD and Ansible Controller match. Any discrepancies here will cause authentication to fail.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Clock Skew<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>SAML assertions are time sensitive. Ensure that the system clocks on both Ansible Controller and Azure AD (Azure&#8217;s end will typically be accurate) are synchronised.&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Role and Attribute Mapping<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>In Ansible Controller&#8217;s SAML settings, ensure that you&#8217;ve correctly mapped Azure AD attributes to Controller&#8217;s user attributes and roles.&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Network Issues<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Confirm there are no network issues preventing Ansible Controller from reaching Azure AD or vice versa.&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Session and Cookie Issues<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Clear cookies and session data in your browser or try a different browser to rule out any session-specific issues.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">AWX-MANAGE Tool<\/h3>\n\n\n\n<ol class=\"wp-block-list\" start=\"1\">\n<li><strong>Configuration<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>View current AAP configuration:&nbsp;<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx-manage print_settings <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Check for pending migrations. This command can be used after the upgrade, to verify the progress of the DB migrations<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>awx-manage showmigrations <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\">\n<li><strong>Database<\/strong>:&nbsp;\n<ul class=\"wp-block-list\">\n<li>Check DB (if the database responds, version etc)<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx-manage check_db <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Clean up old job history (this will remove old jobs and preserve space):&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>awx-manage cleanup_jobs <\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Verify the instances (controller nodes and execution nodes)&nbsp;<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>awx-manage list_instances <\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">AWX Tool<\/h3>\n\n\n\n<p>Remember to add the following after each command:&nbsp;&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--conf.host https:\/\/aap_fqdn --conf.username admin --conf.password \u2018password\u2019 --conf.insecure <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Get Configuration<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx config <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\">\n<li><strong>List All Jobs<\/strong>:&nbsp;<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx jobs list <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"3\">\n<li><strong>Retrieve Details of a Specific Job<\/strong>:&nbsp;<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx jobs get &lt;job_id&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"4\">\n<li><strong>List All Projects<\/strong>:&nbsp;<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx projects list <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"5\">\n<li><strong>Retrieve Details of a Specific Project<\/strong>:&nbsp;<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx projects get &lt;project_id&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"6\">\n<li><strong>List All Inventories<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx inventories list <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"7\">\n<li><strong>Retrieve Details of a Specific Inventory<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx inventories get &lt;inventory_id&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"8\">\n<li><strong>List All Hosts within an Inventory<\/strong>:&nbsp;<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx hosts list --inventory &lt;inventory_id&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"9\">\n<li><strong>Ad-hoc Commands<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx ad_hoc_commands create --inventory &lt;inventory_id&gt; --module-name &lt;module_name&gt; --module-args \"&lt;module_arguments&gt;\" <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"10\">\n<li><strong>List All Job Templates<\/strong>:&nbsp;<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx job_templates list <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"11\">\n<li><strong>Launch a Job Template<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx job_templates launch &lt;template_id&gt; --monitor <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"12\">\n<li><strong>List All Users<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx users list <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"13\">\n<li><strong>Retrieve Details of a Specific User<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx users get &lt;user_id&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"14\">\n<li><strong>Create a New User<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx users create --username &lt;username&gt; --password &lt;password&gt; --email &lt;email&gt; <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"15\">\n<li><strong>List All Organisations<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx organizations list <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"16\">\n<li><strong>Ping the AWX API<\/strong>:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>awx ping <\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Ansible Private Automation Hub<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Check Pulp Worker Status<\/strong>: Use systemctl to check the status of the Pulp workers.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl status pulpcore-worker@*.service <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\">\n<li><strong>Review Pulp Worker Logs<\/strong>: The logs can give insight into any issues the workers might be facing.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>journalctl -u pulpcore-worker@*.service <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"3\">\n<li><strong>Restart Pulp Workers<\/strong>: If a worker seems to be stuck or malfunctioning, you can restart it.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl restart pulpcore-worker@*.service <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"4\">\n<li><strong>Clean Up Old Tasks<\/strong>: Sometimes, cleaning up old completed tasks can help.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>pulpcore-manager handle-artifact-checksums  <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"5\">\n<li><strong>Database Issues<\/strong>: Check the status of the PostgreSQL database that Pulp uses. If there are connectivity issues, Pulp tasks will fail.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"6\">\n<li><strong>Disk Space<\/strong>: Ensure that there&#8217;s enough disk space where Pulp stores its content and artifacts. Running out of space can cause tasks to fail.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"7\">\n<li><strong>Check the Number of Workers<\/strong>: If you&#8217;re dealing with a large number of tasks or large-sized content, you might need to scale up the number of Pulp workers.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"8\">\n<li><strong>SELinux<\/strong>: SELinux policy denials can interfere with the operation of services, including Pulp. Check for any relevant AVC denials.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>ausearch -m AVC -ts recent <\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"9\">\n<li><strong>Check Connectivity to External Repositories<\/strong>: If you&#8217;re having issues syncing from external repositories, ensure network connectivity, and that firewalls or proxies aren&#8217;t blocking the connection.<\/li>\n<\/ol>\n\n\n\n<p>Are you facing challenges with Ansible Automation Platform troubleshooting? Our team of experienced experts is here to help you resolve any issues swiftly and efficiently. Whether it&#8217;s communication problems, configuration errors, or any other technical difficulties, we have the knowledge and expertise to assist you.&nbsp;<\/p>\n\n\n\n<p>Don&#8217;t let automation issues slow down your operations. <a href=\"https:\/\/www.insentragroup.com\/au\/contact\/\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/www.insentragroup.com\/nz\/contact\/\" rel=\"noreferrer noopener\">Contact us<\/a> today, and let&#8217;s work together to ensure your Ansible Automation Platform runs smoothly and efficiently.\u00a0<\/p>\n\n\n\n<style>\nbody .wp-block-code>code {\n    font-family: Menlo,Consolas,monaco,monospace;\n    color: #000;\n    padding: 30px 40px;\n    border: none;\n    border-radius: 4px;\n    background: #ddd;\n}\n<\/style>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to troubleshoot communication issues and configuration problems in Ansible Automation Platform with expert tips from Sebastian Baszczyj. <\/p>\n","protected":false},"author":67,"featured_media":19337,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[19],"tags":[],"class_list":["post-19336","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-modern-workplace","entry"],"_links":{"self":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts\/19336","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/comments?post=19336"}],"version-history":[{"count":2,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts\/19336\/revisions"}],"predecessor-version":[{"id":23140,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/posts\/19336\/revisions\/23140"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/media\/19337"}],"wp:attachment":[{"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/media?parent=19336"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/categories?post=19336"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.insentragroup.com\/us\/wp-json\/wp\/v2\/tags?post=19336"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}