Citrix released the security bulletin CTX267027 on the 17th of December 2019. This article identifies a vulnerability in Citrix Application Delivery Controller (ADC), Citrix Gateway and Citrix SD-WAN appliances. The vulnerability was labelled CVE-2019-19781 and could allow an unauthenticated attacker to perform arbitrary remote code execution via directory traversal.
The National Vulnerability Database (NVD) also modified the rating for the exploit on the 8th of January to a 9.8 Critical base score. On the 11th January, it was identified the exploit had been transformed into a weaponisation version of the exploit and multiple compromises were identified by FireEye.
How to address the exposure with the latest Citrix firmware
Citrix and Insentra strongly advise all Citrix ADC, Citrix Gateway and Citrix SD-WAN appliances are updated to the latest firmware version to mitigate this vulnerability Citrix ADC Firmware. This may not fully address any exposure your devices have had between the identification of the vulnerability and applying remediation actions.
To apply the latest firmware from Citrix, follow this guide. Once the latest firmware has been applied, it is advisable to run the Verification Tool against the appliance to confirm that the vulnerability has been closed on the device.
Additional recommended steps to ensure the security of the devices has not been compromised
On 22nd of January 2020, a Compromise Scanner Tool was developed and released by Citrix and Fireye. This scanner was created to investigate the Citrix appliances for any known indicators of this vulnerability being exploited. It is strongly recommended to run this scanner on all devices in the advisory to ensure there are no indicators of compromise, this includes all nodes of any HA pairs deployed.
To perform the scan
1. Copy the script to the Citrix appliance using WinSCP
2. Log on to the appliance using a terminal application, e.g. Putty.
3. Enter Shell on the device
4. Browse to the folder where the script was uploaded to
5. Execute the script with the below command
6. Export the resultant txt file for analysis
What to look for in the results of the compromise scanner tool
The results from a scanner will fall into one of three categories if any compromising indicators are found.
What may be impacted on a compromised appliance
Due to the nature of the vulnerability, there are several aspects which may be impacted.
- Modification of the Citrix appliance configuration
- Replacement of the certificates used by the Citrix appliance
- Backdoor installed for future access
- Invasive services on the NetScaler process as part of this compromise is often associated with a cryptocurrency mining utility
The results in the output of the scan will show indicators of vulnerability. Some of the results may show activity which may be legitimate, other areas will show suspicious activity as per the example below, including unauthorised cron job, user creation and communication to blacklisted sites:
What to do if you have been compromised
Upon evidence a compromised device is found, it is recommended to forensically analyse the logs from the appliance to attempt to determine the extent of the exposure. Insentra also recommends one of the following two actions is carried out on the device(s):
Option 1 – Restore, Patch, Release
- Restore the Citrix appliance to a version earlier than the vulnerability discovery
- Restore the device from a date in early December 2019
- Disable networking on the appliance
- Upload the latest firmware to the operating system
- Perform firmware update
- Retest appliance with the verification and compromise scanner tools
- Enable networking on the device
- Migrate device to production
Option 2 – Rebuild, Release
If the process in option 1 cannot be undertaken or is unsuccessful. Option 2 is a ground-up rebuild of the appliance
- Deploy new Citrix appliances on the latest firmware
- Review revision history where possible on the Citrix appliance
- Backup running configuration from the existing device
- Sanitise configuration based on known/required services
- Backup certificates directory
- Configure new Citrix appliances
- Reissue the licence files for the latest instance of the appliance
- Restore certificates
- Apply configurations
- Test and confirm functionality
- Migrate device to production
A rebuild of the services may be deemed the more appropriate action to be undertaken, given a zero-tolerance approach to any external-facing network device being compromised.
What do you need to do
This advisory has covered what the vulnerability is and its severity, it has also provided the actions to follow to ensure any Citrix device in your environment is not at risk of being compromised by this vulnerability.
- How to patch Citrix devices with the latest firmware that close the vulnerability
- How to verify the device is no longer vulnerable to this exploit
- How to detect if your Citrix device has been compromised
- If your device has been compromised, what your options are to remediate the compromised appliance.
If you would like any assistance in performing any of these tasks, whether it be updating the appliance, analysing the output of the compromise scanner, or dealing with a compromised device, please do not hesitate to reach out to Insentra, we will provide any assistance you require.