Skip to main content

Troubleshoot Ransomware Rollback in OneView

Updated: 

The Ransomware Rollback feature in Endpoint Detection and Response (EDR) allows you to revert file changes made by malware or ransomware on Windows endpoints. This article provides a deeper understanding of the EDR backup and remediation solution and how to troubleshoot it.

Checking for the EDR Service

To ensure EDR backups are being created, check that the EDR service is running on an endpoint:

  • In the console.
    1. Go to Manage > Endpoints.
    2. Click on an affected endpoint.
    3. Click See more details.
    4. Check the Agent and Plugins info section for the following
      1. Presence of Endpoint Detection and Response plugin.
      2. Date listed for Agent info last refreshed is the current day.
    5. Example: 
      Agent and plugins
      Agent info last refreshed: 06/23/2023 10:50:09 AM*
      Agent version: 1.2.0.1048
      Endpoint Detection and Response: 1.2.0.387
  • On the endpoint.
    • Check the About screen.
      1. Hold control and right-click the system tray icon on the endpoint and click About.
      2. Verify the Endpoint Detection and Response version in the list.
    • Check with command prompt.
      1. Open command prompt.
      2. Run the following commands.
        1. SC QUERY MBEndpointAgent
        2. SC QUERY flightrecorder
    • Check with Powershell
      1. Open PowerShell.
      2. Run the following command.
        • Get-Service -Name flightrecorder,MBEndpointAgent.

Backups

Before a file is modified or a registry entry is changed, a backup is made in the following folder:

  • C:\ProgramData\ Endpoint Agent\Plugins\EDRPlugin\Backups\

For servers, this path can be modified to a separate drive. For more information, see Configure Endpoint Detection and Response options in OneView.

Notes

  • The folder and contents are self-protected by drivers against attack, preventing even local administrators from deleting the folder.
  • Each backup is encrypted to avoid interference and scanning by other security products. Exclude this folder from other security products to avoid unwanted detections and false positives. For more information, see Network access requirements and firewall settings for OneView.
  • All file types can be backed up (docs, xls, json, xml, exe, dll, etc.)
    • There is a 14-day self-learning process. After that period, for space and performance optimization, backups are ignored by trusted processes. For example:
      • A document edited by Word.exe would be ignored
      • Backups would occur for a document edited by an untrusted or malicious process
  • Files are named like 0000001670324876267_2D7E74B2.frb.
    • The first part of the name contains the backup time with a Unix timestamp. (0000001670324876267 = Tue Dec 06 2022 22:07:56). For information on converting the timestamp, see https://www.unixtimestamp.com/.
    • The second part of the name (2D7E74B2) is randomly generated.
    • The extensions are either Flight Recorder Backup (FRB) and Flight Recorder Backup Registry (FRBR).
    • The file's datetime viewed by Windows, is the creation date of the original or source file, not the backup time.
  • Backups are to a local drive for very fast recovery.
  • Offline backups are still recommended to cover cases of hard drive crashes, theft of device, and for files ignored by Ransomware Rollback.

Remediation and Ransomware Rollback

A Remediation action can be triggered for any suspicious activity alert. When a remediation is triggered, a scan is run to clean the identified processes.

Additionally, remediating a [Ransomware] suspicious activity alert automatically begins the ransomware rollback process.

The rollback uses the processes identified in the alert to identify the files modified by that process, then copying and overwriting files changed with the prior good copies.

This design takes away the need to discover the exact date and time of the start of the attack.

Self cleaning

The backups perform self cleaning to meet the thresholds set in the Endpoint Detection and Response policy configuration.

A task runs every 10 minutes to check the disk quota and delete old backups or unindexed files if the quota is exceeded. This activity is logged in the following location:

  • C:\ProgramData\ Endpoint Agent\Logs\EndpointAgent.txt.

The following is an example of successful cleanups.

INFO FRCoreManager [FRSDK] Running cleanup ALL. RollbackTTL: 72 LearningMode: 2
INFO FRCoreManager [FRSDK] FR cleanup ALL started. Cleaning up events older than 2022-11-28 17:22:34+1100. Current backup files total number: 1579 and disk stats size/free/usage/quota/quota%: 84880125952/1781522432/537416136/695681570/30%
INFO FRCoreManager [FRSDK] FR cleanup ALL finished. Deleted 11574 backup events and 298 backup files. New backup files total number: 1281 and disk size/free/usage/quota/quota%: 84880125952/1902772224/390548912/687996340/30%
INFO FRCoreManager [FRSDK] Next backup cleanup scheduled for 2022-12-02 17:22:33+1100
INFO FRCoreManager Checking for orphaned backup files under "C:\ProgramData\ Endpoint Agent\Plugins\EDRPlugin\Backup"
INFO FRCoreManager Finished checking for orphaned backup files under "C:\ProgramData\ Endpoint Agent\Plugins\EDRPlugin\Backup"

Backup folder very large or exceeds configured quota

This issue was resolved with Endpoint agent version 1.2.0.1054 and the EDR plugin 1.2.0.394. 

Check for updates if the agent or plugin is on a previous version to resolve the issue. If the backup folder is still larger than expected after updating, complete the following:

  1. Check the Windows Services and verify the Endpoint Agent service is Running, as this service controls cleaning.
  2. Enable debug logging. For more information, see Enable debug logging on the Endpoint Agent.
  3. Check the C:\ProgramData\ Endpoint Agent\Logs\EndpointAgent.txt for the following:
    • The self cleaning process is running every 10 minutes. 
      DEBUG FRCoreManager FRCallbackWrapper -- Received Message: Disk size/free/usage/quota/quota%: 510823661568/404060053504/476145280/121360859635/30%
      DEBUG FRCoreManager FRCallbackWrapper -- Received Message: Disk size/free/usage/quota/quota%: 510823661568/404060053504/476145280/121360859635/30%
    • Any log messages containing ERROR FRCoreManager.
  4. Contact Support. 

Manual or emergency cleanup of backup folder

In case it is urgent to free up disk space, complete the following:

  1. Collect diagnostic logs. For more information, see Collect Endpoint Agent diagnostic logs.
  2. Report the issue to Support.
  3. Clean up the backup folder with one of the following methods:
    • Disable and re-enable EDR in the policy settings.
      • Create a Policy with all EDR policy settings disabled. For more information, see Configure Endpoint Detection and Response options in OneView.
      • Create a Group with the new policy assigned.
      • Move the affected endpoints into that group. This will force the EDR plugin to unload and clean up. A reboot may be required.
      • Move the affected endpoints back to the previous group, and EDR will reinstall.
    • Uninstall and reinstall .
      • Use Add or Remove Programs.
      • Use the Discovery and Deployment Tool.

Contacting Support

When submitting a support case, the following information is required:

  • Endpoint names.
  • Diagnostic logs. For more information, see Collect Endpoint Agent diagnostic logs.
    • If you are unable to collect the Diagnostic logs, manually obtain the following files:
      • C:\ProgramData\ Endpoint Agent\Logs\EndpointAgent.txt
      • C:\ProgramData\ Endpoint Agent\Plugins\EDRPlugin\Database\2B455663142B495843A6F3DCB6B55CCE
    •  

Still have questions?

Support is available via chat, phone or by creating a ticket. You can also use our virtual assistant for guided assistance.

Learn more