Remediation Custom Jobs

Last updated on August 19, 2020

Overview

VMware Secure State provides a highly extensible remediation framework for improving cloud security by automating actions across your cloud environments. Users can customize remediation actions – modify existing jobs, create new jobs or delete existing jobs. Through custom remediation jobs, users may author new actions that can target native or custom rules and can be manually or automatically triggered. The framework can be extended for various use cases for security operations and administration, such as applying cloud tags to resources, creating JIRA tickets after taking a remediation action, sending an email after applying a hardened configuration, etc. This document will walk through the process of customizing remediation jobs.

This walkthrough assumes a basic understanding of VSS Remediation. If you’re just getting started with Remediation, you can read more here.


Finding Existing Jobs

The remediation worker with all the native VSS jobs is available to users as part of the worker container. All remediation jobs can be found in a designated directory on the worker. First, you must disable the worker group and run the worker in your preferred environment. Next, use the following commands:

  1. List all the running containers and find the container Id for the worker. docker ps

  2. Connect to the remediation worker container. docker exec -it {Container ID} /bin/sh

  3. Find the directory with all the remediation jobs. cd opt/vss/job-code/

Remediation jobs are Python scripts that follow certain naming and design conventions. Each directory in the job-code folder represents a unique job, and each job contains a .py file with the directory name. The name of the job directory registers itself as the job name on the VSS platform. The .py file is executed by the worker with certain arguments each time a remediation action is triggered.

Every job also includes a minimum_policy.json describing the minimum permissions needed for the job, along with a README file with details about what the job does.

By convention, all jobs are executed via the {Job Name}.py file with the following argument available as sys.args[1]:

{ "cloudAccount": { "provider": string, "roleArn": string, "subscriptionId": string, "applicationId": string, }, "notificationInfo": { "RuleID": string, "RuleName": string, "RuleDisplayName": string, "Level": string, "Service": string, "FindingInfo": { "FindingId": string, "ObjectId": string, "ObjectChain": string, "CloudTags": { "key1": "value1", "key2": "value2", }, "RiskScore": integer, "Region": string, "Service": string } }, "autoRemediate": boolean }

Properties such as "ObjectId" and "CloudAccountId" can be used to identify the targeted resource. Other properties may be used for logging or building logic in custom jobs.


Adding or Modifying a Remediation Job

Users can add new remediation jobs or modify existing jobs by making changes to the worker container. It is recommended to make modifications to jobs by first running the worker container locally, mounting the local directory on the container, and then making changes.

Create a New Job

Begin by navigating to the opt/vss/job-code/ directory. Create a new directory with the name of the desired job. Spaces are not allowed in the name. Next, create a Python script matching the job name i.e. {Job Name}.py file. Refer to one of the other jobs in the same directory for an example. The remediation job will be executed with a JSON string parameter (sys.args[1]) containing all the details of the finding, the object, and the remediation status. The schema of the argument is provided above. You can include other Python files, import other libraries, etc., as long as the execution environment supports it. All standard out and standard error logs will be sent back to VSS and will show up in the Remediation Logs tab and the linked Worker Group Logs tab.

Note: The remediation job directory name MUST match with the .py file for correct setup. In case of a name mismatch, the worker error logs will identify it. Remediation jobs utilize boto3 for making calls to the AWS APIs. Refer to the docs here for supported services and APIs.

For iterative custom job development, it is recommended that you mount your local drive to the worker container so that any changes made locally are automatically synced to the worker.

  1. Copy the remediation jobs from the container to a local folder. docker cp {Container ID}:/opt/vss/job-code {destination folder}

  2. Make updates to the folder here, following the guidelines for creating a new remediation job described above.

  3. Mount the local folder on the worker container by running the container with the command provided in the platform and adding the volume mount parameter: -v "$(pwd)":/opt/vss/job-code

The worker job code is registered with VSS only during worker startup time, so you must restart the worker container after making changes. We recommend users to follow all development best practices when developing new remediation job code. Finally, deploy a worker with the new image. The worker will register the newly created remediation job with VSS, and it will now be available for selection when you create a new remediation.

Note: We only support 1 worker instance per worker group for custom jobs. While the system will seem to work as expected with multiple workers, differences in custom jobs can lead to mismatches in the job code versions, leading to unexpected outcomes when a remediation job is started. In a future release, VSS will support multiple workers with custom jobs, handling job code mismatches.


Modify an Existing Job

Similar to authoring a new job, you can modify any existing job listed in the opt/vss/job-code/ directory. Disable the remediation worker group before making any modifications. You can include other Python files, import other libraries, etc., as long as the execution environment supports it. You can modify it to explore new use cases such as applying a Remediated tag on an object after taking an action upon it, sending an email after a Remediation complete, or even link multiple Remediation actions in a sequence.

For more ideas on possible custom jobs, refer to the VSS Remediation Github page. All VSS Remediation jobs are open sources, so you can contribute to the VSS community by submitting your custom jobs!


Conclusion

VMware Secure State provides an extensible remediation framework that allows users to create new jobs or modify existing jobs, enabling customized remediation actions.