Microsoft Azure Sentinel 12.1.0

tags: python | SIEM

Description

Integration with Microsoft Azure Sentinel supports CDC users by providing the extraction of logs from Sentinel as alerts and observables. This enables CDC users to make informed decisions regarding incident response.

Sentinel is a scalable, cloud-native, Security Information and Event Management (SIEM) solution. It delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for attack detection, threat visibility, proactive hunting, and threat response.

CyberProof supports various observable models depending on configuration. These models help group observables under headings; otherwise, they are left as independent.

Microsoft Sentinel via KQL (Keyword Query Language) enable us to query against data collected and interactively analyze their results. We use KQL queries to retrieve records that match particular criteria, identify trends, analyze patterns, and provide a variety of insights into our data.

CDC Command Lines

• complete_alert_data_cli
  This CLI is used to complete alert information. For example: complete_alert_data_cli --alert_ids=["59889e23-0229-64fe-ec12-7058e1f9be87","45852sdfs3-0229-64fe-ec12-7058e1f9be87"] --alerts_limit=2 --base_events_limit=5. (Default limit is 10)

Workflows

• async_update_alert
  Update alerts in the CDC from updated Sentinel incidents.

• complete_alert_data
  This CLI is used to complete alert information. Example: complete_alert_data_cli --alert_ids=["59889e23-0229-64fe-ec12-7058e1f9be87","45852sdfs3-0229-64fe-ec12-7058e1f9be87"] --alerts_limit=2 --base_events_limit=5. (Default limit is 10)

• create_alert
  Create alerts in the CDC from Sentinel incidents.

• enrich_ala_incident
  Enrich Sentinel incidents by Azure Log Analytics.

• execute_close_incident
  Close an incident in Sentinel.

• fetch_ala_data
  This workflow is used to fetch an incident and its alerts from Azure Log Analytics.

• update_alert
  Update alerts in the CDC from updated Sentinel incidents.

• update_alert_wrapper
  Update alerts in the CDC from updated Sentinel incidents.

Rules

• close_alert_lisenter
  Close incidents; i.e., an incident in Sentinel.

• create_alert_lisenter
  Triggers injecting a new Azure Log Analytics incident to the CDC workflow.

• update_alert_lisenter
  Triggers injecting an updated Azure Log Analytics incident to the CDC workflow.

Sensors

• SentinelSensor
  Sensor to pull incidents from Azure Sentinel.

Poll interval - 30s

• SentinelUpdateSensor
  Sensor to pull updated incidents from Azure Sentinel.

Poll interval - 30s

Triggers

No triggers

Observable New Data Model

Extract Observables from Alert Entities

The new observable data model structure is an object that contains the following fields:
+ type
+ value
+ extraProperties (optional)
+ relatedExtraProperties (optional)

Observable Example:

json { "type": "account", "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "extraProperties": { "Name": { "value": "ADMINISTRATOR" } }, "relatedExtraProperties": { "accountName": { "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699" }, "accountType": { "value": "account" } } }

Schema

DataStore Keys

• The observable fields filled by the schema configuration via ms_sentinel_entities_schema
  DataStore key.
• ms_sentinel_entities_observable_method - When True or when it was not found, extract observables via the old method. Otherwise, do not extract.

Schema Explanation

• The schema contains entity name as the key, and its configuration as the value.
• Using "strong" and "weak" fields, the "value" of the observable is defined.
• The "type" field in the observable will be the entity.
• The "type" and "value" define the unique ID, which prevents observables duplications.
• The "extraProperties" and "relatedExtraProperties" observable fields will be defined according to the relevant schema fields.

Algorithm Flow

• For each alert, extract entities.
• For each entity -> generate observables candidates (assets).
• Choose the main observable using the schema configuration. Iterate over 'strong' list and search for values compatible with observable candidates of 'type' value. If we do not find a match with a 'strong' list, iterate over the 'weak' list and search for values compatible with observable candidates of the 'type' value.
  Note: We take the first value that the algorithm found in a 'strong' or 'weak' list.
• If we do not find a match to the 'strong' or 'weak' list, observables will be extracted via
  the legacy method - which is according to the ms_sentinel_entities_observable_method DataStore key.
• Rest observables candidates (assets) will be part of the observable 'extraProperties'
  and 'relatedExtraProperties', according to the schema configuration.
  Note: If some observable candidate "type" is not compatible with a schema configuration, such that it is not compatible with 'extraProperties' and 'relatedExtraProperties', then observables will automatically be set as 'relatedExtraProperties'.

Schema Example:

Entities schema configuration:

json { "config": { "account": { "strong": [ "AadUserId" ], "weak": [ "id" ], "extra_prop": [ "Name" ], "related_extra_prop": [ "accountName", "accountType" ] } } }

Entity:

json { "id": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "Name": "name_test", "accountName": "USER", "accountType": "account" }

Observables candidates after data manipulation engine (assets):

Asset(type="id",value="0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699"),
Asset(type="Name",value="name_test"),
Asset(type="accountName",value="USER"),
Asset(type="accountType",value=account)

Observable created:
json { "type": "account", "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "extraProperties": { "Name": { "value": "name_test" } }, "relatedExtraProperties": { "accountName": { "value": "USER" }, "accountType": { "value": "account" } } }

Extract Observables from Base Events

The new observable data model structure is an object that contains the following fields:
+ type
+ value
+ extraProperties (optional)
+ relatedExtraProperties (optional)

Observable Example:

json { "type": "ip", "value": "192.12.23.25", "extraProperties": { "EventSourceName": { "value": "Microsoft-Windows-Security-Auditing" }, "EventOriginId": { "value": "8b9ee84d-2b52-4432-b2e8-e86d11358a67" }, "EventID": { "value": 4625 } }, "relatedExtraProperties": { "Type": { "value": "SecurityEvent" }, "AccountType": { "value": "User" }, "LogonTypeName": { "value": "3 - Network" }, "LogonType": { "value": 3 } } }

Schema

DataStore Keys

• The observable fields filled by the schema configuration via the ms_sentinel_schema
  DataStore key.
• ms_sentinel_observable_method - When True or when the key is not defined, extract observables via the old method. Otherwise, do not extract.

Schema Explanation

• The schema contains the final observable type as key, and its configuration as the value.
• "strong", "weak", "extraProperties" and "relatedExtraProperties" are defined via regex.
• Using "strong" and "weak" fields, the "value" of the observable is defined. ['strong' and 'weak'
  originated from Azure sentinel entities type]
  (https://docs.microsoft.com/en-us/azure/sentinel/entities-reference#security-group)
• The "type" field in the observable will be the key from the schema.
• The "type" and "value" defines a unique ID, which prevents observables duplications.
• The "extraProperties" and "relatedExtraProperties" observable fields will be defined according to the relevant schema fields.

Algorithm Flow

• For each base event, create observable candidates (assets).
• Choose the main observable using the schema configuration. Iterate over the 'strong' list and search for values compatible with observable candidates of the 'type' value.
  If we do not find a match with the 'strong' list, iterate over the 'weak' list and search
  for values compatible with observable candidates of the 'type' value.
• If we do not find a match to the 'strong' or 'weak' list, observables will be extracted via
  the legacy method, according to the ms_sentinel_observable_method DataStore key.
  Note: We take the first value that the algorithm finds in the 'strong' or 'weak' list.
• Rest observables candidates (assets) will be part of the observable 'extraProperties'
  and 'relatedExtraProperties', according to the schema configuration.
  Note: If some observable candidate "type" is not compatible with the schema configuration, such that it is not compatible with 'extraProperties' and 'relatedExtraProperties' then observables, observables will be extracted via
  the legacy method, according to the ms_sentinel_observable_method DataStore key.

Schema Example:

Base Event Schema Configuration:
json { "config": { "ip": { "strong": [ "AadUserId" ], "weak": [ "LmPackageIP" ], "extra_prop": [ "^Event" ], "related_extra_prop": [ "Type" ] } } }

Base Event:

{
 "tables": [
 {
 "name": "PrimaryResult",
 "columns": [
 {
 "name": "LmPackageIP",
 "type": "string"
 },
 {
 "name": "EventSourceName",
 "type": "datetime"
 },
 {
 "name": "EventOriginId",
 "type": "string"
 },
 {
 "name": "EventID",
 "type": "integer"
 },
 {
 "name": "Type",
 "type": "string"
 },
 {
 "name": "AccountType",
 "type": "string"
 },
 {
 "name": "LogonTypeName",
 "type": "string"
 },
 {
 "name": "LogonType",
 "type": "integer"
 }
 ],
 "rows": [
 [
 "192.12.23.25",
 "Microsoft-Windows-Security-Auditing",
 "8b9ee84d-2b52-4432-b2e8-e86d11358a67",
 4625,
 "SecurityEvent",
 "User",
 "3 - Network",
 "3"
 ]
 ]
 }
 ]
}

Observables candidates after data manipulation engine (assets):

Asset(type="LmPackageIP",value="192.12.23.25"),
Asset(type="EventSourceName",value="Microsoft-Windows-Security-Auditing"),
Asset(type="EventOriginId",value="8b9ee84d-2b52-4432-b2e8-e86d11358a67"),
Asset(type="EventID",value=4625)
Asset(type="Type",value="SecurityEvent"),
Asset(type="AccountType",value="User"),
Asset(type="LogonTypeName",value="3 - Network")
Asset(type="LogonType",value="3")

Observable created:
json { "type": "ip", "value": "192.12.23.25", "extraProperties": { "EventSourceName": { "value": "Microsoft-Windows-Security-Auditing" }, "EventOriginId": { "value": "8b9ee84d-2b52-4432-b2e8-e86d11358a67" }, "EventID": { "value": 4625 } }, "relatedExtraProperties": { "Type": { "value": "SecurityEvent" }, "AccountType": { "value": "User" }, "LogonTypeName": { "value": "3 - Network" }, "LogonType": { "value": 3 } } }

Observable New Data Model Using Azure Blob

The new observable data model structure is an object that contains the following fields:
+ type
+ value
+ extraProperties (optional)
+ relatedExtraProperties (optional)

Observable Example:

json { "type": "account", "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "extraProperties": { "Name": { "value": "ADMINISTRATOR" } }, "relatedExtraProperties": { "accountName": { "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699" }, "accountType": { "value": "account" } } }

Schema

Schema Overview

• The schema contains configuration for observables extraction, according to the new data model.
• The schema is saved in Azure Blob and in the Redis cache, in order to improve performance while fetching schema details.
• The schema is written in 'yaml' format.

Schema Structure:

`json
config:

• product: qradar
  type: base_events
  name: example
  strong: []
  weak: []
  extra_prop: []
  related_extra_prop: []`

Parameters Explanations:

• "product" - name of specific integration.
• "type" - will be "entities" or "base_events". This field indicates if the observable extraction is from base entities or base_events.
• "name" - used for observable type.
• Using "strong" and "weak" fields, the "value" of the observable is defined.
• "extraProperties" and "relatedExtraProperties" observable fields will be defined according to the relevant schema fields.

Schema Example:

`json
config:

• product: qradar
  type: base_events
  name: host
  strong:
• host_name
• host_address
  weak: []
  extra_prop:
• dest
  related_extra_prop:
• app
• product: sentinel
  type: entities
  name: host
  strong:
• id
• host_id
  weak: []
  extra_prop:
• host_address
  related_extra_prop:
• host_ip
• product: sentinel
  type: entities
  name: url
  strong:
• uri4
• uri_path
  weak: []
  extra_prop:
• dest
  related_extra_prop:
• app`

DataStore Keys

• ms_sentinel_observable_method - When True, extract observables via the old method. Otherwise, do not extract.
• ms_sentinel_entities_observable_method - When True, extract observables via the old method. Otherwise, do not extract.

Algorithm Flow

• Choose whether to extract observables using the observable new data model from Azure using the 'is_observable_new_data_model' checkbox in Pack UI. The default is to use the old method.
• Extract Sentinel records from Redis. If not found in Redis, search in Azure Blob.
  If the schema is not found in Azure Blob, continue observables extraction according to boolean DataStore key "ms_sentinel_observable_method" or "ms_sentinel_entities_observable_method", for
  entities or base events, respectively.
• Observable extraction engine will convert the schema to the schema described in Observable new
  data model - in order to achieve fast access to observables types, as the "name" is used for defining observables type.

Note:

Configuring and fetching schema in Azure Blob will be by using REPLACE/EXTEND/GET CLIs. Please refer to the Azure Storage Integration documentation for more information about setting and getting observables schema.

Known Issues

• Asynchronous actions do not remove defaults or None values while sending the data to the CDC. Currently, the AMQP service, which collects messages from RabbitMQ and pushes them to ST2, does not handle these cases, and fails to push these triggers to ST2. This should be fixed in new ucf-microservices releases.