Microsoft Azure Sentinel 11.1.2

Microsoft Azure Sentinel - 11.1.2

tags: python | SIEM

Table of Contents

• Description
• CDC Command Lines
• Workflows
• Rules
• Sensors
• Triggers
• Observable new data model
• Observable new data model using Azure Blob
• Known Issues

Description

Integration with Microsoft Azure Sentinel supports CDC users by providing the extraction of logs from Sentinel as alerts and observables. This enables CDC users to make informed decisions regarding incident response.

Sentinel is a scalable, cloud-native, Security Information and Event Management (SIEM) solution. It delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for attack detection, threat visibility, proactive hunting, and threat response.

CyberProof supports various observable models depending on configuration. These models help group observables under headings; otherwise, they are left as independent.

Microsoft Sentinel via KQL (Keyword Query Language) enable us to query against data collected and interactively analyze their results. We use KQL queries to retrieve records that match particular criteria, identify trends, analyze patterns, and provide a variety of insights into our data.

CDC Command Lines

• complete_alert_data_cli
  This CLI is used to complete alert information. Example: complete_alert_data_cli --alert_ids=["59889e23-0229-64fe-ec12-7058e1f9be87","45852sdfs3-0229-64fe-ec12-7058e1f9be87"] --alerts_limit=2 --base_events_limit=5. (Default limit is 10)

Workflows

• async_update_alert
  Update alerts in the CDC from updated Sentinel incidents.

• complete_alert_data
  This CLI is used to complete alert information. Example: complete_alert_data_cli --alert_ids=["59889e23-0229-64fe-ec12-7058e1f9be87","45852sdfs3-0229-64fe-ec12-7058e1f9be87"] --alerts_limit=2 --base_events_limit=5. (Default limit is 10)

• create_alert
  Create alerts in the CDC from Sentinel incidents.

• enrich_ala_incident
  Enrich Sentinel incidents by Azure Log Analytics.

• execute_close_incident
  Close an incident in Sentinel.

• fetch_ala_data
  This workflow is used to fetch an incident and its alerts from Azure Log Analytics.

• update_alert
  Update alerts in the CDC from updated Sentinel incidents.

• update_alert_wrapper
  Update alerts in the CDC from updated Sentinel incidents.

Rules

• close_alert_lisenter
  Close incidents; i.e., an incident in Sentinel.

• create_alert_lisenter
  Triggers injecting a new Azure Log Analytics incident to the CDC workflow.

• update_alert_lisenter
  Triggers injecting an updated Azure Log Analytics incident to the CDC workflow.

Sensors

• SentinelSensor
  Sensor to pull incidents from Azure Sentinel.

Poll interval - 30s

• SentinelUpdateSensor
  Sensor to pull updated incidents from Azure Sentinel.

Poll interval - 30s

Triggers

No triggers

Observable new data model

Extract observables from Alert Entities

• The new observable data model structure is an objec which contains the following fields:
  • type
  • value
  • extraProperties (optional)
  • relatedExtraProperties (optional)

Observable Example:

json { "type": "account", "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "extraProperties": { "Name": { "value": "ADMINISTRATOR" } }, "relatedExtraProperties": { "accountName": { "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699" }, "accountType": { "value": "account" } } }

Schema

DataStore keys

• The observable fields filled by the schema configuration via ms_sentinel_entities_schema
  DataStore key.
• ms_sentinel_entities_observable_method - When True or when it was not found - extract
  observables via old method.
  Otherwise, do not extract.

Schema Explanation

• The schema contains entity name as the key, and its configuration as the value.
• Using "strong" and "weak" fields the "value" of the observable is defined.
• "type" field in the observable will be the entity.
• "type" and "value" define unique id which prevent from observables duplications.
• "extraProperties" and "relatedExtraProperties" observable fields will be defined according to the
  relevant schema fields.

Algorithm Flow

• For each alert, extract Entities.
• For each entity -> generate observables candidates (assets).
• Choose the main observable using the schema configuration. Iterate over 'strong' list and search
  for value compatible with observable candidates 'type' value.
  If we didn't find a match with 'strong' list - Iterate over 'weak' list and search
  for value compatible with observable candidates 'type' value.
  Note: we are taking the first value that the algorithm found in 'strong' or 'weak' list.
• If we didn't find match to 'strong' list or to 'weak' list, observables will be extracted via
  legacy method - according to ms_sentinel_entities_observable_method DataStore key.
• Rest observables candidates (assets) will be part of the observable 'extraProperties'
  and 'relatedExtraProperties' according to schema configuration.
  Note: If some observable candidate "type" is not compatible with schema configuration, such that
  it's not compatible with 'extraProperties' and 'relatedExtraProperties' then observables will be
  set as 'relatedExtraProperties' automatically.

Schema example:

Entities Schema configuration:

json { "config": { "account": { "strong": [ "AadUserId" ], "weak": [ "id" ], "extra_prop": [ "Name" ], "related_extra_prop": [ "accountName", "accountType" ] } } }

Entity:

json { "id": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "Name": "name_test", "accountName": "USER", "accountType": "account" }

Observables candidates after data manipulation engine(assets):

Asset(type="id",value="0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699"),
Asset(type="Name",value="name_test"),
Asset(type="accountName",value="USER"),
Asset(type="accountType",value=account)

Observable Created:
json { "type": "account", "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "extraProperties": { "Name": { "value": "name_test" } }, "relatedExtraProperties": { "accountName": { "value": "USER" }, "accountType": { "value": "account" } } }

Extract observables from Base Events

• The new observable data model structure is an objec which contains the following fields:
  • type
  • value
  • extraProperties (optional)
  • relatedExtraProperties (optional)

Observable Example:

json { "type": "ip", "value": "192.12.23.25", "extraProperties": { "EventSourceName": { "value": "Microsoft-Windows-Security-Auditing" }, "EventOriginId": { "value": "8b9ee84d-2b52-4432-b2e8-e86d11358a67" }, "EventID": { "value": 4625 } }, "relatedExtraProperties": { "Type": { "value": "SecurityEvent" }, "AccountType": { "value": "User" }, "LogonTypeName": { "value": "3 - Network" }, "LogonType": { "value": 3 } } }

Schema

DataStore keys

• The observable fields filled by the schema configuration via ms_sentinel_schema
  DataStore key.
• ms_sentinel_observable_method - When True or when key is not defined - extract observables via
  old method. Otherwise, do not extract.

Schema Explanation

• The schema contains the final observable type as key, and its configuration as the value.
• "strong", "weak", "extraProperties" and "relatedExtraProperties" are defined via regex.
• Using "strong" and "weak" fields the "value" of the observable is defined. ['strong' and 'weak'
  originated from Azure sentinel entities type]
  (https://docs.microsoft.com/en-us/azure/sentinel/entities-reference#security-group)
• "type" field in the observable will be the key from the schema.
• "type" and "value" define unique id which prevent from observables duplications.
• "extraProperties" and "relatedExtraProperties" observable fields will be defined according to the
  relevant schema fields.

Algorithm Flow

• For each base event, create observable candidates (assets).
• Choose the main observable using the schema configuration. Iterate over 'strong' list and search
  for value compatible with observable candidates 'type' value.
  If we didn't find a match with 'strong' list - Iterate over 'weak' list and search
  for value compatible with observable candidates 'type' value.
• If we didn't find match to 'strong' list or to 'weak' list, observables will be extracted via
  legacy method - according to ms_sentinel_observable_method DataStore key.
  Note: we are taking the first value that the algorithm found in 'strong' or 'weak' list.
• Rest observables candidates (assets) will be part of the observable 'extraProperties'
  and 'relatedExtraProperties' according to schema configuration.
  Note: If some observable candidate "type" is not compatible with schema configuration, such that
  it's not compatible with 'extraProperties' and 'relatedExtraProperties' then observables, observables will be extracted via
  legacy method - according to ms_sentinel_observable_method DataStore key.

Schema example:

Base Event Schema Configuration:
json { "config": { "ip": { "strong": [ "AadUserId" ], "weak": [ "LmPackageIP" ], "extra_prop": [ "^Event" ], "related_extra_prop": [ "Type" ] } } }

Base Event:

{
 "tables": [
 {
 "name": "PrimaryResult",
 "columns": [
 {
 "name": "LmPackageIP",
 "type": "string"
 },
 {
 "name": "EventSourceName",
 "type": "datetime"
 },
 {
 "name": "EventOriginId",
 "type": "string"
 },
 {
 "name": "EventID",
 "type": "integer"
 },
 {
 "name": "Type",
 "type": "string"
 },
 {
 "name": "AccountType",
 "type": "string"
 },
 {
 "name": "LogonTypeName",
 "type": "string"
 },
 {
 "name": "LogonType",
 "type": "integer"
 }
 ],
 "rows": [
 [
 "192.12.23.25",
 "Microsoft-Windows-Security-Auditing",
 "8b9ee84d-2b52-4432-b2e8-e86d11358a67",
 4625,
 "SecurityEvent",
 "User",
 "3 - Network",
 "3"
 ]
 ]
 }
 ]
}

Observables candidates after data manipulation engine(assets):

Asset(type="LmPackageIP",value="192.12.23.25"),
Asset(type="EventSourceName",value="Microsoft-Windows-Security-Auditing"),
Asset(type="EventOriginId",value="8b9ee84d-2b52-4432-b2e8-e86d11358a67"),
Asset(type="EventID",value=4625)
Asset(type="Type",value="SecurityEvent"),
Asset(type="AccountType",value="User"),
Asset(type="LogonTypeName",value="3 - Network")
Asset(type="LogonType",value="3")

Observable Created:
json { "type": "ip", "value": "192.12.23.25", "extraProperties": { "EventSourceName": { "value": "Microsoft-Windows-Security-Auditing" }, "EventOriginId": { "value": "8b9ee84d-2b52-4432-b2e8-e86d11358a67" }, "EventID": { "value": 4625 } }, "relatedExtraProperties": { "Type": { "value": "SecurityEvent" }, "AccountType": { "value": "User" }, "LogonTypeName": { "value": "3 - Network" }, "LogonType": { "value": 3 } } }

Observable new data model using Azure Blob

• The new observable data model structure is an objec which contains the following fields:
  • type
  • value
  • extraProperties (optional)
  • relatedExtraProperties (optional)

Observable Example:

json { "type": "account", "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699", "extraProperties": { "Name": { "value": "ADMINISTRATOR" } }, "relatedExtraProperties": { "accountName": { "value": "0ed0f0f7-ba9e-4dca-a14e-2d8d2d0c8699" }, "accountType": { "value": "account" } } }

Schema

Schema Overview

• The schema contains configuration for observables extraction according to new data model
• The schema is saved in Azure blob and in Redis cache in order to improve performance while
  fetching schema details.
• The schema is written in 'yaml' format.

Schema structure:

`json
config:

• product: qradar
  type: base_events
  name: example
  strong: []
  weak: []
  extra_prop: []
  related_extra_prop: []`

Parameters Explanations:

• "product" - name of specific integration.
• "type" - will be "entities" or "base_events". This field indicates if the observable extraction is
  from base entities or base_events.
• "name" - used for observable type.
• Using "strong" and "weak" fields, the "value" of the observable is defined.
• "extraProperties" and "relatedExtraProperties" observable fields will be defined according to the
  relevant schema fields.

Schema example:

`json
config:

• product: qradar
  type: base_events
  name: host
  strong:
• host_name
• host_address
  weak: []
  extra_prop:
• dest
  related_extra_prop:
• app
• product: sentinel
  type: entities
  name: host
  strong:
• id
• host_id
  weak: []
  extra_prop:
• host_address
  related_extra_prop:
• host_ip
• product: sentinel
  type: entities
  name: url
  strong:
• uri4
• uri_path
  weak: []
  extra_prop:
• dest
  related_extra_prop:
• app`

DataStore keys

• ms_sentinel_observable_method - When True - extract observables via old method.
  Otherwise, do not extract.
• ms_sentinel_entities_observable_method - When True - extract observables via old method.
  Otherwise, do not extract.

Algorithm Flow

• Choose if to extract observables using observable new data model from Azure using 'is_observable_new_data_model' checkbox in
  Pack UI. Default is using old method.
• Extract sentinel records from Redis, if not found in redis - search in Azure blob.
  If schema is not found in Azure blob, continue observables extraction according to boolean
  DataStore key "ms_sentinel_observable_method" or "ms_sentinel_entities_observable_method" for
  entities or base events respectively.
• Observable extraction engine will convert the schema to the schema describes in Observable new
  data model - in order to achieve fast access to observables types, as
  the "name" is used for defining observables type.

Note:

Configuring and fetching schema in Azure blob will be using REPLACE/EXTEND/GET CLIs. Please refer to
Azure storage integration documentation for more information about setting and getting observables
schema.

Known Issues

• Asynchronous actions do not remove defaults or None values while sending the data to CDC.
  Currently, AMQP service which collect messages from RabbitMQ and push them to ST2, does not
  handle these cases, and fails to push these triggers to ST2.
  It should be fixed in new ucf-microservices releases.