Cloud Log Forwarding
This page contains instructions on how to forward logs from various cloud based log sources to BluSapphire. There may be minor differences on the data collected on various sources. Beware.
This page showcases multiple methods & instructions that guides you in setting up O365/Azure AD Integration and Configuration.
Before shipping logs from Azure Event Hubs to BluSapphire, you must stream the logs you want to be analyzed to Azure Event Hubs. Make sure to stream your logs to the same event hub.
- Azure Monitor collects logs for most Microsoft Azure services and streams the data to Azure Event Hubs.
- Azure Event Hub is a data streaming platform and event ingestion service for Microsoft Azure. Event Hub streams the logs collected by Azure Monitor.
- With further integration between BluSapphire SIEM and Azure Event Hubs, logs can be received and processed from an event hub.
Type of logs sent through Azure Event Hubs includes, but not limited to the following:
- Azure Active Directory (AD) logs, including audit logs and sign-in logs
- Azure Monitor logs
- Azure Application Gateway logs
- Azure SQL Database logs
- Microsoft Defender Advanced Threat Protection (ATP) logs
- Microsoft Intune logs
To configure streaming for the logs you want to collect into event hubs, please follow Microsoft documentation:
- 1.Azure AD logs: See instructions from Microsoft on streaming Azure Active Directory Logs to an Azure Event Hub.
- 2.Azure Monitor logs: See instructions from Microsoft on creating Diagnostic Settings to Send Logs, make sure to select Event Hub as the destination to stream logs to Event Hub.
- 3.Azure Application Gateway logs: See instructions from Microsoft on enabling Logging for Application Gateway, make sure to select Event Hub as the destination to stream logs to Event Hub.
- 4.Azure SQL Database logs: See instructions from Microsoft on setting up auditing for your database, make sure to select Event Hub as the destination.
- 5.Microsoft Defender ATP logs: See instructions from Microsoft on configuring Microsoft Defender ATP to stream Advanced Hunting events to your Azure Event Hubs.
- 6.Microsoft Intune logs: See instructions from Microsoft on sending log data to event hubs, in Intune.
As the document progresses, create an event hub using the Azure portal.
A resource group is a logical collection of Azure resources. All resources are deployed and managed in a resource group. To create a resource group:
1) Login to the Azure portal: https://portal.azure.com/
2) Click Resource groups in the left navigation pane and Click Create on Resource groups page.
3) For Subscription, select the name of the Azure subscription in which you want to create the resource group.
4) Enter a unique name for the resource group, The system immediately checks to see if the name is available in the currently selected Azure subscription.
5) Select a Region for the resource group. Click Review + Create. Click Create again on next page.
An Event Hub namespace provides a unique scoping container, referenced by its fully qualified domain name, in which you create one or more event hubs. To create a namespace in your resource group using the portal, complete the following steps:
1) In the Azure portal, click Create a resource at the top left of the screen.
2) Search for Event Hubs, and then click the Create button in the Event Hubs page.
3) On the Create namespace page, complete the following steps:
3.1 Select the subscription in which you want to create the namespace.
3.2 Select the Resource group for the namespace.
3.3 Enter a name for the namespace. The system immediately checks to see if the name is available.
3.4 Select a location for the namespace and choose the pricing tier “Standard" – required for additional, custom consumer groups other than $Default group.
3.5 Click Review + Create and then Create. This will take few minutes for the system to fully provision the resources.
3.6 Refresh the Event Hubs page to see the event hub namespace. You can check the status of the event hub creation in the alerts. Select the namespace. You see the home page for your Event Hubs Namespace in the portal.
To Create an Event Hub within the namespace, follow these steps:
1) In the Event Hubs Namespace page, click Event Hubs in the left menu.
2) At the top of the window, click + Event Hub.
3) Choose an appropriate name for your event hub, then click Create
4) After the event hub is created, you see it in the list of event hubs.
Note: This step is not required for all types of logs. Some log sources automatically create their own event hub like insights-logs-auditlogs, insights-logs-signinlogs, insights-activity- logs, others, within the namespace you created above. We recommend creating an Event Hub for sake of clarity.
To Generate Shared Access Policy for Event Hub Namespace you’ve created
1) Select event hub namespace and go to Shared access policies, and then click +Add. Enter the Policy name, check the Manage box, and then click Create.
2) Click on the Shared Access policies you just created, make a note of the connection string primary key field for later use.
3) BluSapphire collector requires the SAS Policy name (created in Step04 - 1) and the connection string primary key when creating the credential in “blusapphire-sas". Copy the connection string primary key and policy name to a text editor for later use.
Note: When the event hub namespace is created, Azure will also create a default Shared Access Policy named RootManageSharedAcessKey.
4) Select an event hub namespace and go to respective Event Hub.
5) From the selected event hub, go to Consumer group within that event hub. Click +Consumer group to create a new consumer group.
Note: Please ensure you select Standard tier while creating Event Hub Namespace, this allows creating multiple Consumer Groups.
Following information is to be shared and required to setup BluSapphire Collector: These details can be obtained from steps performed earlier from Step01 to Step04
Required Information | Description |
|---|---|
Event Hub Namespace | Name of the Azure event hub namespace used while creating |
Event Hub Name | Name of the Azure event hub created |
Policy | Shared Access (SAS) Policy Name created |
Primary Key | Name of the primary key |
Connection String Primary Key | Connection String of Primary Key |
Consumer Group | Name (list) of all the newly created consumer groups |
To configure streaming for the logs you want to collect into event hubs, please follow Microsoft documentation:
- 1.Azure AD logs: See instructions from Microsoft on streaming Azure Active Directory Logs to an Azure Event Hub.
- 2.Azure Monitor logs: See instructions from Microsoft on creating Diagnostic Settings to Send Logs, make sure to select Event Hub as the destination to stream logs to Event Hub.
- 3.Azure Application Gateway logs: See instructions from Microsoft on enabling Logging for Application Gateway, make sure to select Event Hub as the destination to stream logs to Event Hub.
- 4.Azure SQL Database logs: See instructions from Microsoft on setting up auditing for your database, make sure to select Event Hub as the destination.
- 5.Microsoft Defender ATP logs: See instructions from Microsoft on configuring Microsoft Defender ATP to stream Advanced Hunting events to your Azure Event Hubs.
- 6.Microsoft Intune logs: See instructions from Microsoft on sending log data to event hubs, in Intune.
Note: While configuring each of the above log stream, make sure you choose “Stream to an event-hub” in the “destination details” and select the correct Subscription, Event Hub Namespace, Event Hub, Policy name.
Subscription of "Azure Event Hub" in Azure cloud, subscribe to all the event-types that is needed to be ingested into BluSapphire platform.
Create an event hub using Azure portal
Azure Event Hubs is a Big Data streaming platform and event ingestion service, capable of receiving and processing millions of events per second. Event Hubs can process, and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/ storage adapters.
As the document progresses, create an event hub using the Azure portal.
Create a resource group
A resource group is a logical collection of Azure resources. All resources are deployed and managed in a resource group. To create a resource group:
1. Sign in to the Azure portal.
2. In the left navigation, click Resource groups. Then click Add.
3. For Subscription, select the name of the Azure subscription in which you want to create the resource group.
4. Type a unique name for the resource group. The system immediately checks to see if the name is available in the currently selected Azure subscription.
5. Select a region for the resource group.
6. Select Review + Create.
7. On the Review + Create page, select Create.
Create an Event Hubs namespace
An Event Hubs namespace provides a unique scoping container, referenced by its fully qualified domain name, in which you create one or more event hubs. To create a namespace in your resource group using the portal, do the following actions:
1) In the Azure portal and click Create a resource at the top left of the screen.
2) Select All services in the left menu and select star (*) next to Event Hubs in the Analytics category. Confirm that Event Hubs is added to FAVORITES in the left navigational menu.
3) Select Event Hubs under FAVORITES in the left navigational menu, and select Add on the toolbar.
4) On the Create namespace page, take the following steps:
5) Select the subscription in which you want to create the namespace.
6) Select the resource group you created in the previous step.
7) Enter a name for the namespace. The system immediately checks to see if the name is available.
8) Select a location for the namespace.
9) Choose the pricing tier (Basic or Standard).
10) Leave the throughput units settings as it is. To learn about throughput units, see Event Hubs scalability.
11) Select Review + Create at the bottom of the page.
12) On the Review + Create page, review the settings, and select Create. Wait for the deployment to complete.
13) On the Deployment page, select Go to resource to navigate to the page for your namespace.
14) Confirm that you see the Event Hubs Namespace page similar to the following example:
Create an event hub
To create an event hub within the namespace, do the following actions:
1) On the Event Hubs Namespace page, select Event Hubs in the left menu.
2) At the top of the window, click + Event Hub.
3) Type a name for your event hub, then click Create.
4) You can check the status of the event hub creation in alerts. After the event hub is created, you see it in the list of event hubs as shown in the following image:
Enabling Log Service onto Event Hubs:
Azure offers multiple subscription opportunities based on the business requirements.
O365 component provides multiple logging capabilities such as Audit logs, Message tracing, SharePoint logs, Etc. If a log source is to be onboarded onto BluSapphire, the log source to be enabled over event hubs before forwarding.
Customer/ Partner to decide on what events should be consumed by the SIEM, since within Office365 suite, there are multiple event-types which may be consumed, few examples to quote are audit logs, sign-in logs, metrics Etc.
Once the customer decides on the event-types which are to be ingested, customer needs to connect those event-types to the EventHub according to the instructions.
Let us take an example of onboarding Azure AD logs via Event Hubs:
Prerequisites
- An Azure AD tenant.
- A user who is a global administrator or security administrator for the Azure AD tenant.
- An Event Hubs namespace and an event hub in your Azure subscription. Stream logs to an event hub
1. Sign in to the Azure portal.
2. Select Azure Active Directory > Monitoring > Audit logs.
3. Select Export Settings.
4. In the Diagnostics settings pane, do either of the following:
a) To change existing settings, select Edit setting.
b) To add new settings, select Add diagnostics setting.
c) You can have up to three settings.
Select the Stream to an event hub check box, and then select Event Hub/Configure.
5) Select the Azure subscription and Event Hubs namespace that you want to route the logs to. The subscription and Event Hubs namespace must both be associated with the Azure AD tenant that the logs stream from. You can also specify an event hub within the Event Hubs namespace to which logs should be sent. If no event hub is specified, an event hub is created in the namespace with the default name insights-logs-audit.
6) Select OK to exit the event hub configuration.
7) Do either or both of the following:
a) To send audit logs to the event hub, select the AuditLogs check box.
b) To send sign-in logs to the event hub, select the SignInLogs check box.
8) Select Save to save the setting.
9) After about 15 minutes, verify that events are displayed in your event hub. To do so, go to the event hub from the portal and verify that the incoming messages count is greater than zero.
Access data from your event hub
At this point our collector can connect to the EventHub and pull the corresponding events. There is some access Key mojo which is need to be configured and do before our collector will get access to the EventHub but that is based on the external address of the EventHub and we can instruct the customer in that regards.
This section provides steps on configuring Microsoft O365 for shipping log, following data can be collected and audited.
- Azure Active Directory
- Exchange
- SharePoint
- DLP
- General (Other Workloads)
- An active subscription for Microsoft 365 with administrative privileges
- Machine with Filebeat (version 8 or greater) installed, to ship logs to BluSapphire Gateway Collector, preferably a windows machine.
1) Navigate to portal.azure.com and sign-in with an administrative account.
2) Open the pop-out menu from the top left corner of the web page and click on Azure Active Directory.
3) Under the Manage section on the left-hand side of the page, select App registrations > New registration
4) Provide Application Name, and then adjust options as necessary. The default settings should be fine if working with a single M365 tenant without a URI proxy
Note: Take note of the Application (client) ID and the Directory (tenant) ID. These will be used later while configuring Filebeat O365 Service Module.
1) From newly created application overview page, navigate to Certificates & secrets. A client secret will be needed for your Filebeat configuration in order to pull data from the management API.
2) Select + New client secret and provide a description of your choosing and the certificate expiry.
3) Make note of the Value of the new client secret, as this will only be shown once.
1) To allow interaction with the management API, configure app permissions by navigating to API Permissions on the left hand side of the page.
2) Select + Add a permission and scroll down to find the “Office 365 Management APIs” widget.
3) From the Application permissions tab, enable the ActivityFeed.Read, ActivityFeed.ReadDlp and ServiceHealth.Read permissions.
1) Move on to machine hosting filebeat service, and navigate to the modules.d directory, open the “o365.yml.disabled ” file.
2) Content of the file should be something like below, change the enabled parameter value to "true" and update "var.application_id, var.tenants, var.client_secret" parameter values with “Application ID, TenantID, TenantName (from Step01 – 4), & Client Secret (from Step02 – 3)” generated earlier.
// o365.yml
# Module: o365
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-module-o365.html
- module: o365
audit:
enabled: true
# Set the application_id (also known as client ID):
var.application_id: <<Application ID>> // Provide 'Application ID' created earlier
# Configure the tenants to monitor:
# Use the tenant ID (also known as directory ID) and the domain name.
# var.tenants:
# - id: "tenant_id_1"
# name: "mydomain.onmicrosoft.com"
# - id: "tenant_id_2"
# name: "mycompany.com"
var.tenants:
- id: <<TenantID>> // Provide 'TenantID' created earlier
name: <<TenantName>> // Provide 'TenantName' created earlier
# List of content-types to fetch. By default all known content-types
# are retrieved:
# var.content_type:
# - "Audit.AzureActiveDirectory"
# - "Audit.Exchange"
# - "Audit.SharePoint"
# - "Audit.General"
# - "DLP.All"
# Use the following settings to enable certificate-based authentication:
#var.certificate: "C:\ProgramData\filebeat\COMODomainValidationSecureServerCA.crt"
# var.key: "/path/to/private_key.pem"
# var.key_passphrase: "myPrivateKeyPassword"
# Client-secret based authentication:
# Comment the following line if using certificate authentication.
var.client_secret: <<Client Secret>> // Provide 'ClientSecret' generated earlier
# Advanced settings, use with care:
# var.api:
# # Settings for custom endpoints:
# authentication_endpoint: "https://login.microsoftonline.us/"
# resource: "https://manage.office365.us"
#
# max_retention: 168h
# max_requests_per_minute: 2000
# poll_interval: 3m
3) Save “o365.yml.disabled” file and rename it as “o365.yml”
4) Open main Filebeat configuration file "filebeat.yml" and update “output – logstash – hosts” config parameter in "filebeat.yml" configuration file with BluSapphire Gateway Collector IP & Port (As shared by deployment team).
// Filebeat.yml
# Filebeat Configuration for O365 module
filebeat.config:
modules:
enabled: true
path: modules.d/*.yml
fields:
codec: plain
type: o365
fields_under_root: true
#For version 6.x and lower uncomment the line below and remove the line after it
#filebeat.registry_file: 'C:\ProgramData\Filebeat'
filebeat.registry.path: 'C:\ProgramData\filebeat'
#The following processors are to ensure compatibility with version 7
processors:
- rename:
fields:
- from: "agent"
to: "beat_agent"
ignore_missing: true
- rename:
fields:
- from: "log.file.path"
to: "source"
ignore_missing: true
############################# Output ##########################################
output:
logstash:
hosts: ["<<LISTENER-HOST>>:<PORT>"] // Provide BluSapphire Collector IP & PORT
5) Save the “filebeat.yml” configuration file, restart Filebeat service to load the new configuration.
Note: Having all the above configuration steps performed, and Filebeat service is running and It may take some time to get the logs populated in the BluSapphire portal.
What is required for BluSapphire?
- tenant_domain:
- client_secret
- client_id: aka application id (GUID)
- directory_id: aka tenant id (GUID)
the following content types will be pulled from the API
Kindly follow the URL for the required content types: https://docs.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-reference#working-with-the-office-365-management-activity-api
Permissions required on the following content type based on the choice of customer/ requirement.
- Audit.AzureActiveDirectory
- Audit.Exchange
- Audit.SharePoint
- Audit.General
When you create an application that needs access to secured services like the Office 365 Management APIs, you need to provide a way to let the service know if your application has rights to access it. The Office 365 Management APIs use Azure AD to provide authentication services that you can use to grant rights for your application to access them.
There are four key steps:
1) Register your application in Azure AD. To allow your application access to the Office 365 Management APIs, you need to register your application in Azure AD. This allows you to establish an identity for your application and specify the permission levels it needs to access the APIs.
2) Get Office 365 tenant admin consent. An Office 365 tenant admin must explicitly grant consent to allow your application to access their tenant data by means of the Office 365 Management APIs. The consent process is a browser-based experience that requires the tenant admin to sign into the Azure AD consent UI and review the access permissions that your application is requesting, and then either grant or deny the request. After consent is granted, the UI redirects the user back to your application with an authorization code in the URL. Your application makes a service-to-service call to Azure AD to exchange this authorization code for an access token, which contains information about both the tenant admin and your application. The tenant ID must be extracted from the access token and stored for future use.
3) Request access tokens from Azure AD. Using your application's credentials as configured in Azure AD, your application requests additional access tokens for a consented tenant on an ongoing basis, without the need for further tenant admin interaction. These access tokens are called app-only tokens because they do not include information about the tenant admin.
4) Call the Office 365 Management APIs. The app-only access tokens are passed to the Office 365 Management APIs to authenticate and authorize your application.
The following diagram shows the sequence of consent and access token requests.
Before you can access data through the Office 365 Management Activity API, you must enable unified audit logging for your Office 365 organization. You do this by turning on the Office 365 audit log. For instructions around enabling unified audit logging for your Office 365 organization kindly follow the link: https://docs.microsoft.com/en-us/microsoft-365/compliance/turn-audit-log-search-on-or-off?view=o365-worldwide
Note: Enabling unified audit logging isn't required if you're only using the Office 365 Service Communications API.
Register your application in Azure AD
The Office 365 Management APIs use Azure AD to provide secure authentication to Office 365 tenant data. To access the Office 365 Management APIs, you need to register your app in Azure AD, and as part of the configuration, you will specify the permission levels your app needs to access the APIs.
Utilize the Azure Management Portal to register your application in Azure AD
Post you have Microsoft tenant with required subscriptions in place, you can register your application in Azure AD.
1) Sign into the Azure management portal, using the credential of your Microsoft tenant that has the subscription to Office 365 you wish to use. You can also access the Azure Management Portal via a link that appears in the left navigation pane in the Office admin portal.
2) In the left navigation panel, choose Active Directory (1). Make sure the Directory tab (2) is selected, and then select the directory name (3).
3) On the directory page, select Applications. Azure AD displays a list of the applications currently installed in your tenancy.
4) Choose Add.
5) Select Add an application my organization is developing.
6) Enter the NAME of your application and specify the Type as WEB APPLICATION AND/OR WEB API.
7) Enter the appropriate App properties: o
- SIGN-ON URL. The URL where users can sign in and use your app. You can change this later as needed.
- APP ID URI. The URI used as a unique logical identifier for your app. The URI must be in a verified custom domain for an external user to grant your app access to their data in Windows Azure AD. For example, if your Microsoft tenant is contoso.onmicrosoft.com, the APP ID URI could be https://app.contoso.onmicrosoft.com.
8) Your app is now registered with Azure AD, and has been assigned a client ID. However, there are several important aspects of your app left to configure.
Configure your application properties in Azure AD
Now that your application is registered, there are several important properties you must specify that determine how your application functions within Azure AD and how tenant admins will grant consent to allow your application to access their data by using the Office 365 Management APIs.
For more information about Azure AD application configuration in general, kindly navigate the following link:
1) CLIENT ID. This value is automatically generated by Azure AD. Your application will use this value when requesting consent from tenant admins and when requesting app-only tokens from Azure AD.
2) APPLICATION IS MULTI-TENANT. This property must be set to YES to allow tenant admins to grant consent to your app to access their data by using the Office 365 Management APIs. If this property is set to NO, your application will only be able to access your own tenant's data.
3) REPLY URL. This is the URL that a tenant admin will be redirected to after granting consent to allow your application to access their data by using the Office 365 Management APIs. You can configure multiple reply URLs as needed. Azure automatically sets the first one to match the sign-on URL you specified when you created the application, but you can change this value as needed.
Please make sure to choose Save after making any changes to these properties.
Generate a new key for your application Keys, also known as client secrets, are used when exchanging an authorization code for an access token.
1) In the Azure Management Portal, select your application and choose Configure in the top menu. Scroll down to keys.
2) Select the duration for your key and choose Save.
3) Azure displays the app secret only after saving it. Select the Clipboard icon to copy the client secret to the Clipboard.
Important Note:
Azure only displays the client secret at the time you initially generate it. You cannot navigate back to this page and retrieve the client secret later.
Configure an X.509 certificate to enable service-to-service calls
An application that is running in the background, such as a daemon or service, can use client credentials to request app-only access tokens without repeatedly requesting consent from the tenant admin after initial consent is granted.
For more information kindly find the following link: https://docs.microsoft.com/en-us/previous-versions/azure/dn645543(v=azure.100)?redirectedfrom=MSDN
You must configure an X.509 certificate with your application to be used as client credentials when requesting app-only access tokens from Azure AD. There are two steps to the process:
1) Obtain an X.509 certificate. You can use a self-signed certificate, or a certificate issued by publicly trusted certificate authority.
2) Modify your application manifest to include the thumbprint and public key of your certificate.
The following instructions show you how to use the Visual Studio or Windows SDK makecert tool to generate a self-signed certificate and export the public key to a base64-encoded file.
From the command line, run the following:
makecert -r -pe -n "CN=MyCompanyName MyAppName Cert" -b 03/15/2015 -e 03/15/2017 -ss my -len 2048
Note: When you are generating the X.509 certificate, make sure the key length is at least 2048. Shorter key lengths are not accepted as valid keys.
1) Open the Certificates MMC snap-in and connect to your user account.
2) Find the new certificate in the Personal folder and export the public key to a base64-encoded file (for example, mycompanyname.cer). Your application will use this certificate to communicate with Azure AD, so make sure you retain access to the private key as well.
Note: You can use Windows PowerShell to extract the thumbprint and base64-encoded public key. Other platforms provide similar tools to retrieve properties of certificates.
3) From the Windows PowerShell prompt, type and run the following:
PowerShellCopy
$cer = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$cer.Import("mycer.cer")
$bin = $cer.GetRawCertData()
$base64Value = [System.Convert]::ToBase64String($bin)
$bin = $cer.GetCertHash()
$base64Thumbprint = [System.Convert]::ToBase64String($bin)
$keyid = [System.Guid]::NewGuid().ToString()
4) Store the values for $base64Thumbprint, $base64Value, and $keyid to be used when you update your application manifest in the next set of steps.
Using the values extracted from the certificate and the generated key ID, you must now update your application manifest in Azure AD.
5) In the Azure Management Portal, select your application and choose Configure in the top menu.
6) In the command bar, choose Manage manifest, and then choose Download Manifest.
7) Open the downloaded manifest for editing and replace the empty KeyCredentials property with the following JSON:
JSONCopy
"keyCredentials": [ { "customKeyIdentifier" : "$base64Thumbprint_from_above", "keyId": "$keyid_from_above", "type": "AsymmetricX509Cert", "usage": "Verify", "value": "$base64Value_from_above" } ],
"keyCredentials": [
{
"customKeyIdentifier" : "$base64Thumbprint_from_above",
"keyId": "$keyid_from_above",
"type": "AsymmetricX509Cert",
"usage": "Verify",
"value": "$base64Value_from_above"
}
],
Note: The KeyCredentials property is a collection, making it possible to upload multiple X.509 certificates for rollover scenarios or delete certificates for compromise scenarios.
8) Save your changes and upload the updated manifest by choosing Manage manifest in the command bar, choosing Upload manifest, browsing to your updated manifest file, and then selecting it.
Specify the permissions your app requires to access the Office 365 Management APIs
Finally, you need to specify exactly what permissions your app requires of the Office 365 Management APIs. To do so, you add access to the Office 365 Management APIs to your app, and then you specify the permission(s) you need. In the Azure Management Portal, select your application, and choose Configure in the top menu. Scroll down to permissions to other applications and choose Add application.
Select the Office 365 Management APIs (1) so that it appears in the Selected column (2), and then select the check mark in the lower right (3) to save your selection and return to the main configuration page for your application.
The Office Management APIs now appear in the list of applications to which your application requires permissions. Under both Application Permissions and Delegated Permissions, select the permissions your application requires. Refer to the specific API reference for more details about each permission.
Note: There are currently four unused permissions related to activity reports and threat intelligence that will be removed in the future. Do not select any of these permissions because they are unnecessary.
Choose Save to save the configuration.
Get Office 365 tenant admin consent
Now that your application is configured with the permissions it needs to use the Office 365 Management APIs, a tenant admin must explicitly grant your application these permissions in order to access their tenant's data by using the APIs. To grant consent, the tenant admin must sign into Azure AD by using the following specially constructed URL, where they can review your application's requested permissions. This step is not required when using the APIs to access data from your own tenant.
The redirect URL must match or be a sub-path under one of the Reply URLs configured for your application in Azure AD.
For example:
You can test the consent URL by pasting it into a browser and signing in using the credentials of an Office 365 admin for a tenant other than the tenant that you used to register the application. You will see the request to grant your application permission to use the Office Management APIs.
After choosing Accept, you are redirected to the specified page, and there will be a code in the query string.
For example:
Your application uses this authorization code to obtain an access token from Azure AD, from which the tenant ID can be extracted. After you have extracted and stored the tenant ID, you can obtain subsequent access tokens without requiring the tenant admin to sign in.
Request access tokens from Azure AD
There are two methods for requesting access tokens from Azure AD: The Authorization Code Grant Flow involves a tenant admin granting explicit consent, which returns an authorization code to your application. Your application then exchanges the authorization code for an access token. This method is required to obtain the initial consent that your application needs to access the tenant data by using the API, and this first access token is needed in order to obtain and store the tenant ID.
The Client Credentials Grant Flow allows your application to request subsequent access tokens as old ones expire, without requiring the tenant admin to sign in and explicitly grant consent. This method must be used for applications that run continuously in the background calling the APIs once the initial tenant admin consent has been granted.
Request an access token using the authorization code
After a tenant admin grants consent, your application receives an authorization code as a query string parameter when Azure AD redirects the tenant admin to your designated URL.
Your application makes an HTTP REST POST to Azure AD to exchange the authorization code for an access token. Because the tenant ID is not yet known, the POST will be to the "common" endpoint, which does not have the tenant ID embedded in the URL:
The body of the POST contains the following:
JSONCopy
resource=https%3A%2F%2Fmanage.office.com&client_id=a6099727-6b7b-482c-b509-1df309acc563 &redirect_uri= http%3A%2F%2Fwww.mycompany.com%2Fmyapp%2F &client_secret={your_client_key}&grant_type=authorization_code&code= AAABAAAAvPM1KaPlrEqdFSB...
Sample request
JSON
Content-Type: application/x-www-form-urlencoded
Host: login.windows.net
Content-Length: 944
resource=https%3A%2F%2Fmanage.office.com&client_id=a6099727-6b7b-482c-b509-1df309acc563 &redirect_uri= http%3A%2F%2Fwww.mycompany.com%2Fmyapp%2F &client_secret={your_client_key}&grant_type=authorization_code&code=AAABAAAAvPM1KaPlrEqdFSB...
The body of the response will include several properties, including the access token.
Sample response
JSON
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 3265
{"expires_in":"3599","token_type":"Bearer","scope":"ActivityFeed.Read ActivityReports.Read ServiceHealth.Read","expires_on":"1438290275","not_before":"1438286375","resource":"https://manage.office.com","access_token":"eyJ0eX...","refresh_token":"AAABAAA...","id_token":"eyJ0eXAi..."}
The access token that is returned is a JWT token that includes information about both the admin that granted consent and the application requesting access. The following shows an example of an un-encoded token. Your application must extract the tenant ID "tid" from this token and store it so that it can be used to request additional access tokens as they expire, without further admin interaction.
Sample token
JSON:
{ "aud": "https://manage.office.com", "iss": "https://sts.windows.net/41463f53-8812-40f4-890f-865bf6e35190/", "iat": 1427246416, "nbf": 1427246416, "exp": 1427250316, "ver": "1.0", "tid": "41463f53-8812-40f4-890f-865bf6e35190", "amr": [ "pwd" ], "oid": "1cef1fdb-ff52-48c4-8e4e-dfb5ea83d357", "upn": "[email protected]", "puid": "1003BFFD8EC47CA6", "sub": "7XpD5OWAXM1OWmKiVKh1FOkKXV4N3OSRol6mz1pxxhU", "given_name": "John", "family_name": "Doe", "name": "Contoso, Inc.", "unique_name": "[email protected]", "appid": "a6099727-6b7b-482c-b509-1df309acc563", "appidacr": "1", "scp": "ActivityFeed.Read ServiceHealth.Read", "acr": "1" }
{
"aud": "https://manage.office.com",
"iss": "https://sts.windows.net/41463f53-8812-40f4-890f-865bf6e35190/",
"iat": 1427246416,
"nbf": 1427246416,
"exp": 1427250316,
"ver": "1.0",
"tid": "41463f53-8812-40f4-890f-865bf6e35190",
"amr": [
"pwd"
],
"oid": "1cef1fdb-ff52-48c4-8e4e-dfb5ea83d357",
"upn": "[email protected]",
"puid": "1003BFFD8EC47CA6",
"sub": "7XpD5OWAXM1OWmKiVKh1FOkKXV4N3OSRol6mz1pxxhU",
"given_name": "John",
"family_name": "Doe",
"name": "Contoso, Inc.",
"unique_name": "[email protected]",
"appid": "a6099727-6b7b-482c-b509-1df309acc563",
"appidacr": "1",
"scp": "ActivityFeed.Read ServiceHealth.Read",
"acr": "1"
}
Request an access token by using client credentials
After the tenant ID is known, your application can make service-to-service calls to Azure AD to request additional access tokens as they expire. These tokens include information only about the requesting application and not about the admin that originally granted consent. Service-to-service calls require that your application use an X.509 certificate to create client assertion in the form of a base64-encoded, SHA256 signed JWT bearer token. When you are developing your application in .NET, you can use the Azure AD Authentication Library (ADAL) to create client assertions. Other development platforms should have similar libraries. An un-encoded JWT token consists of a header and payload that have the following properties.
JSON
HEADER:
{ "alg": "RS256", "x5t": "{thumbprint of your X.509 certificate used to sign the token", } PAYLOAD: { "aud": "https://login.windows.net/{tenantid}/oauth2/token", "iss": "{your app client ID}", "sub": "{your app client ID}" "jti": "{random GUID}", "nbf": {epoch time, before which the token is not valid}, "exp": {epoch time, after which the token is not valid}, }
{
"alg": "RS256",
"x5t": "{thumbprint of your X.509 certificate used to sign the token",
}
PAYLOAD:
{
"aud": "https://login.windows.net/{tenantid}/oauth2/token",
"iss": "{your app client ID}",
"sub": "{your app client ID}"
"jti": "{random GUID}",
"nbf": {epoch time, before which the token is not valid},
"exp": {epoch time, after which the token is not valid},
}
SAMPLE JWT TOKEN:
HEADER:
{ "alg": "RS256", "x5t": "YyfshJC3rPQ-kpGo5dUaiY5t3iU", } PAYLOAD: { "aud": "https://login.windows.net/41463f53-8812-40f4-890f-865bf6e35190/oauth2/token", "iss": "a6099727-6b7b-482c-b509-1df309acc563", "sub": "a6099727-6b7b-482c-b509-1df309acc563" "jti": "0ce254c4-81b1-4a2e-8436-9a8c3b49dfb9", "nbf": 1427248048, "exp": 1427248648, }
{
"alg": "RS256",
"x5t": "YyfshJC3rPQ-kpGo5dUaiY5t3iU",
}
PAYLOAD:
{
"aud": "https://login.windows.net/41463f53-8812-40f4-890f-865bf6e35190/oauth2/token",
"iss": "a6099727-6b7b-482c-b509-1df309acc563",
"sub": "a6099727-6b7b-482c-b509-1df309acc563"
"jti": "0ce254c4-81b1-4a2e-8436-9a8c3b49dfb9",
"nbf": 1427248048,
"exp": 1427248648,
}
The client assertion is then passed to Azure AD as part of a service-to-service call to request an access token. When using client credentials to request an access token, use an HTTP POST to a tenant-specific endpoint, where the previously extracted and stored tenant ID is embedded in the URL.
HTTP:
The body of the POST contains the following:
JSONCopy
resource=https%3A%2F%2Fmanage.office.com&client_id={your_app_client_id}&grant_type=client_credentials&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&client_assertion={encoded_signed_JWT_token}
Sample request:
JSON
Content-Type: application/x-www-form-urlencoded
Host: login.windows.net
Content-Length: 994
resource=https%3A%2F%2Fmanage.office.com&client_id= a6099727-6b7b-482c-b509-1df309acc563&grant_type=client_credentials &client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&client_assertion=eyJhbGciOiJSUzI1NiIsIng1dCI6Ill5ZnNoSkMzclBRLWtwR281ZFVhaVk1dDNpVSJ9.eyJhdWQiOiJodHRwczpcL1wvbG9naW4ud2luZG93cy5uZXRcLzQxNDYzZjUzLTg4MTItNDBmNC04OTBmLTg2NWJmNmUzNTE5MFwvb2F1dGgyXC90b2tlbiIsImV4cCI6MTQyNzI0ODY0OCwiaXNzIjoiYTYwOTk3MjctNmI3Yi00ODJjLWI1MDktMWRmMzA5YWNjNTYzIiwianRpIjoiMGNlMjU0YzQtODFiMS00YTJlLTg0MzYtOWE4YzNiNDlkZmI5IiwibmJmIjoxNDI3MjQ4MDQ4LCJzdWIiOiJhNjA5OTcyNy02YjdiLTQ4MmMtYjUwOS0xZGYzMDlhY2M1NjMifQ.vfDrmCjiXgoj2JrTkwyOpr-NOeQTzlXQcGlKGNpLLe0oh4Zvjdcim5C7E0UbI3Z2yb9uKQdx9G7GeqS-gVc9kNV_XSSNP4wEQj3iYNKpf_JD2ikUVIWBkOg41BiTuknRJAYOMjiuBE2a6Wyk-vPCs_JMd7Sr-N3LiNZ-TjluuVzWHfok_HWz_wH8AzdoMF3S0HtrjNd9Ld5eI7MVMt4OTpRfh-Syofi7Ow0HN07nKT5FYeC_ThBpGiIoODnMQQtDA2tM7D3D6OlLQRgLfI8ir73PVXWL7V7Zj2RcOiooIeXx38dvuSwYreJYtdphmrDBZ2ehqtduzUZhaHL1iDvLlw
The response will be the same as before, but the token will not have the same properties, because it does not contain properties of the admin that granted consent.
Sample response
JSONCopy
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 1276
{"token_type":"Bearer","expires_in":"3599","expires_on":"1431659094","not_before":"1431655194","resource":"https://manage.office.com","access_token":"eyJ0eXAiOiJKV1QiL..."} Sample access token JSONCopy { "aud": "https://manage.office.com", "iss": "https://sts.windows.net/41463f53-8812-40f4-890f-865bf6e35190/", "iat": 1431655194, "nbf": 1431655194, "exp": 1431659094, "ver": "1.0", "tid": "41463f53-8812-40f4-890f-865bf6e35190", "roles": [ "ServiceHealth.Read", "ActivityFeed.Read" ], "oid": "67cb0334-e242-4783-8028-0f39132fb5ad", "sub": "67cb0334-e242-4783-8028-0f39132fb5ad", "idp": "https://sts.windows.net/41463f53-8812-40f4-890f-865bf6e35190/", "appid": "a6099727-6b7b-482c-b509-1df309acc563", "appidacr": "1" }
{
"token_type":"Bearer","expires_in":"3599","expires_on":"1431659094","not_before":"1431655194","resource":"https://manage.office.com","access_token":"eyJ0eXAiOiJKV1QiL..."}
Sample access token
JSONCopy
{
"aud": "https://manage.office.com",
"iss": "https://sts.windows.net/41463f53-8812-40f4-890f-865bf6e35190/",
"iat": 1431655194,
"nbf": 1431655194,
"exp": 1431659094,
"ver": "1.0",
"tid": "41463f53-8812-40f4-890f-865bf6e35190",
"roles": [
"ServiceHealth.Read",
"ActivityFeed.Read"
],
"oid": "67cb0334-e242-4783-8028-0f39132fb5ad",
"sub": "67cb0334-e242-4783-8028-0f39132fb5ad",
"idp": "https://sts.windows.net/41463f53-8812-40f4-890f-865bf6e35190/",
"appid": "a6099727-6b7b-482c-b509-1df309acc563",
"appidacr": "1"
}
Build your app
Now that you have registered your app in Azure AD and configured it with the necessary permissions, you are ready to build your app. The following are some of the key aspects to consider when designing and building your app:
The consent experiences to obtain consent from your customers, you must direct them in a browser to the Azure AD website, using the specially constructed URL described previously, and you must have a website to which Azure AD will redirect the admin once they grant consent. This website must extract the authorization code from the URL and use it to request an access token from which it can obtain the tenant ID.
Store the tenant ID in your system. This will be needed when requesting access tokens from Azure AD and when calling the Office Management APIs.
Managing access tokens. You will need a component that requests and manages access tokens as needed. If your app calls the APIs periodically, it can request tokens on demand, or if it calls the APIs continuously to retrieve data, it can request tokens at regular intervals (for example, every 45 minutes).
Implement a webhook listener as needed by the particular API you are using.
Data retrieval and storage. You will need a component that retrieves data for each tenant, either by using continuous polling or in response to webhook notifications, depending on the particular API you are using.
- 1.Umbrella logs are CSV formatted, compressed (gzip), and saved every ten minutes.
- 2.Umbrella reports are based on logged data. When the Log Management page displayed the upgrade option.a. Navigate to Admin > Log Management.b. Click Upgrade.
- 1.You can also configure logging so that logs are also stored to an Amazon S3 bucket either your own or one managed by Cisco.
- 2.By default, Umbrella saves your event data logs to Cisco's California location. however, you can change the location of the data warehouse at any time.
- 3.Logging to Amazon S3.
- 4.Give Full administrative access to Cisco Umbrella.
- 5.Enable Logging:-a. Navigate to Admin > Log Management and select Use a Cisco-managed Amazon S3 bucket.
b. Select a Region and a Retention Duration
c. Click Save and then Continue to confirm your settings
d. When activation is complete, the Amazon S3 Summary page appears.
e. Copy credentials from this page and store them in a safe place. This is the only time that the Access and Secret keys are made available to you. These keys are required to access your S3 bucket and download logs. If you lose these keys they must be regenerated.
f. Once keys are copied and safe, check Got it and then click Continue.
g. Summary page
h. Please share keys to BluSapphire.
Note: - if you need more information please find below link
- 1.Create a Duo account, and then contact Duo support to request Admin API access.
- 2.After you are granted the appropriate role, sign in to Duo Admin Panel, and then on the left panel, click Applications.
- 3.Click Protect an Application, and then locate the entry for Admin API in the applications list.
- 4.In the “DUO Admin API” Click Protect to configure the application.
5. Copy the keys – Once the “Duo Admin API” application is created, you’ll need to copy the hostname and key values.
6. In the Permissions section, select which permissions you want to grant to the Admin API application.
7. Click Save Changes.
8. Please share keys with BluSapphire.
Note: - if you need more information, please refer to the below link.
- 1.Log in to the Cisco AMP for Endpoints portal as an administrator.
- 2.Click Accounts > API Credentials.
- 3.In the API Credentials pane, click New API Credential.
- 4.In the Application name field, type a name, and then select Read & Write.
- 5.You must have read & write access to manage event streams on your Cisco AMP for Endpoints platform. If its to fetch, only read access is needed
- 6.Click Create.
- 7.From the API Key Details section, copy the values for the 3rd Party API Client ID and the API Key. You need these values to fetch logs.
- 1.Run the CLI command logconfig.
- 2.Select the option new.
- 3.Choose the log file type for this subscription, this will be "1" for IronPort Text Mail Logs, or any other log file type of your choice.
- 4.Enter the name for the log file.
- 5.Select the appropriate log level. Typically you would need to select "3" for Informational, or any other log level of your choice.
- 6.When prompted 'Choose the method to retrieve the logs', select "3" for SCP Push.
- 7.Enter in the IP address or DNS hostname to deliver the logs to.
- 8.Enter the port to connect to on the remote host.
- 9.Enter the directory on remote host to place logs.
- 10.Enter in a filename to use for log files.
- 11.Configure, if needed, system based unique identifiers like $hostname, $serialnumber to append to the log filename.
- 12.Set Maximum filesize before transferring.
- 13.Configure time-based rollover of the log files, if applicable.
- 14.When asked "Do you want to enable host key checking?", enter "Y".
- 15.You are then presented the "Please place the following SSH key(s) into your authorized_keys file so that the log files may be uploaded."
- 16.Copy that key, as you will need to put the SSH key in your 'authorized_keys' file on the Syslog server. Paste the key given from logconfig to $HOME/.ssh/authorized_keys file on the Syslog server.
- 17.From the ESA, run the CLI command commit to save and commit configuration changes
Configure Syslog Server:
Step 1: Configure Syslog Server
Go to Logs & Reports > Configuration > Syslog Server and click Add to configure the syslog server according to parameters given below.
Enter server details.
Name
Enter a unique name for the syslog server.
IP Address / Domain
Specify the IP address (IPv4 / IPv6) or domain name of the syslog server. Logs from the device will be sent to the server.
Port
Specify the port number for communication with the syslog server. The device will send logs using the configured port.
Facility
Select syslog facility for logs to be sent to the syslog server.
Facility indicates to the syslog server the source of a log such as operating system, the process or an application. It is defined by the syslog protocol.
The device supports several syslog facilities for received log.
Available Options:
DAEMON
Daemon logs (information of services running in device as daemon).
KERNEL
Kernel log
LOCAL0 - LOCAL7
Log level information.
USER
Logging based on users who are connected to the server.
Severity Level
Specify severity levels of logs.
Severity level is the severity of the log that has been generated.
The device logs all the messages at and above the logging severity level you select. For example, select ERROR to log all messages tagged as ERROR, as well as any messages tagged with CRITICAL, ALERT and EMERGENCY and select DEBUG to log all messages.
The device supports following severity levels:
EMERGENCY - System is not usable
ALERT - Action must be taken immediately
CRITCAL - Critical condition
ERROR - Error condition
WARNING - Warning condition
NOTIFICATION - Normal but significant condition
INFORMATION - Informational
DEBUG - Debug level messages.
Format
The device produces logs in the specified format. The device currently produces logs in device standard format.
Click OK to save the configuration.
Note:
Repeat above steps if you want to add multiple syslog servers. Maximum five syslog servers can be added.
Step 2: Specify Logs to be Stored in Syslog Server
Go to Logs & Reports > Configuration > Log Settings and specify the kinds of logs to be recorded in the syslog server configured in Step 1. Check against the required types of logs.
The Falcon SIEM Connector provides users a turnkey, SIEM-consumable data stream. The Falcon SIEM Connector:
· Maintains the connection to the CrowdStrike Event Streaming API and your SIEM
· Manages the data-stream pointer to prevent data loss
Prerequisites:
Before using the Falcon SIEM Connector, you’ll want to first define the API client and set its scope. Refer to this guide to getting access to the CrowdStrike API for setting up a new API client key. For the new API client, make sure the scope includes read access for Event streams.
The CrowdStrike Falcon SIEM Connector (SIEM Connector) runs as a service on a local Linux server.
The resource requirements (CPU/Memory/Hard drive) are minimal and the system can be a VM.
· Supported OS (64-bit only):
o CentOS/RHEL 6.x-7.x
o Ubuntu 14.x
o Ubuntu 16.04
o Ubuntu 18.04
· Connectivity: Internet connectivity and ability to connect the CrowdStrike Cloud (HTTPS/TCP 443)
· Authorization: Crowdstrike API Event Streaming scope access
· Time: The date and time on the host running the Falcon SIEM Connector must be current (NTP is recommended)
Installation and Configuration:
To get started, you need to download the rpm install packages for the SIEM Connector from the CrowdStrike Falcon UI. For a more comprehensive guide, please visit the SIEM Connector Feature Guide
Download the package for your operating system to the Linux server you’d like to use.
Open a terminal and run the installation command where <installer package> is the installer that you had downloaded :
· CentOS:
sudo rpm -Uvh <installer package>
· Ubuntu:
sudo dpkg -i <installer package>
The last step before starting the SIEM Connector is to pick a configuration. There are a couple of decisions to make. The SIEM connector can: