Azure Event Hubs is a Big Data streaming platform and event ingestion service, capable of receiving and processing millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters.
The strategy used in this scenario is to have Azure Activity Log sending events to an Event Hub where a Logic App sends them to our Splunk HEC environment.
Advantages of this approach include:
Low latency since the Azure Activity Log is streamed into the Event Hub. The Logic App is then triggered and posts the data to the Splunk HEC.
Minimal code is required, and there is no server infrastructure to deploy.
This post describes the steps through how to:
- Step 1: Create an Event Hub.
- Step 2: Export activity logs to an Event Hub using Azure Activity Log export profile.
- Step 3: Create a Logic App to read from the Event Hub and send events to Splunk HEC.
Step 4: Test and Troubleshoot Logic App
Requirements
The user who configures the setting must have appropriate access permissions to the subscriptions (Logic App reads from the Event Hub using the Event Hub’s shared access key.)
Step 1 — Create an Event Hub
In the Azure portal, select Create a resource > type Event Hubs.
Under Create Namespace, enter a new namespace. The system immediately checks to see if the name is available:
Choose the “Standard” pricing tier, an Azure subscription, resource group, and location for the new resource. Click Create to create the namespace. You may have to wait a few minutes for the system to fully provision the resources.
Once deployment is succeeded click the namespace you just created from the notification window:
Select Shared access policies, and then click RootManageSharedAccessKey.
Click the copy button to copy the RootManageSharedAccessKey connection string to the clipboard:
In a temporary location, such as Notepad, keep a copy the Event Hub name and either the primary or secondary Event Hub connection string.
The Logic App requires these values. For the Event Hub connection string, you can use the RootManageSharedAccessKey connection string or create a separate one. The connection string you use must start with Endpoint=sb:// and be for a policy that has the Manage access policy
Step 2 — Export Activity Logs to Event Hub
To enable streaming of the Activity Log, you pick an Event Hub Namespace and a shared access policy for that namespace. An Event Hub is created in that namespace when the first new Activity Log event occurs.
The user who configures the setting must have the appropriate RBAC to access the subscriptions.
In the Azure portal, select the blade Monitor > Activity Log.
Click the Export to Event Hub button at the top of the page:
Select the Subscription to export from, and then click Select all in the Regions drop-down to select events for resources in all regions. Click the Export to an event hub check box.
Click Service bus namespace, and then select the Subscription with the event hub, the event hub namespace, and an event hub policy name.
Click OK and then Save to save these settings. The settings are immediately be applied to your subscription.
Step 3 — Create Logic App
Once the activity logs are writing to the event hub, you create a Logic App to collect the logs from the event hub and send them to the Splunk HEC.
The Logic App includes the following:
- An Event Hub connector trigger to read from the Event Hub.
- A Parse JSON action to extract the JSON events.
- A Compose action to convert the JSON to an object.
- An HTTP connector to post the data to the Splunk HEC.
Logic App Requirements
Before creating your Logic App, make sure you have the following information from previous steps:
- Event Hub name
- Event Hub connection string (either the primary or secondary) for the Event Hub namespace.
- Splunk HTTP Event Collector URL and Port #
- Splunk HEC Token
Create a new blank Logic App
In the Azure portal, choose Create a resource > type Logic App.
Enter the settings in the table below:
Select Create. When Deployment Succeeded notification displays, click on Go to resource to open your Logic App.
Under Templates, choose Blank Logic App.
The Logic Apps Designer now shows you available connectors and their triggers, which you use for starting your logic app workflow.
Add Event Hub trigger
In the search box for the Logic App Designer, type “event hubs” for your filter. Select the trigger Event Hubs — When events are available in Event Hub.
You can either browse the Event Hubs list of current subscription or manually enter connection information. If you choose the second one, enter a name for your connection and then the connection string that you already copied. Select Create.
After you create the connection, edit the settings for the trigger. Start by selecting insights-operational-logs from the Event Hub name drop-down. (This Event Hub name was created automatically when first event comes in)
Change Content type to application/json. Maximum events count to 175 and Interval to 1 minute.
Add Parse JSON action
The output from the Event Hub contains a JSON payload with an array of records. The Parse JSON action is used to extract just the array of records for sending to Log Analytics workspace.
Click New step > Choose an action
In the search box, type “parse json” for your filter. Select the action Data Operations — Parse JSON.
Click in the Content field and then select Body.
Copy and paste the schema text (in the file) into the Schema field. This schema matches the output from the Event Hub action:
{
“properties”: {
“body”: {
“properties”: {
“ContentData”: {
“type”: “string”
},
“Properties”: {
“properties”: {
“ProfileName”: {
“type”: “string”
},
“x-opt-enqueued-time”: {
“type”: “string”
},
“x-opt-offset”: {
“type”: “string”
},
“x-opt-sequence-number”: {
“type”: “number”
}
},
“type”: “object”
},
“SystemProperties”: {
“properties”: {
“EnqueuedTimeUtc”: {
“type”: “string”
},
“Offset”: {
“type”: “string”
},
“PartitionKey”: {},
“SequenceNumber”: {
“type”: “number”
}
},
“type”: “object”
}
},
“type”: “object”
},
“headers”: {
“properties”: {
“Cache-Control”: {
“type”: “string”
},
“Content-Length”: {
“type”: “string”
},
“Content-Type”: {
“type”: “string”
},
“Date”: {
“type”: “string”
},
“Expires”: {
“type”: “string”
},
“Location”: {
“type”: “string”
},
“Pragma”: {
“type”: “string”
},
“Retry-After”: {
“type”: “string”
},
“Timing-Allow-Origin”: {
“type”: “string”
},
“Transfer-Encoding”: {
“type”: “string”
},
“Vary”: {
“type”: “string”
},
“X-AspNet-Version”: {
“type”: “string”
},
“X-Powered-By”: {
“type”: “string”
},
“x-ms-request-id”: {
“type”: “string”
}
},
“type”: “object”
}
},
“type”: “object”
}
Note: You can get a sample payload by clicking Run and looking at the Raw Output from the Event Hub. You can then use this output with Use sample payload to generate schema in the Parse JSON activity to generate the schema also.
Add Compose action
The Compose action takes the JSON output and creates an object that can be used by the HTTP action.
Click New step > Add an action
Type compose for your filter and then select the action Data Operations — Compose.
Click the Inputs field and select Body under the Parse JSON activity.
Add HTTP action
The HTTP action takes the object from the Compose action and sends it to the Splunk HEC.
Click New step > Add an action
Type “http” for your filter and then select the action HTTP:
Enter the settings in the table below:
Click Save to save the changes you’ve made to your Logic App.
Step 4 — Test and troubleshoot the Logic App
With the workflow complete, you can test in the designer to verify that it’s working without error.
In the Logic App Designer, click Run to test the Logic App. Each step in the Logic App shows a status icon, with a white check mark in a green circle indicating success.
To see detailed information on each step, click on the step name to expand it. Click on Show raw inputs and Show raw outputs to see more information on the data received and sent at each step.
Enable High throughput
By last, go to the Logic App main page clicking on its name:
Select Settings > Workflow settings
Under Runtime options, enable the High throughput option for processing a large number of concurrent runs.
The default number of action executions that run within a single logic app run is 100000 over a 5-minute period. While this default limit seems like a large value, if you have a scenario where you ingested a batch of records and need to iterate over this batch, you may exhaust this threshold.
To avoid hitting this upper limit, Microsoft has provided a way to enable High Throughput from the Workflow settings menu.
With this setting enabled, a single logic app run can include 300000 action executions within a 5-minute period, providing 3 times more executions than the default value.
Happy Splunking!
