Azure: Send Linux VM Logs to an Event Hub

Why do I need this?

Azure Monitor is a comprehensive solution for storing and analyzing telemetry data from resources in your Azure environment. It has several features to monitor Linux VM metrics and logs, but the richer functionality requires storing log data in a Log Analytics Workspace.

Depending on your organization’s existing logging solutions, risk profile, and future in the cloud, you may require logs to be sent to an external tool such as Splunk or LogRhythm instead. When this is the case, most vendors offer a simple agent to install on the VM that simply connects back to the logging mothership. However, if networking restrictions do not allow the agent to communicate directly with the central logging tool, logs can be sent to an Event Hub for retrieval instead. All the VM needs is permission to reach an HTTP endpoint!

Great, how do I do it?

The easiest way to accomplish this is to install the Azure Diagnostics Extension on the VM and configure it to output log data to an Event Hub sink. A colleague sent me an ARM template to accomplish this on a Windows VM, however, the Linux implementation was slightly different and I want to highlight the nuances in this blog post.

Authentication

The primary difference between configuring the Windows and Linux diagnostic agents is authenticating to the Event Hub. The Windows version uses an Event Hub’s Shared Access Policy Name and its corresponding Key.

The Linux version is more complex and requires a full SAS URL. In Azure, SAS translates to Shared Access Signature. A SAS URL is a special URL that includes a SAS token. Look out, these tokens expire! Make sure you plan appropriate expiration times or generate new tokens as part of a CI/CD pipeline as needed.

If you have never created a SAS URL before or if you’re not a developer and simply need to get logs off of a VM and into an Event Hub, then this can get pretty confusing. This Microsoft documentation is comprehensive, but here is a simplified version of how an Event Hub SAS URL is built:

First, a SAS token is generated:

sig=<signature-string-xxxxxxxx>&se=<expiry>&skn=<keyName>&sr=<URL-encoded-resourceURI>

Each variable of the token following an & has a unique property:

sig – The signature (SHA-256 hash) of the token
se – The token’s expiration date in seconds since 00:00:00 UTC on 1/1/1970
skn – The name of the shared access policy
sr – The encoded URI of the resource being accessed

Next, it needs to be combined with an Event Hub URI:

An Event Hub URI without a token attached looks like this:

https://<Namespace>.servicebus.windows.net/<Event-Hub>/Messages

With the token attached, it should look something like this:

https://<Namespace>.servicebus.windows.net/<Event-Hub>?sr=namespace.servicebus.windows.net%2event-hub&sig=<signature-string-xxxxxxxx>&se=<expiry>skn=<keyName>

Notice that in the SAS URL, the “Messages” portion of the Event Hub URI is not included.

Creating these SAS URLs would be time consuming without a script, so I’ve created a PowerShell script that generates a token and combines it with an Event Hub URI.

How to use this scrip t:

Ensure that you have a namespace and at least one Event Hub created.
Create a Shared Access Policy for the Event Hub. The policy must have permission to Send. Save the policy name and key.

Add SAS Policy to Event Hub — Create the policy for the individual Event Hub, not the parent Namespace.

Add the event hub information in the script’s parameters and run it!

    # Event Hub Information:
    [string]       $EventHubNamespace = "yourNamespace",
    [string]       $EventHubName = "eventHub",
    [int]          $Expiration = 30000, # Token expires now+30000 seconds
    
    # Shared Access Policy Information:
    [string]       $AccessPolicyKey = "generatedKey",
    [string]       $AccessPolicyName = "sendPolicy"

Configuring the VM

Now that you have a SAS URL, it needs to be placed into the VM configuration. You can hard code it into your template, but you should really use a parameter for extensibility. Because the URL is somewhat sensitive, you should import it as a Key Vault secret. If these are new terms or you haven’t worked with ARM templates before, I would recommended that you review the template structure and syntax before proceeding.

Linux VMs can send logs and metrics, so having at least two Event Hubs will simplify data retrieval. For simplicity’s sake, the demo in this blog post will focus on setting up just one event hub for metrics. The process is exactly the same for a second (or third, fourth, fifth…) Event Hub and again, is also captured in the documentation.

Prerequisites for this demo:

An Event Hub namespace and an Event Hub for metrics
An Event Hub Shared Access Policy and Key with permission to “Send”
A SAS URL generated using this Event Hub, Shared Access Policy, and Key (Using the script above.)

First, download the ARM template of a Linux VM with the diagnostics extension enabled. If this is a brand new deployment, you can use the Azure portal to configure the VM and download a template at the final step before deployment. To activate the diagnostics extension, enable OS guest diagnostics on the VM:

Enable OS Guest Diagnostics and choose a storage account. — These settings are under the “Management” tab of the VM setup. Yes, you need a storage account, too.

Once you have the ARM template downloaded, it’s time to update the PrivateConfig and PublicConfig sections. Search for these two sections inside of the JSON file.

PrivateConfig

In the PrivateConfig section, under the storageAccountName and storageAccoutnSasToken parameters, add a section for sinksConfig:

"sinksConfig": {
    "sink": [
      {
        "name": "MetricsEventHub",
        "type": "EventHub",
        "sasURL": "https://<Namespace>.servicebus.windows.net/MetricsEventHub?sr=namespace.servicebus.windows.net%2metricseventhub&sig=<signature-string-xxxxxxxx>&se=<expiry>skn=<keyName>"
      }
    ]
  }

PublicConfig

In the PublicConfig section, add a sinks section inside of the performanceCounters:

    "performanceCounters": {
        "sinks": "MetricsEventHub",

To see the full context for these config sections, visit the documentation and search the page for “sink”

Deploy!

Visit your Event Hub resource page to ensure that messages are successfully being sent to your Event Hub.

Okay, how about a template?

Glad you asked. This GitHub link contains a template, parameters file, and a deploy script. For security’s sake, the template uses a parameter for the SAS URL, the parameters file references a Key Vault, and the SAS URL is saved as a secret inside of a Key Vault. The deploy script retrieves the secret from the Key Vault during deployment, so make sure your Key Vault Access Policies support this:

To close the loop on this process, here’s another PowerShell script to create a Shared Access Policy, generate a SAS URL and save it as a secret in a Key Vault.