Run notebooks on the Microsoft Sentinel data lake (preview)

Jupyter notebooks provide an interactive environment for exploring, analyzing, and visualizing data in the Microsoft Sentinel data lake. With notebooks, you can write and execute code, document your workflow, and view results—all in one place. This makes it easy to perform data exploration, build advanced analytics solutions, and share insights with others. By leveraging Python and Apache Spark within Visual Studio Code, notebooks help you transform raw security data into actionable intelligence.

This article shows you how to explore and interact with data lake data using Jupyter notebooks in Visual Studio Code.

Prerequisites

Onboard to the Microsoft Sentinel data lake

To use notebooks in the Microsoft Sentinel data lake, you must first onboard to the data lake. If you haven't onboarded to the Microsoft Sentinel data lake, see Onboarding to Microsoft Sentinel data lake. If you have recently onboarded to the data lake, it may take some time until sufficient volume of data is ingested before you can create meaningful analyses using notebooks.

Permissions

Microsoft Entra ID roles provide broad access across all workspaces in the data lake. Alternatively you can grant access to individual workspaces using Azure RBAC roles. Users with Azure RBAC permissions to Microsoft Sentinel workspaces can run notebooks against those workspaces in the data lake tier. For more information, see Roles and permissions in Microsoft Sentinel.

To create new custom tables in the analytics tier, the data lake managed identity must be assigned the Log Analytics Contributor role in the Log Analytics workspace.

To assign the role, follow the steps below:

1. In the Azure portal, navigate to the Log Analytics workspace that you want to assign the role to.
2. Select Access control (IAM) in the left navigation pane.
3. Select Add role assignment.
4. In the Role table, select Log Analytics Contributor, then select Next
5. Select Managed identity, then select Select members.
6. Your data lake managed identity is a system assigned managed identity named msg-resources-<guid>. Select the managed identity, then select Select.
7. Select Review and assign.

For more information on assigning roles to managed identities, see Assign Azure roles using the Azure portal.

Install Visual Studio Code and the Microsoft Sentinel extension

If you don't already have Visual Studio Code, download and install Visual Studio Code for Mac, Linux, or Windows.

The Microsoft Sentinel extension for Visual Studio Code (VS Code) is installed from the extensions marketplace. To install the extension, follow these steps:

1. Select the Extensions Marketplace in the left toolbar.
2. Search for Sentinel.
3. Select the Microsoft Sentinel extension and select Install.
4. After the extension is installed, the Microsoft Sentinel shield icon appears in the left toolbar.

Install the GitHub Copilot extension for Visual Studio Code to enable code completion and suggestions in notebooks.

1. Search for GitHub Copilot in the Extensions Marketplace and install it.
2. After installation, sign in to GitHub Copilot using your GitHub account.

Explore data lake tier tables

After installing the Microsoft Sentinel extension, you can start exploring data lake tier tables and creating Jupyter notebooks to analyze the data.

Sign in to the Microsoft Sentinel extension

1. Select the Microsoft Sentinel shield icon in the left toolbar.

2. A dialog appears with the following text The extension "Microsoft Sentinel" wants to sign in using Microsoft. Select Allow.

   

3. Select your account name to complete the sign in.

   

View data lake tables and jobs

Once you sign in, the Microsoft Sentinel extension displays a list of Lake tables and Jobs in the left pane. Select a table to see the column definitions.

For information on Jobs, see Jobs and Scheduling.

Create a new notebook

1. To create a new notebook, use one of the following methods.

2. Enter > in the search box or press Ctrl+Shift+P and then enter Create New Jupyter Notebook.

3. Select File > New File, then select Jupyter Notebook from the dropdown.
   

4. In the new notebook, paste the following code into the first cell.

   from sentinel_lake.providers import MicrosoftSentinelProvider
   data_provider = MicrosoftSentinelProvider(spark)
   
   table_name = "EntraGroups"  
   df = data_provider.read_table(table_name)  
   df.select("displayName", "groupTypes", "mail", "mailNickname", "description", "tenantId").show(100,   truncate=False)  
   

5. Select the Run triangle to execute the code in the notebook. The results are displayed in the output pane below the code cell.
   

6. Select Microsoft Sentinel from the list for a list of runtime pools.

7. Select Medium to run the notebook in the medium sized runtime pool. For more information on the different runtimes, see Selecting the appropriate Microsoft Sentinel runtime.

When the session is started, the code in the notebook runs and the results are displayed in the output pane below the code cell, for example:

For sample notebooks that demonstrate how to interact with the Microsoft Sentinel data lake, see Sample notebooks for Microsoft Sentinel data lake.

Use GitHub Copilot in notebooks

Use GitHub Copilot to help you write code in notebooks. GitHub Copilot provides code suggestions and autocompletion based on the context of your code. To use GitHub Copilot, ensure that you have the GitHub Copilot extension installed in Visual Studio Code.

Copy code from the Sample notebooks for Microsoft Sentinel data lake and save it in your notebooks folder to provide context for GitHub Copilot. GitHub Copilot will then be able to suggest code completions based on the context of your notebook.

The following example shows GitHub Copilot generating a code review.

Microsoft Sentinel Provider class

To connect to the Microsoft Sentinel data lake, use the SentinelLakeProvider class. This class is part of the access_module.data_loader module and provides methods to interact with the data lake. To use this class, import it and create an instance of the class using a spark session.

from sentinel_lake.providers import MicrosoftSentinelProvider
data_provider = MicrosoftSentinelProvider(spark)

For more information on the available methods, see Microsoft Sentinel Provider class reference.

Select the appropriate runtime pool

There are three runtime pools available to run your Jupyter notebooks in the Microsoft Sentinel extension. Each pool is designed for different workloads and performance requirements. The choice of runtime pool affects the performance, cost, and execution time of your Spark jobs.

View Logs

Logs can be viewed in the Output pane of Visual Studio Code.

1. In the Output pane, select Microsoft Sentinel from the drop-down.
2. Select Debug to include detailed log entries.

Jobs and scheduling

You can schedule jobs to run at specific times or intervals using the Microsoft Sentinel extension for Visual Studio Code. Jobs allow you to automate data processing tasks to summarize, transform, or analyze data in the Microsoft Sentinel data lake. Jobs are also used to process data and write results to custom tables in the data lake tier or analytics tier. For more information on creating and managing jobs, see Create and manage Jupyter notebook jobs.

Service parameters and limits for VS Code Notebooks

The following section lists the service parameters and limits for Microsoft Sentinel data lake (Preview) when using VS Code Notebooks.

Troubleshooting

The following table lists common errors you may encounter when working with notebooks, their root causes and suggested actions to resolve them.

Spark compute

VS Code Runtime

Interactive notebooks

Library

Jobs

Related content

• Create and manage notebook jobs
• Sample notebooks for Microsoft Sentinel data lake
• Microsoft Sentinel Provider class reference
• Microsoft Sentinel data lake overview
• Microsoft Sentinel data lake roles and permissions.