Data Engineering documentation in Microsoft Fabric

About Data Engineering

Overview

• What is Data engineering?
• What is a lakehouse?

Concept

• Lakehouse and Delta tables
• Workspace roles and permissions
• Choose a data store

How-To Guide

• Delta optimization and V-Order
• Load to tables

Get started with lakehouse

Get started

• Create a lakehouse
• Navigate the lakehouse explorer

How-To Guide

• Get data into lakehouse using notebook

Tutorial

• Step 1 - Create a workspace
• Step 2 - Create a lakehouse
• Step 3 - Ingest data into lakehouse
• Step 4 - Transform data into delta tables
• Step 5 - Build a report
• Step 6 - Clean up resources

Get started with notebooks

Get started

• Create and use notebooks

How-To Guide

• Develop and run notebooks
• Explore lakehouse data with a notebook
• Visualize notebooks
• Notebooks source control and deployment
• Author and run T-SQL Notebooks

Query with API for GraphQL

Overview

• What is Fabric API for GraphQL?

Get started

• Create an API for GraphQL and add data

How-To Guide

• Manage relationships
• Connect to applications
• Query multiple data sources
• GraphQL source control and deployment pipelines
• Monitoring dashboard and logging (preview)

VS Code integration

Concept

• What is Data Engineering VS Code extension?

How-To Guide

• Use notebooks from VS Code
• Use container image from VS Code
• Use Spark job definitions from VS Code
• Explore lakehouse from VS Code
• Develop notebook from VS Code web experience

Apache Spark for Data Engineering

Concept

• Apache Spark job definition
• What is Apache Spark advisor?

Get started

• Create a Spark job definition
• Create a Spark job definition with a Fabric API
• Spark job definition Git integration

How-To Guide

• Schedule and run a Spark job definition
• Manage Apache Spark libraries
• How to use Fabric MSSparkUtils

Get started with environment

Get started

• Microsoft Fabric environments

How-To Guide

• Manage Spark compute in environments
• Manage libraries in environments
• Environment Git integration and deployment pipeline
• Migrate libraries and Spark properties to an environment

Fabric Livy API

Overview

• What is the Livy API?

Get started

• Create and run Spark jobs using the Livy API
• Submit Livy API session jobs
• Submit Livy API batch jobs

Optimize with Materialized lake views

Overview

• What are materialized lake views?

Get started

• Get started with materialized lake views
• Implement Medallion architecture with materialized lake views
• Manage materialized lake views