Scribd
Upload
19 views4 pages

Create External Database Objects

This document discusses how to create external database objects in Azure Synapse Analytics serverless SQL pools to simplify querying data stored in Azure Data Lake Storage Gen2. It covers creating a database, external data sources, file formats, and external tables to provide a relational view of files in the data lake.

Uploaded by

Chandresh Padmani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
19 views4 pages

Create External Database Objects

This document discusses how to create external database objects in Azure Synapse Analytics serverless SQL pools to simplify querying data stored in Azure Data Lake Storage Gen2. It covers creating a database, external data sources, file formats, and external tables to provide a relational view of files in the data lake.

Uploaded by

Chandresh Padmani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

21/05/2024, 16:54 Create external database objects - Training | Microsoft Learn

Ｒ Previous Unit 4 of 7 Ｓ Next Ｔ

＂ 100 XP

Create external database objects


6 minutes

You can use the OPENROWSET function in SQL queries that run in the default master database
of the built-in serverless SQL pool to explore data in the data lake. However, sometimes you
may want to create a custom database that contains some objects that make it easier to work
with external data in the data lake that you need to query frequently.

Creating a database
You can create a database in a serverless SQL pool just as you would in a SQL Server instance.
You can use the graphical interface in Synapse Studio, or a CREATE DATABASE statement. One
consideration is to set the collation of your database so that it supports conversion of text data
in files to appropriate Transact-SQL data types.
The following example code creates a database named salesDB with a collation that makes it
easier to import UTF-8 encoded text data into VARCHAR columns.

SQL

CREATE DATABASE SalesDB


COLLATE Latin1_General_100_BIN2_UTF8

Creating an external data source


You can use the OPENROWSET function with a BULK path to query file data from your own
database, just as you can in the master database; but if you plan to query data in the same
location frequently, it's more efficient to define an external data source that references that
location. For example, the following code creates a data source named files for the
hypothetical https://mydatalake.blob.core.windows.net/data/files/ folder:

SQL

CREATE EXTERNAL DATA SOURCE files


WITH (
LOCATION = 'https://mydatalake.blob.core.windows.net/data/files/'
)

https://learn.microsoft.com/en-us/training/modules/query-data-lake-using-azure-synapse-serverless-sql-pools/4-external-objects 1/4
21/05/2024, 16:54 Create external database objects - Training | Microsoft Learn

One benefit of an external data source, is that you can simplify an OPENROWSET query to use
the combination of the data source and the relative path to the folders or files you want to
query:

SQL

SELECT *
FROM
OPENROWSET(
BULK 'orders/*.csv',
DATA_SOURCE = 'files',
FORMAT = 'csv',
PARSER_VERSION = '2.0'
) AS orders

In this example, the BULK parameter is used to specify the relative path for all .csv files in the
orders folder, which is a subfolder of the files folder referenced by the data source.

Another benefit of using a data source is that you can assign a credential for the data source
to use when accessing the underlying storage, enabling you to provide access to data through
SQL without permitting users to access the data directly in the storage account. For example,
the following code creates a credential that uses a shared access signature (SAS) to
authenticate against the underlying Azure storage account hosting the data lake.

SQL

CREATE DATABASE SCOPED CREDENTIAL sqlcred


WITH
IDENTITY='SHARED ACCESS SIGNATURE',
SECRET = 'sv=xxx...';
GO

CREATE EXTERNAL DATA SOURCE secureFiles


WITH (
LOCATION = 'https://mydatalake.blob.core.windows.net/data/secureFiles/'
CREDENTIAL = sqlcred
);
GO

 Tip

In addition to SAS authentication, you can define credentials that use managed identity
(the Microsoft Entra identity used by your Azure Synapse workspace), a specific Microsoft
Entra principal, or passthrough authentication based on the identity of the user running
the query (which is the default type of authentication). To learn more about using
credentials in a serverless SQL pool, see the Control storage account access for
https://learn.microsoft.com/en-us/training/modules/query-data-lake-using-azure-synapse-serverless-sql-pools/4-external-objects 2/4
21/05/2024, 16:54 Create external database objects - Training | Microsoft Learn

serverless SQL pool in Azure Synapse Analytics article in Azure Synapse Analytics
documentation.

Creating an external file format


While an external data source simplifies the code needed to access files with the
OPENROWSET function, you still need to provide format details for the file being access; which
may include multiple settings for delimited text files. You can encapsulate these settings in an
external file format, like this:

SQL

CREATE EXTERNAL FILE FORMAT CsvFormat


WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS(
FIELD_TERMINATOR = ',',
STRING_DELIMITER = '"'
)
);
GO

After creating file formats for the specific data files you need to work with, you can use the file
format to create external tables, as discussed next.

Creating an external table


When you need to perform a lot of analysis or reporting from files in the data lake, using the
OPENROWSET function can result in complex code that includes data sources and file paths. To
simplify access to the data, you can encapsulate the files in an external table; which users and
reporting applications can query using a standard SQL SELECT statement just like any other
database table. To create an external table, use the CREATE EXTERNAL TABLE statement,
specifying the column schema as for a standard table, and including a WITH clause specifying
the external data source, relative path, and external file format for your data.

SQL

CREATE EXTERNAL TABLE dbo.products


(
product_id INT,
product_name VARCHAR(20),
list_price DECIMAL(5,2)
)
WITH

https://learn.microsoft.com/en-us/training/modules/query-data-lake-using-azure-synapse-serverless-sql-pools/4-external-objects 3/4
21/05/2024, 16:54 Create external database objects - Training | Microsoft Learn

(
DATA_SOURCE = files,
LOCATION = 'products/*.csv',
FILE_FORMAT = CsvFormat
);
GO

-- query the table


SELECT * FROM dbo.products;

By creating a database that contains the external objects discussed in this unit, you can
provide a relational database layer over files in a data lake, making it easier for many data
analysts and reporting tools to access the data by using standard SQL query semantics.

Next unit: Exercise - Query files using a serverless SQL pool


Continue Ｔ

https://learn.microsoft.com/en-us/training/modules/query-data-lake-using-azure-synapse-serverless-sql-pools/4-external-objects 4/4

You might also like