AG4 Constructing+a+polyglot+data+solution Ed2
AG4 Constructing+a+polyglot+data+solution Ed2
[Edition 02]
Atul Kumar
Oracle ACE & Cloud Expert
[email protected] 1
Contents
1 Introduction .................................................................................................................................................. 3
2 Documentation Links .................................................................................................................................. 4
3 Creating database resources in Azure.................................................................................................... 5
3.1 Create an Azure SQL Database server resource .......................................................................................... 5
3.2 Create an Azure Cosmos DB account resource............................................................................................. 9
3.3 Create a Storage account ..................................................................................................................................13
4 Import and validate data ........................................................................................................................ 16
4.1 Upload image blobs ...........................................................................................................................................16
4.2 Upload an SQL .bacpac file ...............................................................................................................................20
4.3 Import an SQL database ...................................................................................................................................23
4.4 Use an imported SQL database .......................................................................................................................26
5 Open and configure a .NET web application ...................................................................................... 33
5.1 Open and build the web application .............................................................................................................33
5.2 Update the SQL connection string .................................................................................................................35
5.3 Update the blob base URL ................................................................................................................................38
5.4 Validate the web application ..........................................................................................................................39
6 Migrating SQL data to Azure Cosmos DB ............................................................................................ 42
6.1 Create a migration project ...............................................................................................................................42
6.2 Create a .NET class .............................................................................................................................................46
6.3 Get SQL database records by using Entity Framework ............................................................................52
6.4 Insert items into Azure Cosmos DB ...............................................................................................................54
6.5 Perform a migration ..........................................................................................................................................57
6.6 Validate the migration ......................................................................................................................................59
7 Accessing Azure Cosmos DB by using .NET ........................................................................................ 63
7.1 Update library with the Cosmos SDK and references ..............................................................................63
7.2 Write .NET code to connect to Azure Cosmos DB ......................................................................................64
7.3 Update the Azure Cosmos DB connection string .......................................................................................75
7.4 Update .NET application startup logic ..........................................................................................................77
7.5 Validate that the .NET application successfully connects to Azure Cosmos DB ................................79
8 Summary ..................................................................................................................................................... 82
[email protected] 2
1 INTRODUCTION
[email protected] 3
2 DOCUMENTATION LINKS
1. Azure Cosmos DB: https://azure.microsoft.com/en-in/services/cosmos-db/
2. Azure Database Migration Service: https://docs.microsoft.com/en-us/azure/dms/
3. Azure SQL Database: https://docs.microsoft.com/en-us/azure/azure-sql/
[email protected] 4
3 CREATING DATABASE RESOURCES IN AZURE
In this section, we will create a Azure SQL Database server (logical server) resource, Azure
Cosmos DB account and a Storage account to use throughout the remainder of the lab.
[email protected] 5
4. From the Create SQL Database Server blade, observe the tabs from the blade, such as
ii. In the Resource group section, select Create new, enter PolyglotData(or any acc. To
iii. In the Server name text box, enter k21polysqlsrvr (or any name whichever is available)
v. In the Server admin login text box, enter testuser (or according to your choice)
vi. In the Password text box, enter TestPa55w.rd.(or according to your choice)
[email protected] 6
5. From the Networking tab, perform the following actions:
i. In the Allow Azure services and resources to access this server section, select
Yes.
[email protected] 7
6. From the Review + Create tab, review the options that you selected during the previous
steps.
7. Select Create to create the SQL Database server by using your specified configuration.
[email protected] 8
Note: At this point in the lab, we are only creating the Azure SQL logical server. We will create
[email protected] 9
4. From the Create Azure Cosmos DB Account blade, observe the tabs from the blade,
ii. In the Resource group section, select PolyglotData from the list.
vi. In the Location drop-down list, select the (US) East US region.
viii. In the Apply Free Tier Discount section, select Do Not Apply.
[email protected] 10
xi. In the Multi-region Writes section, select Disable.
8. From the Review + Create tab, review the options that you selected during the previous
steps.
9. Select Create to create the Azure Cosmos DB account by using your specified
configuration.
Note: Wait for the creation task to complete before you move forward with this lab.
10. In the Azure portal's navigation pane, select the Resource groups link.
11. From the Resource groups blade, find and then select the PolyglotData resource group
[email protected] 11
12. From the PolyglotData blade, select the k21polycosmos[yourname]** Azure Cosmos DB
13. From the Azure Cosmos DB account blade, find the Settings section from the blade, and
14. In the Keys pane, record the value in the PRIMARY CONNECTION STRING text box.
[email protected] 12
3.3 Create a Storage account
1. Sign-in to the the Azure portal (https://portal.azure.com ).
4. From the Storage accounts blade, get your list of storage account instances.
7. Select the Basics tab, and in the tab area, perform the following actions:
ii. In the Resource group section, select PolyglotData from the list.
iii. In the Storage account name text box, enter polystor (or any name).
iv. In the Location drop-down list, select the (US) East US region.
vi. In the Account kind drop-down list, select StorageV2 (general purpose v2).
[email protected] 14
8. Select Create to create the storage account by using your specified configuration.
[email protected] 15
4 IMPORT AND VALIDATE DATA
In this section, you will import all the resources that you'll use with your web application.
2. From the Resource groups blade, find and select the PolyglotData resource group that
3. From the PolyglotData blade, select the k21polystor[yourname]** storage account that
4. From the Storage account blade, select the Containers link in the Blob service section
ii. In the Public access level drop-down list, select Blob (anonymous read access
7. Back in the Containers section, select the newly created images container.
8. From the Container blade, find the Settings section from the blade, and then select the
Properties link.
9. In the Properties pane, record (copy and save somewhere with you) the value in the URL
[email protected] 17
10. Find and select the Overview link from the blade.
Pre-requisite: Download a Folder from the given Github link for some samples packages:
[email protected] 18
https://github.com/MicrosoftLearning/AZ-204-DevelopingSolutionsforMicrosoftAzure
ii. In the File Explorer window, browse to the Location where you downloaded the
iii. Ensure that Overwrite if files already exist is selected, and then select Upload.
[email protected] 19
4.2 Upload an SQL .bacpac file
1. From the Resource groups blade, find and select the PolyglotData resource group that
2. From the PolyglotData blade, select the k21polystor[yourname]** storage account that
3. From the Storage account blade, select the Containers link in the Blob service section
ii. In the Public access level drop-down list, select Private (no anonymous access).
[email protected] 20
6. Back in the Containers section, select the newly created databases container.
[email protected] 21
ii. In the File Explorer window, browse to Downloaded Github Folder
Open.
iii. Ensure that Overwrite if files already exist is selected, and then select Upload.
[email protected] 22
4.3 Import an SQL database
1. From the Resource groups blade, find and select the PolyglotData resource group that
2. From the PolyglotData blade, select the k21polysqlsrvr[yourname]** SQL server that
[email protected] 23
4. From the Import database blade, perform the following actions:
iii. From the Storage accounts blade, select the k21polystor[yourname]** storage
iv. From the Containers blade, select the databases container that you created earlier
in this lab.
v. From the Container blade, select the AdventureWorks.bacpac blob that you
created earlier in this lab, and then select Select to close the blade.
vi. Back from the Import database blade, leave the Pricing tier option set to its default
value.
viii. Leave the Collation text box set to its default value.
[email protected] 24
x. In the Password text box, enter TestPa55w.rd.
Note: Wait for the database to be created before you continue with this lab. If you receive a
firewall-related error on the import step, it means you did not correctly configure the Allow Azure
services to access server setting on your SQL Server earlier in the lab. Review your settings,
delete the empty AdventureWorks database, and then attempt your import again.
[email protected] 25
4.4 Use an imported SQL
database
1. From the Resource groups blade, find and select the PolyglotData resource group that
2. From the PolyglotData blade, select the k21polysqlsrvr[yourname]** SQL server that
3. From the SQL server blade, find the Security section from the blade, and then select the
4. In the Firewalls and virtual networks pane, perform the following actions:
[email protected] 26
ii. Select Save.
Note: This step will ensure that your local machine will have access to the databases that
5. From the Resource groups blade, find and select the PolyglotData resource group that
6. From the PolyglotData blade, select the AdventureWorks SQL database that you
[email protected] 27
7. From the SQL database blade, find the Settings section from the blade, and then select
[email protected] 28
8. In the Connection strings pane, record the value in the ADO.NET (SQL Authentication)
9. Update the connection string in your file, that you recorded by performing the following
actions:
i. Within the connection string, find the your_username placeholder and replace it with
ii. Within the connection string, find the your_password placeholder and replace it with
10. Find and select the Query editor (preview) link from the blade.
[email protected] 29
11. In the Query editor pane, perform the following actions:
[email protected] 30
12. In the open query editor, enter the following query:
13. Select Run to run the query, and then observe the results.
Note: This query will return a list of models from the home page of the web application.
[email protected] 31
14. In the query editor, replace the existing query with the following query:
Note: This query will return a list of products that are associated with each model.
[email protected] 32
5 OPEN AND CONFIGURE A .NET WEB APPLICATION
In this section, you will configure your ASP.NET web application to connect to your resources in
Azure.
[email protected] 33
3. In the File Explorer window that opens, browse to
4. In the Visual Studio Code window, right-click or activate the shortcut menu for the
[email protected] 34
5. At the open command prompt, enter the following command, and then select Enter to build
dotnet build
Note: The dotnet build command will automatically restore any missing NuGet packages
6. Observe the results of the build printed in the terminal. The build should complete
7. Select Kill Terminal or the Recycle Bin icon to close the currently open terminal and any
associated processes.
AdventureWorks.Web project.
[email protected] 35
2. Open the appsettings.json file.
[email protected] 36
3. In the JavaScript Object Notation (JSON) object on line 3, find the
empty:
4. Update the value of the AdventureWorksSqlContext property by setting its value to the
ADO.NET (SQL Authentication) connection string of the SQL database that you
Note: It's important that you use your updated connection string here. The original connection
string copied from the portal won't have the username and password necessary to connect to
the SQL database. So make sure to put your SQL Server username and password.
[email protected] 37
Note: Please remove the curly brackets { } before typing the password.
2. Update the value of the BlobContainerUrl property by setting its value to the URL property
of the Azure Storage blob container named images that you recorded earlier in this lab.
[email protected] 38
3. Save the appsettings.json file using Ctrl+S.
command, and then select Enter to switch your terminal context to the
AdventureWorks.Web folder:
cd .\AdventureWorks.Web\
2. At the command prompt, enter the following command, and then select Enter to run the
dotnet run
[email protected] 39
Note: The dotnet run command will automatically build any changes to the project and then
start the web application without a debugger attached. The command will output the URL
4. Now open a web browser window, browse to the currently running web application
(http://localhost:5000)
5. In the web application, observe the list of models displayed from the front page.
[email protected] 40
6. Find the Water Bottle model, and then select View Details.
7. From the Water Bottle product detail page, find Add to Cart, and then observe that the
9. Return to the Visual Studio Code window, and then select Kill Terminal or the Recycle
Bin icon to close the currently open terminal and any associated processes.
[email protected] 41
6 MIGRATING SQL DATA TO AZURE COSMOS DB
In this section, you used Entity Framework and the .NET SDK for Azure Cosmos DB to migrate
shortcut menu for the Explorer pane, and then select Open in Terminal.
2. At the open command prompt, enter the following command, and then select Enter to
create a new .NET console project named AdventureWorks.Migrate in a folder with the
same name:
Note: The dotnet new command will create a new console project in a folder with the same name
as the project.
3. At the command prompt, enter the following command, and then select Enter to add a
[email protected] 42
dotnet add .\AdventureWorks.Migrate\AdventureWorks.Migrate.csproj reference
.\AdventureWorks.Models\AdventureWorks.Models.csproj
Note: The dotnet add reference command will add a reference to the model classes contained in
4. At the command prompt, enter the following command, and then select Enter to add a
.\AdventureWorks.Context\AdventureWorks.Context.csproj
Note: The dotnet add reference command will add a reference to the context classes contained
5. At the command prompt, enter the following command, and then select Enter to switch your
[email protected] 43
cd .\AdventureWorks.Migrate\
6. At the command prompt, enter the following command, and then select Enter to import
Microsoft.EntityFrameworkCore.SqlServer.
7. At the command prompt, enter the following command, and then select Enter to import
[email protected] 44
Note: The dotnet add package command will add the Microsoft.Azure.Cosmos package from
8. At the command prompt, enter the following command, and then select Enter to build the
dotnet build
9. Select Kill Terminal or the Recycle Bin icon to close the currently open terminal and any
associated processes.
[email protected] 45
6.2 Create a .NET class
1. In the Explorer pane of the Visual Studio Code window, expand the
AdventureWorks.Migrate project.
3. From the code editor tab for the Program.cs file, delete all the code in the existing file.
using AdventureWorks.Context;
using AdventureWorks.Models;
[email protected] 46
5. Add the following line of code to import the Microsoft.Azure.Cosmos namespace from the
using Microsoft.Azure.Cosmos;
using Microsoft.EntityFrameworkCore;
7. Add the following lines of code to add using directives for the built-in namespaces that will
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
[email protected] 47
{
9. Within the Program class, enter the following line of code to create a new string constant
named sqlDBConnectionString:
10. Update the sqlDBConnectionString string constant by setting its value to the ADO.NET
(SQL Authentication) connection string of the SQL database that you recorded earlier in
this lab.
[email protected] 48
Note: It's important that you use your updated connection string here. The original connection
string copied from the portal won't have the username and password necessary to connect to
the SQL database. So make sure to put your SQL Server username and password.
11. Within the Program class, enter the following line of code to create a new string constant
named cosmosDBConnectionString:
12. Update the cosmosDBConnectionString string constant by setting its value to the
PRIMARY CONNECTION STRING of the Azure Cosmos DB account that you recorded
13. Within the Program class, enter the following code to create a new asynchronous Main
method:
[email protected] 49
public static async Task Main(string[] args)
14. Within the Main method, add the following line of code to print an introductory message to
the console:
[email protected] 50
16. In the Visual Studio Code window, access the shortcut menu or right-click or activate the
shortcut menu for the Explorer pane, and then select Open in Terminal.
17. At the open command prompt, enter the following command, and then select Enter to
cd .\AdventureWorks.Migrate\
18. At the command prompt, enter the following command, and then select Enter to build the
dotnet build
19. Select Kill Terminal or the Recycle Bin icon to close the currently open terminal and any
associated processes.
[email protected] 51
6.3 Get SQL database records by
using Entity Framework
1. Within the Main method of the Program class within the Program.cs file, add the following
AdventureWorksSqlContext(sqlDBConnectionString);
2. Within the Main method, add the following block of code to issue a language-integrated
query (LINQ) to get all Models and child Products from the database and store them in an
.ToListAsync<Model>();
3. Within the Main method, add the following line of code to print the number of records
5. In the Visual Studio Code window, access the shortcut menu or right-click or activate the
shortcut menu for the Explorer pane, and then select Open in Terminal.
6. At the open command prompt, enter the following command, and then select Enter to
cd .\AdventureWorks.Migrate\
7. Select Kill Terminal or the Recycle Bin icon to close the currently open terminal and any
associated processes.
[email protected] 53
6.4 Insert items into Azure
Cosmos DB
1. Within the Main method of the Program class within the Program.cs file, add the following
line of code to create a new instance of the CosmosClient class, passing in the
2. Within the Main method, add the following line of code to create a new database named
3. Within the Main method, add the following block of code to create a new container named
Online if it doesn't already exist in the Azure Cosmos DB account with a partition key path
partitionKeyPath: $"/{nameof(Model.Category)}",
throughput: 1000
[email protected] 54
);
4. Within the Main method, add the following line of code to create an int variable named
count:
int count = 0;
5. Within the Main method, add the following block of code to create a foreach loop that
[email protected] 55
6. Within the foreach loop in the Main method, add the following line of code to upsert the
object into the Azure Cosmos DB collection and save the result in a variable of type
7. Within the foreach loop contained in the Main method, add the following line of code to
Id: {document.ActivityId}]");
[email protected] 56
8. Back within the Main method (outside of the foreach loop), add the following line of code to
10. In the Visual Studio Code window, access the shortcut menu or right-click or activate the
shortcut menu for the Explorer pane, and then select New Terminal.
11. At the open command prompt, enter the following command, and then select Enter to
cd .\AdventureWorks.Migrate\
dotnet run
[email protected] 57
Note: The dotnet run command will start the console application.
2. Observe the various data that prints to the screen, including initial SQL record count,
individual upsert activity identifiers, and final Azure Cosmos DB document count.
3. Select Kill Terminal or the Recycle Bin icon to close the currently open terminal and any
associated processes.
[email protected] 58
6.6 Validate the migration
1. Return to your web browser window with the Azure portal.
2. In the Azure portal's navigation pane, select the Resource groups link.
3. From the Resource groups blade, find and select the PolyglotData resource group that
5. From the Azure Cosmos DB account blade, find and select the Data Explorer or
[email protected] 59
6. In the Data Explorer pane, expand the Retail database node.
7. Expand the Online container node, and then select New SQL Query.
[email protected] 60
8. From the query tab, enter the following text:
9. Select Execute Query, and then observe the list of JSON models that the query returns.
10. Back in the query editor, replace the existing text with the following text:
[email protected] 61
SELECT VALUE COUNT(1) FROM models
11. Select Execute Query, and then observe the result of the COUNT aggregate operation.
[email protected] 62
7 ACCESSING AZURE COSMOS DB BY USING .NET
In this exercise, you will write C# code to query an Azure Cosmos DB collection by using the .NET
SDK.
shortcut menu for the Explorer pane, and then select New Terminal.
2. At the open command prompt, enter the following command, and then select Enter to
cd .\AdventureWorks.Context\
3. At the command prompt, enter the following command, and then select Enter to import
[email protected] 63
Note: The dotnet add package command will add the Microsoft.Azure.Cosmos package
4. At the command prompt, enter the following command, and then select Enter to build the
dotnet build
5. Select Kill Terminal or the Recycle Bin icon to close the currently open terminal and any
associated processes.
AdventureWorks.Context project.
[email protected] 64
2. Access the shortcut menu or right-click or activate the shortcut menu for the
4. From the code editor tab for the AdventureWorksCosmosContext.cs file, add the
using AdventureWorks.Models;
[email protected] 65
5. Add the following lines of code to import the Microsoft.Azure.Cosmos and
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.Linq;
6. Add the following lines of code to add using directives for the built-in namespaces that will
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
[email protected] 66
7. Enter the following code to add an AdventureWorks.Context namespace block:
namespace AdventureWorks.Context
8. Within the AdventureWorks.Context namespace, enter the following code to create a new
AdventureWorksCosmosContext class:
[email protected] 67
{
IAdventureWorksProductContext interface:
Note: Please ignore the error showing on adding the previous command as this will be
10. Within the AdventureWorksCosmosContext class, enter the following line of code to
[email protected] 68
private readonly Container _container;
11. Within the AdventureWorksCosmosContext class, add a new constructor with the
following signature:
12. Within the constructor, add the following block of code to create a new instance of the
CosmosClient class and then obtain both a Database and Container instance from the
client:
.GetDatabase(database)
.GetContainer(container);
[email protected] 69
13. Within the AdventureWorksCosmosContext class, add a new FindModelAsync method
14. Within the FindModelAsync method, add the following blocks of code to create a LINQ
query, transform it into an iterator, iterate over the result set, and then return the single item
.ToFeedIterator<Model>();
while (iterator.HasMoreResults)
matches.AddRange(next);
return matches.SingleOrDefault();
[email protected] 70
15. Within the AdventureWorksCosmosContext class, add a new GetModelsAsync method
[email protected] 71
16. Within the GetModelsAsync method, add the following blocks of code to run an SQL
query, get the query result iterator, iterate over the result set, and then return the union of
all results:
while (iterator.HasMoreResults)
matches.AddRange(next);
return matches;
[email protected] 72
17. Within the AdventureWorksCosmosContext class, add a new FindProductAsync
18. Within the FindProductAsync method, add the following blocks of code to run an SQL
query, get the query result iterator, iterate over the result set, and then return the single
[email protected] 73
19. Save the AdventureWorksCosmosContext.cs file.
20. In the Visual Studio Code window, access the shortcut menu or right-click or activate the
shortcut menu for the Explorer pane, and then select New Terminal.
21. At the open command prompt, enter the following command, and then select Enter to
cd .\AdventureWorks.Context\
22. At the command prompt, enter the following command, and then select Enter to build the
dotnet build
23. Select Kill Terminal or the Recycle Bin icon to close the currently open terminal and any
associated processes.
[email protected] 74
7.3 Update the Azure Cosmos
DB connection string
1. In the Explorer pane of the Visual Studio Code window, expand the
AdventureWorks.Web project.
[email protected] 75
3. In the JSON object on line 4, find the
value is empty:
the PRIMARY CONNECTION STRING of the Azure Cosmos DB account that you recorded
[email protected] 76
7.4 Update .NET application
startup logic
1. In the Explorer pane of the Visual Studio Code window, expand the
AdventureWorks.Web project.
[email protected] 77
Note: The current product service uses SQL as its database.
5. Within the ConfigureProductService method, add the following block of code to change
services.AddScoped<IAdventureWorksProductContext,
AdventureWorksCosmosContext>(provider =>
new AdventureWorksCosmosContext(
_configuration.GetConnectionString(nameof(AdventureWorksCosmosContext))
);
[email protected] 78
7.5 Validate that the .NET
application successfully
connects to Azure Cosmos
DB
1. In the Visual Studio Code window, access the shortcut menu or right-click or activate the
shortcut menu for the Explorer pane, and then select New Terminal.
2. At the open command prompt, enter the following command, and then select Enter to
cd .\AdventureWorks.Web\
3. At the command prompt, enter the following command, and then select Enter to run the
dotnet run
Note: The dotnet run command will automatically build any changes to the project and then start
the web application without a debugger attached. The command will output the URL of the
4. Now open you web browser window, browse to the currently running web application
(http://localhost:5000).
5. In the web application, observe the list of models displayed from the front page.
[email protected] 79
6. Find the Touring-1000 model, and then select View Details.
7. From the Touring-1000 product detail page, perform the following actions:
[email protected] 80
i. In the Select options list, select Touring-1000 Yellow, 50, $2,384.07.
ii. Find Add to Cart, and then observe that the checkout functionality is still disabled.
9. Return to the Visual Studio Code window, and then select Kill Terminal or the Recycle
Bin icon to close the currently open terminal and any associated processes.
10. Close the active applications and delete the resources you created in this guide.
[email protected] 81
8 SUMMARY
This completes Step by Step guide covering:
How to Create instances of various database services by using the Azure portal.