14/7/2021 How to handle 100 million rows with SQL Server BCP

SQLShack SQL Server training

Español


How to handle 100 million rows with SQL Server

BCP
April 9, 2021 by Esat Erkec

In this article, we will explore the Bulk Copy Program tool that is also known as the SQL Server BCP
tool. BCP allows us to export data into particular flat-file formats from SQL Server, and it also enables
us to transfer data between different SQL Server instances or into the SQL Azure.

Introduction
Either exporting or importing the data is among the most needed operations of the database persons.
In order to achieve this operation on SQL Server, we have various tool alternatives. Such as SQL Server
Integration Services (SSIS), SQL Server Management Studio (SSMS), OPENROWSET function, and BCP
are the first options that come to our minds. SQL Server BCP is a very simple command-line tool that
exports table data into flat files from SQL Server, or it can import data into the database tables. How‐
ever, when we want to use the BCP for the giant data transferring operations we might face some per‐
formance problems. For this reason, in the next sections of this article, we will mainly focus on how to
improve the performance of the data transferring operations for the tables which have a huge number
of rows.

Pre-requirements
For the examples in this article, we will create a table and populate it with 100 million rows. The fol‐
lowing script
creates the SalesPerson table and we can also use ApexSQL Generate to generate 100
million test data.

CREATE TABLE [dbo].[SalesPerson](

[SalesPerson_Id] [int] IDENTITY(1,1) NOT NULL,
[First_name] [varchar](50) NOT NULL,
[Last name] [varchar](50) NOT NULL,
https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 1/12
14/7/2021 How to handle 100 million rows with SQL Server BCP
_
[Email] [varchar](255) NOT NULL,
[Phone] [varchar](25) NULL,
[Active] [tinyint] NOT NULL,
[Store_id] [int] NOT NULL,
[Manager_id] [int] NOT NULL,
PRIMARY KEY CLUSTERED
(
[SalesPerson_Id] ASC
))

On the other hand, we will use the Adventureworks sample database for the first look examples.

A first look at BCP

As we stated before, the BCP is a command-line tool and as a first step, we need to check the installed
version of
this tool. The bcp /v command gives the version information of this utility.

If the installed version is older than the last version, we can download and install the latest version
from the
Microsoft website. The main capability of the SQL Server BCP is not much complex
because it
can only run with several arguments. The syntax of the BCP is like below:

bcp {table|view|”query”} {out|queryout|in|format} {data_file|nul}

{[optional_argument]…}

For example, if we want to export any data of a table to a text file, we have to specify the table name,
the out option, and the data file. The following command will export the Production
table into the
specified text file.

bcp AdventureWorks2017.Production.Product out C:\ExportedData\Product.txt -

S localhost -T –w

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 2/12
14/7/2021 How to handle 100 million rows with SQL Server BCP

In the above script, we have also used some additional parameters:

-S: Server Name

-T: Use the trusted connection

-w: Performs the bulk copy operation using Unicode characters

After the exporting operation, the text file will look as below:

At the same time, we can export result sets of the queries through the queryout parameter so that
we can
filter the data or join the tables before exporting operation.

bcp ” SELECT p.[FirstName] ,p.[LastName],e.BirthDate FROM

AdventureWorks2017.[HumanResources].[Employee] e
INNER JOIN
AdventureWorks2017.[Person].[Person] p ON p.[BusinessEntityID] = e.[Busi-
nessEntityID] WHERE
e.BirthDate > ‘19800101’” queryout
C:\ExportedData\QueryData.txt -S localhost -T -w
https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 3/12
14/7/2021 How to handle 100 million rows with SQL Server BCP
p y

Using the BCP to import data into the SQL Azure

We can use BCP to import data into SQL Azure. The following command will import the Production
table text data into the SQL Azure. The only change is to use in the argument and it specifies copy
the
data from a file into the database table.

bcp TestDB.dbo.Product in C:\ExportedData\Product.txt -S

tcp:esat1.database.windows.net –U username –P password -w

The imported data can be seen in the query editor of SQL Azure.
https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 4/12
14/7/2021 How to handle 100 million rows with SQL Server BCP

Use the SQL Server BCP to export big tables data

In this example, we will export the SalesPerson table data into a text file. This table includes
100 mil‐
lion rows and it’s size is about 7.5 GB.

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 5/12
14/7/2021 How to handle 100 million rows with SQL Server BCP

In our first testing, we will run the SQL Server BCP with default values in order to export 100 M rows.

bcp SalesTest.dbo.SalesPerson out C:\ExportedData\SalesPerson.txt -S local-

host -T -w

As we can see in the above image, the export operation has completed about 588.079 seconds and it
has exported 170.045 rows per second. The network packet size configuration allows us to specify
how much bytes data is sent out by the SQL Server. The –a parameter changes the packet size
individually for the bcp data transfer session and might help to increase the performance of data
transfer
operations. The default packet size is 4096 bytes and we will increase this number to
32.728
bytes and it will affect the performance of the data exporting positively.

bcp SalesTest.dbo.SalesPerson out C:\ExportedData\SalesPerson.txt -S local-

host -T -w -a 32768

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 6/12
14/7/2021 How to handle 100 million rows with SQL Server BCP

After changing the packet size parameter of the BCP the data transfer duration has decreased and the
number of rows transferred per second has increased so increasing the packet size may have an im‐
provement in data transfer. As a
last, using the fast disk systems for the exported file location, chang‐
ing the packet size parameter of the BCP, and using the fast NIC card will improve the export
performance.

Use the SQL Server BCP to import a huge amount

of data into tables
In general, performance bottlenecks are experienced related to BCP during the importing of external
data into the
SQL tables. Under the full recovery model, the first thing, we need to consider is the
workload that will occur on the log file during the data importing operation because if we don’t deter‐
mine any batch size for the data import operation, the whole operation will be done in a big single
transaction. In this circumstance, all imported data will be fully written into the transaction log file of
the database.

Before starting the first testing with BCP, we will take a look at the log file size of the database and
then we
will start the data import operation for 100 M rows.

SELECT CAST(ROUND((total_log_size_in_bytes)*1.0/1024/1024,2,2) AS FLOAT)

AS [Total Log Size]  
FROM sys.dm_db_log_space_usage;

bcp Sales.dbo.SalesPerson in C:\ExportedData\SalesPerson.txt -S localhost -

T –w

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 7/12
14/7/2021 How to handle 100 million rows with SQL Server BCP

After the completion of the data import operation, the log file size has reached 27.144 megabytes.

SELECT CAST(ROUND((total_log_size_in_bytes)*1.0/1024/1024,2,2) AS FLOAT)

AS [Total Log Size]  
FROM sys.dm_db_log_space_usage;

On the other hand, using the simple recovery model or bulk-logged recovery model may enable the
minimal logging
mode. For SQL Server to enable the minimum logging option, the target table must
meet the following conditions:

The table is not being replicated

The table is not memory-optimized

In addition to these conditions, we need to use the TABLOCK hint in the BCP command, and also the
table must be empty if it includes a clustered index.

After changing the database recovery model to bulk-logged, we drop and re-create the target table
and re-execute the
following bcp command with the TABLOCK hint.

bcp Sales.dbo.SalesPerson in C:\ExportedData\SalesPerson.txt -S localhost -

T h”TABLOCK”
https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 8/12
14/7/2021 How to handle 100 million rows with SQL Server BCP
T -w -h”TABLOCK”

After the completion of the data import operation, the log file size has only reached 200 megabytes.

In this test, we have realized that SQL Server minimizes the log file activity and it increases the perfor‐
mance of
the bulk copy operation.

Use the native data format to import or export

data with SQL Server bcp
Microsoft recommends using native data formats when the data will be transferred between two SQL
Server instances
for the identical tables if the table data does not contain any extended/double-byte
character set (DBCS) characters. Thus, SQL Server avoids unnecessary conversion of data types to and
from character format. To use native data format in BCP, we need to replace the –w parameter with
the –n parameter.

bcp Sales.dbo.SalesPerson in C:\ExportedData\SalesPerson.txt -S localhost -

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 9/12
14/7/2021 How to handle 100 million rows with SQL Server BCP
T -n -h”TABLOCK”

After using the native data format in BCP, the data import performance has boosted because the un‐
necessary data conversions are eliminated.

The following chart shows how the minimal logging mode and using the native data format option af‐
fect the data
importing performance.

Conclusion
In this article, we have explored the SQL Server BCP tool and also we have focused on how to improve
its performance with some changes. Minimal logging mode and using the native formats dramatically
increase the performance of the
BCP. At the same time, the packet size parameter of the BCP can af‐
fect the performance of the data transfer performance of the BCP.

See more
To generate millions of rows of test data quickly, consider ApexSQL Generate, a test data generator
specifically designed for SQL Server developers

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 10/12
14/7/2021 How to handle 100 million rows with SQL Server BCP

Esat Erkec
Esat Erkec is a SQL Server professional who began his career 8+ years ago as
a Software Developer. He is a SQL Server Microsoft Certified Solutions Expert.

Most of his career has been focused on SQL Server Database Administration and
Development. His current interests are in database administration and Business
Intelligence. You can find him on LinkedIn.

View all posts by Esat Erkec

Related Posts:
1. How to import data from an Excel file to a SQL Server database
2. How to import/export data to SQL Server using the SQL Server Import and Export Wizard
3. How to export data from SQL Server to a Flat file
4. An introduction to the bcp Utility (bulk copy program) in SQL Server
5. The BCP (Bulk Copy Program) command in action

Migration, Performance

3,195 Views
ALSO ON SQL SHACK

An overview of Azure Best author award in Learn MySQL: Install Export i

Data Lake Analytics … 2020 MySQL server 8.0.19 … constra
5 months ago • 1 comment 6 months ago • 1 comment 3 months ago • 2 comments 3 months a

This article will describe an Best author award in 2020 This article will show how This artic
overview of the Azure Data we can install MySQL export the
Lake Analytics and U-SQL. Server using noinstall ZIP … indexes a

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 11/12
14/7/2021 How to handle 100 million rows with SQL Server BCP

2 Comments SQL Shack 🔒 Disqus' Privacy Policy  Ronald Martinez

 Recommend 2 t Tweet f Share Sort by Best

Join the discussion…

BayerArsenal • 3 months ago

Hi Esat,

Thank you for putting up this article regarding the BCP tool, its usage explanation and the ways
to use this tool for exporting and importing table with 100 millions rows with screenshot
demonstrations. A very insightful information needed during this time period.

I noticed a couple of things that may need to be updated: you mentioned the packet size is
32.728 bytes, but in the BCP screen it showed 32756 bytes. Second, the log file size after data
import is mentioned 27.144MB but the screenshot shows 27143.99MB. Last, before the
Conclusion section, you have mentioned about a chart showing how the minimal logging mode
and using the native data format option affect the data importing performance, but the chart is
not displayed.

Keep it going Sir! Thank you.

△ ▽ • Reply • Share ›

Esat Erkeç > BayerArsenal • 3 months ago

Thank @BayerArsenal your feedbacks. I fixed the chart
△ ▽ • Reply • Share ›

✉S d ⚠ S

© 2021 Quest Software Inc. ALL RIGHTS RESERVED.   |   GDPR   |   Terms of Use   |   Privacy

https://www.sqlshack.com/how-to-handle-100-million-rows-with-sql-server-bcp/ 12/12