Getting Started with MASM and Visual Studio 2019

Updated 3/18/2022
http://www.asmirvine.com/gettingStartedVS2019/index.htm

This tutorial assumes that you are using either the 7th or 8th Edition of Assembly
Language for x86 Processors, and you are using Visual Studio 2019.

Before starting the tutorials listed below, please visit our GitHub project
page (surferkip/asmbook) to download the following files:

 Irvine.zip - code examples and required libraries

 Project32_VS2019.zip - sample 32-bit Visual Studio project, configured for
assembly language
 Project64_VS2019.zip - sample 64-bit Visual Studio project, configured for
assembly language

Topics:

 Tutorial: Building and running a 32-bit program

 Tutorial: Building and running a 64-bit program
 Building 16-bit programs
 Syntax highlighting in your source code
 Using the Visual Studio debugger
 EXE Programs blocked by antivirus software

Found an error in this document? Please email the author.

Required Setup for 32-bit Applications

After you have downloaded and installed the VS 2019 Community Edition, you may
need to install the Visual C++ language option. First, let's see if it has already been
installed (as often happens in college computer labs). Select File >> New >> Project
from the Visual Studio menu. You will see this Create a new project dialog window.
Look for C++ in the Language dropdown list:

Note: If you do not see Visual C++ in the list, close Visual Studio and run a separate
program named the Visual Studio Installer. If your computer is in a college laboratory,
your account may not have sufficient privileges to run this program, so you can ask
your lab supervisor to do this.

(If you run the VS installer, select the Desktop development with C++ button in the
installer window, look at the Summary list on the right side to verify that VC++ is
selected, and click the Modify button in the lower right corner of the window.)

The Visual C++ language includes the Microsoft Assembler (MASM). To verify that
MASM is installed, open a Windows Explorer window and look for the file
named ml.exe in the Visual Studio installation directory, such as C:\Program Files
(x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.xx.xxxx\bin\
HostX64\x86. (The "x" characters above indicate digits in the version number of your
current VS installation.)

The Book's Example Programs

At the top of this document, we explained how to download the file named Irvine.zip
and extract it into the C:\Irvine folder. Unless you have some objection to using that
location, do not alter the path. (Note to lab administrators: you can designate c:\
Irvine directory as read-only.).

The folllowing files should appear in the c:\Irvine directory:

Filename Description
b16.asm, Blank templates for 16-bit and 32-bit assembly language
b32.asm source files
GraphWin.inc Include file for writing Windows applications
Include file used with the Irvine16 link library (16-bit
Irvine16.inc
applications)
Irvine16.lib 16-bit link function library used with this book
Include file used with the Irvine32 link library (32-bit
Irvine32.inc
applications)
Irvine32.lib Irvine's 32-bit link library
Kernel32.lib 32-bit link library for Windows API
Link16.exe 16-bit Microsoft linker
Macros.inc Irvine's macro include file (see Chapter 10)
make16_vs2019.
Visual Studio 2019 batch file for building 16-bit applications
bat
Small-sized include file containing MS-Windows definitions,
SmallWin.inc
used by Irvine32.inc
User32.lib MS-Windows basic I/O link library
VirtualKeys.inc Keyboard code definitions file, used by Irvine32.inc
A subdirectory named Examples will contain all the example programs shown in the
book, source code for the book's 16-, 32-, and 64-bit libraries, and two sample
projects for earlier versions of Visual Studio.

Setting up Visual Studio

Select the C++ Configuration

Visual Studio supports multiple programming languages and application types. The
C++ programming language configuration most closely matches that of assembly
language programming, so we suggest the following steps:

1. Select Tools >> Import and Export Settings from the menu
2. Select the "Import selected environment settings" radio button
3. Select the "No, just import..." radio button
4. Select "Visual C++" from the Default Settings List and click the Next button
5. Click the Finish button, then click the Close button
6. Notice the tabs on the left and right sides of the Visual Studio workspace.
Close the Server Explorer, Toolbox, and Properties tabs. (Optionally, you can
use the mouse to drag the Solution Explorer tool window to the right side of
the workspace.) If you should accidentally close the Solution Explorer window
in the future, you can bring it back: select View from the menu, and locate
Solution Explorer in the list of views.

Optional Step: Set the tab indent size

Start Visual Studio and select Options from the Tools menu. Select and expand
the Text Editor item, select All Languages, and select Tabs. Optionally, you may
want to select the Insert spaces radio button:
I prefer to set the Tab Size and Indent Size values to 5.

Tutorial: Building and running a 32-bit program

Now you're ready to open and build your first 32-bit project.

Opening a Project

Visual Studio requires assembly language source files to belong to a project, which is
a kind of container. A project holds configuration information such as the locations of
the assembler, linker, and required libraries. A project has its own folder, and it holds
the names and locations of all files belonging to it.

If you have not already done so,Right-click here to download a zip file containing an
up-to-date Visual Studio 2019 project that has been configured for assembly
language. After downloading this file, un-zip it into your working directory. It contains
a sample asm test file named AddTwo.asm.

Follow these steps:

1. Start Visual Studio.

2. Open our sample Visual Studio project file by
selecting File/Open/Project from the Visual Studio menu.
3. Navigate to your working folder where you unzipped our project file, and
select the file named Project.sln.
4. Once the project has been opened, you will see the project name in Visual
Studio's Solution Explorer window. You should also see an assembly language
source file in the project named AddTwo.asm. Double-click the file name to
open it in the editor.

You should see the following program in the editor window:

; AddTwo.asm - adds two 32-bit

integers.
; Chapter 3 example

.386
.model flat,stdcall
.stack 4096
ExitProcess
proto,dwExitCode:dword

.code
main proc
mov eax,5

add eax,6

invoke ExitProcess,0
main endp
end main

In the future, you can use this file as a starting point to create new programs by
copying it and renaming the copy in the Solution Explorer window.

Adding a File to a Project: If you need to add an .asm file to an open project, do
the following: (1) Right-click the project name in the Visual Studio window, select Add,
select Existing Item. (2) In the Add Existing Item dialog window, browse to the
location of the file you want to add, select the filename, and click the Add button to
close the dialog window.

Build the Program

Now you will build (assemble and link) the sample program. Select Build
Project from the Build menu. In the Output window for Visual Studio at the bottom of
the screen, you should see messages similar to the following, indicating the build
progress:

1>------ Build started: Project: Project, Configuration: Debug Win32

------
1>Assembling AddTwo.asm...
1>Project.vcxproj -> ...\Project32_VS2019\Debug\Project.exe
========== Build: 1 succeeded, 0 failed, 0 skipped ==========

If you do not see these messages, the project has probably not been modified since it
was last built. No problem--just select Rebuild Project from the Build menu.

Run the Program in Debug Mode

The easiest way to run your first program is to use the debugger. First, you must set
a breakpoint. When you set a breakpoint in a program, you can use the debugger to
execute the program a full speed (more or less) until it reaches the breakpoint. At
that point, the debugger drops into single-step mode. Here's how to do it:

1. Make sure the ASM source code file is open in the editor window.

2. Click the mouse along the border to the left of the mov eax,5 statement. A

large red dot should appear in the margin.

 3. Select Start Debugging from the Debug menu. The program should run and

pause on the line with the breakpoint. (Optionally, you can close the

Diagnostic Tools, Autos, and Call Stack windows.)

4. Press the F10 key (called Step Over) to execute the current statement.

Continue pressing F10 until the program is about to execute

the invoke statement.

5. A small black window icon should appear on either your Windows desktop or

status bar. The window should be blank because this program does not

display any output.

6. Press F10 one more time to end the program.

You can remove a breakpoint by clicking its dot with the mouse. Take a few minutes
to experiment with the Debug menu commands. Set more breakpoints and run the
program again.

Here's what your program will look like when paused at the breakpoint:

Running a program from the Command Prompt: When you assembled and
linked the project, a file named Project.exe was created inside the project's \Debug
folder. This file executes when you run the project. You can execute any EXE by
double-clicking its name inside Windows Explorer, but it will often just flash on the
screen and disappear. That is because Windows Explorer does not pause the display
before closing the command window. On the other hand, you can open a Command
prompt window, move to the Debug directory, and run Project.exe by typing "Project"
(without the quotes). You will need to do some reading on Windows shell commands
if you plan to use the command line.

To remove a source file from the Visual Studio window, right-click its filename and
select Remove. The file will not be deleted from the file system. On the other hand, if
you want to delete the file, select it and press the Del key.

Registers

Soon you will want to display CPU registers when debugging your programs. Here's
how to make them visible: First, under the Tools >> Options menu, select Debbuging
in the left panel, and select Enable address-level debugging. Next, set a breakpoint in
your source code on an executable statement, run your program in Debug mode,
select Windows from the Debug menu, and then select Registers from the drop-down
list.

If you do not see the Registers command in the Debug >> Windows drop-down menu
(which seems to be the case for the VS2019 Community Edition, there is a way to add
a Registers command button to your Debug toolbar. Here's how to do it:

1. While not debugging, select Customize from the Tools menu.

2. Click the Commands tab, select the Toolbar tab, and select Debug from the
list of toolbars.
3. Click the Add Command button. In the Categories list, select Debug.
4. Select Registers from the list of commands, click the OK button to close the
dialog window.
5. Click the Close button to close the Customize dialog. You should now see a
new button on the Debug toolbar that looks like a small rectangle containing
"0X" when you begin debugging a program.

The Registers window may appear docked to the top of the workspace, but you may
find it helpful to float the window on top of your workspace. Just grab the window
header with the mouse and pull it to the center area. You will also want to display the
CPU flags. To do that, right click inside the Registers window and check the
word Flags from the popup menu.

You can interrupt a debugging session at any time by selecting Stop Debugging from
the Debug menu. You can do the same by clicking the maroon-colored square button
on the toolbar. To remove a breakpoint from a program, click its red dot to make it
disappear.

A reminder, you might want to review our tutorial: Using the Visual Studio debugger

Building and Running Other Programs

Suppose you want to run another example program, or possibly create your own
program. You can remove the existing assembly language file from the Solution
Explorer window and insert a new .asm file into the project.

 To remove a program from a project without deleting the file, right-click its
name in the Solution Explorer window. In the context menu, select Remove. If
 you change your mind and decide to add it back to the project, right-click in
the same window, select Add, select Existing item, and select the file you
want to add.

Adding a File to a Project

An easy way to add an assembly language source file to an open project is to drag its
filename with the mouse from a Windows Explorer window onto the name of your
project in the Solution Explorer window. The physical file will not be copied--the
project only holds a reference to the file's location. Try this now:

1. Remove the AddTwo.asm file from your project.

2. Add a reference to the file Examples\ch03\AddTwoSum.asm to the project.
3. Build and run the project.

Copying a Source File

One way to make a copy of an existing source code file is to use Windows Explorer to
copy the file into your project directory. Then, right-click the project name in Solution
Explorer, select Add, select Existing Item, and select the filename.

Return to top

Tutorial: Building and running a 64-bit program

In this tutorial, we will show you how to assemble, link, and run a sample 64-bit
program. We assume you have already completed our tutorial entitled Building a 32-
Bit Assembly Language Program.

Do the following steps, in order:

1. Right-click here to download the Project64_VS2019.zip file and unzip it into

your working directory.
2. In Visual Studio 2019, select Open Project from the File menu, navigate to the
Project64_VS2019 folder, and select the file named Project.sln.
3. You are about to add an existing source code file to the project. To do that,
right-click on Project in the Solution Explorer window, select Add,
select Existing Item, navigate to the book's Examples\ch03\64 bit" folder,
select AddTwoSum_64.asm, and click the Add button to close the dialog
window.
4. Open the AddTwoSum_64.asm file for editing by double-clicking its filename in
 the Solution Explorer window.

You should see the following program in the editor window:

; AddTwoSum_64.asm - Chapter 3 example.

ExitProcess proto

.data
sum qword 0

.code
main proc
mov rax,5
add rax,6
mov sum,rax

mov ecx,0
call ExitProcess

main endp
end
(Notice that the program's entry point is the main procedure. If you wish to use a
different name for your startup procedure in your own programs, you can modify this
option by selecting Properties from the Project menu, and then selecting Linker /
Advanced / Entry Point.)

Build the Program

Select Build Project from the Build menu. You should see text written to Visual
Studio's output window like the following:
1>------ Build started: Project: Project, Configuration: Debug x64
------
1> Assembling AddTwoSum_64.asm...
1> Project64_VS2019.vcxproj -> ...\Project64_VS2019\x64\Debug\
Project.exe
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped
==========
If you do not see these messages, the project has probably not been modified since it
was last built. No problem--just select Rebuild Project from the Build menu.

You use the same Visual Studio commands to run and debug 64-bit programs as you
would for 32-bit programs.

Return to top
Building 16-bit programs (Chapters 14-17)
Only Chapters 14 through 17 require you to build 16-bit applications. Except for a few
exceptions, which are noted in the book, your 16-bit applications will run under the
32-bit versions of Windows (Windows XP, Windows Vista, Windows 7).

If you're interested in running 16-bit programs under 64-bit Windows, you will need to
enable a feature named NTVDM.) Click here to read a web site with instructions on
how to do this. Another alternative you may wish to explore is to install a virtual
machine (using a free program named VirtualBox from Oracle) and install 32-bit
Windows on the virtual machine.

The book's example programs in Chapters 1-13 have been successfully tested in 32-
bit Windows 7,8, and 10. On the other hand, many programs in Chapters 14-17 will
not run in any Microsoft OS later than Windows 98, because they rely on direct access
to hardware and system memory. You cannot directly run 16-bit applications in any
64-bit version of Windows.

If you plan to build 16-bit applications, you need to add two new commands to the
Visual Studio Tools menu. To add a command, select External Tools from the Tools
menu. The following dialog will appear, although many of the items in your list on the
left side will be missing. Download the batch file here (rename the .txt extension
to .bat after downloading): make16_vs2019.txt.
Step 1: Create the Build 16-bit ASM Command

Click the Add button and fill in the Title, Command, Arguments, and Initial directory
fields as shown in the screen snapshot. If you click the buttons with arrows on the
right side of the Arguments and Initial directory fields, a convenient list appears. You
can select an item without having to worry about spelling:

Click the Apply button to save the command.

Step 2: Create the Run 16-bit ASM Command

Click the Add button again, and create a new command named Run 16-bit ASM:
Uncheck the "Close on exit" option and click the OK button to save the command and
close the External Tools dialog.

Testing Your new 16-Bit Commands

To test your new 16-bit commands, close any Visual Studio project that happens to be
open. Then, select File | Open | File from the menu and choose the file named 16-
bit.asm from the ch03 folder in the book's example programs. Select Build 16-bit
ASM from the Tools menu. The following command window should appear, showing
the successful execution of the assembler and linker, followed by a listing of all files
related to this program:
Press a key to close the window. Next, you will run the program. Select Run 16-bit
ASM from the Tools menu. The following window will appear, although the contents
of all registers except EAX will be different:

Press a key to close the window.

You have completed the setup for building and running 16-bit assembly language
programs.

Return to top

Syntax highlighting in your source code

When a text editor uses syntax highlighting, language keywords, strings, and other
elements appear in different colors. Visual Studio highlights MASM reserved words
and strings, as shown in the following example:
This won't happen automatically, but you can create a syntax definition file named
Usertype.dat that contains MASM keywords. Then when Visual Studio starts, it reads
the syntax file and highlights MASM keywords.

If you decide to use Visual Studio's built-in MASM syntax highlighter, here are the
required steps to set it up:

1) Download this Usertype.dat file (enclosed in a ZIP file) given here to a folder in
which you have read/write permissions. Extract it from the zip archive.

2) Close Visual Studio.

3) Copy Usertype.dat to the C:\Program Files (x86)\Microsoft Visual Studio\2019\

Community\Common7\IDE folder.

Windows will display a confirmation dialog before copying the file.

4) Open Visual Studio, select Options from the Tools menu, select Text Editor, and
select File Extension. On the right side of the dialog (shown below), enter asm as
the extension, select Microsoft Visual C++ from the Editor list, and click
the Add button. Click the OK button to save your changes.
Open your project and display an ASM file. You should see syntax highlighting in the
editor. There is a glitch in the highlighting--assembly language comment line starts
start with a semicolon, which C++ doesn't recognize. But this is a simple workaround:
add an extra // right after the semicolon, like this, which will cause the comments to
appear in their usual green color:

;// AddTwo.asm - adds two 32-bit integers.

;// Chapter 3 example

Return to top

Was your program's EXE file blocked by an Antivirus

scanner?
Antivirus scanner software has improved greatly in recent years, and so have the
number of viruses (one website reports 50,000 at present). Because of this, your
computer's antivirus scanner may report a false positive when you build your
program, and refuse to let you run it. There are a few workarounds: (1) You can add
your project's bin/debug folder into an exclusion list in your antivirus configuration.
This is my preferred approach, and it nearly always works. (2) You can suspend your
realtime antivirus scanner software, but this will leave you open to malware for a
short time. If you choose this option, be sure to temporarily disconnect your computer
from the Internet.

Return to top
Setup an Assembly
Project on Visual
Studio 2019
numnita Uncategorized February 16, 2020 8 Minutes

(Updated: Feb, 18, 2022)

The Microsoft Visual Studio come with the built-in Macro

Assembler and provide many debugging capabilities (Register
& Memory View, Breakpoints and Step over). However there is
no explicit assembly project template to choose from. In this
article, I will describe the steps required to set up an assembly
project on the Visual Studio 2019.

I had compiled the knowledge from many sources and do many

trial and error.

The Sample Projects (for quick

reference)
The sample projects of MASM (both 64-bit Assembly and 32-bit
Assembly) can be downloaded from the Github links below:

https://github.com/nthana/SampleASM64/

https://github.com/nthana/SampleASM32/
In these samples, I also demonstrate how to call the Win32
“MessageBox” API from the Assembly codes.

1. Installing Visual Studio 2019

Visual Studio 2019 could be downloaded from the
site: https://visualstudio.microsoft.com/vs/

The Community 2019 edition is free of charge.

When prompting to select an installed language, I choose C+

+. I’m not sure whether selecting other languages would work
or not for assembly project.

2. Creating a C++ project as a

base project
Create a C++ empty project.
Named the project: “MASM”. So that, in the later step, we can
save this project as a project template for further use. Set the
save location to any preferred folder (I have precreated the
folder C:\VSProj). Select “Place solution and project in the
same directory” to prevent two nested folders with the same
name.

The Visual Studio will create the folder with the same project
name automatically.

3. Setup the Build Customization

as Assembly Language
Right click at the project name > Build Dependencies > Build
Customizations…
Select the “masm”, then OK.
Right click at the project name > properties > Linker >
System > SubSystem : Windows
This will prevent the assembly program to show the console
window ( the black text-mode window). Or if you want the
console window you can leave it at the default: Console
mode.

I prefer the Windows mode because it has look-and-feel of

native windows app. For debugging, just use breakpoint and
watch windows to look at register and memory values at real-
time. However, if we want to test the text mode input/output,
we should use the console mode.
4. Creating an Assembly Source
File
Right click at the project name > Add > New Item…
Name the file: “Main.asm” . The selected item type is not
important. Any type can be selected. But it is important to
name it with the “.asm” file extension.
Write a test program. You can copy the code from below. I
adapted the code from the wonderful book : Assembly
Language for x86 Processors by Kip Irvine. I highly
recommend this book for anyone start learning the Assembly
language.

Firstly, We will create the 32-bit Assembly Code. And then at

the appendix B, we will create the 64-bit Assembly Code.

The 32-bit Assembly Code :

.386
.model flat, stdcall
.stack 4096

ExitProcess PROTO, dwExitCode:DWORD

.data
myList DWORD 2, 3, 5, 8

.code
main PROC
mov eax, 7
add eax, 8
INVOKE ExitProcess, eax
main ENDP

END main ;specify the program's entry point

To build the 32-bit Assembly, make sure that “x86” is selected

in the “solution platform” drop down of the toolbar.
Press the green, play button: (run with) Local Windows
Debugger.
The program will sum 7 with 8 and return the result 15 as
shown in the debug output window.

5. (Optional) Setup Listing File

The listing file is useful for examining what the assembler has
generated. To setup the listing file generation, right click the
project name > Properties > Microsoft Macro Assembler >
Listing File > Assembled Code Listing File : $(IntDir)$
(ProjectName).lst

Note: The page “Microsoft Macro Assembler” will appear only

when there are some assembly files in the project.

Build (and/or run) the project again. You will see the listing file
generated at the project’s debug folder.
Drag the listing file into Visual Studio to view it. The listing file
content will be like this:
6. (Optional) Setup for Release
Mode Build
This section could be skip because it is quite complex. Do
it when you need to use the release mode for release build. Or
you could come to this section later when in need.

In Visual Studio, there is a separate release mode build

configuration. The release mode configuration could be set
differently from the debug mode configuration. For example,
the release mode may have more optimizations and less
debugging features.

If the release mode is needed, we should do the same as the

step 3 and 5. Additionally, we need to customize the safe
exception handler configurations.

6.1 Setup Subsystem (Windows or Console)

and Listing File Generation
Go to the project properties > change the configuration mode
to : Release.
Choosing your prefer subsystem (Windows or Console).
Setting up the listing file generation.
Changing the active configuration to Release.

Try building and running the program using the green play
button. You will see the error message: “fatal error:
LNK1281: Unable to generate SAFESEH image.”.
The next step 5.2 will solve this error.
6.2 Fully Enable or Disable the Safe
Exception Handlers
In the release build mode, the default option of the linker is
that every images (object files) should have safe exception
handlers. However, the default option of the assembler
(compiler) is not to use safe exception handlers. These
conflicts cause the above error. Now, we have two choices to
fix the error.

Way 1: Enable the safe exception handlers in the assembler

(compiler).
Way 2: Or we could entirely disable the use of safe exception
handler at the linker.
Linker > Advance > Image has Safe Exception Handlers : No

Choose one way but not both!

I preferred way 1 because it looks safer to have safe exception

handlers. But, in fact, I am not 100% sure because I don’t have
much knowledge about safe exception handlers.

After that we should be able to build and run the program in

the release mode without any error.

Caution on switching between Debug and Release : When

changing the active configuration in the toolbar and then open
the project property pages, the property pages may not be
conformed to the current active configuration. This could
cause confusion when changing some project settings appear
to not affect the current build. If this occur, please ensure the
matching of active configuration and the property pages.

7. Save as a Project Template

After these long steps, we might not want to do them every
time to create a new Assembly project. Fortunately, Visual
Studio can save the project as a project template.

First, edit the Main.asm to contain only template code:

.386
.model flat, stdcall
.stack 4096

ExitProcess PROTO, dwExitCode:DWORD

.data

.code
main PROC

INVOKE ExitProcess, 0
main ENDP

END main ;specify the program's entry point

Try running it to make sure that the program works correctly.

If you still use release mode (from step 5), switch back to the
debug mode.
Click: Project > Export Template …
Choose our “MASM” project from the drop down list.

Describe the template description.

Finished!

After that when you create a new project, you could select
MASM as a project template.
Appendix A: Debugging
Debugging is straightforward in Visual Studio. To set a
breakpoint, just clicking the grey area in front of the line of the
source code. When running, the program will pause before the
execution of the breakpoint line.

To look at registers, select Debug > Windows > Registers

To look at variables, select Debug > Windows > Watch. Then

type the variable name.
Or select Debug > Windows > Memory > Memory 1. Then
type “&varname” at the address box.
An example of pausing at the breakpoint of the Visual Studio
Debugger for Assembly Language

Also, we can debug step-by-step using “Step Into” and “Step

Over”.
Appendix B: Setup the 64-Bit
Assembly Project
This section describes additional steps needed to setup a 64-
bit assembly project. You should finish the prior steps before
doing these steps.

1. Select the 64-bit platform

At the toolbar, set the Solution Platform to “x64”.
2. Creating the 64-bit Assembly Source Code
Copy the code below to replace the old code in main.asm:
The 64-Bit Assembly Code

ExitProcess PROTO

.data

.code
main PROC

sub rsp, 28h ;reserved the stack area as parameter passing

area
mov rcx, 12345678 ; specify Exit Code
call ExitProcess
main ENDP

END

Please note that the 32-bit and 64-bit assembly code are
different in many ways.

In 64-bit Assembly Code:

 Some directives, such as “.386”, are not used.
 PROTO doesn’t have parameter list specified.
 END doesn’t have “main” start symbol specified.
 When calling Win32 API such as “ExitProcess”,
parameters will be passed on registers (rcx, rdx, r8,
r9). (see details in Appendix C)

3. Specifying the “main” start symbol (entry

point)
When Build, you may get the error: unresolved external symbol
mainCRTStartup.
To fix this error:
 Open the project properties page (see image below) by
choosing Project > Properties.

 Then make sure that the “Active(Debug)”

Configuration and “x64” Platform are selected.

 Set the Linker > Advance > Entry Point to “main“

If you also want to use the release mode build, do the same for
the configuration: “Release”, platform: “x64”.
You can also choose the SubSystem: Console or Windows as
you preferred.

Run the program. The return code from calling ExitProcess

will be displayed on the output windows correctly.

Appendix C: Microsoft x64 calling

convention
Microsoft had defined the way that the 64-bit assembly code
should call the Win32 API. This specification is
called “Microsoft x64 calling convention”. Some essential
points are:
 The first four arguments are passed by registers (rcx,
rdx, r8, r9).

 Some stack memory, called the shadow area, must be

reserved (at least 4*8 bytes)

 16-byte stack pointer alignment (rsp = stack pointer

register) is needed before calling any win32 API.

That are the reasons of this line in the 64-bit assembly code:

sub rsp, 28h ;reserved shadow area

We need to reserve at least 32 bytes (20h) as the shadow area.

And main proc always start at the address ending 8h. So, to do
16-byte alignment, we need to add another (8h) bytes. That’s
why we subtract rsp by (28h) bytes. (the stack grows down)

Consider the following code. This code calls the MessageBox

Win32 API.

ExitProcess PROTO
MessageBoxA PROTO

.data
myText db "Hello World", 0
myCaption db "Message Box Testing", 0

.code
main PROC
sub rsp, 28h ; reserved shadow area
 mov rcx, 0
lea rdx, myText
lea r8, myCaption
mov r9, 0
call MessageBoxA

mov rcx, 12345678 ; the exit code

call ExitProcess

END

I have tried to depict the call stack memory layout in the

following diagram. Notice that rsp-28h will align the stack
pointer into 16-byte alignment beautifully, because the main()
procedure is always start with the stack pointer having
address ending with 8h. You can see this fact yourself by
debugging the above code, running it step by step and
observing register values in the register window.

Stack Area of x64 Calling Convention

To verify that the alignment requirement is needed, try
changing the “sub rsp, 28h” into “sub rsp, 20h“. This will
cause the Access Violation run-time error, when calling
MessageBox():

Some Win32 APIs are not that strict. For example, the
ExitProcess don’t cause the Access Violation Exception. But to
respect the x64 calling convention, every 64-bit assembly code
that call Win32 API should have some statement like “sub rsp,
28h” to reserve shadow area and adjust stack pointer
alignment.

Please note that if use full x64 prolog/epilog code, the “sub
rsp, 28h” must be changed to “sub rsp, 20h” because the prior
“push” instruction has changed the rsp by 8h bytes. So, need
to subtract only 20h.
Normal x64 Prolog/Epilog Code

.code
main PROC
; usual x86-64 Prolog code
push rbp ; save caller's rbp
mov rbp, rsp ; use rbp as a frame pointer
sub rsp, 20h ; reserved as Win32 parameters' shadow area

; Do Something

; usual x86-64 Epilog code

 mov rsp, rbp ; restore stack pointer
pop rbp ; restore caller's rbp
ret
main ENDP

CS/ECE 252 Spring 2009