Systems Programming with C# and .NET: Building robust system solutions with C# 12 and .NET 8

Systems Programming with C# and .NET

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Associate Group Product Manager: Kunal Sawant

Publishing Product Manager: Akash Sharma

Book Project Manager: Deeksha Thakkar

Senior Editor: Esha Banerjee

Technical Editor: Jubit Pincy

Copy Editor: Safis Editing

Proofreader: Esha Banerjee

Indexer: Hemangini Bari

Production Designer: Gokul Raj S.T

DevRel Marketing Coordinator: Sonia Chauhan

First published: July 2024

Production reference: 1120724

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK

ISBN: 978-1-83508-268-3

www.packtpub.com

To my amazing daughter Emma and my wonderful wife Diana. They say books are written in solitary, but that is entirely untrue. I would not have been able to write this if it hadn’t been for your support. This work is as much yours as it is mine. I would not have been the man I am today if it weren’t for you. Thank you.

- Dennis

Contributors

About the author

Dennis Vroegop is a programmer, no matter what his business card states. He has been programming computers since the early 1980s and still gets a kick whenever he sees his software running. After graduating with a degree in Business Informatics, he has worked in many roles over the years while retaining his passion for developing great software. These days, he works as an interim IT manager or CTO, helping companies get their software development in shape and making the developers happy about their work again.

He has been awarded the Microsoft MVP Award every year since 2006. In that role, he has been working with the C# team in Redmond on design sessions and has helped shape the language (a little bit). Dennis is a sought-after international speaker and public figure who is always ready to teach new generations about programming. Apart from his computer-related activities, Dennis plays the guitar and sings in a classic rock cover band named The Total Amateurs, which says all you need to know about their skills.

Dennis lives with his wife, Diana, and they have a wonderful daughter, Emma.

About the reviewers

Ankit Srivastava is a seasoned Senior Developer at Walmart; boasting seven years of extensive experience in software development. He specializes in .NET Development, Windows Development, WPF, WCF, REST API, .NET Core, .NET Standard, Python, and Linux. Ankit earned his Bachelor of Technology degree in Information Technology from Harcourt Butler Technological Institute and holds certifications in C#, C/C++, Python, Linux, Java, and SQL. Throughout his career, he has made significant contributions to diverse domains, including Semiconductors, Automotive, Storage, Chemical Heat Exchangers, and Health and Wellness.

Sarita Nag earned her master’s degree in Computer Application from KIIT University, Orissa, India. She started her career as a Software Engineer at Thomson Reuters and since then she has worked for many multinational companies. She is currently working as a Senior Developer at Fiserv, New Jersey, USA.

Sarita has 10+ years of experience in the various phases of Software Development Life cycle (SDLC) and Agile methodologies. She has also worked in domains such as Tax and Accounting, Auditing, Financial, Communication, and Customer relationships.

She is passionate about taking on new challenges and learning new things. Besides Development and Coding, Sarita enjoys spending time with her family. Her hobbies include adventure sports and photography.

Table of Contents

Preface

Overview of Systems Programming

Let’s define systems programming

When is a system user-facing and when is it not?

A better definition

Using C# and .NET in systems programming

Higher-level languages for systems programming

Kernel mode and user mode

Why use .NET?

What is .NET anyway?

.NET, .NET Framework, .NET Standard – what is all this?

Programming languages – a choice to make

Now what?

Setting up your development environment

1

The One with the Low-Level Secrets

Technical requirements

What are low-level APIs, and how do they differ from higher-level abstractions?

Overview of .NET Core runtime components (CLR, BCL)

CLR

BCL

Using P/Invoke to call low-level APIs

Dealing with errors

Issues when debugging code with low-level APIs

Error handling

Interoperability

Debugging tools

Compatibility and portability

Documentation and community support

Next steps

2

The One Where Speed Matters

Technical requirements

Setting up the stage

Accessibility

Hosting costs

Planned obsolescence

Energy usage

Which integer is the fastest?

The CTS

Value types and reference types

Classes and structs

Floating-point numbers

Where types live – the difference between value types and reference types

The stack and the heap

Boxing and unboxing

Hidden boxing and unboxing

Choosing the right data structures and algorithms

Arrays, Lists, and LinkedLists

Stacks and queues

HashSets and lists

SortedList, SortedDictionary, and Dictionary

Dictionary or last of tuples/objects

For versus ForEach

Strings

Use StringBuilder for concatenation

Interning strings

Use String.Concat or String.Join

Comparison

Preallocating StringBuilder

Writing unsafe code

Compiler optimizations

Aggressive optimization

The optimize flag

Next steps

3

The One with the Memory Games

Technical requirements

GC and its generations

The LOH

Finalizers

IDisposable

Memory-saving tips and tricks

Unsafe code and pointers in C#

Next steps

4

The One with the Thread Tangles

Technical requirements

Concurrency and threading – the basics

The beginnings of concurrency – the IRQ

Cooperative and preemptive multitasking

Threads in C#

Win32 threads

.NET threads

Tasks and Parallel Library – the TPL

Async/await

Task.Wait() and Task.Result

Synchronizing threads

Synchronization – how do we do that?

Synchronization with async/await

Canceling a thread

Thread-safe programming techniques

Lock()

Records

Avoid static members and classes

Using the volatile keyword

Concurrent collections in .NET

Next steps

5

The One with the Filesystem Chronicles

Technical requirements

File writing basics

FileStream

Even faster – Win32

File reading basics

Reading binary data

Directory operations

The Path class

The Directory class

The DirectoryInfo class

File system monitoring

Asynchronous I/O

The naïve approach

Using CancellationTokens

BufferedStream

File system security

Encryption basics

Symmetric encryption and decryption

Asymmetric encryption and decryption

File compression

Compressing some data

Decompressing some data

Serialization – JSON and Binary

JSON serialization

Binary serialization

Next steps

6

The One Where Processes Whisper

Technical requirements

Overview of IPC and its importance in modern computing

Windows Messages

A sample

Working with pipes for local IPC

Named pipes

Anonymous pipes

Using sockets to establish network-based IPC

Networking 101

A TCP-based chat app

UDP

Using shared memory to exchange data between processes

Overview of RPCs and how to use them for IPC

JSON RPC

Overview of gRPC and how to use it for IPC

Differences between JSON RPC and gRPC

Next steps

7

The One with the Operating System Tango

Technical requirement

The Windows Registry

What is the Windows Registry?

How to access and store data with the Windows Registry

Comparing the Windows Registry to JSON settings files

Worker Services

Docker support

Dissecting the Worker Service

Controlling the lifetime of the service

Wrapping up Worker Services

WMI

How to use WMI

Reading the CPU temperature

Reading the BIOS

Controlling the Windows Update service

Watching USB devices

Registry and WMI – risks and how to avoid them

The Windows Registry

Potential risks when dealing with WMI

Next steps

8

The One with the Network Navigation

Technical requirements

The fundamentals

A walk down the OSI layers

Exploring the System.Net namespace

Understanding HTTP/HTTPS

FTP

Email protocols

Working with the System.Net.Sockets namespace

Steps to take when using sockets

IPv4 and IPv6

Looking up time with sockets

Async, non-blocking networking

Making asynchronous calls

Networking performance

Connection pooling

Caching

Compression and serialization

Keep-alive connections

Networking errors and time-outs

Using the HTTPClient wisely

Implementing retries with Polly

The circuit breaker pattern

Validating network availability

Monitoring and logging

Next steps

9

The One with the Hardware Handshakes

Technical requirements

Connecting to serial ports

The path to the hardware

Why do we care?

A word about parity, data sizes, and stop bits

Working with an Arduino

Receiving serial data with .NET

Faking a serial device

Making it foolproof

Reasons things go haywire

Hardening your code

Next steps

10

The One with the Systems Check-Ups

Technical requirements

Available logging frameworks

Default logger in .NET

NLog

Serilog

Comparing the logging frameworks

Monitoring your applications

Monitoring with Seq

Performance counters

Prometheus

Other platforms for monitoring

What you should be monitoring or logging

Next steps

11

The One with the Debugging Dances

Technical requirements

Introducing debugging

Debugging and profiling – an overview

Debugging

Profiling

Debugging 101

Debug builds versus Release builds

Breakpoints

Debug windows

Diagnostic Tools

Debugging multithreaded and asynchronous code

Parallel Watch

Debugging deadlocks with Parallel Stacks and Thread windows

Profiling application performance

The prime application

Profiling in Visual Studio

Benchmarking different solutions

Other tools

Debugging tools

Profiling tools

Next steps

12

The One with the Security Safeguards

Technical requirements

Security for system programmers

What could happen if we have a vulnerability?

How to protect yourself

Working with strings

Protecting settings

Reading encrypted data

Where are the keys?

Handling strings in memory

Using key management

Using the Azure Key Vault

Using environment variables

Using the right privilege level

Admin-level scenarios

Impersonating as an admin

How to transmit network data securely

How HTTPS works

Certificates and certificate authorities

Creating a development certificate

Securing TCP streams

Next steps

13

The One with the Deployment Dramas

Technical requirements

From development to production

Publishing and file copy

Publish using Visual Studio

Publishing using the CLI

Using Azure DevOps and GitHub

Deploying to Azure

Enabling continuous integration in Azure DevOps

Enabling CI from GitHub

Building installers with Visual Studio

Building a simple installer

Writing a Custom Action

Incorporating the custom action in the setup

Using Docker

Adding Docker support to your background worker

Deploying your Docker images

Production-ready Docker repository

Next steps

14

The One with the Linux Leaps

Technical requirements

An overview of Linux

A short history of Linux

What is Linux?

A quick primer to use Linux

Basic commands

Elevated privileges

Developing for Linux

Installing .NET on Linux

Running a .NET background worker on Linux

Make your code cross-platform

How code can help you

Writing services for Linux

The service description

Installing the service

Uninstalling the service

Handling signals

Summing up

Let’s recap

Index

Other Books You May Enjoy

Preface

Most people think of graphical user interface (GUI) applications when they think of software. Software is the code that the user interacts with. But these days, that’s not true anymore. All modern applications, web servers, web applications, and mobile solutions mostly rely on hidden, unseen system software. This is the software that is built for other software. It is dormant until needed, then it does its job and goes back to sleep. These programs are the unsung heroes of our ecosystem, doing the work in the background. At the same time, GUI systems stand in the limelight. However, do not underestimate these hard-working systems: they must be extremely fast, reliable, and safe. Therefore, they are essential to good working systems and are hard to write. This book teaches you all you need to know to write these applications.

Who this book is for

People writing systems software are not junior developers. Ideally, you have a couple of years of experience developing software with C# and .NET. I will not explain what a variable is or how a while-loop differs from a for-loop. You know how to use NuGet. If I ask you to switch from Debug to Release mode in Visual Studio, you know what I am asking you to do.

But I do not expect you to know what instructions a CPU uses. I will explain those when we reach that point in the book. So there is no need to be on that low level just yet.

This book is for people who want to write system software. System software is software that is usually not visible to the regular user. However, it is essential to the good working of the complete software ecosystem running on your systems.

This means that you must have a passion for programs that run fast and are stable. This also means that the software we write is not the easiest to maintain: readability often decreases as performance increases. This is not for the faint-hearted: writing this kind of software is hard-core development. But if you are curious about how your programs really work deep inside the heart of the machine, this is the book for you.

The lessons learned here can, of course, be applied to all sorts of projects. Performance and stability can benefit all programs. So, if you are ready to take your C# and .NET skills to the next level, follow along!

What this book covers

Overview of Systems Programming sets the stage and explains what systems programming is all about.

Chapter 1

, The One with the Low-Level Secrets, dives into the low-level APIs, the BCL and CLR, and how to use Win32 APIs.

Chapter 2

, The One Where Speed Matters, examines how to make your software perform as fast as possible.

Chapter 3

, The One with the Memory Games, talks about memory handling, the garbage collector, and how to be as memory efficient as possible.

Chapter 4

, The One with the Thread Tangles, looks at threads and asynchronous programming.

Chapter 5

, The One with the Filesystem Chronicles , teaches input/output, file handling, encryption, and compression of files.

Chapter 6

, The One Where Processes Whisper, talks about how to make programs communicate on one machine or over a network.

Chapter 7

, The One with the Operating System Tango, dives into the operating system’s services and how to use them.

Chapter 8

, The One with the Network Navigation, discusses everything we need to know about networking in your application, both as a server and a client.

Chapter 9

, The One with the Hardware Handshakes, deals with connecting to outside hardware and communicating with other devices.

Chapter 10

, The One with the Systems Check-Ups, talks about logging and monitoring your software.

Chapter 11

, The One with the Debugging Dances, is all about debugging your software.

Chapter 12

, The One with the Security Safeguards, talks about the security of your software.

Chapter 13

, The One with the Deployment Dramas, teaches you how to deploy your software to the production machines.

Chapter 14

, The One with the Linux Leaps, discusses the operating system that most of our software will run on: Linux.

To get the most out of this book

I use Visual Studio 2022 as the main software development tool in this book. It is advisable for you to have a working knowledge of this, including creating console applications, class libraries, and worker services. You do not need to know what a Worker Service is as long as you can create a default one.

Each chapter might have software that you may want to try out. You’ll find the details explained in the Technical requirements section of the concerned chapter.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Systems-Programming-with-C-Sharp-and-.NET

. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/

. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: One of them is the Share option. We have set it to FileShare.Delete

A block of code is set as follows:

using var serialPort = new SerialPort(

    COM3,

    9600,

    Parity.None,

    8,

    StopBits.One);

serialPort.Open();

try

{

    serialPort.Write([42],0, 1);

}

finally

{

    serialPort.Close();

}

Any command-line input or output is written as follows:

docker tag image13workerfordocker:dev localhost:5000/image13workerfordocker:dev

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: Compact object representations: Sometimes, you can save some memory by smartly combining data into other data structures.

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected]

and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata

and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected]

with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com

.

Share Your Thoughts

Once you’ve read Systems Programming with C# and .NET, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page

for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/978-1-83508-268-3

Submit your proof of purchase

That’s it! We’ll send your free PDF and other benefits to your email directly

Overview of Systems Programming

So, you want to learn about systems programming in .NET, using C#. At least, I assume you want to learn that; you probably read the title of this book and decided that this was a good match. Maybe you have dived into systems programming a bit and want to get better at it. Or, perhaps you haven’t touched that subject and want to start. Or, maybe you picked the wrong book. If the latter is the case, I hope you still have your receipt so you can return this book and get something else. For all others: welcome!

Let’s define systems programming

Before we go into the nitty gritty details of systems programming, we need to set the stage. We need to have a common understanding of a couple of things. For instance, what does the term systems programming even mean? What is it for? Who is it for?

Let me get started with a definition.

Systems programming is the programming of systems. That might technically be correct, but I don’t think it helps us move forward.

Let us break it down: what is a system?

That one is easy. We have been building systems for ages, so we understand what we mean by a system.

Let me show you one definition:

A system is a set or arrangement of things that are related or connected so as to form a unity or organic whole. It is a collection of components or parts that interact with each other to function. This term is used in various fields such as physics, biology, computer science, and business management, each with slightly different connotations.

Great. But this definition is a bit broad. We might want to focus on computer science or software development. No problem; there are several definitions to choose from as well:

A system is a collection of software components that interact to perform a specific function or set of functions.

That is a lot better. If we dive into this a bit further, we can distinguish between different groups of systems:

Software systems: This is an integrated set of software components that work together to carry out a specific function or set of functions. Those components can be a database server, micro-services, and a frontend. Those components form the complete system, such as a CRM system, source control repository system, and others like that.

Operating systems (OSs): You probably know what an OS is. I think you have seen that term so often that there is a fair chance you didn’t even realize it is a system. But it most definitely is an OS that contains many parts and components, such as drivers, tools, helpers, and logs. Together, they deliver a system you as a user can use to run your software on, independently of the hardware.

Distributed systems: We often refer to loosely connected components on a network as a distributed system. Each part is isolated from the others, but they must collaborate to achieve something worthwhile. For example, Azure DevOps runs on many different servers in the Azure cloud. All the components run potentially on different servers and machines, and these components can even be running in different parts of the world. However, they work together to form a complete solution for the end user.

Embedded systems: An embedded system is usually a combination of hardware and software. The components are tightly coupled with each other. Developers usually write the software to match specific specifications so it uses the hardware best. Think, for instance, about the systems in your car. If you have a reasonably recent car, you undoubtedly have an entertainment system on board. The word system in entertainment system is a bit of a giveaway: it consists of many distinct components. There is very likely a device that can collect electromagnetic waves from the air (we call that a radio). That device is connected to some software that interprets those waves and turns them into an electrical signal to feed the speakers. Next to that, a component shows you, as the user, what you are listening to. I am sure you can find a lot of other systems in your car and probably in your TV, your phone, or your refrigerator.

There are many more examples, but I hope you see that a system always consists of individual components that are not useful on their own but, when combined, deliver a solution to a problem.

But hold on. We are not done yet.

Given these definitions and examples, you might think that the art of systems programming is just the programming of these systems, and you would not be wrong. But that is, in general, not what systems programming means. It most certainly is not what I mean by that.

Very, very roughly, we can divide software into two types:

User-facing software: This is software written to be used by people. It has a user interface (UI) with buttons, lists, labels, and more. People interact with the software by using various means of input modalities.

Software-facing software: This is software designed to be used by other software. There are no UIs since we have no users. We could say other components are the users, but when I say users, I mean people. Software-facing software interacts with other components through APIs, RPC (Remote Procedure Calls) calls, file transfer, and many other ways. No humans are involved in this.

It is the second type we are most interested in here in this book – software meant to be used by other software.

When is a system user-facing and when is it not?

It is not always clear when people are the primary users of some code or when other processes are. We could be very rigorous and say that anything with a UI is user-oriented; anything else is systems-oriented. That will make life easier for us if we want a clear definition. However, in the real world, the boundaries tend to blur.

Let me give you an example. Have a look at this Visual Studio Solution:

Figure 0.1: Solution Explorer with the Calculator project

We have a very, very simple solution here. It has a main program called MyAwesomeCalculator that contains the main code. This is the entry point of our app, using the console as the UI. All logic is in the MathFunctions class library. This is where the magic happens.

If we go back to our definition of Systems programming, we could say that writing the MathFunctions class library is part of Systems programming. After all, no user will ever interact with the classes and interfaces in that library. It is the code in MyAwesomeCalculator that actually uses it.

Great! This means writing the MathFunctions library is systems programming! Well, not so fast. We might come to another conclusion if we look at the sequence diagram that explains the flow. Figure 0.2 shows this sequence diagram.

Figure 0.2: Sequence diagram for our calculations

As you can see in Figure 0.2, the user initiates an operation: they want to add up two numbers. They enter it in the UI of the Main class. The Main class then instantiates an instance of the Adder class. After that creation, the Main class calls the AddUp(a,b) method. The result is passed back to the Main class and shown to the user. After all this, we could discard the Adder instance.

Great. Where are the boundaries? If we look at it this way, we could say that the code in Adder and, thus, in the MathFunctions library is immediately tied to user actions. So, it is user-facing code instead of systems-facing code.

I still like to use the question of who is using the code to determine what kind of software we are writing. But apparently, this is not enough. We need to go a bit deeper.

The code in MyAwesomeCalculator and MathFunctions are in separate assemblies. The user interacts with one assembly; the other is accessed through code only. But they can still be seen as one. If we run the application, the runtime creates AppDomain for us.

AppDomain in .NET is different than AppDomain in .NET Framework. The latter had more ways to isolate code from each other. That was nice, but it was a typical Windows feature. That did not translate well to other platforms. So, to make .NET applications run on other platforms, they needed to redesign this. This results in AppDomain being less restrictive than it used to be. Still, AppDomain remains a logical boundary between different processes. Code runs in one app domain and cannot access other app domains directly.

Here, we have another clue: our MyAwesomeCalculator app and the associated MathFunctions assembly all run in the same AppDomain. To the OS, they are one. Since we decided that actual people use the Main method, the same applies to all other pieces of code in that particular AppDomain.

Let’s rewrite our solution a bit. See the following screenshot.

Figure 0.3: Our solution with a worker process

We removed the class library with the code that did all the work. Instead, we created a new project. That project is a worker process. Technically, I should have kept that class library and referenced that, but I wanted to keep things simple.

A worker process is a background process that runs all the time (not technically true, but for now, this is true enough). It just sits there doing nothing. Then, suddenly, something of interest happens, and it comes to life, does its job, and goes to idle mode again.

As shown in Figure 0.4, the sequence diagram in this case is also slightly different.

Figure 0.4: Sequence diagram for the new revised architecture

MyAwesomeCalculator and the MathFunctionServices worker are now independent of each other. They each run in their own AppDomain. When the user wants to perform the calculation, they enter this in the UI, which invokes the service. The Worker class picks up the command, creates an instance of the Adder class, calls the AddUp method, and then calls the MyAwesomeCalculator again with the results.

As you can see, the calls between all classes are synchronous (designated by a line with a solid arrowhead) except for the call between Main and Worker. That is asynchronous (designated by a line and an open arrowhead).

That makes sense; the calculator cannot know whether the command has arrived or the service is listening. It just does a fire-and-forget, crosses its digital fingers, and hopes for the best.

This is more like it. This is genuinely writing software used by other software (I am talking about MathFunctionServices here, not MyAwesomeCalculator).

I have not shown you how the code in Main calls Worker and how the result flows back from Worker to Main. After all, they are in separate app domains. So, they cannot share memory, right? That is correct. I did not show you that. But do not worry. I have a couple of chapters dedicated to this.

It is important to realize that MathFunctionServices does not have a UI in the ordinary sense of the word. No user ever touches this code. It lies there, dormant, until its services are required. If we compare that to the first example, we see the differences. That first example had all code loaded on the user’s demand, and it somehow all responded to the users’ actions.

A better definition

So, if we combine all of this, we can determine that systems programming is the art of writing components that can perform a function or a set of functions but interact only with other components.

That is what this book is all about. We will learn how to write software that is to be consumed by other software. That is a whole other way of looking at software, requirements, design considerations, and more compared to software meant for humans.

Writing software for software means other ways of thinking about communications, performance, memory usage, security, and so on. All those topics are covered here in this book. Now, you might say: But wait a minute. Software written for users should also keep performance in mind! You are right, but software communicating with software has unique needs.

Later chapters show how you can achieve the desired performance and explain why this is important. Let us agree that a component, potentially called thousands of times per second, could use more thought about performance than a screen with a button that a user might click once an hour. I am exaggerating here, but I am sure you get the point.

The same applies to memory consumption. I believe we should always write all software with memory consumption in mind. However, a component that gets used frequently by many other systems tends to be much more vulnerable to issues with memory than other software programs.

Performance and memory pressure are essential when we think about writing embedded systems. Embedded software usually runs on very limited hardware, so we have to try and take advantage of every trick in the book to get it running as fast as possible and using as little memory as possible.

As promised, we will spend much time looking at ways to communicate with these types of software.

To me, Systems programming is the purest form of software development. It is all about algorithms, tweaks, and trying out every trick in the book to get the most out of it. systems programming is the major league of software development. When you have this covered, all other software you write will also benefit from your newfound knowledge. What you learn when writing systems software will become second nature, and you will improve your overall software writing skills. Does this sound exciting? Then, let’s get started!

Using C# and .NET in systems programming

We already run into a problem. You most likely are a C# developer. Maybe you are a VB.Net developer. But no matter what language, you are a .NET developer. After all, that is what this book is about.

Traditionally, Systems programming is done in Assembly, C, and C++. Systems programming has always been the realm of hardcore developers who know the systems they are working on inside out. In the early 50s of the last century, people wrote systems software using switches. A switch in the up position meant a 1, and a switch in the down position meant a 0. These early computers had 8, 16, or even more switches that pointed to the memory address to read or write. Then, 8 switches represented all the bits in a byte for that memory address. Above these switches, there were little lights (no, not LEDS: that invention happened later). Those little lights, if illuminated, meant a 1 in that byte (and a 0 if not illuminated). That way, you could read the contents of that memory address.

Do not worry; that kind of low-level programming is not the topic of this book. If you are interested, there are good remakes of the original Altair 8800 that started a company called Microsoft. You can program that computer in this way: use the switches and lights on the front panel to enter your software. That is how Bill Gates and Paul Allen wrote their first software. But we have other tools at our disposal.

Since systems software relies on efficient, fast, and memory-aware code, people often use programming languages close to the metal. That usually means using language such as machine code – such as the switches I mentioned earlier. Assembly language is another language used, especially in the seventies and eighties of the last century. C and later C++ are other examples of languages that can take advantage of the specifics of the hardware. Most parts of Windows, for instance, are written in C.

However, systems developers do not restrict themselves to low-level languages only. Let me give you an example.

Higher-level languages for systems programming

In 1965, IBM published a manual called PL/I Language Specifications. C28-6571. This relatively obscure title is a fascinating read: it outlines the specifications of the PL/I programming language. PL/I, a sort of abbreviation for Programming Language One, is a higher-level programming language. It contains block structures to allow for recursion, many different datatypes, exception handling, and many other features we take for granted today. It truly was a high-level language. However, they used it to write parts of the early OSs inside IBM. Remember, this was in the sixties when every microsecond counted. Machines were extremely slow compared to modern systems, so they had to utilize every trick in the book to make things work. Yet, a high-level language was considered appropriate. That means there is no reason not to use a high-level language today, especially considering memory profilers’ compiler techniques and advantages.

Kernel mode and user mode

OSs and drivers are usually not built using .NET. The reason for this is that drivers and most parts of the OS run in kernel mode.

The CPU in your computer can run in two modes: kernel or system mode and user mode. User mode is where most of the applications run. The CPU shields the applications from using other memory or process spaces. The CPU protects the applications by placing them in a sandbox. That is precisely what you would want: it would be very undesirable for a program to snoop around in another application’s memory. The processor handles this level of security.

Kernel mode, however, does not have those limitations. Software running in kernel mode is less restricted, controlled, and trusted. That makes sense: parts of an OS should be able to run in all parts of the system, including in the space of other applications.

However, to run in kernel, the compiled code needs to have certain flags set, and the layout of the binaries should be very specific. That is the problem we face. Our C# code relies heavily on the .NET Runtime, and that runtime is not built to be used in Kernel mode. So, even if we could compile our code so that the OS would accept it, it still would not work due to the app not loading the runtime.

There are ways around this. There are ways to pre-compile and include the runtime classes in your binary. Then, you can modify that binary to run in kernel mode. However, the results may vary, and the whole thing would be unreliable. Unreliable code is the exact opposite of what a device driver or OS part should be, so we will not get into this in this book. It’s a hack, not a standard way of working.

Although this book does not deal with kernel-mode apps, I want to give you some insight. Especially since systems programming is usually programming close to the metal, so to speak, we are interacting with systems that are running in kernel mode.

Kernel mode is a mode in the CPU. A system can request the CPU to turn on kernel mode. If the code requesting it has the proper privileges, the CPU will do so, thus unlocking parts of the memory previously unavailable. The code does what it needs to do, and then the CPU returns to user mode. Since the code is still in memory doing all sorts of things, it is quite wrong to say an app is a kernel or user-mode app. Some apps can switch the CPU into that state, but the app is almost always running in mixed mode: most of the time, it is in user mode, sometimes kernel mode. Oh, and when I say CPU, I mean logical CPU. This toggling happens on that level, not on the chip itself (but it can also do that).

I have Adobe Creative Cloud installed on my machine. We all know Photoshop, Illustrator, and Premiere, but these apps are meant to be accessed through the Creative Cloud app. This app monitors the system and launches any app you need when you need it. It also updates the background and keeps track of your fonts, files, colors, and other things like that.

Whenever you read something like runs in the background, you might expect some systems programming going on, and indeed, there is.

For example, I get this image if I start Performance Monitor on my system and add the % Privileged Time and % User Time counters for the Adobe Desktop Service process.

Figure 0.5: Performance Monitor showing kernel and user times

The red line in Figure 0.5 shows how much time the Adobe Desktop Service spends in user time. The green line, however, shows how long the service is running in privileged time, and privileged time is just a fancy term for kernel time.

As you can see, this app is doing much work in the kernel time. Although I have to admit, I have no clue what it is doing there, but I am sure it is all for a good reason.

We will encounter kernel mode later in other chapters but we will not build apps that run in it.

Why use .NET?

So, we established that we cannot build an OS or a device driver in .NET. That might lead to the question: Can we use .NET for systems programming? The answer is a big yes. Otherwise, this would have been a very thin and short book.

Shall we have a look at our recently discovered definition of systems programming? Writing software used by other software, as a part of a bigger system that works together to achieve a certain goal. I have shortened the definition, but it is all about this.

Looking at it this way, we can use .NET to write that software. Better yet: I bet .NET is one of the best choices to do so.

.NET offers many advantages over plain C or even C++ (not the managed kind of C++, that is still .NET.)

Back in the day, when we used .NET-Framework-based applications, it would have been a bad idea to use that for systems programming. However, with the introduction of the latest versions of .NET, many disadvantages have been taken care of. With many disadvantages out of the way, .NET-based systems are a viable choice for these kinds of systems.

C and C++ still are excellent languages for low-level systems code. However, C# and .NET Core have their advantages as well.

This table lays out the differences.