Optimized Computing in C++: Mastering Concurrency, Multithreading, and Parallel Programming

Optimized Computing in C++

Mastering Concurrency, Multithreading, and Parallel Programming

Copyright © 2024 by NOB TREX L.L.C.

All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law.

Contents

1 Introduction to High-Performance Computing in C++

1.1 The Evolution of High-Performance Computing

1.2 Understanding Performance in Computing

1.3 The Role of C++ in High-Performance Computing

1.4 Challenges and Goals of High-Performance Computing

1.5 Introduction to Concurrency, Multithreading, and Parallel Programming

1.6 Basic Concepts of Concurrency and Parallelism

1.7 Hardware Considerations for High-

Performance Computing

1.8 Software and Tools Ecosystem for HPC in C++

1.9 Overview of C++ Features Relevant to HPC

1.10 Real-World Applications of High-Performance Computing

1.11 Setting the Stage for Advanced HPC Topics in C++

2 Fundamentals of Concurrency in C++

2.1 Concurrency vs. Parallelism: Understanding the Difference

2.2 Basic Building Blocks of C++ Concurrency

2.3 Creating Threads in C++

2.4 Passing Data to Threads and Returning Results

2.5 Joining and Detaching Threads

2.6 Thread Safety and Data Races

2.7 Mutexes and Locks: Basic Synchronization Primitives

2.8 Condition Variables for Synchronizing Operations

2.9 Atomic Operations and Memory Models in C++

2.10 Deadlocks: Detection, Prevention, and Avoidance

2.11 Futures and Promises for Asynchronous Programming

2.12 Task-Based Concurrency with std::async

3 Deep Dive into C++ Threads and Multithreading

3.1 Introduction to Threads in C++

3.2 Thread Management and Lifecycle

3.3 Advanced Thread Synchronization Techniques

3.4 Design Patterns for Multithreading

3.5 Thread Pools and Work Stealing

3.6 Implementing Safe Concurrent Data Structures

3.7 Parallel Algorithms in the C++ Standard Library

3.8 Thread Local Storage for Data Isolation

3.9 Best Practices for Error Handling in Multithreaded Environments

3.10 Performance Considerations and Tuning

3.11 Integrating Multithreading with Other Parallel Programming Models

3.12 Case Study: Designing a Multithreaded Application in C++

4 Parallel Programming Models and Patterns in C++

4.1 An Overview of Parallel Programming Models

4.2 Functional Parallelism in C++

4.3 Data Parallelism: Theory and Practice

4.4 Task Parallelism and Decomposition Strategies

4.5 Pipeline Parallelism for Throughput Optimization

4.6 The Fork-Join Model and Its Implementation in C++

4.7 Using C++17 and C++20 Parallel Algorithms

4.8 SIMD and Vectorization: Leveraging CPU Architectures

4.9 GPGPU Programming with C++ for Massive Parallelism

4.10 Distributed Computing with MPI and C++

4.11 Design Patterns for Parallel Programming in C++

4.12 Performance Optimization Techniques in Parallel Programs

5 Synchronization and Communication Between Threads

5.1 Understanding Synchronization: The Basics

5.2 Mutex Types and Lock Management in C++

5.3 Lock-Free Programming and Atomic Operations

5.4 Condition Variables and Producer-Consumer Problems

5.5 Barriers and Latches for Synchronization Points

5.6 Semaphores: Usage and Applications

5.7 Message Passing for Inter-Thread Communication

5.8 Signal Handling and Event-Driven Programming

5.9 Designing Reliable and Scalable Synchronization Primitives

5.10 Avoiding Common Pitfalls: Deadlocks, Starvation, and Livelocks

5.11 Optimizing for Performance: Balancing Concurrency and Resource Use

5.12 Case Studies: Solving Complex Synchronization Challenges

6 Optimizing Concurrent Data Structures and Algorithms

6.1 Introduction to Concurrent Data Structures

6.2 Atomic Operations and Their Impact on Data Structures

6.3 Design Principles for Concurrent Data Structures

6.4 Concurrent Queues and Stacks: Implementation and Use Cases

6.5 Building and Using Lock-Free Data Structures

6.6 Optimizing Concurrent Hash Tables

6.7 Scalable Memory Allocation for Concurrent Structures

6.8 Concurrent Trees: Types, Implementations, and Applications

6.9 Vectorization and Parallelism in Algorithm Design

6.10 Dynamic Workload Balancing for Concurrent Algorithms

6.11 Benchmarking and Profiling Concurrent Data Structures

6.12 Case Studies: Performance Optimization in Real-World Applications

7 Concurrency in Modern C++: The latest features and standards

7.1 Overview of Concurrency Features in Modern C++

7.2 C++11 and C++14: The Foundation of Modern Concurrency

7.3 C++17 Enhancements in Concurrency

7.4 Exploring C++20’s Concurrency and Parallelism Features

7.5 Understanding the C++23 Proposals for Concurrency

7.6 The std::jthread in C++20 for Easier Thread Management

7.7 Cooperative Interruption with std::

stop_token and std::stop_source

7.8 Simplifying Concurrency with C++20 Coroutines

7.9 Latches and Barriers in C++20 for Multi-Thread Synchronization

7.10 Atomic Smart Pointers and Other Atomic Operations in C++20

7.11 Executors in C++: The Future of Task Execution

7.12 Concurrency Best Practices with Modern C++

8 Debugging and Profiling Concurrent and Parallel C++ Programs

8.1 The Challenges of Debugging Concurrent and Parallel Programs

8.2 Understanding the Tools: Debuggers and Profilers

8.3 Detecting and Addressing Data Races and Deadlocks

8.4 Debugging with Condition Variables and Atomic Operations

8.5 Profiling for Performance: Identifying Bottlenecks

8.6 Using Sanitizers to Detect Concurrency Issues

8.7 Visualizing Program Execution with Trace Tools

8.8 Optimizing Thread Utilization and Synchronization

8.9 Memory Management Issues in Concurrent Programs

8.10 Leveraging Testing Techniques for Concurrent Applications

8.11 Case Study: Debugging a Real-World Multithreading Issue

8.12 Best Practices for Developing and Debugging Concurrent Code

9 High-Performance Computing using GPU with C++

9.1 Introduction to GPU Computing and Its Advantages

9.2 Understanding GPU Architecture and Execution Model

9.3 Setting Up Your Environment for GPU Programming with C++

9.4 Basic Concepts of CUDA and OpenCL in C++

9.5 Writing Your First GPU Program with C++

9.6 Memory Management on GPUs

9.7 Optimizing GPU Kernels for High Performance

9.8 Understanding and Utilizing Warp Programming

9.9 Integrating GPU Computing into Existing C++ Applications

9.10 Debugging and Profiling GPU Programs

9.11 Advanced Topics: Dynamic Parallelism and Unified Memory

9.12 Case Studies: Real-World Applications Using GPU Computing

10 Scalable Parallel Computing with MPI in C++

10.1 Introduction to MPI and Its Role in Parallel Computing

10.2 Setting Up an MPI Environment for C++ Development

10.3 Basic MPI Concepts: Communicators, Ranks, and Tags

10.4 Point-to-Point Communication in MPI

10.5 Collective Communication and Data Distribution

10.6 Synchronization and Barriers in MPI

10.7 Advanced MPI Features: Groups, Communicators, and Derived Data Types

10.8 Optimizing MPI Applications: Techniques and Best Practices

10.9 Hybrid Programming: Combining MPI with Threads

10.10 Debugging and Profiling MPI Applications

10.11 Case Studies: Scalable Parallel Algorithms with MPI

10.12 Future Directions in MPI and Parallel Computing

Introduction

High-Performance Computing (HPC) is a domain of computing that involves aggregating computing power in a way that delivers significantly higher performance than one could get out of a typical desktop computer or workstation. The objective is to solve large problems in science, engineering, or business. In HPC, C++ has emerged as a leading programming language, thanks to its combination of high-level abstraction and low-level control over system resources. This book, Optimized Computing in C++: Mastering Concurrency, Multithreading, and Parallel Programming, is designed to serve as a comprehensive guide aimed at imparting a deep understanding of how to utilize C++ for high-performance applications effectively.

The book is formulated with the specific objective of bridging the gap between the conventional C++ programming approaches and the demanding requirements of High-Performance Computing. It begins with an exploration of the theoretical foundations of HPC and gradually advances towards more complex concepts such as concurrency, multithreading, and parallel programming in C++. The content is meticulously organized to introduce readers to the basics before moving into sophisticated programming models, patterns, synchronization mechanisms, and optimizing techniques that are essential for developing scalable, efficient, and reliable HPC applications in C++.

The substance of this book carefully balances theoretical knowledge with practical application. It is enriched with code snippets, examples, and case studies to foster an engaging learning experience. Topics such as GPU programming, debugging, and profiling concurrent and parallel C++ programs are particularly emphasized to equip readers with the skills necessary to tackle real-world challenges in HPC. Furthermore, modern C++ standards (C++11, C++14, C++17, and C++20) and their relevance to HPC are thoroughly explored to ensure readers are well-versed in the latest advancements in the language that enhance concurrent and parallel programming paradigms.

This book is tailored for a variety of readers, ranging from intermediate C++ programmers seeking to expand their knowledge in HPC, to experienced software developers aspiring to optimize their applications for high performance. Additionally, it serves as a valuable resource for researchers, scientists, and engineers who employ C++ in their computational work, providing them with in-depth insights and strategies for leveraging concurrency, multithreading, and parallelism in their projects.

In essence, Optimized Computing in C++: Mastering Concurrency, Multithreading, and Parallel Programming stands as a pivotal educational resource. It endeavors not only to disseminate knowledge but also to inspire innovation and proficiency in the use of C++ for High-Performance Computing. Through this book, readers will gain the confidence and competence to design and implement powerful C++ applications that can leverage the full potential of modern hardware architectures, thereby unlocking new possibilities in their respective fields of study or work.

Chapter 1

Introduction to High-Performance Computing in C++

High-Performance Computing (HPC) has become indispensable for tackling complex computational problems across various scientific, engineering, and business domains. C++, with its fine balance of high-level abstraction and low-level control, is particularly suited for developing scalable, efficient, and reliable HPC applications. This chapter lays the foundational knowledge necessary to understand the role and potential of C++ in the HPC landscape, discussing the evolution of HPC, the challenges it addresses, and the significance of concurrency, multithreading, and parallel programming practices in maximizing computational performance.

1.1

The Evolution of High-Performance Computing

High-Performance Computing (HPC) has undergone significant transformations since its inception, evolving in response to both technological advances and expanding computational demands across diverse fields. This evolution can be traced back to the early supercomputers, designed and developed in the 1960s, which marked the beginning of an era dedicated to solving complex scientific and military calculations that were beyond the capabilities of conventional computing systems of the time.

Initially, these supercomputers were characterized by their use of highly specialized, custom hardware and architectures. They were singular, powerful machines such as the Control Data Corporation’s CDC 6600, which in 1964 became known as the fastest computer in the world. The CDC 6600’s speed was due to its unique architectural design, which included a central processing unit (CPU) capable of handling ten million floating-point operations per second (10 MFLOPS). This era emphasized the power of individual, centralized computing units to perform large-scale calculations.

As technology progressed, so did the design and capabilities of HPC systems. The 1980s saw the introduction of vector processors, which greatly enhanced the computer’s ability to perform mathematical calculations by applying a single operation to multiple data points simultaneously. This vector processing capability was a significant step toward modern parallel computing techniques, allowing for substantial increases in computational speed and efficiency.

The concept of parallel computing, wherein multiple calculations are carried out simultaneously, marked a pivotal moment in the evolution of HPC. It was facilitated by advances in both software and hardware, including the development of sophisticated parallelizing compilers and the advent of multi-processor architectures. This period also witnessed the rise of massively parallel processing (MPP) systems, characterized by their ability to harness hundreds or even thousands of processors to work in tandem on a single problem, dramatically scaling computational power.

In parallel to these hardware advancements, the 1990s and early 2000s saw significant progress in network technologies and distributed computing, leading to the emergence of cluster computing and grid computing. Clusters, comprising groups of standard, off-the-shelf computers networked together to function as a single cohesive unit, became a cost-effective alternative to traditional supercomputers. Similarly, grid computing leveraged the internet to connect multiple clusters and other resources dispersed geographically, further expanding the computational capabilities available to scientists and engineers.

The introduction of General-Purpose computing on Graphics Processing Units (GPGPU) marked another transformative phase in HPC. Initially designed for rendering graphics in video games, GPUs were repurposed for general computing tasks, capitalizing on their parallel processing capabilities. This innovation significantly lowered the cost and physical footprint of HPC solutions while providing unparalleled acceleration for specific types of parallel computations.

Today, HPC is characterized by a heterogeneous computing environment that integrates various technologies, including traditional CPUs, GPUs, and specialized accelerators like Field-Programmable Gate Arrays (FPGAs) and Tensor Processing Units (TPUs). This hybrid approach allows for the optimization of different tasks according to the most suitable computing resource, highlighting the adaptability and continuous innovation underlying the field of HPC. Additionally, the advent of cloud computing has democratized access to HPC resources, making supercomputing capabilities available to a broader range of users and applications.

The endless pursuit of computational speed, efficiency, and scalability has driven the evolution of HPC from monolithic supercomputers to the diverse, interconnected, and adaptable computing landscapes of today. This journey highlights the collective ingenuity in overcoming the limitations of existing technologies and the unwavering commitment to addressing some of the most challenging computational problems faced by humanity.

1.2

Understanding Performance in Computing

In High-Performance Computing (HPC), performance is a multifaceted metric, crucial for evaluating how effectively a computing system accomplishes its task. Generally, performance can be assessed through various dimensions, including but not limited to, computation speed, efficiency, scalability, reliability, and energy consumption. Each of these dimensions contributes to the overall performance of the system in fulfilling the computational requirements of advanced scientific, engineering, and business applications.

Computation speed, often measured in floating-point operations per second (FLOPS), remains one of the most cited benchmarks for evaluating the performance of an HPC system. It quantifies the number of arithmetic operations a system can perform in a second, providing a raw gauge of its computational power. However, focusing solely on FLOPS can overlook other essential performance considerations, such as how efficiently the system utilizes its computational resources.

Efficiency in computing performance relates to the ratio of the actual performance to the theoretical maximum performance that the hardware architecture could achieve. It reflects how well the system manages its computational and memory resources, indicating the level of waste or underutilization. High efficiency is essential for sustainable and cost-effective HPC operations, especially in scenarios involving extensive data processing tasks or simulations.

Scalability, another critical performance metric, measures a system’s ability to maintain or increase performance proportionally to the addition of resources. It is pivotal in HPC environments due to the ever-growing complexity and size of computational problems. A scalable system can exploit parallel processing and distribute workloads efficiently across multiple computing nodes, ensuring enhanced performance as the system scales.

Reliability in the context of HPC refers to the system’s ability to operate without failure over its expected operational lifetime. This aspect of performance is paramount, given the high costs associated with downtime or incorrect results in critical applications. Mechanisms to ensure reliability include error detection and correction, fault tolerance techniques, and robust system design.

Lastly, energy consumption has emerged as a significant concern in HPC due to the environmental and economic implications of operating large-scale computing facilities. As the demand for computational power grows, so does the necessity for energy-efficient designs that minimize power usage while maintaining high performance. Optimizing energy consumption involves hardware innovations, efficient algorithms, and software solutions that reduce the energy footprint of computing operations.

In this section, we have discussed multiple dimensions of performance in computing, highlighting their relevance in the context of High-Performance Computing. It is imperative to consider these various facets when designing, implementing, and evaluating HPC systems and applications. Achieving optimal performance in an HPC environment is not merely about maximizing computation speed; it encompasses efficiency, scalability, reliability, and responsible energy usage. As we proceed, we will delve deeper into how C++ and its features can be leveraged to address these performance dimensions, setting the stage for the development of efficient, scalable, and reliable HPC applications.

1.3

The Role of C++ in High-Performance Computing

C++ emerges as a paramount language in the realm of High-Performance Computing (HPC) due to its intrinsic support for low-level memory manipulation and high-level object-oriented programming paradigms. This unique amalgamation enables developers to write code that closely interacts with the hardware while maintaining the abstraction necessary for complex, large-scale application development. The role of C++ in HPC can be dissected into several critical areas, including performance optimization, code efficiency, and the extensive ecosystem of libraries and tools that support parallel programming.

Performance optimization in C++ is significantly facilitated by its language constructs, such as inline functions and templates, which allow for compile-time polymorphism. This diminishes the overhead typically associated with dynamic polymorphism, making C++ especially suitable for performance-critical applications. The use of inline functions, for example, eliminates the function call overhead, thereby streamlining the execution flow, especially in tight loops which are pervasive in computational science applications.

1

inline

double

square

(

double

x

)

{

2

return

x

*

x

;

3

}

Another aspect where C++ aids in HPC is through its explicit control over memory management. Unlike languages with automatic garbage collection, C++ grants developers the ability to precisely manage memory allocation and deallocation. This control is crucial in HPC, where efficient memory use is paramount due to the immense data sizes and the need for rapid access and computation.

1

double

*

largeArray

=

new

double

[1000000];

2

//

Perform

operations

3

delete

[]

largeArray

;

In addition to manual memory management, the introduction of smart pointers in modern C++ (C++11 and onwards) simplifies resource management while ensuring memory safety. Smart pointers, like std::unique_ptr and std::shared_ptr, handle automatic resource deallocation, thereby reducing common memory management errors, such as leaks and dangling pointers.

1

#

include

<

memory

>

2

std

::

unique_ptr

<

double

[]>

largeArray

(

new

double

[1000000])

;

The extensive library ecosystem in C++ significantly propels its application in HPC. Libraries such as Intel Threading Building Blocks (TBB), OpenMP, and MPI (Message Passing Interface) facilitate the development of parallel and distributed applications by abstracting complex tasks, such as thread management and inter-process communication. These libraries not only speed up the development process but also enhance the application’s performance on multi-core and distributed systems.

Furthermore, the ongoing evolution of the C++ language, with standards adding features like lambda expressions, auto keyword, and concurrency API, continuously enhances its suitability for HPC applications. These features enable more expressive, clean, and maintainable code, which is crucial for the development of scalable and complex HPC software.

1

std

::

vector

<

int

>

vec

=

{1,

2,

3,

4,

5};

2

std

::

for_each

(

vec

.

begin

()

,

vec

.

end

()

,

[](

int

&

x

)

{

3

x

=

square

(

x

)

;

4

})

;

In terms of real-world application, C++ is used extensively in simulation, modeling, and data analysis applications across numerous fields such as physics, chemistry, biology, and financial modeling. The computational requirements of these applications are immense, dealing with large data sets and requiring significant processing power. C++’s efficiency, combined with its ability to harness the full potential of underlying hardware, makes it an ideal choice for such demanding computational tasks.

To encapsulate, C++ stands at the forefront of HPC owing to its balance between low-level hardware control and high-level programming constructs. Its efficiency, coupled with a robust ecosystem of libraries and ongoing language evolution, empowers developers to tackle the myriad challenges posed by HPC applications. As computational demands continue to escalate, the role of C++ in driving forward scientific and industrial advancements is undeniably pivotal.

1.4

Challenges and Goals of High-Performance Computing

High-Performance Computing (HPC) operates at the frontier of computational ability, addressing some of the most complex and data-intensive problems across various disciplines. While HPC represents a powerful tool for scientific discovery and innovation, it also brings forth a unique set of challenges that must be navigated to unlock its full potential. Concurrently, the overarching goals of HPC reflect its ambition to not only surmount these challenges but also to anchor computational practices that are both efficient and sustainable.

One of the primary challenges in HPC is the scalability of algorithms and applications. As the size of the computational problems grows, it becomes imperative for applications to efficiently utilize an increasing number of processors or computing nodes. This requires algorithms that can scale their performance linearly or near-linearly with the addition of resources, a property known as strong scalability. However, achieving this level of scalability is inherently complex due to issues such as communication overhead between processors, memory access patterns, and algorithmic inefficiencies that become magnified at scale.

Addressing the memory bottleneck issue requires innovative approaches to memory management and access. Traditional single-threaded applications may not effectively leverage the memory hierarchy in multi-core and many-core systems, leading to significant bottlenecks.

Minimizing communication overhead is crucial for enhancing scalability. As the number of processors increases, the cost of data exchange and synchronization among them can escalate, undermining overall performance.

Algorithmic optimization for parallel environments is another critical challenge. Many algorithms need to be fundamentally redesigned to harness parallel architectures effectively.

Another significant challenge lies in energy efficiency. As HPC systems grow in complexity and size, their energy consumption escalates, posing sustainability concerns and increasing operational costs. Achieving high computation performance while minimizing energy use requires a holistic approach, including hardware innovations, energy-efficient coding practices, and algorithms optimized for power consumption.

The goals of High-Performance Computing are multifaceted, reflecting the diverse needs of the research, engineering, and business communities it serves. One such goal is to achieve exascale computing, which entails developing computing systems capable of performing at least an exaflop, or 10¹⁸ floating-point operations per second. Achieving this milestone would open new horizons for scientific modeling, simulation, and data analysis, enabling breakthroughs in fields such as climate research, genomics, and materials science.

Enhancing computational efficiency through the development of more sophisticated algorithms and software tools that can effectively utilize exascale and beyond capabilities.

Accelerating scientific discovery by providing the computational resources necessary to tackle unsolved problems in various domains.

Ensuring accessibility and usability for a broad range of users, from researchers and engineers to industry professionals, is essential for maximizing the impact of HPC.

To surmount these challenges and achieve its ambitious goals, the HPC community continues to innovate across various fronts. This includes advancements in parallel computing architectures, development of scalable and energy-efficient algorithms, and the creation of comprehensive software ecosystems that facilitate programming for HPC environments. Meanwhile, interdisciplinary collaboration among computer scientists, domain scientists, and engineers plays a critical role in driving the evolution of HPC and ensuring its continued relevance and impact in addressing the most pressing computational challenges of our time.

1.5

Introduction to Concurrency, Multithreading, and Parallel Programming

Let’s start with an exploration of the core concepts that underpin high-performance computing (HPC) in C++: concurrency, multithreading, and parallel programming. These concepts are pivotal for developing applications that efficiently leverage the available computational power to solve complex problems in reduced timeframes.

Concurrency in computing refers to the ability of a system to manage multiple tasks at the same time. It is a broader notion that encompasses not just parallel execution of code, but also the interleaving of processes on systems that may not support parallel execution due to hardware constraints. Concurrency is about structuring software for potential parallelism even if the execution may not be truly simultaneous.

Multithreading is a specific instance of concurrency that involves multiple threads of execution within a single process. Threads are lighter weight than processes — they share the same memory space and resources, but can execute different parts of a program simultaneously. This makes multithreading a powerful tool for improving the responsiveness and computational throughput of an application.

Parallel programming, on the other hand, explicitly focuses on executing multiple operations at the same time. It is a subset of concurrency that specifically aims at simultaneous execution, leveraging multiple processors or cores to perform different tasks or the same task on different data concurrently. Parallelism is the cornerstone of high-performance computing, enabling significant reductions in processing times by dividing large tasks into smaller ones and solving them concurrently.

Concurrency enables the efficient management of multiple tasks.

Multithreading allows for multiple threads of execution within a single process, enhancing computational throughput.

Parallel programming divides large tasks into smaller ones to be solved simultaneously, crucial for achieving high performance.

The distinction between these concepts is crucial for understanding how to design and implement HPC applications in C++. While all three are related and often used together, choosing the appropriate model based on the problem domain, hardware capabilities, and performance goals is vital.

Let’s consider a basic example in C++ that demonstrates the difference between sequential and parallel execution. Suppose we have a simple task: to increment a counter a million times. Sequentially, this would be implemented as follows:

1

long

long

counter

=

0;

2

for

(

long

long

i

=

0;

i

<

1000000;

++

i

)

{

3

++

counter

;

4

}

To implement a parallel version of this task, we might split the task across multiple threads:

1

#

include

<

thread

>

2

#

include

<

vector

>

3

4

void

increment

(

long

long

&

counter

)

{

5

for

(

long

long

i

=

0;

i

<

500000;

++

i

)

{

6

++

counter

;

7

}

8

}

9

10

int

main

()

{

11

long

long

counter

=

0;

12

std

::

vector

<

std

::

thread

>

threads

;

13

14

//

Create

two

threads

to

do

the

task

in

parallel

15

threads

.

push_back

(

std

::

thread

(

increment

,

std

::

ref

(

counter

)

)

)

;

16

threads

.

push_back

(

std

::

thread

(

increment

,

std

::

ref

(

counter

)

)

)

;

17

18

for

(

auto

&

t

:

threads

)

{

19

t

.

join

()

;

20

}

21

}

This code demonstrates a basic form of parallel programming by splitting a task into two parts and processing them simultaneously on two threads. However, it introduces potential data races on the shared counter variable, highlighting a common challenge in concurrent and parallel programming: ensuring the correctness of access to shared resources.

To manage concurrency effectively, several patterns and synchronisation mechanisms have been developed. These include locks, mutexes, semaphores, and condition variables, which help to coordinate access to shared resources among concurrent threads, preventing data races and ensuring data integrity.

Output (Sequential version): 1000000

Output (Multithreaded version): Varies due to data race

The output of the parallel version illustrates the non-deterministic nature of concurrent executions, where the outcome may vary across different