POSIX thread (pthread) libraries

The POSIX thread libraries are a standards based thread API for C/C++. It allows
one to spawn a new concurrent process flow. It is most effective on multi-
processor or multi-core systems where the process flow can be scheduled to run
on another processor thus gaining speed through parallel or distributed
processing. Threads require less overhead than "forking" or spawning a new
process because the system does not initialize a new system virtual memory
space and environment for the process. While most effective on a multiprocessor
system, gains are also found on uniprocessor systems which exploit latency in
I/O and other system functions which may halt process execution. (One thread
may execute while another is waiting for I/O or some other system latency.)
Parallel programming technologies such as MPI and PVM are used in a
distributed computing environment while threads are limited to a single computer
system. All threads within a process share the same address space. A thread is
spawned by defining a function and its arguments which will be processed in the
thread. The purpose of using the POSIX thread library in your software is to
execute software faster.

Thread Basics:

• Thread operations include thread creation, termination, synchronization

(joins,blocking), scheduling, data management and process interaction.
• A thread does not maintain a list of created threads, nor does it know the
thread that created it.
• All threads within a process share the same address space.
• Threads in the same process share:
o Process instructions
o Most data
o open files (descriptors)
o signals and signal handlers
o current working directory
o User and group id
• Each thread has a unique:
o Thread ID
o set of registers, stack pointer
o stack for local variables, return addresses
o signal mask
o priority
o Return value: errno
• pthread functions return "0" if OK.
Thread Creation and Termination:

Example: pthread1.c

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

void *print_message_function( void *ptr );

main()
{
pthread_t thread1, thread2;
char *message1 = "Thread 1";
char *message2 = "Thread 2";
int iret1, iret2;

/* Create independent threads each of which will execute function */

iret1 = pthread_create( &thread1, NULL, print_message_function,

(void*) message1);
iret2 = pthread_create( &thread2, NULL, print_message_function,
(void*) message2);

/* Wait till threads are complete before main continues. Unless we

*/
/* wait we run the risk of executing an exit which will terminate
*/
/* the process and all threads before the threads have completed.
*/

pthread_join( thread1, NULL);

pthread_join( thread2, NULL);

printf("Thread 1 returns: %d\n",iret1);

printf("Thread 2 returns: %d\n",iret2);
exit(0);
}

void *print_message_function( void *ptr )

{
char *message;
message = (char *) ptr;
printf("%s \n", message);
}

Compile:

• C compiler: cc -lpthread pthread1.c

or
• C++ compiler: g++ -lpthread pthread1.c
Run: ./a.out
Results:
Thread 1
Thread 2
Thread 1 returns: 0
Thread 2 returns: 0

Details:

• In this example the same function is used in each thread. The arguments
are different. The functions need not be the same.
• Threads terminate by explicitly calling pthread_exit, by letting the function
return, or by a call to the function exit which will terminate the process
including any threads.
• Function call: pthread_create - create a new thread

int pthread_create(pthread_t * thread,

const pthread_attr_t * attr,
void * (*start_routine)(void *),
void *arg);

Arguments:

o thread - returns the thread id. (unsigned long int defined in

bits/pthreadtypes.h)
o attr - Set to NULL if default thread attributes are used. (else define
members of the struct pthread_attr_t defined in
bits/pthreadtypes.h) Attributes include:
 detached state (joinable? Default: PTHREAD_CREATE_JOINABLE.
Other option: PTHREAD_CREATE_DETACHED)
 scheduling policy (real-time?
PTHREAD_INHERIT_SCHED,PTHREAD_EXPLICIT_SCHED,SCHED_
OTHER)
 scheduling parameter
 inheritsched attribute (Default: PTHREAD_EXPLICIT_SCHED Inherit
from parent thread: PTHREAD_INHERIT_SCHED)
 scope (Kernel threads: PTHREAD_SCOPE_SYSTEM User threads:
PTHREAD_SCOPE_PROCESS Pick one or the other not both.)
 guard size
 stack address (See unistd.h and bits/posix_opt.h
_POSIX_THREAD_ATTR_STACKADDR)
 stack size (default minimum PTHREAD_STACK_SIZE set in pthread.h),
o void * (*start_routine) - pointer to the function to be threaded.
Function has a single argument: pointer to void.
 o *arg - pointer to argument of function. To pass multiple arguments,
send a pointer to a structure.
• Function call: pthread_join - wait for termination of another thread

int pthread_join(pthread_t th, void **thread_return);

Arguments:

o th - thread suspended until the thread identified by th terminates,

either by calling pthread_exit() or by being cancelled.
o thread_return - If thread_return is not NULL, the return value of th
is stored in the location pointed to by thread_return.
• Function call: pthread_exit - terminate the calling thread

void pthread_exit(void *retval);

Arguments:

o retval - Return value of thread.

This routine kills the thread. The pthread_exit function never returns. If
the thread is not detached, the thread id and return value may be
examined from another thread by using pthread_join.
Note: the return pointer *retval, must not be of local scope otherwise it
would cease to exist once the thread terminates.

• [C++ pitfalls]: The above sample program will compile with the GNU C
and C++ compiler g++. The following function pointer representation below
will work for C but not C++. Note the subtle differences and avoid the
pitfall below:

void print_message_function( void *ptr );

...
...
iret1 = pthread_create( &thread1, NULL, (void*)&print_message_function,
(void*) message1);
...
...

Thread Synchronization:

The threads library provides three synchronization mechanisms:

 • mutexes - Mutual exclusion lock: Block access to variables by other
threads. This enforces exclusive access by a thread to a variable or set of
variables.
• joins - Make a thread wait till others are complete (terminated).
• condition variables - data type pthread_cond_t

Mutexes:
Mutexes are used to prevent data inconsistencies due to operations by multiple
threads upon the same memory area performed at the same time or to prevent
race conditions where an order of operation upon the memory is expected. A
contention or race condition often occurs when two or more threads need to
perform operations on the same memory area, but the results of computations
depends on the order in which these operations are performed. Mutexes are
used for serializing shared resources such as memory. Anytime a global
resource is accessed by more than one thread the resource should have a Mutex
associated with it. One can apply a mutex to protect a segment of memory
("critical region") from other threads. Mutexes can be applied only to threads in a
single process and do not work between processes as do semaphores.

Example threaded function:

Without Mutex With Mutex

view source view source

print? print?
1 int counter=0; /* Note scope of variable and mutex are the
01
2 same */
pthread_mutex_t mutex1 =
3 /* Function C */ 02
PTHREAD_MUTEX_INITIALIZER;
4 void functionC()
03 int counter=0;
5 {
04
6
05 /* Function C */
7 counter++
06 void functionC()
8 07 {
9 } 08 pthread_mutex_lock( &mutex1 );
09 counter++
10 pthread_mutex_unlock( &mutex1 );
11 }
Possible execution sequence
Thread 1 Thread 2 Thread 1 Thread 2
counter = counter =
counter = 0 counter = 0
0 0
 counter = counter = Thread 2 locked out.
counter = 1 Thread 1 has exclusive use of variable
1 1
counter
counter = 2

If register load and store operations for the incrementing of variable counter
occurs with unfortunate timing, it is theoretically possible to have each thread
increment and overwrite the same variable with the same value. Another
possibility is that thread two would first increment counter locking out thread one
until complete and then thread one would increment it to 2.

Sequence Thread 1 Thread 2

1 counter = 0 counter=0
Thread 1 locked out.
2 counter = 1
Thread 2 has exclusive use of variable counter
3 counter = 2

Code listing: mutex1.c

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

void *functionC();
pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;
int counter = 0;

main()
{
int rc1, rc2;
pthread_t thread1, thread2;

/* Create independent threads each of which will execute functionC */

if( (rc1=pthread_create( &thread1, NULL, &functionC, NULL)) )

{
printf("Thread creation failed: %d\n", rc1);
}

if( (rc2=pthread_create( &thread2, NULL, &functionC, NULL)) )

{
printf("Thread creation failed: %d\n", rc2);
}

/* Wait till threads are complete before main continues. Unless we

*/
/* wait we run the risk of executing an exit which will terminate
*/
 /* the process and all threads before the threads have completed.
*/

pthread_join( thread1, NULL);

pthread_join( thread2, NULL);

exit(0);
}

void *functionC()
{
pthread_mutex_lock( &mutex1 );
counter++;
printf("Counter value: %d\n",counter);
pthread_mutex_unlock( &mutex1 );
}

Compile: cc -lpthread mutex1.c

Run: ./a.out
Results:

Counter value: 1
Counter value: 2

When a mutex lock is attempted against a mutex which is held by another thread,
the thread is blocked until the mutex is unlocked. When a thread terminates, the
mutex does not unless explicitly unlocked. Nothing happens by default.

Man Pages:

• pthread_mutex_lock() - acquire a lock on the specified mutex variable. If

the mutex is already locked by another thread, this call will block the
calling thread until the mutex is unlocked.
• pthread_mutex_unlock() - unlock a mutex variable. An error is returned if
mutex is already unlocked or owned by another thread.
• pthread_mutex_trylock() - attempt to lock a mutex or will return error code
if busy. Useful for preventing deadlock conditions.

Joins:
A join is performed when one wants to wait for a thread to finish. A thread calling
routine may launch multiple threads then wait for them to finish to get the results.
One waits for the completion of the threads with a join.

Sample code: join1.c

#include <stdio.h>
#include <pthread.h>

#define NTHREADS 10
void *thread_function(void *);
pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;
int counter = 0;

main()
{
pthread_t thread_id[NTHREADS];
int i, j;

for(i=0; i < NTHREADS; i++)

{
pthread_create( &thread_id[i], NULL, thread_function, NULL );
}

for(j=0; j < NTHREADS; j++)

{
pthread_join( thread_id[j], NULL);
}

/* Now that all threads are complete I can print the final result.
*/
/* Without the join I could be printing a value before all the
threads */
/* have been completed.
*/

printf("Final counter value: %d\n", counter);

}

void *thread_function(void *dummyPtr)

{
printf("Thread number %ld\n", pthread_self());
pthread_mutex_lock( &mutex1 );
counter++;
pthread_mutex_unlock( &mutex1 );
}

Compile: cc -lpthread join1.c

Run: ./a.out
Results:

Thread number 1026

Thread number 2051
Thread number 3076
Thread number 4101
Thread number 5126
Thread number 6151
Thread number 7176
Thread number 8201
 Thread number 9226
Thread number 10251
Final counter value: 10
Man Pages:

• pthread_create() - create a new thread

• pthread_join() - wait for termination of another thread
• pthread_self() - return identifier of current thread

Condition Variables:

A condition variable is a variable of type pthread_cond_t and is used with the

appropriate functions for waiting and later, process continuation. The condition
variable mechanism allows threads to suspend execution and relinquish the
processor until some condition is true. A condition variable must always be
associated with a mutex to avoid a race condition created by one thread
preparing to wait and another thread which may signal the condition before the
first thread actually waits on it resulting in a deadlock. The thread will be
perpetually waiting for a signal that is never sent. Any mutex can be used, there
is no explicit link between the mutex and the condition variable.

Man pages of functions used in conjunction with the condition variable:

• Creating/Destroying:
o pthread_cond_init
o pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
o pthread_cond_destroy
• Waiting on condition:
o pthread_cond_wait - unlocks the mutex and waits for the condition
variable cond to be signaled.
o pthread_cond_timedwait - place limit on how long it will block.
• Waking thread based on condition:
o pthread_cond_signal - restarts one of the threads that are waiting
on the condition variable cond.
o pthread_cond_broadcast - wake up all threads blocked by the
specified condition variable.

Example code: cond1.c

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

pthread_mutex_t count_mutex = PTHREAD_MUTEX_INITIALIZER;

pthread_cond_t condition_var = PTHREAD_COND_INITIALIZER;
void *functionCount1();
void *functionCount2();
int count = 0;
#define COUNT_DONE 10
#define COUNT_HALT1 3
#define COUNT_HALT2 6

main()
{
pthread_t thread1, thread2;

pthread_create( &thread1, NULL, &functionCount1, NULL);

pthread_create( &thread2, NULL, &functionCount2, NULL);

pthread_join( thread1, NULL);

pthread_join( thread2, NULL);

printf("Final count: %d\n",count);

exit(0);
}

// Write numbers 1-3 and 8-10 as permitted by functionCount2()

void *functionCount1()
{
for(;;)
{
// Lock mutex and then wait for signal to relase mutex
pthread_mutex_lock( &count_mutex );

// Wait while functionCount2() operates on count

// mutex unlocked if condition varialbe in functionCount2()
signaled.
pthread_cond_wait( &condition_var, &count_mutex );
count++;
printf("Counter value functionCount1: %d\n",count);

pthread_mutex_unlock( &count_mutex );

if(count >= COUNT_DONE) return(NULL);

}
}

// Write numbers 4-7

void *functionCount2()
{
for(;;)
{
pthread_mutex_lock( &count_mutex );

if( count < COUNT_HALT1 || count > COUNT_HALT2 )

{
// Condition of if statement has been met.
// Signal to free waiting thread by freeing the mutex.
 // Note: functionCount1() is now permitted to modify "count".
pthread_cond_signal( &condition_var );
}
else
{
count++;
printf("Counter value functionCount2: %d\n",count);
}

pthread_mutex_unlock( &count_mutex );

if(count >= COUNT_DONE) return(NULL);

}

Compile: cc -lpthread cond1.c

Run: ./a.out
Results:

Counter value functionCount1: 1

Counter value functionCount1: 2
Counter value functionCount1: 3
Counter value functionCount2: 4
Counter value functionCount2: 5
Counter value functionCount2: 6
Counter value functionCount2: 7
Counter value functionCount1: 8
Counter value functionCount1: 9
Counter value functionCount1: 10
Final count: 10

Note that functionCount1() was halted while count was between the values
COUNT_HALT1 and COUNT_HALT2. The only thing that has been ensures is
that functionCount2 will increment the count between the values
COUNT_HALT1 and COUNT_HALT2. Everything else is random.

The logic conditions (the "if" and "while" statements) must be chosen to insure
that the "signal" is executed if the "wait" is ever processed. Poor software logic
can also lead to a deadlock condition.

Note: Race conditions abound with this example because count is used as the
condition and can't be locked in the while statement without causing deadlock.

Thread Scheduling:
When this option is enabled, each thread may have its own scheduling
properties. Scheduling attributes may be specified:

• during thread creation

• by dynamically by changing the attributes of a thread already created
• by defining the effect of a mutex on the thread's scheduling when creating
a mutex
• by dynamically changing the scheduling of a thread during synchronization
operations.

The threads library provides default values that are sufficient for most cases.
Thread Pitfalls:

• Race conditions: While the code may appear on the screen in the order
you wish the code to execute, threads are scheduled by the operating
system and are executed at random. It cannot be assumed that threads
are executed in the order they are created. They may also execute at
different speeds. When threads are executing (racing to complete) they
may give unexpected results (race condition). Mutexes and joins must be
utilized to achieve a predictable execution order and outcome.
• Thread safe code: The threaded routines must call functions which are
"thread safe". This means that there are no static or global variables which
other threads may clobber or read assuming single threaded operation. If
static or global variables are used then mutexes must be applied or the
functions must be re-written to avoid the use of these variables. In C, local
variables are dynamically allocated on the stack. Therefore, any function
that does not use static data or other shared resources is thread-safe.
Thread-unsafe functions may be used by only one thread at a time in a
program and the uniqueness of the thread must be ensured. Many non-
reentrant functions return a pointer to static data. This can be avoided by
returning dynamically allocated data or using caller-provided storage. An
example of a non-thread safe function is strtok which is also not re-
entrant. The "thread safe" version is the re-entrant version strtok_r.
• Mutex Deadlock: This condition occurs when a mutex is applied but then
not "unlocked". This causes program execution to halt indefinitely. It can
also be caused by poor application of mutexes or joins. Be careful when
applying two or more mutexes to a section of code. If the first
pthread_mutex_lock is applied and the second pthread_mutex_lock fails
due to another thread applying a mutex, the first mutex may eventually
lock all other threads from accessing data including the thread which holds
the second mutex. The threads may wait indefinitely for the resource to
become free causing a deadlock. It is best to test and if failure occurs, free
the resources and stall before retrying.
...
pthread_mutex_lock(&mutex_1);
while ( pthread_mutex_trylock(&mutex_2) ) /* Test if already locked
*/
{
pthread_mutex_unlock(&mutex_1); /* Free resource to avoid deadlock
*/
...
/* stall here */
...
pthread_mutex_lock(&mutex_1);
}
count++;
pthread_mutex_unlock(&mutex_1);
pthread_mutex_unlock(&mutex_2);
...

The order of applying the mutex is also important. The following code segment
illustrates a potential for deadlock:

void *function1()
{
...
pthread_mutex_lock(&lock1); // Execution step 1
pthread_mutex_lock(&lock2); // Execution step 3 DEADLOCK!!!
...
...
pthread_mutex_lock(&lock2);
pthread_mutex_lock(&lock1);
...
}

void *function2()
{
...
pthread_mutex_lock(&lock2); // Execution step 2
pthread_mutex_lock(&lock1);
...
...
pthread_mutex_lock(&lock1);
pthread_mutex_lock(&lock2);
...
}

main()
{
...
pthread_create(&thread1, NULL, function1, NULL);
pthread_create(&thread2, NULL, function2, NULL);
...
}
 • If function1 acquires the first mutex and function2 acquires the second,
all resources are tied up and locked.
• Condition Variable Deadlock: The logic conditions (the "if" and "while"
statements) must be chosen to insure that the "signal" is executed if the
"wait" is ever processed.

Process:
fork():

The fork() system call will spawn a new child process which is an identical
process to the parent except that has a new system process ID. The process is
copied in memory from the parent and a new process structure is assigned by
the kernel. The return value of the function is which discriminates the two threads
of execution. A zero is returned by the fork function in the child's process.

The environment, resource limits, umask, controlling terminal, current working

directory, root directory, signal masks and other process resources are also
duplicated from the parent in the forked child process.

Example:

#include <iostream>
#include <string>

// Required by for routine

#include <sys/types.h>
#include <unistd.h>

#include <stdlib.h> // Declaration for exit()

using namespace std;

int globalVariable = 2;

main()
{
string sIdentifier;
int iStackVariable = 20;

pid_t pID = fork();

if (pID == 0) // child
{
// Code only executed by child process
 sIdentifier = "Child Process: ";
globalVariable++;
iStackVariable++;
}
else if (pID < 0) // failed to fork
{
cerr << "Failed to fork" << endl;
exit(1);
// Throw exception
}
else // parent
{
// Code only executed by parent process

sIdentifier = "Parent Process:";

}

// Code executed by both parent and child.

cout << sIdentifier;

cout << " Global variable: " << globalVariable;
cout << " Stack variable: " << iStackVariable << endl;
}

Compile: g++ -o ForkTest ForkTest.cpp

Run: ForkTest

Parent Process: Global variable: 2 Stack variable: 20

Child Process: Global variable: 3 Stack variable: 21

[Potential Pitfall]: Some memory duplicated by a forked process such as file

pointers, will cause intermixed output from both processes. Use the wait()
function so that the processes do not access the file at the same time or open
unique file descriptors. Some like stdout or stderr will be shared unless
synchronized using wait() or some other mechanism. The file close on exit is
another gotcha. A terminating process will close files before exiting. File locks set
by the parent process are not inherited by the child process.

[Potential Pitfall]: Race conditions can be created due to the unpredictability of

when the kernel scheduler runs portions or time slices of the process. One can
use wait(). the use of sleep() does not guarentee reliability of execution on a
heavily loaded system as the scheduler behavior is not predictable by the
application.
Note on exit() vs _exit(): The C library function exit() calls the kernel system
call _exit() internally. The kernel system call _exit() will cause the kernel to
close descriptors, free memory, and perform the kernel terminating process
clean-up. The C library function exit() call will flush I/O buffers and perform
aditional clean-up before calling _exit() internally. The function exit(status)
causes the executable to return "status" as the return code for main(). When
exit(status) is called by a child process, it allows the parent process to
examine the terminating status of the child (if it terminates first). Without this call
(or a call from main() to return()) and specifying the status argument, the
process will not return a value.

#include <stdlib.h> #include <unistd.h>

void exit(int status); void _exit(int status);

Man Pages:

• fork - create a child process

vfork():

The vfork() function is the same as fork() except that it does not make a copy
of the address space. The memory is shared reducing the overhead of spawning
a new process with a unique copy of all the memory. This is typically used when
using fork() to exec() a process and terminate. The vfork() function also
executes the child process first and resumes the parent process when the child
terminates.

#include <iostream>
#include <string>

// Required by for routine

#include <sys/types.h>
#include <unistd.h>

using namespace std;

int globalVariable = 2;

main()
{
string sIdentifier;
int iStackVariable = 20;

pid_t pID = vfork();

if (pID == 0) // child
 {
// Code only executed by child process

sIdentifier = "Child Process: ";

globalVariable++;
iStackVariable++;
cout << sIdentifier;
cout << " Global variable: " << globalVariable;
cout << " Stack variable: " << iStackVariable << endl;
_exit(0);
}
else if (pID < 0) // failed to fork
{
cerr << "Failed to fork" << endl;
exit(1);
// Throw exception
}
else // parent
{
// Code only executed by parent process

sIdentifier = "Parent Process:";

}

// executed only by parent

cout << sIdentifier;

cout << " Global variable: " << globalVariable;
cout << " Stack variable: " << iStackVariable << endl;
exit(0);
}

Compile: g++ -o VForkTest VForkTest.cpp

Run: VForkTest

Child Process: Global variable: 3 Stack variable: 21

Parent Process: Global variable: 3 Stack variable: 21

Note: The child process executed first, updated the variables which are shared
between the processes and NOT unique, and then the parent process executes
using variables which the child has updated.

[Potential Pitfall]: A deadlock condition may occur if the child process does not
terminate, the parent process will not proceed.

Man Pages:

• vfork - create a child process and block parent

• _exit - - terminate the current process
clone():

The function clone() creates a new child process which shares memory, file
descriptors and signal handlers with the parent. It implements threads and thus
launches a function as a child. The child terminates when the parent terminates.
See the YoLinux POSIX threads tutorial

Man Pages:

• clone - create a child process

wait():

The parent process will often want to wait until all child processes have been
completed. this can be implemented with the wait() function call.

wait(): Blocks calling process until the child process terminates. If child process
has already teminated, the wait() call returns immediately. if the calling process
has multiple child processes, the function returns when one returns.

waitpid(): Options available to block calling process for a particular child process
not the first one.

Code snipet:
#include <sys/wait.h>

...

pid_t pID = <I>set to child process id with call to fork OR:</I>

// If set <-1, wait for any child process whose
process group ID = abs(pID)
// If = -1, wait for any child process. Same as
wait().
// If = 0, wait for any child process whose process
group ID is same as calling process.
// If > 0, wait for the child whose process ID = pID.

...
int childExitStatus;

pid_t ws = waitpid( pID, &childExitStatus, WNOHANG);

if( WIFEXITED(childExitStatus) )
{
// Child process exited thus exec failed.
// LOG failure of exec in child process.
cout << "Result of waitpid: Child process exited thus exec
failed." << endl;
}
OR

...
int childExitStatus;

pid_t ws = waitpid( pID, &childExitStatus, 0);

if( !WIFEXITED(childExitStatus) )
{
cerr << "waitpid() exited with an error: Status= "
<< WEXITSTATUS(childExitStatus)
<< endl;
}
else if( WIFSIGNALED(childExitStatus) )
{
cerr << "waitpid() exited due to a signal: "
<< WTERMSIG(childExitStatus)
<< endl;
}

Notes:

• See man page for options: WNOHANG, WUNTRACED.

• See man page for return macros: WIFEXITED(), WEXITSTATUS(),
WIFSIGNALED(), WTERMSIG(), WIFSTOPPED(), WSTOPSIG().
• See man page for errors: ECHILD, EINVAL, EINTR. (Also see sample of
error processing below.)

Man Pages:

• wait / waitpid - wait for process termination

Set system group ID and process ID:

Avoids orphaned process group when parent terminates. When parent dies, this
will be a zombie. (No parent process. Parent=1) Instead, create a new process
group for the child. Later process the group is terminated to stop all spawned
processes. Thus all subsequent processes should be of this group if they are to
be terminated by the process group id. Process group leader has the same
process id and group process id. If not changed then the process group is that of
the parent. Set the process group id to that of the child process.
#include <sys/types.h>
#include <unistd.h>

#ifdef __gnu_linux__
pid_t pgid = setpgid(child_pID, child_pID);
#endif

...
...

if( pgid < 0)

{
cout << "Failed to set process group ID" << endl;
_exit(0); // If exec fails then exit forked process.

Use the setgid call to set the group id of the current process. Requires
root access.

The macro testing for __gnu_linux__ is for cross platform support as man
other OS's use a different system call.

Man Pages:

• setpgid/getpgid setpgrp/getpgrp - set process group

• setsid - creates a session and sets the process group ID
• getuid/geteuid - get user identity
• setgid - set group identity
• getgid/getegid - get group (real/effective) identity
• setreuid/setregid - set real user or group identity

Kill all processes in a process group:

This is the real reason to set up a process group. One may kill all the processes
in the process group without having to keep track of how many processes have
been forked and all of their process id's.

See /usr/include/bits/signum.h for list of signals.

...

int killReturn = killpg( pID, SIGKILL); // Kill child process group

if( killReturn == ESRCH) // pid does not exist

{
cout << "Group does not exist!" << endl;
}
else if( killReturn == EPERM) // No permission to send signal
{
cout << "No permission to send signal!" << endl;
}
else
cout << "Signal sent. All Ok!" << endl;

...
Man Pages:

• killpg - send signal to a process group

• kill - send signal to a process
• signal (2) - ANSI C signal handling
• signal (7) - List of available signals
• sigaction - POSIX signal handling functions.
• pause (2) - wait for signal
• raise (3) - send a signal to a current process

system() and popen():

The system() call will execute an OS shell command as described by a character

command string. This function is implemented using fork(), exec() and
waitpid(). The command string is executed by calling /bin/sh -c command-
string. During execution of the command, SIGCHLD will be blocked, and
SIGINT and SIGQUIT will be ignored. The call "blocks" and waits for the task to
be performed before continuing.

#include <stdio.h>
#include <stdlib.h>
main()
{
system("ls -l");
printf("Command done!");
}

The statement "Command done!" will not print untill the "ls -l" command
has completed.

The popen() call opens a process by creating a pipe, forking, and invoking the
shell (bourne shell on Linux). The advantage to using popen() is that it will allow
one to interrogate the results of the command issued.

This example opens a pipe which executes the shell command "ls -l". The
results are read and printed out.

#include <stdio.h>
main()
{
 FILE *fpipe;
char *command="ls -l";
char line[256];

if ( !(fpipe = (FILE*)popen(command,"r")) )
{ // If fpipe is NULL
perror("Problems with pipe");
exit(1);
}

while ( fgets( line, sizeof line, fpipe))

{
printf("%s", line);
}
pclose(fpipe);
}

#include <stdio.h>
main()
{
FILE *fpipe;
char *command="ls -l";
char line[256];

if ( !(fpipe = (FILE*)popen(command,"r")) )
{ // If fpipe is NULL
perror("Problems with pipe");
exit(1);
}

while ( fgets( line, sizeof line, fpipe))

{
printf("%s", line);
}
pclose(fpipe);
}

The second argument to popen:

• r: Read from stdin (command results)

• w: write to stdout (command)
• For stderr: command="ls -l 2>&1","w");

Man Pages:

• system - execute a shell command

• popen - process I/O

exec() functions and execve():

The exec() family of functions will initiate a program from within a program. They
are also various front-end functions to execve().

The functions return an integer error code. (0=Ok/-1=Fail).

execl() and execlp():

The function call "execl()" initiates a new program in the same environment in
which it is operating. An executable (with fully qualified path. i.e. /bin/ls) and
arguments are passed to the function. Note that "arg0" is the command/file name
to execute.

int execl(const char *path, const char *arg0, const char *arg1, const
char *arg2, ... const char *argn, (char *) 0);

#include <unistd.h>
main()
{
execl("/bin/ls", "/bin/ls", "-r", "-t", "-l", (char *) 0);
}
Where all function arguments are null terminated strings. The list of
arguments is terminated by NULL.

The routine execlp() will perform the same purpose except that it will use
environment variable PATH to determine which executable to process. Thus a fully
qualified path name would not have to be used. The first argument to the function
could instead be "ls". The function execlp() can also take the fully qualified
name as it also resolves explicitly.

Man Pages:

• execl / execlp - execute

execv() and execvp():

This is the same as execl() except that the arguments are passed as null
terminated array of pointers to char. The first element "argv[0]" is the command
name.

int execv(const char *path, char *const argv[]);

example:

#include <unistd.h>
main()
{
char *args[] = {"/bin/ls", "-r", "-t", "-l", (char *) 0 };

execv("/bin/ls", args);
}
The routine execvp() will perform the same purpose except that it will use
environment variable PATH to determine which executable to process. Thus a fully
qualified path name would not have to be used. The first argument to the function
could instead be "ls". The function execvp() can also take the fully qualified
name as it also resolves explicitly.

Man Pages:

• execv / execvp - execute

execve():

The function call "execve()" executes a process in an environment which it

assigns.

Set the environment variables:

Assignment:

char *env[] = { "USER=user1", "PATH=/usr/bin:/bin:/opt/bin", (char *)

0 };

Read from file:

#include <iostream>
#include <fstream>
#include <string>
#include <vector>

// Required by for routine

#include <sys/types.h>
#include <unistd.h>

using namespace std;

// Class definition:

class CReadEnvironmentVariablesFile
{
 public:
CReadEnvironmentVariablesFile() { m_NumberOfEnvironmentVariables=0;
};
~CReadEnvironmentVariablesFile();
char **ReadFile(string& envFile);
private:
int m_NumberOfEnvironmentVariables;
char **m_envp;
};

Call execve:

string getErrMsg(int errnum);

main()
{
string envFile("environment_variables.conf");
CReadEnvironmentVariablesFile readEnvFile;
char **Env_envp = readEnvFile.ReadFile(envFile);

// Command to execute
char *Env_argv[] = { "/bin/ls", "-l", "-a", (char *) 0 };

pid_t pID = fork();

if (pID == 0) // child
{
// This version of exec accepts environment variables.
// Function call does not return on success.

int execReturn = execve (Env_argv[0], Env_argv, Env_envp);

cout << "Failure! execve error code=" << execReturn << endl;

cout << getErrMsg(execReturn) << endl;

_exit(0); // If exec fails then exit forked process.

}
else if (pID < 0) // failed to fork
{
cerr << "Failed to fork" << endl;
}
else // parent
{
cout << "Parent Process" << endl;
}
}

Handle errors:

string getErrMsg(int errnum)

{
 switch ( errnum ) {

#ifdef EACCES
case EACCES :
{
return "EACCES Permission denied";
}
#endif

#ifdef EPERM
case EPERM :
{
return "EPERM Not super-user";
}
#endif

#ifdef E2BIG
case E2BIG :
{
return "E2BIG Arg list too long";
}
#endif

#ifdef ENOEXEC
case ENOEXEC :
{
return "ENOEXEC Exec format error";
}
#endif

#ifdef EFAULT
case EFAULT :
{
return "EFAULT Bad address";
}
#endif

#ifdef ENAMETOOLONG
case ENAMETOOLONG :
{
return "ENAMETOOLONG path name is too long ";
}
#endif

#ifdef ENOENT
case ENOENT :
{
return "ENOENT No such file or directory";
}
#endif

#ifdef ENOMEM
case ENOMEM :
{
return "ENOMEM Not enough core";
}
#endif

#ifdef ENOTDIR
case ENOTDIR :
{
return "ENOTDIR Not a directory";
}
#endif

#ifdef ELOOP
case ELOOP :
{
return "ELOOP Too many symbolic links";
}
#endif

#ifdef ETXTBSY
case ETXTBSY :
{
return "ETXTBSY Text file busy";
}
#endif

#ifdef EIO
case EIO :
{
return "EIO I/O error";
}
#endif

#ifdef ENFILE
case ENFILE :
{
return "ENFILE Too many open files in system";
}
#endif

#ifdef EINVAL
case EINVAL :
{
return "EINVAL Invalid argument";
}
#endif

#ifdef EISDIR
case EISDIR :
{
return "EISDIR Is a directory";
}
#endif

#ifdef ELIBBAD
case ELIBBAD :
{
return "ELIBBAD Accessing a corrupted shared lib";
}
#endif
 default :
{
std::string errorMsg(strerror(errnum));
if ( errnum ) return errorMsg;
}
}
}

Man Pages:

• strerror / strerror_r - return string describing error code

• errno - number of last error
• perror - print a system error message

Data File: environment_variables.conf

PATH=/usr/bin:/bin
MANPATH=/opt/man
LANG=C
DISPLAY=:0.0

Man Pages:

• execve - execute with given environment

Note: Don't mix malloc() and new. Choose one form of memory allocation and
stick with it.

Malloc:

..
...

int ii;

m_envp = (char **) calloc((m_NumberOfEnvironmentVariables+1),

sizeof(char **));

...
// Allocate arrays of character strings.
int ii;
for(ii=0; ii < NumberOfEnvironmentVariables; ii++)
{
// NULL terminated
m_envp[ii] = (char *) malloc(vEnvironmentVariables[ii].size()+1);
strcpy( m_envp[ii], vEnvironmentVariables[ii].c_str());
}

// must terminate with null

 m_envp[ii] = (char*) 0;

return m_envp;
}
...

Free:

...

// Free array's of characters

for(ii=0; ii < m_NumberOfEnvironmentVariables; ii++)

{
free(m_envp[ii]);
}

// Free array of pointers.

free(m_envp);

...

Socket
Socket programming and the C BSD API
C and C++ sockets programming with examples of the BSD API on the Linux
platform

Sockets are an inter-process network communication implementation using a

Internet Protocol (IP) stack on an Ethernet transport. Sockets are language and
protocol independent and available to "C", Perl, Python, Ruby and Java (and
more) programmers. The "C" language BSD API is used on Linux, all popular
variants of Unix, Microsoft Windows (NT,2000,XP,... and later) and even
embedded OSs like VxWorks. It is by far the most popular implementation of
inter-process network communication.

Sockets allow one process to communicate with another whether it is local on the
same computer system or remote over the network. Many other higher level
protocols are built upon sockets technology.

The sockets API provides many configuration options so we will try and cover the
socket API components and then give examples of a few implementations. It
would be very difficult to cover all variations of its use.
Sockets utilize the following standard protocols:
Protocol Description
IP Internet Protocol provides network routing using IP addressing eg
192.168.1.204
UDP User Datagram Protocol - IP with ports to distinguish among
processes running on same host. No data verification.
TCP Transmission Control Protocol - IP with ports to distinguish
among processes running on same host. Connection oriented,
stream transfer, full duplex, reliable with data verification.

BSD socket API:

Typically one configures a socket server to which a socket client may attach and
communicate. The IP protocol layer will also require that the domain name or IP
addresses of the communicating processes be made known as well. Within the
IP protocol it is also important to provide the mechanism used: TCP or UDP.

The BSD is a "C" programming API. Examples shown are compiled using the
GNU C++ compiler on Linux:

Basic steps in using a socket:

• Socket include files:

Include File Description
sys/types.h Types used in sys/socket.h and netinet/in.h
netinet/in.h Internet domain address structures and functions
netinet/tcp.h Socket option macro definitions, TCP headers, enums, etc
sys/socket.h Structures and functions used for socket API.i accept(),
bind(), connect(), listen(), recv(), send(),
setsockopt(), shutdown(), etc ...

netdb.h Used for domain/DNS hostname lookup

sys/select.h Used by the select(), pselect() functions and defines
FD_CLR, FD_ISSET, FD_SET, FD_ZERO macros

sys/time.h select() uses argument of type struct timeval and

pselect() uses struct timespec defined by this include file.
 arpa/inet.h Definitions for internet operations. Prototypes functions such
as htonl(), htons(), ntohl(), ntohs(), inet_addr(),
inet_ntoa(), etc ...

unistd.h Defines constants and types

errno.h Defines sytem error numbers
• Create the socket instance:

Open a socket using TCP: Basic declarations and call to "socket".

#include <iostream>
#include <sys/types.h> // Types used in sys/socket.h and netinet/in.h
#include <netinet/in.h> // Internet domain address structures and
functions
#include <sys/socket.h> // Structures and functions used for socket API
#include <netdb.h> // Used for domain/DNS hostname lookup
#include <unistd.h>
#include <errno.h>

using namespace std;

main()
{
int socketHandle;

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, IPPROTO_IP)) < 0)

{
close(socketHandle);
exit(EXIT_FAILURE);
}

...
...
}

• Socket function prototype:

int socketHandle = socket(int socket_family, int socket_type, int

protocol);

Choose socket communications family/domain:

o Internet IPV4: AF_INET

o Internet IPV6: AF_INET6
o Unix path name (communicating processes are on the same
system): AF_UNIX
 Choose socket type:

o TCP: SOCK_STREAM
o UDP: SOCK_DGRAM
o Raw protocol at network layer: SOCK_RAW

Choose socket protocol: (See /etc/protocols)

o Internet Protocol (IP): 0 or IPPROTO_IP

o ICMP: 1
o ...

Also see:

o socket man page

o protocols man page
• Configure the socket as a client or server:

Comparison of sequence of BSD API calls:

Socket Server Socket Client

socket() socket()
bind() gethostbyname()
listen()
accept() connect()
recv()/send() recv()/send()
close() close()

This is specific to whether the application is a socket client or a socket

server.

o Socket server:
 bind(): bind the socket to a local socket address. This
assigns a name to the socket.
 listen(): listen for connections on a socket created with
"socket()" and "bind()" and accept incoming connections.
This is used for TCP and not UDP. Zero is returned on
success.
 accept(): accept a connection on a socket. Accept the first
connection request on the queue of pending connections,
create a new connected socket with mostly the same
 properties as defined by the call to "socket()", and allocate a
new file descriptor for the socket, which is returned. The
newly created socket is no longer in the listening state. Note
this call blocks until a client connects.

...
...
#define MAXHOSTNAME 256
...
...

struct sockaddr_in socketAddress;

char sysHost[MAXHOSTNAME+1]; // Hostname of this computer we are
running on
struct hostNamePtr *hPtr;
int portNumber = 8080;

bzero(&socketInfo, sizeof(sockaddr_in)); // Clear structure memory

// Get system information

gethostname(sysHost, MAXHOSTNAME); // Get the name of this computer

we are running on
if((hPtr = gethostbyname(sysHost)) == NULL)
{
cerr << "System hostname misconfigured." << endl;
exit(EXIT_FAILURE);
}

// Load system information into socket data structures

socketInfo.sin_family = AF_INET;
// Use any address available to the system. This is a typical
configuration for a server.
// Note that this is where the socket client and socket server
differ.
// A socket client will specify the server address to connect to.
socketInfo.sin_addr.s_addr = htonl(INADDR_ANY); // Translate long
integer to network byte order.
socketInfo.sin_port = htons(portNumber); // Set port number

// Bind the socket to a local socket address

if( bind(socketHandle, (struct sockaddr *) &socketInfo,

sizeof(struct sockaddr_in)) < 0)
{
close(socketHandle);
perror("bind");
exit(EXIT_FAILURE);
}

listen(socketHandle, 1);

int socketConnection;
if( (socketConnection = accept(socketHandle, NULL, NULL)) < 0)
{
close(socketHandle);
exit(EXIT_FAILURE);
}

...
...

// read/write to socket here

o Socket functions:
 bind():
Function prototype:

int bind(int sockfd, struct sockaddr *my_addr, socklen_t

addrlen);

Bind arguments:

 int sockfd: Socket file descriptor. Returned by call to

"socket".
 struct sockaddr: Socket information structure
 socklen_t addrlen: Size of structure
 Returns 0: Sucess, -1: Failure and errno may be set.

Also see the bind man page

 listen():
Function prototype:

int listen(int s, int backlog);

Listen arguments:

 int s: Socket file descriptor. Returned by call to

"socket". Identifies a bound but unconnected socket.
 int backlog: Set maximum length of the queue of
pending connections for the listening socket. A reasonable
value is 10. Actual maximum permissible: SOMAXCONN
Example: int iret = listen(socketHandle, SOMAXCONN);
The include file sys/socket.h will include
/usr/include/bits/socket.h which defines the default value
for SOMAXCONN as 128.
The actual value set for the operating system: cat
/proc/sys/net/core/somaxconn
In kernels before 2.4.25, this limit was a hard coded value
and thus would require a kernel recompile with the
 SOMAXCONN value as defined in /usr/include/linux/socket.h

For very heavy server use, modify the system limit in the
proc file and set "backlog" to the same value (eg. 512).
 Returns 0: Sucess, -1: Failure and errno may be set.

Also see the listen man page

 accept():
Function prototype:

int accept(int s, struct sockaddr *addr, socklen_t *addrlen);

Accept arguments:

 int s: Socket file descriptor. Returned by call to

"socket".
 struct sockaddr *addr: Pointer to a sockaddr structure.
This structure is filled in with the address of the connecting
entity.
 socklen_t *addrlen: initially contains the size of the
structure pointed to by addr; on return it will contain the
actual length (in bytes) of the address returned. When addr
is NULL nothing is filled in.
 Returns:
 Success: non-negative integer, which is a
descriptor of the accepted socket. Argument "addrlen"
will have a return value.
 Fail: -1, errno may be set

Also see the accept man page

o [Potential Pitfall]: If you get the following message:

o bind: Address already in use
o The solution is to choose a different port or kill the process
which is
using the port and creating the conflict. You may have to be root to see all
processes with netstat.
o netstat -punta | grep 8080
o tcp 0 0 :::8080 :::* LISTEN
o Socket client:
 connect(): initiate a connection with a remote entity on a
socket. Zero is returned on success. Support both TCP
(SOCK_STREAM) and UDP (SOCK_DGRAM). For SOCK_STREAM, an
 actual connection is made. For SOCK_DGRAM the address is
the address to which datagrams are sent and received.

...
...

struct sockaddr_in remoteSocketInfo;

struct hostent *hPtr;
int socketHandle;
char *remoteHost="dev.megacorp.com";
int portNumber = 8080;

bzero(&remoteSocketInfo, sizeof(sockaddr_in)); // Clear structure

memory

// Get system information

if((hPtr = gethostbyname(remoteHost)) == NULL)

{
cerr << "System DNS name resolution not configured properly." <<
endl;
cerr << "Error number: " << ECONNREFUSED << endl;
exit(EXIT_FAILURE);
}

// Load system information for remote socket server into socket data
structures

memcpy((char *)&remoteSocketInfo.sin_addr, hPtr->h_addr, hPtr-

>h_length);
remoteSocketInfo.sin_family = AF_INET;
remoteSocketInfo.sin_port = htons((u_short)portNumber); // Set
port number

if( (connect(socketHandle, (struct sockaddr *)&remoteSocketInfo,

sizeof(sockaddr_in)) < 0)
{
close(socketHandle);
exit(EXIT_FAILURE);
}

...
...
o Connect function prototype:
int connect(int sockfd, const struct sockaddr *serv_addr,
socklen_t addrlen);
Connect arguments: (Same as server's bind() arguments)

 int sockfd: Socket file descriptor. Returned by call to

"socket".
 struct sockaddr: Socket information structure
 socklen_t addrlen: Size of structure
 Returns 0: Sucess, -1: Failure and errno may be set.
Zero is returned upon success and on error, -1 and errno is set appropriately.
Also see the connect man page

o The sockaddr_in data structure: /usr/include/linux/in.h

/* Internet address. */
struct in_addr {
__u32 s_addr; /* Defined as 32 or 64 bit address (system
dependent) */
};

/* Structure describing an Internet (IP) socket address. */

#define __SOCK_SIZE__ 16 /* sizeof(struct sockaddr)
*/
struct sockaddr_in {
sa_family_t sin_family; /* Address family
*/
unsigned short int sin_port; /* Port number
*/
struct in_addr sin_addr; /* Internet address
*/

/* Pad to size of `struct sockaddr'. */

unsigned char __pad[__SOCK_SIZE__ - sizeof(short int) -
sizeof(unsigned short int) - sizeof(struct
in_addr)];
};

•
o Note:
 IP addresses: Note the specific IP address could be
specified:
 #include <arpa/inet.h> // IP from string
conversion

 socketInfo.sin_addr.s_addr =
inet_addr("127.0.0.1");

 cout << inet_ntoa(socketInfo.sin_addr) << endl;

or bind to all network interfaces available:

socketInfo.sin_addr.s_addr = htonl(INADDR_ANY);

 Port numbers: Note the specific port can be specified:

 socketInfo.sin_port = htons(8080);

 cout << ntohs(socketInfo.sin_port) << endl;
 or let the system define one for you:

socketInfo.sin_port = htons(INADDR_ANY);

• Read from or write to socket:

Use send() and recv(), or write() and read(), or sendto() and recvfrom() to
read/write to/from a socket.

o TCP recv() or UDP recvfrom(): receive a message from a socket

char *pcIpAddress;
unsigned short shPort;

...
...

if (iSocketType == SOCK_STREAM)
{
rc = recv(socketHandle, (char*) _pcMessage, (int) _iMessageLength,
0);
if ( rc == 0 )
{
cerr << "ERROR! Socket closed" << endl;
}
else if (rc == -1)
{
cerr << "ERROR! Socket error" << endl;
closeSocket();
}
}
else if (iSocketType == SOCK_DGRAM)
{
int iLength;
struct sockaddr_in stReceiveAddr;
iLength = (int) sizeof(struct sockaddr_in);
memset((void*) &stReceiveAddr, 0, iLength);

rc = recvfrom(socketHandle, (char*) _pcMessage, (int)

_iMessageLength, 0,
(struct sockaddr *) &stReceiveAddr, (socklen_t*)
&iLength))
if{ rc == 0 )
{
cerr << "ERROR! Socket closed" << endl;
}
else if (rc == -1)
{
cerr << "ERROR! Socket error" << endl;
closeSocket();
}
}

pcIpAddress = inet_ntoa(stReceiveAddr.sin_addr);
shPort = ntohs(stReceiveAddr.sin_port);

cout << "Socket Received: " << _iNumRead << " bytes from "
<< pcIpAddress << ":" << shPort << endl;

...
...

read(): read a specific number of bytes from a file descriptor

int rc = 0; // Actual number of bytes read by function read()
int count = 0; // Running total "count" of bytes read
int numToRead = 32; // Number of bytes we want to read each pass
char buf[512];

...
...

while(bcount < numToRead)

{
// rc is the number of bytes returned.
if( (rc = read(socketHandle, buf, numToRead - count)) > 0);
{
count += rc;
buf += rc; // Set buffer pointer for next read
}
else if(rc < 0)
{
close(socketHandle);
exit(EXIT_FAILURE);
}

cout << "Number of bytes read: " << count << endl;
cout << "Received: " << buf << endl;
}

...
...

send(): send a message from a socket. Used only when in a connected state.
The only difference between send and write is the presence of flags. With zero
flags parameter, send is equivalent to write.
#include <string.h>

...
...

char buf[512];

strcpy(buf,"Message to send");

...
...
 send(socketHandle, buf, strlen(buf)+1, 0);

...
...
TCP send() or UDP sendto():
int iSocketType;
int iBytesSent = 0;
char *pMessage = "message to send";
int iMessageLength = 16; // number of bytes (includes NULL
termination)
sockaddr pSendAddress;

...
...

if (iSocketType == SOCK_STREAM)
{
if ((iBytesSent = send(socketHandle, (char*) pMessage, (int)
iMessageLength, 0)) < 0 )
{
cerr << "Send failed with error " << errno << endl;
close(socketHandle);
}
}
else if (iSocketType == SOCK_DGRAM)
{
if ((iBytesSent = sendto(socketHandle, (char*) pMessage,
(int) iMessageLength,
0,
(struct sockaddr*) pSendAddress,
(int) sizeof(struct sockaddr_in))) < 0 )
{
cerr << "Sendto failed with error " << errno << endl;
close();
}
}
else
{
// Failed - Socket type not defined
}

...
...
Close the socket when done:
#include <unistd.h>

...

close(socketHandle);

...

Socket Server:
Simple Socket Server:

#include <iostream>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#define MAXHOSTNAME 256
using namespace std;

main()
{
struct sockaddr_in socketInfo;
char sysHost[MAXHOSTNAME+1]; // Hostname of this computer we are
running on
struct hostent *hPtr;
int socketHandle;
int portNumber = 8080;

bzero(&socketInfo, sizeof(sockaddr_in)); // Clear structure memory

// Get system information

gethostname(sysHost, MAXHOSTNAME); // Get the name of this computer

we are running on
if((hPtr = gethostbyname(sysHost)) == NULL)
{
cerr << "System hostname misconfigured." << endl;
exit(EXIT_FAILURE);
}

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, 0)) < 0)

{
close(socketHandle);
exit(EXIT_FAILURE);
}

// Load system information into socket data structures

socketInfo.sin_family = AF_INET;
socketInfo.sin_addr.s_addr = htonl(INADDR_ANY); // Use any address
available to the system
socketInfo.sin_port = htons(portNumber); // Set port number

// Bind the socket to a local socket address

if( bind(socketHandle, (struct sockaddr *) &socketInfo,

sizeof(socketInfo)) < 0)
{
close(socketHandle);
perror("bind");
exit(EXIT_FAILURE);
}

listen(socketHandle, 1);
 int socketConnection;
if( (socketConnection = accept(socketHandle, NULL, NULL)) < 0)
{
exit(EXIT_FAILURE);
}
close(socketHandle);

int rc = 0; // Actual number of bytes read

char buf[512];

// rc is the number of characters returned.

// Note this is not typical. Typically one would only specify the
number
// of bytes to read a fixed header which would include the number of
bytes
// to read. See "Tips and Best Practices" below.

rc = recv(socketConnection, buf, 512, 0);

buf[rc]= (char) NULL; // Null terminate string

cout << "Number of bytes read: " << rc << endl;

cout << "Received: " << buf << endl;
}

Forking Socket Server:

In order to accept connections while processing previous connections, use fork()

to handle each connection.

Use establish() and get_connection() to allow multiple connections.

File: serverFork.cpp
#include <iostream>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <netdb.h>
#include <errno.h>
#include <unistd.h>
#include <signal.h>
#include <netinet/in.h>
#define MAXHOSTNAME 256
using namespace std;

// Catch signals from child processes

void handleSig(int signum)
{
while(waitpid(-1, NULL, WNOHANG) > 0);
}

main()
{
 struct sockaddr_in socketInfo;
char sysHost[MAXHOSTNAME+1]; // Hostname of this computer we are
running on
struct hostent *hPtr;
int socketHandle;
int portNumber = 8080;

signal(SIGCHLD, handleSig);

bzero(&socketInfo, sizeof(sockaddr_in)); // Clear structure memory

// Get system information

gethostname(sysHost, MAXHOSTNAME); // Get the name of this computer

we are running on
if((hPtr = gethostbyname(sysHost)) == NULL)
{
cerr << "System hostname misconfigured." << endl;
exit(EXIT_FAILURE);
}

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, 0)) < 0)

{
close(socketHandle);
exit(EXIT_FAILURE);
}

// Load system information into socket data structures

socketInfo.sin_family = AF_INET;
socketInfo.sin_addr.s_addr = htonl(INADDR_ANY); // Use any address
available to the system
socketInfo.sin_port = htons(portNumber); // Set port number

// Bind the socket to a local socket address

if( bind(socketHandle, (struct sockaddr *) &socketInfo,

sizeof(struct sockaddr_in)) < 0)
{
close(socketHandle);
perror("bind");
exit(EXIT_FAILURE);
}

listen(socketHandle, 1);

int socketConnection;
for(;;) // infinite loop to handle remote connections. This should
be limited.
{
if( (socketConnection = accept(socketHandle, NULL, NULL)) < 0)
{
close(socketHandle);
if(errno == EINTR) continue;
perror("accept");
 exit(EXIT_FAILURE);
}
switch(fork())
{
case -1:
perror("fork");
close(socketHandle);
close(socketConnection);
exit(EXIT_FAILURE);
case 0: // Child process - do stuff
close(socketHandle);
// Do your server stuff like read/write messages to the
socket here!
exit(0);
default: // Parent process, look for another connection
close(socketConnection);
continue;
}
}

For more on the use of the fork() function see the YoLinux.com fork() tutorial.

Socket Client:

Simple Socket client:

File: client.cpp
#include <iostream>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <unistd.h>
#include <errno.h>
#define MAXHOSTNAME 256
using namespace std;

main()
{
struct sockaddr_in remoteSocketInfo;
struct hostent *hPtr;
int socketHandle;
char *remoteHost="localhost";
int portNumber = 8080;

bzero(&remoteSocketInfo, sizeof(sockaddr_in)); // Clear structure

memory

// Get system information

if((hPtr = gethostbyname(remoteHost)) == NULL)

{
 cerr << "System DNS name resolution not configured properly." <<
endl;
cerr << "Error number: " << ECONNREFUSED << endl;
exit(EXIT_FAILURE);
}

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, 0)) < 0)

{
close(socketHandle);
exit(EXIT_FAILURE);
}

// Load system information into socket data structures

memcpy((char *)&remoteSocketInfo.sin_addr, hPtr->h_addr, hPtr-

>h_length);
remoteSocketInfo.sin_family = AF_INET;
remoteSocketInfo.sin_port = htons((u_short)portNumber); // Set
port number

if(connect(socketHandle, (struct sockaddr *)&remoteSocketInfo,

sizeof(sockaddr_in)) < 0)
{
close(socketHandle);
exit(EXIT_FAILURE);
}

int rc = 0; // Actual number of bytes read by function read()

char buf[512];

strcpy(buf,"Message to send");
send(socketHandle, buf, strlen(buf)+1, 0);
}

Test Simple Client and Server Socket program:

Note that this runs on a single system using "localhost".

Compile the simple client and the simple server:

• g++ server.cpp -o server

• g++ client.cpp -o client

Start the server: ./server

This will block at the "accept()" call and await a connection from a client.

Start the client: ./client

This will connect to the server and write the message "Message to send". The
server will receive the message and write it out.
Name Resolution and Network Information Lookup:
Network information data lookup for:

• DNS: Name resolution associates IP address to a hostname using DNS

(Domain Name System) name servers. Name resolution invokes a series
of library routines to query the name servers.
• TCP/IP ports and associated services
• Network Protocols
• Network name information

Function Description
gethostname(char *name, size_t len) returns hostname of local host
getservbyname(const char returns a structure of type servent for the
*name,const char *proto) given host name and protocol
gethostbyname(const char *name) returns a structure of type hostent for the
given host name
getservbyport(int port, const char returns a servent structure for the line that
*proto) matches the port port given in network byte
order using protocol proto. If proto is NULL,
any protocol will be matched.
getservent(void) returns a structure servent containing the
broken out fields from the line in
/etc/services

getprotobyname(const char *name) returns a structure protoent containing the

broken out fields from the line in
/etc/protocols

getprotobynumber(int proto) returns a protoent structure for the line that

matches the protocol number
getprotoent(void) returns a structure protoent containing the
broken out fields from the line in
/etc/protocols

getnetbyname(const char *name) a structure netent containing the broken out

fields from the line in /etc/networks
getnetbyaddr(long net, int type) returns a netent structure for the line that
matches the network number net of type
"type"
getaddrinfo(const char *node, const Returns 0 if it succeeds or an error code.
char *service, const struct addrinfo Network address and service translation
*hints, struct addrinfo **res) freeaddrinfo() frees the memory that was
void freeaddrinfo(struct addrinfo *res) allocated by getaddrinfo()
getnetent(void) returns a structure netent containing the
broken out fields from the line in
/etc/networks
Data Structures returned:

hostent defined in <netdb.h>

struct hostent {
char *h_name; /* official name of host */
char **h_aliases; /* alias list */
int h_addrtype; /* host address type */
int h_length; /* length of address */
char **h_addr_list; /* list of addresses */
}
Lookup of data in /etc/hosts or from DNS resolution.

servent defined in <netdb.h>

struct servent {
char *s_name; /* official service name */
char **s_aliases; /* alias list */
int s_port; /* port number */
char *s_proto; /* protocol to use */
}
Lookup of data in /etc/services

protoent defined in <netdb.h>

struct protoent {
char *p_name; /* official protocol name */
char **p_aliases; /* alias list */
int p_proto; /* protocol number */
}
Lookup of data in /etc/protocols

netent defined in <netdb.h>

struct netent {
char *n_name; /* official network name */
char **n_aliases; /* alias list */
int n_addrtype; /* net address type */
unsigned long int n_net; /* network number */
}
Lookup of data in /etc/networks

Socket Configuration Options:

Socket options:
One can "get" (read) the current socket options or "set" them to new values. The
default values are obtained from the OS:

Level Option Type Default Description

IPPROTO_IP TCP_NODELAY int 0 Don't delay send to
coalesce packets. If
set, disable the
Nagle algorithm.
When not set, data
is buffered until
there is a sufficient
amount to send out,
thereby avoiding the
frequent sending of
small packets,
which results in
poor utilization of
the network. Don't
use with
TCP_CORK. This
option is overridden
by TCP_CORK
IPPROTO_IP TCP_MAXSEG int 536 Maximum segment
size for outgoing
TCP packets. TCP
will impose its
minimum and
maximum bounds
over the value
provided.
IPPROTO_IP TCP_CORK int 0 Control sending of
partial frames. If
set, don't send out
partial frames. Not
cross platform.
IPPROTO_IP TCP_KEEPIDLE int 7200 When the
SO_KEEPALIVE
option is enabled,
TCP probes a
connection that has
been idle for some
 amount of time. The
default value for this
idle period is 2
hours. The
TCP_KEEPIDLE
option can be used
to affect this value
for a given socket,
and specifies the
number of seconds
of idle time between
keepalive probes.
Not cross platform.
This option takes an
int value, with a
range of 1 to 32767.
IPPROTO_IP TCP_KEEPINTVL int 75 Specifies the
interval between
packets that are
sent to validate the
connection.
Not cross platform.
IPPROTO_IP TCP_KEEPCNT int 9 When the
SO_KEEPALIVE
option is enabled,
TCP probes a
connection that has
been idle for some
amount of time. If
the remote system
does not respond to
a keepalive probe,
TCP retransmits the
probe a certain
number of times
before a connection
is considered to be
broken. The
TCP_KEEPCNT
option can be used
to affect this value
for a given socket,
 and specifies the
maximum number
of keepalive probes
to be sent. This
option takes an int
value, with a range
of 1 to 32767. Not
cross platform.
IPPROTO_IP TCP_SYNCNT int 5 Number of SYN
retransmits that
TCP should send
before aborting the
attempt to connect.
It cannot exceed
255.
IPPROTO_IP TCP_LINGER2 int 60 Life time of
orphaned FIN-
WAIT-2 state. Not
to be confused with
option SO_LINGER
Not cross platform.
SOL_SOCKET SO_REUSEADDR int 0 Allow local address
(bool) reuse. If a problem
is encountered
when attempting to
bind to a port which
has been closed but
not released (may
take up to 2 minutes
as defined by
TIME_WAIT). Apply
the
SO_REUSEADDR
socket option to
release the
resource
immediately and to
get around the
TIME_WAIT state.
0 = disables, 1 =
enables
SOL_SOCKET SO_REUSEPORT int 0 This option is
(bool) AF_INET socket-
specific. This option
allows multiple
processes to share
a port. All incoming
multicast or
broadcast UDP
datagrams that are
destined for the port
are delivered to all
sockets that are
bound to the port.
All processes that
share the port must
specify this option.
0 = disables, 1 =
enables
SOL_SOCKET SO_ERROR int 0 When an error
(bool) occurs on a socket,
set error variable
so_error and notify
process
0 = disables, 1 =
enables
SOL_SOCKET SO_BROADCAST int 0 Permit sending of
(bool) broadcast
datagrams
0 = disables, 1 =
enables
SOL_SOCKET SO_SNDBUF int 16384 Send buffer size
(value)
SOL_SOCKET SO_RCVBUF int 87380 Receive buffer size
(value)
SOL_SOCKET SO_KEEPALIVE int 0 Periodically test if
(bool) connection is alive
0 = disables, 1 =
enables
SOL_SOCKET SO_SNDTIMEO timeval 0 Set timeout period
(struct) 0 for socket send.
 Disable by setting
timeval.tv_sec = 0
sec, timeval.tv_usec
= 0 usec (default)
Affects write()
writev() send()
sendto() and
sendmsg()
SOL_SOCKET SO_RCVTIMEO timeval 0 Set timeout period
(struct) 0 for socket receive.
Disable by setting
timeval.tv_sec = 0
sec, timeval.tv_usec
= 0 usec (default)
Affects read()
readv() recv()
recvfrom() and
recvmsg()
SOL_SOCKET SO_LINGER linger 0 Specifies how close
(struct) 0 function will operate
for connection
protocols (TCP)
l_onoff: 0 =
disables, 1 =
enables
l_linger: 0 = unsent
data discarded, 1 =
close() does not
return untill all
unsent data is
transmitted or
remote connection
is closed
Structure defined in
sys/socket.h

SOL_SOCKET SO_RCVLOWAT int 1 Specifies number of

(value) bytes used as a
threshold by
select() to
consider a socket
read ready
 SOL_SOCKET SO_SNDLOWAT int 1 Specifies number of
(value) bytes used as a
threshold by
select() to
consider a socket
write ready
SOL_SOCKET SO_TYPE int undefined Specifies socket
(value) type (e.g., tcp
(SOCK_STREAM),
udp
(SOCK_DGRAM),
etc.)
For use with
getsockopt() only.
IPPROTO_IP macro defines are found in /usr/include/netinet/tcp.h
SOL_SOCKET macro defines require /usr/include/sys/socket.h

For a full list of options see the TCP man page

For a full list of IP options see the IP(7) man page

Function Prototypes:
int getsockopt(int s, int level, int optname, void *optval,
socklen_t *optlen);
int setsockopt(int s, int level, int optname, const void *optval,
socklen_t optlen);
getsockopt/setsockopt arguments:

• int sockfd: Socket file descriptor. Returned by call to

"socket".
• int level: See table above
• int optname: See table above
• void *optval: Pointer to value or data structure
• optlen: Length of "optval"
• Returns 0: Sucess, -1: Failure and errno may be set.

Code to read socket options:

File: printSocketOptions.c
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <errno.h>
#include <stdio.h>
int main()
{
int socketHandle;

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, IPPROTO_IP)) < 0)

{
close(socketHandle);
perror("socket");
}

int iSocketOption = 0;
int iSocketOptionLen = sizeof(int);;

struct linger SocketOptionLinger;

int iSocketOptionLingerLen = sizeof(struct linger);;

getsockopt(socketHandle, IPPROTO_TCP, TCP_NODELAY, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_NODELAY = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_MAXSEG, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_MAXSEG = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_CORK, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_CORK = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_KEEPIDLE, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_KEEPIDLE = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_KEEPINTVL, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_KEEPINTVL = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_KEEPCNT, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_KEEPCNT = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_SYNCNT, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_SYNCNT = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_LINGER2, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket TCP_LINGER2 = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_REUSEADDR, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket SO_REUSEADDR = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_ERROR, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket SO_ERROR = %d\n", iSocketOption);
 getsockopt(socketHandle, SOL_SOCKET, SO_BROADCAST, (char
*)&iSocketOption, &iSocketOptionLen);
printf("Socket SO_BROADCAST = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_KEEPALIVE, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket SO_KEEPALIVE = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_SNDBUF, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket SO_SNDBUF = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_RCVBUF, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket SO_RCVBUF = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_LINGER, (char

*)&SocketOptionLinger, &iSocketOptionLingerLen);
printf("Socket SO_LINGER = %d time = %d\n",
SocketOptionLinger.l_onoff, SocketOptionLinger.l_linger);

getsockopt(socketHandle, SOL_SOCKET, SO_RCVLOWAT, (char

*)&iSocketOption, &iSocketOptionLen);
printf("Socket SO_RCVLOWAT = %d\n", iSocketOption);
}

Compile: gcc -o printSocketOptions printSocketOptions.c

getsockopt man page: get a particular socket option for the specified
socket.

Set socket options:

• Socket "keep-alive":

int iOption = 1; // Turn on keep-alive, 0 = disables, 1 = enables

if (setsockopt(socketHandle, SOL_SOCKET, SO_KEEPALIVE, (const char *)

&iOption, sizeof(int)) == SOCKET_ERROR)
{
cerr << "Set keepalive: Keepalive option failed" << endl;
}

Set socket client options:

• Socket re-use:

int iOption = 0; // Reuse address option to set, 0 = disables, 1 =

enables
if (setsockopt(socketHandle, SOL_SOCKET, SO_REUSEADDR, (const char *)
&iOption, sizeof(int)) == SOCKET_ERROR)
{
cerr << "Set reuse address: Client set reuse address option
failed" << endl;
}

• When a socket connection is closed with a call to close(), shutdown() or

exit(), both the client and server will send a FIN (final) packet and will
then send an acknowledgment (ACK) that they received the packet. The
side which initiates the closure will be in a TIME_WAIT state until the
process has been completed. This time out period is generally 2-4 minutes
in duration. It is hoped that all packets are received in a timely manner and
the entire time out duration is not required. When an application is
abnormally terminated, the TIME_WAIT period is entered for the full
duration.

Setting the SO_REUSEADDR option explicitly allows a process to bind a port in

the TIME_WAIT state. This is to avoid the error "bind: Address Already in
Use". One caviat is that the process can not be to the same address and
port as the previous connection. If it is, the SO_REUSEADDR option will not
help and the duration of the TIME_WAIT will be in effect.
For more info see How to avoid the "Address Already in Use" error.

Solution: Enable socket linger:

linger Option;
Option.l_onoff = 1;
Option.l_linger = 0;

if(setsockopt(socketHandle, SOL_SOCKET, SO_LINGER, (const char *)

&Option, sizeof(linger)) == -1)
{
cerr << "Set SO_LINGER option failed" << endl;
}

This allows the socket to die quickly and allow the address to be reused again.
Warning: This linger configuration specified may/will result in data loss upon
socket termination, thus it would not have the robustness required for a banking
transaction but would be ok for a recreational app.

• Broadcast:

int iOption = 0; // Broadcast option to set, 0 = disables, 1 = enables

if (setsockopt(socketHandle, SOL_SOCKET, SO_BROADCAST, (const char *)
&iOption, sizeof(int)) == SOCKET_ERROR)
{
cerr << "Set reuse address: Client set reuse address option
failed" << endl;
}

Struct: remoteSocketInfo.sin_addr.s_addr = htonl(INADDR_BROADCAST);

setsockopt man page: set a particular socket option for the specified socket.

Test Socket Availability:

Function to test if a socket or set of sockets has data and can be read (Test so
you don't get blocked on a read) or written.

#include <sys/select.h>
#include <sys/time.h>

...
...

bool isReadyToRead(int _socketHandle, const long

&_lWaitTimeMicroseconds)
{
int iSelectReturn = 0; // Number of sockets meeting the criteria
given to select()
timeval timeToWait;
int fd_max = -1; // Max socket descriptor to limit search
plus one.
fd_set readSetOfSockets; // Bitset representing the socket we want
to read
// 32-bit mask representing 0-31
descriptors where each
// bit reflects the socket descriptor
based on its bit position.

timeToWait.tv_sec = 0;
timeToWait.tv_usec = _lWaitTimeMicroseconds;

FD_ZERO(&readSetOfSockets);
FD_SET(_socketHandle, &readSetOfSockets);

if(_socketHandle > fd_max)

{
fd_max = _socketHandle;
}

iSelectReturn = select(fd_max + 1, &readSetOfSockets, (fd_set*) 0,

(fd_set*) 0, &timeToWait);

// iSelectReturn -1: ERROR, 0: no data, >0: Number of descriptors

found which pass test given to select()
 if ( iSelectReturn == 0 ) // Not ready to read. No valid
descriptors
{
return false;
}
else if ( iSelectReturn < 0 ) // Handle error
{
cerr << "*** Failed with error " << errno << " ***" << endl;

close(_socketHandle);
return false;
}

// Got here because iSelectReturn > 0 thus data available on at

least one descriptor
// Is our socket in the return list of readable sockets
if ( FD_ISSET(_socketHandle, &readSetOfSockets) )
{
return true;
}
else
{
return false;
}

return false;
}

Arguments to select():

1. int fd_max: Highest socket descriptor number. When

opening a socket with the call socket(), the function returns a
socket descriptor number. The call to select() will loop through all
of the socket descriptors from zero up to fd_max to perform the
"test".
2. fd_set *readSetOfSockets: This is a pointer to the variable
holding the set of bits representing the set of sockets to test for
readability. (Read will not block)
The default number of bytes detected for the socket to be
considered ready to read is 1. To change this default use (in this
example 8 bytes):

int nBytes = 8;
setsockopt(socketHandle, SOL_SOCKET, SO_RCVLOWAT,(const char *)
&nBytes, sizeof(int))

3. fd_set *writeSetOfSockets: This is a pointer to the variable

holding the set of bits representing the set of sockets to test for
writeability. (Write will not block)
 4. fd_set *exceptfds: This is a pointer to the variable holding
the set of bits representing the set of sockets to test for exceptions.
5. struct timeval *timeout: This structure holds the upper
bound on the amount of time elapsed before select returns. It may
be zero, causing select to return immediately. If timeout is NULL
(no timeout), select can block indefinitely.
6. struct timeval {
7. long tv_sec; /* seconds */
8. long tv_usec; /* microseconds */
9. };

Note: Any of the tests (read/write/exceptions) can be set to NULL to ignore

that test.

Also see the select man page

Port Numbers:

The use of port numbers below 1024 are limited to the root process. For a list of
port numbers with established uses see the file /etc/services.

Posts are 16 bit identifiers. Many are reserved and managed by the Internet
Assigned Numbers Authority (IANA).
See RFC 1700.

Host and Network Byte Order:

Note that when transferring data between different platforms or with Java, the
byte order endianess will have to be considered. The network (the neutral
medium) byte order is Big Endian and the byte order to which data is usually
marshalled.

Host processor byte order:

Host Processor Endianness

Intel x86 processor family Little endian
Power PC processor family Big endian
SUN SPARC Big endian
Mips Big endian (IRIX)
Mips Little endian (NT)
Note that it is the processor architecture which determines the endianness and
NOT the OS. The exception is for processors which support both big and little
endian byte ordering, such as the MIPS processor.

Also note: Java processes and stores data in big endian byte ordering on any
platform.

Character data is not subject to this problem as each character is one byte in
length but integer is.

Integer data can be converted from/to host or network byte order with the
following routines:

Function Description
Network to host byte order conversion for long integer data
ntohl()
(uint32_t)
Network to host byte order conversion for short integer data
ntohs()
(uint16_t)
Host to network byte order conversion for long integer data
htonl()
(uint32_t)
Host to network byte order conversion for short integer data
htons()
(uint16_t)
Requires #include <arpa/inet.h>
The routines are aware of the processor they are running on and thus
either perform a conversion or not, depending if it is required.
Man pages:

• ntohl/htonl, ntohs/htons: convert values between host and network byte

order

Note that one uses these calls for portable code on any platform. The port of the
library to that platform determines whether a byte swap will occur and not your
source code.
Include file/code snipit to determine the processor endianess:
File: is_bigendian.h
#ifndef __IS_BIGENDIAN__
#define __IS_BIGENDIAN__

const int bsti = 1; // Byte swap test integer

#define is_bigendian() ( (*(char*)&bsti) == 0 )

#endif // __IS_BIGENDIAN__
Code snipit to swap bytes "in place" and convert endian:

#include <netdb.h>
#include "is_bigendian.h"

/**
In-place swapping of bytes to match endianness of hardware

@param[in/out] *object : memory to swap in-place

@param[in] _size : length in bytes
*/
void swapbytes(void *_object, size_t _size)
{
unsigned char *start, *end;

if(!is_bigendian())
{
for ( start = (unsigned char *)_object, end = start + _size - 1;
start < end; ++start, --end )
{
unsigned char swap = *start;
*start = *end;
*end = swap;
}
}
}

Swaping bit field structures:

#include <netdb.h>
#include "is_bigendian.h"

void swapbytes(void *_object, size_t _size);

struct ExampleA
{
#ifdef BIGENDIAN
unsigned int a:1;
unsigned int b:1;
unsigned int c:1;
unsigned int d:1;
unsigned int e:4;
unsigned int f:8;
unsigned int g:8;
unsigned int h:8;
#else
unsigned int h:8;
unsigned int g:8;
unsigned int f:8;
unsigned int e:4;
unsigned int d:1;
unsigned int c:1;
unsigned int b:1;
unsigned int a:1;
#endif
};

...
...

// Bits:
// |B31....B25 |B24....B16 |B15....B8 |B7....B0 | Big endian
// |B7....B0 |B15....B8 |B24....B16 |B31....B25 | Little endian

// Intel host to network:

// Reverse bit field structure and then byte swap.
// Just byteswap for int.

// Code body

struct ExampleA exampleA;

char tmpStore[4];

...

// assign member variables:

exampleA.a = 0;
exampleA.b = 0;
exampleA.c = 1;
exampleA.d = 1;
exampleA.e = 3;
...

// Use memcpy() because we can't cast a bitfield

memcpy(&tmpStore, &exampleA, sizeof(ExampleA));

swapbytes((void *)&tmpStore, sizeof(ExampleA));

...

A bit reversal routine (use as needed):

unsigned long reverseBitsUint32l(unsigned long x)

{
unsigned long tmplg = 0;
int ii = 0;

for(h = ii = 0; ii < 32; ii++)

{
h = (tmplg << 1) + (x & 1);
x >>= 1;
}

return tmplg;
}

Socket Tips and Best Practices:

 • Re-connect: If a socket client attempts to connect to a socket server and
fails, one should attempt to re-attach after a given waiting period, a fixed
number of times.

...
...

int iTry = 0;
int mRetrymax = 10;
int mRetrywait = 2;

// If this client can not connect to the server, try again after period
"Retrywait".

while ( connect(socketHandle, (struct sockaddr *)&remoteSocketInfo,

sizeof(sockaddr_in)) < 0 )
{
iTry++;
if( iTry > mRetrymax )
{
cerr << "Failed to connect to server. Exceeded maximum
allowable attempts: " << mRetrymax << endl;
break; // Done retrying!
}
sleep( mRetrywait );
}

...

• Check return codes: Just because the socket was opened doesn't mean
it will stay open. Dropped connections do happen. Check all read/write
socket API function return codes:
o return code > 0: Data received/sent
o return code == 0: Socket closed
o return code == -1: Check system errno and interpret (or call
"perror()")

For even more robust code for a socket client, close, then open a new
socket connection for return codes 0 or -1.

• Read entire message: (Variable and fixed length messages) When

sending messages using a socket, be aware that the other side of
connection must know how much to read. This means that you must:
o Send fixed message sizes of a known number of bytes where all
messages sent are of the same size
or
o Send a message with a header of a known size where the message
size is sent in the header. Thus read the fixed size header and
 determine the size of the whole message then read the remaining
bytes.

Note that with a UDP client/server communications, the socket read will
read the whole frame sent. This is not necessarily true for a TCP stream.
The function used to read the stream will return the number of bytes read.
If the required number of bytes have not been read, repeat the read for the
remaining bytes until the whole message has been read.

In this example, our message header is a single integer used to store the
message size in the first 32 bits of the message.

• Check return codes: Just because the socket was opened doesn't mean
it will stay open. Dropped connections do happen. Check all read/write
socket API function return codes:
o return code > 0: Data received/sent
o return code == 0: Socket closed
o return code == -1: Check system errno and interpret (or call
"perror()")

For even more robust code for a socket client, close, then open a new
socket connection for return codes 0 or -1.

• Read entire message: (Variable and fixed length messages) When

sending messages using a socket, be aware that the other side of
connection must know how much to read. This means that you must:
o Send fixed message sizes of a known number of bytes where all
messages sent are of the same size
or
o Send a message with a header of a known size where the message
size is sent in the header. Thus read the fixed size header and
determine the size of the whole message then read the remaining
bytes.

Note that with a UDP client/server communications, the socket read will
read the whole frame sent. This is not necessarily true for a TCP stream.
The function used to read the stream will return the number of bytes read.
If the required number of bytes have not been read, repeat the read for the
remaining bytes until the whole message has been read.

In this example, our message header is a single integer used to store the
message size in the first 32 bits of the message.
 • Check return codes: Just because the socket was opened doesn't mean
it will stay open. Dropped connections do happen. Check all read/write
socket API function return codes:
o return code > 0: Data received/sent
o return code == 0: Socket closed
o return code == -1: Check system errno and interpret (or call
"perror()")

For even more robust code for a socket client, close, then open a new
socket connection for return codes 0 or -1.

• Read entire message: (Variable and fixed length messages) When

sending messages using a socket, be aware that the other side of
connection must know how much to read. This means that you must:
o Send fixed message sizes of a known number of bytes where all
messages sent are of the same size
or
o Send a message with a header of a known size where the message
size is sent in the header. Thus read the fixed size header and
determine the size of the whole message then read the remaining
bytes.

Note that with a UDP client/server communications, the socket read will
read the whole frame sent. This is not necessarily true for a TCP stream.
The function used to read the stream will return the number of bytes read.
If the required number of bytes have not been read, repeat the read for the
remaining bytes until the whole message has been read.

In this example, our message header is a single integer used to store the
message size in the first 32 bits of the message.

• This example applies to either client or server TCP sockets.

Another method of ensuring a complete read of a fixed size message is to use

the MSG_WAITALL flag:

...
...

int flags = MSG_WAITALL;

int rc = recv(socketConnection, buffer, length, flags);

...
...
 • On "SOCK_STREAM" (TCP) sockets, this flag requests that the recv()
function block until the full amount of data specified (length) can be
returned. The function may return the smaller amount of data if the socket
is a message-based socket, if a signal is caught or if the connection is
terminated.
• UDP message order: TCP will guarentee that the order of the message
delivery is maintained. UDP will not guarentee the message delivery order
and thus you will have to maintain a counter in the message if the order is
important.
• Signals: The socket layer can issue signals, which if not handled, can
terminate your application. When a socket is terminated at one end, the
other may receive a SIGPIPE signal.
•

Program received signal SIGPIPE, Broken pipe.

0x000000395d00de45 in send () from /lib64/libpthread.so.0
(gdb) where
#0 0x000000395d00de45 in send () from /lib64/libpthread.so.0
#1 0x00000000004969c5 in bla bla bla
...
...

• Note GDB will report a received signal even if it's being ignored by the
application.
In this example, we tried to send() over a broken socket link.

Set up a signal handler at the beginning of your program to handle SIGPIPE:

#include <stdlib.h>
#include <signal.h>

/// Ignore signal with this handler

void handleSigpipe(int signum)
{
cout << "SIGPIPE ignored" << endl;
return;
}

int main()
{
/// Ignore SIGPIPE "Broken pipe" signals when socket connections are
broken.
signal(SIGPIPE, handleSigpipe);

...
...
}
 • Also see:
o C++ Signal class tutorial - YoLinux.com tutorial
o signal: Signal handling
o sigaction: examine and change a signal action

Socket BSD API man pages:

Sockets API:

• socket: establish socket interface

• gethostname: obtain hostname of system
• gethostbyname: returns a structure of type hostent for the given host
name
• bind: bind a name to a socket
• listen: listen for connections on a socket
• accept: accept a connection on a socket
• connect: initiate a connection on a socket
• setsockopt: set a particular socket option for the specified socket.
• close: close a file descriptor
• shutdown: shut down part of a full-duplex connection

Interrogate a socket:

• select: synchronous I/O multiplexing

• FD_ZERO(), FD_CLR(), FD_SET(), FD_ISSET(): Set socket bit masks
• poll: check on the state of a socket in a set of sockets. The set can be
tested to see if any socket can be written to, read from or if an error
occurred.
• getsockopt: retrieve the current value of a particular socket option for the
specified socket.

Read/Write:

• recv/recvfrom/recvmsg: Read socket

• send/sendto/sendmsg: Write socket

Convert:

• ntohl/htonl, ntohs/htons: convert values between host and network byte

order
• inet_pton: Create a network address structure
• inet_ntop: Parse network address structures

Other supporting system calls:

 • exit: Terminate process
• perror: Output explanation of an error code
• protocols: Network Protocols (see /etc/protocols)

Static, Shared Dynamic and Loadable Linux

Libraries
Why libraries are used:

This methodology, also known as "shared components" or "archive libraries",

groups together multiple compiled object code files into a single file known as a
library. Typically C functions/C++ classes and methods which can be shared by
more than one application are broken out of the application's source code,
compiled and bundled into a library. The C standard libraries and C++ STL are
examples of shared components which can be linked with your code. The benefit
is that each and every object file need not be stated when linking because the
developer can reference the individual library. This simplifies the multiple use and
sharing of software components between applications. It also allows application
vendors a way to simply release an API to interface with an application.
Components which are large can be created for dynamic use, thus the library
remain separate from the executable reducing it's size and thus disk space used.
The library components are then called by various applications for use when
needed.

Linux Library Types:

There are two Linux C/C++ library types which can be created:

1. Static libraries (.a): Library of object code which is linked with, and
becomes part of the application.
2. Dynamically linked shared object libraries (.so): There is only one form of
this library but it can be used in two ways.
1. Dynamically linked at run time but statically aware. The libraries
must be available during compile/link phase. The shared objects
are not included into the executable component but are tied to the
execution.
2. Dynamically loaded/unloaded and linked during execution (i.e.
browser plug-in) using the dynamic linking loader system functions.

Library naming conventions:

Libraries are typically names with the prefix "lib". This is true for all the C
standard libraries. When linking, the command line reference to the library will
not contain the library prefix or suffix.

Thus the following link command: gcc src-file.c -lm -lpthread

The libraries referenced in this example for inclusion during linking are the math
library and the thread library. They are found in /usr/lib/libm.a and
/usr/lib/libpthread.a.

Static Libraries: (.a)

How to generate a library:

• Compile: cc -Wall -c ctest1.c ctest2.c

Compiler options:
o -Wall: include warnings. See man page for warnings specified.
• Create library "libctest.a": ar -cvq libctest.a ctest1.o ctest2.o
• List files in library: ar -t libctest.a
• Linking with the library:
o cc -o executable-name prog.c libctest.a
o cc -o executable-name prog.c -L/path/to/library-directory
-lctest
• Example files:
o ctest1.c

•
o ctest1.c
void ctest1(int *i)
{
*i=5;
}

o ctest2.c
void ctest2(int *i)
{
*i=100;
}

o prog.c

#include <stdio.h>
void ctest1(int *);
void ctest2(int *);

int main()
{
 int x;
ctest1(&x);
printf("Valx=%d\n",x);

return 0;
}

Historical note: After creating the library it was once necessary to run the
command: ranlib ctest.a. This created a symbol table within the archive.
Ranlib is now embedded into the "ar" command.

Note for MS/Windows developers: The Linux/Unix ".a" library is conceptually the
same as the Visual C++ static ".lib" libraries.

Dynamically Linked "Shared Object" Libraries: (.so)

How to generate a shared object: (Dynamically linked object library file.) Note
that this is a two step process.

1. Create object code

2. Create library
3. Optional: create default version using a symbolic link.

Library creation example:

gcc -Wall -fPIC -c *.c
gcc -shared -Wl,-soname,libctest.so.1 -o libctest.so.1.0 *.o
mv libctest.so.1.0 /opt/lib
ln -sf /opt/lib/libctest.so.1.0 /opt/lib/libctest.so
ln -sf /opt/lib/libctest.so.1.0 /opt/lib/libctest.so.1

This creates the library libctest.so.1.0 and symbolic links to it.

Compiler options:

• -Wall: include warnings. See man page for warnings

specified.
• -fPIC: Compiler directive to output position independent
code, a characteristic required by shared libraries. Also see "-fpic".
• -shared: Produce a shared object which can then be linked
with other objects to form an executable.
• -W1: Pass options to linker.
In this example the options to be passed on to the linker are: "-
soname libctest.so.1". The name passed with the "-o" option is
passed to gcc.
• Option -o: Output of operation. In this case the name of the
shared object to be output will be "libctest.so.1.0"
 Library Links:

• The link to /opt/lib/libctest.so allows the naming

convention for the compile flag -lctest to work.
• The link to /opt/lib/libctest.so.1 allows the run time
binding to work. See dependency below.

Compile main program and link with shared object library:

Compiling for runtime linking with a dynamically linked libctest.so.1.0:

gcc -Wall -I/path/to/include-files -L/path/to/libraries
prog.c -lctest -o prog
Use:
gcc -Wall -L/opt/lib prog.c -lctest -o prog

Where the name of the library is libctest.so. (This is why you must
create the symbolic links or you will get the error "/usr/bin/ld: cannot find
-lctest".)
The libraries will NOT be included in the executable but will be
dynamically linked during runtime execution.

List Dependencies:

The shared library dependencies of the executable can be listed with the
command: ldd name-of-executable

Example: ldd prog

libctest.so.1 => /opt/lib/libctest.so.1
(0x00002aaaaaaac000)
libc.so.6 => /lib64/tls/libc.so.6 (0x0000003aa4e00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003aa4c00000)

Run Program:

• Set path: export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH

• Run: prog

Man Pages:

• gcc - GNU C compiler

• ld - The GNU Linker
• ldd - List dependencies

Links:
 • LDP: Shared libraries

Library Path:

In order for an executable to find the required libraries to link with during run time,
one must configure the system so that the libraries can be found. Methods
available: (Do at least one of the following)

1. Add library directories to be included during dynamic linking to the file

/etc/ld.so.conf

Sample: /etc/ld.so.conf

/usr/X11R6/lib
/usr/lib
...
..
/usr/lib/sane
/usr/lib/mysql
/opt/lib

Add the library path to this file and then execute the command (as root)
ldconfig to configure the linker run-time bindings.
You can use the "-f file-name" flag to reference another configuration file
if you are developing for different environments.
See man page for command ldconfig.

OR

2. Add specified directory to library cache: (as root)

ldconfig -n /opt/lib
Where /opt/lib is the directory
containing your library libctest.so
(When developing and just adding your current directory: ldconfig -n .
Link with -L.)

This will NOT permanently configure the system to include this directory.
The information will be lost upon system reboot.

OR

3. Specify the environment variable LD_LIBRARY_PATH to point to the directory

paths containing the shared object library. This will specify to the run time
loader that the library paths will be used during execution to resolve
dependencies.
 (Linux/Solaris: LD_LIBRARY_PATH, SGI: LD_LIBRARYN32_PATH, AIX: LIBPATH,
Mac OS X: DYLD_LIBRARY_PATH, HP-UX: SHLIB_PATH)

Example (bash shell): export LD_LIBRARY_PATH=/opt/lib:

$LD_LIBRARY_PATH or add to your ~/.bashrc file:

...
if [ -d /opt/lib ];
then
LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH
fi

...

export LD_LIBRARY_PATH

This instructs the run time loader to look in the path described by the
environment variable LD_LIBRARY_PATH, to resolve shared libraries. This
will include the path /opt/lib.

Library paths used should conform to the "Linux Standard Base" directory
structure.

Library Info:

The command "nm" lists symbols contained in the object file or shared library.

Use the command nm -D libctest.so.1.0

(or nm --dynamic libctest.so.1.0)

0000000000100988 A __bss_start
000000000000068c T ctest1
00000000000006a0 T ctest2
w __cxa_finalize
00000000001007b0 A _DYNAMIC
0000000000100988 A _edata
0000000000100990 A _end
00000000000006f8 T _fini
0000000000100958 A _GLOBAL_OFFSET_TABLE_
w __gmon_start__
00000000000005b0 T _init
w _Jv_RegisterClasses

Man page for nm

Symbol
Description
Type
 The symbol's value is absolute, and will not be changed by
A
further linking.
B Un-initialized data section
D Initialized data section
T Normal code section
Undefined symbol used but not defined. Dependency on
U
another library.
Doubly defined symbol. If found, allow definition in another
W
library to resolve dependency.

Also see: objdump man page

Library Versions:

Library versions should be specified for shared objects if the function interfaces
are expected to change (C++ public/protected class definitions), more or fewer
functions are included in the library, the function prototype changes (return data
type (int, const int, ...) or argument list changes) or data type changes (object
definitions: class data members, inheritance, virtual functions, ...).

The library version can be specified when the shared object library is created. If
the library is expected to be updated, then a library version should be specified.
This is especially important for shared object libraries which are dynamically
linked. This also avoids the Microsoft "DLL hell" problem of conflicting libraries
where a system upgrade which changes a standard library breaks an older
application expecting an older version of the the shared object function.

Versioning occurs with the GNU C/C++ libraries as well. This often make binaries
compiled with one version of the GNU tools incompatible with binaries compiled
with other versions unless those versions also reside on the system. Multiple
versions of the same library can reside on the same system due to versioning.
The version of the library is included in the symbol name so the linker knows
which version to link with.

One can look at the symbol version used: nm csub1.o

00000000 T ctest1

No version is specified in object code by default.

ld and object file layout
There is one GNU C/C++ compiler flag that explicitly deals with symbol
versioning. Specify the version script to use at compile time with the flag:
--version-script=your-version-script-file
Note: This is only useful when creating shared libraries. It is assumed that the
programmer knows which libraries to link with when static linking. Runtime linking
allows opportunity for library incompatibility.

GNU/Linux, see examples of version scripts here:

sysdeps/unix/sysv/linux/Versions

Some symbols may also get version strings from assembler code which appears
in glibc headers files. Look at include/libc-symbols.h.

Example: nm /lib/libc.so.6 | more

00000000 A GCC_3.0
00000000 A GLIBC_2.0
00000000 A GLIBC_2.1
00000000 A GLIBC_2.1.1
00000000 A GLIBC_2.1.2
00000000 A GLIBC_2.1.3
00000000 A GLIBC_2.2
00000000 A GLIBC_2.2.1
00000000 A GLIBC_2.2.2
00000000 A GLIBC_2.2.3
00000000 A GLIBC_2.2.4
...
..

Note the use of a version script.

Library referencing a versioned library: nm /lib/libutil-2.2.5.so

..
...
U strcpy@@GLIBC_2.0
U strncmp@@GLIBC_2.0
U strncpy@@GLIBC_2.0
...
..

Links:

• Symbol versioning
• GNU.org: ld version scripts

Dynamic loading and un-loading of shared libraries using libdl:

These libraries are dynamically loaded / unloaded and linked during execution.
Usefull for creating a "plug-in" architecture.

Prototype include file for the library: ctest.h

#ifndef CTEST_H
#define CTEST_H

#ifdef __cplusplus
extern "C" {
#endif

void ctest1(int *);

void ctest2(int *);

#ifdef __cplusplus
}
#endif

#endif

Use the notation extern "C" so the libraries can be used with C and C++.
This statement prevents the C++ from name mangling and thus creating
"unresolved symbols" when linking.

Load and unload the library libctest.so (created above), dynamically:

#include <stdio.h>
#include <dlfcn.h>
#include "ctest.h"

int main(int argc, char **argv)

{
void *lib_handle;
double (*fn)(int *);
int x;
char *error;

lib_handle = dlopen("/opt/lib/libctest.so", RTLD_LAZY);

if (!lib_handle)
{
fprintf(stderr, "%s\n", dlerror());
exit(1);
}

fn = dlsym(lib_handle, "ctest1");
if ((error = dlerror()) != NULL)
{
fprintf(stderr, "%s\n", error);
exit(1);
}
 (*fn)(&x);
printf("Valx=%d\n",x);

dlclose(lib_handle);
return 0;
}

gcc -rdynamic -o progdl progdl.c -ldl

Explanation:

• dlopen("/opt/lib/libctest.so", RTLD_LAZY);
Open shared library named "libctest.so".
The second argument indicates the binding. See include file dlfcn.h.
Returns NULL if it fails.
Options:
o RTLD_LAZY: If specified, Linux is not concerned about unresolved
symbols until they are referenced.
o RTLD_NOW: All unresolved symbols resolved when dlopen() is
called.
o RTLD_GLOBAL: Make symbol libraries visible.
• dlsym(lib_handle, "ctest1");
Returns address to the function which has been loaded with the shared
library..
Returns NULL if it fails.
Note: When using C++ functions, first use nm to find the "mangled" symbol
name or use the extern "C" construct to avoid name mangling.
i.e. extern "C" void function-name();

Object code location: Object code archive libraries can be located with either
the executable or the loadable library. Object code routines used by both should
not be duplicated in each. This is especially true for code which use static
variables such as singleton classes. A static variable is global and thus can only
be represented once. Including it twice will provide unexpected results. The
programmer can specify that specific object code be linked with the executable
by using linker commands which are passed on by the compiler.

Use the "-Wl" gcc/g++ compiler flag to pass command line arguments on to the
GNU "ld" linker.

Example makefile statement: g++ -rdynamic -o appexe $(OBJ) $(LINKFLAGS)

-Wl,--whole-archive -L{AA_libs} -laa -Wl,--no-whole-archive $(LIBS)
 • --whole-archive: This linker directive specifies that the libraries listed
following this directive (in this case AA_libs) shall be included in the
resulting output even though there may not be any calls requiring its
presence. This option is used to specify libraries which the loadable
libraries will require at run time.
• -no-whole-archive: This needs to be specified whether you list additional
object files or not. The gcc/g++ compiler will add its own list of archive
libraries and you would not want all the object code in the archive library
linked in if not needed. It toggles the behavior back to normal for the rest
of the archive libraries.

Man pages:

• dlopen() - gain access to an executable object file

• dclose() - close a dlopen object
• dlsym() - obtain the address of a symbol from a dlopen object
• dlvsym() - Programming interface to dynamic linking loader.
• dlerror() - get diagnostic information

Links:

• Shared Libraries-Dynamic Loading and Unloading

• GNOME Glib dynamic loading of modules - cross platform API for
dynamically loading "plug-ins".

C++ class objects and dynamic loading:

C++ and name mangling:

When running the above "C" examples with the "C++" compiler one will quickly
find that "C++" function names get mangled and thus will not work unless the
function definitions are protected with extern "C"{}.

Note that the following are not equivalent:

extern "C" extern "C" int functionx();
{
int functionx();
}

The following are equivalent:

extern "C" extern "C" int functionx();
{
extern int functionx();
}
Dynamic loading of C++ classes:

The dynamic library loading routines enable the programmer to load "C"
functions. In C++ we would like to load class member functions. In fact the entire
class may be in the library and we may want to load and have access to the
entire object and all of its member functions. Do this by passing a "C" class
factory function which instantiates the class.

The class ".h" file:

class Abc {

...
...

};

// Class factory "C" functions

typedef Abc* create_t;

typedef void destroy_t(Abc*);

The class ".cpp" file:

Abc::Abc()
{
...
}

extern "C"
{
// These two "C" functions manage the creation and destruction of the
class Abc

Abc* create()
{
return new Abc;
}

void destroy(Abc* p)
{
delete p; // Can use a base class or derived class pointer here
}
}

This file is the source to the library. The "C" functions to instantiate
(create) and destroy a class defined in the dynamically loaded library
where "Abc" is the C++ class.
Main executable which calls the loadable libraries:
// load the symbols
create_t* create_abc = (create_t*) dlsym(lib_handle, "create");

...
 ...

destroy_t* destroy_abc = (destroy_t*) dlsym(lib_handle, "destroy");

...
...

Pitfalls:

• The new/delete of the C++ class should both be provided by the

executable or the library but not split. This is so that there is no surprise if
one overloads new/delete in one or the other.

Links:

• LinuxJournal.com: Dynamic Class Loading for C++ on Linux

• dlopen howto

Comparison to the Microsoft DLL:

The Microsoft Windows equivalent to the Linux / Unix shared object (".so") is the
".dll". The Microsoft Windows DLL file usually has the extension ".dll", but may
also use the extension ".ocx". On the old 16 bit windows, the dynamically linked
libraries were also named with the ".exe" suffix. "Executing" the DLL will load it
into memory.

The Visual C++ .NET IDE wizard will create a DLL framework through the GUI,
and generates a ".def" file. This "module definition file" lists the functions to be
exported. When exporting C++ functions, the C++ mangled names are used.
Using the Visual C++ compiler to generate a ".map" file will allow you to discover
the C++ mangled name to use in the ".def" file. The "SECTIONS" label in the
".def" file will define the portions which are "shared". Unfortunately the generation
of DLLs are tightly coupled to the Microsoft IDE, so much so that I would not
recomend trying to create one without it.

The Microsoft Windows C++ equivalent functions to libdl are the following
functions:

• ::LoadLibrary() - dlopen()
• ::GetProcAddress() - dlsym()
• ::FreeLibrary() - dlclose()

[Potential Pitfall]: Microsoft Visual C++ .NET compilers do not allow the linking
controll that the GNU linker "ld" allows (i.e. --whole-archive, -no-whole-archive).
All symbols need to be resolved by the VC++ compiler for both the loadable
library and the application executable individually and thus it can cause
duplication of libraries when the library is loaded. This is especially bad when
using static variables (i.e. used in singleton patterns) as you will get two memory
locations for the static variable, one used by the loadable library and the other
used by the program executable. This breaks the whole static variable concept
and the singleton pattern. Thus you can not use a static variable which is
referenced by by both the loadable library and the application executable as they
will be unique and different. To use a unique static variable, you must pass a
pointer to that static variable to the other module so that each module (main
executable and DLL library) can use the same instatiation. On MS/Windows you
can use shared memory or a memory mapped file so that the main executable
and DLL library can share a pointer to an address they both will use.

Cross platform (Linux and MS/Windows) C++ code snippet:

Include file declaration: (.h or .hpp)
class Abc{
public:
static Abc* Instance(); // Function declaration. Could also be used
as a public class member function.

private:
static Abc *mInstance; // Singleton. Use this declaration in C++
class member variable declaration.
...
}

C/C++ Function source: (.cpp)

/// Singleton instantiation
Abc* Abc::mInstance = 0; // Use this declaration for C++ class member
variable
// (Defined outside of class definition in
".cpp" file)

// Return unique pointer to instance of Abc or create it if it does not

exist.
// (Unique to both exe and dll)

static Abc* Abc::Instance() // Singleton

{
#ifdef WIN32
// If pointer to instance of Abc exists (true) then return instance
pointer else look for
// instance pointer in memory mapped pointer. If the instance
pointer does not exist in
// memory mapped pointer, return a newly created pointer to an
instance of Abc.

return mInstance ?
mInstance : (mInstance = (Abc*)
 MemoryMappedPointers::getPointer("Abc")) ?
mInstance : (mInstance = (Abc*)
MemoryMappedPointers::createEntry("Abc",(void*)new Abc));
#else
// If pointer to instance of Abc exists (true) then return instance
pointer
// else return a newly created pointer to an instance of Abc.

return mInstance ? mInstance : (mInstance = new Abc);

#endif
}

Windows linker will pull two instances of object, one in exe and one in loadable
module. Specify one for both to use by using memory mapped pointer so both
exe and loadable library point to same variable or object.
Note that the GNU linker does not have this problem.
For more on singletons see the YoLinux.com C++ singleton software design
pattern tutorial.
Cross platform programming of loadable libraries:
#ifndef USE_PRECOMPILED_HEADERS
#ifdef WIN32
#include <direct.h>
#include <windows.h>
#else
#include <sys/types.h>
#include <dlfcn.h>
#endif
#include <iostream>
#endif

using namespace std;

#ifdef WIN32
HINSTANCE lib_handle;
#else
void *lib_handle;
#endif

// Where retType is the pointer to a return type of the function

// This return type can be int, float, double, etc or a struct or
class.

typedef retType* func_t;

// load the library

-------------------------------------------------
#ifdef WIN32
string nameOfLibToLoad("C:\opt\lib\libctest.dll");
lib_handle = LoadLibrary(TEXT(nameOfLibToLoad.c_str()));
if (!lib_handle) {
cerr << "Cannot load library: " << TEXT(nameOfDllToLoad.c_str())
<< endl;
}
 #else
string nameOfLibToLoad("/opt/lib/libctest.so");
lib_handle = dlopen(nameOfLibToLoad.c_str(), RTLD_LAZY);
if (!lib_handle) {
cerr << "Cannot load library: " << dlerror() << endl;
}
#endif

...
...
...

// load the symbols

-------------------------------------------------
#ifdef WIN32
func_t* fn_handle = (func_t*) GetProcAddress(lib_handle,
"superfunctionx");
if (!fn_handle) {
cerr << "Cannot load symbol superfunctionx: " << GetLastError()
<< endl;
}
#else
// reset errors
dlerror();

// load the symbols (handle to function "superfunctionx")

func_t* fn_handle= (func_t*) dlsym(lib_handle, "superfunctionx");
const char* dlsym_error = dlerror();
if (dlsym_error) {
cerr << "Cannot load symbol superfunctionx: " << dlsym_error <<
endl;
}
#endif

...
...
...

// unload the library

-----------------------------------------------

#ifdef WIN32
FreeLibrary(lib_handle);
#else
dlclose(lib_handle);
#endif

Tools:

Man pages:

• ar - create, modify, and extract from archives

• ranlib - generate index to archive
• nm - list symbols from object files
• ld - Linker
• ldconfig - configure dynamic linker run-time bindings
ldconfig -p : Print the lists of directories and candidate libraries stored in
the current cache.
i.e. /sbin/ldconfig -p |grep libGL
• ldd - print shared library dependencies
• gcc/g++ - GNU project C and C++ compiler
• man page to: ld.so - a.out dynamic linker/loader