SYSTEMS PROGRAMMING D M Dhamdhere @)@ Tata McGraw-Hill Published by Tata McGraw Hill Education Private Limited, 7 West Patel Nagar, New Delhi 110 008 Systems Programming Copyright © 2011 by Tata McGraw Hill Education Private Limited. First reprint 2011 RBZCRRBGRCOQZ, No part of this publication may be reproduced or distributed in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise or stored in a database or retrieval system without the prior written permission of the publishers. The program listings (if any) may be entered, stored and executed in a computer system, but they may not be reproduced for publication. This edition can be exported from India only by the publishers, Tata McGraw Hill Education Private Limited. ISBN (13): 978-0-07-133311-5 ISBN (10): 0-07-133311-8 Vice President and Managing Director—MeGraw-Hill Education, Asia Pacific Region: Ajay Shukla Head—Higher Education Publishing and Marketing: Vibha Mahajan Publishing Manager—SEM & Tech Ed.: Shalini Jha Editorial Researcher: Noaman Khan Sr Copy Editor: Nimisha Kapoor Sr Production Manager: Satinder S Baveja Marketing Manager—Higher Education: Vijay S Jagannathan Sr Product Specialist—SEM & Tech Bd.: John Mathews General Manager—Production: Rajender P Ghansela Asst General Manager—Produetion: B L Dogra Information contained in this work has been obtained by Tata McGraw-Hill, from sources believed to be reliable, However, neither Tata McGraw-Hill nor its authors guarantee the accuracy or completeness of any information published herein, and neither Tata McGraw-Hill nor its authors shall be responsible for any errors, omissions, or damages arising out of use of this information. This work is published with the understanding that Tata McGraw-Hill and its authors are supplying information but are not attempting to render engineering or other professional services. If such services are required, the assistance of an appropriate professional should be sought. ‘Typeset at Script Makers, 19, Al-B, DDA Market, Paschim Vihar, New Delhi 110 063 and text printed at Adarsh Printers, C-50-51, Mohan Park, Naveen Shahdara, Delhi ~ 110032 Cover Designer: Meenu Raghav Cover Printer: A. B. OffsetContents Preface x 1._Introduction J 1.1 What is System Software? 2 1.2 Goals of System Software 4 1.2.1 User Convenience 5 1.2.2 Efficient Use 6 1.2.3 Non-interference 6 1.3 System Programs and Systems Programming 8 1.4 The Wonderland of System Software 8 1.4.1 The Program Development and Production Environments 9 1.4.2 Making Software Portable ul 1.4.3 Realizing Benefits of the Internet 12 1.4.4 Treating Programs as Components 13 1.4.5 Quick-and-Dirty Programming 14 1.4.6 The Embedded System Environment 15 1.4.7 Dynamic Specification, Flexibility, and Adaptive Software 16 1.5 Views of System Software 19 1.5.1 The User-Centric View of System Software 20 1.5.2 The System-Centric View of System Software 20 ‘1.6 Overview of the Book 1.7 Summary 2 Test Your Concepts 23 Bibliography 23 Part | : Language Processors pS 2. Overview of Language Processors a 2.1 Programming Languages and Language Processors 28 2.1.1 Kinds of Language Processors 30 2.1.2 Procedure Oriented and Problem Oriented Programming Languages 32 2.2 Language Processing Activities 33 2.2.1 Program Generation 34vi_Contents 2.2.1.1 Program Generators for Specific Application Domains 2.2.1.2 A General Purpose Program Generator 2.22 Progam Execution 2.3 Fundamentals of Language Processing Multi-Pass Organization of Language Processors ‘A Toy Compiler 2.3.2.1 The Front End 2.3.22 The Back End 2.4 Symbol Tables 2.4.1 Symbol Table Entry Formats 2.4.2 Symbol Table Organizations Using Linear Data Structures 2.4.3 Linked List and Tree-Structured Symbol Table Organizations 2.5 Summary Test Your Concepts Exercises liography 3. Assemblers 3.1 Elements of Assembly Language Programming 3.1.1 Assembly Language Statements 3.1.2 Benefits of Assembly Language 3.2 A Simple Assembly Scheme 3.3 Pass Structure of Assemblers 3.4 Design of a Two-Pass Assembler 3.4.1 Advanced Assembler Directives 3.4.2 Pass | of the Assembler 3.4.3 Intermediate Code Forms 3.4.4 Intermediate Code for Imperative Statements 3.4.5 Processing of Declarations and Assembler Directives 3.4.6 Pass II of the Assembler 3.4.7 1m Listing and Error Rk 3.4.8 Some Organizational Issues 3.5 A Single-Pass Assembler for Intel x86 Family Processors ‘LS.1 The Architecture of Intel 8088 3.5.2 Intel 8088 Instructions 3.5.3 The Assembly Language of Intel 8088 3.5.4 Problems of Single-Pass Assembly 3.5.5 Design of the Assembler 3.5.6 Algorithm of the Single-Pass Assembler 3.6 Summary Test Your Concepts Exercises Bibliography BEBEREBEESeeseseeContents _vii 4. Macros and Macro Preprocessors 123 4.1 Introduction 124 4.2 Macro Definition and Call 288 4.3 Macro Expansion 126 4.4 Nested Macro Calls 131 4.5 Advanced Macro Facilities 132 4.5.1 Conditional Expansion 135 4.5.2 Expansion Time Loops 136 4.8.3 Semantic Expansion 138 4.6 Design of a Macro Preprocessor 139 4.6.1 Design Overview 139 4.6.2 Data Structures of the Macro Preprocessor 141 4.6.3 Processing of Macro Definitions 144 4.6.4 Macro Expansion 148 4.6.5 Handling Nested Macro Calls 149 4.6.6 The Stack Data Structure 151 4.6.6.1 The Extended Stack Model ISI 4.6.7 Use of Stacks in Expansion of Nested Macro Calls 153 4.6.8 Design of a Macro Assembler 155 4.7 Summary 157 Test Your Concepts 158 Exercises Bibliography 160 S.Linkers and Loaders 5.1 Introduction 162 5.2 Relocation and Linking Concepts 164 5.2.1 Program Relocation 164 $.2.2 Linking 166 5.2.3 Object Module 168 5.3 Design of a Linker 169 4.3.1 Scheme for Relocation TO 5.3.2 Scheme for Linking 171 5.4 Self-Relocating Programs 173 5.5 Linking in MS DOS. 174 5.5.1 Relocation and Linking Requirements in Segment-Based Addressi 174 5.5.2 Object Module Format 176 5.5.3 Design of the Linker 182 5.6 Linking of Overlay Structured Programs 187 5.7 Dynamic Linking 190 5.8 Loaders 191 5.8.1 Absolute Loaders 191 5.8.2 Relocating Loaders 192 5.9 Summary 193i Grammars 6.1.1 Classification of Grammars 6.1.2 Ambiguity in Grammatic Specification 6.2 Scanning 6.3 Parsing 6.3.1 Top-Down Parsing 6.3.1.1 Practical Top-Down Parsing 6.3.2 Bottom-Up Parsing 6.3.2.1 Operator Precedence Parsing 6.4 Language Processor Development Tools 6.4.1 LEX 6.4.2 YACC 6.5 Summary Test Your Concepts Exercises Bibliography 7. Compilers 7.1 Causes of a Large Semantic Gap 7.2 Binding and Binding Times 7.3 Data Structures Used in Compilers 23.1 Stack 7.3.2 Heap 7.4 Scope Rules 7.5 Memory Allocation 7.5.1 Static and Dynamic Memory Allocation 75.2 ic Me Allocation and Access 7.5.3 Memory Allocation and Deallocation 7.5.3.1 Accessing Local and Nonlocal Variables 7.5.3.2 Symbol Table Requirements 7.5.3.3 Recursion 7.8.3.4 Array Allocation and Access 7.6 Compilation of Expressions 7.6.1 A Toy Code Generator for Expressions 7.6.2 Intermediate Codes for Expressions 1.6.3 Postfix Notation 7.6.4 Triples and Quadruples 7.6.5 Expression Trees 7.7 Compilation of Control Structures 7.7.1 Function and Procedure Calls RBREER EEE x BER 236 237 239 241 & 243 BRSERE 253 255 259 B 261 263 267 268 272 273 282 284 285 289 290Contents _ix 7.8 Code Optimization 296} 7.8.1 Optimizing Transformations 297 7.8.2 Local Optimization 300 7.8.3 Global Optimization 303 7.8.3.1 Program Representation for Global Optimization 304 7.8.3.2 Control Flow Analysis 305 7.8.3.3 Data Flow Analysis 305 7.9 Summary 309 Test Your Concepts 310 Exercises ju Bibliography 313 8. Interpreters 35, 8.1 Benefits of Interpretation 316 8.2 Overview of Interpretation 317 8.2.1 A Toy Interpreter 317 8.2.2 Pure and Impure Interpreters 320 8.3 The Java Language Environment 321 8.3.1 Java Virtual Mache 3D 8.4 Summary 326 Test Your Concepts 327 Exercises 327 Bibliography 327 9. Software Tools 329 9.1 What is a Software Tool? 330 9.2 Software Tools for Program Development 330 9.2.1 Program Design and Coding 331 9.2.2 Program Entry and Editing 331 9.2.3 Program Testing and Debugging 331 9.2.4 Program Performance Tuning 335 9.2.5 Source Code Management and Version Control 336 9.2.6 Program Documentation Aids 339 9.2.7 Design of Software Tools 339 9.3 Editors 341 341 341 9.3.3 Structure Editors 342 9.3.4 Design of an Editor 342 9.4 Debug Monitors 343 9.4.1 Testing Assertions 348 9.5 Programming Environments 345 9.6 User Interfaces 347 9.6.1 Command Dialogs 348 9.6.2 Presentation of Data 349X_Contents 9.6.3 On-Line Help 349 9.6.4 Structure of a User Interface 350 9.6.5 User Interface Management Systems 350 9.7 Summary 351 Test your concepts 352 Exercises 552 Bibliography 353 Part Il: Operating Systems 355 10. Overview of Operating Systems 357 10.1 Fundamental Principles of OS Operation 358 10.2 The Computer 360 10.2.1 The CPU 360 10.2.2 Memory Hierarchy 363 10.2.3 Input/Output 367 10.2.4 Interrupts 368 10.3 OS Interaction with the Computer and User Programs 371 10.3.1 Contfolling Execution of Programs 371 10.3.2 Interrupt Servicing 372 10.3.3 System Calls 376 10.4 Structure of Operating Systems 378 10.4.1 Portability and Extensibility of Operating Systems 378 10.4.2 Kernel-Based Operating Systems 379 10.4.3 Microkernel-Based Operating Systems 380 10.5 Computing Environments and Nature of Computations 381 10.6 Classes of Operating Systems 384 10.7 Batch Processing Systems 386 10.8 Multiprogramming Systems 388 10.8.1 Priority of Programs 390 10.9 Time Sharing Systems 393 10.9.1 Swapping of Programs 396 10.10 Real Time Operating Systems 397 10.10.1 Hard and Soft Real Time Systems 397 10.11 Multiprocessor Operating Systems 398 10.12 Distributed Operating Systems 400 10.13 Virtual Machine Operating Systems 402 10.14 Modern Operating Systems 404 10.15 Summary 405 Test Your Concepts 407 Exercises 407 Bibliography 409Contents _xi Il. Program Management 11.1 Processes and Programs 11.1.1 What is a process? 11.1.2 Relationships Between Processes and Programs 11.1.3 Child Processes 11.1.4 Concurrency and Parallelism 11.2 Implementing Processes 11.2.1 Process States and State Transitions 11.2.2 Events Pertaining to a Process 11.2.3 Process Control Block 11.2.4 Data Sharing, Message Passing and Synchronization Between Processes 11.3 Process Scheduling 11.3.1 Scheduling Terminology and Concepts 11.3.2 Fundamental Techniques of Scheduling 11.3.3 Round-Robin Scheduling with Time Slicing (RR) 11.3.4 Multilevel Scheduling 11.4 Threads: 11.5 Summary Test Your Concepts Exercises Bibliography 12, Memory Management 12.1 Managing the Memory Hierarchy 12.1.1 Static and Dynamic Memory Allocation 12.2 Memory Allocation to a Process 12.2.1 Stacks and Heaps 12.2.2 Reuse of Memory 12.2.2.1 Maintaining a Free List 12.2.2.2 Performing Fresh Allocations Using a Free List 12.2.2.3 Memory Fragmentation 12.2.3 Buddy System Allocator 12.2.4 The Memory Allocation Model 12.2.5 Memory Protection 12.3 Contiguous Memory Allocation 12.4 Virtual Memory 12.5 Virtual Memory Using Paging 12.5.1 Memory Protection 12.5.2 Demand Paging 12.5.3 Page Replacement Algorithms 12.5.4 Controlling Memory Allocation to a Process 12.6 Summary Test Your Concepts Exercises Bibliography ail 412 412 413 414 416 416 417 419 420 421 423 424 425 426 428 429 430 431 431 432 435 436 437 437 438 438 439 443 445 447 449 452 452 487 459 461 462 463xii_Contents 13. File Systems 465 13.1 Overview of File Processing 466 13.1.1 File System and the IOCS 466 13.1.2 File Processing in a Program 467 13.2 Files and File Operations 468 13.3 Fundamental File Organizations 469 469 470 13.3.3 Indexed and Index Sequential File Organizations 470 13.4 Directories 471 13.5 Allocation of Disk Space 474 13.5.1 Linked Allocation 474 13.5.2 Indexed Allocation 475 13.6 File Protection 476 13.7 File System Reliability 477 13.7.1 Recovery Techniques 477 13.8 Implementing an /O Operation 478 13.9 Overview of I/O Organization 478 13.10 Device Level /O 480 13.10.1 /O Programming 480 13.11 The Input-Output Control System 481 13.11.1 Logical Devices 482 13.11.2 IOCS Data Structures 482 13.11.3 Overview of IOCS Operation 483 13.11.4 Device Drivers 484 13.11.5 Disk Scheduling 485 13.12 Buffering of Records 486 13.13 Blocking of Records 492 13.14 Disk and File Caches 495 13.15 Summary 496 Test Your Concepts 497 Exercises 497 Bibliography 498 14. Security and Protection 500 14.1 Overview of Security and Protection 501 14.1.1 Goals of Security and Protection 503 14.1.2 Security and Protection Threats 504 14.2 Security Attacks 504 14.2.1 Trojan Horses, Viruses and Worms 505, 14.2.2 The Buffer Overflow Technique 508 143 Encryption . 510 14.4 Authentication and Password Security Sul 14.5 Protection Structures 512145.1 Access Control Matrix 14.5.2 Access Control Lists (ACLs) 14.6 Classifications of Computer Security 14.7 Security Attacks in Distributed Systems 14.8 Message Security 14.8.1 Preventing Message Replay Attacks 14.9 Authentication of Data and Messages 14.9.1 Certification Authorities and Digital Certificates 14.9.2 Message Authentication Code and Digital Signature 14.10 Summary Test Your Concepts Exercises Bibliography Index Contents _ xiii 513 514 S15 516 S17 520 $21 521 522 523 525 526 526Copyrighted materialPreface Easy development of new programs and applications, and their efficient operation on a computer are the primary concerns of computer users. However, both the concerns cannot be satisfied simultaneously. This situation has led to different kinds of schemes for program development which provide ease of program development to varying degrees and to many schemes for program execution which offer varying levels of execution efficiency. A user has to choose a scheme of each kind to obtain a suitable combination of ease of program development and efficiency of operation. Programs that implement the schemes for program development and program execution are called system programs and the collection of system programs of a computer is called system software, A course on systems programming deals with the fundamental concepts and techniques used in the design and implementation of system programs, and their properties concerning speed of program development and efficiency of operation. Accordingly, a book on systems programming has to focus on numerous topics—support for quick and efficient development of programs, design of adaptive and extensible programs that can be easily modified to provide new functionalities, models for execution of programs written in programming languages, and interactions among user programs, the operating system, and the computer to achieve efficient and secure operation of the computer. My previous book Systems Programming and Operating Systems covered the complete area of system software. It was used for three kinds of courses—systems programming, compilers, and operating systems. However, it was a large book that needed to get even larger to keep pace with developments in this field, so I decided to offer two separate books in this area. This book focuses primarily on systems programming. It can be used for a course on systems programming that contains ‘a small module on operating systems and also for a course on compilers. It is expected that instructors and students of courses on operating systems would use my other book Operating Systems—A Concept- Based Approach. General approach Diversity of computer systems and system programs has been a major challenge in the writing of this text. [have used principles of abstraction and simple models of computer systems and system programs to present the key issues in design of system programs. This way, the text is free of dependence on specific computers, programming languages, or operating systems. Pedagogical features Chapter introduction The chapter introduction motivates the reader by describing the objectives of the chapter and the topics covered in it.xvi_ Preface Figures and boxes Figures depict practical arrangements used to handle user computations and resources, stepwise operation of specific techniques, or comparisons of alternative techniques that project their strengths and weaknesses. Boxes are used to enclose key features of concepts or techniques being discussed. They also serve as overviews or summaries of specific topics. Examples Examples demonstrate the key issues concerning concepts and techniques being discussed. Examples are typeset in a different style to set them apart from the main body of the text, so a reader can skip an example if she does not want the flow of ideas to be interrupted, especially while reading a chapter for the first time. Algorithms Specific details of processing performed by system programs are presented in the form of algorithms. Algorithms are presented in an easy to understand pseudo-code form. Case studies Case studies emphasize practical issues, arrangements and trade-offs in the design and implementation of specific schemes. Case studies are organized as separate sections in individual chapters. Tests of concepts A set of objective and multiple choice questions are provided at the end of each chapter, so that the reader can test the grasp of concepts presented in the chapter. Exercises Exercises are included at the end of each chapter. These include numerical problems based on material covered in the text, as well as challenging conceptual questions which test understanding and also provide deeper insights. Chapter summary The summary included at the end of each chapter highlights the key topics covered in the chapter and their interrelationships. Instructor resources A detailed solutions manual is provided. Organization of the book The first chapter describes goals of system software and discusses the characteristics that distinguish system programs from other kinds of programs. It also introduces the notion of effective utilization of a computer; it is a combination of user convenience, programmer productivity, and efficient and secure operation of a computer that suits a user’s purposes. The rest of the chapter describes the broad spectrum of considerations that influence the design of system programs. These considerations range from user expectations such as high productivity during program development, ease of porting a program from ‘one computer to another, ability to compose new programs by using existing programs as components, ability of software to adapt to its usage environment, and secure and efficient operation of programs. The first chapter also develops two views of system software from the perspectives of ease of programming and efficient operation, respectively. The user-centric view is comprised of system programs that assist a user in fulfilling her computational needs. It includes system programs that transform a user program written in a programming language into a form that can be executed on a computer. The systemcentric view is comprised of programs that achieve effective utilization of a computer system by sharing its resources among user programs and ensuring non-interference during their execution. We call these two kinds of system programs language processors and operating system programs, respectively. The rest of the book is organized into two parts dealing with these two kinds of system programs, respectively.Preface xvii Language Software Processors Tools me | Assemblers Linkers ito Macro Pee Interpreters | | Compilers Chapters of Part I + Part I: Language processors A language processor is a system program that bridges the gap between how a user describes a computation in the form of a program, and how a computer executes a program. This gap is called the specification gap. The figure shows the interrelationship between chapters in this part, Chapter 2 describes how the specification gap influences ease of programming and execution efficiency of programs. It describes two classes of language processors called program generators and translators. \t then describes fundamentals of the language processing activity and the organization of language processors. Remaining chapters of this part discuss details of specific kinds of language processors. Chapter 3 discusses the design of an assembler, which is the translator of a low-level machine-specific language called the assembly language. Chapter 4 discusses the macro facility provided in assembly languages, which enables a programmer to define new operations and data structures of her own choice to simplify design and coding of programs. Chapter 5 discusses linkers and loaders, which are system programs that merge the code of many programs so that they can be executed together and make the merged code ready for execution by the computer. Chapter 6 describes the techniques of scanning and parsing that are used by a language processor to analyse a program written in a programming language. Chapters 7 and 8 discuss compilers and interpreters, which are two models of execution of programs written in programming languages. Finally, Chapter 9 discusses software tools, which are programs that suitably interface a program with other programs to simplify development of new applications. Many of these tools use elements of language processing to achieve their goals. + Part II: Operating systems The operating system controls operation of the computer and organizes execution of programs. Part II comprises five chapters. Chapter 10 describes the fundamentals of an operating system—how it controls operation of the computer, and how it organizes execution of programs, It contains an overview of those features of a computer's architecture that are relevant to the operating system and describes how an operating system functions. It then provides an overview of the concepts and techniques of operating systems used in the classical computing environments. Chapters 11-14 discuss concepts and techniques used in the four key functionalities of operating systems, namely, management of programs, management of memory, file systems, and security and protection in operating systems.xvili_ Preface Using this book Apart from an introduction to computing, this book does not assume the reader to possess any specific background. Hence it can be used by both students of systems programming and by working professionals. Dhananjay Dhamdhere April 2011CHAPTER 1 Introduction A modern computer has powerful capabilities such as a fast CPU, large memory, sophisticated input-output devices, and networking support; however, it has to be instructed through the machine language, which has strings of Os and Is as its instructions. A typical computer user does not wish to interact with the computer at this level. The system software is a collection of programs that bridge the gap between the level at which users wish to interact with the computer and the level at which the computer is capable of operating. It forms a software layer which acts as an intermediary between the user and the computer. It performs two functions: It translates the needs of the user into a form that the computer can understand so that the user’s program can actually get executed on the computer. However, the com- puter has more resources than needed by a program, so many of its resources would remain idle while it is servicing one program. To avoid this problem, the software layer gives the idle resources to some other programs and interleaves execution of all these programs on the computer. This way, the computer can provide service to ‘many users simultaneously. Each program in the system software is called a system program. System pro- grams perform various tasks such as editing a program, compiling it, and arranging for its execution. They also perform various tasks that a user is often unaware of, such as readying a program for execution by linking it with other programs and with functions from libraries, and protecting a program against interference from other programs and users, The term systems programming is used to describe the collec- tion of techniques used in the design of system programs. In this chapter we define the design goals of system programs and discuss the diverse functions performed by them. We then discuss the functions performed by different kinds of system programs. Details of their functioning and design are dis- cussed in subsequent chapters.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.Introduction 5 each computing environment, users desire a specific combination of convenience and efficient use. This is the notion of effective utilization of the computer system. For example, users in an interactive environment would favour convenience, ¢.g., fast re- sponse, to efficient use, whereas in a commercial data processing environment users would prefer efficiency to convenience because it would reduce the cost of comput ing. Hence an operating system simply chooses techniques that provide a matching combination of convenience and efficient use. We find a rich diversity in operating systems, and in system software in general, because effective utilization has a differ- ent flavour in each computing environment. We consider examples of this diversity in later sections of this chapter. Interference with a user's activities may take the form of illegal use or modifi- cation of a user’s programs or data, or denial of resources and services to a user. It could be caused by users of a computer system or by non-users. The system software must incorporate measures to prevent interference of all kinds and from all sources. We discuss important aspects of the three fundamental goals of system software in the following section. 1.2.1 User Convenience Table 1.1 lists many facets of user convenience. In the early days of computing, user convenience was synonymous with bare necessity—the mere ability to execute a program written in a higher level language was considered adequate. However, soon users were demanding better service, which in those days meant only fast response toa user command, ‘Table 1.1 Facets of user convenience Facet Examples Fulfillment of necessity Ability to execute programs, use the file system Good Service ‘Speedy response to computational requests User friendly interfaces Easy-to-use commands, Graphical user interface (GUI) New programming model Concurrent programming ‘Web-oriented features Means to set up web enabled servers Evolution ‘Add new features, use new computer technologies Other facets of user convenience evolved with the use of computers in new fields. Early operating systems had command-line interfaces, which required a user to type in a command and its parameters. Users needed substantial training to learn use of commands, which was acceptable because most users were scientists or computer professionals. However, simpler interfaces were needed to facilitate use of comput- ers by new classes of users. Hence graphical user interfaces (GUIs) were designed. These interfaces used icons on a screen to represent programs and files and inter- preted mouse clicks on the icons and associated menus as commands conceminga You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.Introduction 9 ‘compare two system programs to decide which of them is ‘better’. However, reality is more complex than this conception for several reasons. An axis may not be homogeneous, so its calibration would be subjective and contentious. For example, how do we place the six facets of user convenience shown in Table 1.1 along the convenience axis in a non-controversial manner? Should we consider the ability to evolve more important than having web-oriented features? Similarly, given that a computer has several kinds of resources such as the CPU, memory and input-output devices, how do you assess efficient use of resources? Is one kind of resource more or less important than another kind? We could resolve this difficulty by developing a resource utilization function in which each kind of resource is a parameter. But if we did that we would have to decide how much weightage to assign to each of the resource kinds, and so on. The second reason is that the nature of the computing environment in which a system program is used decides how it rates along each of the axes. For example, the computational needs of a user decide what facet of user convenience and efficiency are relevant. Consequently, a system program cannot be rated uniquely along the three axes. Due to these reasons, the question “Should system program A be preferred over system program B?” does not have a unique answer, In some situations A would be preferred while in some others B would be preferred. The purpase of studying system software is to know which of the two should be preferred in a specific situation. In the following sections we consider aspects which influence answers to this question. After a reader has gained a perspective for answering this question, she should try to answer the higher-level question “Under what situations should system program A be preferred over system program B?" 1.4.1. ‘The Program Development and Production Environments Ina program development environment, users are engaged in developing programs to meet their computational needs. A user makes a trial run of her program to discover the presence of bugs, modifies the program to fix the bugs and makes further trial runs and so on until no new bugs can be discovered. In a production environment, the users merely execute already developed programs on one or more sets of data each to produce useful results; none of the programs face any modification. A compiler and an interpreter are (wo system programs that can be used to execute programs in these environments. The compiler translates a program P written in a higher level programming language L into the machine language of a computer. Thus, it generates a machine language program that can be executed later. The inter- preter does not generate a machine language program; instead it analyzes program P and directly carries out the computation described in it. To understand comparative efficiency of a compiler and an interpreter, let us consider how they function: The compiler analyzes each statement in program P and generates a sequence of instructions in the machine language that would realize -meaning of the statement when executed, by computing values, making decisionsa You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.Introduction 13 this purpose as follows: When a client contacts the-seryer, the client and the server coordinate their activities to download a program in the bytecode form in the client’s computer, and the client initiates its interpretation by a Java virtual machine. This program obtains data from the server periodically. This scheme shifts most of the action to the client's side, thereby reducing the pressure on the server's resources. It is implemented simply by including a Java virtual machine in the client's web browser. 1.4.4 ‘Treating Programs as Components A new computational task to be accomplished may have known subtasks for which programs have been already developed. A program for performing the new task can be developed quickly and cheaply if the existing programs can be treated as its components. This requirement points to a facility that permits programs to be glued together to form larger software systems. A user interface is a prime example of such a facility because it permits ready programs—either system programs provided in the system software of a computer or those developed by users—to be invoked to accomplish larger tasks. A language designed for gluing together existing program components is called a scripting language. Note that the scripting language merely binds together those programs that actually perform the computational task. Hence efficiency of the scripting language is not important. The scripting languages provided in early mainframe computer systems were called job control languages (JCL). In such computer systems, a job consisted of a sequence of programs. The JCL allowed a programmer to indicate which programs constituted a job and implemented their execution in the specified sequence. The shell of Unix and graphical user interfaces (GUIs) of the Apple, Windows and Linux ‘operating systems are current-day descendants of the JCL. Example 1.1 contains an illustration of the Unix shell. A scripting language used in this manner is also called the command language of an operating system. Example 1.1 (Gluing of programs using the Unix shell) The following command in the Unix shell cat alpha | sort | uniq | wc -1 ‘glues together four programs to accomplish counting of unique names in file alpha. These programs are—cat, sort, uniq and we. ‘|’ is the symbol for a Unix pipe, which sends the output of one program as input to another program. Thus, the output of cat is given to sort as its input, the output of sort is given to unig and the output of uniq is given to ve. Program cat reads the names from file alpha and writes each ‘name into its standard output file. The output of cat is the input (o sort, so sort reads these names, sorts them in alphabetical order, and writes them into its output file. ‘unig removes duplicate names appearing in its input and outputs the unique names. ‘wc counts and reports the number of names in its input. (Note that ~1 is a parameter passed to the we program asking it to count the number of lines in its input because unig puts each name in a line by itself.)a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.Introduction 17 100 real elements. The compiler makes use of this type information and generates ap- propriate machine language instructions for accessing these variables. Some elements of array alpha may not be used during an execution of the program; however, the compiler would have allocated memory to them. Consequently, some of the allocated ‘memory may never be used. i, m: integer; ji real; m := 60; nm := 50; alpha [1 .. 100]: real; alpha [1 .. mJ; if m< 50 then j := m~- 10.5; if m< 50 then j := m~ 10.5; else j := m- 56; else j := m - 56; iiejt i := (integer) j + 6; alpha[i] 3 alpha[i] 3 end; end; A program in language L A program in language Ly Figure 1.4 Static and dynamic specification in programs In language L2, types and dimensions of variables are either specified dynamically of inferred from the context of their use. Variable j would be of type real if m is < 50 because value of the expression m~ 10.5 is assigned to it, but it would be of type integer if m 4 50. However, irrespective of j’s type i would be of type integer because of the specification “(integer)” appearing in the right-hand side of the assignment to i. The size of array alpha is determined by the value of m; presumably, this manner of specifying the size avoids wastage of memory due to unused urray elements. Because the type of a variable depends on the value assigned to it, a variable may have different types in different parts of a program. Such a program cannot be compiled. It would have to be interpreted, which would slow down its execution (see Section 1.4.1). Flexibility is the capability to broaden the choice in a specification or decision according to the needs of a user. In the world of programming languages, flexibility is provided through user defined data types. A user can define her own data type for use in a program, say type Tj, by specifying the following: © The values that a variable of type 7, may assume, ‘© The operations that can be performed on variables and values of type Ti. Now the set of types available in the program is comprised of the built-in types of the language, such as types real and integer in Figure 1.4, and the user defined data types such as type Tj. Adaptive software is one that adjusts its own features and behavior according to its environment. As we shall see shortly, it uses both dynamic features and flexibil- ity. The plug-and-play capability of an operating system is an example of adaptive behavior. It allows new devices to be connected and used during the operation of a computer. The operating system implements this capability as follows: The com- puter has a feature through which it alerts the operating system when a device isa You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.Introduction 21 Some of the functions performed by system programs included in the system-centric view are described in the following. Resource sharing is implemented through the following two techniques: resource Partitioning divides the resources among programs such that each program has some resources allocated for its exclusive use, whereas resource scheduling selects one of the many programs that wish to use a scarce resource and allocates the resource to it for some duration of time. Memory, storage deviccs such as disks, and input-output devices such as keyboards and screens are handled by means of resource partitioning, whereas the CPU is subject to scheduling. Scheduling of the CPU results in interleaved execution of programs. An operat- ing system uses a scheduling policy that provides an appropriate combination of user convenience and efficient use of resources in a computing environment. A multi- user application software needs to service many users’ requests simultaneously. An operating system provides multithreading support for this purpose. 1.6 OVERVIEW OF THE BOOK As discussed in earlier sections, the system software of a computer is a collection of system programs and systems programming is the collection of techniques used in the design of system programs. ‘A dominant theme in system programming is the trade-off between static and dy- namic actions. A static action is one that is performed before execution of a program is started. A dynamic action is performed during execution of a program. The action incurs an overhead during the program’s execution; however, it provides flexibility because it can use information that is determined during a program’s execution. We have seen instances of both averhead and flexibility of dynamic decisions when we discussed different kinds of system programs in Section 1.4, So system programming involves a careful analysis of the benefits of static and dynamic decision making and actions. It results in design of appropriate data structures for use in a program and schemes for use in language processors and operating systems. In Section 1.5, we discussed how system programs can be grouped into two views that we called the user-centric and system-centric views of system software. The user-centric view consists of language processors that are used for developing new programs, and system programs that assist in the use of existing programs. It also contains operating system programs that help in execution of user programs; how- ever, these programs and their functions are not as visible to the users. The system- centric view consists of programs that focus on efficient use of a computer system and non-interference with execution of programs and use of resources. It contains programs of the operating system that work harmoniously to achieve a common goal. Hence the special techniques used in their design are called operating system tech- niques. Accordingly, chapters of this book are organized into two parts. The first part discusses the system programming techniques used in language processors and ina You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.Introduction 23 this decision. Such software adjusts its own features and behavior according to its environment. For example, it can switch between the use of a compiler and an inter- preter depending on whether it finds that a program is being executed for production purposes or debugging purposes. Some system programs are used to provide benefits that are not restricted to specific computing environments. A virtual machine is implemented by a system program to provide easy portability of programs. Virtual machines are employed to provide a capability to download programs from the Internet and execute them on any computer, Scripting languages permit new programs to be composed from existing programs and also provide a method of quickly implementing certain kinds of computational tasks. System software can be viewed from two perspectives. The user-centric view fo- cuses on language processors and scripting languages, while the system-centric view focuses on programs that provide effective utilization of a computer, i.e., programs in the operating system. We study language processors and scripting languages in Part 1 of the book and operating systems in Part 2 of the book. TEST YOUR CONCEPTS: 1. Classify each of the following statements as true or false: (a) An abstract view hides unimportant details. (b) User convenience and efficient use of a computer system are completely inde- pendent of one another. (©) Efficient use of a computer system is of primary importance in a program development environment. (@) Use of a virtual machine enhances portability of software. (e) A scripting language can glue together existing programs. (f) User convenience is of primary importance in the embedded systems environ- ment, (g) User defined data types provide flexibility. (h) A just-in-time compiler uses adaptive techniques. (i) The system-centric view of system software contains language processors. BIBLIOGRAPHY ‘The XEN virtual machine product is described in Barham et al. (2003). Rosenblum and Garfinkel (2005) discuss trends in the design of virtual machine monitors. The May 2005 issue of IEEE Computer is a special issue on virtualization technologi Welch (2003), Schwartz et al. (2008), and Beazley (2009) are books devoted to the script- ing languages TeV/Tk, Perl, and Python, respectively. Aycock (2003) discusses just-in-time compilers. 1, Aycock, J. (2003): A brief history of just-in-time, ACM Computing Surveys, 97-113. 5 (2),24 Systems Programming 2. Barham, P., B. Dragovic, K, Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield (2003): “XEN and the art of virtualization,” ACM Symposium ‘on Operating System Principles, 164-177. Beazley, D. M. (2009): Python Essential Reference, fourth edition, Addison-Wesley Professional, Reading, MA. Rosenblum, M. and T. Garfinkel (2005): “Virtual machine monitors: current technol- ogy and future trends," IEEE Computer, 38 (5), 39-47. Schwartz, R.. T. Phoenix, and B. D. Foy (2008): Learning Perl, filth edition, O'Reilly Media, Sebastopol. Smith, J. B. and R. Nair (2005): The Architecture of Virtual Machines. IEEE Com- puter, 38 (5), 32-38. Welch, B. B. (2003): Practical Programming in Tet and Tk, Preni Hall, NJ.PART I LANGUAGE PROCESSORS Chapter 2: Overview of Language Processors Chapter 3: Assemblers Chapter 4: Macros and Macro Preprocessors Chapter 5: Linkers and Loaders Chapter 6: Scanning and Parsing Chapter? : Compilers Chapter 8 : Interpreters Chapter9 : Software ToolsCHAPTER 2 Overview of Language Processors A user would like an arrangement in which she could describe a computation in some convenient manner and the system software of the computer system would implement the computation using the computer. A language processor is a system program that bridges the gap between how a user describes a computation—we call it a specification of the computation— and how a computer executes a program. The ease of specification depends on the language in which the specification is written. A problem oriented programming lan- ‘guage lets a user specify a computation using data and operations that are meaningful in the application area of the computation. A procedure oriented programming lan- guage provides some standard methods of creating data and performing operations and lets the user describe the intended computation by using them. Of the two, use of a problem oriented language provides more user convenience. ‘Two kinds of language processors are used to implement a user’s computation. A program generator converts the specification written by the user into a program in a procedure oriented language, whereas a compiler or interpreter helps in implement- ing a program written in a programming language. A compiler translates a program into the target language, which is either the machine language of a computer or a language that is close to it. An interpreter analyzes a program and itself performs the computation described in it with the help of the computer. The compiler and interpreter suit the needs of different kinds of programming environments. We discuss the fundamentals of a language processor in this chapter—how it analyzes a program input to it and how it synthesizes a program in the target lan- guage. It uses a data structure called a symbol table to store attributes of symbols used in a program. The design of the symbol table is a crucial decision because a language processor accesses it a large number of times, We describe the principle of time-space trade-off used in its design and discuss many symbol table organizations.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.30_Systems Programming of converting specifications across any pair of domains is simpler, so it can be performed more reliably. A system program can be used to perform conversion of specifications across domains. Now the conversion is faster, cheaper and more reliable because it is not performed manually. Of course, correctness of the system program that performs the conversion would have to be ensured, but that is a one-time effort. Use of a programming language combines both these methods as follows: The programming language domain is introduced as an intermediate domain, so the semantic gap between the application and execution domains is split into two smaller semantic gaps—the specification gap and the execution gap (see Figure 2.2), Only the specification gap has to be bridged by the software designer, whereas the execu- tion gap is bridged by the language processor such as a compiler or an interpreter. ‘The language processor also provides a capability to detect and indicate errors in its input, which helps in improving reliability of software. ‘We assume that each domain has specification language. A specification written in a specification language is a program in the specification language. The specifi- cation language of the programming language domain is the programming language itself, The specification language of the execution domain is the machine language of the computer system. We use the terms specification gap and execution gap as follows: * Specification gap is the semantic gap between two specifications of the same task. * Execution gap is the semantic gap between the semantics of programs that perform the same task but are written in different programming languages. We restrict use of the term ‘execution gap’ to situations where one of the two specification languages is closer to the machine language of a computer system. In other situations, the term ‘specification gap’ is more appropriate. 2.1.1 Kinds of Language Processors Definition 2.1 (Language processor) A language processor is a software which bridges a specification or execution gap. We use the term language processing to describe the activity performed by a language processor. As mentioned earlier, a semantic gap is bridged by converting a specification in one domain into a specification in another domain, During this conversion, the language processor points out errors in the input specification and aborts the conversion if errors are present. This capability of a language processor, which we call the diagnostic capability, contributes to reliability of a program written in a programming language by ensuring that it would reach execution only if it is free of specification errors, (We shall discuss the diagnostic capability of language processors in Chapters 6 and 7.)a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.34 Systems Programming ‘© Program generation activity: This activity generates a program from its spec ification. The language in which the specification of the program is written is close to the specification language of an application domain and the target lan- ‘guage is typically a procedure oriented programming language. Thus, program generation bridges the specification gap. ‘© Program execution activity: This activity aims at bridging the execution gap by organizing execution of a program written in a programming language on a computer system. The programming language could be a procedure oriented language or a problem oriented language. 2.2.1 Program Generation Figure 2.6(a) depicts the program generation activity. The program generator is a system program which accepts the specification of a program in some specification language and generates a program in the target language that fulfills the specifica- tion. Use of the, program generator introduces a new domain called the program generator domain between the application and programming language domains (see Figure 2.6(b)). The specification gap faced by the designer of an application program is now the gap between the application domain and the program generator domain. This gap is smaller than the gap between the application domain and the target pro- gramming language domain; the gap is insignificant if the specification language is a problem oriented language. The execution gap between the target language domain and the execution domain is bridged by a compiler or interpreter for the programming language. Errors Program Program |__ Popes w specification generator a Specification ° 0 ( Q 0 0 plication, Program TargetPL Execution omain” generator domain domain domain Figure 2.6 Program generation: (a) Schematic, (b) the program generator domain If the program generator domain is close to the application domain, the speci- fication gap is small. Consequently, the task of writing the specification is simple, which increases the reliability of the generated program. The harder task of bridginga You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.38_ Systems Programming © Ifa source program is modified, the modified program must be translated be- fore it can be executed. We call it retranslation following a modification. sme 8 program Data inH+ + CH |_ J Mie language inker |] Loader stasor 42] C compiler) Asembler meee ; : i ¥ c Assembly M/c language Results program program = program Figure 2.10 A practical arrangement of language processors, Practical arrangements for program execution differ from that depicted in Fig- ure 2.9 for providing higher effectiveness and flexibility. Figure 2.10 shows a prac- tical arrangement for executing C++ programs which uses a number of language processors. The output of each of the language processors can be saved on disk and used repeatedly. Important features of this arrangement are described below. © Preprocessors: One or more preprocessors may be used along with a trans- lator for a programming language to provide a superset of the programming language’s features. Recall from Example 2.1 that use of the C++ preproces- sor along with a C compiler enables use of the C++ language without having to develop a C+ compiler. In Chapter 5 we shall discuss use of a macro preprocessor for providing a superset of the features of an assembly language. Using a sequence of translators: Translation from a programming language to a machine language may be achieved by using two or more translators. The translator used in the first step produces a target program that is not in the machine language. This target program is input to another translator, and so on, until we obtain a target program that is in the machine language. Each of the translators is less complex than a translator for the programming language that would have directly produced a machine language program. This arrangement also has another benefit that is described later. © Linking and loading: target program obtained by translating a program writ- ten in a programming language requires the help of some other programs dur- ing its execution, e.g., programs that perform input-output or standard mathe- matical functions. The linker is a system program that puts all these programs together so that they can execute meaningfully. The loader is a program that loads a ready-to-run program from a file into the computer's memory for exe- cution. If one of the intermediate target programs is in a standard programming lan- guage, say, language PL, the arrangement using a sequence of translators can be