CCD ARRAYS, CAMERAS,

and DISPLAYS
Second edition
TABLE of CONTENTS
1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1. Solid State Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2. Imaging System Applications . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1. General Imagery . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.2. Machine Vision . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.3. Scientific Applications . . . . . . . . . . . . . . . . . . . . . . 8
1.2.4. Military Applications . . . . . . . . . . . . . . . . . . . . . . 8
1. 3. Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4. Image Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.5. Pixels, Datels, Disels, and Resets . . . . . . . . . . . . . . . . . . . 14
1. 6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2. RADIOMETRY and PHOTOMETRY . . . . . . . . . . . . . . . . . . . . 18

2.1. Radiative Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2. Planck's Blackbody Law . . . . . . . . . . . . . . . . . . . . . . . . 21
2. 3. Photometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1. Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.2. Typical Illumination Levels . . . . . . . . . . . . . . . . . . 27
2.4. Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.1. Calibration Sources . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.2. Real Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5. Camera Formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.6. Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.7. Normalization Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2. 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3. SOLID STATE ARRAYS . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3. 1. Photodetection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.1.1. Photogate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3. 1. 2. Photodiode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2. CCD Array Operation . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.3. CCD Array Architecture . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3 .1. Linear Arruys . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3.2. Full Frame Arrays . . . . . . . . . . . . . . . . . . . . . . . . 59
3.3.3. Frame Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.3.4. Interline Transfer . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.3.5. Progressive Scan . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3.6. Time Delay and Integration . . . . . . . . . . . . . . . . . . 70
3.4. Charge Transfer Efficiency . . . . . . . . . . . . . . . . . . . . . . . 74

xiii
xiv CCD ARRAYS, CAMERAS, and DISPLAYS

3.5. Charge Conversion (Output Structure) . . . . . . . . . . . . . . . . 78

3.6. Dark Current . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3. 7. Dark Pixels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.8. Antibloom Drain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.9. CID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.10. CMOS . . . . . . . . . . . . . . . . . _ . . . . . . . . . . . . . . . . . 91
3 .11. Physical Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3 .11.1. Micro lenses . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.11.2. Color Filter Arrays . . . . . . . . . . . . . . . . . . . . . . . 94
3 .11.3. Number of Detectors . . . . . . . . . . . . . . . . . . . . . . 97
3.11.4. Video Chip Size . . . . . . . . . . . . . . . . . . . . . . . . . 98
3.12. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

4. ARRAY PERFORMANCE . . . . . . . . . . . . . . . . . . . . . . . . . 102

4.1. Signal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4. 1.1. Spectral Response . . . . . . . . . . . . . . . . . . . . . . . 104
4.1.2. Responsivity . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.1.3. Minimum Signal . . . . . . . . . . . . . . . . . . . . . . . . 119
4 .1.4. Maximum Signal . . . . . . . . . . . . . . . . . . . . . . . . 120
4.1.5. Dynamic Range . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.2. Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
4.2.1. Shot Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.2.2. Reset Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.2.3. On-chip Amplifier Noise . . . . . . . . . . . . . . . . . . . 128
4.2.4. Off-chip amplifier Noise . . . . . . . . . . . . . . . . . . . 130
4.2.5. Quantization Noise . . . . . . . . . . . . . . . . . . . . . . . 130
4.2.6. Pattern Noise . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.2.7. Photon Transfer . . . . . . . . . . . . . . . . . . . . . . . . 133
4.3. Array Signal-to-Noise Ratio . . . . . . . . . . . . . . . . . . . . . 139
4.4. Correlated Double Sampling . . . . . . . . . . . . . . . . . . . . . 141
4.5. Frame Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.6. Defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.7. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

5. CAMERAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5 .1. Camera Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
5.2. Video Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.2.1. Video Timing . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5.2.2. Broadcast/Non-broadcast Design . . . . . . . . . . . . . . 155
5.2.3. Component/Composite Signals . . . . . . . . . . . . . . . . 156
5.2.4. IRE Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
5.2.5. Digital Television . . . . . . . . . . . . . . . . . . . . . . . 159
 TABLE of CONTENTS xv

5.2.6. HDTV/ATS 160

5.3. Consumer/Broadcast Cameras . . . . . . . . . . . . . . . . . . . . 163
5.3.1. The Knee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
5.3.2. Color Correction . . . . . . . . . . . . . . . . . . . . . . . . 165
5.3.3. Gamma Compensation . . . . . . . . . . . . . . . . . . . . . 169
5.3.4. Aperture Correction . . . . . . . . . . . . . . . . . . . . . . 172
5.4. Industrial/Scientific Cameras . . . . . . . . . . . . . . . . . . . . . 172
5.4.1. Analog-to-Digital Converters . . . . . . . . . . . . . . . . . 174
5.4.2. Intensified Solid State Cameras . . . . . . . . . . . . . . . 176
5.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

6. CAMERA PERFORMANCE . . . . . . . . . . . . . . . . . . . . . . . . 182

6.1. Nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
6.2. Camera Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
6.2.1. Camera SNR . . . . . . . . . . . . . . . . . . . . . . . . . . 186
6.2.2. Camera Dynamic Range . . . . . . . . . . . . . . . . . . . 187
6.2.3. Maximum Signal . . . . . . . . . . . . . . . . . . . . . . . . 189
6.2.4. Minimum Signal . . . . . . . . . . . . . . . . . . . . . . . . 189
6.3. Intensified CCD Camera . . . . . . . . . . . . . . . . . . . . . . . 195
6.3.1. ICCD Signal . . . . . . . . . . . . . . . . . . . . . . . . . . 196
6.3.2. ICCD Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
6.3.3. ICCD SNR . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
6 .4. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

7. CRT-BASED DISPLAYS . . . . . . . . . . . . . . . . . . . . . . . . . . 201

7 .1. The Observer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
7.2. CRT Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
7 .2.1. Monochrome Displays . . . . . . . . . . . . . . . . . . . . . 207
7 .2.2. Color Displays . . . . . . . . . . . . . . . . . . . . . . . . . 207
7.2.3. HDTV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
7.3. Spot Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
7 .4. Disels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
7.5. Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
7 .5.1. Vertical Resolution . . . . . . . . . . . . . . . . . . . . . . . 219
7.5.2. Theoretical Horizontal Resolution . . . . . . . . . . . . . . 219
7.5.3. TV Limiting Resolution . . . . . . . . . . . . . . . . . . . . 220
7.5.4. MTF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
7.6. Addressability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.7. Shades of Gray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
7 .8. Character Recognition . . . . . . . . . . . . . . . . . . . . . . . . . 227
7. 9. Contrast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
7. 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
xvi CCD ARRAYS, CAMERAS, and DISPLAJ:S

8. SAMPLING THEORY . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

8.1. Sampling Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
8.2. Aliasing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
8.3. The Detector as a Sampler . . . . . . . . . . . . . . . . . . . . . . 239
8.3.1. Detector MTF . . . . . . . . . . . . . . . . . . . . . . . . . 239
8.3.2. Detector Array Nyquist Frequency . . . . . . . . . . . . . 242
8.3.3. Image Distortion . . . . . . . . . . . . . . . . . . . . . . . . 245
8.4. Aliasing in Frame Grabbers and Displays . . . . . . . . . . . . . 249
8.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

9. LINEAR SYSTEM THEORY . . . . . . . . 252

9 .1. Linear System Theory . . . . . . . . . . . . . . . . . . . . . . . . . 253
9 .1. 1. Time-varying Signals . . . . . . . . . . . . . . . . . . . . . 254
9.1.2. Spatially Varying Signals . . . . . . . . . . . . . . . . . . . 256
9.2. Electronic Imaging System . . . . . . . . . . . . . . . . . . . . . . 257
9. 3. MTF and PTF Interpretation . . . . . . . . . . . . . . . . . . . . . 259
9.4. Superposition Applied to Optical Systems . . . . . . . . . . . . . 261
9.5. Contrast Transfer Function . . . . . . . . . . . . . . . . . . . . . . 263
9. 6. References 266

10. SYSTEM MTF 267

10 .1. Frequency Domains . . . . . . . . . . . . . . . . . . . . . . . . . . 269
10.2. Optics MTF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
10.3. Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
10.3.1. Diffusion MTF . . . . . . . . . . . . . . . . . . . . . . . . 276
10.3.2. Charge Transfer Efficiency . . . . . . . . . . . . . . . . . 279
10.3.3. TDI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
10.3 .4. Optical Anti-alias Filter . . . . . . . . . . . . . . . . . . . 283
10.4. Optics-detector Subsystem . . . . . . . . . . . . . . . . . . . . . . 286
10.5. Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
10.5.1. Linear Motion . . . . . . . . . . . . . . . . . . . . . . . . . 292
10.5.2. Random Motion (jitter) . . . . . . . . . . . . . . . . . . . 294
10.6. Electronic Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
10.6.1. Anti-alias Analog Filter . . . . . . . . . . . . . . . . . . . 295
10.6.2. Digital Filters . . . . . . . . . . . . . . . . . . . . . . . . . 296
10.6.3. Sample-and-hold . . . . . . . . . . . . . . . . . . . . . . . 299
10.6.4. Post-reconstruction Filter . . . . . . . . . . . . . . . . . . 300
10.6.5. Boost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
10.7. CRT-based Display . . . . . . . . . . . . . . . . . . . . . . . . . . 303
10.8. Human Visual Response . . . . . . . . . . . . . . . . . . . . . . . 306
10. 9. Intensified CCD . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
10.10. Sampling Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
 TABLE of CONTENTS xvii

100110 References 0 0 0 0 o o 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 312

11. IMAGE QUALITY 0 0 0 0 0 o 0 0 0 0 0 0 0 o 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 315

11010 Resolution Metrics 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 317
11020 Optical Resolution Metrics 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 320
11.30 Detector Resolution 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 321
11.40 Electrical Resolution Metrics 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 323
1105 0 MTF-based Resolution 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 324
11.5 ol. Limiting Resolution 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 324
11.5020 MTF/TM Resolution 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 325
11.5030 Shade's Equivalent Resolution 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 325
11060 System Resolution Examples 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 328
11.70 MTFA o 0 o o o o o 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 331
11080 Subjective Quality Factor 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 332
11.9 0 Square-root Integral 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 334
110100 References 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 336

120 MINIMUM RESOLVABLE CONTRAST 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 338

12ol. The Observer 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 339
1201010 Perceived Signal-to-noise Ratio 0 0 0 0 0 0 0 0 0 0 0 0 0 0 340
1201.20 Perceived Resolution 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 344
1201.30 The HVS "Filter" o 0 o o o o o 0 o o o o o 0 0 0 0 0 0 0 0 0 0 345
12o2o Three-dimensional Noise Model 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 346
12030 The MRC Model 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 o 0 351
12.40 SNRth and te o o 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 o o 355
12050 Two-dimensional MRC 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 358
12060 Range Predictions 0 0 o 0 0 0 o 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 359
120601. Contrast Transmittance 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 359
1206020 Johnson Criteria 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 362
12o6o3o Target Transfer Probability Function 0 0 0 0 0 0 0 0 0 0 0 363
12o6.4o Range Prediction Methodology 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 365
1206050 Sampling Effects 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 368
12070 References 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 369

APPENDIX o o o 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 371
Aol. Effective Focal Length 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 371
A020 f-Number 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 371
Ao3o Reference 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 372

INDEX 0 0 0 0 0 o 0 o o o 0 0 0 0 0 0 o 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 373
SYMBOL LIST

SYMBOL DEFINITION UNITS

spectrally averaged absorption coefficient lim

spectral absorption coefficient lim
dark current factor numeric
target displacement in TDI systems mm
display aspect ratio numeric

eye filter 1 llelectrons 2

eye filter m l/electrons2

gamma numeric

I!.V TDI velocity error mmls

I!. f. wavelength interval ~tm
l!.p target-background reflectance difference numeric
/:if, noise equivalent bandwidth Hz
/:l.nM!N!MUM minimum number of photoelectrons numeric
1!.1\. target-background photoelectron difference numeric
/:it time separation between two pulses

charge transfer efficiency numeric

quantum efficiency numeric

17CATIIODE intensifier photocathode quantum efficiency numeric
17cco CCD quantum efficiency numeric
17PHOTOCATIIODE intensifier photocathode quantum efficiency numeric
17scREEN intensifier screen quantum efficiency numeric

wavelength
average wavelength
maximum wavelength
minimum wavelength
selected wavelength
wavelength of peak response

p spectrally averaged reflectance numeric

p(f.) spectral reflectance numeric
PB background reflectance numeric
PT target reflectance numeric

atmospheric absorption lim

component of three-dimensional noise model rms
component of three-dimensional noise model rms
image intensifier tube 1/e spot size mm

xviii
 SYMBOL LIST xix
u,s display 1/e spot size in image space mm
UR rms value of random motion rms mm
UsP<lT display spot 1/e intensity size mm
UT component of three-dimensional noise model rms
Urn component of three-dimensional noise model rms
UTVH component of three-dimensional noise model rms
Uv component of three-dimensional noise model rms

opCATIIODE flux incident onto to intensifier photocathode w

.Pv luminous flux lumens

aL distance target has moved on detector mm

AcArnooE area of photocathode m2
Ao detector area m2
A, area of source in image plane m2
Ao lens area m2
A PIXEL projected CCD pixel on intensifier photocathode m2
As source area m2

c speed oflight, c = 3 x 10' mls

c, first radiation constant c, = 3.7418 x 108 W-p.m4 /m2
c2 second radiation constant c, = 1.4388 x 104 p.m-K
c, third radiation constant c, = 1. 88365 x 1027 photons-p.m3/ s-m2
c sense node capacitance farad
Co target's inherent contrast numeric
CR target's contrast at entrance aperture numeric
CTE charge transfer efficiency numeric

d detector width mm
dec detector pitch mm
dccH horizontal detector pitch mm
dccv vertical detector pitch mm
dH horizontal detector size mm
do selected target dimension mm
dT target detail mm
dv vertical detector size mm
D observer to display distance m
Do optical diameter mm
DASH horizontal detector angular subtense mrad
DASv vertical detector angular subtense mrad

Ea detector band gap eV

~ impurity band gap eV

f20 two-dimensional spatial frequency cycles/mrad

fBOOST boost frequency Hz
f, cutoff frequency of an ideal circuit Hz
fcLOCK pixel clock rate Hz
foe detector cutoff frequency cycles/mm
f,3dB one-half power frequency Hz
f, electrical frequency Hz
xx CCD ARRAYS, CAMERAS, and DISPLAYS

fEYE spatial frequency at eye cycles/deg

f, image spatial frequency cycles/mrn
fN Nyquist frequency cycles/mrn
f. selected spatial frequency cycles/mrn
foe optics cutoff frequency cycles/mrn
fPEAK peak frequency of eye MTF cycles/deg
fRASTER raster frequency cycles/mrn
fs sampling frequency cycles/mrn
fv electrical frequency in the video domain Hz
fv3dB one-half power frequency (video domain) Hz
fvN Nyquist frequency in the video domain Hz
fvs sampling frequency in the video domain Hz
f, horizontal spatial frequency cycles/mrn
fy vertical spatial frequency cycles/mrn
fl focal length m
F f-number numeric
FMAX maximum frame rate Hz
FR frame rate Hz

G on-chip amplifier gain numeric

G, off-chip amplifier gain numeric
GMCP microchannel gain numeric

h Planck's constant, h = 6.626 x 10'34 J-s

he target critical dimension m
H target height m
HMONITOR monitor height m
HFOV horizontal field-of-view mrad

dark current density A/cm2

k Boltzmann's constant, k = 1.38 X 10·23 J/K

k, a constant numeric
KH(f) horizontal summary noise factor
KM luminous efficacy, KM = 683 (photopic) lumens/W
kMcP microchannel excess noise numeric
Kv(O vertical summary noise factor

background luminance lumen/m2-sr

Depletion length l-Im
Diffusion length l-Im
spectral radiant sterance W/(m2-1-1m-sr)
spectral photon sterance photons/(s-m2-1-1m-sr)
target luminance lumen/m2-sr

index numeric
spectral radiant exitance W/(m2-1-1m)
optical magnification numeric
spectral power W/1-1m
spectral photon exitance photons/(s-m2-1-1m)
 SYMBOL LIST xxi
n index numeric
IlcATIIODE photons incident onto intensifier photocathode numeric
Do ARK number of dark electrons numeric
DDETECTOR number of photons incident onto detector numeric
n. number of electrons numeric
OEYE number of photons incident onto eye numeric
DIMAGE number of photons incident onto image plane numeric
llLENS number of photons incident onto lens numeric
OMCP number of photons incident on microchannel plate numeric
n,. number of photoelectrons numeric
llpe-8 background photoelectrons numeric
U,e-T target photoelectrons numeric
llPHOTON-CCD number on photons incident onto CCD numeric
DREAD pixels between active array and sense node numeric
llsCREEN photons incident onto intensifier screen numeric
DwiNDOW number of photons incident onto intensifier numeric
N index numeric
Nso Johnson's 50% probability cycles
NoETECTORs total number of detectors numeric
NH number of horizontal detectors numeric
NUNE number of raster lines numeric
NT equivalent number of cycles on target cycles
NTD! number of TDI elements numeric
NTRANS number of charge transfers numeric
NTV display resolution TVLIPH
Nv number of vertical detectors numeric
NwELL charge well capacity numeric

p display line spacing or pixel spacing rnrnlline

PASH horizontal pixel angular subtense mrad
PASv vertical pixel angular subtense mrad

q electron charge, q = 1.6 X 10" 19 coul

R range to target m
R, distance from lens to source m
R2 distance from lens to detector m
RAVE spectral averaged responsivity V/(JL-l-cm- 2) or DN/(JL-l-cm2)
R. average spectral responsivity A/W
R.(t-) spectral responsivity A/W
Rp peak responsivity A/W
RPHOTOMETRIC responsivity V/lux
Rq spectrally averaged quantum efficiency numeric
R.(t-) spectral quantum efficiency numeric
R. raster resolution lines/mm
RTVL horizontal display resolution TVLIPH
Rv responsivity A/lumen
RvERTICAL vertical display resolution lines

s display spot size (FWHM intensity) mm

xxii CCDARRAYS, CAMERAS, and DISPLAYS

!ARRAY time to clock out the full array

lci.DCK time between pixels
t. eye integration time
~-UNE time to read one pixel line
tiNT integration time
tUNE video active line time s
T absolute temperature Kelvin
T/# T-number numeric
TASPECT target aspect ratio numeric
TATM spectrally averaged atmospheric transmittance numeric
TATM(A) spectral atmospheric transmittance numeric
Tro fiber optic bundle transmittance numeric
Tmum absolute temperature of illuminating source K
TJR-FILTER spectrally averaged transmittance of IR filter numeric
Tomes spectrally averaged optical transmittance numeric
Tomes(>-) spectral optical transmittance numeric
TRELAY LENS relay lens transmittance numeric
TwiNDOW transmittance of intensifier window numeric

ud horizontal spatial frequency (display space) cycles/mm

U; horizontal spatial frequency (image space) cycles/mm
Uic horizontal optical cutoff (image space) cycles/mm
uiD horizontal detector cutoff (image space) cycles/mm
uiN horizontal Nyquist frequency (image space) cycles/mm
Uob horizontal spatial frequency (object space) cycles/mm
u photoresponse nonuniformity numeric

vd vertical spatial frequency (display space) cycles/mm

V; vertical spatial frequency (image space) cycles/mm
Vio vertical detector cutoff (image space) cycles/mm
viN vertical Nyquist frequency (image space) cycles/mm
V(>.) photopic or scotopic eye response numeric
VeAMERA camera output voltage v
VGRID voltage on CRT grid v
VLSB voltage of the least significant bit v
VMAX maximum signal numeric
VMIN minimum signal numeric
VNOISE noise voltage after on-chip amplifier V rms
VouT voltage after on-chip amplifier v
VR viewing ratio numeric
VRESET reset voltage after on-chip amplifier v
VseENE video voltage before gamma corrector v
VsJGNAL signal voltage after on-chip amplifier v
VvmEO video voltage after gamma corrector v
VFOV vertical field-of-view mrad

w target width m
WMONITOR monitor width m

X horizontal distance m
y vertical distance m
 SYMBOL LIST XX111

(nt) noise source 1 rrns electrons

(nAoc) quantization noise rrns electrons
(ficcD-DARK) dark current CCD noise in an ICCD rrns electrons
(ficco-PHOTON) noise before CCD in an ICCD rrns electrons
(ncco) CCD noise in an ICCD electrons rrns
(noARK) dark current shot noise rrns electrons
(nF!DOR) noise floor rrns electrons
(nFPN) fixed pattern noise rrns electrons
(nMcP) microchannel noise rrns electrons
(n,) noise source m rrns electrons
(fioFF-CHIP) off-chip amplifier noise rrns electrons
(fioN-CHIP) on-chip amplifier noise rrns electrons
(nPATI'ERN) pattern noise rrns electrons
(nPC-DARK) intensifier dark current shot noise rrns electrons
(nPC-SHOT) intensifier photon shot noise rrns electrons
(fip,) photon shot noise rrns electrons
(nPRNU) photoresponse nonuniforrnity noise rrns electrons
(nRESET) reset noise rrns electrons
(nsCREEN) intensifier screen noise rrns electrons
(nsHoT) shot noise rrns electrons
(nsvs) system noise rrns electrons
(VN) noise voltage rrns V
1
INTRODUCTION
Charge-coupled devices (CCDs) were invented by Boyle and Smith 1.2 in
1970. Since then, considerable literature3-7 has been written on CCD physics,
fabrication, and operation. However, the array does not create an image by
itself. It requires an optical system to image the scene onto the array's
photosensitive area. The array requires a bias and clock signals. Its output is a
series of analog pulses that represent the scene intensity at a series of discrete
locations.

Devices may be described functionally according to their architecture (frame

transfer, interline transfer, etc.) or by application. Certain architectures lend
themselves to specific applications. For example, astronomical cameras typically
use full frame arrays whereas consumer video systems use interline transfer
devices.

The heart of the solid state camera is the solid state array. It provides the
conversion of light intensity into measurable voltage signals. With appropriate
timing signals, the temporal voltage signal represents spatial light intensities.
When the array output is amplified and formatted into a standard video format,
a solid state camera is created. Because CCDs were the first solid state
detectors, cameras are popularly called CCD cameras even though they may
contain charge injection devices (CIDs) or complementary metal-oxide-
semiconductors (CMOS) as detectors. These are solid state cameras.

The array specifications, while the first place to start an analysis, are only
part of the overall system performance. The system image quality depends on
all the components. Array specifications, capabilities, and limitations are the
basis for the camera specifications. Camera manufacturers cannot change these.
A well-designed camera will not introduce additional noise nor adversely affect
image quality provided by the array.

A camera is of no use by itself. Its value is only known when an image is

evaluated. The camera output may be directly displayed on a monitor, stored on
video tape or disk for later viewing, or processed by a computer. The computer
may be part of a machine vision system, be used to enhance the imagery, or be
used to create hard copies of the imagery. If interpretation of image quality is
performed by an observer, then the observer becomes a critical component of
2 CCD ARRAYS, CAMERAS, and DISPLAYS

the imaging system. Consideration of human visual system attributes should

probably be the starting point of the camera design. But some machine vision
systems may not create a "user friendly" image. These systems are designed
from a traditional approach: resolution, high signal-to-noise ratio, and ease of
operation.

Effective camera design requires an orderly integration of diverse

technologies and languages associated with radiation physics, optics, solid state
sensors, electronic circuitry, and image processing algorithms. Each field is
complex and is a separate discipline.

Electro-optical imaging system analysis is a mathematical construct that

provides an optimum design through appropriate tradeoff analyses. A
comprehensive model includes the target, background, the properties of the
intervening atmosphere, the optical system, detector, electronics, display, and
the human interpretation of the displayed information. While any of these
components can be studied in detail separate! y, the electro-optical imaging
system cannot. Only complete end-to-end analysis (scene-to-observer
interpretation) permits system optimization.

1.1. SOLID STATE DETECTORS

CCD refers to a semiconductor architecture in which charge is read out of

storage areas. The CCD architecture has three basic functions: (a) charge
collection, (b) charge transfer, and (c) the conversion of charge into a
measurable voltage. The basic building block of the CCD is the metal-oxide-
semiconductor (MOS) capacitor. The capacitor is called a gate. By manipulating
the gate voltages, charge can be either stored or transferred. Charge generation
in most devices occurs under a MOS capacitor (also called a photogate). For
some devices (notably interline transfer devices) photodiodes create the charge.
After charge generation, the transfer occurs in the MOS capacitors for all
devices.

Because most sensors operating in the visible region use a CCD type
architecture to read the signal, they are popularly called CCD cameras. For
these devices, charge generation is often considered as the initial function of the
CCD. More explicitly, these cameras should be called solid state cameras with
a CCD readout.
 INTRODUCTION 3

CCDs and detectors can be integrated either monolithically or as hybrids.

Monolithic arrays combine the detector and CCD structure on a single chip. The
most common detectors are sensitive in the visible region of the spectrum. They
use silicon photogates or photodiodes and are monolithic devices. CCDs have
been successfully used for infrared detectors such as Schottky barrier devices
that are sensitive to 1.2 p.m to 5 p.m radiation.

Hybrid arrays avoid some pitfalls associated with growing different materials
on a single chip and provide a convenient bridge between well-developed but
otherwise incompatible technologies. HgCdTe (sensitive to 8 to 12 p.m radiation)
is bump bonded to a CCD readout using indium as the contact and, as such, is
a hybrid array.

With charge-injection devices (CID), the pixels consist of two MOS

capacitors whose gates are separately connected to rows and columns. Usually
the column capacitors are used to integrate charge while the row capacitors
sense the charge after integration. With the CID architecture, each pixel is
addressable, i.e., it is a matrix-addressable device.

CID readout is accomplished by transferring the integrated charge from the

column capacitors to the row capacitors. After this nondestructive signal
readout, the charge moves back to the columns for more integration or is
injected (discarded) back into the silicon substrate. By suspending charge
injection, the user initiates "multiple frame integration" (time lapse exposure)
and can view the image on a display as the optimum exposure develops.
Integration may proceed for a few milliseconds up to several hours. With
individual capacitors on each sensing pixel, blooming is not transported so
overloads cannot propagate.

Recently, active pixels have been introduced. These devices are fabricated
with complementary-metal-oxide-semiconductor (CMOS) technology. The
advantage is that one or more active transistors can be integrated into the pixel.
As such, they become fully addressable (can read selected pixels) and can
perform on-chip image processing.

1.2. IMAGING SYSTEM APPLICATIONS

There are four broad applications: general imagery (which includes both
professional television broadcast and consumer camcorder systems), machine
vision, scientific, and military. Trying to appeal to all four applications,
manufacturers use words such as low noise, high frame rate, high resolution,
4 CCD ARRAYS, CAMERAS, and DISPLAYS

reduced aliasing, and high sensitivity. These words are simply adjectives with
no specific meaning. They only become meaningful when compared (i.e.,
camera A has low noise compared to camera B).

Table 1-1 lists several design categories. While the requirements vary by
category, a camera may be used for multiple applications. For example, a
consumer video camera often is adequate for many scientific experiments. A
specific device may not have all the features listed or may have additional
features not listed. The separation between professional broadcast, consumer
video, machine vision, scientific, and military devices becomes fuzzy as
technology advances.

Color cameras are used for professional television, camcorder, and film
replacement systems. With machine vision systems, color cameras are used to
verify the color consistency of objects such as printed labels or paint mixture
colors. While color may not be the primary concern, it may be necessary for
image analysis when color is the only information to distinguish boundaries.

While consumers demand color camera systems, this is not true for other
applications. Depending on the application, monochrome (black and white)
cameras may be adequate. A monochrome camera has higher sensitivity and
therefore is the camera of choice in low light level conditions.

Image enhancement applies to those systems that have a human observer and
helps the observer extract data. Some images belong to a small precious data set
(e.g., remote sensing imagery from interplanetary probes). The images must be
processed repeatedly to extract every piece of information. Some images are part
of a data stream that are examined once (e.g., real-time video) and others have
become popular and are used routinely as standards. These include the three-bar
or four-bar test patterns, Lena, and the African mandrill.

The camera cannot perfectly reproduce the scene. The array spatially
samples the image, and noise is injected b:y the array electronics. Spatial
sampling creates ambiguity in target edge location and produces moire patterns
when viewing periodic targets. While this is a concern to scientific and military
applications, it typically is of little consequence to the average professional
television broadcast and consumer markets.
 Table 1-1
DESIGN GOALS
DESIGN I
GENERAL IMAGERY MACHINE VISION SCIENTIFIC MILITARY
CATEGORY
i
Image processing Menu-driven Application
Gamma correction Application specific
algorithms multiple options specific

Application specific
Image processing Real time not
Real time with emphasis on high Real time
time usually required
speed operation

For a fixed field-of-view,

Matched to video format
Resolution increased resolution is High resolution High resolution
(e.g., EIA 170)
desired

Dynamic range 8 bits/color 8 bits/color Up to 16 bits 10 or 12 bits

Application specific -
Sensitivity High contrast targets
not necessarily an issue Low noise Low noise
(high signal-to- (noise not necessarily a
noise ratio) dominant design factor)
because lighting can be
controlled
operation operation
~
::a
g
~
~
~
Vl
6 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 1-11isted several design goals by application category. To some extent

these goals are incompatible, thereby dictating design compromises. The demand
for machine vision systems is increasing dramatically. Smaller target detail can
be discerned with magnifying optics. However, this reduces the field-of-view.
For a fixed field-of-view, higher resolution (more pixels) cameras are required
to discern finer detail. When selecting an imaging system, the environment,
camera, data storage, and final image format must be considered (Table 1-2).

Table 1-2
SYSTEM DESIGN CONSIDERATIONS
TRANSMISSION
ENVIRONMENT CAMERA DISPLAY
and STORAGE

Noise
Target size Frame rate Data rate
Hard copy
Target reflectance Sensitivity Type of storage
Soft copy
Distance to target Detector size Storage capacity
Resolution
Atmospheric transmittance Array format \'ideo compression
Lighting conditions Dynamic range
Color capability

Finally, the most convmcmg evidence of system performance is a

photograph of an image. Every time an image is transferred to another device,
that device's tonal transfer and modulation transfer function will affect the
displayed image. Video recorders degrade resolution. Imagery may look slightly
different on different displays. Similarly, hard copies produced by different
printers may appear different, and there is no guarantee that the hard copy will
look the same as the soft copy in every respect. Hard copies should only be
considered as representative of system capability. Even with its over 100-year
history, wet film developing and printing must be controlled with extreme care
to create identical prints.
II II

1.2.1. GENERAL IMAGERY

Cameras for the professional broadcast television and consumer camcorder

markets are designed to operate in real time with an output that is consistent
with a standard broadcast format. The resolution, in terms of array size, is
matched to the bandwidth recommended in the standard. An array that provides
an output of 768 horizontal by 484 vertical pixels creates a satisfactory image
for conventional television. Eight bits (256 intensity levels or gray levels)
provide an acceptable image in the broadcast and camcorder industry. The
largest consumer market presently is the camcorder market.
 INTRODUCTION 7

Because solid state cameras have largely replaced image vacuum tubes, the
terminology associated with these tubes is also used with solid state cameras.
For example, compared to image vacuum tubes, solid state cameras have no
image burn-in, no residual imaging, and usually are not affected by
microphonics.

The current accepted meaning of high definition television (HDTV) is a

television system providing approximately twice the horizontal and vertical
resolution of present NTSC, PAL, and SECAM systems (discussed in Section
5.2.1., Video Timing). HDTV is envisioned as a system that provides high
resolution on large displays and projection screens. The goal is to have world-
wide compatibility so that HDTV receivers can display NTSC, PAL, and
SECAM transmitted imagery. With today's multimedia approach, any new
standard must be compatible with a variety of imaging systems ranging from 35-
mm film to the various motion picture formats.

The Federal Communication Commission (FCC) adopted a HDTV

transmission standard in December 1996 and broadcasting should begin between
1998 and 2000. HDTV camera descriptions and receivers will appear in many
journals over the next few years.

1.2.2. MACHINE VISION

In its simplest version, a machine vision system consists of a light source,
camera, and computer software that rapidly analyzes digitized images with
respect to location, size, flaws, and other preprogrammed data. Unlike other
types of image analysis, a machine vision system also includes a mechanism that
immediately reacts to images that do not conform to the parameters stored in the
computer. For example, defective parts are taken off a production line conveyor
belt.

Machine vision functions include location, inspection, gauging, counting,

identification, recognition, and motion tracking. These systems do not
necessarily need to operate at a standard frame rate. For industrial inspection,
linear arrays operating in the time delay and integration (TDI) mode can be used
to measure objects moving at a high speed on a conveyor belt.

While a multitude of cues are used for target detection, recognition, or

identification, machine vision systems cannot replace the human eye. The eye
processes intensity differences over 11 orders of magnitude, color differences,
and textual cues. The solid state camera, on the other hand, can process limited
8 CCD ARRAYS, CAMERAS, and DISPLAYS

data much faster than the human. Many operations can be performed faster,
cheaper, and more accurately by machines than by humans. Machine vision
systems can operate 24 hours a day without fatigue. They operate consistently
whereas variability exists among human inspectors. Furthermore, these cameras
can operate in harsh environments that may he unfriendly to humans (e.g.,
extreme heat, cold, or ionizing radiation).

1.2.3. SCIENTIFIC APPLICATIONS

For scientific applications, low noise, high responsivity, large dynamic

range, and high resolution are dominant considerations. To exploit a large
dynamic range, scientific cameras may digitize the signal into 12, 14, or 16 bits.
Array linearity and analog-to-digital converter linearity are important. Resolution
is specified by the number of detector elements, and scientific arrays may have
5000 x 5000 detector elements. 8 Theoretically, the array can be any size but
manufacturing considerations may ultimately limit the array size.

Low noise means low-dark current and low readout noise. The dark current
can be minimized by cooling the CCD. Long integration times can increase the
signal value so that the readout noise is small compared to the photon shot noise.

Although low light level cameras have many applications, they tend to be
used for scientific applications. There is no industry-wide definition of a "low
light level" imaging system. To some, it is simply a solid state camera that can
provide a usable image when the lighting conditions are less than 1 lux. To
others, it refers to an intensified camera and is sometimes called a low light
level television (LLLTV) system. An image intensifier amplifies a low light
level image so that it can be sensed by a solid state camera. The image-
intensifier/CCD camera combination is called an intensified CCD or ICCD. The
image intensifier provides tremendous amplification but also introduces
additional noise.

1.2.4. MILITARY APPLICATIONS

The military is interested in detecting, recognizing, and identifying targets

at long distances. This requires high resolution, low noise sensors. Target
detection is a perceptible act. A human determines if the target is present. The
military uses the minimum resolvable contrast (MRC) as a figure of merit.

Solid state cameras are popular because of their ruggedness and small size.
They can easily be mounted on remotely piloted vehicles. They are replacing
wet-film systems used for mapping and photo interpretation.
 INTRODUCTION 9

1.3. CONFIGURATIONS

Imaging systems for the four broad application categories may operate in a
variety of configurations. The precise setup depends on the specific
requirements. Figure 1-1 is representative of a closed circuit television system
where the camera output is continuously displayed. The overall image quality
is determined by the camera capability (which is based on the array
specifications), the bandwidth of the video format used (e.g., EIA 170, NTSC,
PAL, or SECAM), the display performance, and the observer. EIA 170 was
formerly called RS 170 and the NTSC standard is also called RS 170A.

Figure 1-2 illustrates a generic transmission system. The transmitter and

receiver must have sufficient electronic bandwidth to provide the desired image
quality. For remote sensing, the data may be compressed before the link.
Compression may alter the image and the effects of compression will be seen
on selected imagery. However, compression effects are not objectionable in most
imagery.

For remote applications the image may be recorded on a separate video

recorder9 •10 (Figure 1-3) or may be obtained with a camcorder. The most
popular player is the video cassette recorder (VCR). The recorder further
modifies the image by reducing the signal-to-noise ratio (SNR) and image
sharpness. As the desire for portability increases, cameras and recorders will
shrink in size. l0-!3

Camera H Display 1>

Observer

Figure 1-1. A closed circuit television system. Image quality is

determined by an observer.

Camera r- Transmitter ~
Link
Receiver r-- Display 1>
Observer
L----'

Figure 1-2. Generic remote transmission system. The transmitter and

receiver must have adequate electronic bandwidth to support the image
quality created by the solid state camera.
10 CCD ARRAYS, CAMERAS, and DISPLAYS

Camera
Video
Recorder
Display 1>
....__ ____, Observer

Figure 1-3. Imagery can be stored on video tape. However, the recorder'
circuitry may degrade the image quality.

For scientific applications, the camera output is digitized and then processed
by a computer (Figure 1-4). After processing, the image may be presented on
a monitor (soft copy), printed (hard copy), or stored. The digital image can also
be transported to another computer via the Internet, local area net, floppy disc,
or magnetic tape. For remote applications, the digital data may be stored on a
digital recorder (Figure 1-5). 14- 16

Perhaps the most compelling reason for adopting digital technology is the
fact that the quality of digital signals remains intact through copying and
reproduction unless the signals are deliberately altered. Digital signal
"transmission" was first introduced into tape recorders. Because a bit is either
present or not, multiple generation copies retain high image quality.

1>
.___ ___,~ Observer

Camera Computer

Figure 1-4. Most imagery today is enhanced through image processing.

The camera may view a scene directly or may scan a document. The
computer output hard copy may be used in newspapers, advertisements,
or reports.
 INTRODUCTION 11

Digital Digital
Camera - Video Video Computer
Recorder Recorder

Figure 1-5. Digital systems can provide very high quality imagery.
Electronic bandwidth limitations may impose data compression
requirements. Data compression may alter the image but this alteration
may not be obvious on general imagery.

In comparison, analog recorders, such as in video home systems (VHS),

provide very poor quality after just a few generations. The first digital recorder
used a format called D-1 and it became known as the CCIR 601 component
digital standard. Digital recording formats now include 16 D-1, D-2, D-3, D-4,
D-5, and D-6. Error correcting codes can be used to replace missing bits.

New digital recorders combine advanced technologies in electronics, video

compression, and mechanical transport design. Real-time digital video systems
operate at data rates that are faster than most computers. While errors in the
imagery are never considered desirable, the eye is very tolerant of defects. In
comparison, a computer is considered worthless if the error rate was high.

When it comes to displaying images (either hard copy or soft copy), the
range of the camera digitizer is often greater than the display device. A camera
may offer 12, 14, or 16 bits whereas the display may only be 8 bits. Here, a
look-up table is employed to match the camera output to the display. This may
be a simple linear relationship or specific expansion of a part of the camera's
gray scale. False color can also be introduced to enhance the displayed image.
False color is often useful for the human observer but serves no function in
machine vision systems.
12 CCD ARRAYS, CAMERAS, and DISPLAYS

While a machine vision system does not require a monitor (Figure 1-6), a
monitor is often used during system setup and for diagnostic evaluation. That is,
a computer algorithm compares an image to a stored standard. If the target does
not compare favorably to the standard, then the process is changed. For
example, this may mean sending a rejected item for rework or changing the light
intensity.

Camera
H Computer
Il __ Process
Control

Figure 1-6. Machine vision systems do not require a monitor. The

computer output controls the manufacturing process. Monitors are used
for setup and diagnostic evaluation.

1.4. IMAGE QUALITY

Image quality is a subjective impression ranking of imagery from poor to
excellent. It is a somewhat learned skill. It is a perceptual ability, accomplished
by the brain, affected by and incorporating other sensory systems, emotions,
learning, and memory. The relationships are many and not well understood.
Perception varies between individuals and over time. There exist large variations
in an observer's judgment as to the correct rank ordering from best to worst and
therefore image quality cannot be placed on an absolute scale. Visual
psychophysical investigations have not measured all the properties relevant to
imaging systems.

Many formulas exist for predicting image quality. Each is appropriate under
a particular set of viewing conditions. These expressions are typically obtained
from empirical data in which multiple observers view many images with a
known amount of degradation. The observers rank order the imagery from worst
to best and then an equation is derived that relates the ranking scale to the
amount of degradation.

Early metrics were created for film-based cameras. Image quality was
related to the camera lens and film modulation transfer functions (MTFs). With
the advent of televisions, image quality centered on the perception of raster lines
and the minimum SNR required for good imagery. Here, it was assumed that
the studio camera provided a perfect image and only the receiver affected the
image quality.
 INTRODUCTION 13

The system MTF is the major component of system analysis. It describes

how sinusoidal patterns propagate through the system. Because any target can
be decomposed into a Fourier series, the MTF approach indicates how imagery
will appear on the display.

Many tests have provided insight into image metrics that are related to image
quality. Most metrics are related to the system MTF, resolution, or the signal-
to-noise ratio. In general, images with higher MTFs and less noise are judged
as having better image quality. There is no single ideal MTF shape that provides
best image quality.

Digital processing is used for image enhancement and analysis. Because the
pixels are numerical values in a regular array, mathematical transforms can be
applied to the array. The transform can be applied to a single pixel, group of
pixels, or the entire image. Many image processing designers think of images
as an array of numbers that can be manipulated with little regard to who is the
final interpreter.

With a solid state camera system, the lens, array architecture, array noise,
and display characteristics all affect system performance. Only an end-to-end
assessment will determine the overall image quality. For example, there is no
advantage to using a high quality camera if the display cannot produce a faithful
image. Often, the display is the limiting factor in terms of image quality and
resolution. No matter how good the camera is, if the display resolution is poor,
then the overall system resolution is poor. Only if the display's contrast and
spatial resolution are better than the camera will the camera image quality be
preserved.

For consumer applications, system resolution is important with SNR being

secondary. Scientific applications may place equal importance on resolution and
MTF. The military, interested in target detection, couples the system MTF and
system noise to create the perceived signal-to-noise ratio (SNRp).
MTFSYSI:J.l (1-1)
SNRP - k--....:..:..:_ _ - - - - - - - " - - - - - - -
(system noise) (eye spatialfilter)(eye temporalfilter)

where .:11 is the intensity difference between the target and its immediate
background. The factor k is a proportionality constant that depends on the
specific camera characteristics (aperture diameter, focal length, quantum
efficiency, etc.).
14 CCD ARRAYS, CAMERAS, and DISPLAYS

When the perceived SNR is above a threshold value, the target is just
perceived. Selecting a threshold SNR and solving the equation for ill provides
the minimum resolvable contrast (MRC). When the MRC is coupled with the
target description and atmospheric transmittance effects, the target range can be
predicted.

1.5. PIXELS, DATELS, DISELS, and RESELS

The overall system may contain several independent sampling systems. The
array spatially samples the scene, the computer may have its own digitizer, and
the monitor may have a limited resolution. A monitor "pixel" may or may not
represent a "pixel" in camera space. The designer and user must understand the
differences among the sampling lattices.

Sampling theory describes the requirements that lead to the reconstruction

of a digitized signal. The original analog signal is digitized, processed, and then
returned to the analog domain. For electrical signals, each digitized value is
simply called a sample and the digitization rate is the sampling rate.

Electronic imaging systems are more complex and several sampling lattices
are present. The detector output represents a sampling of the scene. The detector
output is digitized and placed in a memory. After image processing, the data are
sent to a display medium. Although the display medium provides an analog
signal, it is typically digitally controlled.

Each device has its own minimum sample size or primary element. Calling
a sample a pixel or pel (picture element) does not seem bad. Unfortunately,
there is no a priori relationship between the various device pixels. The various
digital samples in the processing path are called pixels, datels, and disels
(sometimes called dixels). For analog systems, the minimum size is the resel
(Table 1-3). Each "-el" is mapped onto the next (Figure 1-7). However, there
is no standard definition for a resel. For optical systems it may be the Airy disk
or Rayleigh criterion. For electronic circuits, it is related to the bandwidth but
the definition has not been standardized. When a conversion takes place between
the analog and digital domains, the resel may be different from the digital
sample size. The analog signal rather than the digital sample may limit the
system resolution. In oversampled systems, the resel consists of many samples.
 INTRODUCTION 15

Table 1-3
THE "-ELS"

ELEMENT DESCRIPTION
Pixel or pel
A sample created by a detector.
(picture element)

Datel Each datum is a datel. Datels reside in a computer

(data element) memory.

Disel The smallest element (sample) that a display

(display element) medium can access.

Resel
The smallest signal supported by an analog system.
(resolution element)

Observer

Pixels Date Is Disels

Figure 1-7. Each array is mapped onto the next. The number of
elements in each array may be different. Not every array exists in every
system.

For staring arrays, the total number of pixels is equal to the number of
detectors. The detector's spatial response is determined by the detector's size
(e.g., photosensitive area). If the extent is dH in the horizontal direction and the
optics focal length is fl, then the horizontal detector-angular-subtense (DASH) is

dH (1-2)
DAS --
H fl
16 CCD ARRAYS, CAMERAS, and DISPLAYS

Similarly, in the vertical direction,

(1-3)

Staring arrays are often specified by the detector center-to-center spacing

(pitch). The horizontal pixel-angular-subtense (PASH) is

d
PAS - CCH (1-4)
H fl

where dccH is the horizontal pitch. Similarly, in the vertical direction

dccv (1-5)
PASv = fl

The fill factor is the ratio of areas:

(1-6)

With a staring array that has a 100% fill factor, the PAS is equal to the DAS
(e.g., dH = dccH and dv = dccv).

If the Airy disk (a resel) is much larger than the PAS, then the optical resel
determines the system resolution. If the number of pixels across this resel is
large, the sampling theorem can be satisfied. If the electronic imaging system
output is in a digital format, then the number of datels (samples) equals the
number of pixels. If the camera's analog output is digitized, then the number of
datels is linked to the frame grabber's digitization rate. This number can be
much greater than the number of pixels. This higher number does not create
more resolution in terms of the "-els." However, the higher sampling frequency
may improve the overall system MTF.

Image processing algorithms operate on datels. A datel may not represent a

pixel or a resel. Because there must be two samples to define a frequency, the
Nyquist frequency associated with pixels, datels, and disels may be different. If
there are more datels than pixels, the Nyquist frequency associated with an
image processing algorithm is higher than that for the detector array. Most
image processing books illustrate datels and call them pixels. The image
processing specialist must understand the differences between the sampling
lattices and take into account who will be the final data interpreter.
 INTRODUCTION 17

After image processing, the datels are output to a display medium. For
monitors, each datel is usually mapped, one-to-one, onto each disel. Monitors
are often specified by the number of addressable pixels (defined as disels in this
text). Although the number of addressable pixels (disels) may be large for
cathode ray tube based monitors, the monitor resel may be limited by the
electron beam diameter. With most display media, the finite-sized spots of two
adjacent disels overlap to provide an analog (continuous) image.

Finally, the system designer must be aware of which subsystem limits the
overall system resolution. In some respects, the starting point for system design
should begin with the final interpreter of the data. The minimum "-el" should
be discernible by the interpreter to ensure maximum transfer of information.
However, the observer may not find this image aesthetically pleasing.

1.6. REFERENCES

1. W. S. Boyle and G. E. Smith, "Charge Coupled Semiconductor Devices," Bell Systems

Technical Journal, Vol. 49, pp. 587-593 (1970).
2. G. F. Amelio, M. F. Tompsett, and G. E. Smith, "Experimental Verification of the Charge
Coupled Concept," Bell Systems Technical Journal, Vol. 49, pp. 593-600 (1970).
3. M. J. Howes and D. V. Morgan, eds., Charge-Coupled Devices and Systems, John Wiley and
Sons, NY (1979).
4. C. H. Sequin and M. F. Tompsett, Charge Transfer Devices, Academic Press, NY (1975).
5. E. S. Yang, Microelectronic Devices, McGraw-Hill, NY (1988).
6. E. L. Dereniak and D. G. Crowe, Optical Radiation Detectors, John Wiley and Sons, NY
(1984).
7. A. J. P. Theuwissen, Solid-State Imaging with Charge-Coupled Devices, Kluwer Academic
Publishers, Dordrecht, The Netherlands (1995).
8. S. G. Chamberlain, S. R. Kamasz, F. Ma, W. D. Washkurak, M. Farrier, and P.T. Jenkins,
"A 26.3 Million Pixel CCD Image Sensor," in IEEE Proceedings of the International
Conference on Electron Devices, pp. 151-155, Washington, D.C. December 10, 1995.
9. Z. Q. You and T. H. Edgar, Video Recorders: Principles and Operation, Prentice Hall, New
York, NY (1992).
10. M. Hobbs, Video Cameras and Camcorders, Prentice Hall, New York, NY (1989).
11. K. Tsuneki, T. Ezaki, J. Hirai, and Y. Kubota, "Development of the High-band 8 mm Video
System," IEEE Transactions on Consumer Electronics, Vol. 35(3), pp. 436-440 (1989).
12. M. Oku, I. Aizawa, N. Azuma, S. Okada, K. Hirose, and M. Ozawa, "High Picture Quality
Technologies for and S-VHS Portable VCR," SMPTE Journal, Vol. 98(9), pp. 636-639 (1989).
13. T. Kawamura, S. Kasai, T. Tominaga, H. Sato, and M. Inatsu, "A New Small-Format VTR
Using an 8 mm Cassette," SMPTE Journal, Vol. 96(5), pp. 466-472 (1987).
14. J. Watkinson, The D-3 Digital Video Recorder, Focal Press, Oxford, England (1992).
15. K. Suesada, K. Ishida, J. Takeuchi, I. Ogura, and P. Livingston "D-5: 112-in. Full Bit Rate
Component VTR Format, SMPTE Journal, Vol. 103(8), pp. 507-516 (1994).
16. J. Hamalainen, "Video Recording Goes Digital," IEEE Spectrum, Vol. 32(4), pp. 76-79 (1995).
2
RADIOMETRY and PHOTOMETRY

The radiometric/photometric relationship between the scene and camera is

of primary importance. It determines the output signal. The second
radiometric/photometric consideration is the match between the display's spectral
output and the eye's spectral response. Display manufacturers have considered
this in detail and this reduces the burden on the system designer. However, the
displayed image typically is not a precise reproduction of the scene. While the
displayed image usually represents the scene in spatial detail, the intensity and
color rendition may be different.

Scenes in the visible and near infrared are illuminated by the sun, moon,
starlight, night glow, or artificial sources. Since both the target and its
background are illuminated by the same source, targets are detected when
differences in reflectance exist. The camera's output voltage depends on the
relationship between the scene spectral content and the spectral response of the
camera.

Historically, cameras were designed to operate in the visible region only.

For those systems, it was reasonable to specify responsivity in photometric units
(e.g., volts/lux). However, when the spectral response extends past the visible
region as with silicon photodetectors, the use of photometric units can be
confusing and even misleading.

Radiometry describes the energy or power transfer from a source to a

detector. When normalized to the eye's response, photometric units are used.
Radiometric and photometric quantities are differentiated by subscripts: e is used
for radiometric units, q for photons, and v is U5ed for photometric units.

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

18
 RADIOMETRY and PHOTOMETRY 19

2.1. RADIATIVE TRANSFER

Spectral radiant sterance, Le, is the basic quantity from which all other
radiometric quantities can be derived. It contains both the areal and solid angle
concept 1 that is necessary to calculate the radiant flux incident onto a system. It
is the amount of radiant flux, (watts), per unit wavelength (micrometers)
radiated into a cone of incremental solid angle (steradians) from a source whose
area is measured in meters (Figure 2-1)

w (2-1)

Similarly, Lq is the spectral photon sterance expressed in photons/(s-m2-J.tm-sr).

The variables Le and Lq are invariant for an optical system that has no
absorption or reflections. That is, Le and Lq remain constant as the radiation
transverses through the optical system. Table 2-1 provides the standard
radiometric units.

Solid Angle oO
Source Area
aAs

Figure 2-1. Radiant sterance.

20 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 2-1
STANDARD SPECTRAL RADIOMETRIC UNITS

DEFINITION SYMBOL UNITS

Radiant energy Q. Joule (J)

Radiant flux ~~>. Watts (W)

Spectral radiant intensity I. W /(m2-~tm-sr)

Spectral radiant exitance

(from a source) M. W/(m2 -~tm)
(also radiant emittance)

Spectral radiant incidance

(onto a target) E. W/(m2-~tm)
(also irradiance)

Spectral radiant sterance W/(m2-pm-st)

L.
(also radiance)

For Lambertian sources, the radiance is emitted into a hemisphere. Then the
spectral radiant exitance is related to the spectral radiant sterance by

Le<J. )
- Me<'J.) -----
w (2-2)
1t m 2 -~m-sr

and

L (A.) - MqCA.) photons (2-3)

q 1t
s-m 2 --J-Lm-sr
 RADIOMETRY and PHOTOMETRY 21

2.2. PLANCK'S BLACKBODY LAW

The spectral radiant exitance of an ideal blackbody source whose absolute

temperature is T (Kelvin), can be described by Planck's blackbody radiation law

w (2-4)

where the first radiation constant is c 1 = 3.7418 x 108 W-p.m4/m2, the second
radiation constant is ~ = 1.4388 x 104 p.m-K, and ).. is the wavelength expressed
in micrometers. The value T is also called the color temperature. Figure 2-2
illustrates Planck's spectral radiant exitance in logarithmic coordinates. Since a
photodetector responds linearly to the available power, linear coordinates may
provide an easier representation to interpret (Figure 2-3). Each curve has a
maximum at )..PEAK· Wien's displacement law provides )..PEAK T = 2898 p.m-K. A
source must have an absolute temperature above about 700 K to be perceived by
the human eye.

The spectral photon exitance is simply the spectral radiant exitance divided
by the energy of one photon (he/)..):

(2-5)

or

(2-6)

where his Planck's constant (h = 6.626 x 10-34 J-s), cis the speed of light (c =
3 x 108 m/s), and the third radiation constant is CJ = 1.88365 x 1027 photons-
p.m3/s-m2. Figures 2-4 and 2-5 provide the spectral photon exitance in
logarithmic and linear coordinates, respectively, for sources typically used for
CCD calibration. These curves have a maximum at )..PEAK T = 3670 p.m-K.
22 CCD ARRAYS, CAMERAS, and DISPLAYS

1E+08
1-
z
:$ 1E+06
ow
~~
....I<( 1E+04
~!::
1-><
ow 1E+02
w
D..
U)
1E+OO~---L---L-+~~~~~+-~~~~~

0.1 1 10 100
WAVELENGTH (pm)

Figure 2-2. Planck's spectral radiant exitance plotted in logarithmic

coordinates. The units are W/(m2-p.m). The light line is Wien's law.
1.5E+08
1-
z
~
ow 1.0E+08
~~
....I<(
~!::
.... >< 5.0E+07
ow 4870
w
D..
U) O.OE+OO ..,_4_________ ...,;~,oo _ _ _ _,..
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2
WAVELENGTH (pm)
Figure 2-3. Planck's spectral radiant exitance plotted in linear
coordinates for typical sources used for calibrating ceo arrays. The
units are W/(m2-p.m). The maximum values occur at 1.01, 0.60, and
0.45 p.m for color temperatures of 2856, 4870, and 6500K, respectively.
1.0E+27
z
g 1.0E+25
Ow
if:~ 1.0E+23
....I<(
~ !:: 1.0E+21
t;fii
If 1.0E+19

+----1"'-----------------
U)

1.0E+17
0.1 1 10
WAVELENGTH (pm)
Figure 2-4. Planck's spectral photon exitance forT = 2856, 4870, and
6500 K. The maximum values occur at 1.29, 0. 75, and 0.56 p.m,
respectively. The units are photons/(s-m2-rm).
 RADIOMETRY and PHOTOMETRY 23

4.0E+26
z
0
..... 3.0E+26
Ow
:z:u
ll.z
..Joe( 2.0E+26
~!::: 4870
..... x
ow 1.0E+26
w
ll.
en 2856
O.OE+OO
0.2 0.4 0.6 0.8 1.2

WAVELENGTH (pm)

Figure 2-5. Planck's spectral photon exitance plotted in linear coordi-

nates. The units are photons/(s-m2 -p.m).

2.3. PHOTOMETRY

Photometry describes the radiative transfer from a source to a detector where

the units of radiation have been normalized to the spectral sensitivity of the eye.
Photometry applies to all systems that are sensitive to visible radiation. The
luminous flux emitted by a source is
0.751'm

«<> v - KM J V().) M/).) d). lumens , (2-7)

0.38!lm

where Mp(},) is the power expressed in units of WI p.m and KM is the luminous
efficacy for photopic vision. It is 683 lumens/W at the peak of the photopic
curve (A-:::::0.55 p.m) and 1746 lumens/W for the scotopic region at A-:::::0.505
p.m. Although both photopic and scotopic units are available, usually only the
photopic units are used (Table 2-2 and Figure 2-6). At very low light levels (less
than 5 x 10·3 lux), the eye's rods operate (scotopic response). For light levels
above 5 x 10·2 lux, the eye's cones (photopic response) respond.
24 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 2-2
PHOTOPIC and SCOTOPIC EYE RESPONSE

Wavelength Photopic Scotopic Wavelength Photopic Scotopic

run V(>.) V'(>.) run V(>.) V'(>.)

380 0.00059 570 0.952 0.2076

390 0.00012 0.00221 580 0.870 0.1212
400 0.0004 0.00929 590 0.757 0.0655
410 0.0012 0.03484 600 0.631 0.03315
420 0.0040 0.0966 610 0.503 0.01593
430 O.Q116 0.1998 620 0.381 0.00737
440 0.023 0.3281 630 0.265 0.00335
450 0.038 0.455 640 0.175 0.00150
460 0.060 0.567 650 0.107 0.00067
470 0.091 0.676 660 0.061 0.00031
480 0.139 0.793 670 0.032
490 0.208 0.904 680 0.017
500 0.323 0.982 690 0.0082
510 0.503 0.997 700 0.0041
520 0.710 0.935 710 0.0021
530 0.862 0.811 720 0.00105
540 0.954 0.650 730 0.00052
550 0.995 0.481 740 0.00025
560 0.995 0.3288 750 0.00012

w
en
z 0.8
0 Scotopic
D.
en 0.6
w
a:
w 0.4
~
1-
:5w 0.2
a: 0
0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75
WAVELENGTH (pm)

Figure 2-6. Photopic and scotopic eye responses.

 RADIOMETRY and PHOTOMETRY 25

2.3.1. UNITS

Unfortunately, there is overabundance of terminology being used in the field

of photometry (Table 2-3). The SI units are recommended. Luminance emittance
and illuminance are sometimes used as alternatives to luminous exitance and
luminous incidance, respectively. Brightness and luminance may be used in
place of luminous sterance. For convenience, luminance sterance has been
labeled as the normalized intensity or nit. For Lambertian sources, the
luminance is emitted into a hemisphere whose solid angle is 21r. The luminous
exitance of a Lambertian surface is simply the luminance exitance given in Table
2-3 divided by 1r. For Lambertian sources, the luminance exitance terms are the
apostilb (abs), Lambert (L), and foot-Lambert (fL) for the SI, CGS and English
systems, respectively.

Figure 2-7 illustrates the geometric relationship between the SI, CGS, and
English luminous incidance units. The numeric relationship is provide in Table
2-4.

A= 1m2

-----------1m -----------
Figure 2-7. Geometric relationship between the SI, CGS, and English
luminous incidance units. The solid angle is one steradian. (Not to
scale).
 Table 2-3
STANDARD PHOTOMETRIC UNITS
(The SI units are recommended)

SI and MKS ENGLISH

DEFINITION SYMBOL CGS UNITS
UNITS UNITS

Luminous energy Qv Talbot (T) Talbot (T) Talbot (T)

Luminous flux <l>v Lumen (lm) Lumen (lm) Lumen (lm)

(A
~
Candela (cd) Candela (cd) Candela (cd)
~~ Luminous intensity IV lm/sr lm/sr lm/sr
Q
Luminous exitance
Lux (lx) Phot (ph)
§ (from a source) Mv lumen/m2 lumen/cm2
Footcandle (fc)
lumen/ft2
vi (also luminous emittance)

~
s
~-
Luminous incidance
(onto a target)
(also illuminance)
Ev
Lux (lx)
Lumen/m2
Phot(ph)
lumen/cm2
Footcandle (fc)
lumen/ft2

Nit

~
Luminous sterance Stilb (sb)
LV Lumen/m2-sr candela/ft2
(also brightness and luminance) cd/cm2
or cd/m2
8
\..)

\0
("''
 RADIOMETRY and PHOTOMETRY 27

Table 2-4
CONVERSION BETWEEN SI, CGS, and ENGLISH UNITS

PHOT FOOTCANDLE LUX

1 Phot = 1 929 1 X 104

1 Footcandle = 1.076 X 10-3 1 10.764

1 Lux= 1 X 10-4 0.0929 1

2.3.2. TYPICAL ILLUMINATION LEVELS

Natural lighting levels can vary by over 9 orders of magnitude (Table 2-5).
The minimum level is limited by night glow and the maximum level is provided
by the sun. At very low light levels (less than 5 x 10-3 lux), the eye's rods
operate (scotopic response). For light levels above 5 X 10-2 lux, the eye's cones
(photopic response) respond. Between these two values, both rods and cones are
operating and the eye's response is somewhat between the two values. This
composite response is called the mesopic response. Table 2-6 provides typical
artificial lighting levels.

The eye automatically adapts to the ambient lighting conditions to provide

an optimized image. Cameras may need neutral density filters if the light level
is too high. Cameras may have an automatic iris and shutter speed to optimize
the image on the detector array. If the light level is too low, an intensified
camera may be necessary.
28 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 2-5
NATURAL ILLUMINANCE LEVELS
EYE AVERAGE LUMINOUS
SKY CONDITION
RESPONSE INCIDANCE (lux)

Direct sun Photopic 105

Full day light Photopic 10•

Overcast sky Photopic 103

Very dark day Photopic 102

Twilight Photopic 10

Deep twilight Photopic 1.0

Full moon Photopic 10"'

Quarter moon Mesopic ro-2

Moonless, clear night 10-3
Scotopic
(starlight)

Moonless, overcast
Scotopic 10·•
(night glow)

Table 2-6
TYPICAL ARTIFICIAL ILLUMINANCE LEVELS
AVERAGE LUMINOUS
LOCATION
INCIDANCE (lux)

Hospital operating theater 105

TV studio 103

Shop windows 103

Drafting office 500

Business office 250

Good street lighting 20

Poor street lighting ro·'

 RADIOMETRY and PHOTOMETRY 29

2.4. SOURCES

As shown in Figure 2-3 (page 22), the peak wavelength shifts toward the
blue end of the spectrum as the color temperature increases. This puts more
energy in the visible region. There is approximately 70% more luminous flux
available from a lamp operating at 3200 K than a light operating at 3000 K based
on blackbody curves. This is the basis for using a higher color temperature lamp
for higher illumination levels. However, the lifetime of tungsten halogen bulbs
decreases dramatically with increasing color temperature. It is far better to use
more lamps to increase the luminous flux than to increase the color temperature
and sacrifice lifetime.

2.4.1. CALIBRATION SOURCES

The CIE (Commission lnternationale de 1'Eclairage or International

Commission on Illumination) recommended four illuminants that could be used
for calibration of cameras sensitive in the visible region (Table 2-7).
Commercially available sources simulate these illuminants only over the visible
range (0.38 to 0. 75 f.Lm). The source intensity is not specified; only the effective
color temperature is specified. Illuminants A and D65 are used routinely whereas
illuminants B and C are of historical interest.

Table 2-7
CIE RECOMMENDED ILLUMINANTS
CIE EFFECTIVE COLOR
DESCRIPTION
ILLUMINANT TEMPERATURE

Light from an
A 2856K
incandescent source

B 4870K Average noon sunlight

Average daylight (sun +

c 6770K
sky)

D 65 or Daylight with a corrected

6500 K
D6500 color temperature
30 CCD ARRAYS, CAMERAS, and DISPLAYS

Illuminant A is a tungsten filament whose output follows Planck's blackbody

radiation law. The other illuminants are created by placing specific filters in
front of illuminant A. These filters have spectral characteristics that, when
combined with illuminant A, provide the relative outputs illustrated in Figure 2-
8. On a relative basis, the curves can be approximated by blackbodies whose
color temperatures are 4870, 6770, and 6500 K for illuminant B, C, D65 ,
respectively.

250
~
c;; 200
z
w
..... 150
~
w
;:: 100
.....
~ 50
w
a: 0
0.4 0.5 0.6 0.7 0.8
WAVELENGTH (pm)
Figure 2-8. Relative output of the standard 11luminants. Compare the shapes
to Figure 2-3, page 22.

2.4.2. REAL SOURCES

A tungsten filament bulb emits light that is closely matched to an ideal

blackbody radiator. This is not so for discharge lamps such as fluorescent lamps.
For these lamps, a blackbody curve that approximates the output is fit to the
spectral radiant exitance (Table 2-8). This approximation is for convenience and
should not be used for scientific calculations. The actual spectral photon sterance
must be used for nonideal sources 2 (discussed in the next three sections).
Although many discharge lamps seem uniform white in color, they have peaks
in the emission spectra (Figure 2-9). The apparent solar color temperature
depends upon latitude, time of day, and a variety of environmental conditions
(e.g., aerosols, pollutants, and cloud cover). The standardization of illuminant
C at 6770 K was arbitrary. It was considered representative of "average"
daylight.
 RADIOMETRY and PHOTOMETRY 31

Table 2-8
APPROXIMATE COLOR TEMPERATURE
APPROXIMATE COLOR
SOURCE
TEMPERATURE

Northern sky light 7500K

Average daylight 6500K

Xenon (arc or flash) 6000K

Cool fluorescent lamps 4300K

Studio tungsten lamps 3200K

Warm fluorescent lamps 3000K

Floodlights 3000K

Domestic tungsten lamps 2800-2900K

Sunlight at sunset 2000K

Candle flame 1800K

'S
D.
.5
~l, 1 j1-.. -
==
(b)

0.3 0.4 0.5 0.6 0.7 0.3 0.4 0.5 0.6 0.7
Wavelength (~m)
(c) (d)

Figure 2-9. Relative output of a mercury arc lamp as a function of

pressure. 3 (a) 21 atm, (b) 75 atm, (c) 165 atm, and (d) 285 atm.
32 CCD ARRAYS, CAMERAS, and DISPLAYS

Usually there is no relationship between the color temperature and contrast

of a scene and what is seen on a display. This is not the fault of the camera and
display manufacturers; they strive for compatibility. The observer, who usually
has no knowledge of the original scene, will adjust the display for maximum
visibility and aesthetics.

The video signal is simply a voltage with no color temperature associated

with it. The display gain and offset can be adjusted such that a white target
appears as if it was illuminated by a source whose color temperature is between
3200 K and 10,000 K. Most displays are preset to either 6500 K or 9300 K
(Figure 2-10). As the color temperature increases, whites appear to change from
a yellow tinge to a blue tinge. The perceived color depends on the adapting
illumination (e.g., room lighting) Setting the color temperature to 9300 K
provides aesthetically pleasing imagery and this setting is unrelated to the actual
scene color temperature.

1.2E+09
1-
z 1.0E+09
:$
ow 8.0E+08
ill!~
..JOCS: 6.0E+08
Ql!!::
1->< 4.0E+08
ow
w 2.0E+08
a..
rn
O.OE+OO
0.4 0.45 0.5 0.55 0.6 0.65 0.7
WAVELENGTH (pm)

Figure 2-10. Display color temperature is typically preset at either 6500

or 9300 K. Some displays allow tuning from 3200 to 10,000 K. The
displayed color temperature is independent of the scene color
temperature. A scene illuminated with a 2854 K source (yellow tinge)
can appear bluish when displayed at 9300 K.
 RADIOMETRY and PHOTOMETRY 33

2.5. CAMERA FORMULA

If an imaging system is at a distance R 1 from a source (Figure 2-11), the

number of photons incident onto the optical system of area A, is during time t1NT
is

(2-8)

where the small angle approximation was used (valid when R/ ~ A 0 ) . TATM is
the intervening atmospheric transmittance. The number of on-axis photons
reaching the image plane is:

(2-9)

where ToPTrcs is the system's optical transmittance.

When the image size is much larger than the detector area (A1 ~ A0 ), the
source is said to be resolved or the system is viewing an extended source.
Equivalently, the detector is flood-illuminated. In most solid state camera
applications, the source is resolved. The value A0 is the effective sensitive
detector area. Microlenses (discussed in Section 3.11.1., Microlenses) and
optical anti-alias filters (discussed in Section 10.3.4., Optical Anti-alias Filter)
increase the effective area.

The number of photons incident onto the detector is simply the ratio of the
areas:

AD (2-10)
n DETECTOR = n IMAGE A
I

Using the small angle approximation for paraxial rays,

(2-11)
34 CCD ARRAYS, CAMERAS, and DISPLAY..'>'

Imaging System
Entrance Aperture
Area, Aa

Source

Detector
Area Ao
Area AS
~ R,

Figure 2-11. An imaging system directly viewing a source.

The number of photons reaching the detector becomes:

LqAoAD
(2-12)
nDETECTOR =
2 (1 M )2 TOP11CS TATM t /NT
fl + OP11CS

The optical magnification is MoPTics = R2/R 1. Here, R1 and R2 are related to

the system's effective focal length, fl, by

1 1
+ (2-13)
~ fl

Assuming a circular aperture (Aa = 7rD2/4) and defining the f-number as F =

fl/D (see Appendix):
LA
nDETECTOR =
1t q D T T t (2-14)
4 p2 (1 M )2 0P11CS ATM /NT
+ OP11CS

An off-axis image will have reduced incidance compared to an on-axis image by

cosine48. As required, cosine48 can be added to all the equations.
 RADIOMETRY and PHOTOMETRY 35

Many of the variables are a function of wavelength. The number of

photoelectrons generated in a solid state detector is
'-z
nP• = JRqO. )nDETEcroi'A) d'A (2-15)
'-t

or

(2-16)

where Rq()..) has units of electrons per photon and is simply the detector's
quantum efficiency and t1NT becomes is the integration time. As the source
moves to infinity, Movncs approaches zero.

In photography, shutter speeds (exposure times) vary approximately by a

factor of two (e.g., 1/30, 1/60, 1/125, 1/250, etc.). Thus, changing the shutter
speed by one setting changes 11pe approximately by a factor of 2. f-stops have
been standardized to 1, 1.4, 2, 2.8, 4, 5.6, 8 ..... The ratio of adjacent f-stops
is ..,[2. Changing the lens speed by one f-stop changes the f-number by a factor
of ..,[2. Here, also, the 11pe changes by a factor of 2.

In an actual application, what is of interest is the signal difference produced

by a target and its immediate background. Here, both the target and the
background are assumed to be illuminated by the same source (artificial lighting,
sun, moon, night glow, star light). Let t.p = PT(A.)- p8 (A.) where PT(A.) and p8 (A.)
are the spectral reflectances of the target and background, respectively. Although
the number of electrons is used for solid state array calculations, the camera
output is a voltage:
(2-17)

where
'- 2 ('A)t A
t.n
pe
=JR ('A) 4 t.pEF2q-scENE
q
1t
( M
INT D T
)2 OPTICS
('A)T
ATM'
f'A)d'J... (2-18)
'-t 1+ OPTICS
36 CCD ARRAYS, CAMERAS, and DISPLAYS

The value GcAMERA contains both the array output conversion gain (units of
volts/electron) and the subsequent amplifier gain. Tables 2-5 and 2-6 provide
nominal values for Ee-scENE·

It would appear that by simply decreasing the f-number, 11pe or <lllpe would
increase. Often, lower f-number lens systems have more optical elements and
therefore may have lower transmittance. The value ToPTics/F 2 must increase to
increase 11pe or <l11pe. Lens systems may be described by the T-number:
F
T/# = (2-19)
JTOPT1CS

If T0 pncs is a function of wavelength, then T/# will also be a function of

wavelength. Here ToPTicsiF2 in Equation 2-16 or 2-18 is replaced by 1/(T/#)2 •

For back-of-the-envelope calculations, the atmospheric transmittance is

assumed to have no spectral features TATM(}..) :::::; TATM (discussed in Section
12.6.1., Contrast Transmittance). For detailed calculations, the exact spectral
transmittance4 must be used.

~============================================
Example 2-1
VISUAL THRESHOLD

An object is heated to incandescence. What is the approximate color

temperature for it to be just perceived? Let the object area be 1 cm2 • The
observer is 1 m from the object.

After adapting to the dark at least 60 min, the eye can perceive a few
photons per second (absolute threshold). Here, the pupil dilates to about 8 mm
(A0 = 5 x I0-5 m2). When the eye is dark adapted, only the rods are functioning
and the eye's spectral response is approximately 0.38 ~-tiD to 0.66 ~-tm. The
number of photons per second reaching the eye from a Lambertian source is
0 6611
" m A M (>.. 1) A M (>.. 1)
A d>.. ~ A 1!..>.. (2-20)
nEYE - J
o.3811m
_____!!_

Rt
2
q
1t
'
S
_!?._

R1
2
q

1t
o'
S '
 RADIOMETRY and PHOTOMETRY 37

where ilf. = 0.28 p.m. The peak scotopic eye response occurs at A, = 0.515 p.m.
The observer can just perceive this object at 640 K when he is in a completely
blackened room. As the ambient illumination increases, the pupil constricts (A,
decreases). Furthermore, the eye's detection capability depends on the ambient
lighting. As the light level increases, the target flux must also increase to be
perceptible.

The eye's ability to just discern intensity differences depends on the intensity
of the surrounding illuminance. At 0.01lux the object must provide 10 times
more flux than the background to be discerned by the eye. If the surround is at
700K, than the target must be at 750K to be perceptible. At 1000lux, the object
must provide 1000 times more flux then the surround. If the surround color
temperature is 1000 K, then the target must be heated to 1350 K to be
perceptible. Photometric units are not linearly related to color temperature.

These back-of-the-envelope calculations do not include the eye's spectral

response (Table 2-2, page 24) and therefore only illustrate required intensity
differences. The eye changes its ability to perceive differences with the surround
illumination. This explains, in part, why television receiver imagery appears so
bright in a darkened room and cannot be seen in bright sunlight.
=================================================~

2.6. NORMALIZATION

"Normalization .... is the process of reducing measurement results as nearly

as possible to a common scale" 5 • Normalization is essential to ensure that
appropriate comparisons are made. Figure 2-12 illustrates the relationship
between the spectral response of a system to two different sources. The output
of a system depends on the spectral features of the input and the spectral
response of the imaging system. Simply stated, the camera output depends on
the source used.

Variations in output can also occur if "identical" systems have different

spectral responses. Equation 2-16 (page 35) is integrated over the wavelength
interval of interest. Since arrays may have different spectral responses, an
imaging system whose spectral response is 0.4 to 0. 7 p.m may have a different
responsivity than a system that operates 0.38 to 0. 75 p.m although both systems
are labeled as visible systems. The spectral response of solid state arrays varies
from manufacturer to manufacturer (discussed in Section 4. 1. 1. , Spectral
Response). Systems can be made to appear as equivalent or one can be made to
provide better performance by simply selecting an appropriate source.
38 CCD ARRAYS, CAMERAS, and DISPLAYS

While the sensor may be calibrated with a standard illuminant, it may not
be used with the same illumination. Calibration with a standard illuminant is
useful for comparing camera responsivities under controlled conditions.
However, if the source characteristics in a particular application are significantly
different from the standard illuminant, the selection of one sensor over another
must be done with care. For example, street lighting from an incandescent bulb
is different from that of mercury-arc lights (compare Figure 2-8 with Figure 2-9,
pages 30 and 31, respectively). The only way to determine the relative
responsivities is to perform the calculation indicated in Equation 2-16.

The effect of ambient lighting is particularly noticeable when using low light
level televisions at nighttime. The spectral output of the sky (no moon) and
moon (clear night) are significantly different. Figure 2-13 illustrates the natural
night sky luminous incidence. Since an abundance of photons exists in the near
infrared (0. 7 to 1.1 p.m), most night vision systems (e.g., image intensifiers,
starlight scopes, snooper scopes, etc.) are sensitive in this region.

1- 2.0E+08
::J
D..
1- 1.5E+08
::J
0
w 1.0E+08
~
1-
5.0E+07
:3w
0:: O.OE+OO
0.2 0.4 0.6 0.8 1.2

WAVELENGTH (pm)

Figure 2-12. Sources with different spectral outputs can produce

different system outputs. The 6500-K source provides more in-band
radiant flux than the 4870-K source. The system output will be higher
when viewing the 6500-K source.
 RADIOMETRY and PHOTOMETRY 39

0.4 0.6 0.8 1.0 1.2 1.4

WAVELENGTH (pm)

Figure 2-13. Night sky luminous incidance (From Reference 6).

2.7. NORMALIZATION ISSUES

A camera output depends on both the system spectral response and the color
temperature of the illuminating source (Equation 2-16, page 35). When the
camera spectral response is within the eye's response, then photometric units are
reasonable. When the camera's response is outside the eye's response curve,
photons contribute to signal but the photometry remains constant. Even though
two sources may provide the same luminous incidance, the camera output can
be quite different. Sometimes an infrared filter is used to restrict the wavelength
response. When the filter is added, the average responsivity becomes
'-2
J TIR-FILTER R.(J. . )M.(I.. ) d)..
AI A (2-21)
Rv = - - - - ' - - - - - - - - -
0.75 lumens
683 f V()..)M.(J.. ,T) d)..
0.38

This average type responsivity is useful for comparing the performance of

cameras with similar spectral responses. It is appropriate if the source employed
during actual usage is similar to the one used for the calibration.
40 CCD ARRAYS, CAMERAS, and DISPLAJ'S

For ideal photon detectors the spectral responsivity (expressed in A/W) is:

R.(J.) - __l_ R
Ap P
(2-22)

- 0 elsewhere

where RP is the peak responsivity and Ap is the wavelength at which Rp occurs.

Since Rq is the quantum efficiency, R, = (q>.ihc)Rq. When the wavelength is
expressed in micrometers, Re = (>./1.24)Rq and Rp = Rq/E0 • For silicon, the
band gap is Eo = 1.12 eV resulting in a cutoff wavelength of Ap = 1.24/E0 =
1.1 p.m. The quantity q is the electron charge (q = 1.6 X 10- 19 coul).

The ideal silicon detector response is used to estimate the relative outputs of
two systems: one with an ideal IR filter (transmittance is unity from 0.38 to 0. 70
p.m and zero elsewhere) and one without. The relative outputs are normalized
to that expected when illuminated with a CIE A illuminant (Table 2-9). The
sources are considered ideal blackbodies with exitances that follow Equation 2-4
(page 21). As the source temperature increases, the output increases but not
linearly with color temperature.

Table 2-9
RELATIVE OUTPUT
This is assuming ideal blackbodies and ideal Si spectral response.
A specific sensor may deviate significantly from these values.

SOURCE RELATIVE OUTPUT RELAT!VE OUTPUT

TEMPERATURE (0.38 to 1.1 p.m) (0.38 to 0.70 p.m)

2856 1.00 0.196

3200 1.96 0.482

4870 15.7 6.95

6500 48.9 27.4

6770 56.5 32.4

 RADIOMETRY and PHOTOMETRY 41

Only photons within the visible region affect the number of available lumens.
For the system without the IR filter, the detector is sensing photons whose
wavelengths are greater than 0.7 p.m even though the number of lumens is not
affected by these photons (Figure 2-14). As the color temperature increases, the
flux available to the observer (lumens) increases faster than the output voltage.
As a result, responsivity, expressed in A/lumen, decreases with increasing color
temperature (Table 2-10). This decrease affects the wide spectral response
systems to a greater extent. Specifying the output in lumens is of little value for
an electro-optical sensor whose response extends beyond the eye's response.
Radiometric units are advised here.

The values in Tables 2-9 and 2-10 should be considered as illustrative. Solid
state detectors do not follow the ideal response (discussed in Section 4.1.1.,
Spectral Response). The IR filter cutoff also varies by manufacturer. Only a
detailed spectral response evaluation permits correct comparisons among
detectors. Since there is considerable variation in spectral response and type of
IR filter used, it is extremely difficult to compare systems based on responsivity
values only.

Photopic
4.0E+07 observer
w
0
z
<
1-
3.0E+07
x
w
..J 2.0E+07
~
1-
0 1.0E+07
w
D..
en O.OE+OO
0.4 0.6 0.8 1.2

WAVELENGTH (pm)
Figure 2-14. Spectral relationship between source characteristic, detector
response, and the photopic observer. As the color temperature increases,
the energy available to the observer increases rapidly. Therefore, the
number of lumens increases dramatically.
42 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 2-10
RELATIVE RESPONSIVITY (A/lumen)
A specific sensor may deviate significantly from these values

SOURCE RELATIVE RELATIVE

TEMPERATURE RESPONSIVITY RESPONSIVITY
(0.38 to 1.1 J.Lm) (0.38 to 0.70 J.Lm)
2856 1.00 0.196

3200 0.754 0.185

4870 0.387 0.171

6500 0.314 0.176

6770 0.308 0.177

Example 2-2
SOURCE SELECTION

A camera's signal-to-noise ratio is too small. The engineer can either change
the light source to one with a higher color temperature or add more lamps. What
are the differences?

With an ideal blackbody, the color temperature completely specifies the

luminous exitance. Increasing the color temperature (by increasing the voltage
on the bulb) will increase the luminous flux. For real sources, the color
temperature is used only to denote the relative spectral content. A 50-W bulb
whose output approximates a 3200-K source will have a lower photometric
output than a 1000-W bulb operating at 2856 K. Adding more lamps does not
change the relative spectral content but increases the luminous flux.

=============================================~
 RADIOMETRY and PHOTOMETRY 43

Example 2-3
RELATIVE VERSUS ABSOLUTE OUTPUT

Should a detector be calibrated with a 5- or 1000-W bulb?

If the relative spectral output of the two bulbs is the same (e.g., CIE A), the
responsivity is independent of the source flux. The bulb intensity should be
sufficiently high to produce a good signal-to-noise ratio. It should not be so high
that the detector saturates.

=============================================~

System output does not infer anything about the source other than that an
equivalent blackbody of a certain color temperature would provide the same
output. This is true no matter what output units are used (volts, amps, or any
other arbitrary unit). These units, by themselves, are not very meaningful for
system-to-system comparison. For example, .:1VCAMERA (and equivalently the
camera responsivity) can be increased by increasing the system gain. As such,
it is dangerous to compare system response based on only a few numbers.

Arrays may be specified by their radiometric responsivity (V/(J-m-2) or by

their photometric responsivity (V /lux). The relationship between the two depends
on the spectral content of the source and the spectral response of the array
(further discussed in Section 4.1.2., Responsivity). In principle, the source can
be standardized (e.g., selection of the CIE A illuminant), but the array spectral
response varies among manufacturers and may vary within one manufacturing
environment if different processes are used. Therefore, there is no simple
(universal) relationship between radiometric and photometric responsivities.

The correct analysis method is to perform a radiometric calculation using the

source spectral output (which may not be an ideal blackbody) and using the
spectral response of the camera system. Using standard sources for quality
control on a production line is valid since the spectral response of the detectors
should not change very much from unit to unit. The camera spectral response
must include the optical spectral transmittance of all filters present.

A 3200-K color temperature source is not well matched to the eye's

response. The eye peaks at 0.555 ~-tm but the source peaks at 0.90 ~-tm. With
silicon detectors, the spectral response tends to match the output of a 3200-K
source. Thus CCD detectors will perform rather well with ambient lighting. For
machine vision systems, the infrared-blocking filter is not typically used.
44 CCD ARRAYS, CAMERAS, and DISPLAYS

However, since the human observer is not familiar with infrared images, the
imagery may appear slightly different compared to that obtained with the IR-
blocking filter.

Low light level televisions can have a variety of spectral responses

(discussed in Section 5.4.2., Intensified Solid State Cameras). The methodology
presented in this chapter can be used to calculate responsivity and output
voltage. The results given in Tables 2-9 and 2-10 do not apply to intensified
systems.

2.8. REFERENCES
l. C. L. Wyatt, Radiometric System Design, Chapter:;, Macmillan Publishing Co., New York,
NY (1987).
2. D. Kryskowski and G. H. Suits, "Natural Sources," m Sources of Radiation, G. J. Zissis, ed.,
pp. 151-209, Volume 1, The Infrared and Electro-Optical Systems Handbook, J. S. Accetta and
D. L. Shumaker, eds., copublished by Environmental Research Institute of Michigan, Ann
Arbor, MI, and SPIE Press, Bellingham, WA (1993).
3. E. B. Noel, "Radiation from High Pressure Mercury 1\.rcs," Illumination Engineering, Vol. 36,
pg. 243 (1941).
4. The ONTAR Corporation, 129 University Road, Brookline, MA 02146-4532, offers a variety
of atmospheric transmittance codes.
5. F. E. Nicodemus, "Normalization in Radiometry," Applied Optics, Vol. 12(12), pp. 2960-2973
(1973).
6. Burle Electro-Optics Handbook, TP-135, pg. 73, Burle Industries, Inc., Lancaster, PA (1974).
Formerly known as the RCA Electro-Optics Handbook.
3
SOLID STATE ARRAYS
CCD refers to a semiconductor architecture in which charge is transferred
through storage areas. The CCD architecture has three basic functions: (a)
charge collection, (b) charge transfer, and (c) the conversion of charge into a
measurable voltage. A CCD can be attached to a variety of detectors. The basic
building block of the CCD is the metal-insulator-semiconductor (MIS) capacitor.
It is also called a gate. The most important MIS is the metal-oxide-
semiconductor (MOS). Because the oxide of silicon is an insulator, it is a natural
choice.

For monolithic devices, charge generation is often considered as the initial

function of the CCD. With silicon photodetectors, each absorbed photon creates
an electron-hole pair. Either the electrons or holes can be stored and transferred.
For frame transfer devices, charge generation occurs under a MOS capacitor
(also called a photogate). For some devices (notably interline transfer devices)
photodiodes create the charge. The charge created at a pixel site is in proportion
to the incident light level present. The aggregate effect of all the pixels is to
produce a spatially sampled representation of the continuous scene.

Because most sensors operating in the visible region use a CCD architecture
to move a charge packet, they are popularly called CCD arrays. However, the
charge injection device (CID) does not use a CCD for charge transfer. Rather,
two overlapping silicon MOS capacitors share the same row and column
electrode. Pixel readout occurs by sensing the charge transfer between the
capacitors at the pixel site.

Recently, active pixels have been introduced. These devices are fabricated
with complementary-metal-oxide-semiconductor (CMOS) technology. The
advantage is that one or more active transistors can be integrated into the pixel.
As such, they become fully addressable (can read selected pixels) and can
perform on-chip image processing.

Considerable literature 1· 8 has been written on the physics, fabrication, and

operation of CCDs. The charge transfer physics is essentially the same for all
CCD arrays. However, the number of phases and number and location of the
serial shift readout registers vary by manufacturer. The description that follows
should be considered as illustrative. A particular design may vary significantly

45
46 CCD ARRAYS, CAMERAS, and DISPLAYS

from the simplified diagrams shown in this chapter. A specific device may not
have all the features listed or may have additional features not listed.

Although CCD arrays are commonplace, the fabrication is quite complex.

Theuwissen9 provides an excellent step-by-step procedure for fabricating CCDs.
He provides a 29-step procedure supplemented with detailed diagrams. Actual
CCDs may require over 150 different operations. The complexity depends upon
the array architecture.

Devices may be described functionally according to their architecture (frame

transfer, interline transfer, etc.) or by application. To minimize cost, array
complexity, and electronic processing, the architecture is typically designed for
a specific application. For example, astronomical cameras typically use full
frame arrays whereas video systems generally use interline transfer devices. The
separation between general imagery, machine vision, scientific, and military
devices becomes fuzzy as technology advance~.

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

3 .1. PHOTODETECTION

When an absorbed photon creates an electron-hole pair, the photodetection

process has occurred. To be a useful detector in a solid state array, the
photogenerated carrier must be collected in a storage site. The absorption
coefficient is wavelength specific and decreases with increasing wavelength.
Some of the incident photons will be reflected at the array's surface. The
remainder may pass through layers of electrodes and insulators before reaching
the photoactive material. Thus, the absorption characteristics of the overlying
layers, thickness of the photoactive material, and the location of the storage site
determine the spectral quantum efficiency (discussed in Section 4 .1.1., Spectral
Response).

After photodetection, the stored charge is read out. The array architecture
often is optimized for a specific application. General imagery systems are
designed for interlace operation consistent with a standard video (e.g., EIA 170).
Research devices may operate in the progressive scan mode. This reduces some
image distortion when the object is moving. For scientific applications the output
may be in the slow scan mode, which simply means that the data read out rate
is lower than a standard frame rate (e.g., EIA 170).
 SOLID STATE ARRAYS 47

3.1.1. PHOTOGATE

The photogate is a photoactive MOS capacitor where the photogenerated

carriers are stored in a depletion region. The storage capacity depends upon
substrate doping, gate voltage, and oxide thickness. Very large capacities are
possible. Because photogates have overlaying electrodes, the quantum efficiency
in the blue region of the spectrum is reduced.

3.1.2. PHOTODIODE

With photodiodes, a depletion region is formed between the n- and p-type

regions. The photogenerated carriers are usually stored at the junction until
readout. The width of the junction limits the storage capability. The junction is
initially reverse biased by a MOS gate. As the electrons, 1\, are collected, the
voltage drops by

(3-1)

After readout, the voltage is reset to its original value. If the number of
electrons is large, the voltage is reduced to zero and saturation occurs. Excess
photoelectrons will diffuse to neighboring wells and create blooming.
Photodiodes do not have an overlaying structure and therefore have higher
quantum efficiency than comparable photogates.

With a photodiode, a transfer gate is required to move the stored electrons

into the CCD. Devices with photodiodes are often labeled as CCPD for charge-
coupled photodiodes. While transfer gates are essential for CCPDs, they may
also exist in devices with photogates.

The hole accumulation diode (HAD) is a pinned photodiode that offers the
best features of the photogate and photodiode. It offers the high blue response
of the photodiode and large well capacity of the photogate.

3.2. CCD ARRAY OPERATION

Applying a positive voltage to the CCD gate causes the mobile positive holes
in the p-type silicon to migrate toward the ground electrode because like charges
repel. This region, which is void of positive charge, is the depletion region
(Figure 3-1). If a photon whose energy is greater than the energy gap is
48 CCD ARRAYS, CAMERAS, and DISPLAYS

absorbed in the depletion region, it produces an electron-hole pair. The electron

stays within the depletion region whereas che hole moves to the ground
electrode. The amount of negative charge (electrons) that can be collected is
proportional to the applied voltage, oxide thickness, and gate electrode area. The
total number of electrons that can be stored is called the well capacity.

/ Metal Electrode
Oxide Layer
I
1 Depletion I
Bulk Semiconductor
1 Region I
L _ _ _ _I (P-Type Silicon)

Figure 3-1. Metal-oxide-semiconductor (M OS) gate for p-type silicon.

Although the depletion region (charge well) is shown to have an abrupt
transition, its actual shape is gradual. That is, the well is formed by a
two-dimensional voltage gradient.

The CCD register consists of a series of gates. Manipulation of the gate

voltage in a systematic and sequential manner transfers the electrons from one
gate to the next in a conveyor-belt-like fashion. For charge transfer, the
depletion regions must overlap. The depletion regions are actually gradients and
the gradients must overlap for charge transfer to occur. Efficient overlap occurs
when the gate electrodes overlap (Figure 3-2). The multilevel polysilicon
structure minimizes the gap between adjacent electrodes. Without this structure,
charge transfer would be poor. For clarity, throughout the remainder of the text,
the structure will be illustrated as adjacent, non-overlapping gates.

v1
v2 --~----------------.----------------
V3--~----~----~----+-----4-----.----

Si

• Polysilicon

Figure 3-2. Three-phase CCD. The gates must overlap for efficient
charge transfer.
 SOLID STATE ARRAYS 49

Each gate has its own control voltage that is varied as a function of time.
The voltage is called the clock or clocking signal. When the gate voltage is low,
it acts as a barrier whereas when the voltage is high, charge can be stored.
Initially, a voltage is applied to gate 1 and photoelectrons are collected in well
1 (Figure 3-3b). When a voltage is applied to gate 2, electrons move to well 2
in a waterfall manner (Figure 3-3c). This process is rapid and the charge quickly
equilibrates in the two wells (Figure 3-3d). As the voltage is reduced on gate 1,
the well potential decreases and electrons again flow in a waterfall manner into
well2 (Figure 3-3e). Finally, when gate 1 voltage reaches zero, all the electrons
are in well 2 (Figure 3-3f). This process is repeated many times until the charge
is transferred through the shift register.

The CCD array is a series of column registers (Figure 3-4). The charge is
kept within rows or columns by channel stops or channel blocks and the
depletion regions overlap in one direction only. At the end of each column is a
horizontal register of pixels. This register collects a line at a time and then
transports the charge packets in a serial fashion to an output amplifier. The
entire horizontal serial register must be clocked out to the sense node before the
next line enters the serial register. Therefore, separate vertical and horizontal
clocks are required for all CCD arrays. This process creates a serial data stream
that represents the two-dimensional image.
Gate 1 Gate 2
2 2

I ~-

j
·-Ill
I N 111·-
<11-
Qj Qj c:
Q)
I ~ I ~ I t;.!!!
) __ j _ _ j c:
_a. 0

(a) (b) (c)

2 2 2

(d) (e) (f)

Figure 3-3. Charge transfer between two wells. (a) Adjacent wells. (b)
Charge in welll. (c) After a voltage is applied to gate 2, electrons flow
into well 2. (d) Equilibration of charge. (e) Reduction of gate 1 voltage
causes the electrons in well 1 to transfer into well 2. (f) All electrons
have been transferred to well 2.
50 CCD ARRAYS, CAMERAS, and DISPLAYS

Serial Readout Register

(a) {b)

(c) {d)

Figure 3-4. Representative CCD operation of a 3 x 3 array with three

gates per pixel. Nearly all the photoelectrons generated within a pixel
are stored in the well beneath that pixel. (a) The CCD is illuminated and
an electronic image is created. (b) The columns are shifted down in
parallel, one gate at a time. (c) Once in the serial row register, the
pixels are shifted right to the output node. (d) The entire serial register
must be clocked out before the next packet can be transferred into the
serial readout register.
 SOLID STATE ARRAYS 51

Although any number of transfer sites (gates) per detector area can be used,
it generally varies from two to four. With a four-phase device, the charge is
stored under two or three wells depending upon the clock cycle. Figure 3-5
represents the steady state condition after the gates have switched and the charge
has equilibrated within the wells. It takes a finite time for this to happen. Figure
3-6 illustrates the voltage levels that created the wells shown in Figure 3-5. The
time that the charge has equilibrated is also shown. The clock rate is fcLocK =
1/tcwcK and is the same for all phases. The output video is valid once per clock
pulse (selected as T2) and occurs after the charge has equilibrated (e.g., Figure
3-3b or Figure 3-3f). Only one master clock is required to drive the array. Its
phase must be varied for the gates to operate sequentially. An anti-parallel
clocking scheme is also possible. Here v3 is the inverse of VI and v4 is the
inverse of Vz. That is, when VI is high, v3 is low. The anti-parallel clock
system is easier to implement because phasing is not required. With the anti-
parallel scheme, charge is always stored in two wells. For equal pixel sizes,
four-phase devices offer the largest well capacity compared to the two- or three-
phase systems. With the four-phase system, 50% of the pixel area is available
for the well.
52 CCD ARRAYS, CAMERAS, and DISPLAYS

Gates,
'

/ Stored Charge

.....!1.......!1......-r

.......!1.......!1.....
Ts

Figure 3-5. Charge transfer in a four-phase device after equilibration. This

represents one column. Rows go into the paper. Charge is moved through
storage sites by changing the gate voltages as illustrated in Figure 3-6. A
four-phase array requires four separate clock signals to move a charge
packet to the next pixel. A four-phase device requires eight steps to move
the charge from one pixel to the next. (After Reference 10).
 SOLID STATE ARRAYS 53

_ ___....,,

L.....-----1'
Equilibrated
Charge Times
/
c::::Jc::::Jc::::Jc::::Jc::::Jc::::Jc::::Jc::::Jc::::Jc::::J c::::J
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ T3

__ll~--------------~n~-- Valid
Video

Figure 3-6. Voltage levels for a four-phase system. As the voltage is

applied, the charge packet moves to that well. By sequentially varying the
gate voltage, the charge moves off the horizontal shift register and onto the
sense capacitor. The clock signals are identical for all four phases but offset
in time (phase). The valid video is available once per clock pulse (selected
as T2). (After Reference 10).

The three-phase system (Figure 3-7 and Figure 3-8) stores the charge under
one or two gates. Only 33% of the pixel area is available for the well capacity.
With equal potential wells, a minimum number of three phases are required to
clock out charge packets efficiently.
54 CCD ARRAYS, CAMERAS, and DISPLAYS

,.----i,...--------,----!1-- ~:
Gates ',..

1.- Pixel -.1.- Pixel -.1

Figure 3-7. Charge transfer in a three-phase device. This represents one

column. Rows go into the paper. Six steps are required to move the charge
one pixel. (After Reference 10).
 SOLID STATE ARRAYS 55

tCLOCK

V1
l I
v2
I
v3

~ Equilibrated
c::::J c::::J c::::J c::::J c::::J c::::J c::::J c::::J Charge Times
T1 T2 T3 T4 Ts Ts T1 T2

__....~n. . ___________n_ valid

Video

Figure 3-8. Voltage levels for a three-phase system. The clock signals are
identical for all three phases but offset in time. T2 was selected as the valid
video time. Valid video exists only after the charge has equilibrated. (After
Reference 10).

The well potential depends on the oxide thickness underneath the gate. In
Figure 3-9, the alternating gates have different oxide thicknesses and therefore
will create different well potentials. With different potentials it is possible to
create a two-phase device. Charge will preferentially move to the right-hand side
of the pixel where the oxide layer is thinner. Assume initially that the wells
controlled by V2 are empty. This will occur after a few clock pulses. At time
T~> both clocks are low. When V2 is raised, the potential increases as shown at
time T2 • Because the effective potential increases across the gates, the charge
cascades down to the highest potential. Then V2 is dropped to zero and the
charge is contained under the V2 gate at time T3 • This process is repeated until
the charge is clocked offthe array. The voltage timing is shown in Figure 3-10.
While Figure 3-9 illustrates variations in the oxide layer, the potential wells can
also be manipulated though ion implantation. As with the four-phase device, the
two-phase device can also be operated in the anti-parallel mode: V2 is a mirror
image of V1 • That is, when V 1 is high, V2 is low.
56 CCD ARRAYS, CAMERAS, and DISPLAYS

Ga~~:
1--- Pixel --1--- Pixel --1
~-

I
·-
r / ) ·Cll
-
cu-
<D c:
.... .S!
'.E ~

Figure 3-9. Charge transfer in a two-phase device. Well potential

depends on the oxide thickness. (After Reference 10).

tCLOCK

_....-- Equilibrated
E:::J Charge Times
T4

Jl~----~n~-------- Valid
Video

Figure 3-10. Voltage levels for a two-phase system. (After Reference 10).
 SOLID STATE ARRAYS 57

The virtual phase device requires only one clock (Figure 3-11). Additional
charge wells are created by internally biased gates. Charge is stored either
beneath the active gate or the virtual gate. When V1 is low, the charge will
cascade down to the highest potential (which is beneath the virtual well). When
V 1 is applied at T2 , the active gate potential increases and the charge moves to
the active gate well.

v1
Gate"'
I I
G5 ~n )15 lvn J- Virtual Phase

~-
·- <0
C/l·-

l
<0-
T, ~
Q) c:
Q)
go
_a_
- Stored Charge

Figure 3-11. Charge transfer in a device with a virtual phase. The virtual
well is created by p- and n-material implants. These ions create a fixed bias
and therefore a well with fixed potential. By changing V 1, the active gate
potential can be lower than the virtual well (T 1) or higher (T2). This
represents one column. Rows go into the paper.

The final operating step is to convert the charge packet to a measurable

voltage. This is accomplished by a floating diode or floating diffusion. The
diode, acting as a capacitor, creates a voltage proportional to the number of
electrons [discussed in Section 3.5., Charge Conversion (Output Structure)].
With many arrays, it is possible to shift more than one row of charge into the
serial register. Similarly, it is possible to shift more than one serial register
element into a summing gate just before the output node. This is called charge
grouping, binning, super pixeling, or charge aggregation. Binning increases
58 CCD ARRAYS, CAMERAS, and DISPLAJ'S

signal output and dynamic range at the expense of spatial resolution. Because it
increases the signal-to-noise ratio, binning is useful for low light level
applications for those cases where resolution 1s less important. Serial registers
and the output node require larger capacity charge wells for binning operation.
If the output capacitor is not reset after every pixel, then it can accumulate
charge.

3 .3. CCD ARRAY ARCHITECTURE

Array architecture is driven by the application. Full frame and frame transfer
devices tend to be used for scientific applications. Interline transfer devices are
used in consumer camcorders and professional television systems. Linear arrays,
progressive scan, and time delay and integration (TDI) are used for industrial
applications. Despite an ever increasing demand for color cameras, black and
white cameras are widely used for many scientific and industrial applications.

3.3.1. LINEAR ARRAYS

The simplest arrangement is the linear array or single line of detectors.

While these could be photogates, photodiodes are used more often because they
have higher quantum efficiency. Linear arrays are used in applications where
either the camera or object is moving in a direction perpendicular to the row of
sensors. They are used where rigid control is maintained on object motion such
as in document scanning.

Located next to each sensor is the transfer gate and then the CCD shift
register (Figure 3-12a). The shift register is also light sensitive and is covered
with a metal light shield. Overall pixel size is limited by the photodiode size.
For example, with a three-phase system, the pixel width is three times the width
of a single gate. For a fixed gate size, the active pixel width can be reduced
with a bilinear readout (Figure 3-12b). Because resolution is inversely related
to the detector-to-detector spacing (pixel pitch), the bilinear readout has twice
the resolution. For fixed pixel size, the bilinear readout increases the effective
charge transfer efficiency because the number of transfers is reduced by a factor
of 2 (discussed in Section 3.4., Charge Transfer Efficiency).
 SOLID STATE ARRAYS 59

Photo
Diode

(a)

(b)

Figure 3-12. Linear array. (a) Simple structure and (b) bilinear readout.
The transfer gate clock pulse allows charge to move from the photodiode
into the CCD transfer register.

3.3.2. FULL FRAME ARRAYS

Figure 3-13 illustrates a full frame transfer (FFT) array. After integration,
the image pixels are read out line-by-line through a serial register that then
clocks its contents onto the output sense node (see Figure 3-4, page 50). All
charge must be clocked out of the serial register before the next line can be
transferred. In full frame arrays, the number of pixels is often based upon
powers of 2 (e.g., 512 X 512, 1024 X 1024) to simplify memory mapping.
Scientific arrays have square pixels and this simplifies image processing
algorithms.
60 CCD ARRAYS, CAMERAS, and DISPLAYS

Imaging
Area

Figure 3-13. Full frame architecture. Because photogates are used,

transfer gates are not essential.

During read out, the photosites are continually illuminated resulting in a

smeared image. The smear will be in the direction of the charge transport in the
imaging part of the array. A mechanical or external electronic shutter can be
used to shield the array during readout to avoid smear. When using strobe lights
to illuminate the image, no shutter is necessary if the transfer is between strobe
flashes. If the image integration time is much longer than the readout time, then
the smear may be considered insignificant. This situation often occurs with
astronomical observations.

Data rates are limited by the amplifier bandwidth and, if present, the
conversion capability of the analog-to-digital converter. To increase the effective
readout rate, the array can be divided into subarrays that are read out
simultaneously. In Figure 3-14, the array is divided into four subarrays. Because
they are all read out simultaneously, the effective clock rate increases by a
factor of 4. Software then reconstructs the original image. This is done in a
video processor external to the CCD device where the serial data are decoded
and reformatted.

Large area devices often allow the user to select one subarray for readout.
Thus, the user can trade frame rate against image size. This allows the user to
obtain high frame rates over the area of interest (discussed in Section 4.5.,
Frame Rates).
 SOLID STATE ARRAYS 61

Readout 1I Readout H>

Sense Sense
Node T I I
I
I
I I I I I I
I
I
I I I Node
and and
Am plifier Amplifier

Subarray Subarray

Subarray Subarray

I I 'I '
I
I
I
I I I
'
I
'
I
I I
<J-11.------I I. . . __Readout
Readout
_
I

---.~
H>
Sense Sense
Node Node
and and
Amplifier Amplifier

Figure 3-14. A large array divided into four subarrays. Each subarray
is read out simultaneously to increase the effective data rate. Very large
arrays may have up to 32 parallel outputs. The subimages are
reassembled in electronics external to the device to create the full image.

3.3.3. FRAME TRANSFER

A frame transfer (FT) imager consists of two almost identical arrays, one
devoted to image pixels and one for storage (Figure 3-15). The storage cells are
identical in structure to the light sensitive cells but are covered with a metal light
shield to prevent any light exposure. After the integration cycle, charge is
transferred quickly from the light sensitive pixels to the storage cells. Transfer
time to the shielded area depends upon the array size but is typically less than
500 p.s. Smear is limited to the time it takes to transfer the image to the storage
area. This is much less time than that required by a full frame device.
62 CCD ARRAYS, CAMERAS, and DISPLAYS

Image
Array

Shielded
Storage Array

l I I I Sense Node
1 Readout and Amplifier

Figure 3-15. Basic architecture for a full frame CCD. Frame transfer
devices can also have a split architecture similar to that shown in Figure
3-14.

The array may have a few dummy charge wells between the active and
storage areas. These wells collect any anticipated light leakage and their data are
ignored. Some arrays do not have the integral light shield. These arrays can
either be operated in the full frame mode (e.g., 512 x 512) or in the frame
transfer mode (e.g., 256 x 512). Here, it becomes the user's responsibility to
design the light shield for the frame transfer mode.

Example 3-1
SHIELD DESIGN

A camera manufacturer buys a full frame transfer array (512 x 512) but
wants to use it in a frame transfer mode (256 x 512). An engineer designs an
opaque shield that is placed 1 mm above the array. He is using f/5 optics. What
is the usable size of the array if the pixels are 20 pm square and the shield edge
is centered on the array?
 SOLID STATE ARRAYS 63

When placed near the focal plane, the shield edge will create a shadow that
is approximately the distance from the focal plane (d = 1 mm) divided by the f-
number (F = 5). The full shadow is 200 p,m or 10 pixels. One-half of the
shadow falls on the active pixels and one-half illuminates the pixels under the
shield. This results in a usable array size of 251 x 512. An alignment error of
0.5 mm adds an additional25 pixels. If the alignment error covers active pixels,
the active area is reduced to 226 X 512. If the error covers shielded pixels, the
storage area is reduced to 226 x 512. Either way, the usable array is 226 x 512.

=================================================~

Example 3-2
USEFUL ARRAY SIZE

A consumer wants to operate the camera designed in Example 3-1 under

relatively low illumination levels. He changes the lens to one with f-number of
1.25. Will he be satisfied with the imagery?

The lower f-number increases the shadow to 40 pixels. Assuming an

alignment error of 0.5 mm, the minimum usable area size has dropped to 211 x
512 with a "soft edge" of 40 pixels. Acceptability of this image size depends
upon the application.
=============================================~

3.3.4. INTERLINE TRANSFER

The interline transfer (IL) array consists of photodiodes separated by vertical
transfer registers that are covered by an opaque metal shield (Figure 3-16).
Although photogates could be used, photodiodes offer higher quantum efficiency
(discussed in Section 4.1.1., Spectral Response). After integration, the charge
generated by the photodiodes is transferred to the vertical CCD registers in
about 1 p,s and smear is minimized. The main advantage of interline transfer is
that the transfer from the active sensors to the shielded storage is quick. There
is no need to shutter the incoming light. The electronic shutter open time can be
variable if an antibloom drain is present (discussed in Section 3. 8., Antibloom
Drain). The shields act like a Venetian blind that obscures half the information
that is available in the scene. The area fill-factor may be as low as 20%.
Because the detector area is only 20% of the pixel area, the output voltage is
only 20% of a detector that would completely fill the pixel area. Microlenses
increase the optical fill-factor (discussed in Section 3.11.1., Microlenses).
64 CCD ARRAYS, CAMERAS, and DISPLAJ'S

Sense Node
and Amplifier

Figure 3-16. Interline transfer architecture. The charge is rapidly

transferred to the interline transfer registers via the transfer gate. The
registers may have three or four gates. Interline transfer devices can also
have a split architecture similar to that shown in Figure 3-14 (page 61).

The combination of the capacitances associated with the photodiode and

transfer gate provide a time constant that limits the speed of transfer to the
vertical CCD register. As a result not all the charge is read out and some is left
behind. This residual charge is added to the next frame and results in image lag.
That is, with photodiodes, the remaining charge is drained in successive readouts
causing an after-image. But hole accumulation devices can be completely
emptied and no image lag occurs.

A fraction of the light can leak into the vert teal registers. This effect is most
pronounced when viewing an ambient scene that has a bright light in it. For
professional television applications, the frame interline transfer (FIT) array was
developed to achieve even lower smear values (Figure 3-17).
 SOLID STATE ARRAYS 65

Photo Diodes Shielded Vertical

Transfer Registers

--------
1

I
----L---

Shielded Storage Array

Figure 3-17. Frame interline transfer architecture. Both the vertical

transfer registers and storage area are covered with an opaque mask to
prevent light exposure. The transfer gates are not shown.

A variety of transfer register architectures are available. A register can

operate with virtual, two, three, or four phases. The number of phases selected
depends upon the application. Because interline devices are most often found in
general imagery products, most transfer register designs are based upon standard
video timing (discussed in Section 5.2.1., Video Timing). Figure 3-18 illustrates
a four-phase transfer register that stores charge under two gates. With 2:1
interlace, both fields are collected simultaneously but are read out alternately.
This is called frame integration. With EIA 170 (formerly called RS 170), each
field is read every 1/60 s. Because the fields alternate, the maximum integration
time is 1/30 s for each field.
66 CCD ARRAYS, CAMERAS, and DISPLAYS

Shielded Shielded
Vertical Vertical
Pixel Transfer Transfer
Register Register
'\.

Active
Detectors

(a) (b)

Figure 3-18. Detailed layout of the 2:1 interlaced array. (a) The odd
field is clocked into the vertical transfer register and (b) the even field
is transferred. The vertical transfer register has four gates and charge is
stored under two wells. The pixel is defined by the detector center-to-
center spacing and it includes the shielded vertical register area. The
transfer gate is not shown. (After Reference 11).

Pseudo-interlacing (sometimes called field integration) is shown in Figure 3-

19. Changing the gate voltage shifts the image centroid by one-half pixel in the
vertical direction. This creates 50% overlap between the two fields. The pixels
have twice the vertical extent of standard interline transfer devices and therefore
have twice the sensitivity. An array that appears to have 240 elements in the
vertical direction is clocked so that it creates 480 lines. However, this reduces
the vertical MTF (discussed in Section 8.3.2., Detector Array Nyquist
Frequency). With some devices, the pseudo-interlace device can also operate in
a standard interlace mode.
 SOLID STATE ARRAYS 67

Shielded Shielded
Vertical Vertical
Pixel Transfer Transfer
Register Register
'\

Active
Detectors

(a) (b)

Figure 3-19. Pseudo-interlace. By collecting charge from alternating

active detector sites, the pixel centroid is shifted by one-half pixel. (a)
Odd field and (b) even field. The detector numbers are related to the
"scan" pattern shown in Figure 3-20b. The transfer gate is not shown.
(After Reference 11).

While any architecture can be used to read the charge, the final video format
generally limits the selection. Although the charge is collected in rows and
columns, the camera reformats the data into a serial stream consistent with the
monitor requirements. This serial stream is related to a fictitious "scan" pattern.
Figure 3-20 illustrates the video lines and field-of-view of a standard format
imaging system. Figure 3-20a represents the "scan" pattern created by the
interlace device shown in Figure 3-18 whereas Figure 3-20b represents the
pseudo-interlace "scan" pattern of the device shown in Figure 3-19. Although
the dissected scene areas overlap in Figure 3-20b, the video timing is consistent
with the video format.
68 CCD ARRAYS, CAMERAS, and DISPLAYS

1
----- 2----- ...
3
}odd
••
----·484
••
____ .,.. }odd
485 (b)

(a)

Figure 3-20. Fictitious "scan" patterns. The scan patterns help to

visualize how the scene is dissected by the array architecture. (a)
Interlace "scanning" achieved with the array shown in Figure 3-18. The
solid lines are the odd field and the dashed lines are the even field. (b)
Pseudo-interlace "scanning" provided by the array shown in Figure 3-19.

3.3.5. PROGRESSIVE SCAN

CCD arrays do not scan the image, but it is convenient to represent the
output as if it were scanned. Progressive scan simply means the noninterlaced
or sequential line-by-line scanning of the image. This is important to machine
vision because it supplies accurate timing and has a simple format. Any
application that requires digitization and a computer interface will probably
perform better with progressively scanned imagery.

The primary advantage of progressive scan is that the entire image is

captured at one instant of time. In contrast, interlaced systems collect fields
sequentially in time. Any vertical image movement from field-to-field smears the
interlaced image in a complex fashion. Horizontal movement serrates vertical
lines. With excessive movement, the two fields will have images that are
displaced one from the other (Figure 3-21).
 SOLID STATE ARRAYS 69

If a strobe light is used to stop motion, the image will appear only in the
field that was active during the light pulse. Because of image motion effects,
only one field of data can be used for image processing. This reduces the
vertical resolution by 50% and will increase vertical image aliasing.

(a) (b) (c)

Figure 3-21. Effects of movement. (a) Character moving horizontally,

(b) serration that occurs with an interlace system, and (c) output of a
progressively scanned array. An image can still be smeared with
progressive scanning but it won't be serrated.

Because an entire frame is captured, progressive scan devices do not suffer

from same image motion effects seen with interlaced systems. Thus, progressive
scan cameras are said to have improved vertical resolution over interlaced
systems. Full frame and frame transfer devices inherently provide progressive
scan readout. Although early interline transfer architecture was designed for
interlace scanning, some current devices also provide progressive scan output.

Although the imagery is collected in a sequential method, camera electronics

convert progressively scanned data into a standard video format (discussed in
Section 5.2.2., Broadcast/Non-Broadcast Design). This allows the progressively
scanned camera output to be viewed on any standard interlaced display. For
scientific and industrial applications, the progressive output is captured by a
frame grabber. Once in the computer, algorithms automatically reformat the data
into the format required by the monitor. The camera pixel layout and array
timing do not need to fit a specified format for these applications.
70 CCD ARRAYS, CAMERAS, and DISPLAYS

3.3.6. TIME DELAY and INTEGRATION

Time delay and integration (TDI) is analogous to taking multiple exposures

of the same object and adding them 12 • The addition takes place automatically in
the charge well and the array timing produces the multiple images. Figure 3-22
illustrates a typical TDI application. A simple camera lens inverts the image so
the image moves in the opposite direction from the object. As the image is
swept across the array, the charge packets are clocked at the same rate. The
relative motion between the image and the target can be achieved in many ways.
In airborne reconnaissance, the forward motion of the aircraft provides the
motion with respect to the ground. With objects on a conveyor belt, the camera
is stationary and the objects move. In a document flat bed scanner, the document
is stationary but either the camera moves or a scan mirror moves.

Figure 3-23 illustrates four detectors operating in TDI mode. The charge
increases linearly with the number of detectors in TDI. Shot noise also
increases, but as the square root of the number of TDI elements, NTDI·
Theoretically, this results in an SNR improvement of v'NTm· The well capacity
limits the maximum number of TDI elements that can be used.
Multiple
Sensor
Nodes

Moving Object
Conveyor Belt

Figure 3-22. Typical TDI operation. The pixel clock rate must be
matched to the image velocity.
 SOLID STATE ARRAYS 71

For this concept to work, the charge packet must always be in synch with
the moving image. If there is a mismatch between the image scan velocity and
pixel clock rate, the output is smeared and this adversely affects the in-scan
MTF (discussed in Section 10.3.3., TDI). The accuracy to which the image
velocity is known limits the number of useful TDI stages.

- Moving Ball on
Conveyor Belt

••

••
•
••
••
Figure 3-23. TDI concept. At time T 1, the image is focused onto the
first detector element and creates a charge packet. At T2 the image has
moved to the next detector. Simultaneously, the charge packet is clocked
down to the next pixel site. Here, the image creates additional charge
that is added to the stored packet (created by the first detector). After
four stages, the signal increases fourfold.
72 CCD ARRAYS, CAMERAS, and DISPLAYS

Although Figure 3-22 illustrates two readout registers, an array may have
multiple readout registers. Array sizes may be as large as 2048 by 32 TDI
elements. Additional readouts are required to avoid charge well saturation. The
multiple readout outputs may be summed in electronics external to the TDI
device or may be placed in a horizontal serial register for summing (Figure 3-
24).

Horizontal Shift Readout

r---t> Sense Node
and Amphfrer

t t t
I I I
"S "S "S "S
0
0 0 0
"C "C "C "C
<tl <tl <tl <tl
Ql Ql Ql Ql
a: a: a: a:
(ij (ij (ij (ij
0
'E
Ql
0
'E
Q)
·-e0
Q)
0
'E
Q)
> > > >

~----------- Nrol --------------

Figure 3-24. The array height must match the image height. To avoid
saturation, multiple vertical register readouts may be used.

The vertical readout register must clock out the data before the next line is
loaded into the register. The amplifier bandwidth or analog-to-digital conversion
time limits the pixel rate. For example, if the amplifier can support 107 pixels/s
and the register serves 500 vertical pixels, then the maximum (in-scan) line rate
is 20,000 lines/s. If the amplifier can only handle 2 x 106 pixels/sand the line
rate is maintained at 20,000 lines/s, then the vertical transfer register must be
divided into five registers - each connected to 100 pixels. These registers and
associated outputs operate in parallel (Figure 3-25).
 SOLID STATE ARRAYS 73

Serial Readout
Image Motion Registers
... I

Vertical
Extent
of Image

- Nn>l-
Figure 3-25. The vertical direction may be divided into subarrays to
increase the line rate. The transfer register typically is a three-phase
device.

TDI improves responsivity nonuniformity (discussed in Section 4.2.6.,

Pattern Noise). Variations in individual detector responsivities in the TDI
direction are averaged by 1/NTDI. Multiple outputs are used to increase the
effective pixel rate. Each output is amplified by a separate amplifier that has
unique gain characteristics. If the gains are poorly matched, the image serviced
by one amplifier will have a higher contrast than the remaining image. In TDI
devices, streaking occurs when the average responsivity changes from column-
to-column. 13 Careful selection of arrays will minimize these nonuniformities.
They can also be minimized though processing techniques employed external to
the device.
74 CCD ARRAYS, CAMERAS, and DISPLAYS

~=============================================
Example 3-3
TDI LINE RATE

The pixels in a TDI are 151'm square. The object is moving at 2 m/s and the
object detail of interest is 150 I'm. What is the lens magnification and what is
the pixel clock rate?

Each pixel must view 150 I'm on the target. Because the pixel size is 151'm,
the lens magnification is Mop-ncs = 15/150 = 1110. The TDI pixel clock rate is
(2 m/s)/150 I'm= 133Khz.
==============================================~

3.4. CHARGE TRANSFER EFFICIENCY

The early CCDs transferred electrons in channels that were at the surface.
Surface defects interfere with the transfer and create poor charge transfer
efficiency. These were SCCD or surface channel CCDs. By burying the channel
(BCCD), the charge was no longer affected by surface defects and this improved
the CTE significantly. Nearly all CCDs use buried channel technology today.

As the charge packet moves from storage site to storage site, a few electrons
are left behind. The ability to transfer all the charge is given by the CTE. As
the number of transfers increases, the charge transfer efficiency must also
increase to maintain signal strength.

The number of transfers across the full array is the number of pixels
multiplied by the number of gates. For example, for an array that is 1000 x
1000, the charge packet farthest from the sense node must travel 2000 pixels.
If four gates (four-phase device) are used to minimize cross talk, the farthest
charge packet must pass through 8000 wells (neglecting isolation pixels). The
net efficiency varies with the target location. If the target is in the center of the
array, then it only experiences one-half of the maximum number of transfers on
the array. If the target is at the leading edge, it is read out immediately with
virtually no loss of information. The loss of information is described by the CTE
MTF (discussed in Section 10.3.2., Charge Transfer Efficiency).

If many transfers exist (e.g., 8000) the transfer efficiency must be high
(e.g., 0.999995) so that the net efficiency is reasonable (0.99999SSOOO = 0.96).
Arrays for consumer applications typically have charge transfer efficiencies
greater than 0.9999 and CTEs approach 0.999999 for scientific grade devices.
 SOLID STATE ARRAYS 15

A fraction of the charge, as specified by the charge inefficiency, is left

behind. After the first transfer, E (E is the CTE) remains in the leading well and
(1-E) enters the first trailing well. After the second transfer, the leading well
transfers E of the previous transfer and loses (1-E) to the trailing well. The
trailing well gains the loss from the first well and adds it to the charge
transferred into the trailing well in the last go-around. Charge is not lost, just
rearranged. The fractional values follow a binomial probability distribution:

PR - (~)eR(l- e)N-R, (3-2)

where N is the number of transfers. For the first well, R = N. For the second
well, R = N - 1 and so on.

Figure 3-26 illustrates the output when a large number of pixels are
illuminated. The leading edge loses charge and decreases in value. For pixels
near the end of the pulse, the charge lost to trailing wells is replaced by the
charge acquired from leading wells. Here, the signal strength is maintained.
Finally, charge is lost to wells after the signal and the pulse width is stretched.

Ie adi ng trailing
- r-

r-

n (b)
Inn.
trailing

(a)

,
lr1
(c)

Figure 3-26. Input and output signals after many transfers. (a) Initially
generated charge, (b) charge distribution with low CTE, and (c) charge
distribution with high CTE.
76 CCD ARRAYS, CAMERAS, and DISPLAYS

Consider a very small spot of light that illuminates just one pixel and creates
no electrons. A well, N transfers away, will contain nN electrons. The fractional
amount of charge left is

(3-3)

The fractional amount of charge appearing in the first trailing well after N
transfers is

n(jirsttrailingwell)-N- NeN-1(1-e). (3-4)

no
The fractional signal strength, nNIDo, is plotted in Figure 3-27. The first
trailing well fractional strength, nfirsttrailing weii-N/no, is plotted in Figure 3-28. Most
of the electrons lost from the first well appear in the second and third wells. For
high CTE arrays, a negligible amount appears in successive wells.

...J
0.999999
<
z 0.99999
(!) 0.8
Ci)
...J 0.6 0.9999
<
z
0 0.4
i=
0 0.2
~
u.. 0
0 1000 2000 3000 4000
NUMBER of TRANSFERS
Figure 3-27. Fractional signal as a function of transfer efficiency and
number of transfers.
...J
...J 0.4 0.999

~ 0.3
(!)Z
zQ
;;;it) 0.2
~~
1-u..
....en 0.1
a::
u::
1000 2000 3000 4000
NUMBER of TRANSFERS
Figure 3-28. Fractional signal for the first trailing charge well. With low
CTEs, as the number of transfers increases, more charge spills into the
second trailing well. The charge in the first trailing well decreases.
 SOLID STATE ARRAYS 77

CTE is not a constant value but depends upon the charge packet size. For
very low charge packets, the CTE decreases due to surface state interactions.
This is particularly bothersome when the signal-to-noise ratio is low and the
array is very large (typical of astronomical applications). The CTE also
decreases near saturation due to charge spill effects.

The fraction of charge left behind also depends upon the clocking frequency.
At high frequencies, efficiency is limited by the electron mobility rate. That is,
it takes time for the electrons to move from one storage site to the next. Thus,
there is a tradeoff between frame rate (dictated by clock frequency) and image
quality (affected by CTE). Using smaller pixels partially overcomes the
limitation imposed by the electron mobility rate. However, a smaller pixel has
a smaller charge well capacity.

If the surface states significantly affect CTE, then the fat zero technique
helps. Fat zero puts a known amount of electrons into the well. These electrons
will fill the interface states and thereby improve the CTE for the photoelectrons.
Because the fat zero electrons partially fill the charge well, the remaining well
capacity for the photoelectrons is reduced.

CTE effects in large linear arrays can be minimized by using alternate

readouts. In Figure 3-29, the alternate readouts reduce the number of transfers
by a factor of2. Adjacent pixels experience the same number of transfers so that
the signal is preserved at any particular location.

I
Shift Register

Detectors I I I l 11 I I I I I I I I I

I Shift Register

Figure 3-29. For long arrays, the MTF can be increased by alternating
the readouts or by reading out subarrays (see Figure 3-14, page 61).
78 CCD ARRAYS, CAMERAS, and DISPLAYS

There are three methods to measure CTE: current, optical, and x-ray
injection. Because most devices do not have current injection terminals, the
optical and x-ray injection methods are used more often. With the optical
technique, either a row of detectors or a single detector is illuminated. The ratio
of the output of the first illuminated pixel to the average value of trailing pixels
is measured. The CTE is found using Equation 3-3. The optical technique
requires a spot 14 whose diameter is less than a pixel width. This can only be
achieved with carefully designed optics that are critically aligned. But x-rays 1
generate a known number of electrons in a localized area that is much smaller
than a pixel. The x-ray technique probably provides the highest accuracy.

3.5. CHARGE CONVERSION (OUTPUT STRUCTURE)

Charge is converted to a voltage by a floating diode or floating diffusion.

Figure 3-30 illustrates the last two gates of a four-phase system. The low voltage
gate allows charge transfer only when gate 4 goes to zero. The diode, acting as
a capacitor, is precharged at a reference level. The capacitance, or sense node,
is partially discharged by the amount of charge transferred. The difference in
voltage between the final status of the diode and its precharged value is linearly
proportional to the number of electrons, n.. The signal voltage after the source
follower is
Gq (3-5)
VSIGNAL - V RESET - V OUT - ne -
c
The gain, G, of a source follower amplifier is approximately 1 and q is the
electronic charge (1.6 x 10-19 caul). The charge conversion is q/C. The output
gain conversion or OGC is Gq/C. It typically ranges from 0.1p.V/e- to 10 p.V/e-.
The signal is then amplified and processed by electronics external to the CCD
sensor.
 SOLID STATE ARRAYS 79

r VREF

~Reset Ve

Gate Gate Low

3 4 Voltage

I Vout

Figure 3-30. Output structure. The charge on the floating diffusion is

converted to a voltage by the diffusion capacitance. The low voltage
gate, which operates in a manner similar to a virtual well, allows charge
transfer when the voltage on gate 4 is zero (see Figure 3-11, page 57).

3.6. DARK CURRENT

The CCD output is proportional to the exposure, Lq(A)trNT· The output can
be increased by increasing the integration time and long integration times are
generally used for low light level operation. However, this approach is
ultimately limited by dark current leakage that is integrated along with the
photocurrent. With a large pixel (24 llm square), a dark current density of 1000
pA/cm2 produces 36,000 electrons/pixel/s. If the device has a well capacity of
360,000 electrons, the well fills in 10 s. The dark current is only appreciable
when t1NT is long. It is not usually a problem for consumer applications but is a
concern for scientific applications.

A critical design parameter is dark current noise reduction. There are three
main sources of dark current: (1) thermal generation in the depletion region, (2)
thermal generation and diffusion in the neutral bulk material, and (3) thermal
generation due to surface states. Dark current densities vary significantly among
manufacturers with values ranging from 0.1 nA/cm2 to 10 nA/cm2 in silicon
CCDs. Dark current due to thermally generated electrons can be reduced by
cooling the device. Surface state dark current is minimized with multi-phase
pinning 15 •16 (MPP).
80 CCD ARRAYS, CAMERAS, and DISPLAYS

The number of thermally generated dark current electrons is

JDAD tiN7' (3-6)
nDARK - - '
q
where J 0 is the dark current density. For thennal generation in the depletion
region, It is proportional to
(Ea-ET)
----- (3-7)
JD a rZe kT

where Tis the absolute temperature, &, is the band gap, and Er is the impurity
energy gap. With TDI, the pixels in a TDI column are summed and the dark
current increases:
JDADtiNTN1DI (3-8)
nDARK- q ---

In principle, dark current density can be made negligible with sufficient

cooling. The dark current density decreases approximately twofold for every 8
to 9°C drop in array temperature from ambient. Conversely, the dark current
density increases by a factor of 2 for every 8 to 9°C increase in array
temperature. Because the temperature is in the exponent of Equation 3-7, the
doubling temperature decreases as the array temperature decreases. At low
temperatures, a 1oc change in temperature changes the dark current by a
significant amount. This means that if cooling is desired, the temperature must
be maintained with high precision. Table 3-1 provides the dark current for Texas
Instruments chip TC271 using experimentally 17 obtained values of&,- Er : : : : 0.58
eV and J 0 = 100pA/cm2 at 27°C.

The most common cooler is a thermoelectric cooler (TEC or TE cooler).

Thermoelectric coolers are Peltier devices driven by an electric current that
pumps heat from the CCD to a heat sink. The heat sink is cooled by passive air,
forced air, or forced liquid (usually water). The final array temperature depends
upon the amount of heat generated in the array, the cooling capacity of the
thermoelectric device, and the temperature of the TEC heat sink. Peltier devices
can cool to about -60°C. Thermoelectric coolers can be integrated into the same
package with the array (Figure 3-31). Cooling probably is only worthwhile for
scientific applications. However, many devices contain TE coolers to stabilize
the temperature so that the dark current does not vary with ambient temperature
changes.
 SOLID STATE ARRAYS 81

Table 3-1
DARK CURRENT VALUES for T1 TC271
(Normalized to 100 pA/cm2 at 27°C)
TEMPERATURE DARK PERCENT CHANGE DOUBLING
(oC) CURRENT for lo C TEMPERATURE
(pA/cm2)

60 1136 6.9 10.8

40 276 7.7 9.5
20 55.8 8.9 8.4
0 9.02 10.2 7.2
-20 1.11 11.9 6.2
-40 9.58 x w-z 14.1 5.3
-60 5.33 x w-3 17.0 4.4
-80 1.66 X 104 20.9 3.6

Window

Figure 3-31. Cross section of a two stage TEC integrated into a package.

Although liquid nitrogen (LN, LN2, or LN2) is at -200°C, the typical

operating temperature is -l20°C to -60°C because the CTE and quantum
efficiency drop at very low temperatures. Condensation is a problem and the
arrays must be placed in an evacuated chamber or a chamber that is filled with
a dry atmosphere. Dark current can approach 18 3.5 electrons/pixel!s at -60°C and
0.02 electrons/pixel!h at -1200C. Various cooling techniques and fabrication
concepts are provided in Reference 19.
82 CCD ARRAYS, CAMERAS, and DISPLAYS

Multi-pinned phasing technology CCDs can operate at room temperature

with reduced dark current. 15 •16 MPP devices have dark current density levels as
low as 2 pA/cm2 with a high value of 1 nA/cm2 • Typically, MPP devices have
charge wells that are two to three times smaller than conventional devices and
these devices are more likely to saturate. Some devices operate in either mode.
MPP is the most common architecture used in scientific and medical
applications.

The output amplifier constantly dissipates power. This results in local

heating of the silicon chip. Because dark current is dependent upon temperature,
a small variation in the device temperature profile produces a corresponding
profile in the dark current. To minimize this effect, the output amplifier is often
separated by several isolation pixels to move the amplifier away from the active
sensors.
~=============================================
Example 3-4
DARK CURRENT

A CCD array has a well size of 150,000 electrons, dark current density of
10pA/cm2 , and detector size 8~tm square. The advertised dynamic range is 80
dB. Is the dark current noise significant? Assume that the integration time is
1160 s (16.67 ms).

Using Equation 3-6, the number of dark electrons is

10 X 10- 9 __:!__) (0.0167 S) (8 X 10- 4cm)2

( cm 2
nDARK = --'------'----- ·---- -668e-. (3-9)
1.6 X lQ-19 COU[ _
electron

Assuming Poisson statistics, the dark current shot noise is ....{668 or 26 electrons
rms. The dynamic range is the charge well capacity divided by the noise floor.
With a dynamic range of 10,000:1, the noise tloor is 15 electrons. Because the
noise variances add, the total noise is:

(nSYS) = V262+ 152 = 30 e- rms . (3-10)

Therefore, the dark current will affect the measured SNR at low levels. The
dark current will saturate the well in 3.75 s.
=============================================~
 SOLID STATE ARRAYS 83

Example 3-5
ARRAY COOLING

The dark current density decreases by a factor of 2 for every soc drop in
temperature. What should the array temperature be so that the dark current noise
is 5 electrons rms for the array described in Example 3-4?

The dark current noise must be reduced by a factor of 5/26 = 0.192 and the
dark current must be reduced by (0.1921 or 0.0369. The doubling factor
approximately provides

nDARK-COOL (- TAMBIE~- TCOOL) •

(3-11)
- 2
nDARK-AMBIENT

Then, TAMBIENT- TcooL = 38.1 oc. That is, the array temperature must be 38.1 oc
below the temperature at which the dark current density was originally
measured. Because the dark current is reduced by 0.0369, the time to saturation
increased by 1/0/0369 or 27.1 times to 102 s.

Although this varies by manufacturer, many use TAMBIENT = 20°C. Note that
this is an approximation only and Equation 3-7 should be used to determine the
required temperature. In reality, the user does not precisely select the
temperature but, if possible, cools the array well below the desired temperature.
The lowest possible temperature is limited by the cooling device.

=============================================~

3.7. DARK PIXELS

Many arrays have extra photosites at the end of the array. These shielded
II II

photosites are used to establish a reference dark current level (or dark signal).
The number of dark elements varies with device and manufacturer and ranges
from a few to 25. The average value of the dark current pixels is subtracted
from the active pixels leaving only photogenerated signal. For example, if the
output of the dark pixels is 5 mV, then 5 mV is subtracted from the signal level
of each active pixel.

Figure 3-32a illustrates the output from a single charge well. Light leakage
may occur at the edge of the shield (see Example 3-1, page 62) and partially fill
a few wells. Because of this light leakage, a few pixels (called dummy or
84 CCD ARRAYS, CAMERAS, and DISPLAYS

isolation pixels) are added to the array. Figure 3-32b illustrates the output of
three dark reference detectors, three isolation detectors, and three active pixels.
The average value of the three dark pixels is subtracted from every active pixel
value.

Dark Current
-<D Vsignal
::I Cl
0.(1:1
Photo Electrons
s=
og Vout
0-+----.....__ _ _ _ __
(a)

Dark Current
Reference Pixels Active Pixels

Light Leakage

(b)
Figure 3-32. Typical output. (a) Single pixel and (b) active line with
isolation and dark pixels. The varying voltage on the active pixels is
proportional to the scene intensity. The dark pixels are covered with an
opaque mask. Light can leak around the mask and partially fill the isolation
pixel wells.

While this process is satisfactory for general imagery and machine vision
use, it may be unacceptable for scientific applications. Dark pixels have slightly
different dark current than the active pixels. Furthermore, the dark current value
varies from pixel to pixel. This variation appears as fixed pattern noise
(discussed in Section 4.2.6., Pattern Noise). Removing the average value does
not remove the variability.

In critical scientific applications, only the dark value from a pixel may be
removed from that pixel. That is, the entire array is covered (mechanical shutter
in place) and the individual pixel values are stored in a matrix. Then the shutter
is removed and the stored values are subtracted, pixel by pixel, as a matrix
subtraction. This method ensures that the precise dark voltage is subtracted from
each pixel. This, of course, increases computational complexity. Removal of
dark current values allows maximum use of the analog-to-digital converter
 SOLID STATE ARRAYS 85

dynamic range. This method cannot remove the dark current shot noise. Dark
current shot noise is reduced by cooling.

Additional isolation pixels may exist between the readout register and the
sense node amplifier. These pixels reduce any possible interaction between the
amplifier temperature and first column pixel dark current. They also reduce
amplifier noise interactions.

3.8. ANTIBLOOM DRAIN

When a well fills, charge spills over into adjacent pixels in the same column
resulting in an undesirable overload effect called blooming. Channel stops
prevent spill over onto adjacent columns. Overflow drains or antibloom drains
prevent spill over. The drain can be attached to every pixel or may only operate
on a column of pixels. Any photoelectron that is swept into the drain is instantly
removed. Thus, the location of the drain is important.

If the drain is buried, the long wavelength quantum efficiency is reduced

because long wavelength charge is generated below the wells and near the buried
drain. A buried drain is also called a vertical overflow drain (VOD). Here,
vertical refers to the depth of the device whereas the horizontal and vertical
directions define the pixel size. If the drain is next to the detector, it is a lateral
antibloom drain (Figure 3-33). This takes up real estate and lowers the fill-
factor. .

Figure 3-34 illustrates the potential well created by a lateral drain. When the
charge well fills over the drain barrier, charge flows into the drain. The barrier
height can be controlled by an external voltage so that variable integration times
are possible. Here, the antibloom drain is used for exposure control. The
exposure control can work at any speed. The drain, acting as an electronic
shutter, may be used synchronously or asynchronously. Asynchronous shuttering
is important for use in processes where the motion of objects does not match the
array frame timing or in high light level applications. The charge well is
drained, reset, and then photoelectrons are integrated for the desired time.
86 CCD ARRAYS, CAMERAS, and DISPLAYS

Transfer
Gate

Lateral Shielded
Anti-Bloom Transfer
Drain Register

Figure 3-33. Lateral antibloom drain. The transfer gate controls the
desired integration time.

White
Clip
Level

Well Drain Well Drain

Figure 3-34. Lateral overflow drain. When the number of stored electrons
exceeds the saddlepoint, the excess electrons flow into the drain

In a perfect system, the output is linearly related to the input up to the

antibloom drain limit. In real arrays, because of imperfect drain operation, a
knee is created20 (Figure 3-35). The response above the knee is called knee slope
or knee gain. While scientific cameras operate in a linear fashion, there is no
requirement to do so with consumer cameras. The knee is a selling feature in
the consumer marketplace. The advantage of the knee is that it provides some
 SOLID STATE ARRAYS 87

response above the white clipping level. Each CCD manufacturer claims various
competitive methods of handling wide scene contrasts. These methods usually
involve manipulating the drain potential.

Interline transfer devices may have the drain at the top of a vertical shift
register. The excess charge cannot be drained while image information is in the
vertical shift registers (Figure 3-36). This limits the range of integration times
available. Figure 3-37 illustrates the variable integration afforded by the drain.

1 _White Clip~v~

--
::I
c.
::I
1 ""'-Knee

I
Point

0
I
I

Input Light Intensity -

Figure 3-35. Characteristic knee created by an overflow drain.

88 CCD ARRAYS, CAMERAS, and DISPLAJ:S

Drain
Drain

To Sense Node

(a) (b)

Figure 3-36. Interline transfer device where the drain is at the top of the
vertical shift register. (a) Unwanted charge is sent to the drain and (b)
desired charge is read out.

Field Time
.. I
Transfer _ _ _ _ _ _ _ _ _ _ __..n.___ _.ll_
Gate
1+-----"Shutter"_ _ __..,+,...,_"Shutter" _I
Closed Open ~

Figure 3-37. By judiciously activating the transfer gate, any integration

time can be achieved. A variable integrat 10n time is equivalent to a
variable electronic shutter.
 SOLID STATE ARRAYS 89

3.9. CID

A charge injection device consists of two overlapping MOS capacitors

sharing the same row and column electrode. Figure 3-38 illustrates the pixel
architecture. Physically, the capacitors are orthogonal. Charge is stored under
the horizontally or vertically oriented capacitor. The nearly contiguous pixel
layout provides a fill-factor of 80% or greater.

Column Select

Figure 3-38. CID pixel architecture.

Although the capacitors are physically orthogonal, it is easier to understand

their operation by placing them side-by-side. Figure 3-39 illustrates a functional
diagram of an array and Figure 3-40 illustrates the pixel operation. In Figure 3-
40a, a large voltage is applied to the columns and photogenerated carriers
(usually holes) are stored under the column gate. If the column voltage is
brought to zero, the charge transfers to the row gate (Figure 3-40b). The change
in charge causes a change in the row gate potential which is then amplified and
outputted. If V1 is reapplied to the columns, the charge transfers back to the
column gate. This is non-destructive readout and no charge is lost. Now
additional charge integration is possible. Reset occurs by momentarily setting the
row and column electrodes to ground. This injects the charge into the substrate
(Figure 3-40c).

The conversion of charge into voltage depends upon the pixel, readout line,
and amplifier capacitance. The capacitance is high because all the pixels on a
given row are tied in parallel. Therefore, when compared to a CCD, the charge
conversion is small, yielding a small signal-to-noise ratio. Because the readout
is nondestructive, it can be repeated numerous times. The multiple reads are
averaged together to improve the signal-to-noise ratio.
90 CCD ARRAYS, CAMERAS, and DISPLAYS

Figure 3-39. Functional layout of a 3 x 3 CID array. The row and

column select registers are also called decoders. The select registers and
readout electronics can be fabricated with CMOS technology.

Column Row Column Row Column Row

v1 v2 v3 v2 v3 v3
-I- - I- -I- -I- -I- -I-

~
l..-1 (a) (b)
v1 > v2
I
> v3 *
(c)
\

Figure 3-40. CID pixel operation. (a) Integration, (b) readout, and (c)
injection.
 SOLID STATE ARRAYS 91

Because each pixel sees a different capacitance, CIDs tend to have higher
pattern noise compared to CCDs. However, off-chip algorithms can reduce the
amount of pattern noise. With no charge transfer, CIDs are not sensitive to CTE
effects. Without multiple gates, CIDs have larger well capacities than
comparably sized CCDs. CIDs inherently have antibloom capability. Because
charge is limited to a single pixel, it cannot overflow into neighboring pixels.
Perhaps the greatest advantage of CIDs is random access to any pixel or pixel
cluster. Subframes and binned pixels can be read out at high frame rates.

3.10. CMOS

The main advantage of CMOS imagers is that they are compatible with
mainstream silicon chip technology. Because transistors are made by this
technology, on-chip processing is possible. 21 An active pixel sensor (APS) has
one or more active transistors integrated into the pixel (Figure 3-41).

Column Select

Row------------------------.---~~
Select _______________ _
I

(a)

~- ;ix~l/------------ Output

Column Select
Row
Select -----------------------~--___;~-
~--------------- -.
(b)
1 Voo

,.. I
L------------------
PiXel / Output

Figure 3-41. (a) Passive pixel device and (b) active pixel sensor. CCDs
and CIDs are regarded as passive pixel sensors. CIDs use photogates,
CCDs use either photogates or photodiodes, and CMOS devices typically
use photodiodes.
92 CCD ARRAYS, CAMERAS, and DISPLAYS

With the APS approach, highly integrated image sensors are possible. 22 By
placing processing on the chip, a CMOS camera is typically physically smaller
than a CCD camera which requires clocks, image reformatting, and signal
processing in separate hardware. A sophisticated APS array could create 23 a
"camera-on-a-chip." It is possible to build a frame transfer device where pixels
can be binned to enhance the SNR and provide variable resolution imaging. 24

In 1993, Fossum25 described state-of-the-art active pixel concepts. At that

time they were the double-gate floating surface transistor, charge modulation
device, bulk charge modulation device, based-stored image sensor, and static
induction transistor. That review continued23 in 1997.

The APS contains a photodiode, row-select transistor, and a reset transistor.

As shown in Figure 3-41, by activating a row, the data from the pixels in that
row are simultaneously copied into the columns. Each column will have a load
transistor, column select switch, and a sampling switch. The chip may also have
an analog-to-digital converter and provide correlated double sampling. The
pixels are then reset and a new integration is started. The APS does not rely
upon charge transfer. Rather the photodiode drives a capacitance line. The total
capacitance limits the chip speed. APSs can be fully addressable and subarrays
can be read out at high frame rates just like CIDs.

CMOS devices tend to have higher dark currents due to the highly doped
silicon used. As such it does not appear that CVIOS sensors will replace CCDs
in low noise scientific applications. Because the active devices often take up real
estate, the area for the photosensor is reduced. This leads to reduced sensitivity
that can be partially offset by a microlens. Because each pixel has its own
amplifier, pattern noise is larger. However, more logic can be added to each
pixel and it seems possible to use on-chip signal processing that suppresses
FPN. 26 With charge limited to a single pixel, it cannot overflow into neighboring
pixels and create blooming as seen with CCDs.

While more integration can lead to the camera-on-a chip, production yield
will probably decrease as the number of transistors increase. Consider an array
that contains 640 x 480 pixels (307 ,200 pixels). With a CCD, there are 307,200
elements on the chip (neglecting dark pixels and the readout structure). With an
APS that has a photodiode and three transistors per pixels there are four times
more elements (1 ,228,800 total). If any one of the four elements fails, that pixel
is dead.

A further advantage is their low power consumption. APSs operate from 5-

volt supply or less (compared to 10 V to 15 V for a CCD). This is a very
 SOLID STATE ARRAYS 93

attractive feature for man-portable, battery-operated devices. It appears at this

time, CMOS will compete with CCDs in the general video marketplace where
weight, size, and power consumption are factors.

3.11. PHYSICAL PARAMETERS

All detector arrays are fabricated for specific applications. Microlenses are
used to increase the optical fill factor. Color filter arrays employ filters that are
placed over the detectors to create specific spectral responses that lend
themselves to color imagery. For general video applications, the number of
detectors is matched to the video standard to maximize bandwidth and image
sharpness. The overall array size follows the convention used with vidicons.

3 .11.1. MICROLENSES

Optical fill factor may be less than 100% due to manufacturing constraints
in full transfer devices. In interline devices, the shielded vertical transfer register
can reduce the fill factor to less than 20%. Micro lens assemblies (also called
microlenticular arrays or lenslet arrays) increase the effective optical fill-factor
(Figure 3-42). But it may not reach 100% due to slight misalignment of the
micro lens assembly, imperfections in the micro lens itself, nonsymmetric shielded
areas, and transmission losses. As shown by the camera formula (Equation 2-16,
page 35), the output is directly proportional to the detector area. Increasing the
optical fill factor with a microlens assembly increases the effective detector size
and, therefore, the output voltage.

The photosensitive area is below the gate structure and the ability to collect
the light depends upon gate thickness. The cone of light reaching the microlens
depends upon the f-number of the primary camera lens. Figure 3-42 illustrates
nearly parallel rays falling on the microlens. This case is encountered with high
f-number lens systems. Low f-number primary camera lenses increase the cone
angle and the effective fill-factor decreases with decreasing f-numberY
Microlenses are optimized for most practical f-numbers. As the array size
grows, off-axis detectors do not obtain the same benefit as on-axis detectors. 28

Hyper-HAD is a Sony trademark which indicates the presence of a micro lens

assembly over a hole accumulation diode. While many camera manufacturers
use the Hyper-HAD detectors, they do not identify the detector by Sony's
trademark. This leads the user to believe that there are many different detectors
and hence many different cameras on the market.
94 CCD ARRAYS, CAMERAS, and DISPLA"J'S

Detectors

(a)

Detectors

(b)

Figure 3-42. Optical effect of a microlens assembly. (a) With no

rnicrolens, a significant amount of photon flux is not detected. (b) The
rnicrolens assembly can image nearly all the flux onto the detector when
a high f-number lens is used. These lenslets can either be grown on the
array during the fabrication process or manufactured out of a material
such as quartz and placed on the array surface during packaging.

3.11.2. COLOR FILTER ARRAYS

The subjective sensation of color can be created from three primary colors.
By adjusting the intensity of each primary (additive mixing), a full gamut
(rainbow) of colors is experienced. This approach is used on all color displays.
They have red, green, and blue phosphors that, when appropriately excited,
produce a wide spectrum of perceived colors. The CIE committee standardized
a color perception model for the human observer in 1931. The literature29 is rich
with visual data that form the basis for color camera design.
 SOLID STATE ARRAYS 95

The "color" signals sent to the display must be generated by three detectors,
each sensitive to a primary or its complement. The primary additive colors are
red, green, and blue (R, G, B) and their complementary colors are yellow, cyan
and magenta (Ye, Cy, Mg). For high quality color imagery, three separate
detectors (discussed in Section 5. 3. 3., Color Correction) are used whereas for
consumer applications, a single array is used. The detectors are covered with
different filters that, with the detector response, can approximate the primaries
or their complements (Figure 3-43). A single array with filters is called a color
filter array (CFA).

1.0 .-----,----,--G-/""7""'i;\:-;---..,----,----,
f. ............................. j.......:/··1···············\+······························+-······························+················
0.9 ··················

9: o.8 1·····························1··-······················· +···,·jl·····················li\\·······················/·····L,.........:~········_··;R~···!······,.,..,.--.....················1

6
[ 0.7 1---/~,.,:;;'\.---+-/1----+\!\:---/-/-+--~"!-"'~----'1

~ ::: ············/··· ··~\ !/ \//············+!;······························+······························!

i :: v
a: 0.2
1
............................................
/,\ 1\
I·······\\·············+································!······························!
0.1 ................................. 77 ··'S..._-, .... ~

0.0
0.4 0.5 0.6 0.7
Wavelength (~m)

(a)

0.9
0.8
3lc 0.7
0
c. 0.6
Ill
CD
a: 0.5
CD
>
:;:::
0.4
ca 0.3
Qj
a: 0.2

0.1

0.5 0.6
Wavelength (~)

(b)

Figure 3-43. Desired spectral response for the three primaries and their
complements. (a) The primaries and (b) their complements.
96 CCD ARRAYS, CAMERAS, and DISPLAl:S

While CF As can have filters with the appropriate spectral transmittance to

create the primaries, it is more efficient to create the complementary colors.
Complementary filters have higher transmittances than the primary filters. The
luminance channel is identical to the green channel and is often labeled Y.
"White" (no color filter) is represented by W = R + G +B.

A linear set of equations relates the primaries to their complements. These

equations are employed (called matrixing) in cameras to provide either or both
output formats. Matrixing can convert RGB into other color coordinates:
Ye- R+ G = W·· B
Mg- R+B = W- G (3-12)

Cy-G+B-W··R.
The arrangement of the color filters for a single array system is either a
stripe or a mosaic pattern (Figure 3-44). The precise layout of the mosaic varies
by manufacturer. One basic CFA patent30 was granted to Bryce E. Bayer at
Eastman Kodak in 1976.

R G B R G B G Ye G Cy G

R G B R G B G Ye G Cy G

R G B R G B G Ye G Cy G

R G B R G B G Ye G Cy G

Stripes

Cy w Ye G G Mg G Mg R G R G

Ye G Cy w Cy Ye Cy Ye G B G B
Cy w Ye G Mg G Mg G R G R G

Ye G Cy w Cy Ye Cy Ye G B G B

Mosaics
Figure 3-44. Representative stripe and mosaic arrays. Although shown
as a full frame device for clarity, they typically are interline transfer
devices. The layout depends upon the manufacturer's philosophy and
cleverness in reducing color aliasing.
 SOLID STATE ARRAYS 97

In many sensors, the number of detectors devoted to each color is different.

The basic reason is that the human visual system (HVS) derives its detail
information primarily from the green portion of the spectrum. That is,
luminance differences are associated with green whereas color perception is
associated with red and blue. The HVS requires only moderate amounts of red
and blue to perceive color. Thus, many sensors have twice as many green as
either red or blue detector elements. An array that has 768 horizontal elements
may devote 384 to green, 192 to red, and 192 to blue. This results in an unequal
sampling of the colors. A birefringent crystal (discussed in Section 10.3.4.,
Optical Anti-alias Filter), inserted between the lens and the array, accommodates
the different sampling rates (discussed in Section 8.3 .2., Detector Array Nyquist
Frequency). The video signal from the CFA is embedded in the architecture of
the CF A. The color video from a single chip CF A must be decoded
(unscrambled) to produce usable R, G, and B signals.

3.11.3. NUMBER OF DETECTORS

Each detector array is designed for a specific application (discussed in

Section 5.2.2., Broadcast/Non-Broadcast Design) The EIA 170 video standard
supports 485 lines (discussed in Section 5.2.1., Video Timing). However, one
line is split between the even and odd fields so that there are only 484
continuous lines. Thus detector arrays designed for EIA 170 compatibility tend
to have 484 detectors in the vertical direction. For convenience, this has been
reduced to 480 detectors. Image processing algorithms are more efficient with
square pixels. With a 4:3 aspect ratio, the desired number of detectors is 640 x
480. Less expensive arrays will have a submultiple (for easy interpolation) such
as 320 x 240.

For video applications, several rows or columns are devoted to dark current
and for light leakage. Therefore an array may be 650 X 492 but the light
sensitive part may be 640 x 480. It is manufacturer dependent whether to
specify the array size by the number of active pixels or the total number of
pixels (which includes the dark pixels).

Beat frequencies are very obvious in color images. To avoid beat

frequencies, the color pixel rate should be a multiple of the chrominance
subcarrier frequency (3.579545 MHz). This results in arrays that contain 384,
576, or 768 horizontal detectors. While CFAs are designed for NTSC operation,
the same chip can be used for monochrome video without the CFA. Therefore,
many monochrome arrays also contain 384, 576, or 768 horizontal detectors.
These arrays do not have square pixels. But, from a manufacturing point-of-
view, this reduces the inventory of arrays offered.
98 CCD ARRAYS, CAMERAS, and DISPLAYS

Scientific array size tends to a power of 2 (e.g., 512 x 512, 1024 x 1024)
for easy image processing. There is a perception that "bigger is better" both in
terms of array size and dynamic range. Arrays may reach 8192 x 8192 with a
dynamic range of 16 bits. This array requires (8192)(8192)(16) or 1.07 Gbits
of storage for each image. Image compression schemes may be required if
storage space is limited. The user of these arrays must decide which images are
significant and through data reduction algorithms, store only those that have
value. Otherwise, he will be overwhelmed with mountains of data.

While large format arrays offer the highest resolution, their use is hampered
by readout rate limitations. For example, consumer camcorder systems operating
at 30 frames/s have a data rate of about 10 Mpixels/s. An array with 5120 x
5120 elements operating at 30 frames/s has a data rate of about 768 Mpixels/s.
Large arrays can reduce readout rates by having multiple parallel ports servicing
subarrays. Each subarray requires separate vertical and horizontal clock signals.
The tradeoff is frame rate (speed) versus number of parallel ports (complexity
of CCD design) and interfacing with downstream electronics. Because each
subarray is serviced by different on-chip and off-chip amplifiers, the displayed
image of the subarrays may vary in contrast and level. This is due to differences
in amplifier gains and level adjustments.

3.11.4. VIDEO CHIP SIZE

Vidicon vacuum tubes were originally used for professional television
applications. These were specified by the tube diameter. To minimize distortion
and nonuniformities within the tube, the recommended 31 image size was
considerably less than the overall tube diameter. When CCDs replaced the tubes,
the CCD industry maintained the image sizes but continued to use the tube
format nomenclature (Table 3-2).

Table 3-2
ARRAY SIZE for STANDARD FORMATS
CAMERA STANDARDIZED ARRAY SIZE ARRAY
FORMAT (H XV) DIAGONAL

1 inch 12.8 mm X 9.6 rnm 16mm

2/3 inch 8.8 mm x 6.6 mm 11 mm

112 inch 6.4 mm X 4.8 mm 8 mm

1/3 inch 4.8 mm X 3.6 mm 6mm

114 inch 3.2 mm x 2.4 mm 4mm

 SOLID STATE ARRAYS 99

Although each manufacturer supplies a slightly different array size and pixel
size, nominal sizes for a 768 X 480 array are given in Table 3-3. With interline
transfer devices, approximately one-half of the pixel width is devoted to the
shielded vertical transfer register. That is, the detector active width is one-half
of the pixel width. Thus the active area of the pixel is rectangular in interline
transfer devices. This asymmetry does not appear to affect image quality in
consumer video products significantly.

Table 3-3
NOMINAL PIXEL SIZE for a 768 x 480 ARRAY
Sizes vary by manufacturer. Detector sizes are smaller.

NOMINAL PIXEL
CAMERA
SIZE
FORMAT
(H XV)

1 inch 16.7 Jlm X 20 Jlm

2/3 inch 11.4 Jlm X 13.8 Jlm

112 inch 8.33 Jlm X 10 Jlm

1/3 inch 6.25 Jlm X 7.5 Jlm

1/4 inch 4.17 Jlm X 5 Jlm

The decrease in optical format is related to cost. The price of CCD arrays
is mainly determined by the cost of processing semiconductor wafers. As the
chip size decreases, more devices can be put on a single wafer and this lowers
the price of each individual device. The trend of going to smaller devices will
probably continue as long as the optical and electrical performance of the
imagers does not change. However, smaller pixels reduce the charge well size.
For a fixed flux level and lens f-number, the smaller arrays have reduced
sensitivity.

Smaller chips make for smaller cameras. However, to maintain resolution,

pixels can only be made so small. Here, the tradeoff is among pixel size, optical
focal length, and overall chip size. Further unless the lens f-number is reduced
as the chip size is reduced, the system will move from being detector-limited to
optically-limited (discussed in Section 10.4., Optical-Detector Subsystem). As
this happens, the system MTF will change and smaller detectors will provide a
less sharp image.
100 CCD ARRAYS, CAMERAS, and DISPLAYS

3.12. REFERENCES
1. J. Janesick, T. Elliott, R. Winzemead, J. Pinter, and R. Dyck, "Sandbox CCDs," in Charge-
Coupled Devices and Solid State Optical Sensors V, M M. Blouke, ed., SPIE Proceedings Vol.
2415, pp. 2-42 (1995).
2. J. Janesick and T. Elliott, "History and Advancement of Large Area Array Scientific CCD
Imagers," in Astronomical Society ofthe Pacific Conference Series, Vol. 23, Astronomical CCD
Observing and Reduction, pp. 1-39, BookCrafters (1992).
3. M. J. Howes and D. V. Morgan, eds., Charge-Coupl1·dDevices and Systems, John Wiley and
Sons, New York, NY (1979).
4. C. H. Sequin and M. F. Tompsett, Charge Transfer Devices, Academic Press, New York, NY
(1975).
5. E. S. Yang, Microelectronic Devices, McGraw-Hill, NY (1988).
6. E. L. Dereniak and D. G. Crowe, Optical Radiation Detectors, pp. 186-269, John Wiley and
Sons, New York, NY (1984).
7. A. J. P. Theuwissen, Solid-State Imaging with Charge-Coupled Devices, Kluwer Academic
Publishers, Dordrecht, The Netherlands (1995).
8. M. Kimata and N. Tubouchi, "Charge Transfer Devices," in Infrared Photon Detectors, A.
Rogalski, ed., pp. 99-144, SPIE Press, Bellingham, WA (1995).
9. A. J.P. Theuwissen, Solid-State Imaging with Charge Coupled Devices, pp. 317-348, Kluwer
Academic Publishers, Dordrecht, The Netherlands (1995).
10. A. J. P. Theuwissen, Solid-State Imaging with Charge-Coupled Devices, pp. 54-66, Kluwer
Academic Publishers, Dordrecht, The Netherlands (1995).
11. A. J. P. Theuwissen, Solid-State Imaging with Charge Coupled Devices, pp. 161-165, Kluwer
Academic Publishers, Dordrecht, The Netherlands (1995).
12. H.-S. Wong, Y. L. Yao, and E. S. Schlig, "TDI Charge-Coupled Devices: Design and
Applications," IBM Journal Research Development, Vol. 36(1), pp. 83-106 (1992).
13. T. S. Lomheim and L. S. Kalman, "Analytical Modeling and Digital Simulation of Scanning
Charge-Coupled Device Imaging Systems," in Electro-Optical Displays, M. A. Karim, ed., pp.
551-560, Marcel Dekker, New York (1992).
14. G. Wan, X. Gong, and Z. Luo, "Studies on the Measurement of Charge Transfer Efficiency
and Photoresponse Nonuniformity of Linear Charge-coupled Devices," Optical Engineering,
Vol. 34 (11), pp.3254-3260 (1995).
15. J. Janesick, "Open Pinned-Phase CCD Technology," in EUV, X-ray, and Gamma Ray
Instruments for Astronomy and Atomic Physics, C. J. Hailey and 0. H. Siegmund, eds., SPIE
Proceedings Vol. 1159, pp 363-373 (1989).
16. J. Janesick, T. Elliot, G. Fraschetti, S. Collins, M. Blouke, and B. Corey, "CCD Pinning
Technologies," in Optical Sensors and Electronic Photography, M. M. Blouke, ed., SPIE
Proceedings Vol. 1071, pp. 153-169 (1989).
17. J. S. Campbell, "TC271 Characterization Report" in Area Array Image Sensor Products, Texas
Instruments Data Catalog, pp. B-43 to B-46 (1996).
18. J. R. Janesick, T. Elliott, S. Collins, M. M. Blouke, and J. Freeman, "Scientific Charge-
coupled Devices," Optical Engineering, Vol. 26(8), pp. 692-714 (1987).
19. C. Buil, CCD Astronomy, pp. 125-152, Willmann-Bell, Richmond, VA (1991).
20. S. Kawai, M. Morimoto, N. Mutoh, and N. Teranishi, "Photo Response Analysis in CCD
Image Sensors with a VOD Structure," IEEE Transacticms on Electron Devices, Vol. 40(4), pp.
652-655 (1995).
21. M. Schanz, W. Brockherde, R. Hauschild, B. Hosticka. and M. Schwarz, "Smart CMOS Image
Sensor Arrays," IEEE Transactions on Electron Devices, Vol. 44(10), pp. 1699-1704 (1997).
22. S. K. Mendis, S. E. Kemeny, and E. R. Fossum, "A 128 x 128 CMOS Active Pixel Sensor
for Highly Integrated Imaging Systems," IEEE IEDM Technical Digest, pp. 583-586 (1993).
 SOLID STATE ARRAYS 101

23. E. R. Fossum, "CMOS Image Sensors: Electronic Camera-on-a-chip," IEEE Transactions on

Electron Devices, Vol. 44(10), pp. 1689-1698 (1997).
24. Z. Zhou, B. Pain, and E. R. Fossum, "Frame-transfer CMOS Active Pixel Sensor with Pixel
Binning," IEEE Transactions on Electron Devices, Vol. 44(10), pp. 1764-1768 (1997).
25. E. R. Fossum, "Active Pixel Sensors: Are CCD's Dinosaurs?, in Charge-coupled Devices and
Solid State Optical Sensors Ill, M. M. Blouke, ed., SPIE Proceedings Vol. 1900, pp 2-14
(1993).
26. J. Nakamura, T. Nomoto, T. Nakamura, and E. R. Fossum, "On-focal-plane Signal Processing
for Current-modeActive Pixel Sensors," IEEE Transactions on Electron Devices, Vol. 44(10),
pp. 1747-1757 (1997).
27. J. Furukawa, I. Hirota, Y. Takamura, T. Walda, Y. Keigo, A. Izumi, K. Nishibori, T. Tatebe,
S. Kitayama, M. Shimura, and H. Matsui, "A %-inch 380k Pixel (Effective) IT-CCD Image
Sensor," IEEE Transactions on Consumer Electronics, Vol. CE-38(3), pp. 595-600 (1992).
28. M. Deguchi, T. Maruyama, F. Yamasaki, T. Hamamoto, and A. Izumi, "Microlens Design
Using Simulation Program for CCD Image Sensor," IEEE Transactions on Consumer
Electronics, Vol. CE-38(3), pp. 583-589 (1992).
29. See, for example, J. Peddie, High-Resolution Graphics Display Systems, Chapter 2,
Windcrest/McGraw-Hill, New York, NY (1993) or A. R. Robertson and J. F. Fisher "Color
Vision, Representation, and Reproduction," in Television Engineering Handbook, K. B. Benson,
ed., Chapter 2, McGraw Hill, New York, NY (1985).
30. B. E. Bayer, U.S. patent #3,971,065 (1976).
31. R. G. Neuhauser, "Photosensitive Camera Tubes and Devices," in Television Engineering
Handbook, K. B. Benson, ed., page 11.34, McGraw-Hill, New York, NY (1986).
4
ARRAY PERFORMANCE
The most common array performance measures are read noise, charge well
capacity, and responsivity. From these the minimum signal, maximum signal,
signal-to-noise ratio, and dynamic range can be calculated. Full characterization
includes quantifying the various noise sources, charge transfer efficiency,
spectral quantum efficiency, linearity, and pixel nonuniformity. These additional
metrics are necessary for the most critical scientific applications.

Because the array is the basic building block of the camera, camera
terminology is often used for array specifications. Additional performance
measures exist at the system level. These include the noise equivalent irradiance
(NEI) and noise equivalent differential reflectance (NE.1p) and are discussed in
Chapter 6, Camera Performance.

The magnitude of each noise component must be quantified and its effect on
system performance must be understood. Noise sources may be a function of the
detector temperature. Predicted system performance may deviate significantly
from actual performance if significant 1/f noise or other noise is present. There
are a myriad of factors involved in system optimization. It is essential to
understand what limits the system performance so that intelligent improvements
can be made.

Janesick 1 uses three methods to characterize CCD performance: (1) photon

transfer, (2) x-ray transfer, and (3) photon standard. The photon transfer
technique (described in this chapter) appears to be the most valuable in
calibrating, characterizing, and optimizing arrays for general video and industrial
applications. Additional information, necessary for characterizing scientific
arrays, is obtained from the x-ray transfer and photon standard methods.

Soft x-ray photons produce a known number of electron-hole pairs in a

localized area of the CCD. Because the number and location of electron hole-
pairs are known precisely, it is easy to determine the charge transfer efficiency.
With the photon standard technique, the CCD array views a source with a
known spectral distribution. Using calibrated spectral filters or a
monochrometer, the spectral quantum efficiency is determined.

102
 ARRAY PERFORMANCE 103

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

4.1. SIGNAL

Device specifications depend, in part, upon the application. Arrays for

general video applications may have responsivity expressed in units of Vllux.
For scientific applications, the units may be in V/(J-cm-2) or, if a digital output
is available, DN/(J-cm-2) where DN refers to a digital number. For example, in
an 8-bit system the digital numbers range from zero to 255. These units are
incomplete descriptors unless the device spectral response and source spectral
characteristics are furnished.

The maximum output occurs when the charge wells are filled. The exposure
that produces this value is the saturation equivalent exposure (SEE). With this
definition, it is assumed that the dark current produces an insignificant number
of electrons so that only photoelectrons fill the well.

Sensitivity suggests something about the lowest signal that can be detected.
It is usually defined as the input signal that produces a signal-to-noise ratio of
one. This exposure is the noise equivalent exposure (NEE). The minimum signal
is typically given as equivalent electrons rms. It is only one of many
performance parameters used to describe system noise performance.

Figure 4-1 illustrates a signal transfer diagram. The shutter represents the
integration time, t1NT. The detector converts the incident photons, noETECTOR• into
electrons at a rate determined by the quantum efficiency. A voltage is created
when the electrons are transferred to the sense node capacitance. The signal
after the source follower amplifier is
Gq (4-1)
VsiGNAL - n, C •

where n, is the total number of electrons in the charge packet. It includes both
photoelectrons and dark current electrons. The output gain conversion, Gq/C,
is typically 0.1 p.V /e- to 10 p.V /e-. The source follower amplifier gain, G, is near
unity and, therefore, it is sometimes omitted from radiometric equations.
Amplifiers, external to the CCD device, amplify the signal to a useful voltage.
These amplifiers are said to be off-sensor or off-chip.
104 CCD ARRAYS, CAMERAS, and DISPLAYS

Shutter
(Integration Sense Node
Time) (Charge Off-Chip

"=~
Detector Conversion) Amplifier

VCAMERA
Digital
(Analog Output) Output

Figure 4-1. Representative signal transfer diagram. Many systems do not

have a shutter, but it represents t 1NT. The output may be specified as the
number of electrons, output voltage of the source follower amplifier, or
output of the off-chip amplifier. Some devices provide a digital output.
Vs1oNAL• VcAMERA• and DN can be measured whereas n, is calculated.

4.1.1. SPECTRAL RESPONSE

For an ideal material, when the photon energy is greater than the
semiconductor band gap energy, each photon produces one electron-hole pair
(quantum efficiency is one). However, the absorption coefficient in silicon is
wavelength dependent (Figure 4-2). Long wavelength photons are absorbed
deeper into the substrate than short wavelengths. Very long wavelength photons
may pass through the CCD and not be absorbed. Beyond 1.1 ~-tm, the absorption
is essentially zero because the photon energy is less than the band gap energy.
For a doping concentration of 10 17cm-3 , the depth at which 90% of the incident
photons are absorbed is given in Table 4-1. Because of reflections, the photon
incidance is higher. The values in Table 4-1 apply only to those photons that
enter the material.
 ARRAY PERFORMANCE 105

1()4

1()3

102 ·1019fcm3

10 ·1018fcm3

s1017fcm3

1
0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1
Wavelength (l!m)

Figure 4-2. Silicon absorption coefficient as a function of wavelength

and doping concentration at 25°C. Detectors typically have doping
concentrations less than 10 17 /cm3 • (From Reference 2).

Any photon absorbed within the depletion will yield a quantum efficiency
near unity. However, the depletion region size is finite and long wavelength
photons will be absorbed within the bulk material. An electron generated in the
substrate will experience a three-dimensional random walk until it recombines
or reaches the edge of a depletion region where the electric field exists. If the
diffusion length is zero, all electrons created within the bulk material will
recombine immediately and the quantum efficiency approaches zero for these
photons. As the diffusion length approaches infinity, the electrons eventually
reach a charge well and are stored. Here, the quantum efficiency approaches
one.
106 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 4-1
REPRESENTATIVE 90% ABSORPTION DEPTH
Wavelength (~tm) Depth (~tm)

0.40 0.19

0.45 1.0

0.50 2.3

0.55 3.3

0.60 5.0

0.65 7.6

0.70 8.5

0.75 16

0.80 23

0.85 46

0.90 62

0.95 150

1.00 470

1.05 1500

1.10 7600

Doping controls the diffusion length and the quantum efficiency is

somewhere between these two extremes (Figure 4-3). The quantum efficiency
is dependent upon the gate voltage (low voltages produce small depletion
regions) and the material thickness (long wavelength photons will pass through
thin substrates). Diffusion creates a responsivity that overlaps pixels (Figure 4-
4). The maximum signal and other performance parameters defined in this
chapter are for large area illumination. That is, a large number of detectors are
illuminated and the average output is used. As the image size decreases,
diffusion effects becomes more apparent. Diffusion effects are described by an
MTF (discussed in Section 10.3.1., Diffusion MTF).
 ARRAY PERFORMANCE 107

>-
0
z
w 0.8
0 ideal
u::
u.. 0.6
silicon
w infinite diffusion
:Ill 0.4 length
::I
1-
z 0.2 zero diffusion
<C length
::I
a 0
0.4 0.6 0.8 1.2

WAVELENGTH (pm)

Figure 4-3. Theoretical internal quantum efficiency for silicon

photosensors 250 p.m thick. The long wavelength quantum efficiency
depends upon the thickness of the substrate and the diffusion length.
Most arrays have a relatively large diffusion length so that the quantum
efficiency tends to be near the infinite diffusion length curve. (From
Reference 3).

j.-- Pixel -1- Pixel -1- Pixel -1

I I I Gate
~ ~ ~Electrode

Ull )< i X )(
Figure 4-4. Idealized relative spatial response to long wavelength
photons.

For front illuminated devices with photogates, the spectral responslVlty

deviates from the ideal spectral response due to the polysilicon gate electrodes
(see Figure 3-2, page 48). The transmittance of polysilicon starts to decrease
below 0.6 p.m and becomes opaque at 0.4 p.m (Figure 4-5). The transmittance
depends upon the gate thickness and material. 4 The gate structure is a thin film
and therefore interference effects will cause variation in quantum efficiency that
is wavelength dependent. Interference effects can be minimized through choice
of material and film thickness. Because different manufacturing techniques are
used for each array type, the spectral response varies with manufacturer. If a
manufacturer uses different processes, then the spectral response will vary
within his product line.
108 CCD ARRAYS, CAMERAS, and DISPLAYS

~
c 50
Cll
·u-- 40
:ec
w ~ 30
_E 8....
:::J
....... 20 Front Illuminated
with UV Coating
c
ca
:::J
a OL-L-~~~d_~~~_L-~_L~_L_L~~~~

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1
Wavelength {llm)

~ 60 .-----------------------------------------,
g Front Illuminated
Cll 50
~ E' 4o
w ~ 30
_ Gi
E .......
a.. 20
:::J
c
ca 10
:::J
a oL-L-L-~~_L_L_L~~_J~L_L_L_L_~~~~

0.2 0.3 0.4 0.5 0.6

0.7 0.8 0.9 1.0 1.1
Wavelength (llm)

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1

Wavelength (11m)

Figure 4-5. Typical response curves provided by different

manufacturers. Above 0.8~-tm, the quantum efficiency drops because the
electron-hole pairs, generated deep in the array, may recombine before
reaching a storage site. Below 0.6~-tm. the polysilicon overcoat starts to
become opaque. The variation in spectral response may not be a concern
for many applications. Organic phosphors convert UV photons into
visible photons and thereby extend the device's spectral response. The
ripples in the responsivity are caused by interference effects.
 ARRAY PERFORMANCE 109

Photodiodes, as used in most interline transfer devices, do not have

polysilicon gates. Therefore these devices tend to have a response that
approaches the ideal silicon spectral response. That is, there are no interference
effects and the blue region response is restored. With virtual phase devices, one
poly silicon electrode is replaced by a shallow highly doped layer. Because there
is no electrode over this layer, higher quantum efficiency occurs in the blue
region (see Figure 3-11, page 57). CIDs consist of two overlapping MOS
capacitors sharing the same row and column electrode. Because the electrode
only covers a part of the pixel, the area-averaged spectral response is higher the
CCDs (see Figure 3-38, page 89).

Devices with vertical antibloom drains have reduced red response. The
buried drain is in close proximity to where long photon absorption occurs. Any
electron that enters the drain is instantly removed.

Even with no polysilicon overcoat, UV photons are absorbed at the surface.

These electrons recombine before they reach a storage site. Hence the quantum
efficiency is very low in the UV. The UV response can be enhanced with a UV
fluorescent phosphor. These phosphors are deposited directly onto the array and
they emit light at approximately 0.54 to 058 p,m when excited by 0.12 to 0.45
p,m light. Phosphors radiate in all directions and only the light fluorescing
toward the array is absorbed. Lumogen, which is one of many available organic
phosphors, has an effective quantum efficiency of about 15%. It does not
degrade the quantum efficiency in the visible region because it is transparent at
these wavelengths. The coating does not change the spectral response of the
detector elements. Rather, it converts out-of-band photons into in-band photons.
Therefore, the detector with the coating appears to have a spectral response from
0.12to 1.1 p,m.

Illuminating the array from the back side avoids the polysilicon problem and
increases the quantum efficiency below 0.6 p,m (Figure 4-6). Photons entering
the back side are absorbed in the silicon and diffuse to the depletion region.
However, short wavelength photons are absorbed near the surface and these
electron-hole pairs recombine before reaching a storage site in a thick wafer (see
Table 4-1). Therefore, the wafer is thinned to about 10 p,m to maintain good
spectral responsivity (Figure 4-7). In back-side thinned devices, the incident
photon flux does not have to penetrate the polysilicon gate sandwich structure
and interference effects are much easier to control. With a proper anti-reflection
coating, a quantum efficiency of approximately 85% is possible. However,
silicon has an index of refraction that varies with wavelength making it
somewhat difficult to manufacture an anti-reflection coating that is effective
across the entire spectrum. Coatings that optimize the response in the near IR
110 CCD ARRAYS, CAMERAS, and DISPLAYS

have reduced effectiveness in the visible (Figure 4-8). Lumogen reduces the
effectiveness of the anti-reflection coating. New anti-reflection coatings
overcome this problem and some offer UV response without the use of
lumogen. 6

80

---8
c
70 Back Illuminated

.
!
60 .... uv
Coating

>- 50
(,)
c
CD
·u 40
J
ffi

0
-
E
::J
c
I'll
::J
30

20
"' Front Illuminated

10 "'

0
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1
Wavelength (J..Lm)

Figure 4-6. Representative spectral response of front illuminated and

back illuminated CCD arrays. The spectral response depends upon the
process used to manufacture each device.

Silicon Silicon
Dioxide
Gate
Thinned
Silicon

Incoming
Light

A B
Figure 4-7. (a) Front illuminated and (b) back illuminated arrays. Owing to
their extremely complex and fragile design, back illuminated devices are
usually limited to scientific applications that require high quantum efficiency.
 ARRAY PERFORMANCE 111

100

'#. 80

>
()
c:
Q)
60
·c:;
:0:::
.....
w 40

...c:E
:I
20
ca
:I
0
0
0.3 0.7 0.8 0.9 1.0 1.1
WAVELENGTH (Jim)
Figure 4-8. Effect of different anti-reflection coatings on back
illuminated CCDs. (From Reference 5).

As the wavelength increases, the silicon becomes more transparent. With

thin back illuminated devices, the rear surface and silicon oxide layer create an
optical etalon. This creates constructive and destructive interference that appear
to modulate the spectral responsivity .5 At 0.8 Jlm, the modulation occurs
approximately every 5 nm. This is bothersome in spectroscopic applications. For
other applications, the wave band is sufficiently wide to cover several etalon
fringe cycles and this averages out the effect.

CCD arrays for general and industrial applications are within a sealed
environment for protection. Light must pass though a window (may be glass or
quartz) to reach the array. In addition to reflection losses at all wavelengths, the
glass transmittance decreases for wavelengths below 0.4 JLm. For scientific
applications requiring high sensitivity, the glass can be coated with a broad band
anti-reflection coating. For UV applications, a quartz UV transmitting window
can be used.

4.1.2. RESPONSIVITY

The spectral quantum efficiency is important to the scientific and military

communities. When the array is placed into a general video or industrial camera,
it is convenient to specify the output as a function of incident flux density or
energy density averaged over the spectral response of the array.
112 CCD ARRAYS, CAMERAS, and DISPLAYS

Assume that the array is at a distance of R 1 from an ideal source whose area
is As. This radiometric setup is similar to that described in Section 2.5., Camera
Formula, page 33, except no lens is present. For convenience, radiant quantities
are used rather than photon flux. Assuming a Lambertian source,

n
1A A
= --S -
D
"J-
2
M (.A)
- R .,) t
e (~~. d'11. (4-2)
pe q 2 1t e lNT '
Rl AI

where R,(>-.) is the spectral response expressed in units of A/W. Note that the
detector incidance is E0 (A) = M0 (>-.)As/R?, Then

(4-3)

The array output voltage (after the source follow amplifier) is (G q ~)/C:

(4-4)

or

(4-5)

It is desirable to express the responsivity in the form

(4-6)

The value RAvE is an average response that has units of V/(J-cm-2) and the
quantity in the brackets has units of J/cm2 •
 ARRAY PERFORMANCE 113

Combining the two equations provides

,.2
JM.().) RP,.)d'A
GA -"-'--!- - - - - - (4-7)
C D }.2

j M/'A)d'A
}.I

Note that Rq and RAvE are the for array package. Rq or RAvE is the detector
responsivity multiplied by the window transmittance. Usually the manufacturer
includes the window transmittance in the quoted responsivity.

Experimentally, the most popular approach to measure RAvE is to vary the

exposure (e.g., the integration time). The value RAvE is the slope of the
output/input transformation (Figure 4-9). The bracketed term in Equation 4-6 is
the input. By placing a calibrated radiometer next to the array (at distance R 1
from the source), the incidance, Ee, can be measured, Multiplying by the
integration time provides J/cm2 • These equations are valid over the region that
the array output/input transformation is linear. The incidance can also be
changed by moving the source (varying R 1).

Slope

-"~~ Saturation

--
::I
c.
::I Dark
0 Current

-----No Dark
Current
SEE
Exposure _ _____,~

Figure 4-9. The average responstvtty is the slope of the output-input

transformation. The maximum input or the saturation equivalent exposure
(SEE) is the input that fills the charge wells. SEE is used to define the
dynamic range. Dark current limits the available signal strength. Cooling can
reduce the dark current to a negligible level.
114 CCD ARRAYS, CAMERAS, and DISPLAYS

It is imperative that the source color temperature remains constant.

Assuming an incandescent bulb is used, reducing the bulb voltage will reduce
the output but this changes the color temperature. Changing the source
temperature changes the integrals and this leads to incompatible results (see
Section 2.7., Normalization Issues, page 39). The incidance can only be varied
by inserting neutral density filters.

The value RAvE is an average type responsh ity that depends upon the source
characteristics and the spectral quantum efficiency. While the source can be
standardized (e.g., CIE illuminant A or illuminant D 6500 ), the spectral quantum
efficiency varies by device (see Figure 4-5, page 108). Therefore extreme care
must be exercised when comparing devices solely by the average responsivity.

Both Me(A) and Re(A) are functions of wavelength. If a very small

wavelength increment is selected, MiA) and ~(A) may be considered as
constants. Equation 4-7 can be approximated as

The responsivity (with units of A/W) is related to the quantum efficiency Rq by

Re = (qA/hc) Rq. If the wavelength is measured in micrometers, then Re =
(A/1.24) Rq. At a specific wavelength

RSIGNAL(').. o) =
Gq ~A R ('A ) --
v
C he D '~ 0
Jfcm 2 (4-9)
G
= -A R ('A)
C D e o

or

RSIGNAL(').. 0) = G ~A D R q,0.)
C 1.24 0
-v-2 (4-10)
Jfcm

Here, the detector area is measured in square centimeters. Typically Rq(A0 ) is

evaluated at that wavelength which provides the peak quantum efficiency:
Rq(Ap). The symbol 7J is often used for the quantum efficiency.
 ARRAY PERFORMANCE 115

If the device has a digital output, the number of counts (integer value) is

. [ VCAMERA -2N
DN = mt -] , (4-11)
VMAX

where N is the number of bits provided by the analog-to-digital converter and

int represents the integer value of the bracketed term. The value N is usually 8
for general video applications but may be as high as 16 for scientific cameras.
The value VMAx is the maximum output that corresponds to a full well, NwELL:

Gq (4-12)
vMAX- GlcNWELL.

The responsivity at the peak wavelength is approximated by

DN (4-13)
Jfcm2

This assumes that the analog-to-digital converter input dynamic range exactly
matches V MAx· This may not be the case in real systems. If an antibloom drain
is present, then the responsivity is defined up to the knee point (see Figure 3-35,
page 87) and Equation 4-11 is modified accordingly.

When using arrays for general video, the output is normalized to photometric
units (see Section 2.3.1., Units, page 25). It is convenient to express the array

l
average response as

0.75 11m
VSIGNAL - RPHOTOMETRIC ( 683 - -
AD
1
0.38
f M/'A) V('A) d).. ,
(4-14)

where RPHOTOMETRic has units of V /lux and the bracketed term has units of lux.
Combining Equations 4-4 and 4-14 yields

A A Az
G ____£___!!__
C R2
f L/'A) R,('A) tiNT d'A
RPHOTOMETRIC -
1 ).1
----0.7_s__m_ _ _ _ __
v (4-15)
11 lux
683 -
AD
1
0.38
f M ('A) V('A) d'A
p
116 CCD ARRAYS, CAMERAS, and DISPLAJ:S

Because Mp(A)/A 0 = Ee(A), for a Lambertian source,

l2

GAD
c
f E.(J.. ) R.(). ) tiNT d)..
l v (4-16)
RPHOWMETRJC - ---0--=-.~s-.-
. m-- - - - -
lux
683 f Ee(J..) V(A.) d)..
0.38

RPHOTOMETRrc depends on the integration time and is either 1/60 or 1130 s

depending on the architecture for EIA 170 or NTSC video standards (see Section
3. 3.4., Interline Transfer, page 63). Experimentally, the photometric illuminance
is measured with a calibrated sensor. The input intensity levels are changed
either by moving the source (changing R1) or by inserting neutral density filters
between the source and the array.

The responsivity depends on the spectral response and the spectral content
of the illumination. The apparent variation in output with different light sources
was discussed in Section 2. 7., Normalization Issues, page 39. Selecting an array
based on the responsivity is appropriate if the anticipated scene illumination has
the same color temperature as the calibration temperature. That is, if the
photometric responsivity is measured with a CIE illuminant A and the scene
color temperature is near 2856 K, then selecting an array with the highest
photometric responsivity is appropriate. Other wise, the average photometric
responsivity is used for informational purposes only.

Five numerical examples follow. Additional examples can be found in

Chapter 6.

~=============================================
Example 4-1
MAXIMUM OUTPUT

A device provides a output gain conversion of 6 p.V/e-. The well size is

70,000 electrons. What is the maximum output?

The maximum output is the OGC multiplied by the well capacity or 420 mV.
If the dark current creates 5,000 electrons, then the maximum signal voltage is
reduced to (70,000-5,000)(0.006mV)= 390 mV. This signal may be amplified
by an off-chip amplifier.
 ARRAY PERFORMANCE 117

Example 4-2
DIGITAL RESPONSIVITY

A device with a well capacity of 50,000 electrons has its output digitized
with a 12-bit analog-to-digital converter (ADC). What is the digital step-size
(digital responsivity)?

The step size is the well capacity divided by the number of digital steps.
50,000/2 12 = 12.2 electrons/DN. Equation 4-2 (page 112) provides the number
of electrons as a function of input exposure. The relationship between the ADC
maximum value and charge well capacity assumes an exact match between V MAx
and the ADC input range. If these do not match, an additional amplifier must
be inserted just before the ADC.

=============================================~

Example 4-3
RESPONSIVITY

A 2/3-inch format interline transfer device provides an output gain

conversion of 6pV/e-. If the quantum efficiency is 0.8 at 0.6/lm, what is the
responsivity in units of VI 11J -cm-2 ?

From Table 3-3 (page 99), a 2/3-inch format pixel is approximately 11.4/lm
X 13.8/lm. If the interline transfer device has a 50% fill factor, then A 0 = 7.87
X 10-7 cm2 • The constant he is 1.99 X 10-25 J-m/photon. Using Equation 4-9,
6
06
RSlGNAL(}.)- (6 X 10-6 )( · X IQ- )(7.87 X 10-7 ) (0.8)- 11.4 X 106
• 1.99 x w-zs (4-17)

V
J-cm-2 -
(-;:V) ((J-m)fphoton
m ) (cm2) (electrons)
photons
118 CCD ARRAYS, CAMERAS, and DISPLAYS

~=============================================
Example 4-4
PHOTOMETRIC OUTPUT

An array designed for video applications has a sensitivity of 250 p, V/lux

when measured with CIE illuminant A. What is the expected output if the input
is 1000 lux?

If the color temperature does not change and only the intensity changes, then
the output is (250 p,V)(1000) = 250 mV. However, if the color temperature
changes, the output changes in a nonlinear manner. Similarly, two sources may
produce the same illuminance but VsiGNAL may be quite different. The color
temperature must always be specified when RPffoTaMETRic is quoted.

Implicit in this calculation is that the integration time remains constant.

Because photometric units are used for general video applications, it is
reasonable to assume that tiNT is either 1/60 or 1130 s for EIA 170 compatibility.
=============================================~
~==============================================
Example 4-5
PHOTOMETRIC CONVERSION

An array designed for video applications has a sensitivity of 20 V/p.J-cm-2 •

What is the estimated photometric response? The relative spectral response of
RsiGNAL(A.) is illustrated in Figure 4-10.

Substituting Equation 4-9 into Equation 4-16 provides

.,
f E,(l) RSIGNAL-REU.~A) dl
•• v (4-18)
RPHOTOMETRIC - tiNT RSIGNAL-MAX
0.1S ~"' lux
683 f E,(l) V(l) dl
0.38

Two different spectral responsivities are considered: one as shown in Figure 4-

10 and the other has an additional infrared blocking filter that cuts off at 0. 7 p,m.
Numerical integration provides the value of the bracketed term (Table 4-2). As
the color temperature increases, the bracketed value may either increase or
decrease depending upon the spectral response. With RsiGNAL-MAx = 2000 V /J-m-2
(note change in units), tiNT = 1160 s, RPHoTaMETRic is given in Table 4-3.
 ARRAY PERFORMANCE 119

w 100
(/)
z
0 75
ll.
(/)
w
0::: 50
w
>
i= 25
:5
w
0::: 0
0.35 0.45 0.55 0.65 0.75 0.85 0.95
WAVELENGTH (pm)
Figure 4-10. Relative spectral responsivity of RsiGNAL(A).

Table 4-2
EQUATION 4-18 BRACKETED TERM
COLOR NO INFRARED WITH INFRARED
TEMPERATURE FILTER FILTER

2856K 0.00473 0.00319

6500K 0.00391 0.00353

Table 4-3
PHOTOMETRIC RESPONSIVITY (V flux)

COLOR NO INFRARED WITH INFRARED

TEMPERATURE FILTER FILTER

2856K 0.158 0.106

6500K 0.130 0.117

=============================================~

4.1.3. MINIMUM SIGNAL

The noise equivalent exposure (NEE) is an excellent diagnostic tool for

production testing to verify noise performance. NEE is a poor array-to-array
comparison parameter and should be used cautiously when comparing arrays
with different architectures. This is so because it depends on array spectral
120 CCD ARRAYS, CAMERAS, and DISPLAYS

responsivity and noise. The NEE is the exposure that produces a signal-to-noise
ratio of one. If the measured rms noise on the analog output is VNOISE• then the
NEE is calculated from the radiometric calibration:

NEE- VNOISE J (4-19)

rms.
RAVE

When equated to electrons, NEE is simply the noise value in rms electrons. The
absolute minimum noise level is the noise floor and this value is used most often
for the NEE. Noise sources are discussed in Section 4.2., Noise. Although noise
is arms value, the notation rms is often omitted. When divided by the average
quantum efficiency, the noise equivalent signal (NES) is obtained:

NES - NEE photons. (4-20)

Rq-AVE

4.1.4. MAXIMUM SIGNAL

The maximum signal is that input signal that saturates the charge well and
is called the saturation equivalent exposure (SEE). It is

VMAX J (4-21)
SEE-
RAVE

The well size varies with architecture, number of phases, and pixel size. The
well size is approximately proportional to pixel area (Table 4-4). Small pixels
have small wells. If an antibloom drain is present, the maximum level is taken
as the white clip level (see Figure 3-35, page 87). The maximum value of VMAx
is
Gq (4-22)
VMAx- G i - (NWEu- nDARK) •
c
For back-of-the-envelope calculations, the dark current is considered negligible
(i.e., the device is cooled or MPP is implemented). Then noARK ::::: 0.
 ARRAY PERFORMANCE 121

Table 4-4
NOMINAL WELL CAPACITY for
TEXAS INSTRUMENTS FRAME TRANSFER DEVICES
(Varies by manufacturer and device type)

WELL CAPACITY
FORMAT
(electrons)

2/3 inch 200,000

112 inch 80,000

113 inch 30,000

4.1.5. DYNAMIC RANGE

Dynamic range defined as the maximum signal (peak) divided by the rms
noise. If an antibloom drain is present, the knee value is used as the maximum
(See Figure 3-35, page 87). Expressed as a ratio, the dynamic range is

DR - SEE - VMAX (4-23)

ARRAY NEE V
NOISE

and when expressed in decibels,

DRARRAY- SEE)
20log (- - dB. (4-24)
NEE

When equated to electrons,

NWEu- nDARK
(4-25)
DRARRAY-

where < nsvs > is the system noise measured in electrons rms. The effects of
< nsvs > on dynamic range is further discussed in Section 4. 3., Array Signal-to-
Noise Ratio. Often, only the noise floor is considered in < nsvs > . The noise is
dependent upon amplifier design and is not related to charge well capacity.
122 CCD ARRAYS, CAMERAS, and DISPLAlS

While larger arrays have larger well capacities, this does not necessary mean
that the dynamic range increases by the same amount. As the pixel rate
increases, the amplifier bandwidth increases. Simultaneously, the amplifier noise
increases and the dynamic range decreases.

Sometimes the manufacturer will specify dynamic range by the peak signal
divided by the peak-to-peak noise. This peak-to-peak value includes any noise
spikes and, as such, may be more useful in determining the lowest detectable
signal in some applications. If the noise is purely Gaussian, the peak-to-peak is
assumed to be four to six times the rms value. This multiplicative factor is
author dependent.

If the system noise is less than the ADC's least significant bit (LSB), then
the quantization noise limits the system dynamic range. The rms quantization
noise, < nAoc > , is VLssl,[12 where VLss is the "oltage step corresponding to the
least significant bit. The dynamic range is 2N ,[12. For a 12-bit ADC, the
dynamic range is 14,189:1 or 83 dB. An 8-bit system cannot have a dynamic
range greater than 887:1 or 59 dB.

Example 4-6
ARRAY PARAMETERS

An array has a dynamic range of 3000: 1, average respons1v1ty of 2

V/(pJ/cm2), and a saturation equivalent exposure of 150 nJ/cm2 • These values
apply only at Ap. What are the remaining design parameters?

Maximum output:
The maximum output is RAvE SEE= 300mV.
Noise level:
The noise floor rms value is equal to the maximum output divided by the
dynamic range or 100 p.V rms.
Analog-to-digital Converter:
For a digital output, the ADC must have 12 bits (212 = 4096 levels) to span
the array's full dynamic range. The responsivity in DN is 2 12/SEE = 27.3
DN/(nJ/cm2) at Ap.
============================================~
 ARRAY PERFORMANCE 123

4.2. NOISE

Many books and articles7•13 have been written on noise sources. The level of
detail used in noise modeling depends on the application. Shot noise is due to
the discrete nature of electrons. It occurs when photoelectrons are created and
when dark current electrons are present. Additional noise is added when reading
the charge (reset noise) and introduced by the amplifier (1/f noise and white
noise). If the output is digitized, the inclusion of quantization noise may be
necessary. Switching transients that are coupled though the clock signals also
appear as noise. It can minimized by good circuit design.

Figure 4-1 (page 104) illustrated the signal transfer diagram. The noise
transfer diagram is somewhat different (Figure 4-11). With different transfer
functions, both the optical and electronic subsystems must be considered to
maximize the system signal-to-noise ratio.

Photon Noise
Dark Current Noise
Fixed Pattern Noise Reset 1/1 Noise Iff Noise Quantization

Photo Responsle Non::::~::de Nolise On-Chip Amplifielr Nois:ff.Chip Amplifie\r Noise Nolise

n!_+ Q~ .~ r:l+_+. l.:l :_+

L__jo-L!f-o~o-LJ-o-rro-

Figure 4-11. The various subsystems are considered as ideal elements with
the noise introduced at appropriate locations.

Although the origins of the noise sources are different, they all appear as
variations in the image intensity. Figure 4-12a illustrates the ideal output of the
on-chip amplifier before correlated double sampling. The array is viewing a
uniform source and the output of each pixel is identical. Photon shot noise
produces a temporal variation in the output signal that is proportional to the
square root of the signal level in electrons. Each pixel output will have a slightly
different value (Figure 4-12b).
124 CCD ARRAYS, CAMERAS, and DISPLAYS

Reset

(a)

(b)
Photon
}
Noise

Dark Current
} Noise
(c)

} Reset Noise
(d)

(e)
l.
T Amplifier Noise

(f)
Ill Noise

(g)

Figure 4-12. Noise sources affect the output differently. (a) though (f)
represent the output after the on-chip amplifier. (g) is the output after the
sample-and-hold circuitry. The array is viewing a uniform source that
produces identical outputs (amplitude = A) for each pixel. (a) ideal output,
(b) photon shot noise, (c), dark current shot noise, (d) reset noise, (e)
amplifier noise, (f) amplifier 1/f noise, and (g) quantization noise. These
values change from frame-to-frame. Pattern noise is not shown but is a
variation that does not change significantly each frame. All these processes
occur simultaneously.
 ARRAY PERFORMANCE 125

Under ideal conditions, each pixel would have the same dark current
and this value is subtracted from all pixels leaving the desired signal (see
Section 3.6., Dark Current, page 79). However, dark current also exhibits
fluctuations. Even after subtracting the average value, these fluctuations
remain and create fixed pattern noise (Figure 4-12c). The output capacitor
is reset after each charge is read. Errors in the reset voltage appear as output
signal fluctuations (Figure 4-12d). The amplifier adds white noise (Figure
4-12e) and 1/f noise (Figure 4-12t). Amplifier noise affects both the active
video and reset levels. The analog-to-digital converter introduces
quantization noise (Figure 4-12g). Quantization noise is apparent after image
reconstruction. Only photon shot noise and amplifier noise affect the amplitude
of the signal. With the other noise sources listed, the signal amplitude, A,
remains constant. However, the value of the output fluctuates with all the noise
sources. Because this value is presented on the display, the processes appear as
displayed noise. These noise patterns change from pixel-to-pixel and on the same
pixel from frame-to-frame.

Pattern noise refers to any spatial pattern that does not change significantly
from frame-to-frame. Dark current varies from pixel-to-pixel and this variation
is called fixed pattern noise (FPN). FPN is due to differences in detector size,
doping density, and foreign matter getting trapped during fabrication.
Photoresponse nonuniformity (PRNU) is the variation in pixel responsivity and
is seen when the device is illuminated. This noise is due to differences in
detector size, spectral response, and thickness in coatings. These "noises" are
not noise in the usual sense. PRNU occurs when each pixel has a different
average value. This variation appears as spatial noise to the observer. Frame
averaging will reduce all the noise sources except FPN and PRNU. Although
FPN and PRNU are different, they are sometimes collectively called scene
noise, pixel noise, pixel nonuniformity, or simply pattern noise.

It is customary to specify all noise sources in units of equivalent electrons

at the detector output. Amplifier noise is reduced by the amplifier gain. When
quoting noise levels, it is understood that the noise magnitude is the rms of the
random process producing the noise. Noise powers are considered additive.
126 CCD ARRAYS, CAMERAS, and DISPLAYS

Equivalently, the noise sources are RSSed (root sum of the squares) or added
in quadrature:

(4-26)

where (n2 ) is the noise variance for source i and v(n2;) = (n;) is the standard
deviation measured in rms units.

For system analysis, it may be sufficient to consider only the on-chip

amplifier noise. The array manufacturer usually provides this value and may call
it readout noise, mux noise, noise equivalent electrons, or the noise floor. The
value varies by device and manufacturer. Reset noise can be reduced to a
negligible level with CDS (discussed in Section 4.4., Correlated Double
Sampling). CDS can also reduce the source follower 1/f noise. The off-chip
amplifier is usually a low noise amplifier such that its noise is small compared
to the on-chip amplifier noise. Although always present, quantization noise is
reduced by selecting the appropriated sized analog-to-digital converter (e.g.,
selecting an ADC with many quantization levels). The simplified noise model
is
(4-27)

The noise model does not include effects such as banding and streaking
which are forms of pattern noise found in linear and TDI arrays. Banding can
occur with arrays that have multiple outputs serviced by different nonlinear
amplifiers (see Figure 3-14, page 61). Streaking occurs when the average
responsivity changes from column to column in TDI devices. 14 These effects are
incorporated in the three-dimensional noise model (discussed in Section 12.2. ,
Three-dimensional Noise Model) and are used to predict the performance of
military systems. Only a complete signal-to-noise ratio analysis can determine
which noise source dominates (discussed in Section 4.3., Array Signal-to-noise
Ratio).
 ARRAY PERFORMANCE 127

4.2.1. SHOT NOISE

Both photoelectrons and dark current contribute to shot noise. Using Poisson
statistics, the variance is equal to the mean:

(4-28)

The number of photoelectrons was given by Equation 4-2, page 112, and the
number of dark current electrons was given by Equation 3-6, page 80. These
values should be modified by the CTE:

(n;Hor) - (n:.) + (n~ARK) - (CTE)N npe + (CTE)N nDARK . (4-29)

Because the CTE is high (CTEN = 1), it is usually omitted from most equations.
However, CTE should be included for very large arrays. With TDI, the number
of photoelectrons and dark current electrons increases with the number of TDI
elements, NTDr:
(4-30)

While the dark current average value can be subtracted from the output to
provide only the signal due to photoelectrons, the dark current shot noise cannot.
Cooling the array can reduce the dark current to a negligible value and thereby
reduce dark current shot noise to a negligible level (see Section 3.6., Dark
Current, page 79).

4.2.2. RESET NOISE

The noise associated with resetting the sense node capacitor is often called
kTC noise. This is due to thermal noise generated by the resistance, R, within
the resetting FET. The Johnson noise current variance is

(4-31)
128 CCD ARRAYS, CAMERAS, and DISPLAYS

Because the resistance is in parallel with the sense node capacitor, the noise
equivalent bandwidth is .:lf = RC/4. Then
(4-32)

When expressed as equivalent electrons, the rms noise is

JkTC . (4-33)
(nREsE1) - - - e rms .
q

The value (nRESET) represents the uncertainty in the amount of charge remaining
on the capacitor following reset.

For a 0.01 pF capacitor at room temperature, kTC noise is about 40

electrons rms and a 0.2 pf capacitor produces about 126 e- rms. Reducing the
capacitance reduces the noise. This has an added benefit that the device output
gain conversion, Gq/C, increases. kTC noise may be significantly reduced with
correlated double sampling. While cooling reduces this noise source, cooling is
used primarily to reduce dark current. That is, cooling reduces dark current
noise exponentially whereas reset noise is reduced only by ...[T.

4.2.3. ON-CHIP AMPLIFIER NOISE

Amplifier noise consists of two components: 1/f noise and white noise. If
fKNEE is the frequency at which the 1/f noise equals the white noise, then the
amplifier noise density is

rms volts (4-34)

VON-CHIP - VAMP-NOISE ( 1 + I/KNEE)

The 1/f noise can be minimized through correlated double sampling.

 ARRAY PERFORMANCE 129

When discussing individual noise sources, noise is usually normalized to unit

bandwidth. When modified by electronic subsystems, the total system noise
power is

cr~rs - f S(jelec) 1Hsysifetec)i2 dfetec

0
volts ' (4-35)

where S(fe1ec) is the total noise power spectral density from all sources. The
subscript elec is used to emphasize that this is electrical frequency (Hz).
H,y,(felec) is the frequency response of the system electronics. The variable
H(fe1ec) is used by electronic circuitry designers and its magnitude is identical to
the MTF. The noise equivalent bandwidth is that bandwidth with unity value that
provides the same total noise power. Assuming the noise is white over the
spectral region of interest [S(fe~ec) = SJ, ~felec is

f Sifelec) IHsys Cfetec) 1 df.lec

2
(4-36)
11felec - _o_____- - - - -
8o

The bandwidth ~felec applies only to those noise sources that are white and
cannot be applied to 1/f noise. Although common usage has resulted in calling
~felec the noise bandwidth, it is understood that it is a power equivalency.

If !cLocK is the time between pixels, the bandwidth for an ideal sampled-data
system is

fcLOCK (4-37)
2

When equated to electrons,

(noN-CHIP)- ~q VON-CHIP-AMP-NOISE J11felec e- rms ·

(4-38)

As the clock frequency increases (for faster readout), <DoN-CHIP> increases.

130 CCD ARRAYS, CAMERAS, and DISPLAYS

4.2.4. OFF-CHIP AMPLIFIER NOISE

The off-chip amplifier noise is identical in form to the on-chip amplifier

noise (Equation 4-34). The 1/f knee value may be different for the two
amplifiers. If fKNEE is small, the equivalent number of noise electrons is

(noFF-CHIP)- G~l q VOFF-CHIP-AMP-NOISEJ!lfe/ec e- rms · (4-39)

4.2.5. QUANTIZATION NOISE

The analog-to-digital converter produces discrete output levels. A range of

analog inputs can produce the same output. This uncertainty, or error, produces
an effective noise given by

V rms, (4-40)

where VLSB is the voltage associated with the least significant bit. For an ADC
with N bits, VLSs = VMAxf2N. When expressed in equivalent electrons,
C LSB _
(nADC) - --- -- e rms . (4-41)
GG1 q {fi

When the ADC is matched to the amplifier output, VMAx corresponds to the full
well. Then

(4-42)

Ideally, (nAoc) is less than the noise floor. This is achieved by selecting a high
resolution ADC (large N).
 ARRAY PERFORMANCE 131

Example 4-7
TOTAL NOISE

An array has a noise floor of 20 electrons rms, well capacity of 50,000

electrons, and an 8-bit analog-to-digital converter. Neglecting shot noise and
pattern noise, what is the total noise?

The LSB represents 50,000/2 8 = 195 electrons. The quantization noise is

195/...[12 or 56.3 electrons rms. The total noise is

The array dynamic is range 2500:1 and the ADC only has 256levels. Here, the
ADC noise dominates the array noise. If a 12-bit ADC is used (4096levels), the
LSB represents 12.2 electrons and (nAod = 3.5 electrons rms. The total noise
in electrons is

4.2.6. PATTERN NOISE

Fixed pattern noise (FPN) refers to the pixel-to-pixel variation15 •16 that occurs
when the array is in the dark. It is primarily due to dark current differences.
Significant FPN can be caused by synchronous timing effects at high data rates.
It is a signal-independent noise and is additive to the other noise powers. PRNU
is due to differences in responsivity (when light is applied). It is a signal-
dependent noise and is a multiplicative factor of the photoelectron number.

PRNU be either be specified as a peak-to-peak value or a rms value

referenced to an average value. This average value may either be full well or
one-half full well value. That is, the array is uniformly illuminated and a
histogram of responses is created. The PRNU can either be the rms of the
histogram divided by the average value or the peak-to-peak value divided by the
average value. This definition varies by manufacturer so that the test conditions
must be understood when comparing arrays. For this text, the rms value is used.
132 CCD ARRAYS, CAMERAS, and DISPLAYS

Because dark current becomes negligible by efficiently cooling the array,

PRNU is the dominate pattern component for most arrays. As a multiplicative
noise, PRNU is traditionally expressed as a fraction of the total number of
charge carriers. This approach assumes that the detectors are operating in a
linear region with the only difference being responsivity differences. If U is the
fixed pattern ratio or nonuniformity, then
(4-45)

As with shot noise, CTEN should be added to the equation. Because CTEN = 1
for modest sized arrays, it is omitted from most equations. CTE should be
included for very large arrays. TDI devices, through their inherent averaging,
reduces PRNU by 1/NTDr· PRNU is usually supplied by the manufacturer. Off-
chip gain/level correction algorithms can minimize FPN and PRNU. For system
analysis, the corrected pattern noise value is used.

In principle, FPN and the noise floor can be reduced so that the system is
photon shot noise limited. This provides the theoretical limit:

(nsrs) MIN = {tl;. . (4-46)

However, all systems have some pattern noise. The plot of SNR versus
nonuniformity suggests a desired maximum level of acceptable nonuniformity .17
In Figure 4-13, the noise floor is considered negligible:

(4-47)

The total noise value (nsvs) increases when PRNU is excessive. Typically an
array is selected that has small PRNU. This value can be further reduced by off-
chip electronics. The final value should be sufficiently small so that it does not
contribute to the overall noise. Because cost increases as the PRNU decreases,
the optimum PRNU occurs at the knee of the curve. This occurs when the
photon shot noise is approximately equal to PRNU or U = 1/...[~. For worst
case analysis, the charge well capacity should be used for npe.
 ARRAY PERFORMANCE 133

Ul 10000
I
CJ)
z
0
a::
.....
0
1000
w
....1
w
w
CJ) ....... - .. photon shot noise
0z 100~--------~~--------~----------~
0.01 0.1 10
PRNU (percent)
Figure 4-13. System noise as a function of PRNU for two signal levels.
The noise floor is considered negligible. The noise floor will increase
(nsvs) and the location of the knee. Shot noise is independent of PRNU
(horizontal dashed lines). The value 1/.J~ is 0.4% and 0.3% when 11pe
is 50,000 and 100,000, respectively. These values represent the "knee"
in the curve.

4.2. 7. PHOTON TRANSFER

Although various noise sources exist, for many applications it is sufficient

to consider photon shot noise, noise floor, and PRNU. Here, the total array
noise is

(4-48)

or

(4-49)

Either the rms noise or noise variance can be plotted as a function of signal
level. The graphs are called the photon transfer curve and the mean-variance
curve, respectively. Both graphs convey the same information. They provide18 •19
array noise and saturation level from which the dynamic range can be calculated.
134 CCD ARRAYS, CAMERAS, and DISPLAYS

For very low photon fluxes, the noise floor dominates. As the incident flux
increases, the photon shot noise dominates. Finally, for very high flux levels,
the noise may be dominated by PRNU. As the signal approaches well saturation,
the noise plateaus and then drops abruptly at saturation (Figure 4-14). As the
charge well reaches saturation, electrons are more likely to spill into adjoining
wells (blooming) and, if present, into the overflow drain. As a result, the
number of noise electrons starts to decrease. Figure 4-15 illustrates the rms
noise as a function of photoelectrons when the dynamic range is 60 dB. Here
dynamic range is defined as NwELd < nFLOOR > . With large signals and small
PRNU, the total noise is dominated by photon shot noise. When PRNU is large,
the array noise is dominated by U at high signal levels.

Dark current shot noise only affects those applications where the signal-to-
noise ratio is low (Figure 4-16). At high illumination levels, either photon shot
noise or pattern noise dominate. Because many scientific applications operate in
a low signal environment, cooling will improve performance. General video and
industrial cameras tend to operate in high signal environments and cooling will
have little effect on performance. A full SNR analysis is required before
selecting a cooled camera.

'iii 1000
s
II)
z 100
~
1-
0
w
..J
w
w
II)
0 1~------~------~------~----~
z
10 100 1000 10000 100000
PHOTO ELECTRONS

Figure 4-14. Photon transfer curve. At low input signals, the device
noise is dominated by the noise floor. As the signal increases, the noise
increases due to photon shot noise and pattern noise. As the well reaches
saturation, there is a drop in noise electrons due to spill over and well
capacity limitation.
 ARRAY PERFORMANCE 135

enE 10000
.:.
(/)
z 1000
0 noise floor
a::
1- 100
0
w
..J
w 10
w
!/)
5
z
10 100 1000 10000 100000
PHOTOELECTRONS
(a)

enE 10000
.:.
!/) total noise
z 1000
0 noise floor
a::
1-
0
100
w
..J
w 10
w
!/)
5
z
10 100 1000 10000 100000
PHOTOELECTRONS
(b)
Figure 4-15. Photon transfer curves for (a) U = 2.5% and (b) U =
0.25%. The charge well capacity is 100,000 electrons and the noise
floor is 100 e- rms to produce a dynamic range of 60 dB. The noise floor,
photon shot noise, and PRNU have slopes of 0, 0. 5, and 1, respectively.
Dark noise is considered negligible. The drop at saturation is not shown
(see Figure 4-14).
136 CCD ARRAYS, CAMERAS, and DISPLAYS

Cii 1000 photon

E shot noise
.:.
1/)
z 100
0
0::
1-
0
w
..J 10
w noise floor
w
1/)
0 1
z
10 100 1000 10000 100000
PHOTOELECTRONS
(a)

Cii 1000 photon

s
1/)
shot noise
z 100
0
0::
1-
0 noise floor
w
..J 10
w
w dark current shot noise
1/)
0z 1
10 100 1000 10000 100000
PHOTOELECTRONS
(b)

Figure 4-16. Dark current shot noise effects. The noise floor is 10
electrons rms and PRNU is zero. (a) Dark current shot noise is 20e- rms
and (b) dark current shot noise is 2 e- rms. For large signals, photon shot
noise dominates the array noise. The drop at saturation is not shown
(Figure 4-14).
 ARRAY PERFORMANCE 137

Figure 4-17 illustrates the mean-variance technique. In the shot noise limited
region,

(4-50)

and

G q)
2
2 (4-51)
VNOISE -
(
C nPE •

Then the output gain conversion is

2
Gq VNOISE (4-52)
c VS/GNAL

or the OGC is equal to the noise variance divided by the mean value. It is
simply the slope of the shot noise in Figure 4-17.

1E+08
w
0
zc( 1E+06
~ noise floor
c( 1E+04
>
w
en 1E+02
0
z 1E+OO
10 100 1000 10000 100000 1000000

PHOTOELECTRONS

Figure 4-17a. Mean-variance plot which corresponds to Figure 4-15a.

The slopes for noise floor, photon shot noise, and PRNU are 0, 1, and
2, respectively.
138 CCD ARRAYS, CAMERAS, and DISPLAYS

1E+06
w
0
z
< 1E+04
ii: dark current
~
w
t/)
1E+02
noise floor
0
z 1E+OO
10 100 1000 10000 100000 1000000
PHOTOELECTRONS
Figure 4-17b. Mean-variance plots which corresponds to Figure 4-16a.
The slopes for noise floor, photon shot noise, and PRNU are 0, 1, and
2, respectively.

If both signal and noise are measured after the on-chip amplifier, the
measured photogenerated signal is VsiGNAL = G q npe/C and the noise is
Gq (4-53)
VNOISE - C (nsys) .
When VNOISE = 1, the signal is C/Gq. The shot noise intercept on the photon
transfer curve is 1/0GC (Figure 4-18). The OGC has units of ILV/electrons.
Similarly, if the values are collected after the ADC, when the shot noise is one,
the shot noise signal provides the output gain conversion in units of
DN/electrons.

- 10000
"'E
~ 1000
w

~
noise floor
100L---------------~~--------------
g 1/0GC
w 10
II) ••• • ·photon shot noise
0
z 1~~~~------~------~------
0.1 10 100 1000

SIGNAL VOLTAGE (pV)

Figure 4-18. Photon transfer curve. The noise floor is 250 e- and the
OGC is 5 ILVJe-. When the photon shot noise is one, the signal is
1/0GC. The drop at saturation is not shown (Figure 4-14).
 ARRAY PERFORMANCE 139

4.3. ARRAY SIGNAL-TO-NOISE RATIO

The array signal-to-noise ratio is

SNR- (4-54)

With negligible dark current shot noise,

(4-55)

The charge transfer efficiency factor, CTEN, should be added to r1pe in both the
numerator and denominator. For buried channel devices (the most popular)
CTEN is near one and is typically is omitted. Equation 4-55 suggests that the
maximum SNR occurs when nrE is a maximum. This occurs when the charge
wells are filled. However, as the charge wells fill, electrons can get close to the
surface even for buried channel devices. Thus as nrE approaches NwELL• CTE
can decrease. The best performance is a compromise between high SNR and
high CTE. High CTE is generally only a concern with large devices.

The SNR plots are similar to the photon transfer curves. At low flux levels,
the SNR increases with the signal (noise floor is constant). At moderate levels,
photon shot noise limits the SNR and at high levels, the SNR approaches 1/U.
In Figure 4-19, the SNR is expressed as a power ratio (in dB) which is 20
log(SNR). While the array may have a finite PRNU, it can be minimized off-
chip through an appropriate algorithm.

When photon shot noise dominates, the SNR depends only on the photon
flux level (SNR = ..fiLpe). In Figure 4-19, dynamic range is defined as
NwELd < nFwoR > . The instantaneous dynamic range, which includes all noise
sources, is the SNR. The theoretical maximum SNR is ..fNwELL· The SNR is
equal to the dynamic range only when the system is noise floor limited. This
occurs only for low signal values and small wells (less than about 10,000
electrons). For large well capacity arrays, the actual SNR can never reach the
value suggested by the dynamic range. Dynamic range is used to select an
appropriate analog -to-digital converter resolution (number of bits). This assures
that low contrast targets can be seen.
140 CCD ARRAYS, CAMERAS, and DISPLAYS

The array signal-to-noise ratio was derived for a large target that covers
many pixels. The noise rides on the signal. The camera SNR, which includes
all the noise sources, is provided in Section 6. 2. l. , Camera SNR. In Chapter 12,
Minimum Resolvable Contrast, the average noise associated with the target and
the background is considered.

60

,-
50
m 40
-
a:::
z
30
20
tn
10
0
10 100 1000 10000 100000
PHOTOELECTRONS
(a)

60
50

,m 40
-z
a::: 30
20
tn
10
0
10 100 1000 10000 100000
PHOTOELECTRONS
(b)

Figure 4-19. SNR as a function of noise for (a) U = 2.5% and (b) U =
0.25%. PRNU, photon shot noise, and the noise floor have slopes of 0,
10, and 20, respectively. Dark current shot noise is considered
negligible.
 ARRAY PERFORMANCE 141

4.4. CORRELATED DOUBLE SAMPLING

After a charge packet is read, the floating diffusion capacitor is reset before
the next charge packet arrives. The uncertainty in the amount of charge
remaining on the capacitor following reset appears as a voltage fluctuation (see
Section 4.2.2., Reset Noise, page 127).

Correlated double sampling (CDS) assumes that the same voltage fluctuation
is present on both the video and reset levels. That is, the reset and video signals
are correlated in that both have the same fluctuation. The CDS circuitry may be
integrated into the array package and this makes processing easier for the end
user. For this reason it is included in this chapter.

CDS may be performed 20 •21 in the analog domain by a clamping circuit with
a delay-line processor or by the circuit illustrated in Figure 4-20. When done in
the digital domain, a high speed ADC is required that operates much faster than
the pixel clock. Limitations of the ADC may restrict the use of CDS in some
applications. The digitized video and reset pulses are subtracted and then clocked
out at the pixel clock rate.

The operation of the circuit is understood by examining the timing signals

(Figure 4-21). When the valid reset pulse is high, the reset switch closes and the
signal (reset voltage) is stored on capacitor CR. When the valid video pulse is
high, the valid video switch closes and the signal (video voltage) is stored on
capacitor Cv. The reset voltage is subtracted from the video voltage in a
summing amplifier to leave the CDS corrected signal. CDS also reduces
amplifier 1/f noise.

Valid
Video

I Cv

Figure 4-20. Correlated double sampling circuitry. The switches close

according to the timing shown in Figure 4-21. (After Reference 20).
142 CCD ARRAYS, CAMERAS, and DISPLAYS

amplifier
(a) output

valid reset
(b)
valid video

(c) amplifier output

after CDS

Figure 4-21. Correlated double sampling. (a) Amplifier output signal

before CDS, (b) timing to measure video and reset levels, and (c) the
difference between the video and reset levels after CDS. The dashed
lines indicate the variation in the reset level. The signal level has been
inverted compared to Figure 4-12 (page 124).

4.5. FRAME RATES

For a fixed output clock rate, the time to read an array increases as the array
size increases. Very large arrays cannot be read in real time. Let the active
array size be m x n pixels. Let there be an additional NREAD pixels before the
sense node. Suppose the clock operates at fcLocK pixels/s. The minimum time to
clock out the array is

tARRAY
_ [mn + NREAD
IeweK
I (4-56)

and the theoretical frame rate (Figure 4-22) is

(4-57)
 ARRAY PERFORMANCE 143

The maximum frame rate is limited by NREAD· The actual frame rate may be less
due to electron mobility, on-chip amplifier bandwidth, or clock rate limitations.
For some CCDs and all CIDs and CMOS devices, subarrays (also called
windows) can be read out. As the subarray size decreases, the frame rate
increases, With increased frame rates, the integration time decreases and then
more illumination is required to maintain a specific signal-to-noise ratio.

w 200
1-
~
w 150
:::!!
~
LL 100
w
>
i= 50
:5
w
0:: 0
0 20 40 60 80 100
RELATIVE IMAGE SIZE

Figure 4-22. Frame rate a function of array size.

4.6. DEFECTS

Large arrays sometimes contain defects. Table 4-5 provides representative

definitions. These definitions vary by manufacturer. Arrays with a few defects
are more expensive than arrays with a large number. Depending upon the
application, the location of the defects may be important. For example, the
center ("sweet spot") must be fully operational with increasing defects allowed
as the periphery is approached.

Defects are regions of reduced sensitivity. They can also be regions of

increased dark current. For scientific application such as spectroscopy, the
variation in dark current can affect results - particularly if long integration times
are used. These pixels may be labeled as "hot" or "warm." Hot and warm
detectors do not follow the same cooling curve as "normal" detectors. 22
144 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 4-5
ARRAY DEFECTS

DEFECTS REPRESENTATIVE DEFINITION

A pixel whose output deviates by more than 6%

Point defect compared to adjacent pixels when illuminated to
70% saturation.
Pixels with extremely htgh output voltages.
Hot point defects Typically a pixel whose dark current is 10 times
higher than the average dark current.

Pixels with low output voltage and/or poor

responsivity. Typically a pixel whose output is one-
Dead pixels
half of the others when the background nearly fills
the wells.

A trap interferes with the charge transfer process

Pixel traps and results in either a partial or whole bad column
(either all white or all dark).

Many (typically 10 or more) point defects in a

Column defect
single column. May be caused by pixel traps.

Cluster defect A cluster (grouping) of pixels with point defects.

4.7. REFERENCES
1. J. Janesick, "Introduction to Scientific Charge-Coupled Devices," Short Course SC-70 presented
at the IS&T/SPIE's Symposium on Electronic Imaging: Science & Technology, San Jose CA
(1995). J. Janesick presents this course at numerous symposia.
2. R. H. Dyck, "VLSI Imagers," in VLSI Electronics: Microstructure Science, Volume 5, N. G.
Einspruch, ed., pp. 594, Academic Press, New York (1985).
3. R. H. Dyck, "Design, Fabrication, and Performance of CCD Imagers," in VLSI Electronics:
Microstructure Science, Volume 3, N. G. Einspruch, ed., pp. 70-71, Academic Press, New
York (1982).
4. A. J.P. Theuwissen, Solid-State Imaging with Charge-Coupled Devices, pp. 205-210, Kluwer
Academic Publishers, Dordrecht, The Netherlands (199:5).
5. The Leader in Spectroscopic Detection, pp. 33-34, Princeton Instruments Catalog, 3660
Quakerbridge Road, Trenton, NJ 08619 (April 1997).
6. The Leader in Spectroscopic Detection, pp. 29-30, Princeton Instruments Catalog, 3660
Quakerbridge Road, Trenton, NJ 08619 (April 1997).
7. C. H. Sequin and M. F. Tompsett, Charge Transfer De1ices, Academic Press, New York, NY
(1975).
 ARRAY PERFORMANCE 145

8. E. S. Yang, Microelectronic Devices, McGraw-Hill, NY (1988).

9. E. L. Dereniak and D. G. Crowe, Optical Radiation Detectors, John Wiley and Sons, New
York, NY (1984).
10. D. G. Crowe, P. R. Norton, T. Limperis, and J. Mudar, "Detectors," in Electro-Optical
Components, W. D. Rogatto, pp. 175-283. This is Volume 3 of The Infrared & Electro-Optical
Systems Handbook, J. S. Accetta and D. L. Shumaker, eds., copublished by Environmental
Research Institute of Michigan, Ann Arbor, MI, and SPIE Press, Bellingham, WA (1993).
11. J. R. Janesick, T. Elliott, S. Collins, M. M. Blouke, and J. Freeman, "Scientific Charge-
coupled Devices," Optical Engineering, Vol. 26(8), pp. 692-714 (1987).
12. T. W. McCumin, L. C. Schooley, and G. R. Sims, "Charge-coupled Device Signal Processing
Models and Comparisons," Journal of Electronic Imaging, Vol. 2(2), pp. 100-107 (1994).
13. M. D. Nelson, J. F. Johnson, and T. S. Lomheim, "General Noise Process in Hybrid Infrared
Focal Plane Arrays," Optical Engineering, Vol. 30(11), pp. 1682-1700 (1991).
14. T. S. Lomheim and L. S. Kalman, "Analytical Modeling and Digital Simulation of Scanning
Charge-Coupled Device Imaging Systems," in Electro-Optical Displays, M. A. Karim, ed., pp.
551-560, Marcel Dekker, New York (1992).
15. J. M. Mooney, "Effect of Spatial Noise on the Minimum Resolvable Temperature of a Staring
Array," Applied Optics, Vol. 30(23), pp. 3324-3332, (1991).
16. J. M. Mooney, F. D. Shepherd, W. S. Ewing, J. E. Murguia, and J. Silverman, "Responsivity
Nonuniformity Limited Performance of Infrared Staring Cameras," Optical Engineering, Vol.
28(11), pp. 1151-1161 (1989).
17. G. C. Holst, Electro-Optical Imaging System Analysis, pp. 360-362 and pg. 400-401, JCD
Publishing, Winter Park, FL (1995).
18. J. E. Murguia, J. M. Mooney, and W. S. Ewing, "Diagnostics on a PtSi Infrared Imaging
Array" in Infrared Technology XIV, I. Spiro, ed., SPIE Proceedings Vol. 972, pp. 15-25
(1988).
19. B. Stark, B. Nolting, H. Jahn, and K. Andert, "Method for Determining the Electron Number
in Charge-coupled Measurement Devices," Optical Engineering, Vol. 31(4), pp. 852-856
(1992).
20. A. J. P. Theuwissen, Solid-State Imaging with Charge-Coupled Devices, pp. 228-231, Kluwer
Academic Publishers, Dordrecht, The Netherlands (1995).
21. T. W. McCumin, L. C. Schooley, and G. R. Sims, "Signal Processing for Low-light-level,
High-precision CCD Imaging," in Camera and Input Scanner Systems, W. Chang and J. R.
Milch, eds., SPIE Proceedings Vol. 1448, pp. 225-236 (1991).
22. The Leader in Spectroscopic Detection, pp. 31-32, Princeton Instruments Catalog, 3660
Quakerbridge Road, Trenton, NJ 08619 (April 1997).
5
CAMERAS
Solid state camera terminology is a mix of photographic, sensor, image
processing, and television terminology. As solid state cameras enter emerging
technologies, camera manufacturers adopt the marketplace terminology and add
it to their specifications.

Many specifications include the known problems associated with vacuum

image tubes such as image bum-in, distortion. or microphonics. These effects
are not present in solid state devices. References to these vacuum tube
disadvantages probably will disappear in the future as these tubes disappear.

Video standards were originally developed for televisions applications. The

purpose of a standard is to ensure that devices offered by different
manufacturers will operate together. The standards apply only to the
transmission of video signals. It is the camera manufacturer's responsibility to
create an output signal that is compatible with the standards. Similarly, it is the
display manufacturer's responsibility to ensure that the display can recreate an
adequate image.

Some agencies involved with standards include the Institute of Electrical and
Electronic Engineers (IEEE), Electronics Industry Association (EIA), EIA-Japan
(EIAJ), Joint Electron Devices Engineering Council (JEDEC), Society of
Motion Picture and Television Engineers (SMPTE), International Electro-
technical Commission (IEC), Technical Center of the European Broadcast Union
(EBU), Advanced Television Systems Committee (ATSC), Comite Consultatif
International des Radiocommunications (CCIR) (now ITU-R), and International
Telecommunications Union-Radiocommunications(ITU-R). Major manufacturers
such as Panasonic, Sony, and JVC influence the standards.

There is no industry wide definition of low light level imaging. To some, it

is simply a solid state camera that can provide a usable image when the lighting
conditions are about 1 lux. Lower illumination values may be possible by
cooling the device. To others it refers to an intensified camera. Intensified
cameras are sometimes called low light level television (LLLTV) systems.

Because solid state devices are designed for specific applications, the array
and camera use the same terminology: progressive scan chips are used in

146
 CAMERAS 147

progressive scan cameras, TDI arrays are used in TDI cameras, etc. Through
appropriate clocking, progressive scan cameras can also provide composite video
and be used with consumer displays. Scientific cameras can have both analog
and digital outputs. The corresponding sections in Chapter 3, Solid State Arrays,
should be read with the specific cameras listed here.

In this chapter, camera designs are separated into consumer/broadcast and

industrial/scientific applications. The major features of each design are listed in
the appropriate sections. However, as technology matures, the distinctions blur.
That is, the same camera may be used for a variety of applications. New
cameras offer variable frame rates and data rates that can be selected by the
user. Therefore, the reader must review all sections before selecting a specific
design. Chapter 6, Camera Performance, discusses minimum signal, maximum
signal, dynamic range, and signal-to-noise ratio.

Both monochrome and color cameras are quite complex as evidenced by the
1478-page Television Engineering Handbook, K. B. Benson, ed., McGraw-Hill,
New York (1986). This chapter is an overview of solid state camera operation.
It highlights those features and specifications that are most often reported in
camera data sheets. Together with the preceding chapters, it provides the
necessary information to compare and analyze camera systems.

Selecting the right camera to meet specific requirements may be difficult.

For consumer video, cost is very important. For broadcast systems, image
quality and color are important. Scientific cameras offer low noise and high
resolution. Issues not addressed include environmental conditions such as
temperature, humidity, altitude, shock, vibration, the "-ilities" (e.g., reliability,
maintainability, etc.) and physical attributes (e.g., weight and size).

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

5.1. CAMERA OPERATION

Figure 5-1 illustrates a generic solid state camera. The lens assembly images
the light onto the detector array. General purpose consumer cameras typically
have an infrared blocking filter to limit the camera's spectral response to the
visible spectral band. Depending upon the application (military, industrial, or
scientific), the camera may or may not have this filter. Color cameras have a
spectral response limited to the visible region. Cameras may provide digital
image processing and these cameras have an internal ADC. Analog cameras will
not have an internal ADC. However, they still spatially sample the scene.
148 CCD ARRAYS, CAMERAS, and DISPLAYS

~
0 IE
Optics
_

CCD Array
Gain/Level
Matchedto
..______.ADC
ADC

ADC

Digitlzatlon
.

- ' Image Gamma

- - Post-
Analog

- Processing Correction

Digital Camera
DAC Reconstruction
Filter
f-- Output

Output

Image Reconstruction Post-Reconstruction

Figure 5-1. Generic functional block diagram. Not all components exist
in every camera. The gamma correction circuitry is appropriate when the
video is to be viewed on a CRT-based display. Analog cameras will not
have an internal ADC.

A single chip color camera typically has an unequal number of red, green,
and blue detectors resulting in different sampling frequencies for the different
colors (discussed in Section 8.3.2., Detector Array Nyquist Frequency). Because
the detectors are at discrete locations, target edges and the associated ambiguities
are different for each color. A black-to-white edge will appear to have colors at
the transition and monochrome periodic imagery may appear to have color.
Aliasing is extremely annoying in color systems. A birefringent filter placed
between the lens and the array reduces aliasing (discussed in Section 10. 3 .4. ,
Optical Anti-Alias Filter). Any method to reduce aliasing generally reduces the
overall MTF.

While aliasing is never considered desirable, it is accepted in scientific and

military monochrome applications where the MTF must be maintained. Aliasing
is reduced to an acceptable level in consumer color sensors. For specific
applications such as medical imaging, it may be considered unacceptable. In
medical imagery, an aliased signal may be misinterpreted as a medical
abnormality requiring medication, hospitalization, or surgery.
 CAMERAS 149

Correlated double sampling reduces reset noise and minimizes 1/f noise.
CDS circuitry may be integrated into the array package and this makes
processing easier for the end user.

Image processing or image reconstruction reformats the data into signals and
timing that are consistent with transmission and display requirements. The
advantage of a digital camera is that the output can be directly fed into an image
processor, machine vision system, or digital data capture system. If using an
analog output camera for machine vision, the video signal must pass through an
external analog -to-digital converter. These multiple conversions may modify the
imagery (discussed in Section 8.4., Aliasing in Frame Grabbers and Displays).
Thus, staying within the digital domain ensures maximum image fidelity.

To present a linear transformation from a scene to an observer, a gamma

correction (discussed in Section 5.3.3., Gamma Compensation) algorithm
provides compensation for the monitor's nonlinear response. The concern with
gamma should be limited to applications where cathode ray tube (CRT) based
displays are used. As the display industry moves from CRTs to flat panel
displays, gamma correction is not required. However, gamma provides some
image enhancement and therefore will probably be used with flat-panel displays.
Gamma correction is generally not preferred for machine vision or scientific
applications.

The digital-to-analog converter (DAC) and sample-and-hold circuitry

converts the digital data into an analog stair-step signal. The post-reconstruction
filter removes the stair-step appearance and creates a smooth analog signal.
After the post-reconstruction filter, the spatial scene sampling by the detectors
is barely noticeable.

5.2. VIDEO FORMATS

In 1941, the Federal Trade Commission (FCC) adopted the 525-line, 30-Hz
frame standard for broadcast television that was proposed by the NTSC
(National Television Systems Committee). The standard still exists today with
only minor modifications. Although originally a monochrome standard, it has
been modified for color transmission. Amazingly, color video fits within the
video electronic bandwidth originally set aside for black-and-white television.
NTSC initially selected a 60-Hz field rate. It was later changed to 59.94 Hz for
color systems to avoid interactions (beat frequencies) between the audio and
color subcarrier frequencies. The monochrome standard is often called EIA 170
(originally called RS 170) and the color format is simply known as NTSC
150 CCD ARRAYS, CAMERAS, and DISPLAYS

(originally called EIA 170A or RS 170A). The timing for these two standards is
nearly the same. 1

Worldwide, three color broadcast standards exist: NTSC, PAL, and

SECAM. NTSC is used in the United States, Canada, Central America, some
of South America, and Japan. The phase alteration line (PAL) system was
standardized in most European countries in 1967 and is also found in China.
Many of the PAL standards were written by the CCIR. As a result, PAL is
sometimes called CCIR. In 1967, France introduced SECAM (Sequentiel colour
avec memoire). SECAM is used in France, Russia, and the former Soviet bloc
nations. Reference 2 provides a complete worldwide listing. These standards are
incompatible for a variety of reasons.

Video transmission bandwidth restrictions and the perception of good

imagery on a display dictated the standard formats. That is, camera design is
based upon the perception of displayed information. The interlace and scan times
were selected to minimize electronic bandwidth and still provide a cosmetically
pleasing image to the viewer with no flicker effects. Two fields are interwoven
on the display. The eye's persistence time combined with the CRT phosphor
persistence blends the two separate images into a perceived full image.

In the NTSC and EIA 170 standards, a frame is composed of 525 scan lines
separated into two sequential fields (even and odd). Because time is required for
vertical retrace, only 485 lines are displayed. There are 242.5 active lines per
field. Because field 1 ends in a half-line, it is called the odd field. Figure 5-2
illustrates how the information is presented on a display. For commercial
television, the aspect ratio (horizontal to vertical extent) is 4:3.

When a camera is designed to be compatible with a transmission standard,

the manufacturer uses the broadcast terminology as part of the camera
specifications. Similarly, consumer display manufacturers use the same
terminology.
 CAMERAS 151

Second Half
of Line 243
Line 244, Field 2-.__ ~/
Line 245, Field 2 ""'- r--:.::.. ..... .~ . ..-- f.-- Line 1, Field 1
_.... -:-./ I;v
;,..- Line 2, Field 1
Line 246, Field 2 \ I' ~::-~
..... Line 3, Field 1
f\-
=
... , '
.......
-- ./
'
''
''
Line 485, Field 2 \ .........
First Half of ''
Line 243

--------------
I\
- -- - ''
~·
~ Line 242, Field 1

Figure 5-2. Presentation of EIA 170 video on a display. Standard timing

is defined for video transmission only. Solid state arrays may not read
out data in this precise order. The camera image processing algorithm
formats the data into a serial stream that is consistent with the video
standard.

5.2.1. VIDEO TIMING

Table 5-1 provides the relevant parameters for the monochrome and color
video standards. The standards specify the line frequency and blanking period.
The scanning frequency is the frame rate multiplied by the number of lines. The
total line time is calculated from the line frequency (e.g., 1115,750 = 63.492/Ls)
and the active line time is the total line time minus the horizontal blanking
period. The blanking period has a tolerance of about ±0.25% so that the active
time line also varies slightly. Table 5-l provides the minimum active time line
where the maximum blanking period has been selected.
152 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 5-l
STANDARD VIDEO TIMING
FRAME LINES SCANNING TOTAL MINIMUM
ACTIVE
FORMAT RATE PER FREQIIENCY LINE ACTIVE LINE
LINES
(Hz) FRAME (kHz) TIME (}ts) TIME (}ts)

EIA 170 30 525 485 15.750 63.492 52.092

NTSC 29.97 525 485 15.7343 63.555 52.456

PAL 25 625 575 15.625 64.0 51.7

SECAM 25 625 575 15.625 64.0 51.7

For the monochrome video transmission (See Section 5 .2.2., Broadcast/Non-

broadcast Design), the picture element was originally defined 3 as the smallest
area being delineated by an electrical signal. f';ote that this is the definition of
a resel (analog element) and it is referenced to an imaginary pixel. The impulse
response to an ideal band-limited system is

(5-1)

where fc is the video bandwidth cutoff frequency. The pixel length is defined as
the minimum distance between two impulses such that the two are still
discernible. This occurs when the maximum of one falls on the first minimum
(zero) of the next. The separation of these two vertical lines in time is Llt =
112fc. This is the smallest possible dimension in a linear system. For the video
standard, the number of resels ("pixels") per line is

active line time 2)

N - - 2fc(active line time) . (5-
!H

The nominal4 video bandwidths of NTSC, PAL, and SECAM are 4.2, 5.5,
and 6 MHz, respectively. Table 5-2 lists the nominal number of horizontal
elements per line. The quoted resolution (number of horizontal pixels) varies in
the literature and depends upon the precise bandwidth and active line time
selected by the author. Multiplying the pixel number by the number of active
lines yields the elements per frame.
 CAMERAS 153

Table 5-2
NOMINAL NUMBER OF VIDEO RESELS ("PIXELS")
Based upon electrical transmission bandwidth only
NOMINAL ACTIVE NUMBER of NUMBER
ELEMENTS
FORMAT BANDWIDTH LINE TIME HORIZONTAL of ACTIVE
PER FRAME
(MHz) ()<s) ELEMENTS LINES

EIA 170 4.2 52.09 437 485 212,000

NTSC 4.2 52.45 440 485 213,000

PAL 5.5 51.7 569 575 327,000

SECAM 6.0 51.7 620 575 356,500

The full bandwidth is somewhat wider than the nominal value but high
frequency components may be attenuated. For example, NTSC has a full
bandwidth of approximately 4.5 MHz. The total bandwidth allocated for sound,
chrominance, and luminance is 6, 8, and 8MHz for NTSC, PAL, and SECAM,
respectively. Using the allocated bandwidth, the maximum number of horizontal
pixels is 625 for NTSC and 827 for PAL and SECAM.

Current camera design philosophy matches the number of detectors to the

number of pixels supported by the video bandwidth. Based upon electrical
bandwidth considerations, it appears that the number of horizontal detectors
should range from 440 to 625 for NTSC, 560 to 827 for PAL, and 620 to 827
for SECAM compatible cameras.

The actual number of detector elements depends upon the manufacturer's

philosophy. Image processing algorithm complexity is minimized when the
spatial sampling frequency is equal in the horizontal and vertical directions. The
4:3 aspect ratio suggests that there should be 4/3 times more detectors in the
horizontal direction compared to the vertical direction. If the number of elements
in the vertical direction is simply equal to the number of active scan lines, then
the number of horizontal detectors should be about 646 for NTSC, and 766 for
PAL and SECAM.

The number of pixels per line is often chosen so that the horizontal clock
rate is an integer multiple of the color subcarrier. For NTSC, 768 pixels equate
to four times the NTSC color subcarrier (see Section 3 .11.3., Number of
Detectors, page 97).
154 CCD ARRAYS, CAMERAS, and DISPLAtS

This larger number provides more sampling for each color and reduces color
aliasing. For single chip color cameras, resolution is more difficult to define.
The primaries or their complements spatially sample the scene unequally
(discussed in Section 8.3.2., Detector Array Nyquist Frequency). If the chip
contains 768 detectors horizontally, then the number of green, red, and blue
detectors is typically 384, 192, and 192, respectively. Because the eye is most
sensitive to luminance values and the green component represents luminance,
then the green resolution may be taken as the camera resolution. Most chips
contain more green detectors and therefore the resolution is more than just one-
third of an equivalent monochrome system. Through data interpolation, any
number of output data points can be created. This number should not be
confused with the camera spatial resolution. Single chip color cameras cannot
provide the resolution listed in Table 5-2.

To reduce cost, a camera may contain fewer detector elements. Here, the
full bandwidth of the video standard is not exploited. The imagery of these
cameras will be poorer compared to those cameras that fully use the available
bandwidth. Recall that the standard only specifies the timing requirements and
a maximum bandwidth. The timing ensures that the camera output can be
displayed on a standard display.

For the complete system, the horizontal resolution may be limited by the
detector array, bandwidth of the video standard, or by the display. Display
resolution is discussed in Section 7.5., Resolution. It is common for the display
to limit the overall system resolution.

For NTSC and EIA 170 compatible arrays, it is common to have about 480
detectors in the vertical direction. The image formatting algorithm inserts blank
lines (all black) to meet video standards (485 lines). Because many displays
operate in an overscan mode, the blank (black) lines are not noticed.

Resolution is often expressed in TV lines. This visual psychophysical

measure was developed to specify the resolution of monitors. Because of phasing
effects, the number of individual horizontal lines that can be perceived, on the
average, is less than the number of scan lines. The ratio of the average number
to the total number is the Kell factor and is assumed to be 0. 7. Thus, the
vertical resolution is 340 lines for NTSC and 402 lines for PAL and SECAM.
This vertical resolution applies to the monitor capability and not the camera.
However, the camera output must be displayed on a monitor so that the Kell
factor is sometimes included in camera specifications.
 CAMERAS 155

For higher vertical resolution, more lines are required. EIA343A (originally
RS 343A) was created 6 for high resolution closed circuit television cameras
(CCTV). Although the standard encompasses equipment that operates from 675
to 1023 lines, the recommended values are 675, 729, 875, 945, and 1023 lines
per frame (Table 5-3). The military primarily uses 875 lines/frame but it
requires "standards conversion" boxes that can record and present imagery on
EIA 170 compatible equipment. EIA 343A is still a monochrome standard.

Table 5-3
RECOMMENDED EIA 343A VIDEO FORMATS
SCANNING ACTIVE
ACTIVE LINE
LINES/FRAME FREQUENCY LINE
LINES TIME p.s
(kHz) TIME p.s

675 624 20.25 49.38 42.38

729 674 21.87 45.72 38.72

875 809 26.25 38.09 31.09

945 874 28.35 35.27 28.27

1023 946 30.69 32.58 25.58

5.2.2. BROADCAST/NON-BROADCAST DESIGN

The EIA 170 standard was developed for the broadcasting of video signals
over limited bandwidth channels. The FCC dictated that the bandwidth cannot
exceed 6 MHz. To avoid flicker on the television receiver, a 2:1 interlace
concept was adopted. The NTSC standard was constrained to the same
bandwidth limitation.

For compatibility, all broadcast cameras and television receivers are

designed to the timing and bandwidth of the broadcast standard. Using a wider
bandwidth receiver does not improve image quality or resolution because it is
limited by the 6 MHz bandwidth. An inexpensive receiver may have reduced
bandwidth (as do many VHS recorders) and this reduces image quality.
However, to function, this "lower quality" receiver still must operate under the
timing requirements of the video standard.

Table 5-2 (page 153) listed the number of video resels supported by the
video standard. If the camera has more pixels, then the received image
resolution will be limited by the standard's bandwidth. If the camera has fewer
156 CCD ARRAYS, CAMERAS, and DISPLAYS

pixels, then the camera's resolution limits image quality. By using image
compression, high definition television cameras can effectively transmit more
resels than that suggested in Table 5-2. The number of transmitted resels
remains the same (constrained by bandwidth). It is through coding (e.g., image
compression) that the resels contain more information.

Because EIA 170 and NTSC are so pervasive, nearly all cameras are built
to these video standards. This permits saving imagery on off-the-shelf video
recorders and displaying it on just about any monitor. However, for non-
broadcast operation, it is not necessary to adhere to the 6 Mhz bandwidth
limitation. The only requirement is that the timing be identical to the standard
timing. A high resolution monitor (wide bandwidth) can display the imagery
from a camera that has more pixels and a wide bandwidth recorder can store
that imagery.

Departure from the standard video timing requires special monitors and
special recorders. Already digital recorder and digital monitors exist. Computer
monitors now operate in the progressive scan mode. To avoid flicker, the frame
rate exceeds 70Hz. The frame rate is determined by the video drive electronics
and is not generally user-selectable.

To see an image on a monitor, the camera's timing must be consistent with

the monitor requirements. Because the video standards are popular, most
cameras adhere to the appropriate timing. With progressive-scan cameras, the
camera's internal electronics must reformat the data into an interlace mode. If
the camera's output is fed directly into a computer, timing is no longer a
concern. If digital data are transferred, only a sync signal is necessary to
indicate the start of the data. If analog data are transferred, then the frame
capture board should have a resolution that is greater than the camera's. That
is, there should be more datels than pixels.

5.2.3. COMPONENT/COMPOSITE SIGNALS

The term composite is used to denote a video signal that encodes luminance,
color, and timing information onto a single channel. It can be transmitted on a
single wire such as cable television. NTSC, PAL, and SECAM are composite
video signals. The signal must be decoded to create an image.

Component analog video (CAV) carries each part on a separate wire. RGB,
or their complementary values are carried on three separate cables. It is common
for the G to be a composite line and carry the sync signals. Component videos
 CAMERAS 157

do not require encoding and therefore there is less chance that there will be
color or luminance cross coupling. This implies that component imagery will
produce a higher quality image.

As equipment is miniaturized (e.g., hand-held video recorders) and

resolution increased (e.g., HDTV), there seems to a proliferation of
recommended standards. 7 Most standards deal with color encoding (chrominance
encoding). The luminance channel is identical with the green channel and is
labeled Y. The other signals are color and the exact composition depends upon
the standard. SECAM uses the primaries whereas the other formats use the
complements. The chrominance values are related through matrices but the
signals are not compatible due to different line frequencies and sync signals.
Because the eye is most sensitive to luminance, the luminance channel has the
widest bandwidth and the chrominance channels have smaller bandwidths.

Matrixing can convert the RGB into any color coordinate system. With the
ease of digital implementation, multiple outputs are available from many
cameras. That is, a camera may offer RGB, NTSC, and Y/C outputs to provide
flexibility to the user.

5.2.4. IRE UNITS

The Institute of Radio Engineers (IRE) assigned 140 IRE units to the peak-
to-peak composite signal (Figure 5-3). This provides 100 IRE units for the
active image signal and 40 IRE units for blanking. They assumed that the video
is one volt peak-to-peak when terminated by 750. Thus, each IRE unit is equal
to 1/140 = 7.14mV.

To avoid seeing the retrace, NTSC introduces a small setup. With most
systems, the video is usually clamped at the blanking level and the video signal
with setup does not contain a reference black signal. That is, the video does not
contain a signal to indicate the black level. This voltage only occurs when there
is a black object within the field-of-view. The blanking level (retrace) is
sometimes called "blacker-than-black" because it is 7.5 IRE units below the
black level. With the signal standardized to 1 V peak-to-peak, the setup reduces
the active video to 0.66 v.
 158 CCD ARRAYS, CAMERAS, and DISPLAYS

Max Signal Level

/ (Pure White)
10

100 IRE

7.5 IRE "Set-up"

~
\
Blanking/
Level t
40 IRE
Pure
Black

~ ...____Sync Pulse

I
Sync Tip

Figure 5-3. Definition of IRE units. The active video represents a gray
scale divided into 10 equal steps. NTSC assigns 7.5 IRE units to setup.

By convention, the blanking level is assigned a value of 0 v so that the

active video is positive while the sync pulse is negative. The peak-to-peak
amplitude remains at 1 V. As new standards are introduced, 7 the voltage levels
associated with IRE units change (Table 5-4). The composite signal with setup
(NTSC) can introduce ambiguity in signal level because there is no fixed black
reference. Without setup, radiometrically correct imagery is ensured and black-
to-white measurements can be performed with higher accuracy. SMPTE and
EBU have recommended the non-NTSC format for color component video.

When a video signal is expressed as a percentage, it is numerically equal to

the IRE units. 80% video is the same at 80 IRE units. For example, 80% video
is (0.8)(0.714) = 571 mV with the NTSC standard.
 CAMERAS 159

Table 5-4
VIDEO STANDARDS
VOLTS IRE UNITS
FORMAT Sync Synch
Setup Peak Blanking Setup Peak
tip tip

NTSC -0.286 0.0536 0.714 -40 0 7.5 100

Composite
without -0.286 - 0.714 -40 0 - 100
setup

Non-
-0.300 - 0.700 -40 0 - 100
NTSC

5.2.5. DIGITAL TELEVISION

Digital image processing is already present in most transmitters and

receivers. It seems natural to transmit digitally. The major advantage is that
digital signals are relatively immune to noise. Either a bit is present or it is not.
A digital voltage signal, which may have been reduced in value and had noise
added, can be restored to its full value with no noise in the receiver. An all-
digital system avoids the multiple analog-to-digital and digital-to-analog
conversions that degrade image quality.

Digital signal "transmission" was first introduced into tape recorders.

Because a bit is either present or not, multiple generation copies retain high
image quality. In comparison, analog recorders, such as VHS, provide very
poor quality after just a few generations. The first digital recorder used a format
called D-1 and it became known as the CCIR 601 component digital standard.
Digital recording formats 8 now include D-1, D-2, D-3, D-4, and D-5.

The nominal 4.2-MHz bandwidth required by NTSC is not a continuous

spectrum but consists of many bands due to the interaction of the line frequency
(59.94 Hz) and scan frequency (15.7343 KHz). It is this discrete nature that
allows the interleaving of chrominance, luminance, and audio within the
allocated bandwidth. A comb filter, present in the receiver, separates the signals
to minimize the cross talk that appears as aliased color. The composite video
signal can be equated to a signal whose sampling frequency is 3.58 MHz for
NTSC and 4.43 MHz for PAL. SECAM has different effective sampling rates
for each color difference: 4.25 MHz and 4.41 MHz. These are the subcarrier
frequencies. 9
160 CCD ARRAYS, CAMERAS, and DISPLAYS

The most popular sampling frequency is four times the subcarrier frequency
and is simply called 4f,c sampling. It is equal to 14.3 MHz for NTSC and 14.7
MHz for PAL. This allows the post-reconstruction filter to have a more gradual
cutoff with less ripple. This filter is easier to design and is less expensive.
Because the data exist at discrete frequencies, strict adherence to the Nyquist
criterion (discussed in Chapter 8, Sampling Theory) is not required.

In component video there is no subcarrier. A sampling frequency of 13.5

MHz has been selected to accommodate both 525 and 625 line formats. The
luminance channel is sampled at 13.5MHz and the two color difference channels
are sampled at 6. 75 MHz each. Recall that there is less detail in the color
channels so they are sampled at a lower rate. If 13.5 MHz is four times the
frequency of some imaginary subcarrier, then 6.75 MHz is twice that same
imaginary frequency. This is known as 4:2:2 digital component television. If the
three signals are multiplexed into a serial stream, the new sample rate is 27 MHz
(13.5 + 6.75 + 6.75 = 27). In the multiplexed form, component signals can be
transmitted on a single cable.

An image, whose amplitude is digitized into 8 bits, can provide a

cosmetically acceptable image for consumer camcorders. However, nonlinear
circuitry is required to minimize the visibility of quantization contours at low
signal levels. Otherwise, more bits are necessary. Photo quality imagery requires
12 or more bits. For composite signals, the sync pulse must also be digitized.
The actual number of bits available to the imagery is conservatively less. CCIR
601 assigns 16 DN to the black level and 235 DN to the white level. The
remaining values (1 to 15 and 236 to 254 DN) allow room for overshoots and
control of the white and black levels. The values 0 and 255 DN are used for
syncs.

5.2.6. HDTV/ATS
The current accepted meaning of high definition television (HDTV) is a
television system providing approximately twice the horizontal and vertical
resolution of present NTSC, PAL, and SEC AM systems. While the term HDTV
implies a receiver, the industry is equally concerned about transmission
standards. The video format must be standardized before building cameras and
receivers. The acronym HDTV is slowly being replaced with the advanced
television system or ATS.

HDTV is envisioned as a system that provides high resolution on large

displays and projection screens. Its resolution will not be fully appreciated in the
 CAMERAS 161

typical living room where the sofa is nine feet from the television (discussed in
Section 7 .1., The Observer). The Japan Broadcasting Corporation (NHK)
provided the first HDTV demonstration in 1981. The aspect ratio was initially
5:3 but this was later changed to 16:9 to match the various aspect ratios used in
motion pictures.

The goal is to have worldwide compatibility so that HDTV receivers can

display NTSC, PAL, and SECAM transmitted imagery. With today's
multimedia approach, any new standard must be compatible with a variety of
imaging systems ranging from 35-mm film to the various motion picture
formats. The desire is to create composite imagery 10•11 and special effects from
a mixture of film clips, motion pictures, and 35-mm-film photos. Compatible
frame rates allow the broadcasting of motion pictures and the filming of
television.

As various companies entered the HDTV arena, they proposed different

transmission formats 12 • To standardize the format, the Grand Alliance was
created in May 1993. The seven members are: AT&T, David Sarnoff Research
Center, General Instrument Corporation, MIT, North American Philips,
Thomson Consumer Electronics, and Zenith Electronics. The Grand Alliance
collaborates with a variety of agencies involved with standards (see list on page
146).

In April 1994, the Grand Alliance approved two different resolutions for
HDTV (Table 5-5). Both formats have square pixels. The selected line rate
should simplify conversion from NTSC and PAL into HDTV. The lines per
frame of 1125 is 1517 times 525 and 9/5 times 625. Similarly, 787.5 is 3/2
times 525 but is approximately 100179 times 625lines. Thus, with the lower line
rate, it is more difficult to convert PAL into HDTV.

Doubling the resolution, increasing the aspect ratio, and separating color
from luminance requires an increased bandwidth of eight-fold over current
television requirements. With a nominal bandwidth of 4.2 MHz, the required
HDTV bandwidth is approximately 33.6MHz. A 10-bit system will have a data
rate of 336 Mbit/s. At this data rate, transmission losses in conventional wire
cable limit the maximum transmission distance to, perhaps, 1 km. This demands
image compression.
162 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 5-5
GRAND ALLIANCE RECOMMENDED FORMATS
(Recommended in April 1994)
RESOLUTION LINES FRAME
SCAN
Pixels/line x PER RATE
TECHNIQUE
active lines FRAME (Hz)

24 Progressive
1920 X 1080
1125 30 Progressive

60 Interlaced

24 Progressive

1280 X 720 787.5 30 Progressive

60 Progressive

The FCC has allocated only a 6-MHz bandwidth for commercial television
channels. Thus, image compression somewhere between 50:1 and 75:1 is
required. The Grand Alliance recommends using MPEG-2 image compression.
Although current cameras are analog devices, 13 the Grand Alliance
recommended digital transmission.

In December 1996, the FCC adopted a standard proposed by the Advanced

Television Systems Committee. This versatile standard defines four basic digital
formats (Table 5-6). The higher resolution formats will probably be used to
broadcast movies while news and commercials will be broadcasted at the lower
resolution formats. While the standard allows both interlaced and progressive
scan formats, only one will probably survive. It also mandates compressed
digital transmission. The conversion from analog to digital television is a
complex process. 14 Broadcasting should begin November 1, 1998. The status of
HDTV I ATS cameras, standards, and receivers will appear in many journals over
the next few years.
 CAMERAS 163

Table 5-6
PICTURE FORMATS SUPPORTED by ATSC STANDARD
RESOLUTION
FRAME ASPECT SCAN
pixels/line x
RATE (Hz) RATIO TECHNIQUE
active lines

24 16:9 Progressive
1920 X 1080
30 16:9 Progressive

60 16:9 Interlaced
24 16:9 Progressive
1280 X 720 30 16:9 Progressive

60 16:9 Progressive

24 16:9 or 4:3 Progressive

30 16:9 or 4:3 Progressive
704 X 480
60 16:9 or 4:3 Progressive

60 16:9 or 4:3 Interlaced

24 4:3 Progressive

30 4:3 Progressive
640 X 480
60 4:3 Progressive

60 4:3 Interlaced

5.3. CONSUMER/BROADCAST CAMERAS

Figure 5-4 illustrates a typical color camera functional block diagram. The
array output is processed to achieve the standard video signals. This section
describes the knee, color correction, gamma, and aperture correction.
Monochrome cameras tend to be used for scientific and industrial applications
and are described in Section 5 .4., Industrial/Scientific Cameras.
164 CCD ARRAYS, CAMERAS, and DISPLAYS

From
ceo
Array.---..,
R Color Gamma
G Encoder
Correction Correction
B Composite
1--...-,--..J Video

Sync

Figure 5-4. Typical color camera functional block diagram. Sync signals
and other control logic are not shown. The subsystem location and
design vary by manufacturer. Consumer cameras typically have a
composite output that is created by the encoder. Scientific and industrial
cameras may have a component output which is the signal before the
encoder. To increase versatility, many cameras can provide either
NTSC, RGB, or complementary color output. The RGB output provides
easy interfacing with a computer monitor.

5.3.1. THE KNEE

The antibloom drain (see Section 3.8., Antibloom Drain, page 85) creates
a knee in the linear response to brightness. The knee slope and location can be
any value (Figure 5-5). While scientific cameras operate in a linear fashion,
there is no requirement to do so with consumer cameras. Consumer camera
manufacturers will provide a mapping that they think will provide a cosmetically
pleasing image. The knee also simplifies requirements in that an 8-bit ADC can
provide aesthetically pleasing imagery over a wide input dynamic range.

Scientific and industrial applications often require a linear input-output

system. For example, linearity is required for radiometric analysis. Interestingly,
as some manufacturers add more knee controL others are actively designing
algorithms to "de-knee" the signal before image processing.
 CAMERAS 165

Maximum Output
- ----

l
'5
c.
White Clip Level Variable
Slope

'5
0

Variable Location
4 I "

Input

Figure 5-5. The knee extends the dynamic range by compressing high
intensity signals. By decreasing the knee slope, the camera can handle
a larger scene dynamic range.

5.3.2. COLOR CORRECTION

Figure 5-4 illustrated a typical color camera block diagram. The "color"
signals may have originated from a single CFA (see Section 3.11.2., Color
Filter Arrays, page 94) or three separate arrays as illustrated in Figure 5-6.
Because of high cost, three-chip cameras were used almost exclusively for high
quality broadcast applications. Now, three-chip color cameras are entering the
scientific market. Consumer camera designs, driven by cost, tend to use only
one chip. Although this is changing.

The color correction circuitry is an algorithm that can be represented as a

matrix

[~]- (5-3)

When used to improve color reproducibility (i.e., matching the camera spectral
response to the CIE standard observer), the process is called masking. 15 •16 When
manipulating signals into the NTSC standard, it is called matrixing.
166 CCD ARRAYS, CAMERAS, and DISPLAYS

Figure 5-6. A high quality color camera has three separate arrays. One
for each primary color or its complement. Light is spectrally separated
by coatings on the beam splitters and additional filters. The three-chip
camera is listed as CCD X 3 or 3CCD in sales brochures. Each color
samples the scene equally and this minimizes color moire patterns when
compared to a CFA. Although beam splitters are shown, most 3CCD
cameras use prisms for compactness.

Matrixing can be used to alter the spectral response and intensities of the
primaries to compensate for changes in scene color temperature. Intensity
changes are achieved simply by multiplying a primary by a constant. Spectral
changes are approximated by adding various ratios of the primaries together. For
example, if an object is viewed at twilight, the matrix can be adjusted so that the
object appears to be in full sun light (neglecting shadows). A scene illuminated
with a 2600-K source may appear somewhat yellow compared to a scene
illuminated with a 6500-K source. This occurs because a 2600-K source has less
blue than a 6500-K source. With white balance (matrixing) a white object can
appear to be illuminated by different blackbodies - usually ranging from 3200
to 5600K.

Gamma correction is introduced into each of the three color channels. By

convention, the analog signals are represented by Be,, ER, and E8 • After gamma
correction, they are represented by E0 1 , ER 1 , and E8 1 •
 CAMERAS 167

With CFAs, detectors sensitive to the same color are separated resulting in
a low fill factor. The optical anti-alias filter blurs the image and effectively
increases the optical area of the detector. The birefringent filter is placed
between the lens assembly and the detector array. Birefringent crystals break a
beam into two components: the ordinary and the extraordinary. This makes the
detector appear larger optically. While this changes the MTF (discussed in
Section 10.3.4., Optical Anti-Alias Filter), the sensitivity does not change. The
light that is lost to adjoining areas is replaced by light refracted to the detector.

With unequal sampling, monochrome edges may appear colored and small
objects may change color. Sophisticated image processing is required to
overcome the unequal sampling of the colors and to format the output into the
NTSC standard. The algorithm selected depends upon the CFA architecture (see
Figure 3-44, page 61). Algorithms may manipulate the ratios of blue-to-green
and red-to-green, 17 separate the signal into high and low frequency
components, 18 use a pixel shift processor, 19 or add two alternating rows
together20·21 to minimize color artifacts. The decoding algorithm is unique for
each CFA configuration. Clearly, each CFA design requires a new decoding
algorithm. Algorithm design is probably more of an art than a science because
the acceptability of color aliasing is not known until after the camera is built.

~=============================================
Example 5-1
SINGLE COLOR CHIP ALGORITHM

Figure 5-7 illustrates a typical CFA21 where rows are added to minimize
aliasing. The NTSC odd field consists of CCD rows 1 + 2, 3 + 4, etc., and the
NTSC even field consists of 2 + 3, 4 + 5, etc. Tables 5-7 and 5-8 provide the
outputs.

Table 5-7
SUMMING TWO ROWS
CCD ROW OUTPUT LINE
1+2 1 (odd field)

3+4 3 (odd field)

2+3 2 (even field)

4+5 4 (even field)

 168 CCD ARRAYS, CAMERAS, and DISPLAYS

ceo NTSC NTSC

Row Odd Field Even Field
Row Row
Cy Ye Cy Ye

2 G Mg G Mg

3 Cy Ye Cy Ye

4
3[ Mg G Mg G

5 I Cy I Ye I Cy I Ye :
I __ .l _ _ L __I __ ..
J 4

Figure 5-7. Typical CFA (From Reference 21).

Table 5-8
OUTPUT VALUES
TV
PIXEL 1 PIXEL 2 PIXEL 3 PIXEL 4
LINE

Cy+ G Ye+Mg Cy+G Ye+Mg

1
=2G+8 =2R+ G +8 =2G+8 =2R+G+8

Cy+Mg Ye+ G Cy+Mg Ye+G

3
=R+G+28 =R+2G =R +G+28 =R+2G

Cy+ G Mg+Ye Cy+G Mg+Ye

2
=20+8 =2R+ G+8 =2G+8 =2R+G + 8

Mg+Cy Ye+ G Mg +Cy Ye+G

4
= R + G +28 =R+2G = R + G +28 =R+2G

The NTSC signal requires equal R, G, and B information on each line and
this requires line interpolation. For example, pixel 3 on the first line does not
contain any red data. Red must be interpolated from adjoining pixels. The
algorithm selected depends upon the mathematics and the human eye response.
The selection depends upon the resultant imagery. That is, the interpolation
tends to be empirical.
=============================================~
 CAMERAS 169

5.3.3. GAMMA COMPENSATION

A consumer video camera may be purposely made nonlinear to match a

monitor response. Gamma correction optimizes the displayed image when using
CRT-based monitors. The input predistortion (created at the camera) can only
be effective over a strictly defined range. The overall system is linear only when
the correct display unit is used. Gamma correction is nonlinear and should be
inserted after any required image processing. As flat panel displays replace
CRT-based displays, the concern with gamma correction should disappear.

The luminous output of a cathode ray tube is related to its grid voltage by
a power law
(5-4)

If the camera has the inverse gamma (Figure 5-8), then the radiometric fidelity
is preserved at the display.

Vvideo
Analog ,.----
CCD
Array t--
Electronics
VScene Gamma
Correction
1fy
Video Display
Electronics -
VGrid CRT
y
'-----
~:\>
Observer

Camera Display

Figure 5-8. Gamma correction should only be used when the imagery is
directly presented on a CRT-based display.

Although gamma correction circuitry is placed in the camera and it is

numerically equal to the inverse of assumed CRT gamma, popular parlance
simply calls it gamma. From now on, gamma will mean the inverse of the CRT
gamma value. Gamma compresses small signals and some black detail is lost.
In certain high contrast scenes, such as scenes with direct sunlight and shaded
areas, the darker parts will lack contrast. It also changes the noise characteristics
depending upon the scene level.
170 CCD ARRAYS, CAMERAS, and DISPLAYS

Figure 5-9 illustrates a machine vision system or a system where the

imagery is evaluated by a computer. Gamma correction should be inserted
before any image processing. Although standards recommend a gamma
correction value, manufacturers modify these values to create what they perceive
to be a cosmetically pleasing image. The output of an imaging processing work
station is shown on a computer monitor. Although a machine vision system does
not require a monitor, it adds convenience. A gamma correction, internal to the
computer, assures that the correct image is shown on the computer monitor.

Lo;splay
CCD Array Display
and ~
Electronics
Electronics

~------~~------/
Camera Image Processing System Display Drivers and Display

Figure 5-9. Image processing is simplified when gamma has been

removed. The camera gamma should be one if the image is to be
digitized and displayed on a computer monitor. Although not required
for machine vision applications, the monitor offers convenience.

Although CRT displays have a range of gammas, NTSC, PAL, and SECAM
"standardized" the display gamma to 2.2. 2.8, and 2.8, respectively. However,
all three systems use 0.45 as the inverse. For NTSC, the monitor luminance is
proportional to the detector output and target intensity (2.2)(0.45) = 1. With
PAL and SECAM, the luminance still has 22 a residual gamma of 1.27. This
provides some contrast enhancement.

The gamma slope becomes very large for low values of input brightness. It
has therefore become accepted practice to limit the operation of the gamma
compensator to an arbitrary level. SMPTE Standard 240M was developed 23 for
high definition televisions but now is applied to many systems It has since been
replaced by ITU 709.
 CAMERAS 171

Normalized to one, 240M states

V VIDEO - 1.1115 (VSCEN~ 0·45 - 0.1115 when 0.0228 ~ VSCENE ~ 1

(5-6)
when VSCENE<0.0228 ,

where VscENE is the voltage before the gamma corrector and VvmEo is the voltage
after. The toe break intensity occurs at VscENE = 0.0228 and the toe slope is 4
(when LscENE < 0.0228). For gamma correction to be accurate, the black
reference level must be known. If gamma is inserted after the 7. 5 % setup, some
ambiguity exists in the black level resulting in some intensity distortion. While
gamma correction may contain errors, the human eye is very accommodating.
The consumer can also adjust the receiver contrast to optimize the image in
accordance with his personal preferences. However, this luxury is not available
for scientific applications.

Gamma compensation is often performed digitally using a look-up table.

Figure 5-10 illustrates the desired output-input transformation recommended by
SMPTE 240M. Alternate look-up tables are possibleY In fact, some camera
manufacturers21 developed an alternate gamma function to enhance gray level in
a high light intensity region and to obtain high SIN ratio in low light intensity
regions simultaneously. Changing the gamma to minimize the compression of
dark areas is also called black stretching. Cameras may have a continuous
variable gamma from 0.4 to 1 or may be preselected to 0.45 or 1. The user
selects the appropriate value. However, the user rarely knows the CRT gamma.
He will simply adjust the camera until he creates an aesthetically pleasing image.
This setting will vary from individual to individual.

60

-c
z 50

-
1-
:J 30
40

D..
1- 20
:J
0 10 SMPTE 240M

0
0 2 4 6 8 10

INPUT (ON)
Figure 5-10. Gamma output-input transformation recommended by
SMPTE 240M. Expansion of the first 10 levels of an 8-bit ADC system.
Data only exist at integer values. Compression of blacks is obvious.
172 CCD ARRAYS, CAMERAS, and DISPLAYS

5.3.4. APERTURE CORRECTION

Image sharpness is related to the modulation transfer function (discussed in

Chapter 9, Linear System Theory, and Chapter 10, System MTF). By increasing
the MTF at high spatial frequencies, the image usually appears "sharper."

For analysis purposes, a camera is separated into a series of subsystems:

optics, detector, electronics, etc. MTF degradation in any one subsystem will
soften imagery. The degradation can be partially compensated with electronic
boost filters (discussed in Section 10.6.5., Boost). Historically, each subsystem
was called an "aperture." Therefore boost provides "aperture correction." This
should not be confused with the optical system where the optical diameter is the
entrance aperture.

Boost filters also amplify noise so that the signal-to-noise ratio remains
constant. For systems that are contrast limited, boost appears to improve image
quality. However, for noisy images, the advantages of boost are less obvious.
As a result, these filters are used only in high contrast situations (typical of
consumer applications) and are not used in scientific applications where low
signal-to-noise situations are often encountered.

5 .4. INDUSTRIAL/SCIENTIFIC CAMERAS

The operation of industrial and scientific cameras is straightforward. They

are built around the solid state device capabilities. The array output is amplified
and then either processed into a standard video format or left in digital format.
Although these may be color cameras, the majority are monochrome.

Progressive scan simply means the noninterlaced or sequential line-by-line

scanning of the image (see Section 3.3.5., Progressive Scan, page 68). Moving
objects can be captured with a strobe light or a fast shutter. Only a progressive
scan camera will give the full resolution when the illumination time is short.
These cameras can capture imagery asynchronously. That is, the camera timing
is initiated from an external sync pulse that is coincident with the strobe flash
which may signify that the object is in the center of the field-of-view. Although
the image is captured in a progressive scan manner, the camera output is
processed into a standard video output so that it can be displayed on commercial
monitors. For critical applications, a digital output is available.
 CAMERAS 173

TDI offers another method to capture objects moving at a constant velocity.

The operation of TDI arrays (see Section 3.3.6, Time Delay and Integration,
page 70) exactly describes how a TDI camera will operate. The TDI camera
output may not be in a standard format and may need a dedicated computer or
image processing work station. Reference 25 describes a TDI system used for
remote sensing. Here, the aircraft or satellite provides the motion and the
objects on the ground are stationary.

Consumer cameras are specified by the minimum detectable signal and

dynamic range. In addition to these metrics, scientific and industrial cameras are
specified by MTF and the signal-to-noise ratio. Scientific cameras often have a
large dynamic range and may have 10-, 12-, 14-, or 16-bit analog-to-digital
converters. Key to the operation is that the ADC is linear.

Example 5-2
TDI CAMERA

Bottles move on a conveyor belt at the rate of 1200 ft/min. An image

processing system must detect defects without any human intervention. The
bottle must be scrapped if a defect is larger than 0.1 inches. The camera sits 48
inches from the conveyor belt. If the detectors are 20 p.m square, what focal
length lens should be used? What is the camera line rate? If the bottle is 10
inches tall, what should the array size be?

Image processing software accuracy increases as the number of pixels on the

target increases. While one pixel on the target may be sufficient for special
cases, signal-to-noise ratio considerations and phasing effects typically require
that the object cover at least three pixels. Thus, the image of the defect must be
at least 60 p.m.
174 CCD ARRAYS, CAMERAS, and DISPLAYS

The optical magnification is

M - ~ - y2 -
60 11m = 0 0236 (5-7)
oPncs R1 y1 (O.linch)(25,400 llmfinch) · '

where R 1 is the distance from the lens to the object and R2 is the lens to array
distance. The focal length is

(5-8)

The image is moving at (1200 ft/rnin)(1 min/60 s)(304,800 prn/ft)(0.0236) =

143,865 prn/s. Because the detectors are 20 pm, the line rate is 7193lines/s.
This creates a detector dwell time of 1/7193 = 140 ps. That is, the edge of the
defect has moved across the detector element in 140 ps. The pixel clock must
operate at 7193 Hz and the integration time is 140 ps.

The image of the bottle is (10 inches)(25400 prn/inch)(0.0236) = 5994 pm.

With 20 pm square detectors, the array must contain 300 detectors in the vertical
direction. The number of TDI elements depends upon the detector noise level,
light level, and detector integration time. The signal-to-noise ratio increases by
VNTDI·
=============================================~

5.4.1. ANALOG-to-DIGITAL CONVERTERS

An analog-to-digital converter is an essential component of all digital

cameras. For consumer applications, an 8-bit converter is used. Sometimes
seemingly slight deviations from perfection in an ADC can modify image
fidelity2 6 when examined by a trained photointerpreter or by an automatic target
detection algorithm. While a nonlinear ADC may not be visually obvious with
a single readout, it may become obvious if the array has multiple outputs with
each output having its own ADC. Here, any object which spans the two areas
with different ADCs will have a discontinuity in intensity.

ADC linearity is specified by differential nonlinearity (DNL) and integral

nonlinearity (INL) (Figure 5-11). The differential nonlinearity is a test of
missing codes or misplaced codes. Two adjacent codes should be exactly one
LSB apart. Any positive or negative deviation from this is the DNL. If the DNL
exceeds one LSB, there is a missing code. Many nonlinearity specifications
 CAMERAS 175

simply state "no missing codes." This means that the DNL is less than one LSB.
However, 1.01 LSB DNL (fail specification) and 0.99 LSB DNL (pass
specification) probably look the same visually. Poor DNL can affect the
cosmetic quality of images.

Integral nonlinearity is the deviation of the transfer function from an ideal

straight line. Most ADCs are specified with endpoint INL. That is, INL is
specified in terms of the deviation from a straight line between the end points.
Good INL is important for radiometric applications.

z 8
e.
1-
7
::I 6
D.
1-
5
::I 4
_.
0 3
< 2
1-
(5 1
0 0
0 25 50 75 100
ANALOG INPUT (percent of full scale)
(a)

INL
best fit
z 8 ~
e.
1-
7
::I 6
D.
1-
5
::I 4
_.
0 3
< 2
1-
(5 1
0 0
0 25 50 75 100
ANALOG INPUT (percent of full scale)
(b)

Figure 5-11. ADC nonlinearity for 3-bit ADC. (a) Differential

nonlinearity and (b) integral nonlinearity. Missing codes can be caused
by the accumulation of DNLs. Accumulation can also cause INL.
176 CCD ARRAYS, CAMERAS, and DISPLAYS

ADCs exist in most cameras. With commercially available cameras, the

manufacturer has selected the ADC. To ensure that the ADC is linear, the user
must experimentally characterize the performance in a manner consistent with
his application. 27 This is rather difficult if the camera output is an analog signal.
The analog output represents a smoothed output of the ADC. The smoothing can
hide ADC nonlinearities.

The same problem may exist within a frame capture board. The user must
test the board for any nonlinearities. Because the frame capture board usually
resides within a computer, testing is relatively easy.

5.4.2. INTENSIFIED SOLID STATE CAMERAS

Although low light level televisions can be used for many applications, they
tend to be used for military and scientific applications. The term image
intensifier refers to a series of special imaging rubes28 that have been designed
to amplify the number of photons internally. These tubes increase the number
of incident photons by several orders of magnitude. The more popular second
generation (Genii) and third generation (Geniii) intensifier tubes use a
microchannel plate (MCP) for intensification. Third generation is similar in
design to second generation with the primary difference being higher quantum
efficiency.

When photons hit the photocathode, electrons are liberated and thereby
create an electron image. The electrons are focused onto the MCP and about
80% enter the MCP. The MCP consists of thousands of parallel channels (glass
hollow tubes) that are about 10 p,m in diameter. Electrons entering the channel
collide with the coated walls and produce secondary electrons. These electrons
are accelerated through the channel by a high potential gradient. Repeated
collisions provide an electron gain up to several thousand. Emerging electrons
strike the phosphor screen and a visible intensified image is created. The
intensifier is an analog device and an observer can view the resultant image on
the phosphor screen.

The intensifier output can be optically linked29 to a solid state array with
either a relay lens or a fiberoptic bundle (Figure 5-12). When coupled to a
CCD, it is called an ICCD. The intensified camera spectral response is
determined solely by the intensifier photocathode responsivity. Although Figure
5-13 illustrates Genii and Geniii spectral responsivities, a variety of
photocathodes are available. 30 The window material controls the short
wavelength cutoff whereas the photocathode determines the long wavelength
cutoff.
 CAMERAS 177

Photo Cathode
Window

Screen Fiberoptic

Fiberoptic Taper

Figure 5-12. CCDs can be coupled to an image intensifier using either

fiberoptic bundle or relay lens. The tapered fiberoptic bundle (shown
here) tends to be the largest component of the ICCD.

~
175
c:( 150
.§. 125

~ 100
>
i=
75
50
en
z 25
w
t/) 0
0.3 0.4 0.5 0.6 0.7 0.8 0.9

WAVELENGTH (pm)
Figure 5-13. Nominal spectral response of militarized (fielded) Genii
and Geniii image intensifier tubes. Geniii provides a better match to the
night sky illumination (see Figure 2-13, page 39). The responsivity
depends upon the window and photocathode material. Genii tubes use a
multialkali photocathode whereas Geniii tubes use GaAs photocathodes.
178 CCD ARRAYS, CAMERAS, and DISPLAYS

Although many phosphors are available for the intensifier output, a phosphor
should be selected whose peak emission is at the peak of the solid state array
spectral responsivity. A color camera is not a good choice because it has lower
quantum efficiency due to the color-separating spectral filters. Furthermore,
single chip color cameras have unequal sampling densities and this can lead to
a loss in resolution.

The intensified camera can provide high quality, real-time video when the
illumination is greater than 10-3 lux (corresponding to starlight or brighter). For
comparison, standard cameras (non-intensified l are sensitive down to about 1
lux. Thus, intensified cameras offer high quality imagery over an additional
three orders of magnitude in illumination. A three-order magnitude increase in
sensitivity implies that the MCP gain should be about a thousand. However, it
must be an order of magnitude larger to overcome the transmission loss in the
fiber optic bundle or relay lens. For very lo\\ illumination levels, it may be
necessary to increase the integration time and then real-time video is no longer
possible.

By simply changing the voltage on the image intensifier, the gain can be
varied. Thus, operation over a wide range of luminance levels ranging from
starlight to daylight is possible. Gating is probably the biggest advantage of the
intensified camera. Electronic shutter action is produced by pulsing the MCP
voltage. Gating allows the detection of low level signals in the presence of
interfering light sources of much greater energy by temporal discrimination.
Gate times as short as 5 ns FWHM (full width at half maximum) are possible.
This allows the detection of laser diagnostic pulses against very intense
continuous sources. Because full frame transfer or frame transfer CCDs are used
for ICCD applications, the inherent gating capability of the intensifier is used
as a shutter during transfer time. This eliminates image smear.

Because the detector size is typically smaller than the intensifier's resolution,
the intensifier's image must be minified. Fiber-optic bundles reduce camera size
and weight (lenses are relatively heavy). But fiberoptic coupling is complex
because it requires critical alignment of the fibers to the individual detectors.
The fiber area and pitch, ideally, should match the detector size and pitch for
maximum sensitivity and resolution. However, to alleviate critical alignment
issues, the fiber diameter is typically one-half the pixel pitch. This reduces
sensitivity somewhat.

The lens-coupled system avoids the alignment problems associated with

fibers and does not introduce moire patterns. It offers more flexibility because
an image intensifier with an output lens can be connected to an existing solid
 CAMERAS 179

state camera. The lens coupling represents a cost-effective way to add gating
capability to an existing camera. However, a lens-coupled system provide a
much lower throughput (2 x to 10 x lower) than a fiber coupled system.

There are four spatial samplers in the intensified camera: (1) sampling by
the discrete channels in the MCP, (2) the fiberoptic output window within the
intensifier, (3) the input to the fiberoptic minifier, and (4) the detector array. At
these interfaces, the image is re-sampled. Depending upon the location
(alignment) of the fiberoptic minifier, moire patterns can develop and the MTF
can degrade. In addition, manufacturing defects in the fiberoptic can appear31 as
a "chicken wire" effect.

The intensifier tube typically limits the intensified camera resolution.

Increasing the array resolution may have little effect on the system resolution.
The array is typically rectangular whereas the intensifier is round and the array
cannot sense the entire amplified scene. Most commercially available intensified
cameras are based on the standard military 18-mm image intensifier. Larger
tubes (25 mm and 40 mm) are available but are more expensive and increase
coupling complexity.

Intensifiers are electron tubes. Tubes can bloom so that scenes with bright
lights are difficult to image. For example, it is difficult to image shadows in a
parking lot when bright street lights are in the field-of-view. Intensifiers are
subject to the same precautions that should be observed when using most
photosensitive coated pickup tubes. Prolonged exposure to point sources of
bright light may result in permanent image retention.

The intensifier specifications alone do not accurately portray how the system
will work. Only a detailed analysis will predict performance. Resolution of the
array will not describe system resolution if the image intensifier provides the
limiting resolution. While the array photoresponse may be quite uniform, the
overall nonuniformity is limited by the image intensifier. Standard military
image intensifier photocathodes have nonuniformities approaching ± 30%. Only
carefully selected intensifiers with low nonuniforrnity are used as part of the
intensified system.

5.5. REFERENCES
1. The timing requirements can be found in many books. See, for example, D. H. Pritchard,
"Standards and Recommended Practices," in Television Engineering Handbook, K. B. Benson,
ed., pp. 21.40-21.57, McGraw-Hill, New York, NY (1986).
2. D. H. Pritchard, "Standards and Recommended Practices," in Television Engineering
Handbook, K. B. Benson, ed., pp. 21.73-21.74, McGraw-Hill, New York, NY (1986).
180 CCD ARRAYS, CAMERAS, and DISPLAYS

3. B. D. Loughlin, "Monochrome and Color Visual Information Transmission," in Television

Engineering Handbook, K. B. Benson, ed., pp. 4.4 - 4.5, McGraw-Hill, New York, NY
(1986).
4. D. H. Pritchard, "Standards and Recommended Practices," in Television Engineering
Handbook, K. B. Benson, ed., pp. 21.10-21.11, McGraw-Hill, New York, NY (1986).
5. S.C. Hsu, "The Kell Factor: Past and Present," SMPTE Journal, Vol. 95, pp. 206-214(1986).
6. "EIA Standard EIA-343A, Electrical Performance Standards for High Resolution Monochrome
Closed Circuit Television Camera," Electronic Industries Association, 2001 Eye Street, NW
Washington, D.C. 20006 (1969).
7. See, for example, "Solving the Component Puzzle," Tektronix Book# 25W-7009-1, dated 9/90,
Tektronix Inc., Beaverton, OR.
8. J. Hamalainen, "Video Recording Goes Digital," IEEE Spectrum, Vol. 32(4), pp. 76-79(1995).
9. B. D. Loughlin, "Monochrome and Color Visual Information Transmission," in Television
Engineering Handbook, K. B. Benson, ed., pp. 4.24-4.42, McGraw-Hill, New York, NY
(1986).
10. L. J. Thorpe, "HDTV and Film- Issues of Video Signal Dynamic Range," SMPTE Journal,
Vol. 100(10), pp. 780-795 (1991).
11. L. J. Thorpe, "HDTV and Film- Digitization and Extended Dynamic Range," SMPTE Journal,
Vol. 102(6), pp. 486-497 (1993).
12. B. L. Lechner, "HDTV Status and Prospects," ir Seminar Lecture Notes, 1995 SID
International Symposium, pp. F4/4-F4/74, Society for Information Display, Santa Ana, CA
(1995).
13. J. Blankevoort, H. Blom, P. Brouwer, P. Centen, B. 'd Herik, R. Koppe, A. Moelands, J. v
Rooy, F. Stok, and A. Theuwissen, "A High-Performance, Full-Bandwidth HDTV Camera
Applying the First 2.2 Million Pixel Frame Transfer CCD Sensor," SMPTE Journal, Vol.
103(5), pp. 319-329 (1994).
14. B. Bhatt, D. Birks, and D. Hermreck, "Digital Televiston: Making It Work," IEEE Spectrum,
pp. 19-28 (October 1997).
15. K. B. Benson, ed., Television Engineering Handbook, Chapter 2 and pp. 22.29- 22.39,
McGraw-Hill, New York, NY (1986).
16. H. R. Kang, Color Technology for Electronic Imaging Devices, pp. 3-101, SPIE Press,
Bellingham, WA (1997).
17. K. A. Parulski, L. J. D'Luna, and R. H. Hubbard, "A Digital Color CCD Imaging Systems
using Custom VLSI Circuits," IEEE Transactions on Consumer Electronics, Vol. CE-35(8), pp.
382-388 (1989).
18. S. Nishikawa, H. Toyoda, V. Miyakawa, R. Asada, Y. Kitamura, M. Watanabe, T. Kiguchi,
and M. Taniguchi, "Broadcast-Quality TV Camera v.ith Digital Signal Processor," SMPTE
Journal, Vol. 99(9), pp. 727-733 (1990).
19. Y. Takemura, K. Ooi, M. Kimura, K. Sanda, A. Kuboto, and M. Amano, "New Field
Integration Frequency Interleaving Color Television Pickup System for Single Chip CCD
Camera," IEEE Transactions on Electron Devices, Vol. ED-32(8), pp. 1402-1406 (1985).
20. H. Sugiura, K. Asakawa, and J. Fujino, "False Color S1gnal Reduction Method for Single Chip
Color Video Cameras," IEEE Transactions on Consumer Electronics, Vol. 40(2), pp. 100-106
(1994).
21. J .-C. Wang, D.-S. Su, D.-J. Hwung, and J .-C. Lee, "A Single Chip CCD Signal Processor for
Digital Still Cameras," IEEE Transactions on Electron Devices, Vol. 40(3), pp. 476-483
(1994).
22. D. H. Pritchard, "Standards and Recommended Practices," in Television Engineering
Handbook, K. B. Benson, ed., p. 21.46, McGraw-Hill. New York, NY (1986).
23. SMPTE 240M-1988 appears in SMPTE Journal, Vol. 9R, pp. 723-725 September 1989 and can
be obtained from SMPTE, 595 West Hartsdale Ave. White Plains, NY 10607.
 CAMERAS 181
24. A. N. Thiele, "An Improved law of Contrast Gradient for High Definition Television," SMPTE
Journal, Vol. 103(1), pp. 18-25 (1994).
25. T. S. Lomheim and L. S. Kalman, "Analytical Modeling and Digital Simulation of Scanning
Charge-Coupled Device Imaging Systems," in Electro-Optical Displays, M.A. Karim, ed., pp.
551-560, Marcel Dekker, New York (1992).
26. C. Sabolis "Seeing is Believing," Photonics Spectra, pp. 119-124, October 1993.
27. D. H. Sheingold,Analog-Digital Conversion Handbook, pp. 317-337, Prentice-Hall, Englewood
Cliffs, NJ (1986).
28. See, for example, I. Csorba, lrrwge Tubes, Howard Sams, Indianapolis, Indiana (1985).
29. Y. Talmi, "Intensified Array Detectors," in Charge-Transfer Devices in Spectropscopy, J. V.
Sweedler, K. L. Ratzlaff, and M. B. Denton, eds., Chapter 5, VCH Publishers, New York
(1994).
30. I. Csorba, lrrwge Tubes, p. 209, Howard Sams, Indianapolis, Indiana (1985).
31. G. M. Williams, "A High-Performance LLLTV CCD Camera for Nighttime Pilotage," in
Electron Tubes and Irrwge Intensifiers, C. B. Johnson and B. N. Laprade, eds., SPIE
Proceedings Vol. 1655, pp. 14-32 (1992).
6
CAMERA PERFORMANCE
Camera performance metrics are conceptually the same as array metrics (see
Chapter 4, Array Performance, page 102). The camera is limited by the array
noise floor, dark current, and charge well capacity. The camera's fixed pattern
noise (FPN) and photoresponse nonuniformity (PRNU) may be better than the
array pattern noise when correction algorithms are present. Frame averaging and
binning can reduce the random noise floor and thereby appear to increase the
camera's dynamic range.

The signal is modified by the spatial response of the optics and detector
whereas the noise originates in the detector and electronics. Noise characteristics
may be different from signal characteristics. System design requires optimizing
the signal-to-noise ratio.

This chapter covers maximum signal, minimum signal, dynamic range, and
the signal-to-noise ratio. It assumes that the image size is large compared to the
detector's area. Point source detection calculations are more difficult and can be
found in References 1 and 2. Camera metrics, depend in part, on the application
(see Section 5.2.2., Broadcast/Non-broadcast Design, page 155).

Historically, performance metrics were created for broadcast systems. These

metrics may be applied to scientific, machine vision, and military systems.
Some of these metrics are not appropriate for specific applications and therefore
may be considered as superfluous information.

Image intensifier and array characteristics both affect the intensified camera
signal-to-noise ratio. ICCDs have a dynamic range that varies with the image
intensifier gain. This permits fabrication of general purpose ICCDs that operate
in lighting conditions that range from starlight to full sunlight.

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of ContenTs.

182
 CAMERA PERFORMANCE 183

6.1. NOMENCLATURE

Figure 6-1 illustrates a functional block diagram of an analog camera. It is

functionally identical to Figure 4-1 (page 104). Although the off-chip electronics
is shown to have only one amplifier in Figure 4-1, it may consist of several
amplifiers. One of the amplifiers may have a variable gain that allows the user
to increase the signal under low light level conditions. This gain is turned off
(i.e., G2 = 1) when the maximum illumination is measured. The value of G1 is
superfluous because its only purpose is to provide a standard video voltage.

G1 G2
Fixed Variable

VSIGNAL VCAMERA

Figure 6-1. Camera function block diagram. The gain G 2 may be

automatic (AGC) or manually controlled.

The input can be measured at two different places. With the first
measurement, the lens focuses reflected scene illumination, PTEv-scENE• onto the
array where PT is the target reflectivity. Tables 2-5 and 2-6 (page 28) provide
typical values of Ev-scENE· Here, the camera's output depends upon the lens f-
number and transmittance (or equivalently the T-number). In the second
measurement, the faceplate illumination is used. For radiometric inputs, the
scene irradiance or faceplate irradiance is used. In Equation 4-2 (page 112), the
source was represented by Me and it is identical to PTEe-scENE· Because the
manufacturer has no control over the user's selection of lenses, the faceplate
illumination is often specified.

Array parameters are based upon the photon flux irradiating the detector,
Eq-FACEPLATE and the measured output VsiGNAL· The number of photons reaching
the detector is
(6-1)

which is identical to Equation 2-12 (page 34).

184 CCD ARRAYS, CAMERAS, and DISPLAYS

The array quantum efficiency is

(6-2)

The array signal-to-noise is

(6-3)

and the array dynamic range is

VSIGNAL N WELL - n DARK (6-4)

DRARRAY =
VNOISE ,:nsYS)

The array outputs, nrE and < nsvs > , are derived theoretically or are calculated
from the measured values VsiGNAL or VcAMERA. Often only < nFwoR > is used for
<nsvs> to calculate DRARRAY· This provides the highest possible dynamic
range. If the camera does not introduce any additional noise, modify the noise
bandwidth, or minimize the maximum output, the camera SNR and DRcAMERA
will be identical to the array values.

For Lambertian targets,

E P T Ev-SCENE T T z (6-5)
v-FACEPLATE =
4 F 2 ( 1 + MOPTicsi OPTICS AIM UX.

On a clear day (TATM = 1), a white target (PT = I) which is far way (MoPT1cs =
0) provides

P T Ev-SCENE T (6-6)
Ev-FACEPLATE .. F OPTICS lux.
4 2
 CAMERA PERFORMANCE 185

For back-illuminated targets (such as those used for calibration), the flux
emanating from the target is PTEv-scENE· The measurement of faceplate
illumination is usually performed with a calibrated lens. The value PTEv-scENE is
measured with a calibrated radiometer or photometer, VcAMERA is measured, and
then by using Equation 6-6 or 6-7 Ev-FACEPLATE is calculated.

With four possible inputs (scene irradiance, scene illuminance, faceplate

irradiance, and faceplate illuminance) and three outputs (number of electrons,
VsiGNAL• and VcAMERA) there potentially exists 12 input-output performance
parameters. Each metric is appropriate for a specific application. This chapter
will consider only the most widely used metrics. The derivation of the others is
straightforward. For image intensified systems, the scene or faceplate values are
used as the input to the image intensifier.

The maximum and minimum signals depend on the spectral output of the
source and the spectral response of the detector. The source color temperature
is not always listed but is a critical parameter for comparing systems. Although
the CIE illuminant A is used most often, the user should not assume that this
was the source used by the camera manufacturer. Manipulation of the source
(CIE illuminant A, B, C, D 65 , or another unspecified source) and optical filter
characteristics can change the maximum and minimum signal values.

Based on signal detection theory, the minimum illumination would imply that
the signal-to-noise ratio is one. However, the definition of minimum illumination
is manufacturer dependent. Its value may be (a) when the video signal is, say,
30 IRE units, (b) when the SNR is one, or (c) when an observer just perceives
a test pattern. Because of its incredible temporal and spatial integration
capability, the human visual system can perceive SNRs as low as 0.05. This is
further discussed in Section 12.1.1., Perceived Signal-to-Noise Ratio. Therefore,
comparing cameras based on minimum illumination should be approached with
II II

care.

6.2. CAMERA METRICS

Both the signal and noise are usually measured. By using a measured value,
all noise sources are included: shot noise, noise floor, reset noise, pattern noise,
quantization noise, amplifier white noise, and amplifier 1/f noise. The minimum
signal is then calculated using the variable amplifier gain, G2 , and signal-to-noise
ratio.
186 CCD ARRAYS, CAMERAS, and DISPLAYS

6.2.1. CAMERA SNR

For many applications, the object is far away so that MoPTrcs approaches
zero. When measured at the camera output, the SNR is

(6-7)

JS(jelec) IHsYSCf,l.c)l dfe/ec

0
2

where the subscript scene is used to indicate the input is the reflected scene flux.
Noise is inserted by the detector and by the amplifiers (see Figure 4-11, page
123). The electronics MTF, Hsvs(fe1ec), modifies the noise spectra. The noise
power spectral density, S(fe1ec), includes all the noise sources: shot noise, noise
floor, reset noise, pattern noise, quantization noise, amplifier white noise, and
amplifier 1/f noise. The subscript elec emphasizes that electrical frequency
applies to the amplifiers and subsequent electronic filters. The MTFs are
described in Chapter 10, System MTF.

Equation 6-8 is the generalized SNR equation. When equated to electrons,

(6-8)

For a small spectral band

AD
-4F Rp. 0) p -J_').. o> Mq-SCENi'A o>
2
TOP17CS(').. o> !l ')..
tiNT
(6-9)
SNRSCENE - ____:_=----------- - - - - - - - -
(nSYS)
 CAMERA PERFORMANCE 187

The SNR is proportional to the detector area and integration time. Figure 6-2
provides the relative SNR as a function of these parameters. Binning is identical
to increasing the detector area. The output of four binned detectors is the same
as one detector four times larger. For fixed area, increasing the integration time
increases the SNR. For fixed integration time, binning increases the SNR.
Shorter integration times permit higher frame rates (See Section 4.5., Frame
Rates, page 142). Shorter integration times also decrease the SNR.

BIN SIZE
100 -10x10
<
w - 8x8
wa:: - 6x6
><
i=a:: - 4x4
oo
WI-
10 - 3x3
u.o
u.w
WI-
w
0
1
1 10 100
INTEGRATION TIME
Figure 6-2. Relative SNR as a function of bin size (detector area) and
integration time. Only shot noise is considered.

6.2.2. CAMERA DYNAMIC RANGE

Dynamic range is the maximum signal divided by the noise. It depends on

integration time, binning, and frame integration. In addition, the spectral content
of the source and the array spectral responsivity affect the camera output
voltage. Thus, the camera dynamic range can be quite variable depending on test
conditions.

Although the theoretical maximum signal just fills the charge wells, the
manufacturer may limit the maximum output voltage to a lower stored-charge
value. Because the dynamic range is often expressed in decibels,

DRCAMERA-dB - 20 log (~
VMAX] (6-10)
N01se
188 CCD ARRAYS, CAMERAS, and DISPLAYS

and the noise voltage is

(6-11)

Many camera manufacturers list the dynamic range as the signal-to-noise

ratio. This value should not be confused with the actual SNR (see Section 4.3.,
Array Signal-to-noise Ratio, page 139). With most cameras, the noise level is
approximately equivalent to the array read noise. The electronics may increase
the camera noise, and image processing techmques may reduce it somewhat.
Usually the dynamic range is calculated from the measured signal and noise.
Because the read noise is amplifier white noise, it is appropriate to include the
bandwidth as part of the measurements. For standard video compatible cameras,
the bandwidth is equal to the video format bandwidth (see Section 5 .2.1., Video
Timing, page 151).

Example 6-1
DIGITAL DYNAMIC RANGE

A device with a well capacity of 50,000 electrons has a noise floor of 60

electrons rms. What size analog-to-digital converter should be used?

Adding noise powers, the total camera noise is related to the array noise and
ADC noise by

(6-12)

If an ADC is selected so that an LSB is equal to the array noise level, the ADC
increases the camera noise by 4%. The array dynamic range is 50,000/60 = 833
but the smallest available ADC has 10 bits or 1024 levels. This ADC devotes
1.23 bits to array noise. But if the LSB was one-quarter of the array noise level,
the camera noise would increase by only 1 %. The ADC dynamic range must be
greater than 3332:1. The smallest ADC that covers this range contains 12 bits.

=============================================~
 CAMERA PERFORMANCE 189

6.2.3. MAXIMUM SIGNAL

If specifying scene illumination, a f-number must be selected. Historically,

film-based cameras are compared when the lens f-number is 5.6. Figure 6-3
illustrates the relative change in the maximum signal as the f-number is varied.
For consumer video and industrial cameras, the integration is selected to match
the video standard (t1NT = 1/30 for EIA 170). Automatic irises and variable
integration times can increase the maximum level.

The photon level that saturates the charge wells is the camera's maximum
input signal and VwELL = G 1 GqNwELL/C where G2 = 1. However, there is no
requirement that the camera manufacturer set the maximum output voltage equal
to VwELL· If the array has an antibloom drain, the maximum output may be
limited to the white clip level (see Figure 3-35, page 87).

1- 1.5
:::J
c.
1-
:::J
0
w
>
~
:5w
rx: 0.5
4 4.5 5 5.5 6 6.5 7

f-number

Figure 6-3. Relative maximum output normalized to f/5.6.

6.2.4. MINIMUM SIGNAL

The minimum input signal suggests something about the lowest detectable
signal. When using scene illumination, the minimum signal is often specified
with different optics than the optics specified for the maximum signal (Figure
6-4). Faster optics is a reasonable choice because these lenses are used in low
illumination conditions.
190 CCD ARRAYS, CAMERAS, and DISPLAYS

2
1-
::J
D..
1- 1.5
::J
0
w 1
>
i=
<C 0.5
..J
w
0:::
0
0.8 1 1.2 1.4 1.6 1.8 2
f-number

Figure 6-4. Relative minimum output normalized to f/1.4. A camera can

appear to have very high sensitivity if a very low f-number lens is used.

The minimum signal provided with gain "on" (G2 greater than one) is
usually calculated due to the difficulty of performing accurate low level
radiometric and photometric measurements. These values may be provided at
30%, 50%, or 80% video levels. That is, the illumination given produces an
output video that gives 30, 50, or 80 IRE units, respectively (see Section 5.2.4.,
IRE Units, page 157). Although a higher gain amplifier could provide 100%
video, the user can optimize the image by adjusting the gain and level of the
display. That is, the display's internal amplifiers can be used for additional gain.
In this context, the camera system consists of the camera and display.

If G2 is expressed in decibels,

(6-13)

The scene illumination that creates a 30% video signal is

0.3 ·Mv-SCENe(MAX) (6-14)

Mv-ScENi30%video) -

where Mv.scENE(MAX) is the scene illumination that produced the maximum

output. For 50% and 80% video, the factor becomes 0.5 and 0.8, respectively.
 CAMERA PERFORMANCE 191

The input signal that produces a signal-to-noise ratio of one is

( _ DRCAMERA-dB)
20
(6-15)
Mv-SCENe(SNR = 1) = Mv-SCENe(Max) ·10 •

For faceplate illumination, Mv-scENE is replaced with Ev-FACEPLATE· Actual

performance may deviate from the calculated values. The gain used in the
equations is typically a nominal value. Usually more gain is available than that
specified in the data specifications.

The examples in Chapter 4 (pages 116-118, 122, and 131) should be

reviewed prior to reading the following examples.

~=============================================
Example 6-2
MINIMUM SIGNAL ESTIMATION

A camera provides maximum output when the faceplate illumination is 0.65

lux. The dynamic range is 55 dB. What is the equivalent noise signal?

Using Equation 6-15, the noise signal is 1.16 X 10-3 lux. A signal-to-noise
of one provides an extremely noisy image. While a human observer can detect
some objects when the SNR is one, a machine vision system cannot.
=============================================~

Example 6-3
VIDEO SIGNAL ESTIMATION

The camera described in Example 6-2 provides a gain of 20 dB. What is the
SNR at 30% and 80% video?

Using Equations 6-14 and 6-15, at 30% video the faceplate illumination is
0.0195 lux and at 80% video, the faceplate illumination is 0.052 lux. The SNR
is 17 and 45, respectively. With a SNR of 17, the image will appear slightly
noisy but probably would be considered acceptable to most observers. However,
if no additional gain is used, the image will appear very dark on the display.
Here, the display gain must be increased to make the image viewable.
Depending upon the task, a machine vision system may need a higher SNR.
While computer programs are usually insensitive to absolute signal values, the
frame grabber's ADC may not provide enough levels for adequate image
192 CCD ARRAYS, CAMERAS, and DISPLAYS

processing. For example, if the ADC is an 8-bit system, 30% video only
provides digital numbers ranging from 0 to 77 .
=============================================~

Example 6-4
NOISE EQUIVALENT INPUT

In Example 6-2, the minimum signal was specified in photometric units. In

the more general radiometric case, the input that provides SNR = 1 is the noise
equivalent input (NEI). That is, pTMq-scENE is found from Equation 6-9 when
SNRq-SCENE is one. Over a small spectral band,

photons (6-16)
s-m 2 -1-1m

When the SNR is unity, the number of photons is small and photon noise is
negligible (see Figure 4-16b, page 136). If the detector is cooled, dark current
is minimized. With PRNU and FPN correction, the minimum NEI is

NEI .. (6-17)

If TDI is used, then NTDI is multiplied in the denominator. With binning or

super pixeling, the number of pixels combined is multiplied in the denominator.

=============================================~

Example 6-5
NOISE EQUIVALENT REFLECTANCE

The NEI assumes that the target is against a black background. For real
scenes, both the target and background create signals. The target is only visible
when a sufficient reflectance difference exists. For detecting objects at long
ranges, the atmospheric transmittance loss must be included. The signal
difference between the target and its background is
 CAMERA PERFORMANCE 193

}.

!J. - f2R(A.)(pT-pB)M/A.)tiNTADT (A.)T (A.)dA.. (6-18)

npe q p2 OPTICS ATM
}.I
4

Both the target and background are illuminated by the same source. The
signals from both depend on the spectral reflectances that are taken as constants
over the spectral region of interest. Using the same simplifying assumptions as
before, solving for PT - p8 and calling this value the noise equivalent reflectance
difference (NE!J.p) provides

(6-19)

where T ATM is the total atmospheric transmittance and it depends on range. The
value TATM is usually expressed as TATM = exp(-uATMR). Then

Thus, NE!J.p depends on the source illumination, system noise, and the
atmospheric conditions. As the range increases, the target reflectance must
increase to keep the SNR at one. The maximum range occurs when the NE!J.p
is one. At this maximum range,

NEI - M q(A 0 ) e- a ATMRMAx (6-21)

•

The variable Mq(>. )[pT(A)- p8 (A)] is a complicated function that depends on the
cloud cover, atmospheric conditions, sun elevation angle, surface properties of
the target, and the angle of the target with respect to the line-of-sight and sun
angle.
194 CCD ARRAYS, CAMERAS, and DISPLAYS

These back-of-the-envelope approximations are still not complete. They

assume that the total received energy is from the target only. In fact, over long
path lengths, path radiance contributes to the energy detected. Path radiance,
specified by visibility, reduces contrast. The additional light reflected by the
atmospheric particulates introduce photon noise that is independent of the signal
from the target. It is quantified by the sky-to-background ratio (discussed in
Section 12.6.1, Contrast Transmittance).

=================================================~

Example 6-6
MINIMUM PHOTOMETRIC SIGNAL ESTIMATION

A CCD array has a noise floor of 150 electrons rms. If placed into a
camera, what is the estimated minimum faceplate detectable signal?

The minimum signal occurs when the signal-to-noise ratio is one. That is,
npe = (nFLOoR). Several simplifying assumptions are required to obtain a back-of-
the-envelope expression for the minimum signal. If the CIE illuminant D6500 is
used, the spectral photon sterance is approximately constant over the spectral
region, ..:1>-. If the quantum efficiency is constant,

photons (6-22)
s-m 2 -1J.m

Because the energy of one photon is he/A, the faceplate spectral power is

(6-23)

The faceplate illumination is

0.7J.Im

Ev-FACEPLATE =
683
AD
f V(A_)Ep- FACEPLATE(A_)dA lux (6-24)
0.4 J.lm

or
h
E
V-FACEPLATE
(
..nFWOR)
A , t
c 68~
A
JV(A.)
0.1

,
dA. l
ux.
(6-25)
fl a II. /NT D 0.4 II.
 CAMERA PERFORMANCE 195

From Figure 4-5 (page 108) TJ ::::0.3. For EIA 170 compatibility, t1NT = 1/30 s,
and the spectral bandwidth is <l}.. = (0. 7 - 0.4) p.m = 0. 3 p.m. The photometric
integral is approximately 0.2. The constant, he, is approximately 2 x 10'19 J-p.m.
Then
1.36 x w- 12 lux. (6-26)
Ev-FACEPUTE - - - - - -
AD

Assume a 50% fill factor interline transfer array is available. From Table 3-3
(page 99), a 1/z-inch format detector will have an active size of approximately
4.17 p.m X 10 p.m (A 0 = 41.7 X 10' 12 m2). Then the estimated minimum signal
is 3.3 X 10'2 lux.

Consider a target that is pure white, (PT = 1) on a black background (p 8 =

0). At low light levels, a f/1.4lens is often used. Let Tovncs = 0.9 and TATM =
1.0. Using Equation 6-6 (page 184) provides Ev-SCENE = 0.29lux. This target will
have a signal-to-noise ratio of one and therefore be very difficult to see. Real
targets and backgrounds have reflectivities between 0 and 1. When including the
signal-to-noise ratio, Ev-sCENE = 0.29 (SNR)/(PT- p 8 ). Let the required SNR be
20, PT = 0.6, and p 8 = 0.4, then Ev-scENE = 29lux. From Table 2-5 (page 28),
this target can be seen during twilight.

While the assumptions are not unreasonable, the statement that the target can
be seen during twilight is an extrapolation. The spectral output of the D6500
source is substantially different than twilight with color temperature of
approximately 2000 K. Changing the spectral input increases the minimum
illumination. Note that these calculations were based solely upon SNR
considerations. The camera must have sufficient gain so that these signals can
be displayed.

=============================================~

6.3. INTENSIFIED CCD CAMERA

The construction of the ICCD was described in Section 5.4.2., Intensified

Solid State Cameras, page 176. As illustrated in Figure 6-5, the photocathode
creates photoelectrons that are amplified by the MCP. The amplified electrons
are converted back into photons by the phosphor screen. These photons are
relayed to the CCD by either a fiberoptic bundle or a relay lens. The CCD
creates the photoelectrons that are measured. Each conversion has a quantum
efficiency less than unity and each optical element yields a finite transmittance
loss.
196 CCD ARRAYS, CAMERAS, and DISPLAYS

Phosphor Screen
Photo Micro
Cathode Channel Plate \ Fiber-Optic ceo

~o~~~oE~~o~
Figure 6-5. Multiple photon-electron conversions occur in an ICCD.

While the image intensifier amplifies the image, the SNR may be reduced
due to quantum losses and additive noise. For moderate light levels, the ICCD
offers no advantage in SNR performance over the standard CCD camera. Its key
attribute is the ability to gate the scene temporally.

6.3.1. ICCD SIGNAL

The ICCD uses a lens to image the target onto the intensifier faceplate. The
photon flux reaching the photocathode is

(6-27)

Using the same approximations as given in the preceding section, the number
of signal electrons generated by the photocathode is

nCATHODE =
(6-28)

where APIXEL is the projected area of a CCD pixel onto the photocathode.
Geometrically, it is the CCD detector area multiplied by the magnification of the
fiberoptic bundle. However, APIXEL depends on fiberoptic bundle characteristics
and microchannel pore size. It may be somewhere between 50% of the CCD
detector geometric projected area to several times larger. 3 As such, it represents
the largest uncertainty in signal-to-noise ratio calculations. The value A0 is the
center wavelength of the photocathode spectral responsivity and ~A is the
photocathode's spectral bandwidth.
 CAMERA PERFORMANCE 197

After the microchannel plate, the number of electrons is

nMCP - GMcP ncATHODE ·
(6-29)

These electrons are then converted to photons by the phosphor screen

nSCREEN - 11 SCREENnMCP • (6-30)

The photons travel to the CCD via a fiberoptic bundle whose effective
transmittance is Tro· This is a composite value that includes the fiberoptic
transmittance and a coupling loss.
nPHOTON-CCD - TfonSCREEN • (6-3l)

The number of electrons sensed in the CCD is proportional to its quantum

efficiency
(6-32)

or

For a lens-coupled system, TFo is replaced by an effective relay lens

transmittance. 4 Each has its own merits. Transmission losses in a fiberoptic
bundle increase proportionally to the square of the minification ratio. This leads
to a practical minification of about 2:1. Light is lost because the screen is a
Lambertian source whereas the fiberoptic can only accept light over a narrow
cone (acceptance angle). But lenses are large and bulky and may provide a
throughput as low as 2%. Relay lenses also suffer from vignetting. Here, the
"relayed" intensity decreases as the image moves from the center to the
periphery of the image intensifier tube.

Because the lens coupling is less efficient than the fiberoptic coupling, the
number of electrons produced is lower. The fiberoptic coupled system, with its
higher effective transmittance and better sensitivity, can detect lower signals.

6.3.2. ICCD NOISE

The photocathode produces both photon noise and dark noise:

(n;C-SHOT) - (n~ATHODE) + (n;C-DARK) ·
(6-34)
198 CCD ARRAYS, CAMERAS, and DISPLAYS

The MCP amplifies these signals and adds excess noise so that
(n~cp) - kMCPG~cp(n;c-snor) • (6-35)

where kMcP is typically taken as 1.8 but varies with bias voltage. 5•6 The value
kMcP is also reported as an SNR degradation. The noise power is transformed
into photons by the phosphor screen
(6-36)

It is attenuated by the fiberoptic bundle

(n~CD-PHOTON) - iio (n.~CREEN) · (6-37)

This value is converted to electrons by the CC D and the CCD noise is added
(n~cv) - fl~cv(n~CD-PHOTON) + (n~CD-DARK) + (n'i.woR) + (n;RNu) · (6-38 )
The total noise is
/2) -
\nCCD T)
2CCD ...22 2/2 )12
lfo T) SCREENkGMCP\nPC-SHOT + \nCCD-SHOT
)
(6-39)

6.3.3. ICCD SNR

The resultant signal-to-noise ratio is
SNR-
(6-40)

Several limiting cases are of interest. When the gain is very high, the intensifier
noise overwhelms the CCD noise:

SNR- (6-41)

The SNR is independent of gain and CCD characteristics. Under high gain
conditions, an expensive, high quantum efficiency or thinned back illuminated
CCD is not necessary.
 CAMERA PERFORMANCE 199

Cooling the photocathode reduces the dark current:

nCATHODE
SNR =
k
(6-42)
11cATHODEMi>..)A>.. t T A
p2 k TNT OPTICS PIXEL
4

High gain is used for photon counting applications. SNR, in this case, is limited
by the photocathode quantum efficiency, the excess noise factor, and the optical
system.

Under moderate gain situations where the CCD noise is greater than the
intensifier noise,

11 CCD 11 SCREEN Tfo G MCP n CATHODE

SNR- (6-43)
v(n~CD-SHOT) + (n;LOOR) + (n;RNu)

Comparing this with Equation 6-8 (page 186), the ICCD will have lower SNR
than a comparable standard camera. The advantage of the ICCD under these
conditions is the ability to gate the incoming photon flux temporally.

Example 6-7
IMAGE INTENSIFIER NOISE

Is there a relationship between the image intensifier noise equivalent input

and the ICCD noise?

The noise in an image intensifier is due to random emissions (dark current

noise) at the photocathode. When referred to the input (SNR = 1), it is called
the equivalent background illumination (EBI). Using the same methodology as
in Example 6-4, the EBI for an area Ar1xEL is

_ 4 /k pz (nPC-DARK) h (6-44)
EBI = M ('
q "o) uA t TNT A PIXEL
A ,
T p otons.
11 CATHODE OPTICS
200 CCD ARRAYS, CAMERAS, and DISPLAYS

The ICCD will have the same equivalent background input under high gain
situations. Cooling the image intensifier reduces the EBI and it may be
negligible in gated applications. Under low gain situations, the CCD noise
dominates.
============================,=================~

Example 6-8
CCD versus ICCD

Under moderate gain situations, what is the relative SNR of the ICCD to the
CCD?

The ratio of SNRs is

(6-45)

The P20 phosphor screen has a peak emission at 0.56 p.m with a conversion
efficiency of 0.063. An S20 bialkali photocathode's efficiency is 0.2 at 0.53 p.m.
The value ofTro is about 0.25. Combined, these factors give 0.00315. The MCP
gain must be greater than 110.00315 = 317 for the SNR1cco to be greater than
the SNRcco·
============================,=================~

6.4. REFERENCES
1. G. C. Holst, Electro-Optical Imaging System Performance, pp. 48-54, JCD Publishing (1995).
2. M. A. Sartor, "Characterization and Modeling of Microchannel Plate Intensified CCD SNR
Variations with Image Size," in Electron Tubes and Image Intensifiers, C. B. Johnson and B.
N. Laprade, eds., SPIE Proceedings Vol. 1655, pp. 74-84 (1992).
3. Mark Sartor, Xybion Corporation, private communication.
4. Y. Talmi, "Intensified Array Detectors," in Charge-Transfer Devices in Spectroscopy, J. V.
Sweedler, K. L. Ratzlaff, and M. B. Denton, eds., Chapter 5, VCH Publishers, New York
(1994).
5. I. Csorba, Image Tubes, pp. 120-124, Howard Sams, Indianapolis, Indiana (1985).
6. R. J. Hertel, "Signal and Noise Properties of Proximity Focused Image Tubes," in Ultrahigh
Speed and High Speed Photography, Photonics, and Videography, G. L. Stradling, ed., SPIE
Proceedings Vol. 1155, pp. 332-343 (1989).
7
CRT-BASED DISPLAYS
Displays are used for consumer television receivers, computer monitors,
scientific applications, and military applications. A display may be called a
monitor, visual display unit (VDU), video display terminal (VDT), or simply a
television receiver.

The major performance requirements are listed in Table 7-1. This table is
not meant to be all inclusive. It just indicates that different users have different
requirements. As displays are built for each user, they are specified in units
familiar to that user. For example, consumer displays will not have the MTF or
spot size listed. These parameters are provided for militarized displays.

Table 7-1
DISPLAY REQUIREMENTS
APPLICATION REQUIREMENTS

Consumer TV "Good" image quality

Alphanumeric character legibility

Computer
"Good" graphical capability
Scientific and High MTF
military Unity gamma

Television receivers have been available for over 50 years and the
technology has matured. They are considered adequate and little improvement
in cathode ray tube (CRT) technology is expected. Advances in television
displays currently focus on high definition television, flat panel displays, the
legibility of alphanumeric characters, and computer graphics capability. As a
result, display specifications are a mix of CRT terminology, video transmission
standards, alphanumeric character legibility, and graphics terminology.

System resolution depends on the camera response and limitations imposed

by the video standard, display characteristics, and observer acuity. High
resolution displays do not offer any "extra" resolution from a system point of
view if display is not the limiting factor. Rather, high resolution displays just
ensure that all the information available is displayed. In many applications,

201
202 CCD ARRAYS, CAMERAS, and DISPLAYS

system resolution is limited by the human eye. If the observer is too far from
the screen, not all of the image detail can be discerned.

In almost all cases, the output device sets the system dynamic range. Often,
the display is the limiting factor in terms of image quality and resolution. No
matter how good the camera is, if the display resolution is poor, then the overall
system resolution is poor.

Flat panel displays are an emerging technology with liquid crystal displays
(LCDs) prevalent in notebook (laptop) computers. LCDs require only a fraction
of the power required by cathode ray tubes and therefore lend themselves to
battery operated systems. However, CRTs will probably dominate the display
market due to low cost, high resolution, wide color gamut, high luminance, and
long life.

The commercially important flat panels are the light emitting diode (LED)
displays, AC plasma display panels (AC PDP). DC plasma display panels (DC
PDP), AC thin film electroluminescent (ACTFEL) display, DC thin film
electroluminescent (DCTFEL) displays, vacuum fluorescent displays (VFD), and
the liquid crystal displays (LCD).

LCD arrays can either be active matrix (AMLCD) or passive arrays. The
active matrix contains switching devices on each disel to maintain high
brightness and high contrast control. The liquid crystals may be either twisted-
nematic (TN) or supertwisted-nematic (STN). The details of these displays can
be found in References 1 through 6. Although this chapter discusses CRTs,
many concepts apply to all displays.

Emerging technology, new designs, and characterization methods can be

found in the Society of Motion Picture and Television Engineers Journal and the
Proceedings of the Society of Information Display symposia. The Society of
Information Display offers numerous tutorials, complete with published notes,
at their symposia.

In accordance with industry convention, the display sizes and viewing

distances are expressed in inches. The majority of this book uses the metric
system. The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.
 CRT-BASED DISPLAYS 203

7 .1. THE OBSERVER

The observer is the starting point for display design. Design is based on both
perceptual and physical attributes (Table 7-2). For most applications, the design
is driven by perceptual parameters. These parameters are partially related to the
physical parameters. For example, sharpness is related to MTF. But the precise
relationship has not been quantified. While the physical parameters are important
to all applications, they tend to be quantified only for scientific and military
displays. Tables 7-1 and 7-2 illustrate different ways of specifying the same
requirement: the need for a high quality display.

Maximum image detail is only perceived when the observer is at an optimum

viewing distance. If the observer is too far away, the detail is beyond the eye's
limiting resolution. It the observer moves closer to the display than the
optimum, the image does not become clearer because there is no further detail
to see. Display size and resolution requirements are based on an assumed
viewing distance.

Table 7-2
PERCEPTIBLE AND PHYSICAL PARAMETERS

PERCEPTUAL PHYSICAL
PARAMETERS PARAMETERS

Luminance
Resolution
Uniformity
Brightness Addressability
Contrast Gamma
Sharpness Color saturation
Color rendition Color accuracy
Flicker Color convergence (CRT)
Distortion (CRT)
Refresh rate
MTF

For monochrome systems, the optimum viewing distance occurs when the
raster pattern is barely perceptible. At closer distances, the raster pattern
becomes visible and interferes with image interpretation. That is, it is considered
annoying by most. For color CRTs, the observer must be far enough away so
that the individual dots of the three primaries are imperceptible. Otherwise, the
individual color spots become visible and the impression of full color is lost.
204 CCD ARRAYS, CAMERAS, and DISPLAYS

For design purposes, perceptibility occurs when targets are larger than 1 arc
minute. This corresponds to the normal visual acuity of 20/20. This design
criterion is quite good. An observer can adjust his distance accordingly. An
observer with poorer acuity (e.g., 20/40) will simply move closer to the screen
whereas someone with better acuity will probably move further away. Most
systems are observer-limited in that the observer cannot see all the detail
because he is typically too far from the display.

Figure 7-1 illustrates the optimum viewing distance as a function of display

diagonal size for monochrome displays. Television receivers and many computer
monitors are specified by the diagonal dimension. Each active line (raster line)
subtends 1 arc minute (0.291 mrad). At closer distances, the individual raster
lines may become visible. The viewing ratio is the viewing distance divided by
the display height. The typical viewing ratio for computer monitors is three.

Television viewers sit about seven times the picture height or 4.2 times the
diagonal picture dimension. Knowing the dimensions of a typical living room
(sofa on one side, television on the other) suggests that consumer televisions
should have a diagonal picture size of 21 to 25 inches. These were the most
popular television receiver picture sizes. Sizes have changed reflecting changing
television viewing habits.

UJ
w
u.Z
o:J
o::c
Ww
m>
:::!c:e
::)-I
Zll.
UJ
c
0 20 40 60 80 100 120 140

VIEWING DISTANCE (inches)

Figure 7-1. Viewing distance as a function of CRT diagonal picture size

and number of displayed lines per picture height for monochrome
displays. Most CRTs have a 4:3 aspect ratio so that the vertical extent
is 60% of the diagonal.
 CRT-BASED DISPLAYS 205

A major design consideration is the elimination of flicker. EIA 170 and

NTSC use a 2:1 interface technique which is updated at 30Hz. For computer
monitors which display progressive scan imagery, the frame rate (refresh rate)
must be greater than 70 Hz to avoid flicker. Flicker and other design features
are not covered here but can be found in numerous books. 2-6

7.2. CRT OVERVIEW

Figure 7-2 illustrates a typical color cathode ray tube display. Although a
color display is illustrated, the block diagram applies to all CRT displays. The
input video signal is decoded into its components and then digitized. While
digital electronics may be used to process the signal, the CRT is an analog
device. Similarly, displays accepting digital inputs must convert the signal into
an analog voltage to drive the CRT. Numerous transformations occur to create
a visible image. The display matrix and display gain controls allow linear
mapping from input image to visible image. 7

Original
Image
or Data

.....----
Visible
Image

CRT

Figure 7-2. Although a color display is illustrated, the block diagram

applies to all CRT displays. The digital-to-analog converter (DAC)
provides the analog voltages that modulate the electron beam.

The display matrix provides the mapping between the standard video color
components and the actual colors emitted by the phosphors on the screen. This
color correction circuitry is an algorithm that can be represented as a matrix

(7-1)
206 CCD ARRAYS, CAMERAS, and DISPLAYS

With the NTSC format, the electron beams paint an image on the CRT
phosphor in the scan format shown in Figure 5-2, page 151. The color of the
visible image depends on the specific phosphors used. Electron beam density
variations, which represent scene intensity differences, create the visible image.

Monochrome displays have a single electron gun and a single phosphor. A

color CRT must produce three images (three primary colors) to give a full range
of color. To do so requires three electron guns and three phosphors. The three
electron beams scan the CRT screen in the same way as a monochrome beam,
but arrive at the screen at slightly different angles. The beams strike three
different phosphors dots. When the dots (subdisels) are beyond the eye's limiting
resolution, the eye blends them to create the illusion of a full color spectrum.

The output brightness of a CRT depends on the grid voltage and follows a
power law relationship. The slope of this curve, on a log-log scale, is the
display gamma (')'):
(7-2)

Gamma can range 8 from one to five. For design purposes, NTSC standardized
gamma at 2.2. PAL and SECAM standardized gamma at 2.8.

To provide a linear relationship in the complete camera-display system, the

inverse gamma is inserted in consumer cameras as an image processing
algorithm (see Section 5.3.3., Gamma Compensation, page 169). For scientific
applications and computer monitors, the displays will have an internal gamma
correction circuit. Here, the user simply supplies the signal with no gamma
compensation (e.g., 1/'Y = 1). For color displays, each color will have its own
gamma look-up table. This operation is performed in the display matrix.

Digital displays provide excellent alphanumeric and graphic imagery. There

is no distortion and the disels are accurately located because there is no jitter in
timing signals compared to what might be expected with EIA 170. "Digital"
seems to imply high resolution but the displayed resolution is limited by the
electron beam spot size. Although called "digital," the CRT is an analog device.
 CRT-BASED DISPLAYS 207

7 .2.1. MONOCHROME DISPLAYS

In many industrial and scientific applications, monochrome cameras are

used. The military uses monochrome monitors extensively to display imagery
from non-visible scenes (e.g., thermal and radar imagery). Monochrome
displays are found in cockpits and medical imaging instruments. For television
receivers, these are simply called black-and-white TVs. Depending on the
phosphor used, monochrome displays can provide nearly any color desired.
Certain colors (e.g., orange) tend to reduce eye strain. Others (e.g., blue) tend
not to disrupt scoptic vision adaption. Although computers have color monitors,
word processing is often performed in a monochrome format.

Black-and-white scenes can be imaged on a color monitor by assigning a

color palette (false color or pseudo color) to various intensity levels. This does
not increase resolution but draws the observer's attention to selected features.
The eye can easily differentiate color differences but may have trouble
distinguishing small intensity differences.

7 .2.2. COLOR DISPLAYS

A color monitor has three electron beams that scan the CRT faceplate. Near
the faceplate is a metal mask that has a regular pattern of holes (Figure 7-3).
With the dot arrangement, it is called as shadow mask. With slots, it is called
an aperture grille. The three guns are focussed onto the same hole and the
emerging beams strike the CRT faceplate at different locations. At each location
there is a different phosphor. One fluoresces in the blue, one in the red, and one
in the green region of the spectrum. The arrangement is such that the beam
corresponding to the desired color will strike only the phosphor dot producing
that color. The electron guns emit only electrons and have no color associated
with them. Color is only created when the beam hits the phosphor even though
the guns are called red, green, and blue guns.

Hexagonal structure masks (the dots) are inherently more rigid than slots.
However, slots are easier to manufacture and therefore are usually found in
television receivers. The slots have a larger area and therefore aperture-grille
CRTs tend to be brighter than shadow-mask CRTs. With this brightness it is
said that the colors are richer. The distance between dots is less and more
difficult to perceive. Shadow-mask CRTs tend to provide sharper images.
Because computer monitors are viewed at a close distance, the dots are used
most often. The selection of slots over dots depends upon the application.
208 CCD ARRAYS, CAMERAS, and DISPLAYS
Metal Metal
Mask Mask
1\ r-- - r-- - /"""
~ ~n If1
'-- I
.__

lA B G R B G R[s B G R B G R B IG
'

Phosphorus
\I
Phosphorus
on Glass on Glass
Faceplate Faceplate

Figure 7-3. Color CRT mask patterns. (a) Dotted, (b) stripes, and (c)
slotted. Most consumer television receivers use slots whereas computer
displays use dots. The mask design varies with manufacturer.

Dots and stripes cannot be easily compared because the distance between
adjacent spots is different and depends upon orientation and the way measured.
Because there is ,.[3/2 difference, a 0.24-mm aperture-grille pitch is roughly
equal to a 0.28-mm shadow mask.

Figure 7-4 illustrates one possible arrangement of color dots. The dot pitch
is the center-to-center spacing between adjacent like-colored dots. The three dots
create a triad. Because each electron beam passes through the same hole, the
mask pitch is the same as the triad pitch. Table 7-3 relates the triad dot pitch to
the more common definitions of CRT resolution.

Table 7-3
RESOLUTION OF COLOR DISPLAYS
RESOLUTION TRIAD DOT PITCH

Ultra-high < 0.27 mm

High 0.27 - 0.32 mm
Medium 0.32 - 0.48 mm

Low > 0.48 mm

 CRT-BASED DISPLAYS 209

,..'2,'0~\\
\/- \

\888888
---000008
0.2Bmm 088888
---eeeeee
8@8088
Figure 7-4. Arrangement of phosphor dots on a CRT. There is one
shadow-mask hole for each triad. In 1997, most computer monitors had
a triad pitch of 0.28 mm. The triad is outlined.

By varying the ratio of red, green, and blue, a variety of effective color
temperatures can be achieved. The video signal is simply a voltage with no color
temperature associated with it. The display electronics can be adjusted so that
a white target appears as if illuminated by a source with color temperature range
somewhere between 3200 K and 10,000 K. Most displays are preset to either
6500 K or 9300 K. As the color temperature increases, whites appear to change
from a yellow tinge to a blue tinge. The perceived color depends on the adapting
illumination (e.g., room lights). Setting the color temperature to 9300K provides
aesthetically pleasing imagery and this setting is unrelated to the actual scene
colors (see Section 5.3.2., Color Correction, page 165).

Usually there is no relationship between the color temperature and contrast

of a scene and what is seen on a display. This is not the fault of the camera and
display manufacturers; they strive for compatibility. The observer, who usually
has no knowledge of the original scene, will adjust the display for maximum
visibility and aesthetics.
210 CCD ARRAYS, CAMERAS, and DISPLAYS

For consumer television and consumer computer graphics applications,

adequate color imagery is obtained when each primary has 8 bits of intensity
(256 shades of gray). The combination creates 24 bits or 16 million colors.
Here, each intensity level of each primary is considered a color. It does not
indicate that the eye can distinguish all 16 million - only that 16 million
combinations are possible. Finer intensity increments may be used in
commercial graphics displays. These systems may operate at 10 bits or 12 bits
per primary color and provide 2 30 or 236 "colors," respectively.

7.2.3. HDTV
As the name implies, high definition television (HDTV) provides more
resolution than conventional television receivers The HDTV display has a 16:9
aspect ratio compared to the NTSC 4:3 aspect ratio. It is designed to be viewed
at three times the picture height. While the horizontal visual angle subtended by
NTSC systems is about 11 degrees, the HDTV subtends about 33 degrees. This
larger format gives the feeling of teleprescence - that you are there! The
operation of a CRT-based HDTV is the same as any other CRT.

7.3. SPOT SIZE

Display spot size is a complex function of design parameters, phosphor
choice, and operating conditions. It is reasonable to assume that the spot
intensity profile is Gaussian distributed and is radially symmetric with radius r:

I( r 2
(7-3)
L(r) - Ae -z- 05PDT)

As illustrated in Figure 7-5, the spot diameter is defined as the full width at
half-maximum (FWHM) and is:

S = FWHM - V8ln(2) o SPOT = 2.35 o SPOT . (7 -4)

Then
2
-4ln(2) (!.) (7-5)
L(r) = Ae s
 CRT-BASED DISPLAYS 211

The advantage of assuming a Gaussian spot is that many resolution measures are
easily compared and the display MTF is easily specified (discussed in Section
10.7., CRT-based Displays).

1
~
en
z
w
1-
~ 0.5
w
>
~
~
w
0::
0
-3 -2 -1 0 1 2 3
NORMALIZED DISTANCE

Figure 7-5. The spot diameter is the full width at one-half maximum
intensity. The profile has been normalized to <TsPoT·

While the shrinking raster method is listed as a resolution test method, its
primary value is in determining the FWHM spot size. With the shrinking raster
method, every line is activated (Figure 7-6). If the lines are far apart, the
valleys between the lines are visible and the raster pattern is obvious. The lines
are slowly brought together until a flat field is encountered. That is, the entire
screen appears to have a uniform intensity. The resolution is the number of
displayed lines divided by the picture height of the shrunk raster:
X lines (7-6)
R =-
FF y em
212 CCD ARRAYS, CAMERAS, and DISPLAYS

1vcm
Shrunk Raster

Raster = X Lines

Figure 7-6. Shrinking raster resolution test method. The raster is shrunk
until the discrete lines can no longer be discerned.

As the lines coalesce, the MTF drops. Under nominal viewing conditions,
experienced observers can no longer perceive the raster when the luminance
variation (ripple) is less than 5%. The line spacing has been standardized9 to
2usPOT (Figure 7-7). Given a shrunk raster resolution of RFF lines/em, the line
spacing in the shrunk raster is 1/RFF em. Then •JsPoT = 112RFFcm and the spot
size is
235 (7-7)
S = FWHM = -
2RFF

While a flat field condition is desirable, the line spacing may be larger. The
shrinking raster methodology is applicable for CR Ts only and cannot be used for
flat panel displays because the "raster" is fixed in those displays.

Sometimes the spot profile tends to be flat-topped. A Gaussian

approximation is adequate if the standard deviation, usPOT• is calculated from the
full width at the 5% intensity level 10 :
FW@ 5% level FW@ 5% level
CJ SPOT - (7-8)
J81n(20) 4.9
 CRT-BASED DISPUYS 213

output

Gaussian spots FWHM beam diameter

j
raster hne center

Figure 7-7. When the line spacing is 2asroT, the spots appear to merge
and this produces a flat field. The peak-to-peak ripple is 3% and the
MTF associated with the residual raster pattern is 0.014. The output is
the sum of all the discrete line intensities.

The beam cannot be truly Gaussian because the Gaussian distribution is

continuous from - oo to + oo . Amplifiers may reasonably reproduce the Gaussian
beam profile within the -3asPoT to + 3asPoT limits or a smaller portion. No simple
relationship exists between electronic bandwidth and spot size. That is, monitor
performance cannot be determined simply by the electronic bandwidth. The
bandwidth number quoted depends on the manufacturer's philosophy on rise
time issues. Monitors with the same electronic bandwidth may have different
resolutions.

7.4. DISELS
Monitor disels are not related to the camera disels. Each is generated by the
respective designs. Cameras with a standard video output may not have the
resolution suggested by the video standard. For example, a camera containing
an array of 320 x 240 pixels may have its output formatted into EIA 170 timing.
This standard suggests that it can support approximately 437 x 485 pixels (see
Table 5-2, page 153). The signal may be digitized by a frame grabber that
creates 1000 x 485 datels. The displayed image still represents 320 x 240
pixels. See Section 5.2.2., Broadcast/Non-broadcast Design (page 155), for a
discussion of pixels, datels, and disels.
214 CCD ARRAYS, CAMERAS, and DISPLAYS

A disel is the smallest imaginary area that can exist on a display (see Section
1.5., Pixels, Datels, Disels, and Resets, page 14). In the display literature,
"pixel" is used. If necessary, the reader should mentally insert pixel wherever
disel is written. The disel size indicates how many individual pieces of
information the system can carry. It does not indicate the extent to which the
detail will be resolved. The distance between adjoining disels is related to
resolution. As the distance decreases, disels overlap and the resolution becomes
poorer. This is further discussed in Section 7. 6 .. Addressability.

The vertical disel size is defined by the raster pitch. It is simply the display
height divided by the number of active scan lines. This is dictated by the video
standard. With an optimum system, the electron beam diameter and video
bandwidth are matched in the horizontal direction. For television receivers, the
number of monitor disels is matched to the video standard. These values were
given in Table 5-2, page 153.

The definition of a disel for color systems is more complicated. The

representation shown in Figure 7-3 (page 208) implies that the electron beam
passes through one hole only. Due to alignment difficulties between the mask
and phosphor dots and because the phosphor may have imperfections, CRTs
have electron beams that encompass several mask holes. The beam width at the
5% intensity level typically illuminates about 2. 5 holes. Encompassing a non-
integral number of holes minimizes aliasing and color moire patterns. 11 The
shadow mask samples the continuous electron beam to create a sampled output
(Figure 7-8). The phasing effects created by the shadow mask are obvious.

Figure 7-9 illustrates the relationship between the shadow-mask pitch and the
beam profile. Assume the shadow-mask pitch is 0.28mm. If the 5% diameter
covers 2.5 holes, then, using Equation 7-9, asPOT = (0.28)(2.5)/4.9 = 0.143mm.
The spot size (Equation 7-4) isS= (2.35)(0.143) = 0.336mm. In Figure 7-9,
the spots are contiguous and each is considered a disel.
 CRT-BASED DISPLAYS 215

"green" electron
beam profile

- - - - - -shadow mask

green image

_Ll1_ blue image

red image

white image

Figure 7-8. Representative image of a single spot on a color CRT. The

"green" electron beam is shown, The red and blue beams (not shown)
arrive at slightly different angles and provide different intensity profiles.
The "sum" appears white when the dot size is less than the eye's
resolution limit.
 216 CCD ARRAYS, CAMERAS, and DISPLAYS

,_ _ _ _ _ 5Diameter
percent

888
I
Disel

~
888
- - Disel - - - Disel - - - - - Disel -

Figure 7-9. Five percent intensity level illuminating 2.5 shadow-mask

holes. The spot size (FWHM) is equal to the disel size. If the shadow-
mask pitch is X mm, then the beam 5% diameter is 2.5X mm, CTsPoT =
0.51Xmm, and the disel is 1.2Xmm.

The disel format is the arrangement of disels into horizontal and vertical
rows. Table 7-4 lists the common formats found in computer color monitors.
The format represents the CRT controller's ability to place the electron beam in
a particular location. The CRT is an analog device even though the format is
listed as discrete values. These monitors are said to be digitally addressed. The
disel format is (erroneously) called the resolution. The relationship between
resolution and format is discussed in Section 7.6., Addressability.

Disel density is usually specified in units of dots per inch. Table 7-5
provides the approximate relationship between "resolution" and disel density.
Confusion exists because dots also refer to the individual phosphor sites. In this
context, a "dot" is a disel. The horizontal disel density is the number of
horizontal disels (Table 7-4) divided by the honzontal display size.
 CRT-BASED DISPLAYS 217

Table 7-4
FORMAT OF COLOR DISPLAYS
TYPE FORMAT

CGA (color graphics card adapter) 340 x 200 disels

640 x 350 and
EGA (extended graphics adapter)
640 X 400
VGA (video graphics array) 640 X 480
SVGA (super video graphics array) 800 X 600
XGA (extended graphics array)
1024 X 768
Typical of multi-sync displays

High resolution > 1024 X 1024

Table 7-5
TYPICAL DISEL DENSITY
DISEL DENSITY
"RESOLUTION"
(dpi)

Ultra-high > 120

High 71-120

Medium 50-70

Low <50

Figure 7-10 illustrates the "resolution" of three displays. For a fixed format,
as the display size increases, the disel size also increases to yield a decreasing
disel density. Plotted also is the human eye's resolution capability of 1 arc
minute per line expressed at a viewing distance three times the display height.
The 1280 x 960 display provides about the highest perceivable resolution. A
higher disel density will be beyond the eye's resolution limit and therefore
ineffective. While consumer receivers are viewed at 4.2 times the display height,
computer monitors are viewed at a closer distance. Figure 7-10 illustrates that
the observer is an integral component of perceived resolution. When monitors
are designed, a nominal viewing distance is assumed.
218 CCD ARRAYS, CAMERAS, and DISPLAYS

The user is typically interested in performance and not in specific design

specifications. For example, the dot pitch is typically superfluous information
to the user. Disel size and the display modulation transfer function (discussed
in Section 10.7., CRT-based Displays) are usually more important.

"resolution"
140 ultra-high
~- 120
ii)'fi
z r::: 100 high
w:.::: 3
c..!!! 80
...JGI
~ :6 60
a:- 40
20~~~~~~--+-~----~~--~-.~
10 12 14 16 18 20 22
DIAGONAL SCREEN SIZE (inches)
Figure 7-10. "Resolution" of three displays: (1) 640 x 480, (2) 1024 x
768, and (3) 1280 x 960. The observer's resolution is plotted for a
viewing ratio of three. The 1280 X 960 display is near the observer's
ability to see the disels.

7.5. RESOLUTION

Resolution suggests something about the highest spatial frequency that can
be seen. As an electron beam moves across the screen, its intensity can be
modulated. The maximum rate at which the beam can be modulated from full
off to full on defines the maximum number of (monitor) disels that can be
written in a row. The beam moves down vertically to the next line and again
travels left to right. The distance the beam moves down is the vertical extent of
a disel. While this can be any distance, it is usually selected so that the
individual raster lines cannot be seen. Resolution and addressability specify
monitor performance. They are combined into the resolution/addressability ratio
(RAR). The RAR provides guidance to monitor design.

Resolution metrics are intertwined with television standards such that they
are often presented together. Nearly all televisions are built to the same basic
design so that by specifying EIA 170 or NTSC, the television receiver bandwidth
and spot size is implied. In this sense, a video standard has a "resolution."
 CRT-BASED DISPLAYS 219

Resolution is independent of display size. Displays can be built to any size

and the viewing distance affects the perceived resolution. For example, if a
screen is designed for viewing at 45 inches, then it may have a 10-inch diagonal
(see Figure 7-1, page 204). If viewing will be at 90 inches, the entire display
can simply be made larger (20-inch diagonal). The larger display has larger
raster lines. But the video electronic bandwidth is the same and the resolution
remains constant. When viewed at 45 inches, the larger display will appear to
have poorer image quality. During design, the selected display size is based
upon an assumed viewing distance.

7.5.1. VERTICAL RESOLUTION

The vertical resolution is limited by the number of raster lines. For example,
NTSC compatible monitors display 485 lines. However, to see precisely this
number, the test pattern must be perfectly aligned with the raster pattern.
Because the test pattern may be randomly placed, the pattern at Nyquist
frequency can have a zero output if 180° out-of-phase (discussed in Section
8.3.3., Image Distortion). To account for random phases, the Kell factor 12 is
applied to the vertical resolution to provide an average value. A value of 0.7 is
widely used:
RVERTICAL = (activescanlines) (Kellfactor) . (7-9)

Timing considerations force the vertical resolution to be proportional to the

number of vertical lines in the video signal. Therefore, NTSC displays have an
average vertical resolution of (0.7)(485) = 340 lines. PAL and SECAM displays
offer an average of (0.7)(575) = 402 lines of resolution.

7.5.2. THEORETICAL HORIZONTAL RESOLUTION

While the number of horizontal disels was given in Table 5-2, page 153 (see
also Section 5.2.2., Broadcast/Non-broadcast Design, page 155), the theoretical
horizontal resolution is usually normalized to the picture height. If the video
electronics has a bandwidth of fvmEo• then the theoretical number is
(2fvwE0 )(activeline time) TVL (7-10)
R1VL -
aspect ratio PH
220 CCD ARRAYS, CAMERAS, and DISPLAYS

The units are television lines/picture height or TVLIPH. Unfortunately,

careless usage results in using only "lines." Note that there are two TV lines per
cycle. While the video bandwidth can be any value, for consumer applications,
it is matched to the video standard. For NTSC television receivers, the nominal
horizontal resolution is (2)(4.2MHz)(52.45 JLS)/(4/3) = 330TVL/PH. PAL and
SECAM values are approximately 426 and 465 TVLIPH, respectively. These
values change slightly depending on the active I ine time and bandwidth selected.

Displays designed for scientific applications often operate with standard

video format but have a much wider video bandwidth. These monitors can
display more horizontal TV lines per picture height (TVLIPH) than that
suggested by, say, the NTSC video transmission bandwidth. The horizontal
bandwidth can be any value and therefore the theoretical horizontal resolution
can be any value.

7.5.3. TV LIMITING RESOLUTION

Whereas the flat field condition exists when two adjacent lines cannot be
resolved, the TV limiting resolution is a measure of when alternate vertical bars
are just visible (on-off-on-off). The standard resolution test target is a wedge
pattern with spatial frequency increasing toward the apex of the wedge. It is
equivalent to a variable square wave pattern.

The measurement is a perceptual one and the results vary across the
observer population. The industry selected9 the limiting resolution as a bar
spacing of 1.18crsPoT· This result is consistent with the flat field condition. The
flat field condition was determined for two Gaussian beams whereas square
waves are used for the TV limiting resolution test. Suppose the Gaussian beams
in Figure 7-7 (page 213) were separated by an imaginary "black" beam. Then
the beams would be separated by just crsPoT but the intensity distribution would
be identical to Figure 7-7. The black beam does not contribute to the visible
image. For TV limiting resolution, the "on" lines are separated by (2)(1.18
crsPoT). Because bar targets are used, the resultant image is not a precise
Gaussian beam and a larger line separation is required (1.18crsPoT versus crsroT)
for the TV limiting resolution.

The flat field condition and high TV limiting resolution are conflicting
requirements. For high TV limiting (horizontal) resolution, crsroT must be small.
But raster pattern visibility (more precisely, invisibility) suggests that crsPoT
should be large. This is further discussed in Section 7.6., Addressability.
 CRT-BASED DISPLAYS 221

Example 7-1
SPOT SIZE versus TV LIMITING RESOLUTION

Consider a 14-inch diagonal monitor used to display an image encoded in the

EIA 170 format. Assume the raster line spacing is 2usPoT· What is the horizontal
resolution?

The vertical extent of the 4:3 aspect ratio monitor is 8.4". With 485 lines,
each line is separated by 0.0173". The value CJsPoT is 0.0173/2 = 0.00865" and
the spot size (FWHM) is 0.0204".

EIA 170 contains approximately 440 elements in the horizontal direction.

The display width is 11.2 inches and each element width is 11.2/440 = 0.0254".
But TV limiting resolution suggests that a bar width of (1.18)(0.0865) = 0.102"
can be perceived. Because the element width due to the video bandwidth is
narrower than the TV limiting resolution, resolution is limited by the spot size.
That is, resolution is not limited by the EIA 170 bandwidth.

===============================================~

7.5.4. MTF
TV limiting resolution occurs when the bar spacing is 1.18usPoT· Because a
cycle consists of two TV lines, the square wave fundamental frequency is
11(2.35usPoT) and the MTF is 0.028. Conversely, the display resolution is
selected as that spatial frequency that provides an MTF of 0.028. The MTF
equation is found in Section 10.6., CRT-based Displays. Any spatial frequency
could be used to specify resolution and the slightly different values are cited in
the literature. For commercial applications, the resolution is often specified by
the spatial frequency where the MTF is 10%.

But square waves are the most popular test patterns. A specification could
be: The CTF must be greater than 10% when viewing 320 line pairs. This is
equivalent to 640 disels in an on-off-on-off configuration). Note that by using
square wave targets, the system response is the contrast transfer function and not
the MTF (discussed in Section 9.5., Contrast Transfer Function). Careless usage
results in calling the system response to square waves, the MTF. This makes it
extremely difficult to compare specifications or to calculate performance from
published specifications.
222 CCD ARRAYS, CAMERAS, and DISPLAYS

When designing a simulation system, the distance to the display and display
size are selected first. These are based on physical constraints of the overall
system design. The simulation industry generally accepts resolution as that
spatial frequency (line pair) which provides a CTF of 10% at this viewing ratio.
Because the number of lines depends on what is seen visually, the simulation
industry uses units of optical line pairs (OLP)_ This translates into arc-minutes
per line pair.

7 .6. ADDRESSABILITY

Addressability is a characteristic of the display controller and represents the

ability to select and activate a unique area on the screen. It defines how
precisely an electron beam can be positioned on the CRT screen. Addressability
is given as the number of discrete lines per picture height or disels per unit
length. With this definition, the inverse of addressability is the center-to-center
spacing between adjacent disels. Addressability is the image format discussed in
Section 7.4., Disels (page 213), and provided in Table 7-4 (page 217).

Addressability and resolution are independent of each other. If the resolution

is low, successive lines will overwrite preceding lines. If addressability is low,
adjacent raster lines will not merge and they will appear as stripes. This was
illustrated in Figure 7-7 (page 213) where the line spacing is inverse of the
addressability and resolution is the inverse of asPOT· Figure 7-7 illustrates good
addressability (because lines can be placed close together) but poor resolution
(because lines cannot detected when the modulation is low).

There are two opposing design requirements. The first design requirement
is that the raster pattern be imperceptible to the observer. This is called the
adjacent disel requirement. If the display meets this requirement, the picture will
appear uniform and solid. It provides the flat field condition. Alphanumeric
characters will appear well constructed and highly legible. The second design
requirement is the alternating disel requirement. Here, individual lines (one disel
on and one disel off) should be highly visible.

Increased addressability favors the adjacent disel requirement. Placing large

spots close together will eliminate the visibility of the raster. But this also
reduces the visibility of alternating disels. Similarly an increase in resolution
favors the alternating requirement but may make the raster pattern visible.
 CRT-BASED DISPLAYS 223

The resolution/addressability ratio 13 (RAR) is the ratio of FWHM spot size

to disel spacing, P:

RAR=-
s 2.36 a SPOT (7-11)
p p

where P is the disel pitch. For example, if a display is 27.5 em high and it
displays (addresses) 1024 lines, then Pis 27.5 cm/1024lines = 0.27 mm/line.
Assuming a 0.28-mm-wide spot then RAR is 0.28/0.27 = 1.04.

As shown in Figure 7-7 (page 213), the summation of periodically spaced

spots creates a modulated output. Modulation is defined as (V MAX- V MrN)/(V MAX
+ VMIN) where VMAx and VMIN are the maximum and minimum values in the
output. Figure 7-11 illustrates the adjacent disel (on-on-on-on) and alternating
disel (on-off-on-off) modulation as a function of the RAR. A RAR of 1.0 is
considered desirable. In Figure 7-7, the RAR is 1.18.

Before the RAR was defined, display design was based on the
experimentally determined flat field condition. The flat condition occurs when
P = 2crsPoT or RAR = 1.18. Here, the adjacent disel modulation is near zero -
the lines are not visible. The TV limiting resolution describes the alternating
disel situation. Now, P = 1.18crsPoT• RAR = 2, and the alternating disel
modulation is near zero (e.g., the lines are not discernible).

z 0.8
0 alternating
i= 0.6 disels
<
..J
::::) 0.4
c
0 0.2
::!:
0
0 0.5 1.5 2 2.5
RAR
Figure 7-11. Modulation based on Gaussian beam profiles for the
adjacent disel and alternating disels. A RAR of one is considered
desirable. The modulation of the alternating disel is associated with the
display's Nyquist frequency. The modulation of the adjacent disel is
associated with the display's sampling frequency. These are imaginary
frequencies because the CRT is an analog device. They represent the
ability to place a spot at the desired location on the CRT faceplate.
224 CCD ARRAYS, CAMERAS, and DISPLAYS

Note that the modulation transfer function is traditionally defined for

sinusoidal inputs. The residual ripple is not truly sinusoidal and therefore the
modulation is not exactly what would be expected if the input was sinusoidal.
That is, the modulation shown in Figure 7-11 is not a true MTF. The measured
display response may provide different values than those shown in Figure 7-11.

The alternating disel modulation represents one Gaussian beam on and one
off. This is different from one bar on and one bar off. Resolution tests are
performed with square waves. For an analog system, the contrast transfer
function is 417r times greater than the MTF (discussed in Section 9.5., Contrast
Transfer Function).

If the horizontal format supports 640 disels (see Table 7-4, page 217), then
320 cycles should be perceived. But the modulation is not specified at this
resolution. For high quality monitors, the 320 cycles are easily visible (RAR <
1). With average monitors, the 320 cycles are visible but with reduced
=
modulation (RAR 1). On a poorly designed monitor, the cycles will be barely
perceived (RAR > 1).

While any disel format can be specified, the RAR determines if the disels
are visible. For the configuration shown in Figure 7-9, (page 216), if the
shadow mask pitch is 0.28mm, then the disels will be separated by 0.336 mm
when the RAR is one. If using a 13-inch diagonal display, the width is 264mm
and 786 horizontal disels are viewable when the RAR = 1. More disels can be
displayed, but with reduced modulation (Table 7-6 and Figure 7-12). Thus the
disel format, by itself, does not fully characterize the display capability. For
optimized displays, it is reasonable to assume that the RAR is about one.

Table 7-6
ALTERNATING DISEL MODULATION
13-inch diagonal display and 0.28-mm shadow-mask pitch
5% intensity level illuminating 2.5 shadow-mask holes
DISELS RAR MODULATION
800 1.02 0.75
900 1.15 0.61
1000 1.27 0.47

1100 1.40 0.35

1200 1.53 0.25
 CRT-BASED DISPLAYS 225

ADDRESSABLE DISELS

Figure 7-12. 13-inch diagonal display alternating disel (Gaussian beam)

modulation. The shadow-mask pitch is 0.28mm. The 5% intensity level
illuminates 2.5 shadow-mask holes.

ANSI!HFS 100-1988 requires that the adjacent disel modulation be less than
20%. This deviates significantly from the flat field condition (adjacent disel
modulation is 0.014). ISO 9241 recommends that the alternating disel
modulation be greater than 0.4 for monochrome and greater than 0.7 for color
systems. For monochrome systems, the RAR will vary from 0.8 to 1.31. For
color systems the RAR varies from 0.8 to 1.04. These standards were developed
to assure legibility of alphanumeric characters.

If the RAR is too high, then the output of two adjacent "on" lines is much
greater then the output of two separated lines due to the summing effect (Figure
7-13). For the flat field condition (Figure 7-7, page 213), the RAR is 1.18 and
the summed lines are 27% brighter than an individual line. If the RAR is very
large, the intensity of several adjacent lines "on" can be significantly brighter
than a single line "on". Thus, there is also a tradeoff in intensity with RAR.

Multi-sync monitors automatically adjust the internal line rate to the video
line rate. These monitors can display any format from 525 lines up to 1225
lines. Because the displayed image size remains constant, the vertical line
spacing changes. At the 1225 line format, the raster lines are close together and
form a uniform (flat) field. At the lower line rates, the individual raster lines are
separated and may be perceived. Because the same video amplifiers are used for
all line rates, the horizontal resolution is independent of the line rate.
226 CCD ARRAYS, CAMERAS, and DISPLAYS

That is, the RAR in the horizontal direction stays constant whereas the RAR in
the vertical direction changes with the number of lines. If the vertical RAR is
one at 1225 lines, then it must be 525/125 = 0.43 at 525 lines.

1.4
1-
:::J 1.3
c..
1-
:::J 1.2
0
1.1
c
w 1
:!
:! 0.9
:::J
U)
0.8
0.8 0.9 1.1 1.2 1.3 1.4

RAR
Figure 7-13. Summed output of contiguous lines "on". Ideally, the
intensity should remain constant whether one or multiple lines are "on".

~=============================================
Example 7-2
DISPLAY RESOLUTION

A multi-sync monitor has a horizontal resolution of 800 TV lines per picture

height. If the display diagonal is 14", what is the spot size? Are the raster lines
visible?

For a 4:3 aspect ratio display, the CRT vertical extent is 284 mm. The
estimated spot size is 284/800 = 0.355 mm and UsroT is S/2.35 = 0.335/2.35
=0.15 mm. If the number of displayed lines is NuNE• the RAR is

RAR - s NUNE = 0.355 N - NUNE (7-12)

H 284 LINE 800
MONITOR

With 485 displayed lines (EIA 170), the RAR is 0.61. From Figure 7-11 the
raster pattern (adjacent disels) is quite visible. For EIA 343A with 809 active
lines, the RAR is at the desired value of one. At 946 active lines, the RAR
increases to 1.18. Here the alternating disel modulation starts to drop and the
vertical resolution starts to decrease.
============================================~
 CRT-BASED DISPLAYS 227

7.7. SHADES OF GRAY

An often used, but widely misunderstood, concept is "shades of gray."

While displays may be specified in shades of gray where each step is square root
of 2, human vision cannot. The misunderstanding originated with the thought
that the eye requires a ..j2 increment to just detect an increment in luminance.
Studies of visual perception clearly indicate that just noticeable increments in
luminance vary with background luminance level, size and shape of the target,
the ambient luminance, and a variety of other variables. To use a single number
to characterize the human eye is too simplistic. The ..j2 approach is merely
another way to indicate display dynamic range. No other inference should be
made about the definition.

7.8. CHARACTER RECOGNITION

Computer monitors are designed to for maximum legibility of alphanumeric

characters and enhanced graphical capability. Readable characters can be formed
in any block of disels from 5 x 7 to 9 x 16 and still be aesthetically pleasing.
More complex symbols such as Japanese characters may require 16 x 16 disels.

The ability to see detail is related to the RAR. Figure 7-14 illustrates three
letters and the resultant intensity traces. The RAR must be near one so that a
reasonable contrast ratio exists between "on" and "off" disels. With reasonable
contrast, the inner detail of the character is seen and the character is legible.
Similarly, with a reasonable contrast ratio, adjacent letters will appear as
separate letters. With characters, the alternating disel pattern is called one stroke
separated by a space.

Figure 7-15 illustrates how reduced modulation affects character legibility.

For easy interpretation, the system contains a shadow mask but only white and
black can be seen. The image is magnified so that the individual phosphor dots
can be seen. The sampled nature of the Gaussian electron beam is evident. As
the beam width increases, the modulation decreases and the RAR increases. The
characters become blurry as the RAR increases.
 228 CCD ARRAYS, CAMERAS, and DISPLAYS

Slit
Photometer
.
Scan
Path

Slit Photometer
Scan Path Luminance

(Within) (Between)

Figure 7-14. The RAR must be near one so that the inner details of
characters are visible and the double width appears solid. Similarly with
RAR near one, two adjacent characters appear separate. Each dot
represents a disel. Lower RARs will make the characters appear very
sharp. However, the raster pattern will become visible and the characters
will not appear solid.

7. 9. CONTRAST

Selection of one minute of arc as a measure of human visual acuity provided

back-of-the-envelope calculations for selecting disel size, display size, and
viewing distance. However, the eye's response varies according to the displayed
contrast.

The air-glass interface and the phosphor reflect ambient light. This reduces
contrast between disels and reduces the display's color gamut. As a result, most
displays are measured in a darkened room where the reflected light is less than
1 % of the projected light. Typically the contrast is measured with a
checkerboard pattern consisting of 16 alternating rectangles. The average value
of the eight white rectangles and the average value of the eight black rectangles
are used to calculate the contrast ratio.
 CRT-BASED DISPLAYS 229

Figure 7-15. Complex symbols are more legible when the display
modulation is high. Each dot represents a hole in the shadow mask. Top:
modulation= 0.5 at the Nyquist frequency. Bottom: modulation= 0.1
at the Nyquist frequency. When viewed at 10 feet, the individual dots
are no longer perceptible. At this distance, the top characters appear
sharp (RAR = 1. 25) and the bottom ones appear fuzzy (RAR = 1. 8). By
courtesy of the Mitre Corporation.
230 CCD ARRAYS, CAMERAS, and DISPLAYS

The perceived contrast ratio is

LWHITE + LAMBIENT· reflectivity

Contrast ratio - (7-13)

where LwHrrE and LsLAcK represent the luminances from the white and black
rectangles, respectively. The reflections are the same regardless if a disel is on
or off. About 4 percent of the ambient light (LAMBIENT) is reflected at the air-
glass interface. The CRT phosphor is an excellent Lambertian reflector with a
typical reflectivity of 70%. "Black" displays minimize these reflections and
thereby increase the perceived contrast.

Unfortunately, specifications that are based on no ambient lighting can only

be achieved in a totally darkened room. Safety considerations require some
lighting in all rooms so that the specification can never be achieved in practice.

The contrast ratio for typical displays is about 20: 1 under normal ambient
lighting conditions. This decreases as the ambient light increases. The eye
operates in the reverse way. As the ambient lighting increases, more contrast is
necessary to see target-background differences. Thus, as the displayed contrast
decreases in bright light, the eye requires more contrast. This explains why
television imagery is so difficult to see in sunlight and why it appears so bright
in a darkened room.

7.10. REFERENCES
1. L. E. Tannas Jr., ed., Flat panel Displays and CRTs, Van Nostrand, New York, NY (1985).
2. J. A. Castellano, Handbook of Display Technology, Academic Press, New York, NY (1992).
3. S. Sherr, Electronic Displays, Second edition, Wiley and Sons, New York, NY (1993).
4. J. C. Whitaker, Electronic Displays, McGraw-Hill, New York, NY (1994).
5. J. Peddie, High-Resolution Graphics Display Systems, Windcrest/McGraw-Hill, NY (1994).
6. L. W. MacDonald, ed., Display Systems, Design and Applications, Wiley, NY (1997).
7. W. Cowan, "Displays for Vision Research," in Handbook of Optics, M. Bass, ed., Second
edition, Vol. 1, Chapter27, McGraw-Hill, New York. NY (1995).
8. T. Olson, "Behind Gamma's Disguise," SMPTE Journal, Vol. 104(7), pp. 452-458 (1995).
9. L. M. Biberman, "Image Quality," in Perception of Displayed Information, L. M. Biberman,
ed., pp. 13-18, Plenum Press, New York (1973).
10. P. G. J. Barten, "Spot Size and Current Density Distributions of CRTs," in Proceedings of the
SID, Vol. 25(3), pp. 155-159 (1984).
11. A. Miller, "Suppression of Aliasing Artifacts in Digitally Addressed Shadow-mask CRTs,"
Journal of the Society of Information Display, Vol. 3(3), pp 105-108 (1995).
12. S.C. Hsu, "The Kell Factor: Past and Present," SMPTE Journal, Vol. 95, pp. 206-214(1986).
13. G. M. Murch and R. J. Beaton, "Matching Display Resolution and Addressability to Human
Visual Capacity," Displays, Vol. 9, pp. 23-26 (Jan 19R8).
8
SAMPLING THEORY
Sampling is an inherent feature of all electronic imaging systems. The scene
is spatially sampled in both directions due to the discrete locations of the
detector elements. This sampling creates ambiguity in target edges and produces
moire patterns when viewing periodic targets. Aliasing becomes obvious when
(a) the image size approaches the detector size and (b) the detectors are in a
periodic lattice (the normal situation). Spatial aliasing is rarely seen in
photographs or motion pictures because the grains are randomly dispersed in the
photographic emulsion.

If the detector size and spacings are different in the horizontal and vertical
directions, sampling effects will be different in the two directions. This leads to
moire patterns that are a function of the target orientation with respect to the
array axis. Although sampling is two-dimensional, for simplicity sampling
effects will be presented in one-dimension.

The highest frequency that can be faithfully reconstructed is one-half the

sampling rate. Any input signal above the Nyquist frequency, fN (which is
defined as one-half the sampling frequency, f5), will be aliased down to a lower
frequency. That is, an undersampled signal will appear as a lower frequency
after reconstruction (Figure 8-1).

input aliased signal

Figure 8-1. An undersampled sinusoid will appear as a lower frequency

after reconstruction. The sampling frequency is f5 = 1/T. When Tis the
measured in mrad, mm, or time, the sampling frequency is expressed in
cycles/mrad, cycles/mm, or Hz, respectively.

231
232 CCD ARRAYS, CAMERAS, and DISPLAYS

After aliasing, the original signal can never be recovered. It would appear
that aliasing is an extremely serious problem. However, the extent of the
problem depends upon the final interpreter of the data.

Signals can be undersampled or oversampled. Undersampling is a term used

to denote that the input frequency is greater than the Nyquist frequency. It does
not imply that the sampling rate is inadequate for any specific application.
Similarly, oversampling does not imply that there is excessive sampling. It
simply means that there are more samples available than that required by the
Nyquist criterion.

After aliasing, the original signal can never be recovered. Undersampling

creates moire patterns (Figure 8-2). Diagonal lines appear to have jagged edges
or "jaggies." Periodic structures are rare in nature and aliasing is seldom
reported when viewing natural scenery although aliasing is always present. It
may become apparent when viewing periodic targets such as test patterns, picket
fences, plowed fields, railroad tracks, and Venetian blinds. Distortion effects can
only be analyzed on a case-by-case basis. Because the aliasing occurs at the
detector, the signal must be band-limited by the optical system to prevent it.

We have become accustomed to the aliasing in commercial televisions.

Periodic horizontal lines are distorted due to the raster. Cloth patterns such as
herringbones and stripes produce moire patterns. Cross-color effects occur in
color imagery. Many video tape recordings are undersampled to keep the price
modest and yet the imagery is considered acceptable when observed at normal
viewing distances.

Figure 8-2. A raster scan system creates moire patterns when viewing
wedges or star bursts.
 SAMPLING THEORY 233

Aliasing is reduced to an acceptable level in consumer color sensors. For

specific applications such as medical imaging, it may be considered
unacceptable. In medical imagery, an aliased signal may be misinterpreted as a
medical abnormality that requires medication, hospitalization, or surgery.

While aliasing is never considered desirable, it is accepted in scientific and

military monochrome applications where the MTF must be maintained. High
MTF is related to image sharpness. Aliasing becomes bothersome when the
scene geometric properties must be maintained as with mapping. It affects the
performance of most image processing algorithms.

Sampling theory states that the frequency can be unambiguously recovered

for all input frequencies below Nyquist frequency. It was developed for band-
limited electrical circuits. The extension to imaging systems is straightforward.
However, the eye is primarily sensitive to intensity variations.

Sampling theory is traditionally presented from a modulation transfer

function viewpoint. MTF theory is presented in Chapter 9 and the subsystem
MTFs are discussed in Chapter 10. Linear system theory is used for system
analysis because of the wealth of mathematical tools available. Approximations
are used to account for sampling effects. While these approximations are
adequate to describe the MTFs, they do not indicate how imagery may be
distorted.

Sampled data systems are nonlinear and do not have unique MTFs. 1-6 The
"MTF" depends on the phase relationships of the scene with the sampling
lattice. Superposition does not hold and any MTF derived for the sampler
cannot, in principle, be used to predict results for general scenery. To account
for the nonlinear sampling process, a sample-scene MTF is added (discussed in
Section 10.10., Sampling Effects). As the sampling rate increases, the MTF
becomes better defined. As the sampling rate approaches infinity, the system
becomes an analog system and the MTF is well defined. Equivalently, as the
detector size and detector pitch decrease, signal fidelity increases. Reference 7
provides an in-depth discussion on sampling effects. This chapter highlights
those results.

The symbols used in this book are summarized in the Symbol List (page
xviii) and it appears after the Table of Contents.
234 CCD ARRAYS, CAMERAS, and DISPLAYS

8.1. SAMPLING THEOREM

The sampling theorem, as introduced by Shannon, 8 was applied to

information theory. He stated that if a time-varying function, v(t), contains no
frequencies higher than fMAx (Hz), it is completely determined by giving its
ordinates at a series of points spaced 1/2fMAx sec apart. The original function
can be reconstructed by using an ideal low-pass filter. To just satisfy Shannon's
theorem, the sampling frequency, f5 , must be 2fMAx· Shannon's work is an
extension of others9 and the sampling theorem is often called the Whittaker-
Shannon theorem.

Three conditions must be met to satisfy the sampling theorem. The signal
must be band-limited, the signal must be sampled at an adequate rate, and a low-
pass reconstruction filter must be present. When any of these conditions are not
present, the reconstructed analog data will not appear exactly as the originals.

In a sampled-data system, the sampling frequency interacts with the signal

to create sum and difference frequencies. Any input frequency, f0 , will appear
as nf5 ± fo after sampling (n = -oo to + oo ). Figure 8-3 illustrates a band-limited
system with frequency components replicated by the sampling process. The base
band (-fH to fH) is replicated at nf5 • To avoid distortion, the lowest possible
sampling frequency is that value where the base band adjoins the first side band
(Figure 8-3c). This leads to the sampling theorem that a band-limited system
must be sampled at twice the highest frequency (f5 ~ 2fH) to avoid distortion in
the reconstructed image. The sampling theorem applies to any periodic input
frequency. The units can be either spatial frequency or electrical frequency.

After digitization, the data reside in data ;mays (e.g., computer memory
location) with nonspecific units. The user assigns units to the arrays during
image reconstruction. That is, the data are read out of the memory in a manner
consistent with display requirements. Because monitors are analog devices, the
data are transformed into an analog signal using a sample-and-hold circuit. This
creates a "blocky" image and can be removed by a reconstruction filter
(discussed in Section 10.6.4., Post-Reconstruction Filter). If the original signal
was oversampled and if the post-reconstruction filter limits frequencies to fN,
then the reconstructed image can be identical to the original image.
 SAMPLING THEORY 235

LL
1-
:e
w
>
~
<
...J
w
0::::

fH
RELATIVE FREQUENCY
(a)

LL
1-
:e
w
2!
1-
:3w
0::::

fH fs-fH f5 +fH 2f 5-fH 2f 5 +fH

RELATIVE FREQUENCY
(b)

LL
1-
:e
w
2!
1-
<
...J
w
0::::

fs f5 +fH 2f 5 2f 5 +fH 3f5

RELATIVE FREQUENCY

(c)
Figure 8-3. Sampling replicates frequencies at nfs ±f. (a) Original band-
limited signal, (b) frequency spectrum after sampling, and (c) when fs =
2fH, the bands just adjoin.
236 CCD ARRAYS, CAMERAS, and DISPLAYS

8.2. ALIASING

As the sampling frequency decreases, the first side band starts to overlap the
base band. It is the summation of these power spectra that create the distorted
image (Figure 8-4). The overlaid region creates distortion in the reconstructed
image. This is aliasing. Once aliasing has occurred, it cannot be removed.
Within an overlapping band, there is an ambiguity in frequency. It is impossible
to tell whether the reconstructed frequency resulted from an input frequency of
fo or nfs ± fo.

All frequency components above fN are folded back into the base band so
that the base band contains

aBASE-BAND = L O(nfs ±f) where nfs ±j~JN • (8-1)

n--oo

To avoid aliasing in the electronic domain, the signal may be passed though
a low-pass filter (anti-aliasing filter). These filters cannot remove the aliasing
that has taken place at the detector. They can only prevent further aliasing that
might occur in the downstream analog-to-digital converter. Figure 8-5 illustrates
the cutoff features of an ideal filter. The ideal filter passes all frequencies below
fN and attenuates all frequencies above fN. The ideal filter is, of course,
unrealizable. 10 It also can produce undesirable effects such as ringing.

::!!
::;)
0:::
t-
o
w
ll.
UJ
0:::
w
3:
0
ll.

RELATIVE FREQUENCY

Figure 8-4. Aliasing alters both the signal <md noise base band spectra.
In a real system, the signal and noise are not necessarily band-limited
and some aliasing occurs.
 SAMPLING THEORY 237
sample
frequency
0.8

LL 0.6
1-
:::! 0.4
0.2

0
0 0.25 0.5 0.75 1.25 1.5 1.75 2

NORMALIZED SPATIAL FREQUENCY

Figure 8-5. An undersampled system must be band-limited to avoid

aliasing. The ideal anti-aliasing filter passes all the signals below fN and
no signal above fN. Whether aliasing is considered undesirable depends
on the application.

In solid state cameras, the sampling frequency is determined by the detector

pitch. That is, the scene is optically sampled by the detectors. If the scene
contains frequencies greater then the array Nyquist frequency, aliasing will
occur. Optical band-limiting can be achieved by using small diameter optics,
defocusing, blurring the image, or by inserting a birefringent crystal between the
lens and array. The birefringent crystal changes the effective sampling rate and
is found in almost all single chip color cameras (discussed in Section 10.3 .4.,
Optical Anti-Alias Filter). Unfortunately, these approaches also degrade the
MTF (reduce image sharpness) in the base band and typically are considered
undesirable for scientific applications. For medical applications, where aliased
imagery may be misinterpreted as an abnormality, reduced aliasing usually
outweighs any MTF degradation.

According to the sampling theorem, any sampled frequency, f, where f ~

fN, can be uniquely recovered. However, the square wave is the most popular
test target and it is characterized by its fundamental frequency f0 • When
expanded into a Fourier series, the square wave consists of an infinite number
of frequencies (discussed in Section 9.5., Contrast Transfer Function). Although
the square wave fundamental may be oversampled, the higher harmonics will
not. The appearance of the square wave after reconstruction depends on the
relative values of the optical, detector, and electronic MTFs.

Although no universal method exists for quantifying aliasing, it is reasonable

to assume that it is proportional to the MTF that exists above fN. In Figure 8-6
a significant MTF exists above the Nyquist frequency. Whether this aliasing is
objectionable, depends on the camera application.
238 CCD ARRAYS, CAMERAS, and DISPLAYS

If a system is Nyquist frequency limited, then the Nyquist frequency is used

as a measure of resolution. Because no frequency can exist above the Nyquist
frequency, many researchers represent the MTF as zero above the Nyquist
frequency (Figure 8-7). This representation may be too restrictive for modeling
purposes.

Nyquist
frequency
0.8

LL 0.6
!i 0.4

0.2

0~------~----~-------+------~
0 0.25 0.5 0.75

NORMALIZED SPATIAL FREQUENCY

Figure 8-6. Signals above Nyquist frequency are aliased down to the
base band. The area bounded by the MTF above fN may be considered
as an "aliasing" metric. 11

Nyquist Sample
frequency frequency
0.8

LL 0.6
1-
::E 0.4

0.2

0
0 0.25 0.5 0.75 1.25 1.5 1.75 2
NORMALIZED FREQUENCY
Figure 8-7. MTF representation of an undersampled system as a function
of f/fN. This restrictive representation erroneously suggests that there is
no response above fN.
 SAMPLING THEORY 239

8.3. The DETECTOR as a SAMPLER

Figure 8-1 (page 231) illustrated a signal sampled at discrete points. It

typifies extremely small detectors and flash analog-to-digital converters.
Detectors have a finite size and integrate the signal spatially. This spatial
integration reduces the modulation and is the detector MTF.

Sections 8-1 and 8-2 described sampling in generic terms. The variable fwas
used to denote frequency. There are four different frequency domains that can
exist with an electro-optical imaging system (discussed in Section 10 .1.,
Frequency Domains). Imagery at the detector plane is described by image-space
spatial frequency. The horizontal and vertical variables are represented by ui and
vi, respectively. They have units of cycles/rom.

8.3.1. DETECTOR MTF

Figure 8-8 illustrates the spatial integration afforded by a staring array. In

Figure 8-8a, the detector width is one-half of the center-to-center spacing
(detector pitch). That is, only 50% of the input sinusoid is detected. The
detector averages the signal over the range indicated and the heavy lines are the
detector outputs. The output MTF is (VMAx- VM 1N)/(VMAx + VM 1N) where VMAx
and VMIN are the maximum and minimum voltages, respectively. As the detector
size increases, it integrates over a larger region and the MTF decreases
(compare Figure 8-8a with 8-8b). As the phase changes between the detector
location and the peak of the sinusoid, the output voltages change. This creates
a variable MTF caused by sampling.

The detector MTF cannot exist by itself. Rather, the detector MTF must also
have the optical MTF to make a complete imaging system. In the horizontal
direction, the MTF of a single rectangular detector is

(8-2)

where d" is the physical extent of the photosensitive surface.

240 CCD ARRAYS, CAMERAS, and DISPLAYS

Idetector! petectorl

(a)

Idetector detector detector detector

(b)

Figure 8-8. A detector spatially integrates the signal. Figure 8-1 (page
231) represents the output of extremely small detectors. (a) Detector
width is one-half of the center-to-center spacing and (b) detector width
is equal to the center-to-center spacing. (a) Typifies an interline transfer
CCD array in the horizontal direction (see Figure 3-16, page 64) and (b)
is representative of frame transfer arrays. The heavy lines are the
detector output voltages. As the phase changes, the output also changes.

Figure 8-9 illustrates the detector MTF in one dimension. The MTF is
equal to zero when ui = k/dH. The first zero (k = 1) is considered the detector
cutoff, uiD. Any input spatial frequency above detector cutoff will be aliased
down to a lower frequency. It is customary to plot the MTF only up to the first
zero (Figure 8-1 0). The optical system cutoff limits the absolute highest spatial
frequency that can be faithfully imaged. The detector cutoff may be either higher
(optics-limited system) or lower (detector-limited system). Most CCD-based
imaging systems are detector-limited (discussed in Section 10.4., Optics-Detector
Subsystem).
 SAMPLING THEORY 241

0.8

LL 0.6
1-
::!! 0.4

0.2

0
0 0.5 1.5 2 2.5 3
NORMALIZED SPATIAL FREQUENCY

Figure 8-9 Detector MTF as a function of normalized spatial frequency

0

u/u; 0 o Detector cutoff is U;o = lido

Oo8

LL 006
1-
::!! Oo4

Oo2

0
0 Oo25 Oo5 Oo75
NORMALIZED SPATIAL FREQUENCY

Figure 8-10 Typical detector MTF representation as a function of

0

normalized spatial frequency u/U;oo This representation erroneously

suggests that the detector does not respond to frequencies greater than
U;oo
242 CCD ARRAYS, CAMERAS, and DISPLAl'S

8.3.2. DETECTOR ARRAY NYQUIST FREQUENCY

A staring array acts a sampler whose sampling rate is 1/dcc (cycles/mm)

where dec is the detector center-to-center spacing (detector pitch). The pitch in
the horizontal and vertical directions, dccH and dccv. respectively, may be
different. Detector arrays are specified by the fill factor which is the detector
area divided by the cell area. It is dHdvldccHdCI v· Staring arrays can faithfully
reproduce horizontal signals up to u;N = 1/(2dccH). Although the individual
detectors can reproduce higher spatial frequencies, the horizontal scene spectrum
is sampled at U;s = lldccH (Figure 8-11).

Microlenses increase the effective detector size but do not affect the Nyquist
frequency. By increasing the detector area, the detector cutoff decreases. With
microlenses, the array in Figure 8-11b can optically appear as the array in
Figure 8-lla.

Staring arrays are inherently undersampled when compared to the detector

spatial frequency cutoff. Any scene spatial frequency above u;N or V;N will
appear as a lower spatial frequency in the reconstructed image. To avoid
aliasing, the scene frequencies must be optically band-limited similar to that
shown in Figure 8-5 (page 237).

Most single chip color filter arrays have an unequal number of red, green,
and blue detectors. A typical array designed for NTSC operation will have 768
detectors in the horizontal direction with 384 detectors sensitive to green, 192
sensitive to red, and 192 sensitive to the blue region of the spectrum. Suppose
the arrangement is G-B-G-R-G-B-G-R. The "green" detectors will have a center-
to-center spacing as shown in Figure 8-11b.

The spacing of the "blue" and "red" detectors is twice the "green" detector
spacing. This produces a "blue" and "red" array Nyquist frequency that is one-
half of the "green" array Nyquist frequency (Figure 8-12). Other detector
layouts will create different array Nyquist frequencies. The "color" Nyquist
frequencies can be different in the horizontal and vertical directions. These
unequal array Nyquist frequencies create color aliasing in single chip cameras
that is wavelength specific. Black-and-white scenes can appear as green, red, or
blue imagery. 12 A birefringent crystal, inserted between the lens and the array,
changes the effective sampling rate and minimizes color aliasing (discussed in
Section 10.3.4., Optical Anti-Alias Filter).
 SAMPLING THEORY 243

rn
1++1
dCCH

1 Sampling:
0.8
LL 0.6
1-
:::t 0.4
0.2
0
0 0.25 0.5 0.75 1
NORMALIZED SPATIAL FREQUENCY
(a)

D1+- D
+I
D
dCCH

1 , Sampling
0.8
LL 0.6
1-
:::t 0.4
0.2
0
0 0.25 0.5 0.75 1
NORMALIZED SPATIAL FREQUENCY
(b)

Figure 8-11. Two arrays with different horizontal center-to-center

spacings. The detector size, dH, is the same for both. (a) dH/dccH = 1.
This typifies frame transfer devices. (b) dH/dccH = 0.5. This is
representative of interline transfer CCD arrays. The detector MTF is
plotted as a function of uJuiD.
244 CCD ARRAYS, CAMERAS, and DISPLAtS

Aliasing in the vertical direction can be reduced by summing alternate

detector outputs (field integration) in a two-field camera system (see Figure 3-
19, page 67). Here, the effective detector height is 2dv and the cutoff is 112dv
(Figure 8-13). This eliminates aliasing but also reduces the vertical MTF and,
therefore, reduces image sharpness in the vertical direction. Horizontal sampling
still produces aliasing.

1r--:----.:
0.8 'lii
·s .
C"~·
LL 0.6 >- r::: '
1- Z111
: ::::J I

:::E 0.4 r:::cr.

I!! '
Cll
1!!-
0.2 ~

0 0.1 0.2 0.3 0.4 0.5

NORMALIZED SPATIAL FREQUENCY

Figure 8-12. Horizontal detector MTF normalized to uJuiD. Unequally

spaced red, green, and blue sensitive detectors creates different array
Nyquist frequencies. This creates different amounts of aliasing and
black-and-white scenes may break into color. The spacing between the
"red" and "blue" detectors is twice the "green" detector spacing.

1 Nyquist Sampling
LL frequency frequency
1- 0.8 without
:::IE
..J 0.6
<(
0
0.4
~
w 0.2
row summing
>
0
0 0.25 0.5 0.75

NORMALIZED SPATIAL FREQUENCY

Figure 8-13. Alternate detector summing can reduce aliasing in the

vertical direction. Vertical MTF with and without summing normalized
to u/u; 0 .
 SAMPLING THEORY 245

8.3.3. IMAGE DISTORTION

The square wave (bar pattern) is the most popular test target. It is
characterized by its fundamental frequency only. The expansion of a square
wave into a Fourier series clearly shows that it consists of an infinite number of
sinusoidal frequencies. Although the square wave fundamental frequency may
be oversampled, the higher harmonics may not. During digitization, the higher
order frequencies will be aliased down to lower frequencies and the square wave
will change its appearance. There will be intensity variations from bar-to-bar and
the bar width will not remain constant.

Sampling replicates frequencies. When two frequencies are close together,

they create a beat frequency that is equal to the difference. The fundamental and
its replication create a beat frequency of fsEAT = (fs- f0 ) - f0 • The beat frequency
period lasts for N input frequency cycles:

N= (8-3)

Figure 8-14 illustrates Nasa function of f0 /fN. As f0 /fN approaches one, the beat
frequency becomes obvious. Lomheim et. al. 13 created these beat frequencies
when viewing multiple bar patterns with a CCD camera.

An ideal reconstruction filter will eliminate the beat frequency. However,

ideal filters do not exist. Real filters will reduce the MTF and soften edges.
Here a square wave may appear as a sinusoid. Thus, there is a tradeoff between
edge sharpness and aliasing.

en
w
10
..J
(.) 8
>
(.)
u. 6
0
0:: 4
w
m 2
:::iE
~
z 0
0.5 0.6 0.7 0.8 0.9
NORMALIZED FREQUENCY
Figure 8-14. Number of input frequency cycles required to see one
complete beat frequency cycle as a function of f0 /fN.
246 CCD ARRAYS, CAMERAS, and DISPLAYS

Figures 8-15 through 8-18 illustrate an ideal staring array output when the
optical MTF is unity for all spatial frequencies and therefore represents a worst
case scenario. That is, all higher harmonics that would have been attenuated by
the optical system are available for aliasing by the detector. For these figures,
dH/dccH is one. The individual detectors sample the square wave in the same
manner as illustrated in Figure 8-8b (page 240). Section 10.4., Optics-Detector
Subsystem describes how the optical MTF modifies the imagery.

If u.,/u;N= 0.952, the beat frequency is equal to 9.9 cycles of the input
frequency (Figure 8-15). Although a square wave consists of an infinite number
of frequencies, it is characterized by its fundamental, u.,. The target must contain
at least 10 cycles to see the entire beat pattern. A 4-bar pattern may either be
replicated or nearly disappear depending on the phase.

Standard characterization targets, however, consist of several bars.

Therefore the beat pattern may not be seen when viewing a 3-bar or 4-bar
target. The image of the bar pattern would have to be moved ± 1h d (detector
extent) to change the output from a maximum value (in-phase) to a minimum
value (out-of-phase). This can be proven by selecting just four adjoining bars in
Figure 8-15.

~out of phase-1 ~ in phase -1

Figure 8-15. Beat frequency produced by an ideal staring system when
U0 /U;N= 0.952. The value N = 9.9 and dH/dccH = 1. The light line is the
input and the heavy line is the detector output. The beat frequency
envelope is shown.

When U0 /U;N is less than about 0.6 (Figure 8-16), the beat frequency is not
obvious. Now the output nearly replicates the input but there is some slight
variation in pulse width and amplitude. In the region where u.,/u;N is
approximately between 0.6 and 0.9, adjacent bar amplitudes are always less than
the input amplitude (Figure 8-17).
 SAMPLING THEORY 247

When U /U;N is less than about 0.6, a 4-bar pattern will always be seen
0

(select any four adjoining bars in Figure 8-16). When llofU;N < 0.6, phasing
effects are minimal and a phase adjustment of ± 1/2d in image space will not
affect an observer's ability to resolve a 4-bar target. In the region where f0 /fN
is approximately between 0.6 and 0.9, 4-bar targets will never look correct
(Figure 8-17). One or two bars may be either much wider than the others or one
or two bars may be of lower intensity than the others. This phasing effect is a
measurement concern and is not normally included in modeling.

Input frequencies of U0 = u;N/k, where k is an integer, are faithfully

reproduced (i.e., no beat frequencies). When k = 1, as the target moves from
in-phase to out-of-phase, the output will vary from a maximum to zero.
Selection of u;N/k targets avoids the beat frequency problem but significantly
limits the number of spatial frequencies selected.

Figure 8-16. Ideal staring system output when U0 1U;N = 0.522. The value
N = 0.54 and dH/dccH = 1. The light line is the input and the heavy line
is the detector output. The output nearly replicates the input when U0 1U;N
< 0.6.

Figure 8-17. Ideal staring system output when U0 /U;N = 0. 811. The value
N = 2.14. The light line is the input and the heavy line is the detector
output. The output never looks quite right when U0 /u;N is between 0.6
and 0.9.
248 CCD ARRAYS, CAMERAS, and DISPLAYS

Signals with frequencies above Nyquist frequency will be aliased down to

lower frequencies. This would be evident if an in finitely long periodic target was
viewed. However, when u., is less than about 1. 15 uiN• it is possible to select a
phase such that four adjoining bars appear to be faithfully reproduced (Figure
8-18). These targets can be resolved although the underlying fundamental
frequency has been aliased to a lower frequency .

Figure 8-18. Ideal staring system output when U 0 /uiN = 1.094. The value
N = 5. 8. The light line is the input and the heavy line is the detector
output. By selecting the appropriate phase, the output appears to
replicate an input 4-bar pattern.

Systems can detect signals with spatial frequencies above cutoff but cannot
faithfully reproduce them. For example, patterns above the Nyquist frequency
are aliased to a frequency below Nyquist and a 4-bar pattern may appear as a
distorted 3-bar pattern.

The phasing effects shown in Figures 8-15 through 8-18 "disappear" when
the target is moving. Here, each frame provides a different image. For each
target whose spatial frequency is above Nyquist frequency, one, two, or three
bars are visible depending on the relative phase. As the pattern moves, a
different set of bars is visible. The eye blends these individual patterns so that
you perceive four bars. This prompted some researchers 14 to develop a dynamic
test in which the 4-bar target moves.
 SAMPLING THEORY 249

Example 8-1
SYSTEM CUTOFF

A staring array consists of detectors that are 10 p,m x 10 p,m in size. The
detector pitch is 15 p.m. The focal length is 15 em. The aperture diameter is 3
em and the average wavelength is 0.5 p,m. What is the system cutoff?

System cutoff is the smaller of the optical cutoff, detector cutoff, or Nyquist
frequency. In image space, the optical cutoff is uic = D0 /(}..fl) = 400 cycles/mm.
The detector cutoff is uic = 1/dH = 100cycles/mm. The detector pitch provides
sampling every 15 p,m for an effective sampling rate at 66.7 cycles/mm. Because
the Nyquist frequency is one-half the sampling frequency, the system cutoff is
33.3 cycles/mm.

Because object space is related to image space by the lens focal length, the
optical cutoff is 30 cycles/rnrad, detector cutoff is 15 cycles/rnrad, sampling is
10 cycles/rnrad, and the Nyquist frequency is 5 cycles/rnrad.

Because the MTF remains zero for all frequencies higher than the optical
cutoff, any scene frequency above uic will appear as a uniform blob with no
modulation. Spatial frequencies higher than the detector cutoff or the Nyquist
frequency (but less than ud will be aliased down to lower frequencies.
=============================================~

8.4. ALIASING in FRAME GRABBERS and DISPLAYS

For many applications, the camera output is an analog signal that is then
re-sampled by a frame grabber (frame capture device). If the frame grabber's
sampling rate is sufficiently high and the analog signal is band-limited, the
reconstructed signal will replicate the analog signal in frequency, amplitude, and
pulse width.

In principle, the clock rate of an analog-to-digital converter can be set at any

rate. However, to conserve on memory requirements and minimize clock rates,
some frame grabbers tend to just satisfy the Nyquist frequency of a standard
video signal. That is, if the video bandwidth is f8 w, then the frame grabber
sampling clock operates at 2f8 w (see Table 5-2, page 153, for f8 w values). A
frame grabber, with its flash converter, can distort the signal as shown in Figure
8-19. The variation in edge locations affects all image processing algorithms.
250 CCD ARRAYS, CAMERAS, and DISPLAYS

The variations in pulse width can only be seen on a monitor because the frame
grabber data reside in a computer memory. Note that output pulse widths are
equally spaced. They just do not match up with the input signal.

I"" ... I"

.. - ...
Figure 8-19. Frame grabber output after sample-and-hold circuitry when
uju;N = 0.952. The light line is the input and the heavy line is the
output. Here, u;N is the Nyquist frequency created by the frame grabber
clock rate. The output is representative of what is seen on a monitor.

Some frame grabbers have an internal anti-alias filter. This filter ensures that
the frame grabber does not produce any additional aliasing. The filter cutoff is
linked to the frame grabber clock and is not related to the camera output. Once
aliasing has occurred in the camera, the frame grabber anti-alias filter cannot
remove it. If the filter does not exist, the frame grabber may create additional
aliasing. 15

Aliasing is rarely discussed with displays, but under certain conditions can
occur. Many displays employ digital techniques to match the electronic signal
to the CRT characteristics. Ideally, the internal analog-to-digital converter
should have a low-pass filter to eliminate possible aliasing. Although good
engineering design recommends that the low-pass filter be included, cost
considerations may eliminate it. With standard video formats, the bandwidth is
limited by the video format and there is no need for the low-pass filter. If a
nonstandard bandwidth is used such as that required by a high resolution CCD
camera, these monitors may exhibit some aliasing. This becomes noticeable as
the array size increases. For example, a high quality monitor can display 1000
x 1000 pixels. If the array consists of 4000 x 4000 elements, the display may
alias the image as depicted in Figure 8-1 (page 231). There is often a loss of
resolution when changing to a lower resolution format.
 SAMPLING THEORY 251

8.5. REFERENCES
1. S. K. Park and R. A. Schowengerdt, "Image Sampling, Reconstruction and the Effect of
Sample-scene Phasing," Applied Optics, Vol. 21(17), pp. 3142-3151 (1982).
2. S. E. Reichenbach, S. K. Park, and R. Narayanswamy, "Characterizing Digital Image
Acquisition Devices," Optical Engineering, Vol. 30(2), pp. 170-177 (1991).
3. W. Wittenstein, J. C. Fontanella, A. R. Newberry, and J. Baars, "The Definition of the OTF
and the Measurement of Aliasing for Sampled Imaging Systems," Optica Acta, Vol. 29(1), pp.
41-50 (1982).
4. J. C. Felz, "Development of the Modulation Transfer Function and Contrast Transfer Function
for Discrete Systems, Particularly Charge Coupled Devices," Optical Engineering, Vol. 29(8),
pp. 893-904 (1990).
5. S. K. Park, R. A. Schowengerdt, and M. Kaczynski, "Modulation Transfer Function Analysis
for Sampled Image Systems," Applied Optics, Vol. 23(15), pp. 2572-2582 (1984).
6. L. deLuca and G. Cardone, "Modulation Transfer Function Cascade Model for a Sampled IR
Imaging System," Applied Optics, Vol. 30(13), pp. 1659-1664 (1991).
7. G. C. Holst, Sampling, Aliasing, and Data Fidelity, JCD Publishing, Winter Park, FL (1998).
8. C. E. Shannon, "Communication in the Presence of Noise," Proceedings of the IRE, Vol. 37,
pp. 10-21 (January 1949).
9. A. B. Jerri, "The Shannon Sampling Theorem - Its Various Extensions and Applications: A
Review," Proceedings of the IEEE, Vol. 85(11), pp. 1565-1595 (1977).
10. G. C. Holst, Sampling, Aliasing, and Data Fidelity, pp. 77-78, JCD Publishing, Winter Park,
FL (1998).
11. G. C. Holst, Sampling, Aliasing, and Data Fidelity, pp. 293-320, JCD Publishing, Winter
Park, FL (1998).
12. J. E. Greivenkamp, "Color Dependent Optical Prefilter for the Suppression of Aliasing
Artifacts," Applied Optics, Vol. 29(5), pp 676-684 (1990).
13. T. S. Lomheim, L. W. Schumann, R. M. Shima, J. S. Thompson, and W. F. Woodward,
"Electro-Optical Hardware Considerations in Measuring the Imaging Capability of Scanned
Time-delay-and-integrate Charge-coupled Imagers," Optical Engineering, Vol. 29(8), pp. 911-
927 (1990).
14. C. M. Webb, "MRTD, How Far Can We Stretch It?" in Infrared Imaging Systems: Design,
Analysis, Modeling, and Testing V, G. C. Holst, ed., SPIE Proceedings Vol. 1689, pp. 297-307
(1994).
15. G. C. Holst, Sampling, Aliasing, and Data Fidelity, pp. 154-156, JCD Publishing, Winter
Park, FL (1998).
9
LINEAR SYSTEM THEORY
Linear system theory was developed for electronic circuitry and has been
extended to optical, electro-optical, and mechanical systems. It forms an
indispensable part of system analysis. For modeling purposes, electronic imaging
systems are characterized as linear spatial-temporal systems that are shift-
invariant with respect to both time and two spatial dimensions. Although space
is three-dimensional, an imaging system displays only two dimensions.

Electrical filters are different from spatial filters in two ways. They are
single-sided in time and must satisfy the causality requirement that no change in
the output may occur before the application of an input. Optical filters are
double-sided in space. Electrical signals may be either positive or negative,
whereas optical intensities are always positive. As a result, optical designers and
circuit designers often use different terminology.

With a linear-shift-invariant (LSI) system, signal processing is linear. A shift

of the input causes a corresponding shift in the output. For electronic circuits,
the shift is in time, whereas for optical systems, the shift is in space. In
additional, the input-to-outputmapping is single-valued. For optical systems, one
addition condition exists: the radiation must be incoherent if the system is to be
described in terms of radiance.

A sampled-data system may be considered "globally" shift invariant on a

macro-scale. For example, with an electronic imaging system, as a target moves
from the top of the field-of-view to the bottom, the image also moves from the
top to the bottom. On a micro-scale, moving a point source across a single
detector does not change the detector output. That is, the system is not shift-
invariant on a micro-scale. Similarly, with electronic circuits, large time shifts
of the input produce corresponding large time shifts in output. A shift during a
sample time, does not change the output. Here also, the system is not shift-
invariant on a micro-scale.

Single-valued mapping only occurs with non-noisy and non-quantized

systems. No system is truly noiseless but can be approximated as one when the
signal has sufficient amplitude. With digital systems, the output is quantized.
The smallest output is the least significant bit and the analog-to-digital converter
generally limits the largest signal. If the signal level is large compared to the

252
 LINEAR SYSTEM THEORY 253

least significant bit, then the system can be treated as quasi-linear over a
restricted region.

In spite of the disclaimers mentioned, systems are treated as quasi-linear

and quasi-shift-invariant over a restricted operating region to take advantage of
the wealth of mathematical tools available. An LSI system merely modifies the
amplitude and phase of the input. These are specified by the modulation transfer
function and phase transfer function.

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

9.1. LINEAR SYSTEM THEORY

A linear system provides single mapping from an input to an output.

Superposition allows the addition of individual inputs to create an unique output.
Let h{ } be a linear operator that maps one function, f(t), into another function,
g(t):

h {f(t)} - g(t) . (9-1)

Let the response to two inputs, f 1(t) and fz(t), be g 1(t) and gz(t):
(9-2)

For a linear system, the response to a sum of inputs is equal to the sum of
responses to each input acting separately. For any arbitrary scale factors, the
superposition principle states
(9-3)

When passing the signal through another linear system, the new operator
provides:
(9-4)
254 CCD ARRAYS, CAMERAS, and DISPLAYS

9.1.1. TIME-VARYING SIGNALS

An electrical signal can be thought of as the sum of an infinite array of

impulses (Dirac delta functions) located inside the signal boundaries. Thus, the
signal can be decomposed into a series of weighted Dirac delta functions (Figure
9-1):

L VIN
1 1
(t )<'i(t - t ) 11t
1
'
(9-5)
t'--~

where o(t - t') is the Dirac delta or impulse function at t = t' and viN(t') is the
signal evaluated at t'.

object

Dirac delta

Figure 9-1. A time-varying signal can be decomposed into a series of

closely spaced impulses with amplitude equal to the signal at that value.
They are shown widely separated for clarity and should not be confused
with sampling.

If v1N(t) is applied to a linear circuit, using h{ } will provide the output

vou·J.t) = L 1
h{viN(t )<'i(t-t )11t
1 1
} • (9-6)
t 1--co

Comparing Equation 9-5 to Equation 9-3, for each t', v1N(t') replaced ai and o(t
- t') replaced fi(t). That is, the input has been separated into a series of
functions a 1f 1(t) + a2fz(t) + ....
 LINEAR SYSTEM THEORY 255

As .dt'--+ 0, this becomes the convolution integral

(9-7)

and is symbolically represented by

v 0 crJ.t) - vllj..t) * h(t) , (9-8)

where * indicates the convolution operator (other texts may use *or ®). The
system impulse response is h{o(t)}. In Figure 9-2, each impulse response is
given for each t'. The individual impulse responses are added to produce the
output. This addition is the definition of superposition.

Figure 9-2. The linear operator, h{ }, transforms each input Dirac delta
into an output impulse response. The sum of the impulse responses
creates the output. Although shown separated for clarity, the impulse
responses have infinitesimal separations.

The convolution theorem of Fourier analysis states that the Fourier transform
of the convolution of two functions equals the product of the transform of the
two functions. That is, multiple convolutions are equivalent to multiplication in
the frequency domain. Using frequency domain analysis is convenient because
multiplications are easier to perform and easier to visualize than convolutions.
Then
(9-9)
256 CCD ARRAYS, CAMERAS, and DISPLAYS

The variable f denotes electrical frequency (Hz). Upper case variables represent
the Fourier transform of the signal (lower case variables). Let the input be an
impulse. The Fourier transform of an impulse response is
(9-10)

That is, all frequencies are present. Because the input to the system contains all
frequencies, the output must be a function of the system only. Any variation in
phase or amplitude must be due to the system itself. The output is

V0 uiJ) = H(j) . (9-11)

9.1.2. SPATIALLY VARYING SIGNALS

The one-dimensional electronic signal decomposition can be easily extended

to two-dimensional imagery. An object can be thought of as the sum of an
infinite array of impulses located inside the target boundaries. Thus, an object
can be decomposed into a two-dimensional array of weighted Dirac delta
functions, o(x - x'), o(y - y'):

o (x,y) = L L o(x 1,y 1)&(x- x') ~(y - y 1) .1x 1 .1y 1 • (9-12)

x1
--0Q y 1--oo

An optical system with operator h{ } produces an image

i(x,y) - L L h{o(x 1 ,y 1)~(x- x')~(y- y 1) .1x 1 .1y 1 } • (9-13)

x 1--oo y'--oo

For small increments, this becomes the convolution integral

i(x,y) = JJo(x ,y 1 1
) h{ &(x- x 1) o(y- y 1)} dx 1 dy 1 • (9-14)
 LINEAR SYSTEM THEORY 257

It is symbolically represented by

i(x,y) - o(x,y) * * h(x,y) , (9-15)

where * * represents the two-dimensional convolution. The function h(x,y) is

the optical system's response to an input impulse. The resulting image is the
point spread function (PSF). For an ideal circular aperture, the central portion
of the PSF is the Airy disk from which the Rayleigh criterion and other
resolution metrics are derived. In one dimension h(x) is the line spread function,
LSF. The LSF is the resultant image produced by the imaging system when
viewing an ideal line. There is no equivalent interpretation for h(t) for electrical
circuits. The function h(t) is simply the impulse response.

The transform of the convolution becomes a multiplication in the frequency

domain

I(u,v) - O(u,v) H(u,v) . (9-16)

If all frequencies are present in the image (i.e., an impulse), O(u,v)= 1, then
l(u,v) = H(u,v). The relationship between spatial frequency (u,v) and electrical
frequency (f) is further discussed in Section 10.1., Frequency Domains.

9.2. ELECTRONIC IMAGING SYSTEM

The electronic imaging system response consists of both the optical response
and the electronic response: h(x,y,t). Time and spatial coordinates are treated
separately. For example, optical elements do not generally change with time and
therefore are characterized only by spatial coordinates. Similarly, electronic
circuitry exhibits only temporal responses. The detector provides the interface
between the spatial and temporal components, and its response depends on both
temporal and spatial quantities. The conversion of two-dimensional optical
information to a one-dimensional electrical response assumes a linear
photodetection process. Implicit in the detector response is the conversion from
input photon flux to output voltage.

The optical transfer function (OTF) plays a key role in the theoretical
evaluation and optimization of an optical system. The modulation transfer
function (MTF) is the magnitude and the phase transfer function (PTF) is the
phase of the complex-valued OTF. In many applications, the OTF is real-valued
and positive so that the OTF and MTF are equal. When focus errors or
aberrations are present, the OTF may become negative or even complex valued.
258 CCD ARRAYS, CAMERAS, and DISPLAYS

Electronic circuitry also can be described by an MTF and PTF. The

combination of the optical MTF and the electronic MTF creates the electronic
imaging system MTF. The MTF is the primary parameter used for system
design, analysis, and specifications. When coupled with the three-dimensional
noise parameters, the MTF and PTF uniquely define system performance
(discussed in Chapter 12, Minimum Resolvable Contrast).

Symbolically

OTF(u,v) = HSPATIAL(u,v) = MTF(u,v) ejPTF(u,v) (9-17)

and

HELECTRONicsif) - MTFI])ejPTF(f) · (9-18)

With appropriate scaling, the electronic frequencies can be converted into spatial
frequencies. This is symbolically represented by f-+u. The electronic circuitry
is assumed to modify the horizontal signal only (although this depends on the
system design). The combination of spatial <md electronic responses is some-
times called the system OTF:

OTFSYSTE~u,v) = MTF(u,v) MTF(j-u)ej[PTF(u,v) +PTF(/-u)] . (9-19)

For mathematical convenience, the horizontal and vertical responses are

considered separable (usually coincident with the detector array axes). The
electrical response is considered to affect only the horizontal response
(9-20)

and
(9-21)

A system is composed of many components that respond to either spatial or

temporal signals. For independent MTFs,
N M
MTFsrswJ-u,v) - II II MTF;(u,v)MTFif-u) . (9-22)
i- 1 j- 1
 LINEAR SYSTEM THEORY 259

While individual lens elements each have their own MTF, the MTF of the lens
system is not usually the product of the individual MTFs. This occurs because
one lens may minimize the aberrations created by another.

9.3. MTF and PTF INTERPRETATION

The system MTF and PTF alter the image as it passes through the circuitry.
For linear-shift-invariant systems, the PTF simply indicates a spatial or temporal
shift with respect to an arbitrarily selected origin. An image where the MTF is
drastically altered is still recognizable, whereas large PTF nonlinearities can
destroy recognizability. Modest PTF nonlinearity may not be noticed visually
except those applications where target geometric properties must be preserved
(i.e., mapping or photograrnmetry). Generally, PTF nonlinearity increases as the
spatial frequency increases. Because the MTF is small at high frequencies, the
nonlinear-phase-shift effect is diminished.

The MTF is the ratio of output modulation to input modulation normalized

to unity at zero frequency. While the modulation changes with system gain, the
MTF does not. The input can be as small as desired (assuming a noiseless
system with high gain) or it can be as large as desired because the system is
assumed not to saturate.

Modulation is the variation of a sinusoidal signal about its average value

(Figure 9-3). It can be considered as the AC amplitude divided by the DC level.
The modulation is:

VMAX - VMIN AC
MODULATION - M - - (9-23)
VMAX + VMIN DC '

where VMAX and VMIN are the maximum and rrummum signal levels,
respectively. The modulation transfer function is the output modulation produced
by the system divided by the input modulation as a function of frequency:

(9-24)

The concept is presented in Figure 9-4. Three input and output signals are
plotted in Figures 9-4a and 9-4b, respectively, and the resultant MTF is shown
in Figure 9-4c. As a ratio, the MTF is a relative measure with values ranging
from zero to one.
260 CCD ARRAYS, CAMERAS, and DISPLAYS

The MTF is a measure of how well a system will faithfully reproduce the
input. The highest frequency that can be faithfully reproduced is the system
cutoff frequency. If the input frequency is above the cutoff, the output will be
proportional to the signal's average value with no modulation.

Figure 9-3. Definition of target modulation. The value do is the extent

of one cycle.

INPUT (IMAGE) OUTPUT (IMAGE)

(a) (b)

iu. 0.5

f,
SPATIAL FREQUENCY
(c)

Figure 9-4. Modulation transfer function. (a) Input signal for three
different spatial frequencies, (b) output for the three frequencies, and (c)
the MTF is the ratio of output-to-input modulation.
 LINEAR SYSTEM THEORY 261

9.4. SUPERPOSITION APPLIED to OPTICAL SYSTEMS

If the system MTF is known, the image for any arbitrary object can be
computed. First, the object is dissected into its constituent spatial frequencies
(i.e., the Fourier transform of the object is obtained). Next, each of these
frequencies is multiplied by the system MTF at that frequency. Then, the
inverse Fourier transform provides the image.

To illustrate the superposition principle and MTF approach, we will show

how an ideal optical system modifies an image. An ideal optical system is, by
definition, a linear-phase-shift system. The most popular test target consists of
a series of bars: typically three or four bars although more may be used. For
illustrative purposes, the periodic bars are assumed of infinite extent. A one-
dimensional square wave, when expanded into a Fourier series about the origin,
contains only odd harmonics:

o(x) ~ -1 + -2 ~ 1 sm
L.-- . (- nx)
21t - n ~ 1, 3, 5, ···, (9-25)
2 1t n do

where do is the period of the square wave. The fundamental frequency U 0 is lld0 •
Note that the peak-to-peak amplitude of the fundamental is 417r times the square
wave amplitude. In the frequency domain, the square wave provides discrete
spatial frequencies of Un = 1/d0 , 3/d0 , 5/d0 , ••• , with amplitudes 217r, 2/37r,
2/57r, ··· , respectively.

Let a circular optical system image the square wave. The MTF for a
circular, clear aperture, diffraction-limited lens is

- 0 elsewhere .

The optical cutoff in image space is uic = D0 /(A. fl), where Do is the aperture
diameter, 'A is the average wavelength, and fl is the focal length.
262 CCD ARRAYS, CAMERAS, and DISPLAYS

By superposition, the diffraction-limited optical MTF and square wave

Fourier series amplitudes are multiplied together at each spatial frequency:
(9-27)

Taking the inverse Fourier transform provides the resultant image. Equivalently,

i(x) - ~ + MTFomcs<uo>[ ~ sin(21t U X)] 0

(9-28)

+ MTF0PTICS(3u
0
)[~
31t
sin(61t u 0
x)] + ....

If llo is greater than U;c/3, only the fundamental of the square wave will be
faithfully reproduced by the optical system. Here, the square wave will appear
as a sine wave. Note that the optical MTF will reduce the image amplitude. As
U0 decreases, the image will look more like a square wave (Figure 9-5). Because
the optics does not pass any frequencies above U;c. the resultant wave form is
a truncation of the original series modified by the optical MTF. This results in
some slight ringing. This ringing is a residual effect of the Gibbs phenomenon.
Note that with imaging systems, the intensity is always positive. As the input
frequency approaches U;c (Figure 9-5d), the modulation is barely present in the
image. However, the average scene intensity approaches 0.5. That is, the scene
detail can no longer be perceived at or above u,c· The scene did not disappear.

r l r\
~ ~
(i) (i)
z z
w
1-
~
J
w
1-
~ J
(a) (b)

~ ll
~
(i)
z
w
1-
~

(c) (d)
Figure 9-5. A square wave modified by a circular aperture as a function
of x/d0 • As the square-wave fundamental frequency increases, the edges
become rounded. It will appear as a sinusoid when U0 ~ U;c/3. (a) U0 =
U;c/40, (b) U0 = U;c/10, (c) U0 = U;c/3, and (d) U0 = 9u;cl10.
 LINEAR SYSTEM THEORY 263

9.5. CONTRAST TRANSFER FUNCTION

The system response to a square-wave target is the contrast transfer function

(CTF) or square-wave response (SWR). The CTF is a convenient measure
because of the availability of square-wave targets. It is not a transfer function
in the same sense that the MTF is and, as such, subsystem CTFs cannot be
cascaded. Figure 9-6 illustrates the relationship between square-wave and
sinusoidal amplitudes. The CTF is typically higher than the MTF at all spatial
frequencies.

A square wave can be expressed either as an infinite sine (Equation 9-25) or

cosine series. The only difference is the arbitrarily selected origin. Coltman1
selected the cosine expansion. The output amplitude of the square wave, whose
fundamental frequency is f is an infinite sum of the input cosine amplitudes
0
,

modified by the system's MTF:

CTF(u) ~ ~ I MTF(u)
MTF(3u 0 ) + MTF;5u) _ .... I (9_29)
3

or

4~~ kMTF[(2k+1)u 0 ]1 (9-30)

CTF(u) - L(-1) .
1t k-0 2k+ 1

By adding and subtracting terms, the MTF at frequency U0 can be expressed as

an infinite sum of CTFs.

t
MTF CTF
T
.i. 1

Figure 9-6. AC components of the CTF and MTF. The CTF is usually
equal to or greater than the MTF.
264 CCD ARRAYS, CAMERAS, and DISPLAYS

Again, assuming a linear-shift-invariant system, the MTF is 1•2 :

CTF(5u)
5
(9-31)
CTF(7u 0 ) CTF(llu 0 )
+ ------"- + + irregular terms I
7 11

or

MTF(u) (9-32)

where k takes on odd values only: 1, 3, 5, .... and~ is -1 or 1 according to:

k-1
(9-33)
Bk = (-l)m(-1) 2 for r = m .

The value r is the number of different prime factors in k. The value m is the
total number of primes into which k can be factored. The value Bk = 0 for r <
m.

Theoretically, to obtain the MTF at frequency U 0 , an infinite number of

square-wave responses must be measured. However, the number required is
limited by the spatial cutoff frequency, uMTF=O• where the MTF approaches zero
and remains zero thereafter. For bar targets with spatial frequency above
113uMTF=O• the MTF is equal to 11'14 times the measured CTF. That is, above 1/a
uMTF=O• only one target is necessary to compute3 the MTF.

For low spatial frequencies, the output nearly replicates the input (Figure 9-
7a). For frequencies near 1/3 cutoff, the output starts to appear sinusoidal (Figure
9-7b). Above 1/3 cutoff (Figure 9-7c), the MTF amplitude is 11'14 times the CTF
amplitude.
 LINEAR SYSTEM THEORY 265

(a) (b) (c)

Figure 9-7. Input and output wave forms for a linear system. (a) Very
low frequency signals are faithfully reproduced, (b) mid-spatial
frequencies tend to look like sinusoids, and (c) input square waves with
spatial frequencies above 1/3 uMTF=o appear as sinusoidal outputs.

The relationship between the CTF and MTF was developed for analog
optical systems and is appropriate for wet-film photographic cameras and image
intensifiers. It can also be applied in the scan direction for analog scanning
devices (vidicons and displays).

For sampled data systems, where aliasing can occur, the relationship should
be used cautiously. CTF measurements are appropriate tests for system
performance verification. Conversion to the MTF that can have significant
mathematical complexity when aliasing is present.

Sampling is present in all solid state imaging systems due to the discrete
location of the detectors. Phasing effects between the detector and the location
of the target introduce problems at nearly all spatial frequencies. The interaction
between the bar frequency and the system sampling frequency produces sum and
difference frequencies that produce variations in amplitude and bar width (see
Section 8.3.3., Image Distortion, page 245).
266 CCD ARRAYS, CAMERAS, and DISPLAYS

The CTF is well behaved (in the sense that the bar width remains constant)
when the bar spatial frequency is proportional to the system Nyquist frequency,

u0 - (9-43)
k

where k is an integer. These spatial frequencies provide the in-phase CTF. The
CTF can be used to assess system perform<mce at the spatial frequencies
selected. It is usually specified only at the Nyquist frequency (k = 1).

9.6. REFERENCES
1. J. W. Coltman, "The Specification of Imaging Properties by Response to a Sine Wave Input,"
Journal of the Optical Society of America, Vol. 44(6), pp. 468-471 (1954).
2. I. Limansky, "A New Resolution Chart for Imaging Systems," The Electronic Engineer, Vol.
27(6), pp. 50-55 (1968).
3. G. C. Holst, Testing and Evaluation of Infrared Imaging Systems, pp. 238-246, JCD
Publishing, Winter Park, FL (1993).
10
SYSTEM MTF
The equations describing array (Chapter 4) and camera (Chapter 6)
performance assume that the image is very large and it illuminates many
detector elements. That is, the object contained only very low spatial frequencies
and that the system MTF is essentially unity over these spatial frequencies.

The optics and detector MTFs modify the signal spatial frequencies. These
frequencies may also be affected by electronic filter MTFs although these
electronic filter functions are typically designed to pass the entire image
frequency spectrum without attenuation.

But noise originates at the detector and within the electronic circuitry. It is
modified by the electronic MTFs only. Detector array noise is fixed but analog
amplifier noise increases as the electronic bandwidth increases. The camera SNR
is maximized when the electronic bandwidth passes the signal with minimal
degradation. Wider bandwidth electronics will increase the noise but not the
signal and thereby reduce the SNR.

For mathematical convenience, a specific target feature can be associated

with a fictitious spatial frequency (Figure 10-1). The SNR for this feature is

(10-1)

where (Ilsys) includes shot, reset, amplifier, quantization, and fixed pattern noise
sources (see Section 4.2., Noise, page 123). The total amplifier noise depends
upon the electronic bandwidth and that depends on the electronic frequency
response associated with the readout signal chain.

A solid state camera consists of optics, an array of pixels, and a signal chain
with electronic frequency response characteristics. The complete imaging system
includes the display:

MTFSYS - MTFOP17CSMTFDETECWRMTFSIGNAL-CHAINMTFDISPLAY 0
<10-2)

267
268 CCD ARRAYS, CAMERAS, and DISPLAYS

The observer processes this information to create a perceived MTF:

MTFPERCEIVED = MTFSY.I MTFHVS • (10-3)

MTFHvs is the human visual system MTF which includes both the eye optical
MTF and neural processing. When MTFPERcErvEo is coupled to system noise,
system responsivity, and the eye integration capabilities, it is possible to predict
the minimum resolvable contrast (described in Chapter 12, Minimum Resolvable
Contrast). MRC is used by the military to predict target detection and
recognition ranges for a given probability of success.
Target

D~allb ~---
1 I
I___ JI

1... dr ...1
1. . .~--
. 2 dr ------..1

Figure 10-1. Fictitious spatial frequency associated with a target feature.

If the image detail size is dr mm at the detector plane, then the fictitious
horizontal frequency is 11o = 1!(2dr) cycles/mm.

Because solid state arrays are undersampled, the highest frequency that can
be reproduced is the array Nyquist frequency, uiN· As a result, it is common
practice to evaluate Equation 10-1 at 11o = uiN· This dictates the smallest target
feature that can be faithfully reproduced. The value uiN should be added to all
MTF curves (see Section 8.3.2., Detector Array Nyquist Frequency, page 242).

The MTFs presented in this chapter apply to all staring arrays. Only the
optics and detector array are sensitive to the scene spectral and spatial content.
The remaining subsystem MTF components only modify the electrical signals.
These signal chain MTFs apply to all imaging systems independent of the system
spectral response.

Optical spatial frequency is two-dimensional with the frequency ranging from

- oo to + oo. Although not truly separable, for convenience, the horizontal
MTF, MTF(u), and vertical MTF, MTF(v), are analyzed separately. Electrical
filters are causal, one-dimensional, and are considered to operate in the direction
of the readout clocking only. Thus, the horizontal and vertical MTFs may be
different.
 SYSTEM MTF 269

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

10.1. FREQUENCY DOMAINS

There are four different locations where spatial frequency domain analysis
is appropriate. They are object space (before the camera optics), image space
(after the camera optics), display, and observer (at the eye) spatial frequencies.
Simple equations relate the spatial frequencies in all these domains.

Figure 10-2 illustrates the spatial frequency associated with a bar target. Bar
patterns are the most common test targets and are characterized by their
fundamental spatial frequency. Using the small angle approximation, the angle
subtended by one cycle (one bar and one space) is d,/R~> where d0 is the spatial
extent of one cycle and R 1 is the distance from the imaging system entrance
aperture to the target. When using a collimator to project the targets at
apparently long ranges, the collimator focal length replaces R 1• Targets placed
in the collimator's focal plane can be described in object space.

Image

Figure 10-2. Correspondence of spatial frequencies in object and image

space. Although the MTF is defined for sinusoidal signals, the bar target
is the most popular test target (shown here). Bar targets are
characterized by their fundamental frequency.
270 CCD ARRAYS, CAMERAS, and DISPLAYS

The horizontal object-space spatial frequency, U00 , is the inverse of the

horizontal target angular subtense and is usually expressed in cycles/mrad:

cycles (10-4)
mrad

A similar equation exists for the vertical object-space spatial frequency. The
object-space domain is used by the military for describing system performance.

Optical designers typically quote spatial frequencies in image space to

specify the resolving capability of lens systems. Photographic cameras and CCD
cameras are typically specified with spatial frequencies in image space which is
the object-space spatial frequency divided by the system focal length:

uob line-pairs cycles (10-5)

u.-- or
I fl mm mm

where ui is the inverse of one cycle in the focal plane of the lens system.
Although used interchangeably, line-pairs suggest square wave targets and cycles
suggest sinusoidal targets. To maintain dimensionality, if U00 is measured in
cycles/mrad then the focal length must be measured in meters to obtain
cycles/mm.

Analog electronic filters are one-dimensional and modify a serial data

stream. The electrical frequency is related to the horizontal field-of-view
(HFOV) and the time it takes to read out one line. For analog filters
immediately following the detector array, where the readout time is tH-LINE•

fe - HFOV ·fl u. Hz . (10-6)

tH-UNE I

For staring arrays with NH horizontal detectors,

(NH- l)dCCH + dH
fe - ui Hz. (10-7)
tH-UNE
 SYSTEM MTF 271

While digital filters operate on a data array, the effective sampling rate is related
to the timing of the readout clock:

1
f.s- - - Hz. (10-8)
e TCLOCK

After the digital-to-analog converter, the serial stream data rate and therefore
the filter design is linked to the video standard. The active line time, tvmEo-LINE•
(Table 10-1) and array size provide the link between image space and video
frequencies:

"v _
J,
HFOV·jl
U; Hz . (10-9)
tVIDEO-UNE

Video standards were discussed in Section 5.2.1., Video Timing (page 151). For
staring arrays,

(10-10)

Table 10-1
STANDARD VIDEO TIMING
TOTAL MINIMUM ACTIVE
FORMAT LINE TIME LINE TIME (JLS)
(JLS) (VIDEO-LINE

EIA 170 63.492 52.092

NTSC 63.555 52.456

PAL 64.0 51.7

SECAM 64.0 51.7

272 CCD ARRAYS, CAMERAS, and DISPLAYS

The field-of-view and monitor size link image spatial frequency to the
horizontal and vertical display spatial frequencies:
HFOV ·jl cycles
ud = u. (10-11)
WMoNnvR I mm

and

VFOV ·jl ~:ycles

vd = v. (10-12)
HMoNnvR I mm

Both the monitor width, WMoNITOR• and height, HMoNITOR• are usually measured
in millimeters. When the monitor aspect ratio is the same as the FOV ratio (the
usual case),

HFOV VFOV
(10-13)

The spatial frequency presented to the observer depends on the observer's

viewing distance and the image size on the display (Figure 10-3). Observer
response is assumed to be rotationally symmetric so that the horizontal and
vertical responses are the same. The usual units 1 are cycles/deg:

1 cycles
ueye
deg (10-14)
2 tan-1 (_!_ _1 )
2 Dud

Here, the arc tangent is expressed in degrees. Because image detail is important,
the small angle approximation provides

!.·ycles (10-15)
ueye = 0.01745Dud
deg

Table 10-2 summarizes the various spatial frequencies.

 SYSTEM MTF 273

Wmonitor

H
Monitor

Figure 10-3. Correspondence of spatial frequencies for the display and

for the eye.

Table 10-2
TRANSFORM PAIR UNITS
SUBSYSTEM
SIGNAL VARIABLES FREQUENCY VARIABLES
RESPONSE

x denotes distance (nun) u; expressed as cycles/nun

Optics, detector
y denotes distance (nun) v; expressed as cycles/nun

II, denotes angle (mrad) U0• expressed as cycles/mrad

Optics, detector
11, denotes angle (mrad) vob expressed as cycles/mrad

Analog electronics t denotes time (sec) f, expressed as cycles/s (Hz)

Video electronics t denotes time (sec) fv expressed as cycles/s (Hz)

x denotes distance (nun) ~ expressed as cycles/mm

Display
y denotes distance (nun) vd expressed as cycles/nun

II, denotes angle (deg) u,,, expressed as cycles/deg

Observer
11, denotes angle (deg) v,,, expressed as cycles/deg
274 CCD ARRAYS, CAMERAS, and DISPLAYS

10.2. OPTICS MTF

Optical systems consist of several lenses or mirror elements with varying

focal lengths and varying indices of refraction. Multiple elements are used to
minimize lens aberrations. While individual element MTFs are used for design
and fabrication, these individual MTFs cannot be cascaded to obtain the optical
system MTF. For modeling purposes, the optical system is treated as a single
lens that has the same effective focal length and aberrations as the lens system.

The optical anti-aliasing filter, which is used to blur the image on a single
chip color filter array, is also part of the lens system. Its performance is best
described by the way it affects the detector MTF. Therefore, its description is
included in the detector section.

Optical spatial frequency is two-dimensional with the frequency ranging from

- oo to + oo. The diffraction-limited MTF for a circular aperture was first
introduced in Section 9.4., Superposition Applied to Optical Systems (page 261).
For an aberration free and radially symmetric optical system, MTFoPTics is the
same in the horizontal and vertical directions. In the horizontal direction,

I
I U. u.
MTForncs<u;) - - cos- (-' ) - - '
n; ";c ";c f ·-1u. 2
1 ·( - ' )
U;c

- 0 elsewhere .

The image-space optics cutoff is uic = D0 /(}, fl) = 1/(FA.), where D0 is the
aperture diameter and fl is the effective focal length. The f-number, F, is equal
to fl/D 0 .

Figure 10-4 illustrates MTF0 .,.ncs as a function of u/uic· Because the cutoff
frequency is wavelength dependent, Equation 10-16 and Figure 10-4 are only
valid for noncoherent monochromatic light. The extension to polychromatic light
is lens specific. Most lens systems are color corrected (achromatized) and
therefore there is no simple way to apply this simple formula to predict the
MTF. As an approximation to the polychromatic MTF, the average wavelength
is used to calculate the cutoff frequency:

(10-17)
 SYSTEM MTF 275

For example, if the system spectral response ranges from 0.4 to 0.7 p.m, then
A.AvE = 0.55JLm.
For most systems operating in the visible spectral region, the optical system
may be considered near diffraction limited and in focus. Approximations for
aberrated and defocused optical systems are in Reference 2. The MTFs for
optical systems with rectangular apertures3 or telescopes with a central
obscuration (Cassegrainian optics) are also available. 4

0.8

u. 0.6
1-
::::E 0.4

0.2

0.25 0.5 0.75

NORMALIZED SPATIAL FREQUENCY

Figure 10-4. MTF 0 PT1cs for a circular aperture normalized to u/u;c·

10.3. DETECTORS

The two-dimensional spatial response of a rectangular detector is

(10-18)

where sinc(x) = sin(7rx)/(7rX) and d" and dv are the photosensitive detector sizes
in the horizontal and vertical directions, respectively. Note that arrays are often
specified by the pixel size. With 100% fill factor arrays, the pixel size is equal
to the detector size. With finite fill factor arrays, the photosensitive detector
dimensions are less than the pixel dimensions. A microlens or an anti-alias filter
may optically modify the effective size (discussed in Section 10.3.4., Optical
Anti-Alias Filter).

Although the detector MTF is valid for all spatial frequencies from - oo to
+ oo, it is typically plotted up to the first zero (called the detector cutoff) which
occurs at U;o = 1/d" (Figure 10-5). Because d" may be different than dv, the
horizontal and vertical MTF can be different.
276 CCD ARRAYS, CAMERAS, and DISPLAYS

A phase shift occurs when ui > 1/dH and v, > lldv. Figure 10-5 illustrates
the MTF without regard to the array Nyquist frequency. The analyst must add
the array Nyquist frequency to the curve (see Section 8.3.2, Detector Array
Nyquist Frequency, page 242). The MTF of virtual phase detectors is a complex
expression. 5

0.8

LL 0.6
1-
:::E 0.4
0.2

0~-------+--------+--------+------~
0 0.25 0.5 0.75
NORMALIZED SPATIAL FREQUENCY

Figure 10-5. Horizontal detector MTF as a function of normalized

spatial frequency dHui. The MTF is usually plotted up to the first zero
(dHui = 1). For staring arrays, the array ~yquist frequency must be
added: uiN = 11(2dccH).

10.3.1. DIFFUSION MTF

As the wavelength increases, photon absorption occurs at increasing depths
in the detector material (see Section 4.1.1., Spectral Response, page 104). A
photoelectron generated deep within the substrate will experience a three-
dimensional random walk until it recombines or reaches the edge of a depletion
region where the pixel electric field exists. A photoelectron generated under one
well may eventually land in an adjoining well. This blurs the image and its
effect is described by an MTF. Beyond 0.8 ~-tm, diffusion (random walk) affects
the detector MTF and the response is expressed by MTFoETEcToRMTFmFFusJoN·
Imagery produced by near-IR sources will appear slightly blurry compared to
imagery produced by a visible source.
 SYSTEM MTF 277

The diffusion MTF 6 for a front-illuminated device is

MTFD/FFUS/0/t U;) (10-19)

1 - exp( -a. ABSLD)
1 + a. ABSLDIFF

where aAss is the spectral absorption coefficient. Figure 4-2 (page 105)
illustrated the absorption coefficient in bulk p-type silicon. The factor L 0 is the
depletion width and L(u) is the frequency-dependent component of the diffusion
length (LmFF) given by

(10-20)

The diffusion length typically ranges between 50 ~-tm and 200 J.tm. The depletion
width is approximately equal to the gate width. For short wavelengths (A < 0.6
~-tm), aAss is large and lateral diffusion is negligible. Here, MTFmFFusroN
approaches unity. For the near IR (A > 0.8 ~-tm), the diffusion MTF may
dominate the detector MTF (Figure 10-6). By symmetry, MTFmFFusroN(u) =
MTFmFFusroN(v;) and L(u) = L(v;).

1
u. 0.6 urn
1- 0.8
::E
z 0.6
0.8 urn
0
u;
:::> 0.4 1.06 urn
u.
u. 0.2
c
0
0 10 20 30 40 50
SPATIAL FREQUENCY (cy/mm)
Figure 10-6. MTFmFFusroN as a function of wavelength. The value L0 =
10 ~-tm and LmFF = 100 J.tm. As the diffusion length increases, the MTF
decreases. For comparison, a 20-~-tm detector will have a cutoff at 50
cycles/mm.
278 CCD ARRAYS, CAMERAS, and DISPLAYS

To account for the effects of lateral diffusion, Schumann and Lomheim7

introduced a trapezoidal response approximation. This MTF only applies in the
direction perpendicular to charge transport:

MTFDIFFUSIOJ_u) ,.. sinc(dCCHUi)sinc[(dccH- ~)U;] , (10-21)

where (3 is the region over which the detector responsivity is flat (Figure 10-7).
At the boundary between pixels, the trapezoidal response is 0.5 indicating equal
probability of an electron going to either pixel.

I I I I
I....,....__D_e_te-c-to-r-1••1 ,. Detector • I,. Detector • I
I I I I
I I I I
I I I I I I
1,.. Center~l
!Center Spacing ! ~-_D~~~~~~n

l )
Silicon Dioxide
I L 5 <::
I
p:= --
1 I

100%
t
Relative
Response
0

1,..
dec
Figure 10-7. Trapezoidal approximation for long -wavelength photon
absorption. Most of the charge goes to the charge well nearest to the
photon absorption site.
 SYSTEM MTF 279

10.3.2. CHARGE TRANSFER EFFICIENCY

CCD performance is characterized by the transfer efficiency and number of

transfers (see Section 3.4., Charge Transfer Efficiency, page 74). Assuming
vertical transfer, an MTF 8 that accounts for the incomplete transfer of electrons
is

(10-22)

where NTRANs is the total number of charge transfers from a detector to the
output amplifier and E is the charge transfer efficiency for each transfer. It is the
number of detectors multiplied by the number of CCD phases. When NTRANs(E-
1) is small, then MTF(U;N) = 1 - 2NTRANs(l -E).

This MTF depends upon the number of charge transfers. If the image is
located next to the readout, the number of transfers is small. But if the image
is at the extreme end of the array, the charge must be transferred across the
entire array and through the serial readout register. For an average response,
NTRANs should be one-half the maximum number of transfers. This is
mathematically equivalent to (MTFcTE)'h.

Transfers may be either in the vertical direction (column readout) or in the

horizontal direction (row readout). Horizontal readout affects MTF(U;) and
vertical readout affects MTF(v). MTFcTE only applies to the transfer readout
direction. Figure 10-8 illustrates MTFcTE at Nyquist frequency for several values
of transfer efficiency. For consumer devices, CTE > 0.9999 and NTRANs >
1500. For scientific applications, CTE > 0.999999 and NTRANs can exceed 5000.

~=============================================
Example 10-1
Number of Transfers

What is the maximum number of transfers that exist in an array that contains
1000 x 1000 detector elements?

For a charge packet that is furthest from the sense node, the charge must be
transferred down one column (1000 pixels) and across 1000 pixels for a total of
2000 pixels. The number of transfers is 4000, 6000, and 8000 for two-phase,
three-phase, and four-phase devices, respectively.
=================================================~
280 CCD ARRAYS, CAMERAS, and DISPLAYS

0.999999

u. 0.6
1-
:::1! 0.4

0.2

0~--~~-----+----~~----+-----~
0 2000 4000 6000 8000 10000
NUMBER OF TRANSFERS
Figure 10-8. MTFcTE at Nyquist frequency for several values of transfer
efficiency. As the number of transfers increases, the efficiency must also
increase to ensure that the MTF is not adversely affected.

10.3.3. TDI

TDI operation was described in Section 3.3.6., Time Delay and Integration,
page 70. It is essential that the clock rate match the image velocity so that the
photogenerated charge packet is always in sync with the image. Any mismatch
will blur the image. Image rotation with respect to the array axes affects both
the horizontal (TDI direction) and vertical (readout direction) MTFs
simultaneously. For simplicity they are treated as approximately separable -
MTF(ui,vi) :::::: MTF(u)MTF(vi).

If the TDI direction is horizontal, velocity errors degrade'~ the detector MTF
by

where NTm is the number of TDI elements. The value dERROR is the difference
between the expected image location and the actual location at the first pixel
(Figure 10-9). After Nm1 elements, the image is displaced NmrdERROR from the
desired location. MTFm1 is simply the MTF of an averaging filter for Nm1
samples displaced dERROR one relative to the next. As an averaging filter, the
effective sampling rate is 1/dERROR·
 SYSTEM MTF 281

.IPGJGJD
I
----..1 I
I
I
I
I

•
1...... I I I
dERROR I I
_.I ~ I I
I
2dERROR • I
----..1I I
1.._. I
3dERROR •
I
1... ...
4dERROR

Figure 10-9. Definition of dERROR· With a velocity mismatch, the target

falls further and further away from the desired location (on the center of
the detector). After four TDI elements, the target is displaced from the
detector center by 4dERROR.

This is a one-dimensional MTF that apples in the TDI direction only. In

image space
(10-24)

where .:lV is the relative velocity error between the image motion and the charge
packet "motion" or "velocity." The charge packet "velocity" is dccH/t1NT where
t1NT is the integration time for each pixel.

Figure 10-10 portrays the MTF for two different relative velocity errors for
32 and 96 TDI stages. As the number of TDI stages increases, dERRoR must
decrease to maintain the MTF. Equivalently, the accuracy of knowing the target
velocity must increase as Nmr increases. Because the image is constantly
moving, a slight image smear occurs 10 because the photogenerated charge is
created at discrete times (e.g., at intervals of lcwcK). This linear motion MTF
(discussed in Section 10.5.1., Linear Motion) is often small and therefore is
usually neglected.
282 CCD ARRAYS, CAMERAS, and DISPLAYS

0.005
.... "' .. "' . . . "' . -. "' . "' ...... .. ..
. .. . "' .......... 0.01
0.9 ...
0.005
u..
1- 0.8
::::!
0.7
0.01 ·. "-,
0.6 - 1 - - - - + - - - - - - - - - - + - - - - - 1
0 0.1 0.2 0.3 0.4 0.5
NORMALIZED SPATIAL FREQUENCY

Figure 10-10. MTF degradation due to mismatch between the image

velocity and the charge packet "velocity." The value dERRoR/dccH equals
0.005 and 0.01. The spatial frequency axis is normalized to dccHui and
the MTF scale is expanded from 0. 6 to 1. 0. The solid lines represent
Nm1 = 32 and the dashed lines are for Nm1 = 96.

TDI is also sensitive to image rotation. If the image motion subtends an

angle() with respect to the TDI rows (TDI direction), the image moves vertically
Nm1dccHtan(8) across the array. The MTF degradation in the vertical direction
is

(10-25)

where MTFm1(vi) is simply the MTF of an averaging filter for Nm1 samples
displaced dccHtan(8) one relative to the next. Figure 10-11 illustrates the vertical
MTF degradation due to image rotation for 32 and 96 stages. The spatial
frequency is normalized to dccHVi where dccH is the horizontal detector pitch.

TDI offers the advantage that the signal-to-noise ratio increases by the
square root of the number of TDI stages. This suggests that many TDI detectors
should be used. The disadvantage is that the scan velocity must be tightly
controlled as the number of TDI stages increases.
 SYSTEM MTF 283

0.5
··---- ... :·-----
... ...... ___ _.... __ . ..
0.8 ... . . .
0.5
LL 0.6
1-
::iil 0.4
0.2

0+-----~------~------~----~------~
0 0.1 0.2 0.3 0.4 0.5
NORMALIZED SPATIAL FREQUENCY

Figure 10-11. Vertical MTF degradation in the readout direction due to

image rotation of 0.5 and 1 deg with respect to the array axis. The
spatial frequency axis is normalized to dccHvi. The solid lines represent
Nm1 = 32 and the dashed lines are for Nm1 = 96.

10.3.4. OPTICAL ANTI-ALIAS FILTER

In color filter arrays, like-colored sensitive detector center-to-center spacings

may be much larger than the detector pitch. This potentially increases aliasing
(see Section 8.3.2., Detector Array Nyquist Frequency, page 242). The optical
anti-alias filter blurs the image and effectively increases the optical area11 of the
detector. The filter, which consists of one or more birefringent crystals, is
placed between the lens assembly and the detector array. Birefringent crystals
break a beam into two components: the ordinary and the extraordinary. As
illustrated in Figure 10-12, the detector collects light from a larger beam. Rays
that would have missed the detector are refracted onto the detector. This makes
the detector appear optically larger. While this changes the system MTF, the
system sensitivity does not change. The light that is lost to adjoining areas is
replaced by light refracted onto the detector.

The filter design depends upon the CFA design. If detectors sensitive to like
colors are next to each other, the filter may not be required. If the detectors
alternate, then an appropriately designed filter will make the detectors appear
twice as large. Similarly, if the detectors are spaced three apart, then the filter
will make the detector appear three times greater. This larger value (the
effective detector size) is used in Equation 10-18 (page 275).
284 CCD ARRAYS, CAMERAS, and DISPLAYS

Detector Effective
Width Width
1.. ...1 1.. -----..1

Optical
Anti-Alias
Filter
I

6
{a) (b)

Figure 10-12. The birefringent filter increases the effective optical area
of the detector. (a) No filter and (b) filter. Rays that would fall between
detectors are refracted onto the detector.

Multiple crystals are used to change the effective detector size both vertically
and horizontally. Because the spacing depends upon the CF A design, there is no
unique anti-alias filter design. Clearly, if the detectors are unequally spaced, the
Nyquist frequency for each color is different. Similarly, with different effective
sizes, the MTFs are also different for each color. Complex algorithms are
required to avoid color aliasing (see Example 5-1, page 167) and to equalize the
MTFs. The anti-alias filter is essential for CFAs to reduce color moire patterns
and can be used in any camera to reduce aliasing.

Figure 10-13 illustrates how the detector MTF is modified by the

birefringent crystal when the detector pitch is twice as large as the element size.
While the filter reduces aliasing, it also reduces the overall system MTF. For
scientific applications, this reduction may be too severe.
 SYSTEM MTF 285

1
0.8

...:IE
LL 0.6
0.4
0.2
0
0 0.25 0.5 0.75
NORMALIZED SPATIAL FREQUENCY
(a)

Nyquist frequency
1
0.8

...:IE
LL 0.6
0.4
0.2
0
0 0.25 0.5 0.75
NORMALIZED SPATIAL FREQUENCY
(b)

1 Nyquist frequency

0.8

...
LL

:iil! 0.4
0.6

0.2

0.25 0.5 0.75

NORMALIZED SPATIAL FREQUENCY
(c)

Figure 10-13. MTFoETECToR modifications due to the anti-alias filter as

a function of dHu;. The detector pitch is twice the detector element size.
(a) No filter, (b) a filter that doubles the detector optical size, and (c) a
filter that quadruples the effective detector size. The filter does not affect
the array Nyquist frequency.
286 CCD ARRAYS, CAMERAS, and DISPLAYS

10.4. OPTICS-DETECTOR SUBSYSTEM

For many solid state cameras, the system MTF is dominated by the optics
and detector MTFs. Here, the electronic MTFs maximize the signal. They are
approximately unity over the spatial frequencies of interest. With electronic
zoom, the effects of the display and observer MTFs can be minimized.
Therefore, it is worthwhile to treat the optics and detector MTFs as a subsystem
assembly and to examine this subsystem MTF.

When viewing a point source, diffraction-limited optics produces a

diffraction pattern that consists of a bull's-eye pattern with concentric rings. The
center is the Airy disk and its diameter is

dAIRY = 2.44 J..fl - 2.44). F . (10-26)

Do

Assume that the detector array has square detectors with 100% fill factor (dH =
dv = dccv = dccH =d). When dAIRY is larger than the detector size, the system
is said to be optics-limited. Here, changes in the optics MTF (i.e., Do or fl)
significantly affect MTFsYSTEM· If dAIRY < d, the system is detector-limited and
changes in the detector size affect MTFsYSTEM at uiN· Figure 10-14 illustrates
MTFoPTicsMTFoETECTOR at Nyquist frequency as a function of d/F>.. It is the ratio
of the optical cutoff to the detector cutoff: uic/uiD = d/F>..

optics detector
0.7 limited , limited detector
0.6
0.5 system
LL 0.4
1-
:::!! 0.3
0.2
0.1
0
0 2 4 6 8 10
d/FA
Figure 10-14. MTF 0 pncsMTFoETEcroR at Nyquist frequency as a function
of d!F>.. The fill factor is 100% (d = d 11 = dv = dccH= dccv). The
vertical line indicates where the Airy disk diameter is equal to the
detector size. The best MTF that can be achieved occurs when
MTFoPTics is negligible compared to MTFoETECTOR· Equivalently,
MTFopncsMTFoETECTOR == MTFoETECfOR = 0.637.
 SYSTEM MTF 287

The MTF at Nyquist frequency is often used as a measure of performance.

As the MTF increases, image quality should increase. Unavoidably, as the MTF
increases, aliasing also increases 12 and image quality suffers. To avoid aliasing,
the MTF at the Nyquist frequency must be zero. This requires an infinite f-
number lens or an optical anti-alias filter (discussed in the previous section).

Smaller chips make smaller cameras (we live in a world were smaller is
presumed more desirable). However, to maintain the MTF the f-number must
decrease. For visible systems (A =
0.5 p.m), the transition from the optics-
limited to detector-limited case occurs when F = d/1.22. If d = 10 p.m, then F
must be greater than 8.2 to enter the optics-limited condition (Figure 10-15).

For a small format chip, modest f-numbers provide an optics-limited camera

system. For low illumination level imagery, the user will probably switch to a
low f-number lens. If the camera enters the detector-limited region, pixelation
becomes evident and image quality suffers. Large format chips are detector-
limited for the more common f-number lenses. The transition from detector-
limited to optics-limited was arbitrarily selected as d = dArRY· The transition is
actually gradual.
format
e2.. 25
-1"
a::
w
20
detector-limited
1-
::2 15 -2/3"
<
c 10
-112"
~ _1/3"
C/)
c 5 optics-limited -1/4"

~
< 0
0 5 10 15 20

f-number
Figure 10-15. Airy disk diameter as a function off-number when )\ =
0.5 p.m. The nominal pixel sizes for the various video formats are from
Table 3-3 (page 99).

With wet-film cameras, in most cases, more scene detail can be discerned
by enlarging the print. This occurs because the silver granules in the film are
very small compared to the Airy disk. That is, the camera system is almost
always optics-limited. Enlarging the image provides more detail. With solid state
cameras, this is not true. Electronic zoom (which is analogous to print
enlargement) provides more detail only up to the point where individual pixels
can be seen. There is no more detail available.
288 CCD ARRAYS, CAMERAS, and DISPLAYS

The output voltage (for square detectors) is given by the simple camera
formula:

(10-27)

Decreasing the focal length, increases the MTF at uiN· It also increases the
camera output (improves sensitivity). But decreasing the focal length increases
the scene spatial frequencies relative to the detector response. That is, the
system's response moves from being detector-limited to being optics-limited.
Thus, a tradeoff exists between sensitivity and image quality (as defined by the
MTF).

While parametric analyses are convenient, absolute values are lost. The
MTF at Nyquist frequency is important. However, it is even more important to
know the relationship between the scene spatial content and the actual value of
the Nyquist frequency. In object space, as the focal length decreases, the
absolute value of the Nyquist frequency also decreases. It is instructive to see
the effects of d/Fft. on computer-generated imagery. Details of the simulation are
provided in Reference 13. Figure 10-16 illustrates the original scene.

Figure 10-16. Image created by the System Image Analyzer software. 14

 SYSTEM MTF 289

When d/E\ = 0.5, MTF(uN) = 0 and the optics have low-pass filtered the
scene. Here, no aliasing can occur. Figure 10-17a provides the MTFs and
Figure 10-17b illustrates the resultant imagery which includes a Butterworth
post-reconstruction filter where N = 10 (discussed in Section 10.6.4., Post-
Reconstruction Filter). While the low-pass filtering action of the optics removed
all aliasing, it also significantly attenuated the in-band frequency amplitudes.

1
0.8
u. 0.6
1-
::1 0.4
0.2
0
0 0.25 0.5 0.75 1
NORMALIZED SPATIAL FREQUENCY
(a)

(b)

Figure 10-17. (a) MTF0 PT1cs and MTFoETECTOR when d/FA. = 0.5 and (b)
reconstructed image including a Butterworth filter where N = 10. The
optics has significantly blurred the image and no aliasing exists. Created
by the System Image Analyzer software. 14
290 CCD ARRAYS, CAMERAS, and DISPLAYS

Figures 10-18 and 10-19 illustrate the cases where d!F>- = 2.44 and 10,
respectively. Note that the MTFs illustrated in Figures 10-18a and 10-19a do not
include the reconstruction filter whereas the imagery includes a 10th-order
Butterworth reconstruction filter.

1
0.8
LL 0.6
1-
:E 0.4
0.2
0
0 0.25 0.5 0.75 1
NORMALIZED SPATIAL FREQUENCY
(a)

(b)

Figure 10-18. (a) MTFoPTics and MTFoETE( TOR when d/FA = 2.44 and (b)
reconstructed image including a Butterworth filter where N = 10.
 SYSTEM MTF 291

Careful examination of Figure 10-19b will reveal that the edges are sharper
compared to Figure 10-18b but aliasing has increased. However, the effect is not
dramatic when using a high order Butterworth reconstruction filter. At Nyquist
frequency, MTF 0 PT1csMTFoETECToR increased from 0.472 to 0.592. In terms of
imagery, whether d/FA = 10 is better than d/FA = 2.44 depends on the scene
spectral content and the reconstruction filter used. The amount of aliased signal
is scene-dependent and the amount of aliasing accepted depends on personal
preference.

1
0.8
u.. 0.6
1-
:::! 0.4
0.2
0
0 0.25 0.5 0.75 1
NORMALIZED SPATIAL FREQUENCY
(a)

(b)

Figure 10-19. (a) MTFoPTics and MTFoETECTOR when d/FA = 10 and (b)
reconstructed image including a Butterworth filter where N = 10.
292 CCD ARRAYS, CAMERAS, and DISPLAYS

10.5. MOTION

Motion blurs imagery. When motion is fast compared to the detector

integration time, details become blurred in a reproduced image. In real-time
imagery, edges may appear fuzzy to the observer even though any one frame
may provide a sharp image. The eye integration time blends many frames. The
effects of motion during the entire integration and human interpretation process
must be considered.

Linear motion includes both target moYement (relative to the imaging

system) and motion of the imaging system (stationary target). High frequency
random motion is simply called jitter. Typical! y, an imaging system is subjected
to linear and random motion simultaneously:

MTFMOTION - MTFUNEAR MTFRANDOM . (10-28)

In a laboratory, the system and test targets may be mounted on a vibration-

isolated stabilized table. Here, no motion is expected so that MTFMonoN = 1.

10.5.1. LINEAR MOTION

The OTF degradation due to horizontal linear motion is:
OTFuNEAiu) - sinc(aLu) , (10-29)

where aL is the distance the target edge moved across the image plane. It is
equal to vR..::lt where vR is the relative image velocity between the sensor and the
target. For scientific and machine vision applications, the exposure time is equal
to the detector integration time. But observer's eye blends many frames of data
and the eye integration time should be used. Although the exact eye integration
time is debatable, the values most often cited are between 0.1 and 0.2s.

Figure 10-20 illustrates the OTF due to linear motion as a fraction of the
detector size, d. Recall that the MTF is the magnitude of the OTF. Linear
motion only affects the MTF in the direction of the motion. As a rule-of-thumb,
when the linear motion causes an image shift less than about 20% of the detector
size, it has minimal effect on system performance.

As the motion increases, a phase reversal occurs which is represented by a

negative OTF. Here, periodic black bars appear white and periodic white bars
appear black. Figure 10-21 illustrates a test pattern smeared by linear motion.
At cutoff (where the system MTF approaches the first zero), the bar pattern
cannot be resolved. Phase reversal occurs above cutoff.
 SYSTEM MTF 293

1
0.8
0.6
LL 0.4
1-
0 0.2
0
-0.2
-0.4
0 0.2 0.4 0.6 0.8
NORMALIZED SPATIAL FREQUENCY

Figure 10-20. OTFuNEAR as a function of normalized spatial frequency

dui. The curves represent values of add. In the laboratory <it. is usually
zero. The negative OTF represents phase reversal. The MTF is always
positive.
First zero

ORIGINAL IMAGE LINEAR MOTION

Figure 10-21. Linear motion smears imagery. Because the motion is
horizontal, the vertical resolution is maintained. When the bar target
frequency is equal to the first zero of OTFuNEAR• the bars all blend
together. Higher spatial frequency bars illustrate phase reversal: There
appear to be two black bars on a gray background. (From Reference 15:
by courtesy of G. L. Conrad).
294 CCD ARRAYS, CAMERAS, and DISPLAYS

10.5.2. RANDOM MOTION (TITTER)

With high frequency motion, it is assumed that the image has moved often
during the integration time so that the central limit theorem is valid. The central
limit theorem says that many random movemenl s can be described by a Gaussian
distribution. The Gaussian MTF is
(10-30)

where uRis the rms random displacement in millimeter. Figure 10-22 illustrates
the random motion MTF. As a rule-of-thumb, when the rms value of the
random motion is less than about 10% of d (detector size), system performance
is not significantly affected. Jitter is considered equal in all directions and is
included in both the horizontal and vertical system MTFs. High frequency
motion blurs imagery (Figure 10-23).

lab=O
1
0.1
0.8

Ll. 0.6
1-
:IE 0.4
0.2

0
0 0.2 0.4 0.6 0.8
NORMALIZED SPATIAL FREQUENCY

Figure 10-22. MTF degradation due to high frequency random

(Gaussian) movement as a function of uRui. In the laboratory, uR is
usually zero.
 SYSTEM MTF 295

(a) (b)

Figure 10-23. Random motion blurs imagery. (a) Original image, (b)
two-dimensional random motion. Generated 14 by SIA.

10.6. ELECTRONIC FILTERS

When filter response is provided as a function of electrical frequency, it is

denoted by H(fJ or H(f). When the electrical frequency is expressed in object
space or image space units, the filter impulse response is labeled as an MTF.
Boost filters may be used to enhance the signal at selected spatial frequencies.
Both analog and digital filters can be used.

10. 6.1. ANTI-ALIAS ANALOG FILTER

Ideally, H(fe) is a filter with MTF of unity up to the array Nyquist frequency
and then drops to zero. This filter maximizes the SNR by attenuating out-of-
band amplifier noise and passing the signal without attenuation. The ideal filter
is unrealizable but can be approximated by Nh-order Butterworth filters:
1
H LOWPASS(je ) - ---;:::========
2N (10-31)
1 f.
+ [ f.3dB ]

where fe3dB is the frequency at which the power is one-half or the amplitude is
0. 707. As N ~ co, HwwrAss(fe) approaches the ideal filter response with the
cutoff frequency of feJdB (Figure 10-24).
296 CCD ARRAYS, CAMERAS, and DISPLAYS

When feJdB is approximately equivalent to the array Nyquist frequency, the

sampling theorem can be satisfied. However, square waves (bar targets) will
appear as sinusoids. Thus, feJdB must be much higher to preserve the square-
wave response.

0.8

u. 0.6
1-
:E 0.4
0.2

0 0.25 0.5 0.75

RELATIVE SPATIAL FREQUENCY

Figure 10-24. Butterworth filters for N = I, 5, 10, and 50. The value
fe3dB = 0.5.

10.6.2. DIGITAL FILTERS

There are many available image processing algorithms. Only a few of these
can be described mathematically in closed-form and only these can be included
in an end-to-end system performance model. The performance of the remaining
algorithms can be inferred only by viewing the system output for a few
representative inputs. Because the system is not spatially invariant, predicting
system performance when viewing any other target is pure conjecture.

The sampling process replicates the frequency spectrum at nf.s (see Section
8.1., Sampling Theorem, page 234). Similarly, digital filter response is
symmetrical about the Nyquist frequency and repeats at multiples of the
sampling frequency. The highest frequency of interest is the Nyquist frequency.

Digital filters process data that reside in a memory. The units assigned to the
filter match the units assigned to the data arrays. With one-to-one mapping of
pixels to datels, the filter sampling frequency is the same as the array sampling
frequency. With this mapping, each filter coefficient processes one pixel value.
Digital filters can be two-dimensional.
 SYSTEM MTF 297

There are two general classes of digital filters 16 : infinite impulse response
(IIR) and finite impulse response (FIR). Both have advantages and
disadvantages. The FIR has a linear phase shift whereas the IIR does not. IIR
filters tend to have excellent amplitude response whereas FIR filters tend to have
more ripple. FIR filters are typically symmetrical in that the weightings are
symmetrical about the center sample. They are also the easiest to implement in
hardware or software.

Figure 10-25 illustrates two FIR filters. The digital filter design software
provides the coefficients, Ai. The central data point is replaced by the digital
filter coefficients as they operate on the neighboring data points. The filter is
then moved one data point and the process is repeated until the entire data set
has been operated on. Edge effects exist with any digital filter. The filter
illustrated in Figure 10-25a requires seven inputs before a valid output can be
achieved. At the very beginning of the data set, there are insufficient data points
to have a valid output at data point 1, 2, or 3. The user must be aware of edge
effects at both the beginning and the end of his data record. In effect, this states
that edges cannot be filtered.

The following MTF equations are only valid where there are no edge
effects. For FIR filters where the multiplicative factors (weightings) are
symmetrical about the center, the filter is mathematically represented by a cosine
series (sometimes called a cosine filter). With its frequency response transposed
to electrical frequency, an FIR filter with an odd number of samples (also called
taps) provides
N-1
2 (10-32)
L
k-0

For an even number of samples,

(10-33)
298 CCD ARRAYS, CAMERAS, and DISPLAYS

DDDDDD D Pixel Values

i l i l t i t Multiplier

7~!
"'!'
'y

Output
(a) Odd Number

D D D D Pixel Values

i "T!, "T! "Tl

7
Multiplier

"'\ 1/
C? Output
(b) Even Number

Figure 10-25. Symmetrical digital filters. (a) 7-tap (odd number) filter
and (b) 4-tap (even number) filter.

The sum of the coefficients should equal unity so that the HoFILTER(fe = 0) is one:
(10-34)

Although the above equations provide the filter response in closed form, the
response of a real filter is limited by the ability to implement the coefficients.
The smallest increment is the LSB.
 SYSTEM MTF 299

With an averaging filter, 9 all the multipliers shown in Figure 10-25 are
equal. When NAvE datels are averaged, the equivalent MTF is

(10-35)

The first zero of this function occurs at f05 /NAvE·

Averaging samples together is called binning or super pixeling. It provides

the same MTF as if the detector elements were proportionally larger. That is,
averaging two samples together provides the same MTF as a single detector
element that is twice as large. Cameras that operate in either the
pseudo-interlace or the progressive scan modes will have a different vertical
MTF for each mode. In the pseudo-interlace mode, two pixels are averaged
together. If dH = dccH• then

sine( NAe< 1·]

MTFDETEcroJir!TFDFILTER - sinc(dnfe) fes
. [t.]
sznc-
fes
(10-36)

where the effective detector width is NAvEdH.

10.6.3. SAMPLE-AND-HOLD

After the digital-to-analog converter (DAC), the analog signal changes only
at discrete times. The sample-and-hold circuitry within the DAC extends the data
into a continuous analog signal. The MTF of a zero-order sample-and-hold when
transposed into image space is:

MTFs&H - sinc(_'2] , (10-37)

uis

where uis = lldccH· Figure 10-26 illustrates the sample-and-hold MTF as a

function of normalized spatial frequency.
300 CCD ARRAYS, CAMERAS, and DISPLAYS

Nyquist frequency
0.8

LL 0.6
1-
:::E 0.4
0.2

0~-------+--------+--------+------~
0 0.25 0.5 0.75

NORMALIZED SPATIAL FREQUENCY

Figure 10-26. Zero-order sample-and-hold MTF normalized to dccHU;.

The sample-and-hold acts as a low-pass filter.

10.6.4. POST-RECONSTRUCTION FILTER

After the digital-to-analog conversion, the image still appears blocky due to
the discrete nature of the conversion. The ideal post-reconstruction filter
removes all the higher order frequencies (Figure 10-27) such that only the
original smooth signal remains. The output is delayed but this is not noticeable
on most imagery.

input

post-reconstruction filter output

Figure 10-27. The post-reconstruction filter removes the blocky (stair-

step) effect created by the sample-and-hold circuitry within the DAC.
 SYSTEM MTF 301

Ideally, H(fv) is a filter with MTF of unity up to the array Nyquist frequency
and then drops to zero. The ideal filter is unrealizable but can be approximated
by N1h-order Butterworth filters:

2 (10-38)
1+ f ]
_v_ N
( fv3dB
where fv3dB is the frequency at which the power is one-half or the amplitude is
0.707. As N-+ oo, HwwPAss(fv) approaches the ideal filter response with the
cutoff frequency of fv3dB. Real filters will have some roll-off and fvJdB should be
sufficiently large so that it does not affect the in-band MTF. The post-
reconstruction filter can be of the same functional form as the anti-alias filter
(see Section 10.6.1., page 295) but serves a different purpose. It removes the
spurious frequencies created by the sampling process.

For analog video output, A low-pass filter is used as a post-reconstruction

filter. This filter affects only the horizontal signal. While the video bandwidth
has been standardized (NTSC, PAL, and SECAM are 4.2, 5.5, and 6 MHz,
respectively), the filter design has not. Thus, different cameras may have
different post-reconstruction filters and therefore different horizontal MTFs even
though they are built to the same video standard. No reconstruction filter is used
in the vertical direction. Here, the display and human visual response provide
the reconstruction. As a result of different detector sizes (dv and dH) and
horizontal analog filters, the MTFs in the vertical and horizontal directions can
be quite different.

Example 10-2
POST-RECONSTRUCTION FILTER

What is the required spatial frequency cutofffor an ideal post-reconstruction

filter if the desired output is EIA 170 and there are 640 horizontal detectors in
the array?
302 CCD ARRAYS, CAMERAS, and DISPLAYS

For EIA 170, tuNE is 52.09 J.tS. The Nyquist frequency in the video domain
is
~ - ..!. 640
= 6.14 Mhz (10-39)
VN 2 52.09 X 10-6

The bandwidth necessary to transmit the detector data must exceed 6.14 MHz
to avoid roll-off effects. This is a greater bandwidth than allowed by the EIA
170 standard. However, the camera output only has to conform to EIA 170
timing so that the information can be displayed on a conventional monitor.

===============================================~

10.6.5. BOOST

MTF degradation caused by the various subsystems can be partially

compensated with electronic boost filters. Boost filters also amplify noise so that
the signal-to-noise ratio may degrade. For systems that are contrast-limited,
boost may improve image quality. However, for noisy images, the advantages
of boost are less obvious. As a result, these filters are used only in high contrast
situations (typical of consumer applications) and are not used in scientific
applications where low signal-to-noise situations are often encountered.
Historically, each subsystem was called an "aperture." Therefore, boost
provided "aperture correction." This should not be confused with the optics
MTF where the optical diameter is the entrance aperture. "Aperture correction"
has been used in video for years.

The boost amplifier can either be an analog or a digital circuit with peaking
that compensates for any specified MTF roll-off. The MTF of a boost filter, by
definition, exceeds one over a limited spatial frequency range. When used with
all the other subsystem MTFs, the resultant MTFsvSTEM is typically less than one
for all spatial frequencies. Excessive boost can cause ringing at sharp edges. The
boost algorithm should enhance signal amplitudes with frequencies below the
Nyquist because reconstruction usually limits the response to the Nyquist
frequency.
 SYSTEM MTF 303
A variety of boost circuits are available. 17 One example is the tuned circuit
with response
1

(10-40)

where Q is the quality factor and equal to boost amplitude when fv = fsoosT·

Although illustrated in the video domain, boost can be placed in any part of
the circuit where a serial stream of data exists. By appropriate selection of
coefficients, digital filters can also provide boost. MTFsoosT can also
approximate the inverse of MTFSYSTEM (without boost). The combination of
MTFsoosT and MTFsYsTEM can provide unity MTF over spatial frequencies of
interest. This suggests that the reproduced image will precisely match the scene
in every spatial detail.

10.7. CRT-BASED DISPLAY

For most electronic imaging systems, the display medium acts as a

reconstruction filter. Displays do not completely attenuate the replicated
frequency amplitudes created by the sampling process. The HVS further reduces
the sampling remnants so that the perceived image appears continuous with
minimal sampling artifacts. Therefore, displays cannot be analyzed in isolation
but must be considered as part of the display /observer system.

The display MTF is a composite that includes both the internal amplifier and
the CRT responses. Implicit in the MTF is the conversion from input voltage to
output display brightness. Although not explicitly stated, the equation implies
radial symmetry and the MTFs are the same in both the vertical and horizontal
directions. That is, the MTF is considered separable and MTFcRT(~,vd) =
MTFcRT(ud)MTFcRT(vd) and MTFcRT(vct) = MTFcRT(Uct). While this may not be
precisely true, the assumption is adequate for most calculations.

The amplifier bandwidth is not a number that is often useful to the user. We
must assume that the manufacturer selected an adequate bandwidth to provide
the display resolution. For example, a 1600 x 1200-disel display must have a
wider bandwidth than a 1280 X 1024-disel display by a factor of approximately
1600/1280= 1.25. MTFcRT includes both the electronics MTF and spot diameter.
For an infinitesimal input, the spot diameter is affected by the video bandwidth.
That is, the minimum spot diameter is the equivalent point spread function.
304 CCD ARRAYS, CAMERAS, and DISPLAYS

It is reasonable to assume that the resultant spot is Gaussian (see Section

7.3., Spot Size, page 210). Then,
2 2 2 -2" 2(-su )2 (10-41)
M'T''F ( ) ..
.~.. CRT'-ud e
-2" a SPOTud _
e
2.35 d

where <TsroT is the standard deviation of the Gaussian beam profile and S is the
spot diameter at FWHM. These parameters must have units of millimeters
because the frequency has units of cycles/mm. The spot diameter is
S - J81n(2) a SPOT - 2.35 a SPOT. (10-42)

If the spot size is not specified, then usPoT can be estimated from the TV
limiting resolution. With the TV limiting resolution test, an observer views a
wedge pattern and selects that spatial frequency at which the converging bar
pattern can no longer be seen. The display industry has standardized the bar
spacing to 2.35usroT·

If NTV/PH is the TV limiting resolution and NTV lines are displayed on a

monitor, then HMoNITOR :::::: 2.35usPoTNTV. When transposed to image space,
_2 ,2( VFOVfl u.) 2
(10-43)
M 'T''F ( ) ..
.~.. CRT'-ui e
2.35 N 7V I

For staring arrays with Nv displayed vertical lines,

-2"2 ((Nv- l)dccv• dv "·)2 1 (10-44)
M
..,..'F ( ) _ 2.35 Nrv
.~., CRT'-ui e

Figure 10-28 illustrates MTFoETEcroR and MTF 01sPLAY· The display spot size
is independent of the detector size and the display resolution is independent of
the number of detector elements. Matching the display resolution (number of
pixels) to the number of detector elements does not uniquely specify the display
MTF as suggested by Equation 10-44. For example, suppose an array contained
1000 elements horizontally. A monitor that can resolve 1000 pixels may not
provide good imagery if the resolution/addressability ratio (RAR) is large.
Equation 10-44 is only valid when RAR :::::: 1. The RAR was described in Section
7.6., Addressability (page 222).

Many consumer CCD cameras match the number of vertical detectors to the
number of active video lines. With consumer televisions, NTV :::::: 0. 7Nv. Thus,
the video standard uniquely defines the CRT MTF. For 100% fill factor arrays,
2
-2" 2(dccv
- - u .) (10-45)
.. e 1.645 I
 SYSTEM MTF 305

At Nyquist frequency, MTFcRT = 0.161. That is, the "standard" design

effectively reduces sampling effects by attenuating the amplitudes above uiN·
However, the in-band amplitudes are unavoidably attenuated also.

0.8

u.. 0.6
1-
::::!! 0.4

0.2

0
0 0.2 0.4 0.6 0.8
NORMALIZED SPATIAL FREQUENCY

Figure 10-28. MTFmsPLAY as a function of dccvVi for four different TV

limiting resolutions when Nv = 480. The display MTF limits most
systems. The Nyquist frequency (not shown) is at dccvVi = 0.5.

~=============================================
Example 10-3
DISPLAY MTF

What should the display resolution be to maximize the system MTF?

Equations 10-43 or 10-44 are used to calculate image MTF when the input
is a sinusoid. The Nyquist frequency is used most often when specifying system
response. For EIA 170, Nv = 485. Inserting dvui = 0.5 into Equation 10-45
provides
2 2( 0.5 485 )2 (10-46)
M 'T'F ( ) = -
.L CRT' UiN e
1t 2.35 N7V

For a 100% fill factor array, the detector MTF is 0.63 at uiN· Figure 10-29
illustrates the detector and display MTF at Nyquist frequency. The display
resolution must increase to 1400 TVLIPH for display MTF to be 0.90 at the
Nyquist frequency. This suggests that most imaging systems are display
resolution limited. This result is based upon the assumption that the beam profile
is Gaussian. A different shaped beam (i.e., flat-topped), will provide a different
relationship between TV limiting resolution and MTF.
306 CCD ARRAYS, CAMERAS, and DISPLAYS

0.7 detector
0.6
0.5
LL 0.4
1-
:2 0.3
0.2
0.1
0
0 200 400 600 800 1000 1200

RESOLUTION (TV LINES)

Figure 10-29. MTFoETEcroRMTFmsrLAY at Nyquist frequency as a

function of TV liming resolution. The detectors are contiguous so that
the center-to-center spacing is equal to the detector element size. The
value Nv = 485. All other MTFs are assumed to be unity at u;N·

============================================~

10.8. HUMAN VISUAL RESPONSE

The observer processes imagery to create a perceived MTF:

MTF PERCEIVED - MTFSYS MTFHVS •
(10-47)

The HVS MTF should only be used to understand the relationship between the
imaging system MTF and the eye response. It is not used to describe sensor
performance. When MTFrERCEivEo is combined with system noise, system
responsivity, and the eye integration capabilities, it is possible to predict the
minimum resolvable contrast (described in Chapter 12, Minimum Resolvable
Contrast). The eye's contrast sensitivity is also included in all image quality
metrics (discussed in Chapter 11, Image Quality).

The sine wave response is used as an approximation to the eye-brain MTF.

It is sometimes called the human visual system MTF or HVS-MTF. The sine
wave response depends on diffraction by the pupil, aberrations of the lens, finite
size of the photoreceptors, ocular tremor, and neural interconnections within the
retina and brain. Diffraction and aberrations vary with ambient luminance,
monitor brightness, and chromatic composition of the light. Because the retina
is composed of rods and cones of varying densities, the location and size of the
object with respect to the fovea significantly affect the sine wave response.
 SYSTEM MTF 307

The sine wave response ignores spatial noise, background luminance,

angular orientation and exposure time. Each of these parameters significantly
affects the interpretation of image visibility. The MTF in the purest sense is
noise independent but the human eye's response is very sensitive to spatial and
temporal noise. Therefore, the sine wave response is only an approximation to
the true response. Furthermore, the overall population exhibits large variations
in response. Any MTF approximation used for the HVS therefore is only a
crude approximation and probably represents the largest uncertainty in the
overall MTF analysis approach. Factors that affect observer performance include
monitor viewing ratio (usually selected to reduce raster effects), target size, and
observation time. Visual psychophysical data can be found in Farrell and
Booth's 18 and Biberman's 19 books.

The usual measurement is to determine the minimum modulation required

to just perceive a sinusoidal target. The modulation is
(10-48)

where LMAx and LMIN are the maximum and minimum luminances, respectively.
The eye's detection capability depends upon the visual angle subtended by the
target size at the observer. As shown in Figure 10-30, in the absence of noise,
the eye's modulation threshold (minimum perceivable modulation) is
characteristically J -shaped. It is labeled as M1 in the visual psychophysical
literature. The eye is most sensitive to spatial frequencies 20 that range between
3 and 5 cycles/deg at typical ambient lighting levels.

zc
0...1 0.1
i=O
:5i}i
::JW
ca: 0.01
OJ:
:::tl-

0.001
0.1 1 10 100
SPATIAL FREQUENCY (cy/deg)

Figure 10-30. A representative modulation threshold function.

308 CCD ARRAYS, CAMERAS, and DISPLAYS

The increase in threshold at low frequencies is due to the eye's inhibitory

signal processing component. The contrast sensitivity is the inverse of the
modulation threshold curve. When normalized to one, the contrast sensitivity or
sine wave response is called the HVS MTF. By convention, the MTF is defined
to be one at zero spatial frequency and to be a monotonically decreasing
function. The way to include the inhibitory response as an MTF component is
not clear.

Nevertheless, various researchers modeled the normalized contrast sensitivity

and labeled it as the HVS MTF (Figure 10-31 ). Nill21 recommended

MTF.,.if_,.) - 2.71 [0.19 + 0.81 [J.:J e -(/.:.,) . (10-49)

Schulze22 recommended
MTFHVS(Jeye) - 2.71 (e -O.ll38f,,. - e-0.32Sf.,.) • (10-50)

Campbell and Robson's high luminance data23 can be approximated by

-1.4 log 10 f.
_2.._ )2 (10-51)
(
MTFHVS(f.eye ) - 10 /pEAl(

deJong and Bakker24 used

MTFHVS(f.eye ) = I'
sin2 2:. /,/.ye
2 1 PEAK
]1/3] •
(10-52)

Neglecting the low frequency inhibitory response, Kornfeld and Lawson25

suggested that
r !.,. (10-53)
MTFHVS(Jeye) - e 17 .4S

where r is a light-level-dependent eye response factor that was presented in

tabular form26 by Ratches et al. A third order polynomial fit provides
r -1.444-0.344log(B)+0.0395log2 (B)+0.00197log 3(B), (10-54)

where B is the monitor brightness in foot-Lamberts.

 SYSTEM MTF 309

0.8

u. 0.6
1-
::!!! 0.4

0.2

0
0 5 10 15 20
SPATIAL FREQUENCY (cy/deg)
Figure 10-31. Various mathematical equations that approximate the HVS
MTF.

Considering the large variation in observer responses, these curves have a

similar shape when feye > fPEAK· The original U.S. Army Night Vision
Laboratory's static performance model 26 and FLIR92 27 use the Komfeld-Lawson
eye model. These models were validated by the U.S. Army when detecting tank-
sized targets at modest ranges. This translates into modest spatial frequencies.
The Komfeld-Lawson model appears to follow all other models (to within a
multiplicative factor) for mid-range spatial frequencies. This multiplicative factor
becomes a normalization issue for the MRC (discussed in Chapter 12, Minimum
Resolvable Contrast).

Insufficient data exist to say with certainty which eye model is best.
Because of this uncertainty, all analyses, no matter which eye model is used,
must only be used for comparative performance purposes.

Two scenarios are possible: (1) the observer is allowed to move his head and
(2) the head is fixed in space. Because the eye's detection capability depends
upon the angular subtense of the target, head movement may provide different
results than if the head is fixed. In laboratory testing, the distance to the monitor
is not usually specified nor limited in any way. To maximize detection capability
(stay on the minimum of the contrast threshold curve), an observer
subconsciously moves toward the monitor to perceive small targets and further
away to see larger targets.

By allowing the observer to adjust the viewing distance, several interrelated

detection criteria are optimized that include striving for apparent edge sharpness
and maximizing perceived signal-to-noise ratio. This apparently results in an
310 CCD ARRAYS, CAMERAS, and DISPLAYS

equal detection capability for all spatial frequencies such that the eye's contrast
sensitivity approaches a constant. This results in a nearly constant MTF that is
called the "non-limiting eye MTF" or MTFHvs = 1. Equation 10-53 is used in
those situations where the head is fixed, such as when a pilot is strapped into a
cockpit seat.

10.9. INTENSIFIED CCD

The intensified CCD (ICCD) is more difficult to analyze. The MTF of an

image intensifier tube (liT) varies with tube type, 28 manufacturing precision, 29
and materials used. Figure 10-32 illustrates a possible range of MTFs. The MTF
should be obtained from the manufacturer before proceeding with system
analysis. It can be approximated by a simple Gaussian expression:
(10-55)

MTF theory will provide an optimistic result of fiber optic coupled ICCDs. It
does not include the CCD mismatch or the chicken wire effects. The MTF
associated with lens coupling is probably more accurate. The MTF of an ICCD
is MTF1NTENSIFIER muitiplied by the appropriate MTFs presented in this chapter.

0.8

LL 0.6
1-
::::E 0.4

0.2

0
0 5 10 15 20 25 30 35
SPATIAL FREQUENCY (cy/mm)

Figure 10-32. Range of MTFs for Genii image intensifier (From

Reference 29).
 SYSTEM MTF 311

10.10. SAMPLING EFFECTS

For a sampled-data system, a wide range of MTF values is possible30-34 for

any given spatial frequency (see Section 8.3.3 ., Image Distortion, page 245). To
account for this variation, a sample-scene MTF may be included. The detector
spatial frequency response is represented by MTFoETEcToRMTFrHAsE· This MTF
provides an average performance response that may be used to calculate the
system response general imagery.

In general, the "MTF" is35

~ cos(_5_ e] ,
uiN
(10-56)

where 8 is the phase angle between the target and the sampling lattice. For
example, when u; = uiN• the MTF is a maximum when 8 = 0 (in-phase) and a
zero when e = 7r/2 (out-of-phase).

To approximate a median value for phasing, 8 is set to 7r/4. Here,

approximately one-half of the time the MTF will be higher and one-half of the
time the MTF will be lower. The median sampling MTF is

MTFMEvw/u) ~cos -
1t -
U; .
(
2 U;s
l (10-57)

At Nyquist frequency, MTFrHASE is 0.707. This is the Kell factor so often

reported when specifying the resolution of monitors. An MTF averaged over all
phases is represented by

(10-58)

Figure 10-33 illustrates the difference between the two equations. Because these
are approximations, they may be considered roughly equal over the range of
interest (zero to the Nyquist frequency). MTFrHASE can be applied to both the
vertical and horizontal MTFs.
312 CCD ARRAYS, CAMERAS, and DISPLAYS

Sampling effects are phase-dependent and, as a result, have no unique MTF.

For system modeling purposes, average MTFs were assigned for some more
common image processing algorithms. Because average MTFs are used, only
an average system performance can be predicted. Performance can vary
dramatically depending upon the target phase with respect to the sampling
lattice. For laboratory measurements, it is common practice to "peak-up" the
target (in-phase relationship) and then MTFrFASE = 1. This is done to get
repeatable results.

0.8

LL 0.6
......
:E 0.4
0.2

0
0 0.25 0.5 0.75 1

NORMALIZED SPATIAL FREQUENCY

Figure 10-33. Average and median scene-sample phase MTFs
normalized to u/u5 • The MTF is defined only up to the Nyquist
frequency.

Frequencies above uiN are aliased down to lower frequencies. Signal aliasing
is ignored in most analyses and has been ignored here. A complete end-to-end
analysis should include both aliased noise and aliased signal36 • These aliased
components lirnit37 the extent to which a sampled image can be sharpened.
Image reconstruction may include data removal (decimation) or data expansion
(interpolation). The effective sampling frequency and aliasing can appear to
change using different image processing algorithms. 3840

10.11. REFERENCES
1. See, for example, R. J. Farrell and J. M. Booth, Design Handbook for Imagery Interpretation
Equipment, Reprinted with corrections, Report D 180- 19063-1, Boeing Aerospace Company,
Seattle, WA, February 1984.
2. G. C. Holst, Testing and Evaluation of Infrared Imaging Systems, pp. 106-108, JCD
Publishing, Winter Park, Fl (1993).
3. G. C. Holst, Testing and Evaluation of Infrared Imaging Systems, pp. 195-200, JCD
Publishing, Winter Park, Fl (1993).
4. G. C. Holst, Testing and Evaluation of Infrared Imaging Systems, pp. 104-105, JCD
Publishing, Winter Park, Fl (1993).
 SYSTEM MTF 313

5. H. H. Hosack, "Aperture Response and Optical Performance of Patterned-Electrode Virtual-

Phase Imagers," IEEE Transactions on Electron Devices, Vol. ED-28(1) pp. 53-63 (1981).
6. D. H. Sieb, "Carrier Diffusion Degradation of Modulation Transfer Function in Charge
Coupled Imagers," IEEE Transactions on Electron Devices, Vol. ED-21(5), pp. 210-217
(1974).
7. L. W. Schumann and T. S. Lomheim, "Modulation Transfer Function and Quantum Efficiency
Correlation at Long Wavelengths (Greater Than 800 nm) in Linear Charge Coupled Imagers,"
Applied Optics, Vol. 28(9), pp. 1701-1709(1989).
8. E. L. Dereniak and D. G. Crowe, Optical Radiation Detectors, pp. 199-203, John Wiley &
Sons, New York (1984).
9. H. V. Kennedy, "Miscellaneous Modulation Transfer Function (MTF) Effects Relating to
Sampling Summing," in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing,
G. C. Holst, ed., SPIE Proceedings Vol. 1488, pp. 165-176 (1991).
10. H.-S. Wong. Y. L. Yao, and E. S. Schlig, "TDI Charge-coupled Devices: Design and
Applications," IBM Journal of Research and Development, Vol. 36(1), pp. 83-105 (1992).
II. J. E. Greivenkamp, "Color Dependent Optical Prefilter for the Suppression of Aliasing
Artifacts," Applied Optics, Vol. 29(5), pp 676-684 (1990).
12. G. C. Holst, Sampling, Aliasing, and Data Fidelity, pp. 308-311, JCD Publishing, Winter
Park, FL (1998).
13. G. C. Holst, Sampling, Aliasing, and Data Fidelity, pp. 259-268, JCD Publishing, Winter
Park, FL (1998).
14. System Image Analyzer is available from JCD Publishing, Winter Park, FL (1995).
15. G. L. Conrad, "Reconnaissance System Performance Predictions Through Image Processing,"
in Airborne Reconnaissance XIV, P A. Henkel, F. R. LaGesse, and W. W. Schurter, eds.,
SPIE Proceedings Vol. 1342, pp. 138-145 (1990).
16. A variety of texts on digital filter design are available. See, for example, Digital Signal
Processing, A. V. Oppenheim and R. W. Schafer, Prentice-Hall, New Jersey (1975).
17. G. C. Holst, Electro-Optical System Peiformance, pp. 146-148, JCD Publishing, Winter Park,
FL (1995).
18. R. J. Farrell and J. M. Booth, Design Handbook for Imagery Interpretation Equipment,
Reprinted with corrections, Report D 180-19063-1, Boeing Aerospace Company, Seattle Wash,
February 1984.
19. L. M. Biberman, ed., Perception of Displayed Information, Plenum Press, New York (1973).
20. B. 0. Hultgren, "Subjective Quality Factor Revisited," in Human Vision and Electronic
Imaging: Models, Methods, and Applications, B. E. Rogowitz and J. P. Allebach, eds., SPIE
Proceedings Vol. 1249, pp. 12-22 (1990).
21. N. Nill, "A Visual Model Weighted Cosine Transform for Image Compression and Quality
Measurements," IEEE Transactions on Communications, Vol. 33(6), pp. 551-557 (1985).
22. T. J. Schulze, "A Procedure for Calculating the Resolution of Electro-Optical Systems," in
Airborne Reconnaissance XIV, P A. Henkel, F. R. LaGesse, and W. W. Schurter, eds., SPIE
Proceedings Vol. 1342, pp. 317-327 (1990).
23. F. W. Campbell and J. G. Robson, "Application of Fourier Analysis to the Visibility of
Gratings," Journal of Physiology, Vol. 197, pp. 551-566 (1968).
24. A. N. deJong and S. J. M. Bakker, "Fast and Objective MRTD Measurements," in Infrared
Systems- Design and Testing, P.R. Hall and J. S. Seeley, eds., SPIE Proceedings Vol. 916,
pp. 127-143 (1988).
25. G. H. Kornfeld and W. R. Lawson, "Visual Perception Model," Journal of the Optical Society
of America, Vol. 61(6), pp. 811-820 (1971).
26. J. Ratches, W. R. Lawson, L. P. Obert, R. J. Bergemann, T. W. Cassidy, and J. M. Swenson,
"Night Vision Laboratory Static Performance Model for Thermal Viewing Systems," U.S.
Army Electronics Command Report ECOM Report 7043, p. 11, Ft. Monmouth, NJ (1975).
314 CCD ARRAYS, CAMERAS, and DISPLAYS

27. FUR92 Thermal Imaging Systems Performance Model, Analyst's Reference Guide, Document
RG5008993, Ft. Belvoir, VA, January 1993.
28. G. M. Williams, Jr., "A High Performance LLLTV CCD Camera for Nighttime Pilotage," in
Electron Tubes and Image Intensifiers, C. B. Johnson and B. N. Laprade, eds., SPIE
Proceedings Vol. 1655, pp. 14-32 (1992).
29. I. P. Csorba, Image Tubes, pp. 79-103, Howard W. Sams, Indianapolis, IN (1985).
30. S. K. Park and R. A. Schowengerdt, "Image Sampling, Reconstruction and the Effect of
Sample-scene Phasing," Applied Optics, Vol. 21(17). pp. 3142-3151 (1982).
31. J. C. Feltz and M. A. Karim, "Modulation Transfer Function of Charge-coupled Devices,"
Applied Optics, Vol. 29(5), pp. 717-722 (1990).
32. J. C. Feltz, "Development of the Modulation Transfer Function and Contrast Transfer Function
for Discrete Systems, Particularly Charge-coupled Devices," Optical Engineering, Vol. 29(8),
pp. 893-904 (1990).
34. L. de Luca and G. Cardone, "Modulation Transfer Function Cascade Model for a Sampled IR
Imaging System," Applied Optics, Vol. 30(13), pp. 1659-1664 (1991).
35. F. A. Rosell, "Effects of Image Sampling," in The Fundamentals of Thermal Imaging Systems,
F. Rosell and G. Harvey, eds., p. 217, NRL Report 8311, Naval Research Laboratory,
Washington D.C. (1979).
36. S. K. Park, "Image Gathering, Interpolation, and Restoration: A Fidelity Analysis," in Visual
Information Processing, F. 0. Huck and R. D. Juday, eds., SPIE Proceedings Vol. 1705, pp.
134-144 (1992).
37. S. K. Park and R. Hazra, "Image Restoration Versus Aliased Noise Enhancement," in Visual
Information Processing Ill, F. 0. Huck and R. D. Juday, eds., SPIE Proceedings Vol. 2239,
pp. 52-62 (1994).
38. I. Ajewole, "A Comparative Analysis of Digital Filters for Image Decimation and
Interpolation," in Applications of Digital Image Processing VIII, A. G. Tescher, ed., SPIE
Proceedings Vol. 575, pp. 2-12 (1985).
38. W. F. Schrieber and D. E. Troxel, "Transforma[ion Between Continuous and Discrete
Representations of Images: A Perceptual Approach,·· TEEE Transactions on Pattern Analysis
and Machine Intelligence, Vol. PAMA-7(2), pp. 178-186 (1985).
39. R. W. Schafer and L. R. Rabiner, "A Digital Signal Processing Approach to Interpolation,"
Proceedings of the IEEE, Vol. 61(8), pp. 692-702 (1973).
40. A. H. Lettington and Q. H. Hong, "Interpolator for Infrared Images," Optical Engineering,
Vol. 33(3), pp. 725-729 (1994).
11
IMAGE QUALITY
Our perception of good image quality is based upon the real-world
experiences of seeing all colors, all intensities, and textures. An imaging system
has a limited field-of-view, limited temporal and spatial resolutions, and presents
a two-dimensional view of a three-dimensional world. In the real-world our eyes
scan the entire scene. Not all of the available information is captured by an
imaging system. Furthermore, an imaging system introduces noise and the loss
of image quality due to noise can only be estimated. Cameras sensitive to
wavelengths less than 0.4 JI-m and greater than 0.7 JI-m provide imagery that we
cannot directly perceive. The quality of the ultraviolet or infrared imagery can
only be estimated because we do not know how it really appears.

Image quality is a subjective impression ranging from poor to excellent. It

is a somewhat learned ability. It is a perceptual one, accomplished by the brain,
affected by and incorporating inputs from other sensory systems, emotions,
learning, and memory. The relationships are many and are still not well
understood. Perceptual quality of the same scene varies between individuals and
temporally for the same individual. Large variations exist in an observer's
judgment as to the correct rank ordering of image quality from poor to best and
therefore image quality cannot be placed on an absolute scale. Visual
psychophysical investigations have not measured all the properties relevant to
imaging systems.

Many formulas exist for predicting image quality. Each is appropriate under
a particular set of viewing conditions. These expressions are typically obtained
from empirical data in which multiple observers view many images with a
known amount of degradation. The observers rank the imagery from worst to
best and then an equation is derived which relates the ranking scale to the
amount of degradation.

If the only metric for image quality was resolution, then we would attempt
to maximize resolution in our system design. Many tests have provided insight
into image quality metrics. In general, images with higher MTFs and less noise
are judged as having better image quality. There is no single ideal MTF shape
that provides best image quality. For example, Kusaka 1 showed that the MTF
that produced the most aesthetically pleasing images depended on the scene
content.

315
316 CCD ARRAYS, CAMERAS, and DISPLAYS

The metrics suggested by Granger and Cupery, Shade, and Barten offer· 6
additional insight on how to optimize an imaging system. Granger and Cupery
developed the Subjective Quality Factor (SQF): an empirically derived
relationship using individuals' responses when viewing many photographs. Shade
used photographs and included high quality TV images. Barten's approach is
more comprehensive in that it includes a variety of display parameters. It now
includes 7 contrast, luminance, viewing ratio, number of scan lines, and noise.

While Barten has incorporated the sampling effects of flat panel displays,?
no model includes the sampling (and associated aliasing) that takes place at the
detector. Until this aliasing is quantified, no metric will predict image quality
fqr sampled data systems. Only an end-to-end system approach can infer overall
image quality.

There are potentially two different system design requirements: (1) good
image quality and (2) performing a specific task. Sometimes these are
equivalent, other times they are not. All image quality metrics incorporate some
form of the system MTF. The underlying assumption is that the image spectrum
is limited by the system MTF. Equivalently, it is assumed that the scene
contains all spatial frequencies and that the displayed image is limited by system
MTF - a reasonable assumption for general imagery. While good image quality
is always desired, a military system is designed to detect and recognize specific
targets (discussed in Chapter 12, Minimum Resolvable Contrast). Optimized
military systems will have high MTF at the (assumed) target frequencies and
other spatial frequencies are considered less important. Computer monitors are
usually designed for alphanumeric legibility.

Often, the display is the limiting factor in terms of image quality and
resolution. No matter how good the electronic tmaging system is, if the display
resolution is poor, then the overall system resolution is poor. If the system is
optics-limited, then a high resolution display does not offer any "extra" system
resolution. A high resolution display just ensures that all the information
available is displayed. System resolution may be limited by the HVS. If the
observer is too far from the screen, not all of the image detail can be discerned.

The HVS appears to operate as a tuned spatial-temporal filter where the

tuning varies according to the task at hand. Because the HVS approximates an
optimum filter, no system performance improvement is expected by precisely
matching the image spectrum to the HVS preferred spectrum. Rather, if the
displayed spectrum is within the limits of the eye spectrum, the HVS will
automatically tune to the image. Clearly, the overall system magnification should
be set such that the frequency of the maximum interest coincides with the peak
 IMAGE QUALITY 317

frequency of the HVS' s contrast sensitivity. Although this implies a specific

frequency, the range of optimization is broad.

A large number of metrics are related to image quality. Most are based on
monochrome imagery such as resolution, MTF, and minimum resolvable
contrast. CRT resolution was presented in Section 7.5., Resolution (page 218).
Color reproduction and tonal transfer issues, which are very important to color
cameras, are not covered here.

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

11.1. RESOLUTION METRICS

An overwhelming majority of image quality discussions center on resolution.

Resolution has been in use so long that it is thought to be fundamental and that
it uniquely determines system performance. There are four different types of
resolution: (1) temporal, which is the ability to separate events in time; (2) gray
scale, which is determined by the analog-to-digital converter design, noise floor;
or the monitor capability, (3) spectral; and (4) spatial. An imaging system
operating at 30 Hz frame rate has a temporal resolution of 1130 sec. Gray-scale
resolution is a measure of the dynamic range. The spectral resolution is simply
the spectral band pass (e.g., UV, visible, or near infrared) ofthe system. This
section covers spatial resolution.

Resolution provides valuable information regarding the finest spatial detail

that can be discerned. A large variety of resolution measures exist8 and the
various definitions may not be interchangeable. An electronic imaging system
is composed of many subsystems and each has its own metric for resolution
(Table 11-1). These can also be divided into analog (Table 11-2) and sampled-
data resolution metrics (Table 11-3).

A single measure of spatial resolution cannot be satisfactorily used to

compare all sensor systems. Spatial resolution does not provide information
about total imaging capability or contrast sensitivity. The minimum resolvable
contrast simultaneously provides contrast sensitivity information and resolution
information for monochrome systems. Resolution does not include the effects of
system noise and is not related to sensitivity.
318 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 11-1
SUBSYSTEM MEASURES of RESOLUTION
SUBSYSTEM RESOLUTION METRIC

Rayleigh criterion
Optics Airy disk diameter
Blur diameter

Detector-angular-subtense
In 1tantaneous-field-of-view
Detectors Effective-instantaneous-field-of-view
Detector pitch
Pixel-angular-subtense

Electronics Bandwidth

Electronic imaging system Limiting resolution

(MTF approach) Nyquist frequency

Displays rv limiting resolution

Table 11-2
RESOLUTION MEASURES for ANALOG SYSTEMS

RESOLUTION DESCRIPTION
Ability to distinguish two
Rayleigh criterion
adjacent point sources

Diffraction-limited diameter
Airy disk
produced by a point source

Actual minimum diameter

Blur diameter
produced by a point source

Limiting Spatial frequency at which MTF= 0.02 to

Resolution 0.10

TV limiting Number of resolved lines per picture

resolution height

The smallest test target

Ground resolved distance (1 cycle) that a photo interpreter can
distinguish

An estimate of the limiting feature size

Ground resolution
seen b~ a photointerpreter

Smallest region that contains unique

Resel
information
 IMAGE QUALITY 319

Table 11-3
RESOLUTION MEASURES for SAMPLED DATA SYSTEMS
RESOLUTION DESCRIPTION

Detector angular subtense Angle subtended by one detector element

Instantaneous field-of-view Angular region over which the detector senses radiation

Nyquist frequency One-half of the sampling frequency

Detector pitch Center-to-center spacing

Effective-instantaneous-field- One-half of the reciprocal of the object-space spatial

of-view frequency at which the MTF is 0.5

Number of detector elements or number of digital data

Pixels, datels, and disels
points

When resolution is defined as the inverse of the system cutoff in angular

space (typical for military applications), the back-of-the-envelope approximation
provides the range at which the target can be "detected":
target size (11-1)
Range ~ --"'---
resolution

This is the maximum range at which a periodic target can be faithfully

reproduced.

Analog measures of spatial resolution may be determined by: (1) the width
of a point source image, (2) the minimum detectable separation of two point
sources, (3) the spatial frequency at which the MTF drops to a defined level, or
(4) the smallest detail resolved by an observer. These measures assume that the
system output image is a replica of the input scene (a linear-shift-invariant
system).

As imaging systems are incorporated into new disciplines, resolution must

be specified in terms used by those industries. For example, photointerpreters
use the ground resolved distance (GRD) when evaluating reconnaissance imagery
captured on film. But as solid state imaging systems replace the wet film
process, the GRD, by default, becomes an imaging system resolution measure.

When viewing aerial imagery that contains a test pattern (such as the U.S.
Air Force 1951 standard 3-bar target), an image analyst determines the smallest
discernible cycle on the ground. This cycle width (bar plus space) is the GRD.
It includes the system MTF and possible degradation by the atmosphere and is
320 CCD ARRAYS, CAMERAS, and DISPLAYS

a function of altitude. The GRD is related 9 •10 to the 10-point Imagery

Interpretability Rating Scale (IIRS). The National Imagery Interpretability Rating
Scale (NIIRS) has replaced the IIRS. 11 • 14

The smallest elements that can be created by a digital subsystem are the
pixel, datel, and disel. These are created by the detector array, ADC, and the
display media, respectively. The resel is the smallest element that can be created
by an analog system. The optics, analog electronics, and video standard (or
transmission link) each have their own resel. System resolution is a convolution
of all -els.
II II

Complex systems cannot be characterized by a single number (for example,

an ~~-el ). Nevertheless, this chapter discusses various measures because they
11

have been in existence so long. They were developed to describe the resolution
of specific subsystems and do not provide an end-to-end performance metric.
Nor do they include aliasing. Aliasing effects are discussed in Reference 15.

11.2. OPTICAL RESOLUTION METRICS

Table 11-4 provides various optical resolutions. As analog metrics, these are
various definitions of a resel. Diffraction measures include the Rayleigh criterion
and the Airy disk diameter. The Airy disk is the bright center of the diffraction
pattern produced by an ideal optical system. In the focal plane of the lens, the
Airy disk diameter is

A.
dAIRY - 2.44-.fl - 2.44A.F . (11-2)
Do

The Rayleigh criterion is a measure of the ability to distinguish two closely

spaced objects when the objects are point sources. Optical aberrations and focus
limitations increase the diffraction-limited spot diameter to the blur diameter.
Optical designers use ray tracing programs to calculate the blur diameter. The
blur diameter size is dependent on how it is specified (i.e., the fraction of
encircled energy 16). den Dekker and van den Bos provide 17 an in-depth review
of optical resolution metrics.

Considerable literature 18 has been written on the image forming capability

of lens systems. Image quality metrics include aberrations, Strehl ratio, and blur
or spot diagrams. These metrics, in one form or another, compare the actual
blur diagram to diffraction-limited spot size. As the blur diameter increases,
image quality decreases and edges become fuzzy. The ability to see this
 IMAGE QUALITY 321

degradation requires a high resolution sensor. Both the human eye and
photographic film visually have this resolution. However, with electronic
imaging systems, the detectors are often too large to see the degradation because
the detectors are often larger than the blur diameter.

Table 11-4
OPTICAL RESOLUTION METRICS (in image space)
DEFINITION
RESOLUTION DESCRIPTION
(usual units)

Ability to distinguish two XRES = 1.22 A F

Rayleigh criterion
adjacent point sources (Calculated: nun)

Diffraction-limited diameter XRES = 2.44 A F

Airy disk
produced by a point source (Calculated: mm)

Actual minimum diameter Calculated from ray tracing

Blur diameter
produced by a point source (nun)

11.3. DETECTOR RESOLUTION

Considering the Airy disk, the wavelength dependence suggests that the
resolution increases as the wavelength decreases (smaller is better). However,
this is only true when the detector size is much smaller than the Airy disk size.
For most electronic imaging systems the reverse is true. The detector is the
same size or larger than the disk diameter (see Section 10.4., Optics-Detector
Subsystem, page 286). This means that the Airy disk is not adequately sampled
and therefore the resolution afforded by the optics cannot be exploited. Systems
designed for star detection are usually optics-limited (optical resel larger than a
pixel), whereas for general imagery the systems are usually detector-limited
(pixel larger than the optical resel)

Detector arrays are often specified by the number of pixels and detector
pitch. These are not meaningful until an optical system is placed in front of the
array. Table 11-5 provides the most common resolution metrics expressed in
object-space units. The detector-angular-subtense (DAS) is often used by the
military to describe the resolution of systems when the detector is the limiting
subsystem. If the detector's element horizontal and vertical dimensions are
different, then the DAS in the two directions is different. Note that the PAS is
different than the DAS (see Section 1.5., Pixels, Datels, Disels, and Resels,
page 14). Only with 100% fill factor arrays are they the same. Spatial sampling
rates are determined by the pitch in image space or the PAS in object space.
322 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 11-5
DETECTOR ELEMENT RESOLUTION MEASURES (in object space)
DEFINITION
RESOLUTION DESCRIPTION
(usual units)

Detector-angular- Angle subtended by one detector d/fl

subtense element (Calculated: mrad)

Instantaneous-field-of- Angular region over which the Measured width at 50%

view detector senses radiat ton amplitude (mrad)

Pixel-angular-subtense Angle subtended by one pixel dcclfl

(Calculated: mrad)

One-half of the reciprocal of the

Effective-instantaneous- Measured or calculated
object-space spatial frequency at
field-of-view (mrad)
which the MTF is 0 5

Ground sampled Projection of detector pitch onto dccRffl

distance the ground (calculated: inches)

One-half of the angle subtended dccf2fl

Nyquist frequency
by detector pitch (calculated: mrad)

If resolution is defined by the DAS, then a vanishingly small detector is

desired. Simultaneously, the f-number must approach zero so the system is not
optics-limited (see Figure 10-15, page 287). Although any detector size, optical
aperture, and focal length can be chosen to select the resolution limit, the same
parameters affect the sensitivity (see Equation 10-27, page 288). With practical
optical systems, the minimum f-number is about 1.0 with a theoretical limit of
0.5. Resolution and sensitivity factors are coupled into a single figure of merit:
the minimum resolvable contrast (discussed in Chapter 12, Minimum Resolvable
Contrast).

The instantaneous-field-of-view (IFOV) is the linear angle from which a

single detector element senses radiation. It is a summary measure that includes
both the optical and detector responses. The IFOV is typically a measured
quantity. Here, a point source transverses the detector element and the detector
output is graphed as a function of angle. The IFOV is the full-width, one-half
maximum amplitude of the resultant signal. For detector-limited systems (large
d/FA), the IFOV is approximately equal to the DAS. Equivalently, if the optical
blur diameter is small compared to the DAS, then they are approximately equal.
 IMAGE QUALITY 323

The detector horizontal cutoff is lion = fl/dH = 1/DAS in object space. With
detector arrays, signal fidelity is limited to the Nyquist frequency. Here, the
effective-instantaneous-field-of-view (EIFOV) offers an alternate measure of
resolution (Figure 11-1). The value uEIFov is the spatial frequency at which the
MTF is 0.5. Calling 1/DAS the cutoff provides a misleading representation.
Recall that the detector can respond to frequencies above 1/DAS (see Figure 8-
9, page 241).

0.8 Nyquist
frequency
LL 0.6
1-
::E 0.4
0.2
0~----~------P-----~--~--P-----~
0 0.1 0.2 0.3 0.4 0.5
llt:Jmv
NORMALIZED SPATIAL FREQUENCY

Figure 11-1. Definition of effective-instantaneous-field-of-view for an

undersampled system. The spatial frequency has been normalized to the
Nyquist frequency, fN. EIFOV = 11(2 uEIFov).

11.4. ELECTRICAL RESOLUTION METRIC

For high data rate systems such as line-scanners, the electronic bandwidth
may limit system response. For electronic circuits, resolution is implied by its
bandwidth. The minimum pulse width is approximately

..
1
N-- (11-3)
MINIMUM 2 BW ,

where rMINIMUM is measured in seconds and BW is measured in Hertz. The 3-dB

point (half-power frequency) is often called the bandwidth. With this minimum
value the output signal looks like the input. Other expressions (e.g., rMINIMUM =
1/BW) simply imply there is a measurable output without any inference about
signal fidelity.
324 CCD ARRAYS, CAMERAS, and DISPLAYS

11.5. MTF-BASED RESOLUTION

Usually systems are designed to have high MTF, and Figure 10-14 (page
286) illustrated the MTF at Nyquist frequency. However, the value of Nyquist
frequency is a measure of resolution; 100% fill factor staring arrays have the
highest Nyquist frequency.

11.5.1. LIMITING RESOLUTION

Other measures also exist such as the limiting resolution. This is the spatial
frequency at which the MTF is, say, 5%. Figure 11-2 illustrates two systems
that have the same limiting resolution. Which system is selected depends on the
specific application. System A is better at high spatial frequencies, and system
B is better at low spatial frequencies. Equivalently, if the scene contains mostly
low frequencies (large objects), then system B should be used. If edge detection
is important (high spatial frequencies), then system A should be considered. This
clearly indicates the difficulty encountered when using resolution exclusively as
a measure of system performance.

URES

Spatial Frequency

Figure 11-2. The MTFs of two different imaging systems with the same
limiting resolution, uREs·
 IMAGE QUALITY 325

11.5.2. MTF/TM RESOLUTION

Early resolution measures were based on wet-film camera characteristics.

The intersection of the system MTF and film threshold modulation (TM)
provided a measure of camera system performance (Figure 11-3). TM is the
minimum modulation required at a given spatial frequency to discern a
difference. The value uREs is the limiting spatial frequency or intersection
frequency.

w
en
z
0
a.
en
w
0::
w
>
~w
0::

SPATIAL FREQUENCY
Figure 11-3. The spatial frequency of the intersection of the film
threshold modulation and MTFsYSTEM is a measure of resolution.

Conceptually, for spatial frequencies above uRES• the film requires more
modulation than that provided by the system. Below uREs the film can easily
discern the scene modulation. Because many measures were in support of aerial
imagery, the intersection was also called the aerial image modulation (AIM).
With the advent of electronic imaging systems, the film TM was replaced with
the HVS's threshold modulation (discussed in Section 11.7., MTFA).

11.5.3. SHADE'S EQUIVALENT RESOLUTION

Shade3 equated apparent image sharpness of a television picture to an

equivalent pass band:

N. - j[MTF(/)]2 df. (11-4)

0
326 CCD ARRAYS, CAMERAS, and DISPLAYS

As reported by Lloyd, 19 Sendall modified Shade's equivalent resolution such

that
1
R - -- - --------
EQ 2N (11-5)
• 2 f[MTFSYS(u)] 2 du
0

The value REQ cannot be directly measured and is a mathematical construct that
expresses overall performance. As REQ decreases, the resolution "improves"
(smaller is better). As an approximation, the system resolution, REQ-sYs• may be
estimated from the component equivalent resolutions, R;. by

..; 2 (11-6)
REQ-SYS .. R1 + R22 + ... + R,.2 •

Shade's approach using the square of the MTF emphasized those spatial
frequencies at which the MTF is relatively high. It appears to be a good measure
for classical systems in which the MTF is decreasing (such as a Gaussian
distribution). The equivalent resolution approach assumes that the system is
completely analog and it ignores sampling effects. Therefore, REQ becomes a
resel. It probably should not be used (nor should any other image quality metric)
to compare systems with significantly different MTFs. Nor should it be used
with systems that can create significant aliasing.

As a summary metric, REQ-svs provides a better indication of system

performance than just a single metric such as the DAS. REQ-SYs probably should
be used in Equation 11-1 (page 319) to obtain a more realistic measure of range
performance. The value REQ cannot be evaluated in closed form for all
subsystems such as boost, defocused optics, or Chebyshev filters. For these
subsystems, Equation 11-6 must be evaluated numerically. If a subsystem MTF
is essentially unity over the spatial frequency region of interest, then that
subsystem REQ can be ignored (e.g., REQ = 0). Table 11-6 provides the
equivalent resolution for several common MTFs.
 IMAGE QUALITY 327

Table 11-6
ONE-DIMENSIONAL EQUIVALENT RESOLUTIONS
SUBSYSTEM MTF REQ

2(
-cos
11
-I(- - -u, 1-( ~J)
"t ) 1.845AF
Optics "tc ";c

Rectangular sinc(d u;) d

detector

r
1 - - whenN- 1
1tf.3dB

N1h-order
low-pass filter
1 (f.
+ f.3dB
1
- - whenN- co
2f.3dB

CRT-based
e -2x a~POTu;
2
2{it OSPOT
display

If the electronics provides adequate bandwidth, it is instructive to see the

relationship between the Airy disk diameter and the detector size. Using the
same parameters as Figure 10-14 (page 286), x = d/(FA.), for a system with
resolution limited by the optics/detector combination,

REQ-SYS - d ( 1.~45r + 1 .
(11-7)
328 CCD ARRAYS, CAMERAS, and DISPLAYS

The equation is graphed in Figure 11-4. As x increases, REQ-svs approaches d.

For small values of x, the system becomes optics-limited and the equivalent
resolution increases. When x = 2.44, REQ-svs = 1.25d and when x = 4, REQ-svs
= 1.10d. Matching the Airy disk to the detector size creates an equivalent
resolution 25% larger than the actual detector size.
optics detector
2 limited limited

1-z
ffiQ
....li- d
e(::;) 1
~cl
::;)(/)
Ow
wa:

0~----~------~------~------~----~
0 2 3 4 5
d!F>.
Figure 11-4. Equivalent resolution normalized to d as a function of
d/F)\. The vertical line indicates where the Airy disk diameter is equal
to the detector size. The best resolution occurs when d/F)\ approaches
infinity. Here the optics MTF approaches unity.

11.6. SYSTEM RESOLUTION EXAMPLES

Example 11-1
PIXELS and DATELS

An imaging system consists of 512 x 512 detector elements. Assume that the
fill factor is 100% and the camera's analog output is digitized at either 1024
samples/line or 2048 samples/line. What is the resolution?

The external analog-to-digital converter creates either two or four datels for
each pixel. In both cases, the pixel size remains the same. Increasing the
sampling rate does not change the system's resolution but improves the
repeatability of all measurements by reducing phasing effects that might occur
in the analog-to-digital converter. Any image processing algorithm that operates
on this higher number must consider the sensor resolution.
 IMAGE QUALITY 329

Cameras with a standard video output may not have the resolution suggested
by the video standard. For example, a camera containing an array of 320 x 240
pixels may have its output formatted into EIA 170 timing. The signal may be
digitized by a frame grabber that creates 640 x 480 datels. The image size is
still 320 x 240 pixels.

===============================================~

Example 11-2
DISELS and PIXELS

A staring array consists of 512 x 512 detectors. Using 4 x electronic zoom

it is presented on a digital monitor that displays 1024 x 1024 disels. What is the
system resolution?

Each pixel is mapped onto four datels that are then mapped one-to-one onto
disels. Here, the imaging system determines the system resolution, not the
number of datels or disels. High quality monitors only ensure that the image
quality is not degraded. Electronic zoom cannot increase resolution. But
decimation or minifying may reduce resolution. Because image processing is
performed on datels, the image analyst must be made aware of the system
resolution.

============================================~

Example 11-3
SYSTEM EQUIVALENT RESOLUTION

A CCD camera has an analog output digitized by a frame grabber. What is

the apparent system resolution for image processing and what is the resolution
displayed on a CRT-based monitor?

This CCD camera contains 480 horizontal detectors. Each detector is 15 ~tm
on 15 Jtm centers (100% fill factor). The optical aperture is 1 inch and the focal
length is 5.6 inches (F = 5.6). The analog electronics is wideband and it does
not significantly affect the resolution. Camera output is in the standard EIA 170
format with a bandwidth of 4.2 MHz and tvmEO-LINE = 52.45 JtS. The frame
grabber provides 640 horizontal samples and a standard consumer CRT-based
monitor is used.
330 CCD ARRAYS, CAMERAS, and DISPLAYS

Assuming A.AvE ""0.5/Lm, REQ for the optics is 1.845A.AvEID0 = 0.0363 mrad.
The DAS is d/fl or 0.105 mrad. Here, d/(FA.) = aD 0 /A. = 5.33. That is, the
optical/detector subsystem is detector-limited. The HFOV is

Nydy 6
480 (15 x 10 ~~-
m)
HFOV- - 50.6 mrad . ( 11 _8)
.fl 3
(5.6 in ) ( 25.4 x 1o- : )

With an active line time of 52.45/Ls, the video bandwidth in object space is
6
/,
BW _ 52.45x10- 4 _ xl06 _ 435 cycles .
tVIDEO-UNE
2 (11-9)
o-VIDEO - HFOV 50.6 X 10" 3 mrad

Assuming a sharp cutoff, REQ "" 112f3ds = 0.115 mrad. The system equivalent
resolution is

REQ-srs- Jo.0363 2 +0.1052 +0.li52- 0.160mrad. (11-10)

Here, the EIA 170 bandwidth significantly affects the resolution. Neglecting
phasing effects, each frame grabber datel represents approximately
(0 .159)(480)/640 = 0.119 mrad. These are overlapping resolution elements
where the center-to-center spacing is (0.105)(480)/640 = 0.0787 mrad.

The CRT equivalent resolution is

REQ - 2{it a 0 - 2{rt ( dec · ) - 0.227 mrad . (11-11)

1.645-jl

The display significantly degrades the resolution. But this is necessary to

suppress the raster and sampling artifacts. Each disel represents overlapping
resolution elements with apparent size

REQ-srs - Jo.1592 + 0.2272 ·• 0.277 mrad . (11-12)

Numerical integration of Equation 11-5 provides REQ-svs = 0.263. For this

example, Equations 11-5 and 11-6 provided similar results.
=============================================~
 IMAGE QUALITY 331

11.7. MTFA

The area bounded by the system MTF and the HVS's modulation threshold
(M1) is called the modulation transfer function area (MTFA) (Figure 11-Sa). As
the MTFA increases, the perceived image quality appears to increase. The
spatial frequency at which the MTF and M 1 intersect is a measure of the overall
resolution. However, the resolution is not unique because M1 depends on the
display viewing distance. The HVS's inhibitory response (see Figure 10-31,
page 309) is omitted from the MTFA approach. Mathematically,
fo

MTFA - f [MTF(f) - M,(f)] df . (11-13)

0

where f0 is the limiting spatial frequency or intersection frequency. The value

M 1 is also called the aerial image modulation and demand modulation function.
It is the minimum modulation required at a given spatial frequency to discern
a difference.

According to Snyder, 20 the MTF A appears to correspond well with

performance in military detection tasks where the targets are embedded in noise.
The noise elevates the HVS's modulation threshold so that the MTFA decreases
with increasing noise (Figure 11-Sb).

For low noise, general imagery, the HVS's contrast sensitivity has little
effect on the MTFA. The integrand in Equation 11-13 can be replaced with just
the system MTF. Here, the high spatial frequencies are weighted too heavily
compared to the low spatial frequencies and Shade's approach appears better.

From an analyst's viewpoint, the convolution theorem states that MTFs

should be multiplied. The contrast sensitivity is proportional to the HVS MTF.
Therefore, the system MTF should be multiplied by the contrast sensitivity or
divided by the inverse (modulation threshold). However, the intersection
frequency can be used as a measure of limiting resolution of the system. It can
be used, for example, to calculate the GRD.
332 CCD ARRAYS, CAMERAS, and DISPLAYS

Modulation
Threshold

(a)

High Noise
Modulation
Threshold

Low Noise
Modulation
Threshold

(b)

Figure 11-5. MTFA. The area between system MTF and M1 up to the
intersection is a measure of image quality. The spatial frequency at the
intersection is a measure of limiting resolution. The area and limiting
resolution depend on the noise level within the imagery. (a) Low noise
and (b) elevated noise.

11.8. SUBJECTIVE QUALITY FACTOR

According to Granger and Cupery, 2 the spatial frequency range important to

image quality is in the region from just below the peak to three times the peak
sensitivity of the eye. The Campbell-Robson high luminance data, and the HVS
models of Nill, Schultz, and delong and Bakker support Granger and Cupery's
approach whereas the Komfeld-Lawson HVS model does not (see Section 10.8.,
Human Visual Response, page 306). Recall that the Kornfeld-Lawson model is
considered only applicable for spatial frequencies greater than fPEAK·
 IMAGE QUALITY 333

In logarithmic units, the normalized subjective quality factor (SQF) is:

log(l2)

SQF - K J MTF5YS(jeye) d[log(jeye)] , (11-14)

log(3)

where K is a normalization constant. In the SQ F approach, only those

frequencies that are very important to the eye are included. The spatial
frequency presented to the eye depends on the image size on the display, the
distance to the display, and electronic zoom. Table 11-7 provides Granger and
Cupery's interpretation of the SQF. These results are based on many observers
viewing noiseless photographs with known MTF degradation.

The SQF is an adequate image quality metric for low noise imagery when
the illumination is fixed at a moderate level. As with the MTFA and Shade's
equivalent pass band, the SQF is intended for general imagery only. It cannot
be used for specific applications such as the legibility of alphanumeric
characters.

Table 11-7
SUBJECTIVE QUALITY FACTOR
SQF SUBJECTIVE IMAGE QUALITY

0.92 Excellent
0.80 Good
0.75 Acceptable
0.50 Unsatisfactory
0.25 Unusable
334 CCD ARRAYS, CAMERAS, and DISPLAYS

11.9. SQUARE-ROOT INTEGRAL

Barten4-7 introduced the square-root integral (SQRI) as a measure of image

quality_ The SQRI is

The multiplication overcomes the theoretical objection raised with the MTFA
where the CSF was subtracted from the MTF. MTFsvs includes all the
subsystem MTFs up to and including the display. Barten modeled the contrast
sensitivity function as

CSFI~ ) - - 1- - af. e -bf,.,. V1 + cebf.,. (11-16)

Veye M eye '
t

where

(11-17a)

100 )0.15 (11-17b)

b- 0.3 1 + - - '
( LAVE

and

c - 0.06 ' (ll-17c)

where feye has units of cycles/degree, LAvE is the average luminance in candelas
per square meter, and w is the angular display size as seen by the observer. The
factor l/ln(2) allows SQRI to be expressed in just-noticeable-difference (JND)
units. Both average luminance level and display size affect the CSF. This CSF
is a good approximation over five orders of magnitude of luminance levels and
for angular display sizes ranging from 0.5° to 60°.
 IMAGE QUALITY 335

This model includes the effects of various display parameters such as

viewing ratio, luminance, resolution, contrast, addressability, and noise. Display
viewing ratio and average luminance are incorporated in the HVS's CSF (via w
and LAvE in the CSF equation). Display resolution (display MTF) is included in
MTFsys·

Contrast loss due to reflected ambient lighting effectively reduces the display
MTF. If .::lL is the additional luminance due to reflected ambient-light, then the
modulation depth is reduced by the factor:

LAVE (11-18)
T) DISPlAY -
LAVE + 11L

This factor is the same for all spatial frequencies of interest and the perceived
effect on image quality is incorporated into the SQRI by multiplying MTFSYS by
V'YIDISPLA Y •
The display parameters dictate the limits of integration. The value fMIN is that
spatial frequency that provides one-half cycle across the display and fMAx is the
maximum spatial frequency of the display and this is related to display
addressability (see Section 7.6., Addressability, page 222). For example, if the
number of active lines is NAcnvE• then

(Kellfactor) NACTIVE
(11-19)
!MAx - DFov ·
2

The small angle approximation was used to calculate field-of-view, DFov•

subtended by the observer. It is the display height, HMoNITOR• divided by the
viewing distance, D.

Noise increases the contrast sensitivity function. Because the SQRI must
decrease with noise, Barten added noise as an additional factor to the HVS
MTF:

(11-20)
336 CCD ARRAYS, CAMERAS, and DISPLAYS

The value Mn can be calculated from the noise power spectral density presented
to the eye. 5 Barten predicted the quality of imagery when both static (pattern
noise) and dynamic (temporally varying) noise are present. He considered both
one-dimensional and two-dimensional noise sources. His approach encompasses
actual sensor operation where different noise components may exist in the
horizontal and vertical directions. As such, he considered some of the three-
dimensional noise components used for military applications (discussed in
Section 12.2., Three-Dimensional Noise Model).

11.10. REFERENCES
1. H. Kusaka, "Consideration of Vision and Picture Quality - Psychological Effects Induced by
Picture Sharpness," in Hu11Uln Vision, Visual Processin!?. and Digital Display, B. E. Rogowitz,
ed., SPIE Proceedings Vol. 1077, pp. 50-55 (1989).
2. E. M. Granger and K. N. Cupery, "An Optical Merit Function (SQF) which Correlates With
Subjective Image Judgments," Photographic Science and Engineering, Vol. 16, pp. 221-230
(1972).
3. 0. H. Shade, Sr., "Image Gradation, Graininess, and Sharpness in Television and Motion
Picture Systems," published in four parts in SMPTE Journal: "Part I: Image Structure and
Transfer Characteristics," Vol. 56(2), pp. 137-171 (1951); "Part II: The Grain Structure of
Motion Pictures - An Analysis of Deviations and Fluctuations of the Sample Number," Vol.
58(2), pp. 181-222 (1952); "Part III: The Grain Structure of Television Images," Vol. 61(2),
pp. 97-164 (1953); "Part IV: Image Analysis in Photographic and Television Systems," Vol.
64(11), pp. 593-617 (1955).
4. P. G. Barten, "Evaluation of Subjective Image Quality with the Square-root Integral Method,"
Journal of the Optical Society of America. A, Vol. 17( 10), pp. 2024-2031 (1990).
5. P. G. Barten, "Evaluation of the Effect of Noise on Subjective Image Quality," in Hu11Uln
Vision, Visual Processing, and Digital Display II, J. P. Allenbach, M. H. Brill, and B. E.
Rogowitz, eds., SPIE Proceedings Vol. 1453, pp. 2-15 (1991).
6. P. G. Barten, "Physical Model for the Contrast Sensitivity of the Human Eye," in Hu11Uln
Vision, Visual Processing, and Digital Display III, B. E. Rogowitz, SPIE Proceedings Vol.
1666, pp. 57-72 (1992).
7. P. G. Barten presents short courses at numerous symposia. See, for example, "Display Image
Quality Evaluation," Application Seminar Notes, SID International Symposium held in Orlando,
FL (May 1995). Published by the Society for Information Display, Santa Ana, CA, or "MTF,
CSF, and SQRI for Image Quality," IS&T/SPIE's Symposium on Electronic Imaging: Science
and Technology, San Jose, CA (February 1995).
8. G. C. Holst, Electro-Optical 111Ulging System Performance, pp. 218-228, JCD Publishing,
Winter Park, FL (1996).
9. Air Standardization Agreement: "Minimum Ground Object Sizes for Imaging Interpretation,"
Air Standardization Co-ordinating Committee report AIR STD 101111 (31 December 1976).
10. Air Standardization Agreement: "Imagery Interpretabiltty Rating Scale," Air Standardization
Co-ordinating Committee report AIR STD 101111 (10 July 1978).
11. K. Riehl and L. Maver, "A Comparison of Two Common Aerial Reconnaissance Image Quality
Measures," in Airborne Reconnaissance XX, W. G. Fishell, A. A. Andraitis, A. C. Crane, Jr.,
and M. S. Fagan, eds., SPIE Proceedings Vol. 2829, pp. 242-254 (1996).
12. J. M. Irvine, "National Imagery Interpretability Rating Scales (NIIRS): Overview and
Methodology," in Airborne Reconnaissance XXI, W. C. Fishell, ed., SPIE Proceedings Vol.
3128, pp. 93-103 (1997).
 IMAGE QUALITY 337
13. G. C. Holst, Sampling, Aliasing, and Data Fidelity, pp. 311-317, JCD Publishing, Winter
Park, FL (1998).
14. J. C. Leachtenauer, W. Malila, J. Irvine, L. Colburn, and N. Salvaggio, General Image-Quality
Equation: GIQE," Applied Optics, Vol. 36(32), pp. 8322-8328 (1997).
15. G. C. Holst, Sampling, Aliasing, and Data Fidelity, pp. 308-311, JCD Publishing, Winter
Park, FL (1998).
16. L. M. Beyer, S. H. Cobb, and L. C. Clune, "Ensquared Power for Obscured Circular Pupils
with Off-Center Imaging," Applied Optics, Vol. 30(25), pp. 3569-3574 (1991).
17. A. J. den Dekker and A. van den Bos, "Resolution: A Survey," Journal of the Optical Society
A, Vol. 14(3), pp. 547-557 (1997).
18. See, for example, W. J. Smith, Modern Optical Engineering, 2"" Edition, McGraw-Hill, New
York (1990).
19. J. M. Lloyd, Thermal Imaging, p. 109, Plenum Press, New York (1975).
20. H. L. Snyder, "Image Quality and Observer Performance," in Perception of Displayed
Information, L. M. Biberman, ed., pp. 87-118, Plenum Press, New York, NY (1973).
12
MINIMUM RESOLVABLE CONTRAST
Most image quality metrics were developed for high contrast imagery. Here,
image quality is closely linked with the MTF. High MTF cameras provide better
image quality and therefore provide better resolution under optimum conditions.
While most resolution metrics are applicable to high contrast targets, the relative
rating of systems may change for low contrast scenes. A high MTF system with
excessive noise will not permit detection of low contrast targets.

The military is interested in detecting and recognizing targets at the greatest

possible range. This is achieved by increasing the system gain until noise is
apparent and the imagery is "snowy." Because the distinction between target and
background is measured by contrast, the signal threshold is expressed as the
minimum resolvable contrast (MRC). MRC applies to monochrome systems
only. It is a measure of an observer's ability to detect a 3-bar target embedded
in noise. The Johnson criterion relates real targets to equivalent bar chart
targets. Thus, the link between real targets and the MRC is the spatial frequency
of the equivalent 3-bar target.

By providing an angular resolution, it is simple to determine the range: R =

(target size)/(resolution). Therefore, for MRC calculations, object-space spatial
frequency is used. The MTFs provided in Chapter 10 were specified in the
image-space spatial frequency domain. The conversion is U00 = flu; and vob = fl
V;, where fl is the effective optical focal length.

In the 1950s, Shade 1 modeled the resolution of photographic film and TV

sensors as a function of light level. His approach is the framework of all models
used today. In the 1970s, Rosell and Willsorr applied Shade's results to thermal
imaging systems and low light level televisions. Since then, most modeling
efforts have concentrated on infrared imaging systems. However, the same
modeling equations apply to all imaging systems. Different forms exist to
accommodate the terminology associated with different technologies (scanning
versus staring, visible versus thermal imaging systems, etc.).

MRC was originally developed for vidicons and low light level televisions.
The approach taken here extends the thermal imaging system performance model
(popularly called FLIR92) methodology 3 to visible systems. FLIR92 calculates
the minimum resolvable temperature (MRTl where the target-background

338
 MRC 339

radiance difference is characterized by a differential temperature. FLIR92

methodology and the three-dimensional noise model4 are reformatted for solid
state cameras. The 3-D noise model was developed to describe the various noise
sources present in thermal imaging systems. These same noises exist in lesser
amounts within solid state cameras.

The MRC approach uses terminology that characterizes human perceptual

capability. The observer does not see the same contrast at the display as that
presented by the target to the camera. Both the displayed contrast and brightness
can be adjusted on the display so the observed contrast may be significantly
different from what is present at the entrance aperture of the imaging system.
Observer threshold is expressed as the perceived signal-to-noise ratio rather than
the perceived contrast.

Within the real world, there is a probability associated with every parameter.
The target contrast is not one number but a range of values that follow a diurnal
cycle. The atmospheric transmittance is not fixed but can change in minutes.
There appear to be an overwhelming set of combinations and permutations.
Therefore, only a few representative target contrasts and a few representative
atmospheric conditions are selected and the performance range is calculated for
these conditions.

The Nyquist frequency limits the highest frequency that can be reproduced.
The specific manner that the Nyquist frequency is included in the model greatly
affects range performance. This is a result of the assumptions used and is not
necessarily representative of actual system performance. While no model is
perfect, models are still excellent for comparative analyses.

The symbols used in this book are summarized in the Symbol List (page
xviii) which appears after the Table of Contents.

12.1. THE OBSERVER

The original Night Vision Laboratory static performance model 5 and FLIR92
use the Komfeld-Lawson eye model (see Section 10.8., Human Visual Response,
page 306). These models were validated by the U.S. Army when detecting tank-
sized targets at modest ranges. Insufficient data exist to say with certainty which
of the eye models available is best. Because of this and other model
uncertainties, range predictions cannot be placed on an absolute scale. All
analyses, no matter which eye model is used, must be used only for comparative
performance purposes.
340 CCD ARRAYS, CAMERAS, and DISPLAYS

12.1.1. PERCEIVED SIGNAL-TO-NOISE RATIO

The human visual system "filters" enhance the displayed SNR. The
perceived signal-to-noise ratio, referred back to the array output, is
MTF lln 1
SNR - k SYS pe • (12-1)
P (nSYS) (eye spatial filter) (eye temporalfilter)

MTFsys includes the camera, electronics, display, and HVS MTFs (see Chapter
10, System MTF, page 267). All calculations are performed with object-space
units (e.g., cycles/mrad). The value .::1{\e is the difference in the number of
photoelectrons generated by the target and its immediate background. The value
k depends on the optical aperture diameter, focal length, detector spectral
quantum efficiency, etc.

This approach differs from the camera signal-to-noise ratio presented in

Section 6.1.2. (page 186). There, the target was considered very large so the
system MTF did not affect the signal strength. The rationale for using the MTF
here, is that observer metrics (i.e., MRC) are based on the visibility of high
frequency bar targets.

Although the HVS's modulation threshold is reported as a J-shaped curve

(see Figure 10-30, page 307), the actual shape depends on the noise power
spectral density. 6-s If the noise is restricted to certain spatial frequencies, then
the detection of targets of comparable spatial frequencies becomes more
difficult. The observer's ability to see a specific spatial frequency target depends
on the noise content in the neighborhood of that spatial frequency (Figure 12-1).
Low spatial frequency noise components will interfere with detecting low
frequency targets (large objects). Mid-spatial frequency noise increases the
modulation threshold curve at mid-frequencies and so on. These noise factors
are included in the MRC model via the three-dimensional noise model. As such,
they are not included in the HVS MTF.

For nearly all imaging systems, square waves (bar targets) are used for
system characterization. Using the CTFsys (See Section 9.5., Contrast Transfer
Function, page 263), the perceived signal-to-noise ratio is

SNR - k' CTFsys llnpe (12-2)

P (nsrs)(eye spatial filter)(eye temporal filter) '

where k' is a constant that includes k and the conversion from MTF to CTF.
 MRC 341

100 low frequency

noise
c
..J
0 10
::r:
II)
w
0:::
::r: high frequency
1-
z noise
0
i= 0.1
~
::J
c 0.01
0
:Iii no noise

0.001 +-----+------+----+-----+
0.1 10 100 1000

SPATIAL FREQUENCY (cy/mrad)

Figure 12-1. Effects of spectral noise on the HVS's modulation

threshold. The MRC should approximately follow the shape of the
modulation threshold curve.

Selecting a threshold value that is required to just perceive a target and

inverting Equation 12-2 provide a minimum detectable target to background
difference:

(nSYS)(eyespatial.filter)(eyetemporal.filter) . ( _)
~nMimMUM-SNRTil~~-------------------------- 12 3
k1 CTFSYS

The system noise may consist of many components and the HVS may
integrate each component differently. The function (3i is the "filter" that
interprets noise source (I\):

v(n~) 131 + .... + (n!)13 m (12-4)

!:>.nMINIMUM = SNRTil ...L..I--'-___:=-,------'--'---=-
k CTFSYS

or

(12-5)
342 CCD ARRAYS, CAMERAS, and DISPLAYS

A test bar target is approximated by a square wave of infinite extent. If the

system is band-limited, then only the fundamental of the square wave has
sufficient amplitude to contribute to the response (valid for high frequencies).
The fundamental's amplitude is 4hr times the square wave amplitude. The HVS
is sensitive to the average value of the first harmonic and the average value of
a half-cycle sine wave is 2hr. Therefore, the conversion from a square wave
(CTFSYS) to a sinusoid (MTFsvs) requires a factor of 8/r when viewed by an
observer:

(12-6)

Following FLIR92 terminology, 3 (n1} is the shot noise. The other noise terms
represent all other noise sources. The three-dimensional noise model quantifies
each (n?}.

In Section 6.2.1., Camera SNR (page 186), a target SNR was calculated.
The background light level was assumed to be zero so that the photon shot noise
was associated with the target only. For target detection, the target exists on a
positive background. Both the target and the background regions provide photons
that the camera converts to photoelectrons. Let ~-T and ~-s be the number of
photoelectrons associated with the target and background, respectively. There
is shot noise associated with both. The "average" shot noise variance is

2\ ( 2 \ npe-T + npe-B (12-7)

( nt; - nsHor; "' - - - -- - ·
2

A variety of definition..; exist for contrast. The contrast, as defined in the

original MRC derivation, is the target luminance minus the background
luminance divided by the background luminance. Because both the target and
background are assumed to be illuminated by the same source (sun, moon, etc.),
the contrast depends on reflectances:

(12-8)
 MRC 343

Historically, the target was always considered darker than the background.
However, the target can be brighter than the background and there does not
appear to be any difference between the detectability of objects that are of
negative or positive contrast.

Substituting Equations 12-7 and 12-8 into Equation 12-6 and calling the
minimum detectable contrast the MRC, provides

2
1t MRC+2
MRC- SNR - - - -
TH 8 kMTFsYS 2npe-B

Because MRC appears on both sides of the equation, it can be solved either
iteratively or as a quadratic equation. The MRC is a family of curves dependent
on the background level (Figure 12-2). MRC approaches a maximum value of
unity as the system MTF approaches zero. Thus, MRC is linked to MTF but the
resolution (i.e., the spatial frequency where the contrast approaches one)
depends on the background.

0.8

0 0.6
0:::
::!!i 0.4
increasing
0.2 light level

0~~~====~~--------~--~
0 0.2 0.4 0.6 0.8

RELATIVE SPATIAL FREQUENCY

Figure 12-2. MRC is a family of curves that depends on the background

light level.
344 CCD ARRAYS, CAMERAS, and DISPLAYS

12.1.2. PERCEIVED RESOLUTION

Perceived resolution depends on the size of the target, ambient lighting, and
target-to-background contrast. These issues were not considered when discussing
display resolution because they are relatively constant.

For signals embedded in noise, the situation is different. The HVS is then
very sensitive to the signal-to-noise ratio. Slight variations affect target visibility.
As illustrated in Figure 12-3, the signal increases faster than the accompanying
photon shot noise. Given a threshold of perceptibility, as the SNR increases,
more detail can be seen because more signals rise above the threshold. Thus,
camera "resolution" depends on background light level. This "resolution" is
strictly due to the HVS's response and is not related to hardware performance.
In contrast, hardware resolution is typically specified as that spatial frequency
where the hardware MTF is, say, 10% and is measured under relatively high
signal-to-noise ratio conditions.

9 increasing
0:: 8 signal
~ 7
c s~-­
w 5
~ 4 r----/
w threshold
~ ~,-----~~--~~~~~~----------
~ ~1-----~~-----+--~======~~~~~--
0 0.2 0.4 0.6 0.8 1
NORMALIZED SPATIAL FREQUENCY
Figure 12-3. As the target signal increases, the SNR increases. For a
fixed threshold, increasing the signal appears to increase the resolution.

Figure 12-4 illustrates the perceived resolution as a function of light level.

The apparent resolution increases with light level. Because the SNR is
proportional to contrast, the "resolution" is also a function of contrast. High
contrast targets produce higher resolution. Again, this is strictly due to the
HVS's response and is not related to hardware limitations.
 MRC 345

10 100
RELATIVE INTENSITY

Figure 12-4. Perceived resolution as a function of light level.

12.1.3. THE HVS "FILTER"

The HVS system is probably the most difficult system to model. Although
many models exist, the two most often employed are the matched filter and the
synchronous integrator model. With the matched filter, the HVS acts like a
tuned circuit. Here, the perceived SNR is affected only by spatial frequencies
that are near the target's spatial frequency. All other spatial frequencies are
ignored. This is not a filter in the usual sense because the signal and noise are
"filtered" by it. Rather, visual psychophysical data suggest that the HVS acts as
if it were a filter that can be described mathematically by a filter function. With
the synchronous integrator model, the HVS integrates over an angular region
defined by the target edges.

Scott and D' Agostino state9 The most frequently encountered

methods ... describe the eye/brain spatial integration using either a matched filter
or a synchronous integrator model ... they yield virtually identical ... predictions
. . . Any potential difference . . . will be lost in the inherent error of the ....
measurements. For these reasons, and because ... a synchronous integrator is
somewhat simpler .... , FLIR92 has been written as a synchronous integrator
model. Russell and Willson2 used the synchronous integrator model for television
systems and their approach is presented in this chapter.
346 CCD ARRAYS, CAMERAS, and DISPLAYS

12.2. THREE-DIMENSIONAL NOISE MODEL

The three-dimensional noise model4 provides the basic framework for

analyzing various noise sources. It was developed to describe the noise in
thermal imaging systems. However, the methodology also applies to solid state
cameras.

The noise is divided into a set of eight components that relate temporal and
spatial noise to a three-dimensional coordinate system (Figure 12-5). This
approach allows full characterization of all noise sources including random
noise, pattern noise, streaks, rain, 1/f noise, and any other artifact. Analyzing
the noise in this manner has the advantage of simplifying the understanding of
complex phenomena by breaking it down into a manageable set of components.
The method simplifies the incorporation of complex noise factors into model
formulations.

H , , ," , ,
, ,,
2

v ,
,
m
,,
-
n
Figure 12-5. Three-dimensional noise model coordinate system
illustrating data set NTVH·

The T-dimension is the temporal dimension representing the video framing

sequence. The other two dimensions (horizontal and vertical) provide spatial
information. For a staring array, n and m indicate detector element locations in
the horizontal and vertical directions, respectively.

Table 12-1 describes the three-dimensional noise components. The subscripts

describe the noise "direction." The value uTVH represents the rms noise of the
three-dimensional data set m x n XN. The value uvH is the rms noise after
averaging in the T-direction. Its data set contains m x n elements. And so on.
 MRC 347

Table 12-1
THREE-DIMENSIONAL NOISE DESCRIPTORS
3-D NOISE
DESCRIPTION
COMPONENT

(TTVH Random 3-D noise

Spatial noise that does not change from frame-to-
UyH
frame

Variations in column averages that change from

(TTH
frame-to-frame (rain)

Variations in row averages that change from

(TTY
frame-to-frame (streaking)

Variations in row averages that are fixed in time

Uy
(horizontal lines or bands)

Variations in column averages that are fixed in

(TH
time (vertical lines)

(TT Frame-to-frame intensity variations (flicker)

Depending on the system design and operation, any one of these noise
components can dominate. The origins of these components are significantly
different and the existence and manifestation depend on the specific design of the
imaging system. Not all of the noise components may be present in every
imaging system.

Assuming the noise components are independent, the total system noise is

(12-10)
348 CCD ARRAYS, CAMERAS, and DISPLAYS

Independence was a fundamental assumption during the development of the

three-dimensional noise model. For nearly all systems, O"T is considered
negligible compared to O"TvH and therefore is omitted from the data set. The
value O"TVH consists of the shot noise and the noise floor. Pattern noise is
incorporated though O"vH• av. and aH. Currently, only aTvH is predicted and the
remaining noise components must be determined from measurements or
estimates. Figures 12-6 through 12-8, generated by the System Image Analyzer, 10
illustrate the effect of these noise sources. Figure 12-6 is the basic image. In
Figure 12-7, FPN, PRNU, and random noise appear similar in a single frame.
Random noise changes from frame-to-frame whereas pattern noise does not.
Figure 12-8 illustrates how aTV or av affect the visibility of horizontal bars.
Scanning systems often exhibit this type of noise. Pixels will always have
different responsivities. When the row-average responsivities are different in
TDI sensors, banding occurs. 11

Figure 12-6. Ideal image with asvs = 0. Generated by System Image

Analyzer software. 10
 MRC 349

Figure 12-7. Image with random, FPN, or PRNU noise.

Figure 12-8. Image with dominant horizontal banding (high uTV or CTy).
350 CCD ARRAYS, CAMERAS, and DISPLAYS

While the three-dimensional noise model was created to describe noise

sources in thermal imaging systems, many of the same noises exist in all
imaging systems. The FLIR92 default values should apply to all imaging
systems built to similar designs.

With all systems, the most prevalent noise source is assumed to be temporal
noise (uTVH). For staring arrays, the next most important noise source is <TvH·
Table 12-2 lists the FLIR92 recommended 3 values for PtSi staring arrays
operating in the 3 to 5/Lm range. Other systems using different detectors may
perform differently. These values were obtained empirically.

For scanning systems, temporal row noise (uTV) and fixed row noise (uv) are
the important contributors. Table 12-3 lists the FLIR92 default values
appropriate for scanning systems with HgCdTe detectors operating in the 8 to
12/Lm region. These values apply to all linear arrays (vertically oriented) with
horizontal scanning.

Table 12-2
DEFAULT VALUES for STARING SYSTEMS
RELATIVE NOISE DEFAULT VALUE

<1vHf<1TvH 0.40

<1TV/<1TVH 0

<Tyf<TTVH 0

<1TH/<1TVH 0

<1H/<1TVH 0

Table 12-3
DEFAULT VALUES for SCANNING SYSTEMS
RELATIVE NOISE DEFAULT VALUES

<1yH/<7TVH 0

<1TV/<1TVH 0.25

<Tyf<TTVH 0.25

<Tmf<TTVH 0

<1Hf<1TVH 0
 MRC 351

12.3. THE MRC MODEL

The MRC is a measure of an observer's ability to detect a signal embedded

in noise. The MRC model is a so-called static model in that the target is
stationary and no search is required (or at least the observer knows where to
look). The observer has an unlimited amount of time for target discrimination.
As with most models, the system is assumed to be linear-shift-invariant.

This section applies the FLIR92 methodology to solid state camera

performance. The fundamental difference between the MRC and MRT is the
method by which the target is characterized. In the infrared spectral region, the
target-to- background differential signal is specified by a temperature difference.
In the visible spectral region, the differential signal is specified by the contrast.
The MRT derivation can be found in the literature. 5•12- 15

Following FLIR92 methodology, the horizontal and vertical MRCs are

1t2 1 MRC + 2 K ( )
MRC tu ) - SNR ----- H Uob
(12-11)
11' ob TH 8 kMTFJu 0 b) 2npe-B

and

The functions KH(U00) and Kv(V00) are summary noise factors that combine the
three-dimensional noise model components with the HVS's spatial and temporal
integration capabilities. The variable MTFH(u00) is the product of all the
horizontal MTFs and MTFv(V00) is the product of all the vertical MTFs (see
Chapter 10, System MTF, page 267). In the original development of the MRC
model, only the HVS spatial and temporal integration factors were considered.
Later models included the HVS MTF.
352 CCD ARRAYS, CAMERAS, and DISPLAYS

The summary noise factors are

(12-13)

and

(12-14)

For staring systems, the default (see Table 12-2) MRCs become

MRCJu ~- 1t2 SNRTH ~ MRC +2 E E (E + __±_) (12-15)

0
8 kMTFJu 0~ 2npe-B v H T 10

and

MRC SNRm ~MRC~2~EE(E

v<vob)-B kMTF ( ) ~-
1t2
v H T+ 10
4)· (12-16)
V Vob pe-B

For scanning systems (applies to linear arrays) the default (see Table 12-3)
MRCs are

MRC tu ) - ~ SNRTH JMRC + 2 .fEE E (12-17)

]{\ ob 8 kMTF fu )
H' ob
2n
pe-B
v T v H

and

E V (E'r"H +_!).
1i' +_!E
4
T
4
(12-18)
 MRC 353

The values Er, EH, and Ev, respectively, represent the temporal, horizontal
spatial and vertical spatial integration afforded by the HVS interpreter. The
model assumes that the HVS temporally integrates perfectly and continuously
over the HVS integration time. The noise is considered uncorrelated from frame-
to-frame so that it adds in quadrature. When the information update rate, FR, is
high enough, and if there is signal correlation between adjacent frames:

(12-19)

A 5: 1 aspect ratio 3-bar target (TASPECT = 5) is used to measure the MRC,

whereas a 7:1 aspect ratio 4-bar target is typically used for MRT measurements.
The bars are oriented vertically for horizontal measurements and horizontally for
vertical measurements. Thus, the equations used depend on the bar orientation.
A vertical and horizontal MRC exists for each bar orientation. Table 12-4 lists
the nomenclature. Before the two-dimensional MRC approach, only EH-H was
used for horizontal MRC measurements. Similarly, only Ev-v was used for
vertical MRC measurements. The additional components, Ev-H and EH-V• are used
in the two-dimensional MRC (discussed in Section 12.5., Two-dimensional
MRC).

Table 12-4
MRC EQUATION NOMENCLATURE

MODEL
MEASUREMENT BAR ORIENTATION
NOMENCLATURE

Horizontal MRC: EH-H

Horizontal MRC Vertical
Vertical MRC: Ev-H

Horizontal MRC: EH-v

Vertical MRC Horizontal
Vertical MRC: Ev-v

The noise filters and noise power spectrum were described in Section 6.2.1.,
Camera SNR (page 186). The noise spectrum is modified by the electronic filters
that occur after the insertion of noise. Because noise is assumed to originate in
the detector, MTFoPTics• MTFMonoN• and MTFoETECTOR are not part of the noise
filters. As appropriate, the remaining MTFs provided in Chapter 10, System
MTF (page 267), are components of the noise filters. For equation brevity, the
horizontal and vertical noise filters are listed as HNF-H(~b) and HNF-v(vob),
respectively. As appropriate, f., fv, and ud are expressed in equivalent object-
space units (u0 b). Because electrical signals are typically read out in a serial data
354 CCD ARRAYS, CAMERAS, and DISPLAYS

stream (consistent with video standard timing), the noise appears in the
horizontal direction only. The variable vobis the transposed variable vd and also
the vertical digital filter response if present.

For horizontal MRC, the bars are oriented vertically and the bar aspect
ratio, TAsPECT• appears in the vertical summary noise factors. For a bar target
with fundamental spatial frequency U 0 , the summary noise factors are

and

E fu )
V-II'- o
=
jl
~
dCCVJIH
0
fv )12 sinc2
NF-V' ob
(T
ASPECTVob
2 uo
l
dv
ob
. (12-21)

For the vertical MRC, the bar aspect ratio is in the horizontal summary
noise factor:

and

(12-23)

In the early literature, only the horizontal MRC was calculated and only
random temporal noise was modeled. With no other noise sources,

MRC ru ) - ~ SNRTH JMRC +2 IE E E (12-24)

II'- ob 8 k MTFII'-fu ob ) 2n
pe-B
v v H T'

where Er. EH, and Ev are given by Equations 12-19, 12-20, and 12-21,
respectively.
 MRC 355

If MTFs are near unity and the noise is white [S(u00) :::: 1] over the spatial
frequencies of interest, then

(12-25)

and

(12-26)

Then

1t2 SNRTH
MRC(u,)-- ----==--
8 kMTF(u,)

The original SNR formulation 16 contained an (a/A) 112 factor where "a" is the
target area at the image plane and "A" is the effective area of the photosensitive
device. The MRC is the inverse of the SNR and therefore should contain
(A/a) 112 • This is identical to the last square-root term in Equation 12-27. "A" is
the photosensitive area that contributes to noise. With vidicons, the "A"
represented the entire image surface. With solid state arrays, it is the effective
pixel size, dccHdccvlfl2 • The target size is (1!u )(TAsPEcr/uo). Equation 12-27
0

reduces to the basic MRC equation used by most researchers 17 •18 prior to the
introduction of the three-dimensional noise model.

12.4. SNRTH and ~

The FLIR92 documentation19 states: SNRm and the eye integration time, t,,
are most often the model parameters used to "tune" an MRTD prediction to a
set of measurements. These values are treated as constants that are adjusted so
that the predicted values match laboratory data.

The 1975 NVL model documentation20 states: ... unfortunately, universal

values for these constants do not exist. The values recommended at this time are
SNRm = 2.25 and t. = 0.2.

FLIR92 documentation continues 19 The luminances associated with about a

0.1 second eye integration time agrees well with the display experiments
356 CCD ARRAYS, CAMERAS, and DISPLAJ'S

conducted in I 988 . . . (darkened room and optimal viewing) . . . In higher

ambient light levels conditions, such as may be encountered with fielded systems,
greater display luminances can be expected and thus a faster eye integration
time may be appropriate ... NVESD recommends setting SNRm equal to 2.5 ...
Though psychophysical data show that SNRm is a function of target frequency,
a value of 2.5 represents a reasonable average

The value te is somewhat nebulous and its value depends on the specific task
at hand and ambient light level. FLIR92 recommends changing te according to
the available light level (Figure 12-9).

4
~~
...,
~~
.......
~
b--.....
0
-5 -4 -3 -2 -1 0 2 3
LOG BRIGHTNESS
Figure 12-9. HVS integration values as a function of light level. FLIR92
documentation recommends using te = 0. I s for laboratory MRT
predictions. The same technique can be applied to MRC.

While many arguments exist for selecting the "correct" values, SNRTH and
te are simply constants. Both SNRTH and te have a range of acceptable values.
No matter what value is selected for t0 , consistency throughout the model must
be maintained. The value te is also used in the linear motion MTF (see Section
10.5.1., Linear Motion, page 292).

A word of clarification is necessary here. SNRTH is approximately the signal-

to-noise value that the HVS perceives the image to be at threshold. It includes,
in part, both temporal and spatial integration effects. It is not the signal-to-noise
ratio of the video signal nor the SNR of the luminance on the display.
Depending on the noise characteristics and the target characteristics, the HVS
can detect2 1 signals with video SNR as low as 0.05. Figure 12-10 provides an
example of the HVS spatial integration capability.
 MRC 357

(a) (b)

(c) (d)

Figure 12-10. Variations in the signal-to-noise ratio, (a) SNR = 2, (b)

SNR = 1, (c) SNR = 0.75, and (d) SNR = 0.5. The imagery was
computer generated with the SNRs listed. The tonal transfer
characteristics of the laser printer (to create the hard copy) and halftone
(to print the image) may have affected the final SNR. The actual SNRs
in this printed copy have not been measured. These images should be
considered as representative of the ability to perceive low SNR imagery.
358 CCD ARRAYS, CAMERAS, and DISPLAYS

The ability to perceive targets embedded in noise depends on the target

spatial extent with respect to the noise spatial frequencies. Large bars are easier
to perceive compared to small targets that "disappear" into the noise. Figure 12-
10 only illustrates the HVS's spatial integration capabilities. With live video, the
random noise is changing from frame to frame and the HVS's temporal
integration provides an additional enhancement of FRt•. For EIA 170 timing, this
is about a factor of three to six.

12.5. TWO-DIMENSIONAL MRC

Although FLIR92 is called two-dimensional, it is a two-directional model.

That is, the threshold is predicted along two orthogonal axes, taken as the
vertical and horizontal directions. With unequal MTFs, the predicted MRC will
be different in the two directions. The two directional values are brought
together with a two-dimensional spatial frequency that approximates the response
of an equivalent radially symmetric system.

As with the MRTs, the two-dimensional MRC is created by taking the

geometric average of the horizontal and vertical MRC frequencies at each
contrast value (Figure 12-11):
(12-28)

In this manner, the two-dimensional MRC is forced to approach asymptotically

the mean values of the vertical and horizontal cutoff frequencies. The two-
dimensional MRC is a mathematical construct that is used for range performance
predictions. It cannot be measured because f20 does not exist.

0.8

0 0.6
0::
:::E 0.4
0.2
oL---~~~~--L_ ________
v1 f 20 u1

RELATIVE FREQUENCY

Figure 12-11. The two-dimensional MRC is mathematically created from

the horizontal and vertical MRCs. It is really a two-directional MRC.
 MRC 359

12.6. RANGE PREDICTIONS

Performance modeling assumes that the target is a single spatial frequency.

This is an approximation because real targets consist of a complex superposition
of spatial frequencies. Targets are actually a continuum of spatial frequencies
from zero on up. The system MTF limits the target spatial frequencies that can
be displayed. This section provides an overview of the range prediction
methodology. A detailed discussion of system range performance is available. 22

The Johnson criterion provides the link between the target angular subtense
and the spatial frequency scale on the MRC graph. The target's inherent
contrast, C0 , is modified by the atmospheric conditions (attenuation and path
radiance). When the target contrast is reduced to the MRC curve, the
performance range is obtained. Depending on the Johnson criterion selected, this
becomes the detection, recognition, or identification range.

12. 6.1. CONTRAST TRANSMITTANCE

Intuitively, it seems that any phenomena that prevent us from visually

perceiving an object will affect all imagery the same way. During a hazy day,
light scattered into the HVS (path radiance) reduces the visual contrast. Here,
objects that are far away appear as a neutral white and we are unable to
distinguish any features. While the HVS acts in this manner, imaging systems
do not. Imaging systems simply respond to radiance differences.

However, real systems have automatic gain circuitry that may be activated
by the path radiance signal. If so, the effect is to reduce system gain. Path
radiance will partially fill the charge wells.

For most applications, the target and background are illuminated by the same
source (sun, moon, etc.). The apparent contrast, CR, is the inherent contrast
modified by the atmosphere. Dropping the wavelength notation for equation
brevity,

C _ !J.L_
-aA,..,R[ ]L
e p T - p B SCENE _ C - - - -
0 L -a.,.,R L L o L (12-29)
e pB SCENE + A 1M 1+ A 1M e a A1JIR
p bLSCENE

where LscENE is the ambient illumination that irradiates the target and
background.
360 CCD ARRAYS, CAMERAS, and DISPLAYS

It is the sky, sun, or moon modified by the intervening atmospheric

transmittance. The value aATM is the atmospheric scattering cross section and
exp(-aATMR) is the atmospheric transmittance. The path radiance, LATM• is due
to light scattered into the line-of-sight and the total amount received is an
integral over the path length. 23 When only scattering is present (a reasonable
approximation for the visible spectral band), LATM =
[1-exp(-aATMR)]I...sKv· The
value LsKv depends on the viewing direction and the location of the sun. The
value p 8 LscENE is the background luminance. For convenience, the LsKviPsLscENE
is called the sky-to-ground ratio (SGR):

1
c- c ·-- (12-30)
R o1 + SGR(eaATMR- 1)

The SGR is approximately 0.2/p 8 for a clear day (LsKv/LscENE 0.2) and 1/p8 =
for an overcast day (LsKviLscENE =
1). When the SGR is one, the received
contrast is simply the inherent contrast reduced by the atmospheric
transmittance:

(12-31)

Meteorological range is defined quantitatively by the Koschmieder formula2 4 :

RVIS = _1_ (-1-) '

(J A7M
1n
CTH
(12-32)

where CTH is the modulation threshold at which 50% of the observers would see
the target. The use of the scattering cross section in this definition rather than
the extinction coefficient implies that absorption of particles at visual
wavelengths is small enough to ignore. This view is probably justified except in
cases of polluted air. Koschmieder set CTH to 0.02 and evaluated aATM at "A=
0.555 ttm.Then

_ 3.912 R
'1' • TA7M e-a,~.ruR - e RYJs (12-33)
~ ransmtttance - -
 MRC 361

Table 12-51ists aATM as a function of standard weather conditions. However,

aATM decreases with increasing wavelength. For visibilities less than 6 km, aATM
can be approximated by
113
<J Ar,Jf..) ., -3.912)
- (-0.55
- )0.585(RVIs) (12-34)
( RVIS A

For much greater visibilities,

(12-35)

Table 12-5
INTERNATIONAL VISIBILITY CODE

SCATTERING
DESIGNATION VISIBILITY
COEFFICIENT' (TATM
Dense fog 0-50 m > 78.2km- 1

Thick fog 50-200m 19.6-78.2km-l

Moderate fog 200-500 m 7.82-19.6km- 1

Light fog 500-1 km 3.92-7.82 km- 1

Thin fog 1-2 km 1.96-3.92 km- 1

Haze 2-4 km 0.978 -1.96km- 1

Light haze 4-lOkm 0.391-0.978km-l

Clear 10-20 km 0.196-0.391 km- 1

Very clear 20-50 km 0.0782 -0.196km- 1

Exceptionally clear >50 km < 0.0782km- 1

362 CCD ARRAYS, CAMERAS, and DISPLAYS

12.6.2. JOHNSON CRITERIA

The Johnson25 methodology has become known as the equivalent bar pattern
approach. Observers viewed military targets through image intensifiers and were
asked to detect, decide the orientation, recognize, and identify the targets. Air
Force tri-bar charts with bars the same contrast as the scaled models were also
viewed and the maximum resolvable bar pattern frequency was determined. The
number of bars per critical object dimension was increased until the bars could
just be individually resolved. In this way, detectability was correlated with the
sensor's threshold bar pattern resolution (Table 12-6). These results became the
foundation for the discrimination methodology used today. Critical dimension
refers to the target dimension that must be discerned for detection, recognition,
or identification. In the early literature, the minimum dimension was called the
critical dimension.

Table 12-6
JOHNSON'S RESlfLTS
(From Reference 25)
CYCLES ACROSS
DISCRIMINATION
MEANING CRITICAL TARGET
LEVEL
DIMENSION

An object is present
Detection 1.0 ± 0.025
(object versus noise).

The object is approximately

symmetrical or unsymmetrical
Orientation and its orientation may be 1.4 ± 0.35
discerned (side view versus
front view).

The class to which the object

Recognition belongs (e.g., tank, tnJck, 4.0 ± 0.80
man).

The object is discerned with

sufficient clarity to specify the
Identification 6.4 ± 1.50
type (e.g., T-52 tank, fnendly
jeep).

Although Johnson used 6.4 cycles across the critical target dimension for
identification, studies with thermal imaging systems at the Night Vision
Laboratory suggested that eight cycles. were more appropriate for identification.
Orientation is a less popular discrimination level. Because today's standards are
based on Johnson's work, they are labeled as the Johnson criteria though they
are not the precise values recommended by him.
 MRC 363

The original Johnson methodology was validated using common module

thermal imaging systems. Those systems typically had a 2:1 difference in
horizontal to vertical resolution. The vertical resolution was limited by the
detector pitch Nyquist frequency. For validation purposes, military vehicles were
used and the critical dimension tended to be in the vertical direction although the
predictions were based on horizontal performance. Converting to a two-
dimension model requires removing the directional bias imposed by the one-
dimensional model. This is achieved by reducing all values by 25%, or
equivalently, multiplyin!f6 all discrimination values by 0.75. This adjustment
then provides the same range performance predictions whether predicted by the
one-dimensional or two-dimensional model. Table 12-7 provides the commonly
used discrimination levels. By convention, each 50% probability is labeled as
N50 (e.g., in one-dimension, N50 is 1, 4, or 8 cycles for detection, recognition,
or identification, respectively).

Table 12-7
COMMON DISCRIMINATION LEVELS
One-dimension Two-dimension
TASK DESCRIPTION
Nso N,o

The blob has a reasonable

Detection probability of being an object 1.0 0.75
being sought.

Object discerned with sufficient

Classical
clarity that its specific class can be 4.0 3.0
recognition
differentiated.

Object discerned with sufficient

Identification clarity to specify the type within 8.0 6.0
the class.

12.6.3. TARGET TRANSFER PROBABILITY FUNCTION

The results of threshold experiments, such as Johnson's, provide an

approximate measure of the 50% probability of discrimination level. Results of
several field tests 20 provided the cumulative probability of discrimination or
target transfer probability function (TTPF) (Table 12-8). The TTPF can be used
for all discrimination tasks by simply multiplying the 50% probability of
performing the task by the TTPF multiplier. For example, from Tables 12-7
and 12-8, the probability of 95% recognition is 2N50 = 2(4) = 8 cycles across
the target's critical dimension.
364 CCD ARRAYS, CAMERAS, and DISPLAYS

Table 12-8
DISCRIMINATION CUMULATIVE PROBABILITY

PROBABILITY of
MULTIPLIER
DISCRIMINATION

1.00 3.0

0.95 2.0

0.80 1.5

0.50 1.0

0.30 0.75

0.10 0.50

0.02 0.25

0 0

An empirical fit 27 to the data provides

P(N) = (12-36)

where

E = 2.7 + 0.7 (_!/-)

Nso
. (12-37)

Figure 12-12 illustrates the full TTPF for the three levels of discrimination
as identified by Johnson. Because of the variability in the population, some
people can only detect the target, others will recognize it, and still a smaller
portion will identify the target. This leads to the wide variations seen in field test
results.
 MRC 365

~
..J
0.8

iii 0.6
<C
m 0.4
0
0:: 0.2
ll..

5 10 15

NUMBER OF CYCLES
(a)

1
~ 0.8
::J
iii 0.6
<C
lXI 0.4

~
ll..
0.2

2 4 6 8 10

NUMBER OF CYCLES
(b)

Figure 12-12. Target transfer probability function for detection,

recognition, and identification. (a) one-dimensional and (b) two-
dimensional. The curves overlap so that when only a small portion of the
population can identify the target, a larger portion will recognize the
target and an even larger portion will detect the target.

12.6.4. RANGE PREDICTION METHODOLOGY

The MRC abscissa is converted into a range scale using a target

discrimination value. With the two-dimensional model, the critical dimension,
he, is the square root of the target area, ATARGET· For rectangular targets, he =
(HW)'h where H and Ware the target's height and width, respectively.
The number of equivalent cycles across the target is

N _ JATARGET f = he f (12-38)
T R 2D R 2D"
366 CCD ARRAYS, CAMERAS, and DISPLAYS

Consider a target that is 2 X 8 m and six cycles are required across the critical
dimension. Then the conversion from spatial frequency to range is

fH
R- - . 4 ~ (12-39)
6 f i2D - --
6 J2D '

where six cycles represent 95% probability of recogmt10n for the two-
dimensional model. The intersection of the MRC curve and the target apparent
contrast, CR (using Equation 12-31, page 360), versus range curve is the range
at which the target can be discerned according to the discrimination level
selected (Figure 12-13).

0.1
0
0:::
:! 0.01

0.001
0 2 4 6 8
SPATIAL FREQUENCY (cy/mrad)
(a)

0.1
0
0:::
:! 0.01

0.001
0 2 3 4 5 6
RANGE (km)
(b)

Figure 12-13. Graphical method to obtain the recognition range. (a)

Two-dimensional MRC and (b) conversion of spatial frequency into
range. The values N50 = 3, h.,= 4 m, and C0 = 0.5. The value CR is a
straight line when plotted in semi-logarithmic coordinates. The visibility
is 6.2 km. The two-dimensional 50% probability recognition range is
approximately 2.5 km.
 MRC 367

The target transfer probability function is used to predict the probability of

range performance. Here, a range is selected and CR is calculated. This value
intersects the MRC curve at what is called the critical frequency. Multiplying
the angular subtense by the critical frequency provides the number of cycles
across the target. Referring to the TTPF (see Figure 12-12), the probability is
determined for that particular range. Then a new range is selected and the
process is repeated until the entire probability function is determined (Figure 12-
14).

apparent target
contrast

0.1
... j .. ..
T
H
0.01
l_
w --1
f20
SPATIAL FREQUENCY

TTPF

p •.......• p

0.5 0.5

OL---------------~-- o~~----~----------
R Nr
RANGE NUMBER OF CYCLES

Figure 12-14. Methodology to determine range performance probability.

368 CCD ARRAYS, CAMERAS, and DISPLAYS

12.6.5. SAMPLING EFFECTS

In Chapter 8, Sampling Theory, page 231, it was stated that the highest
frequency that could be faithfully reproduced is the Nyquist frequency. Solid
state cameras are undersampled systems and are limited to a spatial frequency
dictated by the detector center-to-center spacing. In the development of the MRC
equation, sampling effects were ignored.

Because no frequency can exist above the Nyquist frequency, many

researchers represent the MTF as zero above the Nyquist frequency. This forces
the MRC to approach one at the Nyquist frequency (Figure 12-15). When the
probability of detection is calculated using the target transfer probability
function, the Nyquist limit reduces the range (Figure 12-16).

This range reduction is an artifact of the model. It is real when trying to

discriminate sinusoids or bar targets. Real targets, however, are composed of
many spatial frequencies ranging from zero and on up. Target features that have
spatial frequencies above Nyquist will be distorted. No data exist on the
detection or recognition of distorted imagery and it is, therefore, unwise at this
juncture to extend the MRC model past the Nyquist frequency. The analyst must
recognize that the Nyquist frequency limit may have an overly severe impact on
target discrimination range. The specific manner in which the Nyquist frequency
is included in the model has a dramatic effect on range performance. This is a
result of the assumptions used and is not necessarily representative of actual
system performance.

Nyquist limit

0.8

~ 0.6
:::2 0.4 ',

0 0.25 0.5 0.75

RELATIVE SPATIAL FREQUENCY

Figure 12-15. Representative MRC for a staring array sensor. At

Nyquist frequency, the MRC is assumed to be one. The abrupt increase
at Nyquist frequency is used by some researchers.
 MRC 369

~ 0.8
:J
iii 0.6
<C
m 0.4
Nyquist
0
0::: 0.2 limit
ll.
0
0 0.5 1.5 2
RELATIVE RANGE (km)

Figure 12-16. Probability of detection versus range for a staring array.

The shapes of the curves depend on the MRC, contrast, atmospheric
transmittance, and N50 • Because the MTF is forced to zero at the
Nyquist frequency, the range hits a brick wall at Nyquist frequency.
This is an artifact created by the model assumptions. Targets can be seen
at longer ranges but aliasing will distort edges.

12.7. REFERENCES
1. 0. H. Shade, Sr., "Image Gradation, Graininess, and Sharpness in Television and Motion
Picture Systems," published in four parts in SMPTE Journal: "Part I: Image Structure and
Transfer Characteristics," Vol. 56(2), pp. 137-171 (1951); "Part II: The Grain Structure of
Motion Pictures - An Analysis of Deviations and Fluctuations of the Sample Number," Vol.
58(2), pp. 181-222 (1952); "Part III: The Grain Structure of Television Images," Vol. 61(2),
pp. 97-164 (1953); "Part IV: Image Analysis in Photographic and Television Systems," Vol.
64(11), pp. 593-617 (1952).
2. F. A. Rosell and R. H. Willson, "Performance Synthesis of Electro-Optical Sensors," Air Force
Avionics Laboratory Report AFAL-TR-72-229, Wright Patterson AFB, OH (1972).
3. FUR92 Thermal Imaging Systems Performance Model, User's Guide, NVESD document
UG5008993, Fort Belvoir, VA (1993).
4. J. D' Agostino and C. Webb, "3-D Analysis Framework and Measurement Methodology for
Imaging System Noise," in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing
II, G. C. Holst, ed., SPIE Proceedings Vol. 1488, pp. 110-121 (1991).
5. W. R. Lawson and J. A. Ratches, "The Night Vision Laboratory Static Performance Model
Based on the Matched Filter Concept," in The Fundamentals of Thermal Imaging Systems, F.
Rosell and G. Harvey, eds., NRL Report 8311, pp. 159-179, Naval Research Laboratory,
Washington, D.C. (1979).
6. S. Daly, "Application of a Noise Adaptive Contrast Sensitivity Function in Image Data
Compression," Optical Engineering, Vol. 29(8), pp. 977-987 (1990).
7. H. Pollehn and H. Roehrig, "Effect of Noise on the Modulation Transfer Function of the Visual
Channel," Journal of the Optical Society of America, Vol. 60, pp. 842-848 (1970).
8. A. Van Meeteren and J. M. Valeton, "Effects of Pictorial Noise Interfering With Visual
Detection," Journal of the Optical Society of America A, Vol. 5(3), pp. 438-444 (1988).
370 CCD ARRAYS, CAMERAS, and DISPLAYS

9. L. Scott and J. D' Agostino, "NVEOD FLIR92 Thermal Imaging Systems Performance Model,"
in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing III, G. C. Holst, ed.,
SPIE Proceedings Vol. 1689, pp. 194-203 (1992).
10. System Image Analyzer software is available from JCD Publishing, 2932 Cove Trail, Winter
Park, FL 32789.
11. T. S. Lomheim and L. S. Kalman, "Analytical Modelmg and Digital Simulation of Scanning
Charge-CoupledDeviceimagingSystems," inElectro-OpticalDisplays, M.A. Karim, ed., pp.
551-560, Marcel Dekker, New York (1992).
12. J. A. Ratches, W. R. Lawson, L. P. Obert, R. J. Bergemann, T. W. Cassidy, and J. M.
Swenson, "Night Vision Laboratory Static Performance Model for Thermal Viewing Systems,"
ECOM Report ECOM-7043, p. 34, Fort Monmouth, NJ (1975).
13. J. M. Lloyd, Thermal Imaging Systems, pp. 182-194, Plenum Press, NY (1975).
14. F. A. Rosell, "Laboratory Performance Model," in The Fundamentals of Thermal Imaging
Systems, F. Rosell and G. Harvey, eds., NRL Report 8311, pp. 85-95, Naval Research
Laboratory, Washington, D.C. (1979).
15. R. L. Sendall and F. A. Rosell, "Static Performance Model Based on the Perfect Synchronous
Integrator Model," in The Fundamentals of Thermal Irrwging Systems, F. Rosell and G. Harvey,
eds., NRL Report 8311, pp. 181-230, Naval Research Laboratory, Washington, D.C. (1979).
16. F. A. Rosell, "Video, Display, and Perceived Signal-to-Noise Ratios," in The Fundamentals
of Thermal Imaging Systems, F. Rosell and G. Harve}, eds., NRL Report # 8311, pp. 49-83,
Naval Research Laboratory, Washington, D.C. (1979)
17. W. W. Frame, "Minimum Resolvable and Minimum Detectable Contrast Prediction for Vidicon
Cameras," SMPTE Journal, Vol. 104{1), pp. 21-26 (l 985).
18. W. W. Frame, "Minimum Resolvable and Minimum Detectable Contrast Prediction for
Monochrome Solid-State Imagers," SMPTE Journal, Vol. 96(5), pp. 454-459 (1987).
19. FUR92 Thermal Imaging Systems Performance Model. Analyst's Reference Guide, NVESD
documentRG5008993, pp. ARG-12- 13, Fort Belvoir. VA (1993).
20. J. A. Ratches, W. R. Lawson, L. P. Obert, R. J. Bergemann, T. W. Cassidy, and J. M.
Swenson, "Night Vision Laboratory Static Performance Model for Thermal Viewing Systems,"
ECOM Report ECOM-7043, p. 56, Fort Monmouth, ~J (1975).
21. F. A. Rosell, "Psychophysical Experimentation," in 7he Fundamentals of Thermal Imaging
Systems, F. Rosell and G. Harvey, eds., NRL Report # 8311, p. 225, Naval Research
Laboratory, Washington, D.C. (1979).
22. G. C. Holst, Electro-Optical Imaging System Performance, Chapters 19, 20, and 21, JCD
Publishing, Winter Park, FL (1995).
23. L. Levi, Applied Optics, pp. 118-124, Wiley and Sons (1980).
24. W. E. K. Middleton, Vision Through the Atmasphere, University of Toronto Press (1958).
25. J. Johnson, "Analysis of Imaging Forming Systems," in Proceedings of the Image Intensifier
Symposium, pp. 249-273, Warfare Electrical Engineering Dept., U.S. Army Engineering
Research and Development Laboratories, Ft. Belvoir, VA (1958). This article is reprinted in
Selected Papers on Infrared Design, R. B. Johnson and W. L. Wolfe, eds., SPIE Proceedings
Vol. 513, pp. 761-781 (1985).
26. L. Scott and R. Tomkinson, "An Update on the C 2NVEO FLIR90 and ACQUIRE Sensor
Performance Model," in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing
II, G. C. Holst, ed., SPIE Proceedings Vol. 1488, pp. 99-109 (1991).
27. J. D. Howe, "Electro-Optical Imaging System Performance Prediction," in Electro-Optical
Systems Design, Analysis, and Testing, M. C. Dudzik, ed., p. 92. This is Volume 4 of the
Infrared and Electro-Optical Systems Handbook, J. S Accetta and D. L. Shumaker, eds.,
copublished by Environmental Research Institute of Michigan, Ann Arbor, MI, and SPIE Press,
Bellingham, WA (1993).
APPENDIX

A.l. EFFECTIVE FOCAL LENGTH

Most optical systems are composed of many individual lenses and mirrors.
Each element may have a different refractive index and shape to minimize the
system aberrations. The optical design can be considered as a single thick
element (Figure A-1). For modeling purposes, it is treated as a single thin
element with an effective focal length (Figure 2-6, page 23). The clear aperture
is not necessarily the diameter of an optical element. The clear aperture limits
the amount of scene radiation reaching the detector and is determined by the
optical design.

I
P,

Figure A-1. The optical system can be considered as a single thick lens.
P1 and P2 are the principal surfaces. The effective focal length is
measured from the second principal surface. The clear aperture limits
the amount of scene radiation reaching the detector. Although shown as
planes, the principal surfaces are typically spherical.

A.2. f-NUMBER
The radiometric equations were derived from plane geometry and paraxial
ray approximations. When using solid angles, the image incidance is
proportional to sin2(U') where U' is the maximum angle subtended by the lens.
For paraxial rays, the principal surface is considered a plane. This
representation is shown in most textbooks.

371
372 CCD ARRAYS, CAMERAS, and DISPLAYS

Lens design theory 1 assumes that the principal surface is spherical - every
point on the surface is exactly a focal-length distance away (Figure A-2). This
"sine corrected" lens has zero spherical aberratwn. Thus, sin2(U') is equal to
11(4F2). The numerical aperture is another mea~ure of the energy collected by
the optical system. When the image is in air !refractive index of unity) the
numerical aperture is

1 (A-1)
NA - sinU1 -
2F

Since the largest angle is 1r/2, the smallest theoretical value for F is Vz. This
theoretical limit on F is not obvious from the radiometric equation.

P1

Figure A-2. Principal planes are typically spherical. Sin(U') = D/2fl =

112F (in air).

Not all optical systems have spherical principal planes. Often we assume the
principal plane is flat. For hand-drawn optical layouts, the lenses are
approximated by thin lenses and paraxial rays are used (large f-numbers). Here,
=
sin(O) 0 and Equation A-1 reduces to F = fl/D. Then sin2(U') is 11(4f2+ 1).
Whether 1/(4F2 + 1) or 1/4f2 is used in the radiometric equations depends upon
the f-number definition. If the analyst calculates the f-number from the effective
lens diameter and focal length, then 11(4f2+ 1) should be used. If the optical
designer provides the f-number, then the analyst must consult with him to ensure
that the appropriate factor is used. For large f-numbers, the factors are
approximately equal.

A.3. REFERENCE
1. W. J. Smith, Modem Optical Engineering, second ediuon, pp. 142-145, McGraw-Hill, New
York (1990).
INDEX
Absorption coefficient 104, 277 Camcorder 6
Active pixel sensor 91 Camera formula 33
Addressability 222 Camera operation 147
Adjacent disel requirement 222 Camera performance 182
Advanced television system 160 Cathode ray tube 205
Aerial image modulation 325 Causality 252
Airy disk 16, 257, 286, 320 CAV 156
Aliasing 232, 236 CCIR 150
display 250 CDS 141
frame grabber 250 Center-to-center spacing 16
Alternating disel requirement 222 Channel block 49
Amplifier noise 128 Character recognition 227
Analog-to-digital converter 174 Charge aggregation 57
Anti-aliasing filter 236 Charge conversion 78
Anti-parallel clock 51 Charge grouping 57
Antibloom drain 85 Charge injection device 89
Aperture correction 172, 302 Charge transfer efficiency 74, 279
APS 91 Chicken wire 179
Array CID 89
frame interline transfer 64 CIE 29
frame transfer 61 Clock 49
full frame transfer 59 Cluster defect 144
interline transfer 63 CMOS 91
linear 58 Color correction 165
TDI 70 Color filter array 95, 242, 283
Atmospheric transmittance 360 Color temperature 21, 31, 209
ATS 160 display 32
Column defect 144
Back illuminated 109 Component analog video 156
Bayer 96 Composite video 156
Beat frequency 245 Configuration 9
Bilinear readout 58 Contrast 228
Binning 57, 58, 187, 299 Contrast transfer function 263
Birefringent crystal 283 Contrast transmittance 359
Blackbody 21 Convolution 255, 257
Blanking 151 Cooling 80
Blur diameter 320 Correlated double sampling 141
Boost filter 302 CTE 74
Brightness 26 Cutoff
Broadcast standard detector 240, 275
NTSC 150 optical 261, 27 4
PAL 150
SECAM 150 Dark current 79
Buried channel 74 Dark current noise 79, 127
Butterworth filter 295 Dark pixel 83
DAS 15, 321
Calibration source 29 Date) 14

373
374 CCD ARRAYS, CAMERAS, and DISPLAYS

Dead pixel 144 Expo lUre control 85

Defects 143
Depletion region 48, 105 f-number 34, 371
Depletion width 277 Faceplate illumination 183
Detector Fat zero 77
cutoff 275 Fiber -optic bundle 195
MTF 275 Fiberoptic bundle 176
Detector cutoff 240 Fiberoptic coupling 178
Detector pitch 242 Fictitious spatial frequency 267
Detector-angular-subtense 15, 322 Field integration 66
Detector-limited 287, 321 Fill-factor 16, 63, 242
Differential nonlinearity 174 Filter
Diffusion 78, 276 anti-alias 236
Diffusion length 105 hoost 302
Digital filter 296 digital 296
Digital recorder 11 optical anti-alias 283
Digital television 159 post-reconstruction 300
Dirac delta 254 Finite impulse response 297
Discrimination methodology 362 Fixed pattern noise 131
Disel 14, 214 Flat f1eld 212
Disel density 216 Flicker 205
Display 201 FLIR92 338
aliasing 250 Floatmg diffusion 78
color 207 Floatmg diode 78
color temperature 32 Flux 20,26
flat panel 202 Format 98
monochrome 207 FPN 131
spot size 210 Frame grabber 329
Display MTF 303 aliasing 249
Display space 269 Frame integration 65
Distortion 245 Frame interline transfer 64
Dixel 14 Frame rate 142
Drain 85 Frame transfer 61
Dynamic range 121, 187 Front- sided illuminated 107
Full-frame transfer 59
Effective-instantaneous-field-of-view
322 Gamma 169, 206
EIA 170 149 Gate 45
EIA 170A 150 Gibbs phenomenon 262
EIA 343A 155 Grand Alliance 161
Electronic resolution 323 GRD 319
Electronic shutter 63, 85 Ground resolved distance 319
Electronic zoom 287 Grouping 57
Emittance 20, 26
Equivalent background illumination HAD 47,93
199 HDTV 160
Equivalent bar pattern 362 High Definition Television 160, 210
Etalon 111 Hole accumulation diode 47
Excess noise 198 Hot point defect 144
Exitance 20, 26 Human visual response 306
 INDEX 375

Human visual system 97, 191 Lambertian source 20, 25

HVS 97 Lateral drain 85
integration time 355 Limiting resolution 324
modulation threshold 340 Line spread function 257
synchronous integrator 345 Linear array 58
tuned circuit 345 Linear system theory 252
HVS MTF 306 Linear-shift-invariant 252
Hybrid 3 Low light television 176
Hyper-HAD 93 Luminance 26
Lumogen 109
ICCD 176
IIRS 320 Machine vision 7
Illuminance 26 Matched filter 345
Illuminant Matrixing 96, 157
A, B, C, D 29 Mean-variance 133
Illumination level Metal-insulator-semiconductor 45
artificial 28 Metal-oxide-semiconductor 45
maximum 185 Meteorological range 360
minimum 185 Microchannel plate 17 6, 197
natural 28 Microlens 93
Image distortion 245 Military 8
Image intensifier 176, 311 Minification 197
Image quality 12, 315 Minimum resolvable contrast 338
square-root integral 334 Minimum resolvable temperature 338
subjective quality factor 332 MIS 45
Image space 269 Modulation 259
Imagery Interpretability Rating Scale Modulation threshold 307
320 Modulation transfer function 257,
Impulse 254 259
In-phase 246, 312 Modulation transfer function area
Incidance 20, 26 331
Infinite impulse response 297 Moire pattern 214, 231
Infrared filter 39 Monitor 201, 272
Inslantaneous-field-of-view 322 Monolithic 3, 45
Integral nonlinearity 17 4 MOS 45
Intensified CCD 176, 195 MOS capacitor 45
Intensity 20, 26 Motion 292
Interline transfer 63 MPP 79
IRE units 157 MRC 338
Irradiance 20 two-dimensional 358
Isolation pixel 85 MRT 338
MTF
Jaggies 232 average 312
Jitter 292 averaging filter 299
Johnson criteria 362 boost 302
Johnson noise 127 Butterworth 295
charge transfer efficiency 279
Kell factor 154, 219, 312 CRT 303
Knee 86, 164 detector 239, 275
Koschmieder 360 diffraction-limited 261
KTC noise 127 diffusion 27 6
376 CCD ARRAYS, CAMERAS, and DISPLAYS

MTF Normalization 37
display 303 NTSC 149
eye 306 Nyquist frequency 231,242
Gaussian 294
HVS 306 Object space 269
image intensifier 311 Observer 203
lens 261 Observer space 269
linear motion 292 Optical anti-alias filter 283
median 312 Optical resolution 320
non-limiting 311 Optical transfer function 257
optical anti-alias 283 Optics-limited 287, 321
optics 274 Organic phosphor 109
random motion 294 Out-of-phase 246, 312
Output gain conversion 78, 137
sample and hold 299 Output structure 78
sample-scene 312 Overflow drain 85
separable 268 Oversampling 232
system 286
TDI 280 PAL 150
tuned circuit 303 PAS 16
Multi-pinned phasing 82 Path radiance 359
Pel 14
National Imagery Interpretability Phas~
Rating Scale 320 four 51
NEE 103 three 53
NIIRS 320 cWO 55
Noise virtual 57
1/f 128 Phase reversal 292
ADC 130 Phase transfer function 257
amplifier 128 Phosphor 109, 207
dark current 127 Photodiode 47
excess 198 Photogate 47
fixed pattern 131 Photometric conversion 118
Johnson 127 Photometry 23
kTC 127 Photon standard 102
photoresponse nonuniformity Photon transfer 102, 133
131 Photopic 23
quantization 130 Photo response nonuniformity 131
reset 127 Pitch 16, 242
shot 127 1riad 208
summary factors 351 Pixel 14
three-dimensional 346 Pixel trap 144
white 128 Pixel-angular-subtense 16, 322
Noise equivalent bandwidth 128, 129 Planck's blackbody law 21
Noise equivalent exposure 119 Point defect 144
Noise equivalent input 192 Point spread function 257
Noise equivalent reflectance Poisson statistics 127
difference 193 Polysilicon 107
Noise equivalent signal 120 Post- reconstruction filter 300
Non-destructive readout 89 PRNU 131
 INDEX 377
Professional broadcast television 6 Shrinking raster 211
Progressive scan 68, 156, 172, 205, Shutter speed 35
299 Signal-to-noise ratio 139, 340
Pseudo-interlacing 66, 299 Size
video chip 99
Quantization noise 130 Sky
Quantum efficiency 104, 111 night 39
Sky-to-ground ratio 360
Radiance 20 Smear 60, 68
Radiative transfer 19 SNR 186
Radiometry 18 SNRru 355
Range prediction 359, 365 Source
Raster 203 calibration 29
Rayleigh criterion 320 Spatial frequency 269
Recorder Spectral response 107
digital 11 Square wave 262
Relay lens 176, 195 Square-root integral 334
Reset 14, 320 Square-wave response 263
Reset noise 127 Sterance 20, 26
Resolution 218, 222, 317 Streaking 73
detector 321 Strobe light 68
electronic 323 Subarray 60
limiting 324 Subcarrier frequency 160
optical 320 Subjective quality factor 332
perceived 344 Summary noise factor 351
Shade 326 Super pixeling 57, 299
TV limiting 220 Superposition 255, 261
Resolution/addressability ratio 223, Surface channel 74
304 Synchronous integrator 345
Responsivity 40, 111
Ringing 262, 302 T-number 36
RS 170 149 Taps 297
RS 170A 150 Target transfer probability function
363
Sample-and-hold 299 TDI 70, 173, 280
Sample-scene 312 TEC 80
Sampling 231, 368 Temperature
Sampling theorem 234 color 21, 31
Saturation equivalent exposure 120 Thermoelectric cooler 80
Scan pattern 67 Three-dimensional noise 346
Scene 186 Threshold modulation 325
Scene! 14 Time delay and integration 70
Scientific application 8 Transmittance
Scotopic 23 atmospheric 360
SECAM 150 Tuned circuit 345
SEE 103 TVLIPH 220
Setup 157 Two-dimensional MRC 358
Shade's equivalent resolution 325
Shades of gray 227 Undersampling 232
Shield 61
Shot noise 127 Velocity error 280
378 CCD ARRAYS, CAMERAS, and DISPLAYS

Vertical overflow drain 85

Vertical overflow drain 11
Video
CAV 156
CCIR 150
component 156
composite 156
EIA 170 149
EIA 170A 149
EIA 343A 155
NTSC 149
PAL 150
SECAM 150
Video cassette recorder 9
Video chip size 98
Video formats 149
Video standard 271
Video timing 151, 271
Viewing ratio 204
Virtual phase 57
Visibility code 361
Visual acuity 204
Visual threshold 36

Well capacity 48
White noise 128
Wien's law 21

X-ray transfer 102

Zoom 287