Bayes Tutorial
Bayes Tutorial
G. Larry Bretthorst
Biomedical MR Laboratory
Washington University School Of Medicine,
Campus Box 8227
Room 2313, East Bldg.,
4525 Scott Ave.
St. Louis MO 63110
http://bayes.wustl.edu
Email: [email protected]
[1] Rev. Thomas Bayes (1763), “An Essay Toward Solving a Problem in the Doctrine of Chances,”
Philos. Trans. R. Soc. London, 53, pp. 370-418; reprinted in Biometrika, 45, pp. 293-315 (1958),
and Facsimiles of Two Papers by Bayes, with commentary by W. Edwards Deming, New York,
Hafner, 1963.
[2] G. Larry Bretthorst (1988), “Bayesian Spectrum Analysis and Parameter Estimation,” in Lec-
ture Notes in Statistics, 48, J. Berger, S. Fienberg, J. Gani, K. Krickenberg, and B. Singer
(eds), Springer-Verlag, New York, New York.
[3] G. Larry Bretthorst (1990), “An Introduction to Parameter Estimation Using Bayesian Prob-
ability Theory,” in Maximum Entropy and Bayesian Methods, Dartmouth College 1989, P.
Fougère ed., pp. 53-79, Kluwer Academic Publishers, Dordrecht the Netherlands.
[4] G. Larry Bretthorst (1990), “Bayesian Analysis I. Parameter Estimation Using Quadrature
NMR Models” J. Magn. Reson., 88, pp. 533-551.
[5] G. Larry Bretthorst (1990), “Bayesian Analysis II. Signal Detection And Model Selection” J.
Magn. Reson., 88, pp. 552-570.
[6] G. Larry Bretthorst (1990), “Bayesian Analysis III. Examples Relevant to NMR” J. Magn.
Reson., 88, pp. 571-595.
[7] G. Larry Bretthorst (1991), “Bayesian Analysis. IV. Noise and Computing Time Considera-
tions,” J. Magn. Reson., 93, pp. 369-394.
[8] G. Larry Bretthorst (1992), “Bayesian Analysis. V. Amplitude Estimation for Multiple Well-
Separated Sinusoids,” J. Magn. Reson., 98, pp. 501-523.
[9] G. Larry Bretthorst (1992), “Estimating The Ratio Of Two Amplitudes In Nuclear Magnetic
Resonance Data,” in Maximum Entropy and Bayesian Methods, C. R. Smith et al. (eds.),
pp. 67-77, Kluwer Academic Publishers, the Netherlands.
[10] G. Larry Bretthorst (1993), “On The Difference In Means,” in Physics & Probability Essays in
honor of Edwin T. Jaynes, W. T. Grandy and P. W. Milonni (eds.), pp. 177-194, Cambridge
University Press, England.
[11] G. Larry Bretthorst (1996), “An Introduction To Model Selection Using Bayesian Probability
Theory,” in Maximum Entropy and Bayesian Methods, G. R. Heidbreder, ed., pp. 1-42, Kluwer
Academic Publishers, Printed in the Netherlands.
479
480 BIBLIOGRAPHY
[15] G. Larry Bretthorst, William C. Hutton, Joel R. Garbow, and Joseph J.H. Ackerman (2005),
“Exponential parameter estimation (in NMR) using Bayesian probability theory,” Concepts in
Magnetic Resonance, 27A, Issue 2, pp. 55-63.
[16] G. Larry Bretthorst, William C. Hutton, Joel R. Garbow, and Joseph J. H. Ackerman (2005),
“Exponential model selection (in NMR) using Bayesian probability theory,” Concepts in Mag-
netic Resonance, 27A, Issue 2, pp. 64-72.
[17] G. Larry Bretthorst, William C. Hutton, Joel R. Garbow, and Joseph J.H. Ackerman (2005),
“How accurately can parameters from exponential models be estimated? A Bayesian view,”
Concepts in Magnetic Resonance, 27A, Issue 2, pp. 73-83.
[18] G. Larry Bretthorst, W. C. Hutton, J. R. Garbow, and Joseph J. H. Ackerman (2008), “High
Dynamic Range MRS Time-Domain Signal Analysis,” Magn. Reson. in Med., 62, pp. 1026-1035.
[19] V. Chandramouli, K. Ekberg, W. C. Schumann, S. C. Kalhan, J. Wahren, and B. R. Landau
(1997), “Quantifying gluconeogenesis during fasting,” American Journal of Physiology, 273,
pp. H1209-H1215.
[20] R. T. Cox (1961), “The Algebra of Probable Inference,” Johns Hopkins Univ. Press, Baltimore.
[21] André d’Avignon, G. Larry Bretthorst, Marilyn Emerson Holtzer, and Alfred Holtzer (1998),
“Site-Specific Thermodynamics and Kinetics of a Coiled-Coil Transition by Spin Inversion
Transfer NMR,” Biophysical Journal, 74, pp. 3190-3197.
[22] André d’Avignon, G. Larry Bretthorst, Marilyn Emerson Holtzer, and Alfred Holtzer (1999),
“Thermodynamics and Kinetics of a Folded-Folded Transition at Valine-9 of a GCN4-Like
Leucine Zipper,” Biophysical Journal, 76, pp. 2752-2759.
[23] David Freedman, and Persi Diaconis (1981), “On the histogram as a density estimator: L2
theory,” Zeitschrift f¨r Wahrscheinlichkeitstheorie und verwandte Gebiete, 57, 4, pp. 453-476.
[24] W. R. Gilks, S. Richardson, and D. J. Spiegelhalter (1996), “Markov Chain Monte Carlo in
Practice,” Chapman & Hall, London.
BIBLIOGRAPHY 481
[25] Paul M. Goggans, and Ying Chi (2004), “Using Thermodynamic Integration to Calculate the
Posterior Probability in Bayesian Model Selection Problems,” in Bayesian Inference and Maxi-
mum Entropy Methods in Science and Engineering: 23rd International Workshop, 707, pp. 59-
66.
[26] Marilyn Emerson Holtzer, G. Larry Bretthorst, D. André d’Avignon, Ruth Hogue Angelette,
Lisa Mints, and Alfred Holtzer (2001), “Temperature Dependence of the Folding and Unfolding
Kinetics of the GCN4 Leucine Lipper via 13C alpha-NMR,” Biophysical Journal, 80, pp. 939-
951.
[27] E. T. Jaynes (1968), “Prior Probabilities,” IEEE Transactions on Systems Science and Cyber-
netics, SSC-4, pp. 227-241; reprinted in [30].
[28] E. T. Jaynes (1978), “Where Do We Stand On Maximum Entropy?” in The Maximum Entropy
Formalism, R. D. Levine and M. Tribus Eds., pp. 15-118, Cambridge: MIT Press, Reprinted
in [30].
[29] E. T. Jaynes (1980), “Marginalization and Prior Probabilities,” in Bayesian Analysis in
Econometrics and Statistics, A. Zellner ed., North-Holland Publishing Company, Amsterdam;
reprinted in [30].
[30] E. T. Jaynes (1983), “Papers on Probability, Statistics and Statistical Physics,” a reprint col-
lection, D. Reidel, Dordrecht the Netherlands; second edition Kluwer Academic Publishers,
Dordrecht the Netherlands, 1989.
[31] E. T. Jaynes (1957), “How Does the Brain do Plausible Reasoning?” unpublished Stanford
University Microwave Laboratory Report No. 421; reprinted in Maximum-Entropy and Bayesian
Methods in Science and Engineering 1, pp. 1-24, G. J. Erickson and C. R. Smith Eds., 1988.
[32] E. T. Jaynes (2003), “Probability Theory—The Logic of Science,” edited by G. Larry Bretthorst,
Cambridge University Press, Cambridge UK.
[33] Sir Harold Jeffreys (1939), “Theory of Probability,” Oxford Univ. Press, London; Later editions,
1948, 1961.
[34] John G. Jones, Michael A. Solomon, Suzanne M. Cole, A. Dean Sherry, and Craig R. Mal-
loy (2001) “An integrated 2 H and 13 C NMR study of gluconeogenesis and TCA cycle flux in
humans,” American Journal of Physiology, Endocrinology, and Metabolism, 281, pp. H848-
H856.
[35] John Kotyk, N. G. Hoffman, W. C. Hutton, G. Larry Bretthorst, and J. J. H. Ackerman (1992),
“Comparison of Fourier and Bayesian Analysis of NMR Signals. I. Well-Separated Resonances
(The Single-Frequency Case),” J. Magn. Reson., 98, pp. 483–500.
[36] Pierre Simon Laplace (1814), “A Philosophical Essay on Probabilities,” John Wiley & Sons,
London, Chapman & Hall, Limited 1902. Translated from the 6th edition by F. W. Truscott
and F. L. Emory.
[37] N. Lartillot, and H. Philippe (2006), “Computing Bayes Factors Using Thermodynamic Inte-
gration,” Systematic Biology, 55 (2), pp. 195-207.
482 BIBLIOGRAPHY
[38] D. Le Bihan, and E. Breton (1985), “Imagerie de diffusion in-vivo par rsonance,” Comptes
rendus de l’Acadmie des Sciences (Paris), 301 (15), pp. 1109-1112.
[39] N. R. Lomb (1976), “Least-Squares Frequency Analysis of Unevenly Spaced Data,” Astrophys-
ical and Space Science, 39, pp. 447-462.
[44] K. Merboldt, Wolfgang Hanicke, and Jens Frahm (1969), “Self-diffusion NMR imaging using
stimulated echoes,” Journal of Magnetic Resonance, 64 (3), pp. 479-486.
[45] Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and
Edward Teller (1953), “Equation of State Calculations by Fast Computing Machines,” Journal
of Chemical Physics. The previous link is to the Americain Institute of Physics and if you do
not have access to Science Sitations you many not be able to retrieve this paper.
[46] Radford M. Neal (1993), “Probabilistic Inference Using Markov Chain Monte Carlo Methods,”
technical report CRG-TR-93-1, Dept. of Computer Science, University of Toronto.
[47] Jeffrey J. Neil, and G. Larry Bretthorst (1993), “On the Use of Bayesian Probability Theory for
Analysis of Exponential Decay Data: An Example Taken from Intravoxel Incoherent Motion
Experiments,” Magn. Reson. in Med., 29, pp. 642–647.
[48] H. Nyquist (1924), “Certain Factors Affecting Telegraph Speed,” Bell System Technical Journal,
3, pp. 324-346.
[49] H. Nyquist (1928), “Certain Topics in Telegraph Transmission Theory,” Transactions AIEE, 3,
pp. 617-644.
[50] William H. Press, Saul A. Teukolsky, William T. Vetterling and Brian P. Flannery (1992),
“Numerical Recipes The Art of Scientific Computing Second Edition,” Cambridge University
Press, Cambridge UK.
[51] Emanuel Parzen (1962), “On Estimation of a Probability Density Function and Mode,” Annals
of Mathematical Statistics 33, 1065–1076
[52] Karl Pearson (1895), “Contributions to the Mathematical Theory of Evolution. II. Skew Vari-
ation in Homogeneous Material,” Phil. Trans. R. Soc. A 186, 343–326.
BIBLIOGRAPHY 483
[53] Murray Rosenblatt, “Remarks on Some Nonparametric Estimates of a Density Function,” An-
nals of Mathematical Statistics 27, 832–837 (1956).
[54] Jeffery D. Scargle (1981), “Studies in Astronomical Time Series Analysis I. Random Process In
The Time Domain,” Astrophysical Journal Supplement Series, 45, pp. 1-71.
[55] Jeffery D. Scargle (1982), “Studies in Astronomical Time Series Analysis II. Statistical Aspects
of Spectral Analysis of Unevenly Sampled Data,” Astrophysical Journal, 263, pp. 835-853.
[56] Jeffery D. Scargle (1989), “Studies in Astronomical Time Series Analysis. III. Fourier Trans-
forms, Autocorrelation Functions, and Cross-correlation Functions of Unevenly Spaced Data,”
Astrophysical Journal, 343, pp. 874-887.
[57] Arthur Schuster (1905), “The Periodogram and its Optical Analogy,” Proceedings of the Royal
Society of London, 77, p. 136-140.
[58] Claude E. Shannon (1948), “A Mathematical Theory of Communication,” Bell Syst. Tech. J.,
27, pp. 379-423.
[59] John E. Shore, and Rodney W. Johnson (1981), ”Properties of cross-entropy minimization,”
IEEE Trans. on Information Theory, IT-27, No. 4, pp. 472-482.
[60] John E. Shore and Rodney W. Johnson (1980), “Axiomatic derivation of the principle of maxi-
mum entropy and the principle of minimum cross-entropy,” IEEE Trans. on Information The-
ory, IT-26 (1), pp. 26-37.
[61] Devinderjit Sivia, and John Skilling (2006), “Data Analysis: A Bayesian Tutorial,” Oxford
University Press, USA.
[62] Edward O. Stejskal and Tanner, J. E. (1965), “Spin Diffusion Measurements: Spin Echoes
in the Presence of a Time-Dependent Field Gradient.” Journal of Chemical Physics, 42 (1),
pp. 288-292.
[63] D. G. Taylor and Bushell, M. C. (1985), “The spatial mapping of translational diffusion coeffi-
cients by the NMR imaging technique,” Physics in Medicine and Biology, 30 (4), pp. 345-349.
[64] Myron Tribus (1969), “Rational Descriptions, Decisions and Designs,” Pergamon Press, Oxford.
[65] P. M. Woodward (1953), “Probability and Information Theory, with Applications to Radar,”
McGraw-Hill, N. Y. Second edition (1987); R. E. Krieger Pub. Co., Malabar, Florida.
[66] Arnold Zellner (1971), “An Introduction to Bayesian Inference in Econometrics,” John Wiley
and Sons, New York.
Index
484
INDEX 485
McMC Values, 238 Bayes’ Theorem, 100, 139, 145, 153, 167, 211,
Prob Model, 238 226, 243, 252, 261, 269, 278, 288, 295,
Viewers 306, 314, 315, 317, 318, 331, 333, 343,
Fid Data, 219 370, 399, 407, 439
Fid Model, 221, 236 Bayes.accepted
File, 222, 238 Body, 77
Metabolite, 221 Header, 76
Plot Results, 238 Behrens-Fisher Package, 311
Text, 238 Bayesian Calculations
Widgets Derived Probabilities, 320
Fid Model, 221 Different Mean And Same Variance, 318
Fid Model Viewer, 221 Different Mean And Variance, 319
Load System Metabolite File, 219 Parameter Estimation, 321
Load System Resonance File, 221 Same Mean And Different Variance, 317
Load User Metabolite File, 219 Same Mean And Variance, 315
Load User Resonance File, 221 Model Equation
Shift Left, 221 Different Mean And Same Variance, 318
Shift Right, 221 Different Mean And Variance, 319
Bayes Model, 159, 159 Same Mean And Different Variance, 317
Bayes Test Data Package, 427 Same Mean And Variance, 315
Parameters, 431 Number of data sets, 311
Reports Parameter Listing, 323
Bayes Accepted, 428 Prior Probabilities
Condensed, 429 Different Mean And Same Variance, 318
McMC Values, 429, 431–433 Different Mean And Variance, 319
Viewers Same Mean And Different Variance, 317
Fortran/C Models, 427 Same Means And Same Variance, 315
Image, 428 Reports
Prior Probabilities, 427 Bayes Accepted, 311, 322
Text Data, 430 Condensed, 322
Text Results, 429 Console Log, 322, 323
Widgets McMC Values, 322, 323
# Images, 427 Prob Model, 322
# Slices, 427 Using, 311
Abscissa, 427 Viewers
ArrayDim, 427 File, 322
Build, 427 Plot Results, 322, 324
Get Job, 428 Prior Probabilities, 311
Max Value, 427 Text, 322
Noise SD, 427 Widgets
Parameter Ranges, 428 None, 311
Pe, 427 Big Endian, 471, 473
Ro, 427 Big Magnetization Transfer Package, 259
Run, 428 Bayesian Calculations, 259
Set (server), 428 Files
Status, 428 Bayes Analyze, 264
488 INDEX
Down arrow, 74
Enable Editing, 74
Print, 74
Save (a copy), 74
Save As, 74
Settings, 74
Up arrow, 74
Unknown Number of Exponentials Package
Constant, 143
Find Outliers, 143
Order, 143
Unknown Polynomial Order Package
Set Order, 293, 294
Unknown Order, 293, 294
WorkDir
Creating, 22, 33, 46
Deleting, 22, 33, 46
List, 24, 46
Loading, 46
Name, 46
Popup, 47