Scribd Logo
Upload

Welcome to Scribd!

  • 17 views

    Entropy and Mutual Information: R Pix

    Uploaded by

    Amir Hamza
    The document defines entropy as a measure of uncertainty for discrete random variables. Entropy is calculated as the sum, over all possible outcomes of the random variable, of the probability of each outcome multiplied by the logarithm of the inverse of that probability. This definition is elaborated on to specify that the logarithm base can be any base, with base-2 and base-e units referred to as bits and nats respectively. Examples are given to illustrate entropy calculations for a fair die roll and a binary random variable.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd

    Entropy and Mutual Information: R Pix

    Uploaded by

    Amir Hamza
    17 views1 page
    The document defines entropy as a measure of uncertainty for discrete random variables. Entropy is calculated as the sum, over all possible outcomes of the random variable, of the probability of each outcome multiplied by the logarithm of the inverse of that probability. This definition is elaborated on to specify that the logarithm base can be any base, with base-2 and base-e units referred to as bits and nats respectively. Examples are given to illustrate entropy calculations for a fair die roll and a binary random variable.

    Original Description:

    entropy

    Original Title

    Nouveau Document Texte

    Copyright

    © © All Rights Reserved

    Available Formats

    DOCX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document defines entropy as a measure of uncertainty for discrete random variables. Entropy is calculated as the sum, over all possible outcomes of the random variable, of the probability of each outcome multiplied by the logarithm of the inverse of that probability. This definition is elaborated on to specify that the logarithm base can be any base, with base-2 and base-e units referred to as bits and nats respectively. Examples are given to illustrate entropy calculations for a fair die roll and a binary random variable.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    17 views1 page

    Entropy and Mutual Information: R Pix

    Uploaded by

    Amir Hamza
    The document defines entropy as a measure of uncertainty for discrete random variables. Entropy is calculated as the sum, over all possible outcomes of the random variable, of the probability of each outcome multiplied by the logarithm of the inverse of that probability. This definition is elaborated on to specify that the logarithm base can be any base, with base-2 and base-e units referred to as bits and nats respectively. Examples are given to illustrate entropy calculations for a fair die roll and a binary random variable.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    of 1

    Entropy and mutual information

    1.1 Discrete random variables


    Suppose X is a discrete random variable, that is, one whose range
    R = {x], X2, ... } is finite or countable. Let Pi = PiX = Xi}' (For probabilistic
    terminology consult Appendix A.) The entropy of X is defined by
    1
    H(X) = LPilog-. (1.1)
    i~] Pi
    This definition needs elaboration. First, the base of the logarithm is purposely
    left unspecified. If necessary, however, we shall denote the base-b entropy by
    Hb(X), and say that the entropy of X is being measured in base-b units. Base-
    2 units are called bits (hinary igili), and base-e units are called nats (natural
    digi~). Second, if Pi = 0, the term Pi logpi] in (1.1) is indeterminate; we
    define it to be 0, however. (This convention is by no means arbitrary; see Prob.
    1.1.) Finally, if R is infinite the sum (1.1) may not converge; in this case we
    set H(X) = +00.
    Example 1.1 Let X represent the outcome of a single roll of a fair die. Then
    R = {I, 2, 3,4,5, 6} and Pi = ~ for each i. Here H(X) = log 6 = 2.58
    bits = 1.79 nats. D
    Example 1.2 Let R = {O, 1}, and define X by PiX = O} = p, PiX = 1} =
    1 - p. Then H(X) = - P logp - (1 - p) 10g(1 - p), and so H2(X), as a

    function of ° ::;; P ::;; 1, is identical to the binary entropy function H2(p),


    which was defined in Eq. (0.13). In what follows, we will frequently represent
    the function-P logp - (1 - p) 10g(1 - p), where the bases of the logarithms
    are unspecified, by H(p), and call it the entropy function. Figure 1.1 gives its
    graph (cf. Fig. 0.4). More generally, if p = (p], ... , Pr) is any probability

    You might also like