Scribd
Upload
46 views1 page

A Compilation Target For Probabilistic Programming Languages - 2014 (Paige14) (Dragged)

The document describes probabilistic programming capabilities added to the C programming language. It introduces two key functions: observe, which conditions program execution on the log-likelihood of observed data, and predict, which marks expressions to output for posterior sampling. Examples include a simple Gaussian model, a hidden Markov model, and an infinite mixture of Gaussians model implemented probabilistically in C. The goal is to enable complex probabilistic models to be specified concisely and transparently using a low-level language.

Uploaded by

raqibapp
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
46 views1 page

A Compilation Target For Probabilistic Programming Languages - 2014 (Paige14) (Dragged)

The document describes probabilistic programming capabilities added to the C programming language. It introduces two key functions: observe, which conditions program execution on the log-likelihood of observed data, and predict, which marks expressions to output for posterior sampling. Examples include a simple Gaussian model, a hidden Markov model, and an infinite mixture of Gaussians model implemented probabilistically in C. The goal is to enable complex probabilistic models to be specified concisely and transparently using a low-level language.

Uploaded by

raqibapp
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

A Compilation Target for Probabilistic Programming Languages

#include "probabilistic.h" #include "probabilistic.h"


#define K 3
int main(int argc, char **argv) { #define N 11

double var = 2; /* Markov transition matrix */


double mu = normal_rng(1, 5); static double T[K][K] = { { 0.1, 0.5, 0.4 },
{ 0.2, 0.2, 0.6 },
o b s e r v e (normal_lnp(9, mu, var)); { 0.15, 0.15, 0.7 } };
o b s e r v e (normal_lnp(8, mu, var));
/* Observed data */
p r e d i c t ("mu,%f\n", mu); static double data[N] = { NAN, .9, .8, .7, 0, -.025,
-5, -2, -.1, 0, 0.13 };
return 0;
} /* Prior distribution on initial state */
static double initial_state[K] = { 1.0/3, 1.0/3, 1.0/3 };

/* Per-state mean of Gaussian emission distribution */


Figure 1. This program performs posterior inference over the un- static double state_mean[K] = { -1, 1, 0 };
known mean mu of a Gaussian, conditioned on two data points. /* Generative program for a HMM */
The predict directive formats output using standard printf int main(int argc, char **argv) {
semantics. int states[N];
for (int n=0; n<N; n++) {
states[n] = (n==0) ? discrete_rng(initial_state, K)
2. Probabilistic Programming if (n > 0) {
: discrete_rng(T[states[n-1]], K);

o b s e r v e (normal_lnp(data[n], state_mean[states[n]], 1));


Any program that makes a random choice over the course }
p r e d i c t ("state[%d],%d\n", n, states[n]);
of its execution implicitly defines a prior distribution over }
its random variables; running the program can be inter- return 0;
preted as drawing a sample from the prior. Inference in }

probabilistic programs involves conditioning on observed


data, and characterizing the posterior distribution of the Figure 2. A hidden Markov model (HMM) with 3 underlying
random variables given data. We introduce probabilistic states and Gaussian emissions, observed at 10 discrete time-
programming capabilities into C by providing a library points. We observe 10 data points and predict the marginal distri-
with two primary functions: observe which conditions the bution over the latent state at each time point.
program execution trace given the log-likelihood of a data
point, and predict which marks expressions for which A hidden Markov model example is shown in Figure 2,
we want posterior samples. Any random number genera- in which N = 10 observed data points y1:N are drawn
tor and sampling library can be used for making random from an underlying Markov chain with K latent states, each
choices in the program, any numeric log likelihood value with Gaussian emission distributions with mean µk , and a
can be passed to an observe, and any C expression which (known) K ⇥ K state transition matrix T , such that
can be printed can be reported using predict. The library
includes a single macro which renames main and wraps it z0 ⇠ Discrete([1/K, . . . , 1/K]) (2)
in another function that runs the original in an inner loop in zn |zn 1 ⇠ Discrete(Tzn 1 ) (3)
the forward inference algorithms to be described. yn |zn ⇠ N (µzn , 2
). (4)
Although C is a comparatively low-level language, it can Bayesian nonparametric models can also be represented
nonetheless represent many well-known generative models concisely; in Figure 3 we show a generative program for an
concisely and transparently. Figure 1 shows a simple prob- infinite mixture of Gaussians. We use a Chinese restaurant
abilistic C program for estimating the posterior distribution process (CRP) to sequentially sample non-negative integer
for the mean of a Gaussian, conditioned on two observed partition assignments zn for each data point y1 , . . . , yN .
data points y1 , y2 , corresponding to the model For each partition, mean and variance parameters µzn , z2n
iid are drawn from a normal-gamma prior; the data points yn
µ ⇠ N (1, 5), y1 , y2 ⇠ N (µ, 2). (1)
themselves are drawn from a normal distribution with these
We observe the data y1 , y2 and predict the poste- parameters, defining a full generative model
rior distribution of µ. The functions normal rng and zn ⇠ CRP(↵, z1 , . . . , zn (5)
1)
normal lnp in Figure 1 return (respectively) a normally-
2
distributed random variate and the log probability density 1/ zn ⇠ Gamma(1, 1) (6)
of a particular value, with mean and variance parameters µzn | 2
zn ⇠ N (0, z2n ) (7)
mu and var. The observe statement requires only the log- 2
yn |zn , µzn , ⇠ N (µzn , z2n ). (8)
probability of the data points 8 and 9 conditioned on the zn

current program state; no other information about the like- This program also demonstrates the additional library func-
lihood function or the generative process. In this program tion memoize, which can be used to implement stochastic
we predict the posterior distribution of a single value mu. memoization as described in (Goodman et al., 2008).

You might also like