Bayes Factors Based on Test Statistics and Non-Local Moment Prior Densities

Abstract

We describe Bayes factors based on z, t, $\chi^{2}$, and F statistics when non-local moment prior distributions are used to define alternative hypotheses. The non-local alternative prior distributions are centered on standardized effects. The prior densities include a dispersion parameter that can be used to model prior precision and the variation of effect sizes across replicated experiments. We examine the convergence rates of Bayes factors under true null and true alternative hypotheses and show how these Bayes factors can be used to construct Bayes factor functions. An example illustrates the application of resulting Bayes factors to psychological experiments.

1 Introduction

Bayes factors based on test statistics and first-order non-local prior densities were used in Johnson et al. (2023) (hereafter J23) to define Bayes factor functions (BFFs). The first-order non-local prior densities used to define those Bayes factors contained a single scale parameter that determined the mode of the non-local prior densities used to define the alternative hypotheses (Johnson and Rossell, 2010). BFFs were defined as the mapping of these prior modes to Bayes factors. The use of first-order moment prior densities allowed J23 to obtain closed-form expressions for Bayes factors based on common test statistics, including $z$, $t$, $F$, and $\chi^{2}$ statistics.

In this article we extend the results of J23 by providing closed-form expressions for Bayes factors based on test statistics (BFBOTs) (Johnson, 2005) and alternative hypotheses defined using non-local moment prior densities of arbitrary order. This extension enables the incorporation of subjective prior knowledge regarding the precision of prior estimates of non-null effect sizes. Moreover, it provides a potential mechanism for modeling variation in effect sizes across replicated experiments.

2 BFBOTs and moment prior alternatives

J23 defined Bayes factors using two categories of prior densities: normal-moment prior densities and gamma prior densities. A normal-moment density on a parameter $\lambda$, given hyperparameters $(\tau^{2},r)$, can be expressed as

$$j(\lambda\mid\tau^{2},r) = \frac{(\lambda^{2})^{r}}{(2\tau^{2})^{r+\frac{1}{2}}\Gamma\left(r+\frac{1}{2}\right)}\exp\left(-\frac{\lambda^{2}}{2\tau^{2}}\right),$$ (1)

$$-\infty<\lambda<\infty,\quad\tau,r>0.$$

J23 derived Bayes factors based on $t$ and $z$ statistics by imposing normal-moment prior densities with $r=1$ on the non-centrality parameters of the test statistics under the alternative hypothesis. We extend these results for $r\geq 0$.

BFBOTs for $\chi^{2}_{k}$ and $F_{k,m}$ test statistics were defined by assuming the non-centrality parameter $\lambda$ for each distribution under the alternative hypothesis followed gamma distributions with parameters $k/2+1$ and $1/(2\tau^{2})$, i.e., $G[k/2+1,1/(2\tau^{2})]$. We generalize these results by extending the class of gamma priors to include $G[k/2+r,1/(2\tau^{2})]$ distributions for $r,\tau>0$.

For one-sided tests with positive non-centrality parameters, the prior densities $j^{+}(\mu\mid\tau^{2},r)$ and $j^{-}(\mu\mid\tau^{2},r)$ are defined similarly to the expression in (1), but they are constrained to the positive or negative real line, respectively. This constraint depends on whether the alternative hypothesis necessitates that $\lambda>0$ or $\lambda<0$.

For simplicity we adopt the notation used in J23 and define

• $N(a,b)$ as a normal distribution with mean $a$ and variance $b$.

• $T_{\nu}(\lambda)$ as a $t$ distribution with $\nu$ degrees of freedom and non-centrality parameter $\lambda$.

• $\chi^{2}_{\nu}(\lambda)$ as a $\chi^{2}$ distribution with $\nu$ degrees of freedom and non-centrality parameter $\lambda$.

• $F_{k,m}(\lambda)$ as an $F$ distribution with $(k,m)$ degrees of freedom and non-centrality parameter $\lambda$.

• $G(\alpha,\lambda)$ as a gamma distribution with shape parameter $\alpha$ and rate parameter $\lambda$.

• $J(\tau^{2},r)$ as a normal-moment distribution of order $r$ and rate parameter $\tau^{2}$ and density given in (1).

In addition, we let ${{}_{1}}F_{1}(a,b;z)$ denote the confluent hypergeometric function and ${{}_{2}}F_{1}(a,b,c;z)$ the Gaussian hypergeometric function.

Table 1: Summary of theorems describing Bayes factors based on test statistics. The constants $a$, $b$ and $c$ are defined as $a=\left(1+\tau^{2}\right)^{-\left(r+{\frac{1}{2}}\right)}$, $b=\left(1+\tau^{2}\right)^{-\left(\frac{k}{2}+r\right)}$, and $c=\left[\Gamma(\frac{\nu}{2}+1)\,\Gamma(r+1)\right]/\left[\Gamma(\frac{\nu+1}{2})\,{\Gamma(r+{\frac{1}{2}})}\right]$.

Test	$H_{0}$	$H_{1}$	Prior	$BF_{10}$
Two-sided z	$z\sim N(0,1)$	$z\sim N(\lambda,1)$	$J(\lambda{\,|\,}{\tau^{2}},r)$	$a\,F_{1}\left(r+{\frac{1}{2}},{\frac{1}{2}};\frac{\tau^{2}z^{2}}{2(1+\tau^{2})}\right)$
One-sided z	$z\sim N(0,1)$	$z\sim N(\lambda,1)$	$J^{+}(\lambda{\,|\,}{\tau^{2}},r)$	$a\,\left[{{}_{1}}F_{1}\left(r+{\frac{1}{2}},{\frac{1}{2}};y^{2}\right)+2y\frac{\Gamma(r+1)}{\Gamma(r+{\frac{1}{2}})}{{}_{1}}F_{1}\left(r+{\frac{1}{2}},\frac{3}{2};y^{2}\right)\right]$,
				with $y=\tau z(2+2\tau^{2})^{-{\frac{1}{2}}}$
Two-sided $t$	$t\sim T_{\nu}(0,1)$	$t\sim T_{\nu}(\lambda,1)$	$J(\lambda{\,|\,}{\tau^{2}},r)$	$a\,\left[{{}_{2}}F_{1}\left(\frac{\nu+1}{2},r+{\frac{1}{2}},{\frac{1}{2}};y^{2}\right)+cy\ \,{{}_{2}}F_{1}\left(\frac{\nu}{2}+1,r+1,\frac{3}{2};y^{2}\right)\right]$
				with $y=\tau t\,[(\nu+t^{2})(1+\tau^{2})]^{-{\frac{1}{2}}}$
One-sided $t$	$t\sim T_{\nu}(0,1)$	$t\sim T_{\nu}(\lambda,1)$	$J^{+}(\lambda{\,|\,}{\tau^{2}},r)$	$a\,\left[{{}_{2}}F_{1}\left(\frac{\nu+1}{2},r+{\frac{1}{2}},{\frac{1}{2}};y^{2}\right)+2cy\ \,{{}_{2}}F_{1}\left(\frac{\nu}{2}+1,r+1,\frac{3}{2};y^{2}\right)\right]$
				with $y=\tau t\,[(\nu+t^{2})(1+\tau^{2})]^{-{\frac{1}{2}}}$
$\chi^{2}_{k}$	$h\sim\chi^{2}_{k}(0)$	$h\sim\chi_{k}^{2}(\lambda)$	$G\left(\lambda{\,|\,}\frac{k}{2}+r,\frac{1}{2\tau^{2}}\right)$	$b\ \,{{}_{1}}F_{1}\left(\frac{k}{2}+r,\frac{k}{2};\frac{\tau^{2}h}{2(1+\tau^{2})}\right)$
$F_{k,m}$	$f\sim F_{k,m}(0)$	$f\sim F_{k,m}(\lambda)$	$G\left(\lambda{\,|\,}\frac{k}{2}+r,\frac{1}{2\tau^{2}}\right)$	$b\ \,{{}_{2}}F_{1}\left(\frac{k}{2}+r,\frac{k+m}{2},\frac{k}{2};\frac{k\tau^{2}f}{(1+\tau^{2})(m+kf)}\right)$

Using this notation, the main results of this article are provided in Table 1, which provides explicit expressions for Bayes factors based on $z$, $t$, $\chi^{2}$ and $F$ tests. These expressions are justified by theorems provided in the Supplemental Material.

In common applications of these tests, the convergence rates for the resulting Bayes factors may be summarized as follows.

1. Under true alternative hypotheses (i.e., $\lambda\neq 0$), Bayes factors in favor of null hypotheses decrease exponentially fast to 0 as the sample size $n\rightarrow\infty$.

2. Under true null hypotheses and certain regularity conditions, Bayes factors in favor of alternative hypotheses often decrease at rate $O_{p}(n^{-r-{\frac{1}{2}}})$ for $z$ and $t$ tests and $O_{p}(n^{-r-\frac{k}{2}})$ for $\chi_{k}^{2}$ and $F_{k,m}$ tests.

Sufficient conditions for achieving these rates are provided in the Supplemental Material.

The improved convergence rates obtained for $r>1$ provide a partial motivation for generalizing the class of prior densities considered in J23. The non-local alternative prior densities proposed there correspond to setting $r=1$ in Table 1. The improvement in convergence rates for true null hypotheses can be attributed to the more rapid descent of the non-local prior densities to 0 as the non-centrality parameters converge to $0$.

3 Selection of prior hyperparameters

We now describe strategies for specifying $(\tau^{2},r)$ to define either a Bayes factor or to construct a BFF. We assume that the non-centrality parameter $\lambda$ of the test statistic can be expressed as a function of $\omega$, a standardized effect. That is, $\lambda=\psi(\omega)$. For example, in a $z$ test that a normal mean $\mu$ is 0, the non-centrality parameter satisfies $\lambda=\psi(\omega)=\sqrt{n}\omega$, where $\omega=\mu/\sigma$ and $n$ is the sample size.

Our recommendation for setting $(\tau^{2},r)$ is to fix $\tau^{2}$ conditionally on $r$, and then set $\tau^{2}=\tau^{2}_{\omega,r}$ so that the prior mode on the non-centrality parameter equals $\psi(\omega^{*})$, where $\omega^{*}$ represents a hypothesized value of $\omega$. That is, we define $\tau_{\omega^{*},r}^{2}$ such that

$$\psi(\omega^{*})=\underset{\lambda}{\operatorname{argmax}}\ \pi(\lambda|\tau^{2}_{\omega^{*},r},r).$$ (2)

Here, $\pi$ denotes the prior density on $\lambda$ under the alternative hypothesis. This constraint places the prior mode for $\lambda$ at the specified value of $\psi(\omega^{*})$. In constructing a BFF, $\omega^{*}$ is varied over a plausible range of values for the standardized effect.

Like many default Bayesian testing methods, criteria for setting the scale of the prior density used to define alternative hypotheses remains a topic of active research (e.g., Zellner, 1986; Doucet et al., 2002; Liang et al., 2008; Rouder et al., 2009; Consonni et al., 2018; Pramanik and Johnson, 2023). For the purposes of this article, we simply examine the sensitivity of Bayes factors and BFFs to the choice of $r$. Methods for estimating $r$ from published findings of similar studies or in replicated designs are currently under investigation.

To illustrate this strategy for setting $\tau^{2}$ for given $r$, consider a $t$ test of a null hypothesis $H_{0}:\mu=0$ based on a random sample $x_{1},\dots,x_{n}$, where $x_{i}\sim N(\mu,\sigma)$ and $\sigma^{2}$ is unknown. For this test, $t=\sqrt{n}\bar{x}/s$, where $s^{2}$ is the usual unbiased estimate of $\sigma^{2}$. The distribution of $t$ is

$$t\mid\mu,\sigma\sim T_{\nu}\left(\frac{\sqrt{n}\mu}{\sigma}\right),$$ (3)

where $\nu=n-1$. Under the null hypothesis, $t\sim T_{\nu}(0)$. The non-centrality parameter of the $t$ distribution under the alternative hypothesis is $\lambda=\sqrt{n}\omega$, where we define $\omega=\mu/\sigma$ as the standardized effect.

The prior distribution recommended for $\lambda$ in Table 1 is a normal-moment prior, $J(\tau^{2},r)$. The modes of this density are $\pm\sqrt{2r}\tau$. To define a Bayes factor given a value $\omega^{*}$, we select $\tau^{2}$ so that the modes of the prior occur at $\sqrt{n}\omega^{*}$. That is, we equate $\sqrt{n}\omega^{*}=\sqrt{2r}\tau$ and define $\tau^{2}_{\omega^{*},r}=n{\omega^{*}}^{2}/(2r)$.

A similar procedure can be used to set $\tau^{2}_{\omega^{*},r}$ for other test statistics. Table 2 lists values of $\tau^{2}_{\omega^{*},r}$ for several common statistical tests. This table generalizes the values provided in Table 1 of J23 for the case $r=1$.

Figure 1 illustrates the effect of varying $r$ so that the mode of the prior density on an effect size remains constant. It shows that the prior dispersion around the mode decreases as $r$ increases.

Figure 1: Normal moment prior densities with varying $r$

Table 2: Default choices of $\tau_{\omega,r}$

Test	Statistic	Standardized Effect ($\omega$)	$\tau^{2}_{\omega,r}$
1-sample z	$\frac{\sqrt{n}\bar{x}}{\sigma}$	$\frac{\mu}{\sigma}$	$\frac{n\omega^{2}}{2r}$
1-sample t	$\frac{\sqrt{n}\bar{x}}{s}$	$\frac{\mu}{\sigma}$	$\frac{n\omega^{2}}{2r}$
2-sample z	$\frac{\sqrt{n_{1}n_{2}}(\bar{x}_{1}-\bar{x}_{2})}{\sigma\sqrt{n_{1}+n_{2}}}$	$\frac{\mu_{1}-\mu_{2}}{\sigma}$	$\frac{n_{1}n_{2}\omega^{2}}{2r(n_{1}+n_{2})}$
2-sample t	$\frac{\sqrt{n_{1}n_{2}}(\bar{x}_{1}-\bar{x}_{2})}{s\sqrt{n_{1}+n_{2}}}$	$\frac{\mu_{1}-\mu_{2}}{\sigma}$	$\frac{n_{1}n_{2}\omega^{2}}{2r(n_{1}+n_{2})}$
Multinomial/Poisson	$\chi^{2}_{\nu}=\sum\limits_{i=1}^{k}\frac{(n_{i}-nf_{i}(\hat{\theta}))^{2}}{nf_{i}(\hat{\theta})}$	$\left(\frac{p_{i}-f_{i}(\theta)}{\sqrt{f_{i}(\theta)}}\right)_{k\times 1}$	$\frac{n\omega^{\prime}\omega}{2(\frac{k}{2}+r-1)}=\frac{nk\tilde{\omega}^{2}}{2(\frac{k}{2}+r-1)}$
Linear model	$F_{k,n-p}=\frac{(RSS_{0}-RSS_{1})/k}{[(RSS_{1})/(n-p)]}$	$\frac{\mathbf{L}^{-1}(\mathbf{A}\boldsymbol{\beta}-\mathbf{a})}{\sigma}$	$\frac{n\omega^{\prime}\omega}{2(\frac{k}{2}+r-1)}=\frac{nk\tilde{\omega}^{2}}{4(\frac{k}{2}+r-1)}$
Likelihood Ratio	$\chi^{2}_{k}=-2\log\left[\frac{l(\theta_{r0},\hat{\theta_{s}})}{l(\hat{\theta})}\right]$	${\bf L}^{-1}(\theta_{r}-\theta_{r0})$	$\frac{n\omega^{\prime}\omega}{2(\frac{k}{2}+r-1)}=\frac{nk\tilde{\omega}^{2}}{2(\frac{k}{2}+r-1)}$

4 Examples

To demonstrate the application of the Bayes factors described in Table 1 and the procedure for setting $(\tau^{2},r)$ described in Section 3, we re-analyzed an experiment from the Many Labs 3 project (Ebersole et al., 2016).

The experiment measured the effect on response time for subjects performing the Stroop task (Ebersole et al., 2016). This effect is among the strongest and most widely replicated effects in experimental psychology. The Stroop task requires subjects to identify the color of the type of printed words. This task takes longer when there is discordance between the type’s color and the color’s name. For example, responding red takes longer than blue. After preprocessing the data to account for unusually long response times and incorrect answers (Greenwald et al., 2003), the authors “calculated the average response time for all correct responses separately for congruent and incongruent trials" and then “replaced response latencies for trials with errors using the mean of correct responses in that condition plus 600 ms.” They then “recomputed the means for congruent and incongruent trials overall” and used the difference between these two means divided by the standard deviation of all correct trials regardless of condition” to construct paired $t$ statistics.

Twenty replications of this experiment were replicated in the Many Labs experiment. For illustration, we begin by analyzing results from the first experimental site where the $t$ statistic was 9.38 on $83$ degrees of freedom (Table 3).

The null hypothesis in this study is that the mean difference in response times for the congruous and incongruous conditions is 0. Under the alternative hypothesis, we assume that the population mean difference in response times is $\mu$ and that the observational variance is $\sigma$, and let $\omega=\mu/\sigma$ denote the standardized effect size. Given $\omega>0$, we assume that the non-centrality parameters of the distributions of the $t$ statistic, $\lambda$, is drawn from a normal-moment distribution,

$$J^{+}\left(r+\frac{1}{2},\frac{1}{2\tau^{2}_{\omega,r}}\right),$$ (4)

where, from Table 2, we set

$$\tau^{2}_{\omega,r}=\frac{n\omega^{2}}{2r}.$$ (5)

Figure 2 displays the plot of BFFs obtained for $r=1,5,10$, and $15$ using the theoretical results from Tables 1 and 2.

Several aspects of this figure merit comment. As expected, the BFF curves reflect more evidence in favor of alternative hypotheses corresponding to $\omega\approx 1$ as $r$ increases. This happens because alternative hypotheses concentrate more of their prior mass on the hypothesized values of $\omega$ as $r$ increases. Indeed, as $r\rightarrow\infty$, all priors converge to a point mass on the hypothesized value of $\omega$, and the BFF curves reduce to a plot of the log-likelihood ratio (based on the test statistic). Nonetheless, all BFF curves reflect very strong evidence for alternative hypotheses defined by priors centered on standardized effect sizes greater than about 0.05, and the curves for $r=5,10,15$ are relatively insensitive to the choice of $r$ within this range.

Figure 2: Bayes factor functions for Stroop test for varying $r$.

We now consider how estimates of $r$ might be used to construct BFFs when data from replicated sites is available. In this case, data from 20 Many Labs consortium sites are available; $t$ statistics from the 20 sites are listed in Table 3. For simplicity, we took a naive empirical Bayes approach to estimate $r$ (see Supplemental Materials), leading to an estimate of $r=9.99$. This value of $r$ suggests that standardized effects were relatively consistent across sites.

Figure 3 provides the BFF based on the empirical Bayes estimate of $r$ and the $t$ statistics from all 20 sites. For comparison, the BFF for $r=1$ is also displayed. In this case, using the empirical Bayes estimate of $r$ had a moderate effect in increasing the BFF near its maximum of $\omega=0.89$. With 20 experimental sites each exhibiting very strong evidence for a Stroop effect, the evidence against the null hypothesis is overwhelming except for alternative hypotheses concentrating prior mass on $\omega$ near 0.

Table 3: T statistics and degrees-of-freedom for the Stroop task [t-statistic (degrees of freedom)].

9.38(83)	9.85(118)	7.36(43)	11.62(90)
7.85(95)	12.56(317)	11.01(123)	10.15(130)
13.52(157)	10.14(100)	8.90(116)	10.37(141)
11.68(177)	9.11(118)	16.97(241)	8.82(136)
8.46(88)	5.93(80)	12.17(193)	9.37(94)

Figure 3: The combined BFF using the MOM estimator of $r$ for the 20 sites in the test set is depicted by the solid black line. The combined BFF using $r=1$ is shown by the black dashed line, while the BFFs for each individual study (using $r=1$) are illustrated by the colored lines.

5 Discussion

Bayes factors based on test statistics present compelling advantages over standard methods for calculating Bayes factors. They eliminate computational challenges that arise when Bayes factors are computed from complex statistical models, and they reduce subjectivity when defining prior distributions on model parameters. By indexing Bayes factors based on test statistics according to standardized effects, BFFs eliminate much of the subjectivity associated with specifying a single alternative model. They also provide users with a simple representation of the statistical evidence in favor of alternative hypotheses centered on a range of effect sizes.

The BFFs proposed in J23 lack flexibility in allowing scientists to incorporate the precision of their estimates of effect sizes into Bayesian hypothesis tests. This article addresses this shortcoming by expanding the class of prior distributions used in defining BFFs. In particular, it provides analytic expressions for Bayes factors based on test statistics in conjunction with more general classes of prior distributions and illustrates how BFFs can be constructed from these broader classes. Importantly, we demonstrate how the scale parameters of the prior distributions can be linked so that the modes of prior distributions are located at hypothesized effect sizes.

J23 demonstrated how test statistics from replicated experiments could be combined to generate an aggregated BFF. This article’s results will permit that methodology to account for the dispersion of effect sizes across replications of similar experiments. Efficient and coherent procedures for incorporating such information are currently under investigation.

The BFF package (https://CRAN.R-project.org/package=BFF), available from the CRAN depository, provides R functions to compute the BFFs reported in this article.