Sensitivity analysis for misclassification in logistic regression via likelihood methods and predictive value weighting

2.1. The no-covariate case: basic sensitivity analysis

To begin, consider estimating a crude odds ratio corresponding to a potentially misclassified binary risk factor. Table I revisits data from an often-cited case-control study of the relationship between maternal antibiotic use during pregnancy and the odds of sudden infant death syndrome (SIDS) [5--7, 16]. The observed exposure data reflect self-reported antibiotic use (Z), whereas the aim is to associate the true binary variable of antibiotic use (X) with SIDS status (Y).

The observed odds ratio (OR) associated with Table I is 1.309, corresponding to an estimated ln(OR) (std. error) of 0.269 (0.150). A sensitivity analysis to examine potential effects of misclassification would be based on one or more sets of assumptions about the underlying sensitivity (SE) and specificity (SP) parameters. These consist of SE=Pr(Z =1|X =1) and SP=Pr(Z =0|X =0) or, for differential misclassification [17], SE_y=Pr(Z =1|X =1, Y = y) and SP_y=Pr(Z =0|X =0, Y = y)(y=0, 1).

In this simple 2×2 case-control setting, the most straightforward approach to sensitivity analysis may be to base it directly on writing familiar ‘matrix-method’-type identities in terms of probabilities [3, 18]. In particular, it is easily shown that

where π_y =Pr(X =1|Y = y) and ${\pi }_y^{\ast }=\mathrm{Pr}(Z=1\mid Y=y)$ (y =0,1). Given estimates of ${\pi }_y^{\ast }$ from the observed data, one may supply values for SE_y and SP_y and then calculate the resulting true exposure probabilities via (1). The estimate $\left(\widehat{\mathrm{OR}}\right)$ follows as π₁(1−π₀)[π₀(1−π₁)]⁻¹, where typically we would examine several (SE_y, SP_y) combinations to obtain a range of estimated ORs. The delta method may readily be used to estimate the variance of each ln(OR) estimate, accounting for uncertainty in the estimates ${\widehat{\pi }}_1^{\ast }$ and ${\widehat{\pi }}_0^{\ast }$ based on the observed data.

Equation (1) is also useful in that it implies two important restrictions that should be considered when conducting the analysis:

Choosing values of SE_y and SP_y that are incompatible with these restrictions (upon replacing the ${\pi }_y^{\ast }$’s by their estimates) will generally lead to numerical issues with the sensitivity analysis.

As an alternative procedure for determining the sensitivity analysis-based log(OR) estimate and its standard error, one could also adopt an ML approach. For instance, assume the underlying logistic regression model of interest:

where primary interest is in the log(OR) parameter (β). Observed-data likelihood contributions corresponding to investigator-supplied SE_y and SP_y values are expressible in a traditional manner applicable to covariate measurement error settings [1], with a slight adjustment to account for potentially differential misclassification:

Note that the first term on the right side of (4) is determined by the assumed (SE_y, SP_y) values, whereas the second term follows from model (3) and the last is an estimable nuisance parameter. Numerically maximizing the log-likelihood determined by the contributions in (4) provides $\widehat{\beta }=\ln \left(\widehat{\mathrm{OR}}\right)$ with its standard error estimable by approximating the observed information matrix.

In the following section, we outline a third approach to sensitivity analysis for covariate misclassification that facilitates implementation and generalization, while making an intriguing alternative to the direct ML approach based on equation (4).

2.2. The no-covariate case: predictive value weighting

While basing sensitivity analysis on the matrix-method identity in (1) may offer the most direct route in the no-covariate setting, several authors [10, 11] have proposed methods based on positive and negative predictive values. The basic idea is to ‘reconstruct’ data that might have been observed under no misclassification, by using the observed data (e.g. Table I) together with assumptions about (SE_y, SP_y) to arrive at estimates of PPV_y =Pr(X =1|Z =1, Y = y) and NPV_y =Pr(X =0|Z =0, Y = y). Once these predictive value estimates based on the observed data and the assumed (SE, SP) values are in hand, we note that the following identity analogous to (1) may be applied to execute sensitivity analysis:

This identity underlies the ‘inverse’ matrix method [6], which proves particularly useful when adjusting for differential misclassification in the presence of internal validation data [18, 19].

The use of (5) for sensitivity analysis is less direct than using (1), because of an additional step needed to obtain the appropriate PPV_y and NPV_y values by combining the assumed (SE_y, SP_y) combinations with the estimates of ${\pi }_y^{\ast }$ (y =0,1). One can show that PPV_y and NPV_y are found as the solution to two linear equations, represented compactly as follows:

where

and we replace the ${\pi }_y^{\ast }$’s by observed data-driven estimates. Use of this result in conjunction with (5) would also complicate standard error calculations relative to using (1). However, the estimated predictive values in (6) are the key to the approach advocated henceforth, in which we use them as weights for fitting a desired logistic regression model while adjusting for covariates.

To outline the proposed approach in the no-covariate case, consider viewing each (Z,Y) record in Table I as reflecting two possible (X,Y) records (one with x =0, the other with x =1). Thus, the four possible (Z,Y) pairs yield a total of eight (X,Z,Y) combinations, each of which we weight appropriately as indicated in Table II. The fourth column in Table II defines the predictive value weights that are central to our approach, whereas the two right-most columns provide the estimated values of these weights corresponding to the observed data in Table I for the non-differential and differential example scenarios detailed in Section 3.1.

The appeal of this approach is that the desired model associating Y with the true binary exposure variable (X) may now be fit directly to the (y, x) data in Table II. Formally, we maximize the weighted log-likelihood given by

where the n_zy’s represent the observed-data cell counts where Z = z and Y = y (z, y =0, 1) and l_xy(θ) is the log-likelihood contribution for a (y, x) pair. In practice, this simply means that we fit the logistic regression model (3) to the (y, x) data in Table II, applying the weights (w_xzy). This can be accomplished via standard software, e.g. by direct application of the SAS LOGISTIC procedure with a ‘WEIGHT’ statement [20] to the expanded data set. This straightforward one-step weighting process removes any need for simulation (e.g. [11]). Note that we fit the logistic model to a total of twice the number of observations as originally obtained (i.e. 2288 for the Table I example), since each observation is replicated to allow either x =0 or x =1.

To obtain a standard error accounting for the uncertainty in the observed-data estimates of ${\pi }_y^{\ast }$ (y=0, 1) used to compute the weights, we suggest a standard bootstrap [21] or jackknife [21, 22] procedure. We use the jackknife estimator henceforth, as we find it much less susceptible to numerical problems. Specifically, the potential for conflicts with the necessary restrictions in (2) is very small when employing the jackknife but can be substantial under bootstrap sampling. For each ‘leave one out’ sample from the original observed data, we re-calculate the weights via (6) and re-fit the weighted [summationtext] [summationtext] logistic regression in (7) using standard software. The jackknife standard error is calculated based on the $n={\sum }_{z=0}^1{\sum }_{y=0}^1{n}_{zy}$ resulting θ estimates.

2.3. Outcome misclassification in the no-covariate case

When the outcome (Y) rather than the exposure (X) is potentially misclassified, the problem has been more broadly discussed in the literature [15, 23, 24]. Magder and Hughes [15] facilitated EM algorithm-based ML analysis via weighting of observations in a manner somewhat akin to our proposal for risk factor misclassification in the previous section. We favor a direct ML approach for outcome misclassification, given the straightforward manner in which the appropriate likelihood can be specified and numerically maximized. Assume that we desire to fit model (3), but that an error-prone variable Y* is observed in place of the true binary outcome Y. The sensitivity and specificity of Y* are given by SE_x =Pr(Y* =1|Y =1, X = x) and SP_x =Pr(Y* =0|Y =0, X = x) (x =0, 1). Observations contribute to the likelihood as follows:

with the first term given by the assumed (SE_x, SP_x) values and the second by the desired model (3). This function is readily handled using standard optimization routines such as those available in SAS IML [25], Splus [26], and R [27]. Standard errors for all parameters in model (3) are also readily obtained using such software via close numerical approximations to the observed information matrix. Note that when specifying SE_x and SP_x, restrictions identical to those in (2) should be observed, except with ‘x’ replacing ‘y’ in the subscripts.

2.4. Adjusting for covariates: predictive value weighting and ML

We now turn to the more common setting in which interest lies in one or more covariate-adjusted odds ratios, normally estimable by the following multiple logistic regression model:

As before, we assume that X is subject to misclassification and that one only has access to the error-prone binary variable Z. We assume that the covariates C_j(j =1, …, J) are observed without error.

Although one might initially contemplate covariate-adjusted sensitivity analysis using a generalization of equation (1), there are some fundamental drawbacks to such a strategy. A more promising approach is to generalize the likelihood in equation (4). In this case, the investigator specifies the parameters SE_yc=Pr(Z =1|X =1, Y =y, C=c) and SP_yc=Pr(Z =0|X =0, Y =y, C=c) (x, y, z=0, 1). Observed-data likelihood contributions corresponding to these specified values are given by Pr(Y =y, Z =z|C=c), which may be expressed as follows:

As before, the first term in (10) reflects the assumed (SE_yc, SP_yc) values, whereas the second term reflects the desired model (i.e. (9)). The last term might be modeled, e.g., via a second logistic regression model of X on C.

Alternatively, the predictive value weighting approach introduced in Section 2.2 remains conceptually straightforward and accessible in the covariate-adjusted case based on the following multivariable extensions of (5) and (6):

and

where

and ${\pi }_{y\mathbf{c}}^{\ast }=\mathrm{Pr}(Z=1\mid \mathbf{C}=\mathbf{c},Y=y)$. Here, the predictive value parameter definitions have been generalized in the expected manner, i.e. PPV_yc =Pr(X =1|Z =1, Y = y, C=c) and NPV_yc =Pr(X =0|Z =0, Y = y, C=c). With the (SE_yc, SP_yc) parameters specified by the investigator, the method proceeds exactly as before. We propose estimating the ${\pi }_{y\mathbf{c}}^{\ast }$’s via logistic regression, with attention to model fitting. For example, the assessment of potential interactions between Y and the C_j’s may validate the model

Otherwise, model (13) may be enriched with product terms to account for interactions present in the observed data, in the interest of valid ${\pi }_{y\mathbf{c}}^{\ast }$ estimates for use in (12). Note that predictive value weighting requires a model for Z|(Y, C), whereas ML via (10) requires a model for X|C.

We emphasize again the need for caution with respect to restrictions implied by the extension to equation (1) to incorporate covariates. Analogous to (2), it follows that

for all (y, c) combinations. Investigator-supplied values of SE_yc or SP_yc that fail to honor these restrictions (when replacing the ${\pi }_{y\mathbf{c}}^{\ast }$’s by their estimates) can contribute to numerical problems with ML based on (10), and produce predictive value weights via (12) that are outside the (0, 1) range. Thus, it is reasonable to allow the observed data to inform one’s (SE, SP) choices, as suggested in (14). When all predictor variables are categorical in nature, it is straightforward to incorporate these restrictions into the selection of SE_yc or SP_yc. With one or more continuous predictor (C) variables, we suggest the use of the following adjustments for this purpose:

where ${\mathrm{SE}}_{y\mathbf{c}}^0$ and ${\mathrm{SP}}_{y\mathbf{c}}^0$ are the a priori investigator-supplied values and ${\widehat{\pi }}_{y\mathbf{c}}^{\ast }$ is estimated, e.g., via the logistic regression of Z on (Y, C) in (13). In the case of predictive value weighting, the adjustments in (15) simply equate to setting negative weights produced by the original $({\mathrm{SE}}_{y\mathbf{c}}^0,{\mathrm{SP}}_{y\mathbf{c}}^0)$ values equal to 0, while setting weights in excess of 1 equal to 1.

2.5. Adjusting for covariates: outcome misclassification

For covariate-adjusted sensitivity analysis to outcome misclassification via model (9), we prefer a direct likelihood approach as in Section 2.3. The form for the likelihood contributions becomes

where the second term in the summation follows from (9) and the first term is determined by the user-specified values SE_xc =Pr(Y* =1|Y =1, X = x, C=c) and SP_xc =Pr(Y* =0|Y =0, X = x, C=c) (x=0, 1). Restrictions as in (14) apply, except again with ‘x’ replacing ‘y’ in the subscripts.

2.6. Extensions

An advantage of the predictive value weighting approach is that extensions to more complex scenarios are conceptually straightforward. For example, suppose one was interested in sensitivity analysis in a situation where two binary predictors (X₁ and X₂) are subject to misclassification via corresponding observed surrogates Z₁ and Z₂. The appropriate expanded data set would consist of four (y, x₁, x₂, c) records in place of each observed (y, z₁, z₂, c) record, with each observation weight of the form Pr(X₁ = x₁, X₂ = x₂|Z₁ = z₁, Z₂ = z₂, Y = y, C=c). Though more challenging than in the case of a single misclassified predictor, the calculation of these weights in analogous fashion could be facilitated to the extent that the investigator is willing to make simplifying assumptions about the misclassification processes (e.g. independent and/or non-differential misclassification of X₁ and X₂). Similarly, suppose we were concerned with misclassification of the outcome (Y) in addition to one binary predictor (X). In that case, each observed (y*, z, c) record could be replaced by four (y, x, c) records, with weights of the form P(Y = y, X = x|Y* = y*, Z = z, C=c). Again, estimation of these weights might be facilitated by certain plausible simplifying assumptions. The form of the weights suggests that this latter scenario may only be defensible under cross-sectional study designs.

3.1. The no-covariate case

For the data in Table I, we have ${\widehat{\pi }}_1^{\ast }=0.2163\left(0.0173\right)$ and ${\widehat{\pi }}_0^{\ast }=0.1741\left(0.0157\right)$. Suppose the investigator assumes non-differential misclassification, with SE=0.6 and SP=0.9. It follows from (1) that ${\widehat{\pi }}_1=0.2326\left(0.0346\right)$ and ${\widehat{\pi }}_0=0.1482\left(0.0314\right)$ so that $\widehat{\mathrm{OR}}=1.742$ and $\ln \left(\widehat{\mathrm{OR}}\right)=0.555\left(0.315\right)$, with the latter standard error obtained by the delta method. In contrast, suppose we had specified the following differential conditions: SE₁=0.6, SP₁=0.8, SE₀=0.7, SP₀=0.9. Using (1), we obtain ${\widehat{\pi }}_1=0.0408\left(0.0433\right)$ and ${\widehat{\pi }}_0=0.1235\left(0.0262\right)$ so that $\ln \left(\widehat{\mathrm{OR}}\right)=-1.199(1,133)$, with $\widehat{\mathrm{OR}}=0.302$. Note the dramatic shift in implications assuming differential misclassification.

Applying direct ML (see equation (4)) to the data in Table I yields $\widehat{\beta }=\ln \left(\widehat{\mathrm{OR}}\right)=0.555\left(0.316\right)$ and $\widehat{\beta }=\ln \left(\widehat{\mathrm{OR}}\right)=-1.199\left(1.135\right)$ under the non-differential and differential assumptions outlined above, respectively. Thus, results via the matrix-method identity in (1) with a delta method-based standard error are effectively identical to ML utilizing (4).

Application of the SAS LOGISTIC procedure with a ‘WEIGHT’ statement [20] to the expanded data set in Table II produces solutions identical to those of the matrix-method and likelihood-based approach: $\widehat{\beta }=\ln \left(\widehat{\mathrm{OR}}\right)=0.555$ and $\widehat{\beta }=\ln \left(\widehat{\mathrm{OR}}\right)=-1.199$ for the non-differential and differential cases, respectively. To estimate standard errors accounting for uncertainty in ${\widehat{\pi }}_1^{\ast }$ and ${\widehat{\pi }}_0^{\ast }$, we obtained 1144 $\ln \left(\widehat{\mathrm{OR}}\right)$ estimates using the proposed weighting method, each one calculated by leaving out a unique (z_i, y_i) pair. The jackknife standard error estimate [22] is 0.318 in the non-differential case and 1.170 in the differential case, closely matching the estimates obtained via the matrix-method and ML approaches.

Figure 1 displays ln(OR) estimates determined by predictive value weighting, with bars indicating ±1 jackknife standard error, for several (SE, SP) combinations characterizing non-differential exposure misclassification in the SIDS study data in Table I. Note that this more extensive sensitivity analysis is particularly informative in that it indicates very little change in the ln(OR) estimate as SE varies for a fixed SP (left panel of Figure 1), but dramatic changes as SP varies for a fixed SE (right panel). In exploring a greater variety of (SE, SP) scenarios in practice, one may wish to consider displaying the results via contour plots (e.g. [12]).

To illustrate sensitivity analysis for outcome misclassification, let us now pretend that sampling for the SIDS study had been done in a prospective or cross-sectional manner and that the outcome (Y) rather than the exposure (X) was error-prone. We then view the data in Table I as depicting Y* vs. X. Numerically maximizing the likelihood based on equation (8) assuming non-differential error with SE=0.6 and SP=0.9 yields $\widehat{\beta }=\ln \left(\widehat{\mathrm{OR}}\right)=0.982\left(0.727\right)$, so that $\widehat{\mathrm{OR}}=2.67$. Assuming instead a differential scenario with SE₁=0.6, SP₁=0.8, SE₀=0.7, and SP₀=0.9 gives $\widehat{\beta }=\ln \left(\widehat{\mathrm{OR}}\right)=1.33\left(0.735\right)$, so that $\widehat{\mathrm{OR}}=3.78$. In both of these cases, the natïve OR estimate 1.31 would be markedly attenuated.

3.2. Adjusting for a binary covariate

For the purpose of illustration, Table III shows the original SIDS data of Table I, stratified based on an artificial binary covariate C. Equation (9) gives the desired model, with X as the predictor of primary interest and C as the sole covariate. Estimates of ${\pi }_{y\mathbf{c}}^{\ast }$’s were obtained via model (13), after first verifying non-significance of a Y*C interaction term. These estimates are as follows: ${\widehat{\pi }}_{11}^{\ast }=0.272$, ${\widehat{\pi }}_{10}^{\ast }=0.178$, ${\widehat{\pi }}_{01}^{\ast }=0.233$, ${\widehat{\pi }}_{00}^{\ast }=0.150$.

Table IV provides definitions and estimates for the predictive value weights in this example, based on two distinct sets of assumed SE_yc and SP_yc values. The first case assumes non-differentiality with respect to both Y and C, such that SE_yc =0.7, SP_yc =0.9 (y =0, 1;c =0,1). The second illustrates a differential case in which SE and SP are allowed to vary with Y and C, as follows: SE₁₁ =0.75, SE₁₀ =0.75, SE₀₁ =0.8, SE₀₀ =0.8, SP₁₁ =0.85, SP₁₀ =0.85, SP₀₁ =0.9, SP₀₀ =0.9. The natïve estimates and standard errors for the coefficients corresponding to X and C (β and γ) are 0.206 (0.152) and 0.497 (0.126), respectively. This yields natïve adjusted OR estimates of 1.22 (X) and 1.64 (C). In contrast, in the non-differential case the predictive value weighting approach yields $\widehat{\beta }=0.406\left(0.318\right)$ and $\widehat{\gamma }=0.456\left(0.134\right)$, corresponding to adjusted OR estimates of 1.50 and 1.58 (standard errors obtained via the jackknife). In the differential case, we obtain $\widehat{\beta }=-0.156\left(0.415\right)$ and $\widehat{\gamma }=0.538\left(0.140\right)$, corresponding to adjusted OR estimates of 0.86 and 1.71. Note here that non-differential misclassification implies attenuation of the naïve β estimate, whereas the assumed differential misclassification pattern yields a naïve estimate on the opposite side of the null relative to the adjusted estimate. The estimates of γ are also impacted by misclassification of X, though more modestly than those of β.

3.3. Adjusting for multiple and continuous covariates

In this section, we use an example data set from the logistic regression text by Hosmer and Lemeshow [28]. The original data on 380 subjects with prostate cancer were obtained at The Ohio State University Comprehensive Cancer Center, and were modified to protect confidentiality. We utilize data from 378 patients (two men with Gleason score values of 0 were removed). The outcome (Y) is a binary indicator for whether or not cancer had penetrated the prostatic capsule, and the predictor variables that we consider are the patient’s Gleason score and prostate specific antigen (PSA) value (C₁ and C₂; both continuous), and a binary indicator (X, but observed as Z) for whether nodules were detected via a digital rectal exam. For illustrative purposes, we assume that nodule detection was subject to misclassification.

As described in Section 2.4, estimated ${\pi }_{y\mathbf{c}}^{\ast }$ values were obtained via model (13), with Z as the outcome and Y, C₁, and C₂ as covariates. The ${\widehat{\pi }}_{y\mathbf{c}}^{\ast }$’s ranged from 0.52 to 0.93, with 22 values exceeding 0.9. Table V provides the ‘naïve’ logistic regression results, together with the results of applying the ML and predictive value weighting methods under the assumption that nodule detection was non-differentially misclassified with SE=0.9 and SP=0.8. We performed both analyses with and without adjusting the assumed value of SE_yc upward to equal ${\widehat{\pi }}_{y\mathbf{c}}^{\ast }$ for the 22 subjects with ${\widehat{\pi }}_{y\mathbf{c}}^{\ast }>0.9$ [see equations (15)]. The results in Table V are based on making this adjustment in conjunction with both the ML and predictive value weighting methods, but in both cases they differed only slightly from the unadjusted results. Note that accounting for the misclassification in Z as a surrogate for X markedly moves the naïve OR estimate for nodules away from the null (e.g. e^2.348 =10.46 via weighting, vs the naïve estimate e^1.172 =3.23). Also note that the ML estimates are accompanied by smaller standard errors than those based on predictive value weighting in this example. However, simulation studies (see next section) indicate that this result may not reflect general trends.

Table VI gives predictive value weighting-based estimated log ORs corresponding to true nodule status (X), along with jackknife standard errors, for several assumed (SE, SP) combinations in the prostate cancer example. These results suggest that in this case ln(OR) is much more sensitive to changes in SE given a fixed SP than to changes in SP for a fixed SE, an opposite conclusion to that found in our no-covariate example (Section 3.1; Figure 1). Though not shown in the table, estimated ln(OR)’s corresponding to C₁ and C₂ varied only slightly with the assumed (SE, SP) values, and were similar to the ‘naïve’ estimates for those variables.

3.4. Placing an assumed distribution on (SE, SP)

In previous examples, we calculated standard errors assuming fixed and known investigator-supplied (SE, SP) values. Some authors [10--14] suggest building in additional variability due to uncertainty about these misclassification parameters by specifying underlying joint densities for them and applying imputation-like or Bayesian methods. Such accommodation is readily made using the approach advocated here, although the necessity may be debatable (see Discussion).

To illustrate, we revisit the SIDS example (Table I; Section 3.1). Suppose that we presume non-differential misclassification of exposure, and wish to summarize uncertainty about SE and SP by assuming that they each derive independently from a trapezoidal distribution (see, e.g. [11]) with a minimum of 0.85, maximum of 1, and lower and upper modes of 0.9 and 0.95.

In this case, the goal of sensitivity analysis is to produce one point estimate of the odds ratio, together with an interval estimate that simultaneously takes account of the variability in the observed data (Table I) and the postulated systematic variability of SE and SP. We accomplished this as follows. First, we independently selected 2000 (SE, SP) pairs, with each taken randomly from the assumed trapezoidal distribution. For each pair, we computed the estimated ln(OR) via the predictive value weighting method, together with its jackknife standard error. We then generated 500 random draws from a normal distribution with mean and standard deviation matching that estimated ln(OR) and its associated standard error. Repeating this process for each of the 2000 (SE, SP) pairs and pooling the results produced a histogram of 2000×500 ln(OR) values, which is depicted in Figure 2. The 2.5th, 50th, and 97.5th percentiles of this distribution are −0.053, 0.418, and 1.155, respectively. Exponentiating these produces a median OR estimate of 1.52, with approximate 95 per cent confidence limits of (0.95, 3.17). These may be contrasted with the ‘naïve’ estimates of 1.31 (0.98, 1.71). This approach to interval estimation accounting for both random and systematic variation is akin to that suggested by Fox et al. [11], except that we produced each of our 2000 ln(OR) estimates via a single logistic regression with predictive value weighting, rather than by simulation.