Fast and automatic periacetabular osteotomy fragment pose estimation using intraoperatively implanted fiducials and single-view fluoroscopy

Early navigation systems for PAO, and other pelvic osteotomies, relied on optical trackers (Langlotz et al 1997, Langlotz et al 1998, Mayman et al 2002, Akiyama et al 2010), or custom cutting guides (Radermacher et al 1998, Otsuki et al 2013). These systems only provided intraoperative assistance during performance of the acetabular osteotomies; pose estimates of the relocated fragment were not produced.

More recent systems have focused on reporting the pose of a relocated fragment (Murphy et al 2015, Liu et al 2016b, Murphy et al 2016, Takao et al 2017, De Raedt et al 2018, Grupp et al 2020a). Fragment pose updates may be provided in real-time by directly attaching an optically tracked rigid body to the fragment as demonstrated in (Liu et al 2016b). However, attaching a large rigid body to the acetabular region is challenging, especially when using a minimally invasive technique specialized for PAO (Troelsen et al 2008). In order to estimate fragment poses and avoid the attachment of an extra rigid body, (Murphy et al 2015) and (Takao et al 2017) digitize specific points on the fragment with an optically tracked pointer tool after each adjustment of the fragment. This digitization adds minor overhead to the operative time in (Murphy et al 2015) and causes some ambiguity between rotation and translation in (Takao et al 2017). In (Murphy et al 2015), fragment pose errors ranged from 1.4^° to 1.8^° in rotation and 1.0–2.2 mm in translation.

Eliminating the need for optical tracking systems, (Grupp et al 2020a) used multiple fluoroscopic views to track the acetabular fragment, ipsilateral femur, and pelvis. A multiple-component intensity-based 2D/3D registration of patient anatomy was used, requiring no external objects and maintaining compatibility with any PAO approach. However, the method suffers from several limitations and constraints that interfere with a typical surgical workflow:

• An approximate AP fluoroscopic view, two additional views, and manual annotation of a single anatomical landmark are required to initialize the method.
• Accuracy of the approach degrades as intraoperative fragment shapes differ from preoperatively planned shapes.
• The computation time on state-of-the-art hardware is not real-time, approximately 25 s.

To overcome these limitations, the methods described in this paper leverage implanted BBs and extend RSA-related techniques to automatically track the migration of the acetabular fragment using a single view per adjustment.

Since its introduction in the 1970s, RSA has been used for a variety of applications (Selvik 1990), including the longitudinal analysis of orthopaedic implant migration (Valstar et al 2002), bone growth (Karrholm et al 1984), and even PAO stability (Mechlenburg et al 2007). Recent work in the RSA community has incorporated 2D/3D registration technology to track the movement of bones and implants without relying on inserted BBs (De Bruin et al 2008, Seehaus et al 2012). Similar to (Grupp et al 2020a), these methods require manual input and multiple x-ray views.

The implantation of BBs for intraoperative fragment tracking during PAO was first introduced in (Murphy et al 2013) and (Armand et al 2018). However, this approach is not easily incorporated into a surgical workflow, since it requires: the manual identification of BB correspondences, multiple post-osteotomy views, and a calibrated CBCT C-arm.

Several methods for automatic 3D BB reconstruction have been developed by the CBCT community (Hamming et al 2009, Yaniv 2009, Dang et al 2012, Choi et al 2014). These methods require a calibrated C-arm, leverage more than three projections, or rely on an orbital motion constraint to help establish correspondences. For intraoperative coronary artery reconstruction, epipolar constraints have been applied to automatically prune invalid point correspondences between two (Yang et al 2009), and three (Blondel et al 2006), fluoroscopic views. Structure-from-Motion pipelines operate in a similar fashion and use the dense correspondences found in large photographic collections to reconstruct rigid structures in 3D (Snavely et al 2008, Westoby et al 2012, Schonberger and Frahm 2016).

Methods using a known 3D marker constellation, two 2D x-ray views, and varying levels of manual interaction have been developed for patient positioning and motion compensation in radiation therapy (Schweikard et al 2000, Litzenberg et al 2002, Aubry et al 2004).

Single plane RSA was proposed in order to avoid the less-common, bi-planar, imaging devices used in RSA (Yuan et al 2002). However, the method requires the use of a calibration cage, does not address the establishment of 2D/3D correspondences, and was only evaluated for single object pose recovery.

Automatic pose and correspondence estimation between a single rigid collection of BBs and one fluoroscopic view was described in (Tang et al 2000) and (Kang et al 2013). The poses of custom tailored fiducial objects using lines and ellipses may also be computed automatically in a single view (Jain et al 2005a, Steger et al 2013). These poses are generally restricted to tracking the relative motion of the C-arm, since the relationship between the patient’s anatomy and the intraoperatively inserted fiducial is typically unknown.

The method of (Tang et al 2000) was incorporated into fluoroscopic systems for estimating the poses of multiple BB constellations required for knee kinematics (Tang et al 2004, Ioppolo et al 2007). However, the mechanism used to identify constellation membership and establish 2D/3D correspondences was not described.

The pipeline proposed in this paper is able to accurately, quickly, and automatically provide pose estimates of a relocated bone fragment during PAO. No reliance on external tracking devices is required. Furthermore, the pose estimation method does not require: a calibrated C-arm, multiple-views, a specific constellation pattern, accurate knowledge of the fragment shape, or any manual establishment of correspondence.

Preoperative processing proceeds identically to that in (Grupp et al 2020a), which we briefly describe here. A lower torso CT scan is obtained and resampled to have 1 mm isotropic voxel spacing. An automated method (Krcah et al 2011) is used for an initial segmentation of the pelvis and femurs; any inconsistencies around the femoral head and acetabulum are cleaned up manually. Anatomical landmarks are manually annotated to define the anterior pelvic plane (APP) coordinate system (Nikou et al 2000), and also for later use as initialization of pre-osteotomy pelvis registrations. The origin of the APP is set at the center of the ipsilateral femoral head, and the mapping from APP coordinates to the CT volume coordinates is denoted by $T_{APP}^{V}$. Six additional landmarks are manually annotated in order to create a planned fragment shape, which is only used to visualize the intraoperative movement of the fragment. Examples of the APP axes orientation and a planned fragment shape are shown in figure 1.

A Halifax Biomedical Inc. injection device is used to implant two, four-BB constellations onto the ipsilateral side of the patient’s pelvis, with one constellation lying on the area expected to lie on the acetabular fragment and the other on the larger pelvis fragment. BB injection is conducted after performing soft-tissue dissection, but prior to osteotomy. The injection device may be maneuvered either with direct sight or fluoroscopic guidance to ensure that the BB constellations will lie on on separate bone fragments after performance of the osteotomies. For the experiments conducted in this paper, fluoroscopy was not used during the BB insertion process. Although at least three BBs must not be colinear within each constellation in order to later track their rigid movement (West et al 2001), this constraint is not difficult to satisfy in practice due to the 3D curvature of the bone.

Three distinct fluoroscopic views are collected while the patient anatomy remains stationary. For each view, a variation of the radial symmetry algorithm (Loy and Zelinsky 2003) is used to automatically locate each BB in 2D. Parameter values and details of this approach are listed in supplementary section S-1.1. The 3D locations of each BB are constructed by recovering the relative pose information of each view, establishing inter-view BB correspondences, and performing triangulation (Hartley and Zisserman 2003).

Using the strategy laid out in (Grupp et al 2020a), relative poses between the three views are recovered by performing 2D/3D rigid registrations of the patient’s preoperative pelvis to each view. Since some of our pre-osteotomy views have excessive pelvic tilt and violate the approximate AP view assumption, we select more than the single landmark described in (Grupp et al 2020a) for initialization of the pipeline. Supplementary section S-1.2 describes the parameters used for the intensity-based registrations.

BB correspondences are automatically established using a combination of anatomical information and the multiple-view geometry between the three C-arm poses. Two of the views are selected to create a candidate set of two-view, single-BB, correspondences and triangulated 3D points. Although we have made no assumptions about the geometry of these views, one of the views was always an approximate AP orientation with a variable amount of pelvic tilt. The candidate correspondences are created by first considering all possible combinations of single-BB correspondences between the two views, and pruning candidates that result in a triangulated point located more than 10 mm away from the pelvis surface. The red sphere shown in figure 4 is an example of a correspondence pruned in this way. Candidate three-view correspondences are constructed by pairing each of the remaining two-view correspondences with every 2D BB detection in the third view. For each candidate three-view correspondence, the two-view 3D triangulation is re-projected into the third view and the distance to the hypothesized 2D match is recorded. Intuitively, re-projection distances for valid correspondences should be smaller than distances from invalid matches, as shown with the green and yellow re-projections in figure 4. Correct three-view correspondences are established by greedily selecting the candidate correspondences with minimum re-projection distances in the third view. The final 3D reconstructions are triangulated using the correct three-view correspondences. In this way, the third view is used to enforce consistency and refine the 3D triangulation. A visual example is shown in figure 4 and a more formal description is located in supplementary section S-1.3.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

After pruning reconstructed BBs on the contralateral side, the process is completed by classifying the remaining BBs as fragment/non-fragment using a K-Means clustering of the BB positions (K = 2).

Annotation speed and computation time is not a critical factor at this point in the procedure, since the BB constellations are not required until the fragment has been relocated; osteotomies may be performed immediately after the three fluoroscopic views are obtained.

After osteotomies have been performed and the acetabular fragment has been relocated, a single fluoroscopic image may be used to recover the fragment’s pose with respect to the APP, Δ_APP . Once the poses of the ilium and fragment BB constellations with respect to the C-arm, $T_C^{IL}$ and $T_C^{FR}$, are recovered, Δ_APP is computed as in (1).

$$\begin{equation} \Delta_{APP} = T_V^{APP} T_C^{FR} T_{IL}^C T_{APP}^{V} \end{equation} \tag{ 1 }$$

Since the BB constellations are constructed in the original pelvis volume coordinate frame, the composition of $T_C^{FR} T_{IL}^C$ is valid and maps points on the preoperative fragment region to their adjusted locations. Using Δ_APP , the current pose of the fragment may be visualized (figure 1), and pose parameters or biomechanical (e.g. LCE) angles may also be displayed.

As was the case for each fluoroscopic view used for pre-osteotomy BB reconstruction, the radial symmetry algorithm is used to detect each BB in the 2D fluoroscopic image automatically. Since the 3D/2D BB correspondences are not yet established, it is not feasible to directly apply classic PnP approaches (Hartley and Zisserman 2003) for calculation of $T_C^{IL}$ or $T_C^{FR}$. Since manual identification is tedious and error-prone, an automatic method to establish correspondences is the appropriate intraoperative strategy. One possible, although naïve, approach would be to enumerate over all possible correspondences and their poses. Digitally Reconstructed Radiographs (DRRs) and similarities with the fluoroscopic image would be computed for each candidate pose, with the actual pose implied by the best similarity score. When exactly all 8 BBs are detected in the view and 4 correspondences are used to establish a candidate pose, state-of-the-art GPUs may be capable of efficiently computing the 1,680 similarity scores of all possible ilium BB correspondences. However, the tractability of performing a brute-force search over similarity scores quickly diminishes as additional BBs are detected. Additional detections may result from false detections on the screws and K-wires used to fix the fragment, and are possibly exacerbated during bilateral cases due to detections on the contralateral side. Examples of 2D BB detections in fluoroscopy images are shown in supplementary Figures S-10, S-11, and S-12, with the number of BB detections per image varying between 7 and 21. For specimen 1, there were 13, 12, and 21 total detections in projections 1, 2, and 3, yielding 17,160, 11,880, and 143,640 possible ilium BB correspondences, respectively. Even with state-of-the-art GPUs, the sheer number of possible poses precludes the brute-force strategy from working in an intraoperatively compatible timeframe. However, we shall describe a procedure for pruning the number of candidate poses by several orders of magnitude, enabling the required similarity scores to be intraoperatively computed through GPU acceleration.

2.3.1. General pose pruning strategy

2.3.2. Ilium BB pose estimation

The pose of the ilium BB constellation is recovered first and is then used to assist with establishing the pose of the fragment BB constellation. A set of 129 uniformly spaced source-to-detector ratios is used for each ilium P3P invocation: $\left \{0.6 + 0.003\,125k | k = 0, 1, \dots, 128 \right \}$. Using the APP coordinate frame, a reference AP orientation of the pre-osteotomy pelvis, with respect to the C-arm, is constructed and used for pruning anatomically implausible ilium poses. The AP orientation has the following properties: the patient is supine with the x-ray detector placed anteriorly, the AP axis is parallel to the C-arm depth axis, the IS axis is parallel to the 2D image row axis with the top of the image more superior than the bottom, and the LR axis is parallel to the 2D image column axis. Each candidate P3P pose is examined to obtain the difference in orientation from the reference AP pose and an Euler decomposition is used to obtain rotation angles about each anatomical axis. Poses are pruned when the magnitude of any Euler angle is greater than 60^°. Using such a large range of allowable angles permits all reasonable C-arm orientations while eliminating highly unlikely poses, such as those that place the detector beneath, or nearly orthogonal with, the surface of the operating table. An example of a pose pruned using this logic is shown in the top row of figure 5. Using each remaining candidate ilium pose, the original fragment BB constellation is projected into the view; e.g. where the fragment BBs would be located in 2D had the fragment not been moved. Since the majority of fragment movement consists of rotation, the re-projected fragment BBs should lie nearby to 2D BB detections. Poses are pruned when less than 3 of the fragment BBs are projected inside the bounds of the 2D image. For each projected fragment BB, the distance to the nearest 2D detection is calculated, and the three BBs with the smallest nearest distances are recorded. When the mean distance associated with these three BBs is greater than 200 pixels the candidate ilium pose is pruned.

Zoom In Zoom Out Reset image size

Once the general pruning strategy has been completed for the ilium BB constellation, the naïve brute-force, intensity-based, approach is used to select the best ilium pose from the remaining candidates. The bottom row of figure 5 shows several examples of image similarity calculated from poses derived from different correspondences. This pose is used as initialization for an intensity-based 2D/3D registration of the pre-osteotomy pelvis to the fluoroscopic image. Details of the intensity-based registration parameters are listed in supplementary section S-1.2. Using the pose estimate computed during the intensity-based registration, final ilium BB correspondences are established by re-projecting the 3D ilium BBs into 2D. Correspondences are greedily assigned based on the minimum 2D distances between projected BB locations and detected 2D BB locations. However, no correspondence is established for projected BBs with minimum distances greater than 10.5 pixels. When less than two correspondences are established we consider the algorithm to have failed in establishing the ilium pose and no further processing is performed. The ilium pose is set to the intensity-based registration pose when exactly two correspondences are established. When three or four correspondences are established, the ilium pose is refined by optimizing over the corresponding ilium BB re-projection distances starting from the intensity-based pose as the initial guess.

The set of 2D BB detections is pruned down to exclude: BBs already matched to the ilium, and any BBs that are distant from the expected location of the fragment. A BB is considered distant if the closest, re-projected, fragment BB is greater than 200 pixels away. This is a variation of the process previously used for pruning ilium poses by re-projection of 3D fragment BBs.

2.3.3. Fragment BB pose estimation

The fragment pose recovery is started by conducting the general pruning strategy over candidate fragment BB correspondences and poses. Since approximate depth of the BBs is known from the ilium pose recovery process, only 33 source-to-detector ratios are passed to the P3P solver. A reference source-to-detector ratio, $\hat{r}$, is computed by mapping the centroid of the fragment 3D BB constellation into the C-arm coordinate frame using the ilium pose. The source-to-detector ratios are then uniformly sampled about this reference: $\left \{\hat{r} \pm 0.003\,125k | k = 0, 1, \dots, 16 \right \}$. Using each solution produced by the P3P solver, the relative pose of the fragment is computed using (1). Any relative pose with rotation magnitude greater than 60^° or translation magnitude greater than 30 mm is pruned. Figure 6 shows an example fragment pose that was pruned in this fashion.

Zoom In Zoom Out Reset image size

Due to the difficult nature of the chiseling process, the true shape of the acetabular fragment usually differs from the preoperatively planned shape. For this reason, image similarities are not used to select the best candidate returned from the general pruning process. Instead, the best candidate is selected by choosing the pose yielding the largest number of matching BBs and the smallest mean re-projection distance. The match criterion used for ilium matches is reused here. When less than 3 BBs are matched, the method reports failure. However, the fragment pose is refined by an optimization over re-projection distances if at least 3 BBs are matched. The optimization is regularized by the translation magnitude of the fragment pose relative to the APP. This regularization is reasonable, since the fragment movement is believed to consist primarily of rotation and the approximate depth is known from the ilium pose.

It is important to note that the approach described here only requires correspondences to be established for three ilium BBs and three fragment BBs. Therefore, the proposed method provides some robustness to occlusion, since it is unlikely that more than one BB from a single constellation will be occluded for any given view. Likewise, it is still feasible to obtain fragment pose estimates when a single BB (per constellation) becomes dislodged from the bone.

Surgeries performed on three, non-dysplastic, cadaveric specimens were used to evaluate the proposed method. Specimens 1, 2, and 3 were aged 89, 87, 94 and were male, female, and male, respectively. Preoperative processing and planning was performed bilaterally for each specimen and six PAOs were performed by our surgeon co-author, B.A.M. The Halifax injector, using BBs of 1 mm diameter, was only used during surgeries for specimens 2 and 3. For specimen 1, bone burs were created on the surface of the pelvis, and 1.5 mm diameter BBs were affixed with cyanoacrylate. The larger BBs were also inserted into specimens 2 and 3, but were only used for ground truth calculations. For both BB insertion approaches, direct visual guidance was used to ensure a sufficient separation of the two BB constellations. A comprehensive discussion of BB insertion and the fragment pose ground truth protocol is found in (Grupp et al 2020a). Ground truth poses for specimen 1 were calculated using the 2D/3D known BB constellation approach, whereas specimens 2 and 3 use the 3D/3D method. For specimen 1, the 3D BB positions were manually digitized from a postoperative CT scan and mapped into the preoperative CT frame using a rigid registration between the two images. Pre-osteotomy and post-osteotomy poses of the BB constellations were recovered using a series of 2D/3D, paired point, registrations between the 3D BBs and manually annotated 2D BB locations in the fluoroscopic views. The ground truth fragment pose was defined as the rigid movement of the fragment BBs between the pre-ostetomy and post-osteotomy poses. In addition to the pre and postoperative CT scans for specimens 2 and 3, a CT scan was also collected after BB implantation, but prior to any osteotomies. The positions of the pre and post-osteotomy BBs were manually identified in the corresponding CT scans. A paired point 3D/3D rigid registration between the pre and post-osteotomy BBs was used to calculate the ground truth pose of the fragment.

Three fluoroscopic views were used to reconstruct the pre-osteotomy 3D BB constellations for each surgery. Pose estimation of the relocated fragment was conducted on 3 separate fluoroscopic images, each with different viewing geometries. All fluoroscopy was obtained using a Siemens CIOS Fusion C-arm with 30 inch flat panel detector. The intrinsic parameters of the C-arm were naïvely approximated using metadata present in DICOM images produced by the system. A source-to-detector distance of 1020 mm, isotropic pixel spacings of 0.194 mm, no skew or distortion, and a principal point at the center of the projection were used. For C-arms with minimal distortion, previous work has demonstrated that approximate intrinsics yield accurate reconstructions of small objects and accurate relative pose estimates (Jain et al 2005b). The approximate intrinsics used in this paper are therefore reasonable, as the reconstructions are of small BBs and the final relative fragment pose estimates are not with respect to the C-arm coordinate frame.

A video example of the full pipeline, from preoperative planning to intraoperative pose estimation is available online at: https://youtu.be/0E0U9G81q8g.

For pre-osteotomy BB reconstruction, there were no missed detections in the 2D images and a single false detection in one image. Table 1 summarizes the reconstruction errors. The mean reconstruction error of the larger BBs implanted into specimen 1 was 2.6 mm. For specimens 2 and 3, the mean reconstruction error of the smaller, injected, BBs was 1.4 mm. The mean computation time for the entire reconstruction pipeline was 8.3 ± 0.4 seconds. When excluding the 2D/3D full pelvis registration time required for relative pose recovery of the C-arm, the BB detection, correspondence establishment and reconstruction took 0.7 ± 0.1 s. Timing measurements were conducted using a single NVIDIA P100 (PCIe) GPU and seven cores of an Intel Xeon E5-2680 v4 CPU. A detailed breakdown of these runtimes is found in supplementary table S-2.

Thumbnails of each fluoroscopy image used for BB reconstruction during the cadaver experiments are found in supplementary Figures S-7, S-8, and S-9.

On average, 99.6% of the maximum number of ilium poses are pruned by the P3P solver step. Using anatomical constraints, the remaining poses are pruned by an average of 95.9%. After this pruning, an average of 57 candidate ilium poses were used for the exhaustive image similarity searches. The maximum number of fragment poses were pruned by an average of 97.3% using the P3P solver, and anatomical constraints pruned 81.2% of the remaining poses on average. In order to determine the final fragment BB correspondences, an average of 20 fragment poses remained to be searched after pruning. A summary of the maximum number of ilium and fragment poses considered, and the actual poses considered due to pruning, is shown in supplementary table S-3.

Pose estimation was successfully performed on 18 total views (3 per surgery). The distributions of fragment rotation, LCE angle, and translation errors is shown in figure 7. Errors in rotation were below 3^° for 12 of the 18 cases, with a mean of 2.4^°. When the rotation errors were decomposed about anatomical axes, only rotation about the IS axis had errors greater than 3^°. In terms of both mean and standard deviation, rotation measurements about the AP axis were the most accurate, followed by LR, and then IS. The maximum 3D LCE angle error was 1.8^° and the mean was 1.0^°. The mean translation error was 2.1 mm, and was less than 3 mm for 15 of the 18 estimates. Mean translation errors about the anatomical axes were all within 0.2 mm of each other, and the maximum difference between standard deviations was 0.3 mm. An entire listing of errors for each pose estimate is shown in supplementary table S-4.

Zoom In Zoom Out Reset image size

All four ilium BBs were matched in 6 of the 18 cases and all four fragment BBs were matched in 16 of the 18 cases. The mean rotation, translation, and LCE angle errors for estimates with 4 ilium BBs matched were 1.7^°, 2.1 mm, and 1.0^°, respectively. With less than 4 ilium BBs matched, the mean errors were 2.8^°, 2.0 mm, and 1.1^°, respectively. With 4 fragment BBs matched, the mean rotation, translation, and LCE angle errors were 2.3^°, 2.1 mm, and 1.0^°, respectively. The mean errors were 3.1^°, 1.5 mm, and 1.6^°, when less than 4 fragment BBs matched. Supplementary table S-5 includes a full summary of the number of BB detections and matches in each image.

In the third view for the left side of specimen 1, one ilium BB was outside the image bounds and not detected. On the left side of specimen 2, one of the ilium BBs was occluded by K-wire in each view and therefore not detected. Analysis of the postoperative CT revealed that this BB was actually dislodged by either: performance of the ilium osteotomy or insertion of the K-wire. The missed ilium detections in views 1 and 2 on the right side of specimen 2, were occluded by screws. Occlusion by K-wire also caused the missed ilium detection in view 2 on the right side of specimen 3. However, according to the postoperative CT this ilium BB was also displaced from the bone. The missed fragment BB detections were caused by K-wire occlusion. There were no missed detections of ipsilateral BBs that were present in the scene and not occluded by screws or K-wires.

False detections corresponding to contralateral BBs were observed in 8 of the projections, with an average of 5 detections over these projections. An average of 4 false BB detections were caused by the screws visible in the 6 projections used for specimen 1. No other false BB detections were reported.

The mean computation time for the single-view pose estimation was 0.7 ± 0.2 seconds, and was measured using the same hardware used to record reconstruction times. Supplementary table S-6 provides a detailed breakdown of these runtimes.

Thumbnails of each fluoroscopy image used for fragment pose estimation during the cadaver experiments are found in supplementary figures S-10, S-11, and S-12.

We are grateful for the assistance of Mr Demetries Boston during the cadaveric surgeries. This research was supported by NIH/NIBIB grants R01EB006 839, R21EB020 113, MEXT/JSPS KAKENHI 26 108 004, Johns Hopkins University Internal Funds, and a Johns Hopkins University Applied Physics Laboratory Graduate Student Fellowship. Part of this research project was conducted using computational resources at the Maryland Advanced Research Computing Center (MARCC). Finally, we would also like to thank the anonymous reviewers for their insightful comments which helped to improve the quality of this manuscript.