Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos

To predict the instance shape deformation $\Delta V_{\text{ins}}$ and appearance of the object, we extract a feature vector $\phi_{t}$ for each frame of the video using a pre-trained DINO-ViT [12] with frozen weights, as mentioned previously. We assume the instance shape and appearance remain the same throughout the video, and hence take the average image features across all frames, denoted as $\bar{\phi}$, when querying the MLPs, $f_{\Delta V}$ and $f_{\text{a}}$.

In order to extract the motion information more effectively from the input video clip, we design a pair of spatial and temporal transformer-based motion encoders, $E_{\text{s}}$ and $E_{\text{t}}$, that aggregate a set of bone-specific local features first spatially across each frame and then temporally across the entire sequence, eventually obtaining the distribution parameters $\hat{\mu}$ and $\hat{\Sigma}$ of the motion latent VAE.

Specifically, given each frame $I_{t}$ in the input clip, we first construct a bone-specific feature descriptor $\nu_{t,b}=(\phi_{t},\Phi_{t}(\mathbf{u}_{t,b}),b,\mathbf{J}_{b},\mathbf{u}_{t,b})$ for each bone $b=2,...,B$ and each timestamp $t$. Here, $\phi_{t}$ denotes the same global image feature as before. $\mathbf{J}_{b}$ denotes the 3D location of the center of the bone $b$ at rest-pose, which projects to the pixel location $\mathbf{u}_{t,b}$ in the image space, given the rigid pose $\hat{\xi}_{t,1}$ predicted separately. In addition to the global feature $\phi_{t}$, we also sample an auxiliary bone-specific local feature vector $\Phi_{t}(\mathbf{u}_{t,b})$ from the DINO-ViT key token map $\Phi_{t}$ at the projected pixel location $\mathbf{u}_{t,b}$.

The spatial transformer encoder $E_{\text{s}}$ then fuses these bone-specific feature descriptors $\{\nu_{t,b}\}_{b=2}^{B}$ into a single feature vector $\nu_{t,*}$ summarizing the articulated pose of the animal in each frame $t$:

In practice, we prepend a learnable token to the list of descriptors, and take the first output token of the transformer as the pose feature $\nu_{t,*}$. We call this $E_{\text{s}}$ a spatial transformer as it extracts the spatial geometric features in each input frame that capture the pose information, conditioned on the given skeleton.

Next, we design a second temporal transformer encoder $E_{\text{t}}$, inspired by [59], which operates along the temporal dimension and maps the entire sequence of pose features $\{\nu_{t,*}\}_{t=1}^{T}$ into the motion latent space. Similarly to the $E_{\text{s}}$, $E_{\text{t}}$ fuses the pose feature sequence to predict the VAE distribution parameters:

Using the reparametrization trick [37], we then sample a latent code from the Gaussian distribution $z\sim\mathcal{N}(\hat{\mu},\hat{\Sigma})$, which will be decoded into a sequence of articulated poses $\{\hat{\xi}_{t}\}_{t=1}^{T}$ characterizing the 3D motion of the animal in the clip.

Symmetric to the motion encoders, the motion decoder also consists of a temporal decoder $D_{\text{t}}$ that first decodes $z$ into a sequence of pose features $\{z_{t}\}_{t=1}^{T}$, and a spatial decoder $D_{\text{s}}$ that further decodes each pose feature $z_{t}$ to a set of bone rotations $\{\hat{\xi}_{t,b}\}_{b=2}^{B}$.

Specifically, we query the temporal transformer decoder $D_{\text{t}}$ with a sequence of timestamps $\mathcal{T}$, and use $z$ as both the key token and the value token to obtain a sequence of pose features:

Similarly, given each pose feature $z_{t}$, we then query the spatial transformer decoder $D_{\text{s}}$ with a sequence of bone indices $\mathcal{B}$ to produce the bone rotations:

In practice, the rigid pose $\hat{\xi}_{t,1}$ is predicted by a separate network and is not modeled by this motion VAE, since it is entangled with arbitrary camera motions that are difficult to disentangle in dynamic scenes.

We then deform the predicted instance mesh $\hat{V}_{\text{ins}}$ using these articulated pose sequence $\{\hat{\xi}_{t}\}_{t=1}^{T}$ with the skinning equation $\hat{V}_{t}=g(\hat{V}_{\text{ins}},\hat{\xi}_{t})$, and render the RGB frames $\{\hat{I}_{t}\}_{t=1}^{T}$ and masks $\{\hat{M}_{t}\}_{t=1}^{T}$ using a differentiable renderer [42].

We train the entire model by minimizing the reconstruction losses on the object masks $\hat{M}_{t}$ and RGB frames $\hat{I}_{t}$:

where distance transform $\texttt{dt}(\cdot)$ is used in the second term of the mask loss with a weight $\lambda_{\text{dt}}$ for more effective gradients [34, 81, 80], and $\odot$ denotes the Hadamard product. The RGB loss is only computed inside the intersection of the predicted and ground-truth masks $\tilde{M}_{t}=\hat{M}_{t}\odot M_{t}$. To exploit the temporal consistency of the motion in the videos, we further enforce a temporal smoothness constraint between the predicted poses $\hat{\xi}_{t}$ of consecutive frames: $\mathcal{R}_{\text{temp}}=\sum_{t=2}^{T}\|\hat{\xi}_{t}-\hat{\xi}_{t-1}\|_{2}^{2}$. We also inherit the multi-hypothesis viewpoint prediction mechanism with the hypothesis loss $\mathcal{L}_{\text{hyp}}$ and the shape regularizers $\mathcal{R}_{\text{shape}}=\lambda_{\text{Eik}}\mathcal{R}_{\text{Eik}}+\lambda_{\text{art}}\mathcal{R}_{\text{art}}+\lambda_{\text{def}}\mathcal{R}_{\text{def}}$ [80] with balancing weights $\lambda$’s, which include the Eikonal constraint $\mathcal{R}_{\text{Eik}}$ on the SDF MLP for the base shape, and magnitude regularizers $\mathcal{R}_{\text{art}}$ on the bone rotations $\hat{\xi}_{2:B}$ and $\mathcal{R}_{\text{def}}$ on the vertex deformations $\Delta V_{\text{ins}}$.

Instead of relying on external pose annotations or prior shape models to learn the 3D model from monocular videos, we seek a much cheaper alternative solution for establishing correspondences across different instances. We distill semantic correspondences from self-supervised image features, such as DINO [12]. As shown in prior work [4, 80, 92], after a simple PCA reduction, these image features reveal robust part-level correspondences across different instances with varying poses and appearance. To exploit these correspondences, we additionally optimize a feature field in the canonical space using a coordinate MLP $\psi(\mathbf{x})\in\mathbb{R}^{D}$, which is rendered into an 2D feature image $\hat{\Phi}_{t}\in\mathbb{R}^{D\times H\times W}$ given the posed mesh $\hat{V}_{t}$, with the same procedure as rendering the appearance of the object described above. We then encourage this rendered feature map $\hat{\Phi}_{t}$ to match the feature map $\Phi^{\prime}_{t}$ pre-extracted from the input frame $I_{t}$ using DINO-ViT with PCA reduction: $\mathcal{L}_{\text{feat},t}=\|\tilde{M}_{t}\odot(\hat{\Phi}_{t}-\Phi^{\prime}_{t})\|_{2}^{2}.$ Intuitively, this enforces the model to establish correspondences across all training video instances through the same canonical feature field, hence disentangling the shape and pose in each monocular frame.

Similarly to the conventional VAE, we also minimize the Kullback–Leibler (KL) divergence between the learned motion latent distribution and a standard Gaussian distribution:

where $\mu_{i}$ and $\sigma_{i}^{2}$ are elements of the predicted distribution parameters $\hat{\mu}$ and $\hat{\Sigma}$.

As learning 3D articulated motions from unstructured video clips without labels is extremely ill-posed, we devise a two-stage schedule for robust and efficient training. In the first stage, we pre-train the monocular 3D reconstruction model using a single-image pose predictor $\tilde{\xi}_{t}=f_{\xi}^{\text{sin}}(\phi_{t})$. Inspired by but unlike [80], we train this model to re-render entire video clips with the temporal smoothness constraint $\mathcal{R}_{\text{temp}}$ and temporal feature averaging $\bar{\phi}$, rather than independent images. The total loss in the first stage is given by:

where $\mathcal{L}_{\text{recon},t}=\mathcal{L}_{\text{im},t}+\lambda_{\text{m}}\mathcal{L}_{\text{m},t}+\lambda_{\text{f}}\mathcal{L}_{\text{feat},t}$ summarizes the reconstruction losses on each frame. After this stage, we obtain an accurate monocular 3D reconstruction model, which outperforms the baseline [80] as shown in Table 4, largely owing to the training on videos instead of independent images. More importantly, the model has now learned a reasonable space of articulated poses, on top of which learning a motion generative model is much more efficient.

In the second stage, we replace the monocular pose predictor $f_{\xi}^{\text{sin}}$ with the spatio-temporal transformer-based motion VAE $f_{\xi}$ detailed in Section 3.2, which encodes the entire video clip and generates the entire sequence of articulated poses at once. Empirically, training the motion VAE from scratch with an expensive rendering step in the loop is inefficient. To facilitate training efficiency, we recycle pose predictions $\tilde{\xi}_{t}$ from the first stage to guide the predictions of the VAE decoder $\hat{\xi}_{t}$ using a teacher loss $\mathcal{L}_{\text{teacher}}=\sum_{t=1}^{T}\|\hat{\xi}_{t}-\tilde{\xi}_{t}\|_{2}^{2}$. The final training objective for the second stage is thus:

During inference time, we can generate diverse 3D motion sequences by sampling from the learned motion VAE latent space. Furthermore, when given a single 2D image of a new animal instance unseen at training, our model can reconstruct its 3D shape and appearance in a feed-forward manner, and generate 4D animations fully automatically within a few seconds, as illustrated in Figure 3.

To train our model, we collected an AnimalMotion dataset consisting of video clips of several quadruped animal categories extracted from the Internet. The statistics of the dataset are summarized in Table 1. As pre-processing, we first detect and segment the animal instances in the videos using the off-the-shelf segmentation model of PointRend [38]. To remove occlusion between different instances, we calculated the extent of mask overlap in each frame and exclude crops where two or more masks overlap with each other. We further apply a smoothing kernel to the sequence of bounding boxes to avoid jittering. The non-occluded instances are then cropped and resized to $256\times 256$. The original videos are all at 30fps. To ensure sufficient motion in each sequence, we remove frames with minimal motion, measured by the magnitude of optical flows within the instance mask estimated from RAFT [75]. To conduct quantitative evaluations and comparisons, we also use PASCAL VOC [20] which contains $108$ images of horses, and APT-36K [90] which contains $81$ video clips of horses, each consisting of $15$ frames. Both datasets provide 2D keypoint annotations for each animal in the image, allowing us to evaluate the geometric accuracy of the reconstructed shapes and generated motions.

The encoders and decoders of the motion VAE model ($E_{\text{s}}$, $E_{\text{t}}$, $D_{\text{s}}$, $D_{\text{t}}$) from Section 3.2 are implemented as stacked transformers [77] with 4 transformer blocks and a latent dimension of 256. We use a sinusoidal function for positional encoding following [59]. For the remaining architectures, we base our implementation on top of [80]. We train the model for $120$ epochs for the first stage, which takes roughly $10$ hours on $8$ A6000 GPUs, and another $180$ epochs for the second stage, which takes another $48$ hours. We use a sequence length of $T=10$ for training. During inference, we can generate longer sequences by connecting multiple samples and optimizing transition latent codes for smooth interpolation. For visualization, following prior work [80], we finetune (only) the appearance network $f_{\text{a}}$ for $100$ iterations on each test image, taking less than 10 seconds, as the model struggles to predict detailed texture in a single feedforward pass. More details are included in the sup. mat.

After training, we can generate 3D motion sequences by sampling the motion latent space VAE, and render 4D animations with the textured mesh reconstructed from a single 2D image, as shown in Figure 3. It also generalizes to horse-like artifacts, such as carousel horses, which the model has never seen during training. The model can be trained on a wide range of animal species besides horses, including giraffes, zebras and cows, capturing category-specific prior distributions of 3D motions, as shown in Figure 5. Because the datasets for these categories are limited in size and diversity, as in [80], in the first stage of the training, we fine-tune from the model trained on horses. Additional animation results are provided in the supplementary video.

Our method is the first to learn a generative model of 3D animal motions from raw videos without pose annotations or prior shape models. We compare with one of the most recent 4D generative models, 4D-fy[8], which has publicly released code. Specifically, we provide the model with a list of prompts, which are enriched by ChatGPT [57] from a list of basic prompts describing horse motions, such as “a horse is running/walking/jumping/eating”¹¹1The full list of prompts are included in the supplementary material.. We generate $20$ 4D instances from 4D-fy, and $20$ from our method (without text condition). Note that it takes $12$ hours to generate one 4D-fy instance on one GPU, whereas our model generates 4D animations within a few seconds in a single forward pass. We first compute the Motion Strength to assess the motion magnitude of the generated videos. We use Flowformer [29] to estimate optical flow strengths between consecutive frames of a generated video, and then compute the average of the largest $5$% optical flows as the Motion Strength. We present them in random pairs side by side to $33$ participants, and ask them to select one that shows “a more plausible 3D horse motion sequence”. As reported in Table 2, users preferred the 4D instances generated by our method over 4D-fy $83.0$% of the time. We show a visual comparison in Figure 4. Notably, 4D-fy produces nearly static animals without perceptible motions despite heavy prompt engineering, whereas our method generates much more plausible motion sequences.

Further assessing the quality of the generated 3D motions quantitatively is difficult due to the lack of (1) ground-truth measurements of 3D animal motions, and (2) robust evaluation metrics for generative models. To evaluate and compare different variants of our model, we design a new metric, bi-directional Motion Chamfer Distance (MCD), computed between a set of generated motion sequences projected to 2D image space and a set of 2D keypoint sequences annotated from videos in APT-36K [90]. Since the skeleton automatically discovered by our model is different from the 17 keypoints annotated in APT-36K, we first perform 3D reconstruction on all the images in APT-36K, and optimize a linear transformation that maps the 2D projections of the predicted 3D joints to the annotated 2D keypoints following [34]. To compute MCD, we generate $1,400$ random motion sequences by sampling from the learned motion VAE, each consisting of $10$ frames of 3D articulated poses. We then project these generated 3D poses to 2D using the viewpoints estimated from APT-36K, and apply the previously optimized transformation to align with the annotated keypoints. For each annotated keypoint sequence in the test set, we find the closest generated motion sequence measured by keypoint MSE averaged across all frames, and vice versa for each generated sequence. We then compute MCD based on the MSE between the closest sequence pairs. In essence, MCD measures the fidelity of generated motions by comparing the sampled distribution to that of the real motion sequences annotated from videos. Table 3 compares the results of our final model with two ablated variants.