Do as I Do: Pose Guided Human Motion Copy (2024)

2.1 Image Synthesis

Earlier research resorts to Variational Autoencoder[15] and Auto-Regressive models[16] for image synthesis. Recently, the proposal and application [17, 18] of Generative Adversarial Networks (GANs)[19] have led to great advancement in image generation. Technically, GANs utilize a generator-discriminator architecture, where the generator produces images and the discriminator distinguishes between real and fake images. The generator and discriminator are iteratively optimized in a two-player min-max game. Conditional GANs synthesize the images under a given conditional input (e.g., class labels). Isola et al.[20] consider the conditional GAN as a general solution to accomplish image synthesis tasks such as image reconstruction, style transfer, and image coloring. Rather than generating a vanilla image,[21, 18] propose a two-stage GAN to produce a high-resolution image. Upon initiating photo-realistic images, Gao L et al.[22] propose a lightweight network structure that contains a generator and two discriminators to generate two images with different sizes in a feed-forward process. GANs have made remarkable progress in recent years on many tasks [23, 24, 25, 26]. However, it is well known that GANs are difficult to train and the training process is usually unstable. Towards an easy-to-train and stable GAN, Arjovsky et al.[27, 28] propose a WGAN that introduces a novel Wasserstein loss. Their proposed Wasserstein metric has a superior smoothing property compared to KL divergence of GANs, which can theoretically solve the gradient vanishing problem. A drawback of these methods lies in requiring a considerable amount of samples to train a model, which might limit their applications in human motion copy where we may not have a large number of training samples available.

2.2 Human Motion Copy

Existing approaches for human motion copy can be roughly categorized into three groups, namely pose-guided, warping-guided, and no-intermediary methods.

Pose-Guided Human Motion Copy. [29] is the first seminal work of human motion copy, which proposes a two-stage detailed generation from coarse to fine. Since then, a great deal of research has been conducted on human motion copy.Pumarola et al. [30, 31, 32] employ generators and discriminators to reconstruct the target person image with arbitrary poses. Esser et al. [33, 13] propose a unique conditional U-Net, which regulates the output of the variant auto-encoder on appearance. However, these researches are extremely reliant on large-scale training samples, which is difficult to fulfill in practical applications. Ren et al. [34, 35] achieve great image quality with posture augmentation and novel image refinement. Ghafoor et al. [8] proposed a novel video-to-video action transfer framework, which consists of a cascaded sequence of action transfer block with multi-resolution structure similarity loss. Yang et al. [36] perform human video motion transfer in an unsupervised manner, which utilizes the invariance of three orthogonal variation factors, including motion, structure, and view. Nonetheless, these methods fail to take into account the importance of maintaining facial details during the transfer process of human motion. Although Chan et al. [6] introduce a face enhancement module, due to the overfitting problem of GAN, it is not effective in generating satisfactory faces. In contrast, our body parts enhancement polishes the generated face with a self-supervised training scheme and refines the key body portion using dedicated local GANs.

Warping-Guided Human Motion Copy. Dong et al.[37, 9, 38] disentangle the human image into action and appearance, and then perform motion imitation by a warping GAN that distorts the image according to reference poses. Similar to the above method, Shysheya et al. [39, 40] introduce an attention mechanism between pose skeleton and image to generate UV coordinates and then warps patch-level human texture maps to adapt the UV coordinates. However, these methods are limited by the diversity of texture maps, resulting in blurs and artifacts of the generated video. Han et al. [41] focus on learning an appearance flow that warps the clothing of a target person to the corresponding area of the source person. Wei et al. [10] warp the motion of the target human image and then refine the details. Nevertheless, the warping-based motion copy method, by nature, has difficulties in coping with rapid human motion. Moreover, these methods disregard the temporal consistency across frames, resulting in discontinuous video and visual artifacts.

No-Intermediary Human Motion Copy. There are also attempts that direct their efforts at motion copy without any intermediaries (i.e., poses or meshes). Joo et al. [11] employ two specific losses to constrain the GAN which generates a fusion image (one’s identity with another’s shape). However, it deeply concentrates on upper body motion copy (without legs and feet) and eye style transfer. In contrast, our model does not only achieve human whole-body motion copy but also boldly tries motion copy between animals. To the best of our knowledge, we are the first to replicate movements in other articulated objects of the same species, including fish and mice.

3.1 Pose Extraction and Foreground-Background Separation

Pose Extraction. The goal of motion copy is to learn a mapping between a given video of the source person and the target video of the target person. Unfortunately, each frame of the two videos has millions of pixels, making it extremely difficult to acquire the mapping directly. Inspired by the rapid development of pose estimation techniques[42, 43, 44], we utilize pose skeleton sequence as the intermediary for motion copy. The pose sequence unambiguously indicates the motions and can be used to guide body appearance generation. To this end, we shift to learn a mapping from the poses to the body appearance sequence. Particularly, we adopt pre-trained pose detectors OpenPose[45] and DCPose [46] to extract poses from videos.

Foreground and Background Separation. The pose skeleton clearly characterizes the motion, however, we believe it is too ambitious to synthesize a full frame (foreground and background) directly conditioned on a desired pose. Instead, an important step of our pipeline is to compute a mask matrix $M$, which is leveraged to explicitly disentangle each video frame into foreground and background. We devise a generator to concentrate on synthesizing only the foreground sequence from poses. This facilitates our model to avoid considering a large number of background pixels in the pose-to-appearance generation, resulting in a more realistic appearance of the human and faster convergence for the network. Specifically, we adopt the off-the-shelf Mask-RCNN [47] to obtain the mask matrix $M$. In addition, we employ image inpainting technology[48] to fill the removed foreground pixels in the background.

3.2 Pose-to-appearance Generation and Local Enhancement

Now, we consider how to generate an appealing body foreground sequence upon a given pose sequence. Technically, we design a pose-to-appearance generation GAN (appearance GAN), consisting of a generator that incorporates perceptual loss and Gromov-Wasserstein loss, and a discriminator that exerts spatio-temporal dual constraints.

Dense Skip Connections in Generator. The structure of the generator is illustrated in Fig.3, where we engage in a U-shaped architecture with multiple encoder-decoder layers. In conventional U-Net, a decoder layer solely connects to one symmetry encoder layer [49, 50]. These relatively isolated relationships between different level encoder-decoder layers lead to insufficient spatial information modeling in the encoding and decoding process. Explicitly, during the encoding process of conventional U-Net architecture, consecutive convolutions in the encoder would inevitably drop some low-level detailed features. To tackle the challenge, we devise dense skip connections in the U-shaped architecture. Our motivation is to preserve rich features from multiple levels rather than using only one level feature in the foreground generation. Therefore, as shown in Fig.3, instead of connecting a decoder at layer $i$ with only the symmetric encoder at layer $i$, we add extra skip connections from the encoders at layers $\{1,2,\cdots,i-1\}$ to the decoder at layer $i$. For example, decoder layer De-layer4 not only receives the feature information from the hop connection of encoder layer En-layer4 (as in the conventional U-Net), but also receives the feature information from encoder layers {En-layer1, En-layer2, En-layer3}. In this way, each decoder could integrate multi-level latent features and is able to access lower-level features.

Gromov-Wasserstein Loss and Perceptual Loss to Facilitate Appearance Generation. In the training phase, we extract the pose sequence and foreground sequence from the video of the target person, and train our generator network to learn how to capture the mapping function from the pose sequence to the corresponding foreground sequence of the target person. Existing methods typically address this pose-to-appearance problem with a conventional GAN, and measure the discrepancy between the generated foreground frame and the ground truth frame via a pixel-wise L2 loss. Such approaches, by nature, require a large number of training samples to reach convergence. To alleviate this issue, we propose a Gromov-Wasserstein loss that preserves the distance-structure of the feature space instead of the conventional pixel-wise L2 loss. Particularly, the Gromov-Wasserstein loss enforces that the generated fake frames should have the same feature distance structure as their corresponding ground truth frames. Put differently, if two ground truth frames $F_{i}$ and $F_{j}$ are close to each other in the image feature space, the generated fake frames for them should also be close to each other. Conversely, if $F_{i}$ and $F_{j}$ are far apart in the image feature space, the generated fake frames for them should also be far apart. In this way, we are able to train the network in a pairwise manner, where the training samples are multiplied. Besides the Gromov-Wasserstein loss, we also add a perceptual loss that further forces the generated frame to be consistent with the ground truth frame.

Inspired by [51, 52], an entropy regularization term is introduced to ensure tractability and reversible backpropagation in the optimal transport loss optimization. In addition, we utilize the Sinkhorn algorithm and the projected gradient descent method [51] to solve the entropy-regularized Gromov-Wasserstein loss. Technically, the process of optimizing Gromov-Wasserstein loss is outlined in Algorithm1.

Discriminator in Pose-to-Appearance Generation.  (1) Recalling previous approaches for motion copy, they typically employ a spatial discriminator that concentrates on the quality of each frame and fails to explicitly consider video continuity. (2) When we watch videos, we tend to take care of the quality of frames and continuity across frames. We believe it is crucial to jointly take into account spatial consistency and temporal continuity. Based on the two observations and heuristics above, we present a spatial-temporal dual constraint, consisting of a quality discriminator $D_{q}$ and a temporal discriminator $D_{t}$. Specifically, (1) the quality discriminator $D_{q}$ enforces the forged foreground image to approach the ground truth. (2) the temporal discriminator $D_{t}$ captures the temporal information across frames using a set of parallel dilation convolutions. $D_{q}$ takes $(P_{i},{F}_{i})$ or $(P_{i},{\overline{{F}}}_{i})$ as the input while $D_{t}$ absorbs $(P_{t-1}^{t+1},{F}_{t-1}^{t+1})$ or $(P_{t-1}^{t+1},{\overline{{F}}}_{t-1}^{t+1})$. Note that $P_{i}$ stands for pose of the $i^{th}$ frame and $\overline{{F}}_{i}$ denotes the generated foreground for $P_{i}$. $P_{t-1}^{t+1}$ and ${\overline{{F}}}_{t-1}^{t+1}$ represent $\langle P_{t-1},P_{t},P_{t+1}\rangle$ and $\langle\overline{{F}}_{t-1},\overline{{F}}_{t},\overline{{F}}_{t+1}\rangle$, respectively. Both of the two discriminators are trained to output binary labels, real or fake. Overall, the generator strives to create more lifelike videos to fool the dual discriminator, while the discriminator tries its best to distinguish between generated video and the ground truth. Model performance is iteratively optimized in a two-player min-max game fashion.

The image quality often appears imperfect when the fine-grained local details are missing. We scrutinized and implemented state-of-the-art methods following their released code and parameter settings [6, 13, 7, 9, 10]. A significant insight we gain from the experiments is that current methods still have difficulties in generating detailed face, natural hands, and clear feet. After obtaining the initial body appearance using the proposed pose-to-appearance GAN network, we further employ a self-supervised face enhancement component and multiple local GANs to polish the details of local parts.

Self Supervised Face Enhancement with Vector Field. Intuitively, the face images of the same person with similar face orientations should look similar to each other. Therefore, we search from the given videos of the target person to identify face images that have similar face orientations as the synthesized image. In particular, we choose multiple images with the closest face orientations rather than using only one image that has the closest face orientation as the synthesized image, making it more robust to noise.

For the measurement of face orientation similarity, an intuitive approach is to compute the similarity between facial features. However, facial features usually convey too much information irrelevant to face orientation, e.g., color and eye shape. To tackle the problem, a viable method, as shown in Fig.4, is to represent the face orientation with a face vector field. As shown in Fig.4, we employ six vectors, including $v_{1}$: right eye $\rightarrow$ left eye, $v_{2}$: left eye $\rightarrow$ nose, $v_{3}$: right eye$\rightarrow$ nose, $v_{4}$: right ear$\rightarrow$ left ear, $v_{5}$: nose$\rightarrow$ right ear, $v_{6}$: nose$\rightarrow$ left ear. Given two face orientations $\{v_{i}\}_{i=1}^{6}$ and $\{\hat{v}_{i}\}_{i=1}^{6}$, their similarity can be conveniently computed as:

3.3 Episodic Memory for Experience Replay

For the pose-to-appearance generation, we adopt the theoretical Gromov-Wasserstein loss to mitigate the issue of insufficient training samples. Furthermore, inspired by lifelong learning, we introduce an episodic memory component for appearance generation, which propels continuous learning and the accumulation of past knowledge over a lifetime. More specifically, we store previous poor generations in the episodic memory and replay these poor generations periodically in training. This enforces the network to be able to consistently learn from its own mistakes and accumulate experiences. Interestingly, the mechanism is similar to our human brain that occasionally recaps significant moments recorded in our memory. The entire procedure of memory replay is formulated in Algorithm2. We may describe the high-level idea as:

In the first epoch, we utilize all training samples to train the pose-to-appearance generation network (Algorithm 2 lines 12-14). We then select all the poorly generated samples (when the perceptual loss exceeds a threshold) and randomly select a few other samples, and put them into the episodic memory (Algorithm 2 lines 15-19). In the following epochs, we retrain all the training samples for the second time. During the second-time training, we randomly select several samples from the episodic memory and replay (retrain) them to update the parameters of the pose-to-appearance generation network per K epochs (Algorithm 2 lines 6-11).

Memory replay would keep the model from catastrophic forgetting, continuously improving the generated frames by learning from past poor generations. However, overfitting problems may arise if the training samples in the memory are revisited too frequently. Following [54], our memory replay is designed to be executed only occasionally.

3.4 Foreground and Background Fusion

3.5 Loss Functions

4.1 Experimental Settings

4.2 Comparison with State-of-the-art Methods on Multiple Metrics (RQ1)

To quantitatively compare the performance of our method and existing approaches, we divide the applications into two scenes: Image Reconstruction and Motion Imitation. For Image Reconstruction, we perform self-mimicry experiments in which persons imitate actions from themselves. In other words, we feed the pose skeleton of a subject into the network and output the human image of the same subject. We adopt Structural Similarity (SSIM) [59] as a low-level metric, Peak Signal to Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS) [60] as the perceptual level metrics to evaluate the quality of the generated image sequence.For Motion Imitation, we perform cross-mimicry where persons imitate the movements of others. Put differently, we input the pose skeleton of a subject into the network and output the human image of another subject.We utilize the Inception Score (IS) [61] and Frechet Inception Distance score (FID) [62] to examine the differences between the generated images and the ground truth images. In addition, following the method in [10], we employ Temporal Consistency Metric [63] to measure the temporal continuity of the generated video. The experimental results on the ComplexMotion and iPER datasets are summarized in TableI. From the table, we have the following observations. (1) Among the existing methods, C2F-FWN [10] and FakeMotion [55] achieve the current state-of-the-art performance on both the two datasets. (2) FakeVideo is able to outperform state-of-the-art approaches in both Image Reconstruction and Motion Imitation. For example, in image reconstruction and motion imitation, FakeVideo gains 7.2% and 12.4% improvements on PSNR and FID metrics respectively. The significant performance improvements suggest the potential of FakeVideo to perform motion copy. The experimental results on the SoloDance datasets are summarized in TableIII. From the table, we have the following observations. (1) Among the existing methods, C2F-FWN [10] achieves the current state-of-the-art performance on SoloDance datasets. (2) FakeVideo is able to outperform state-of-the-art approaches. For example, FakeVideo gains 4.4% and 4% improvements on PSNR and FID metrics respectively. The performance improvements suggest the potential of FakeVideo to perform motion copy.

4.3 Visual Comparison with State-of-the-art Methods (RQ2)

Further, we visualize the generated results of state-of-the-art approaches on three datasets, which are depicted in Fig.6. Empirically, PoseWarp [13] and FSV2V [7] may result in distorted body shapes and absent limbs. We conjecture the reasons are that they do not consider fusing information across multiple scales, leading to inevitable information dropping in the generation process. EDN [6] achieves realistic visual results. However, the generated human faces of EDN usually have blurred facial parts. LWGAN [9] and C2F-FWN [10] could effectively copy the motions according to the optical flow, however, they have difficulties in generating fine-grained clothes and hairs. In contrast, our method yields a more realistic human body and plausible local details. More visual results of our method are demonstrated in Fig.7. We see that our method consistently generates realistic frames.

As shown in Fig.6, the first column (Target Person) illustrates the target person, the second column (Source Person) demonstrates the source person, the third column presents desired poses, which are obtained from videos of the source person (not the target person), and the remaining columns represent the generated frames of the target person. As shown in Fig.6 and Fig.7, we would like to clarify that the source person and the target person are not the same individual, they are different in faces, body shapes, clothes, and even in gender. Fig.6 and Fig.7 contain multiple source and target persons pairs from three datasets.

4.4 Study on Key Components of FakeVideo (RQ3)

In this subsection, we study the effects of the different components of our method. With this goal in mind, we tried (1) removing dense skip connections from the pose-to-appearance generation GAN, (2) utilizing different kinds of loss functions in the generation network, (3) removing memory module from our framework, (4) using different face enhancement strategies, and (5) removing multiple local GANs which are responsible for local enhancement.

Removing dense skip connections from the pose-to-appearance GAN. We first investigate the effect of removing the dense skip connections in the pose-to-appearance GAN. From TableII, we observe that the performance of the method degrades significantly upon the removal of dense skip connections. This is consistent with our intuition that the dense skip connections could integrate multi-level latent features and is able to access lower-level pose details, contributing to better performance.

Utilizing different kinds of loss functions. Then, we examine the effects of different kinds of loss functions. In the pose-to-appearance generation GAN, we employ a perceptual loss $\mathcal{L}_{p}$ and a Gromov-Wasserstein Loss $\mathcal{L}_{\textit{GW}}$. To study the impacts of the two loss functions, we conduct comparing experiments on using $\mathcal{L}_{p}$, $\mathcal{L}_{\textit{GW}}$, and the combined losses $\mathcal{L}_{p}+\mathcal{L}_{\textit{GW}}$, respectively. The empirical results are elaborated in TableIV. The combined losses (i.e., $\mathcal{L}_{p}+\mathcal{L}_{\textit{GW}}$) achieves the best performance, while using $\mathcal{L}_{\textit{GW}}$ alone yields better results than using $\mathcal{L}_{p}$ alone.

Removing memory module from our framework. In order to evaluate the effectiveness of our memory module, we further try removing the episodic memory module from our framework. As shown in TableV, with the memory module, our approach achieves 0.14 and 0.24 higher PSNR scores than its counterpart without the memory module on the two datasets. These evidences show that the memory module does play an important role in boosting the generation quality of our method.

Using different face enhancement strategies. For self-supervised face enhancement, we consider two schemes to select similar face images for the generated face: facial similarity computed using VGG image features and using the proposed face vector field. The results are demonstrated in TableII. Empirically, the face vector field strategy achieves significantly better performance, which is in accordance with our intuition. The image features of faces actually emphasize the appearance information (i.e., colors and eye shapes) while ignoring face orientation information,which has difficulties in effectively selecting similar face images to enhance the generated face. Our face vector field strategy is capable of accurately representing more fine-grained face orientation details, which is conducive to selecting similar face images that are more valuable to compensate for the facial details of the generated face.

We also examine the influence on the number of face vectors, as shown in TableII. We observe that the quality of the generated image gradually increases with the growing number of face vectors. We also ablate the number of similar face images selected, as shown in TableII. We find that three images might provide sufficient facial information, resulting in informed self-supervised face enhancement.

Removing local GANs from the local enhancement module. Finally, we remove the multiple local GANs to examine their contributions. As shown in TableII, FID significantly increases from $48.03$ to $61.49$ upon the removal of the local GANs. This dramatic image quality degeneration highlights the effectiveness of the local GANs in the local refinement. Particularly, the residual images of human body parts generated by the local GANs reveal the difference in color and texture details between the generated image of the body part and the ground truth, facilitating the generation of more lifelike local images.

4.5 Motion Copy on Other Articulated Objects (RQ4)

In addition to performing motion copy on humans, we are curious whether our approach could be generalized to other articulated objects including zebra fish and mouse. To this end, we further conduct experiments on fish [56] and mouse [57] datasets. Interestingly, our method could be adapted to copy motions of fish and mouse. Empirical results are demonstrated in Fig.8. Take fish as an example, in the training stage, we first employ Lie-X [56] to detect the desired poses of the fish from given videos. Then, we disentangle the frames of the video of the target fish into foreground and background using Mask-RCNN [47]. Thereafter, we feed the desired poses into our pose-to-appearance generation network, where the network architecture remains the same but with the feature size adapted to fit fish. We would like to point out that we do not enhance the details of the generated frames, since a zebra fish is small in body size and the pose-to-appearance generation seems to be sufficient to yield realistic fish frames. Finally, we couple the generated foreground and the background, obtaining the entire fish videos. In the inference stage, the network is fed with the desired poses from another fish and we could synthesize a lively video of the target fish, where the target fish swims and acts like another fish. Experiments on fish and mouse show that our method is able to copy motions of other articulated objects.

4.6 Discussion about the Computational Time Comparison

Motion transfer models could be classified into two categories: dedicated-purpose models and general-purpose models. Specifically, the dedicated-purpose models excel in generating fake videos of a specific person and offer high video quality at the expense of longer training time. In contrast, general-purpose models have the capability to generate fake videos of any person, necessitating less training time but yielding less satisfactory generation results compared to dedicated-purpose models. In this paper, we concentrate on dedicated-purpose models. We would like to emphasize that despite the longer training time required for our dedicated-purpose model, it offers a shorter inference time. Empirical results are represented in Table VI. Specifically, during the inference phase, EDN [6] achieves an average Frames Per Second (FPS) of 14.29, [55] achieves an average FPS of 15, and our method has an average FPS of 25.25.

4.7 User study

We conduct a user study, engaging a cohort of 25 volunteers, to meticulously evaluate the quality of the generated results. Each participant is presented with six clusters of generated results, and subsequently requested to discern and designate the best results within each group. Ultimately, we collect a total of 25 responses, the results are shown in Fig. 9. We can see that the proposed FakeVideo gets the highest rating and significantly outperforms other methods (EDN [6], PoseWarp [13], FSV2V [7], C2F-FWN [10], and LWGAN [38]).

4.8 Failure Case Analysis

Interestingly, we also observed a few failure cases. The failure cases are shown in Figure 10. The first failure case is shown in Fig. 10 (a). When the hands of the source character are undetectable, the generated hands are not realistic enough. The second failure case is shown in Fig. 10 (b). If there are artifacts in the input source image, such as elongated arms and lower legs, this will result in a target image with missing forearms and misaligned lower legs.In the subsequent work, we plan to implement the following measures to further improve the generation:

(1) We will develop a more accurate human pose estimation framework, which plays an important role in the task of human motion copy.

(2) Additionally, we will enhance the network structure. Specifically, in cases where a generated limb of the synthesized frame is missing, the network will strive to generate a limb that aligns with the target person, thereby ensuring a more coherent output.