Face swapping has gained significant attention for its varied applications. The majority of previous face swapping approaches have relied on the seesaw game training scheme, which often leads to the instability of the model training and results in undesired samples with blended identities due to the target identity leakage problem. This paper introduces the Shape Agnostic Masked AutoEncoder (SAMAE) training scheme, a novel self-supervised approach designed to enhance face swapping model training
Our training scheme addresses the limitations of traditional training methods by circumventing the conventional seesaw game and introducing clear ground truth through its self-reconstruction training regime. It effectively mitigates identity leakage by masking facial regions of the input images and utilizing learned disentangled identity and non-identity features. Additionally, we tackle the shape misalignment problem with new techniques including perforation confusion and random mesh scaling, and establishes a new state-of-the-art, surpassing other baseline methods, preserving both identity and non-identity attributes, without sacrificing on either aspect.
We focus on our research field as portrait manipulation. So we guide you guys to the our sibling works in the field of portrait manipulation.
RobustSwap introduces a StyleGAN2-based face-swapping method with minimizing source attribute leakage.
MagiCapture introduces a diffusion-based portrait personalization method with a novel training scheme.
@article{lee2024selfswapper,
title={SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder},
author={Lee, Jaeseong and Hyung, Junha and Jeong, Sohyun and Choo, Jaegul},
journal={arXiv preprint arXiv:2402.07370},
year={2024}
}