Repurposing GANs for One-shot Semantic Part Segmentation

Pitchaporn Rewatbowornwong*
Nontawat Tritrong*

VISTEC - Vidyasirimedhi Institute of Science and Technology₁

Rayong, Thailand

CVPR 2021 (Oral)

*Equal contribution

Segmentation results trained with only one label

Abstract

While GANs have shown success in realistic image generation, the idea of using GANs for other tasks unrelated to synthesis is underexplored. Do GANs learn meaningful structural parts of objects during their attempt to reproduce those objects? In this work, we test this hypothesis and propose a simple and effective approach based on GANs for semantic part segmentation that requires as few as one label example along with an unlabeled dataset. Our key idea is to leverage a trained GAN to extract pixel-wise representation from the input image and use it as feature vectors for a segmentation network. Our experiments demonstrate that GANs representation is "readily discriminative" and produces surprisingly good results that are comparable to those from supervised baselines trained with significantly more labels. We believe this novel repurposing of GANs underlies a new class of unsupervised representation learning that is applicable to many other tasks.

Few-shot segmentation results

1-shot

5-shot

10-shot

Auto-shot segmentation results

trained on a dataset auto-generated by our method

1 manual label

5 manual labels

10 manual labels

Input

1 manual label

5 manual labels

10 manual labels

Input

1 manual label

5 manual labels

10 manual labels

Auto-shot segmentation using 10 manual labels

BibTex (Extended Version)

Copied!
@article{article,
author = {Rewatbowornwong, Pitchaporn and Tritrong, Nontawat and Suwajanakorn, Supasorn},
year = {2022},
month = {08},
pages = {1-12},
title = {Repurposing GANs for One-shot Semantic Part Segmentation},
volume = {PP},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
doi = {10.1109/TPAMI.2022.3201285}
}

BibTex (Conference Version)

Copied!
@inproceedings{Tritrong2021RepurposeGANs,
    author = {Tritrong, Nontawat and Rewatbowornwong, Pitchaporn and Suwajanakorn, Supasorn},
    title = {Repurposing GANs for One-shot Semantic Part Segmentation},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 
    year = {2021},
}

Video credits

Horses of Iceland. "5 gaits of the Icelandic horse." YouTube, 10 Oct. 2017, https://youtu.be/RV9P0w8vZi8/.
J Utah. "Driving Downtown - Downtown Atlanta 4K - USA." YouTube, 21 Sep. 2016, https://youtu.be/fGJhH0P-izs/.
Jozef Toth. "Car showcase GTA5." YouTube, 1 Feb. 2019, https://youtu.be/UMYif-0D0BU/.
PBS NewsHour. "WATCH: Former Vice President Joe Biden speaks as vote count continues on Election Night." YouTube, 3 Nov. 2020, https://youtu.be/smLvFa7ECp0/.
People. "The Final Interview With The Obamas (Full Interview) | PEN | People." YouTube, 20 Dec. 2016, https://youtu.be/iH1ZJVqJO3Y/.
Stock Freelr. "Horses Running in Farm House [Free To Use]." YouTube, 23 Jul. 2020, https://youtu.be/e6jOczzmKk0/.
Y Combinator. “Mark Zuckerberg : How to Build the Future.” YouTube, 17 Aug. 2016, https://youtu.be/Lb4IcGF5iTQ/.