Show, Match and Segment: Joint Learning of Semantic Matching and Object Co-segmentation


Abstract

We present an approach for jointly matching and segmenting object instances of the same category within a collection of images. In contrast to existing algorithms that tackle the tasks of semantic matching and object co-segmentation in isolation, our method exploits the complementary nature of the two tasks. The key insights of our method are two-fold. First, the estimated dense correspondence field from semantic matching provides supervision for object co-segmentation by enforcing consistency between the predicted masks from a pair of images. Second, the predicted object masks from object co-segmentation in turn allow us to reduce the adverse effects due to background clutters for improving semantic matching. Our model is end-to-end trainable and does not require strong supervision from manually annotated correspondences and foreground object masks. Furthermore, for semantic matching, we exploit the cycle consistency property and enforce the predicted geometric transformations to be geometrically plausible and consistent across multiple images. For object co-segmentation, motivated by the classic idea of histogram matching, we propose a perceptual contrastive loss that allows the model to segment the co-occurrent objects from an image collection. We validate the efficacy of our approach on four benchmark datasets: TSS, Internet, PF-PASCAL, and PF-WILLOW, and show that our algorithm performs favorably against the state-of-the-art methods on both semantic matching and object co-segmentation tasks.
Paper

Submitted to PAMI
ACCV 2018
Citation

Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, and Jia-Bin Huang, "Show, Match and Segment: Joint Learning of Semantic Matching and Object Co-segmentation", arXiv, 2019.

Yun-Chun Chen, Po-Hsiang Huang, Li-Yu Yu, Jia-Bin Huang, Ming-Hsuan Yang, and Yen-Yu Lin, "Deep Semantic Matching with Foreground Detection and Cycle-Consistency", in Asian Conference on Computer Vision (ACCV), 2018.


BibTex
@inproceedings{MaCoSNet,
  author    = {Chen, Yun-Chun and Lin, Yen-Yu and Yang, Ming-Hsuan and Huang, Jia-Bin}, 
  title     = {Show, Match and Segment: Joint Learning of Semantic Matching and Object Co-segmentation},
  booktitle = {arXiv},
  year      = {2019}
}
@inproceedings{WeakMatchNet,
  author    = {Chen, Yun-Chun and Huang, Po-Hsiang and Yu, Li-Yu and Huang, Jia-Bin and Yang, Ming-Hsuan and Lin, Yen-Yu}, 
  title     = {Deep Semantic Matching with Foreground Detection and Cycle-Consistency},
  booktitle = {Asian Conference on Computer Vision (ACCV)},
  year      = {2018}
}
      
Code

Network Architecture

Results
TSS - Semantic Matching

TSS - Object Co-segmentation

Internet

PF-PASCAL

PF-WILLOW





References