We consider a problem: Can a machine learn from a few labeled pixels to predict every pixel in a new image?
This task is extremely challenging (see Fig. 1) as a single body part could contain visually distinctive areas
(e.g. head consists of eyes, noses and mouths); different body parts might look similar and undistinguishable
(e.g., upper arms v.s. lower arms). It could be even more difficult if we do not provide any precise location
but only the occurrence of body parts in the image. This problem is dubbed weakly-supervised segmentation, where
the goal is to classify every pixel into semantic categories using only partial / weak supervision. There are many
forms of weak annotations which are cheap but not perfect, e.g. image-level tags, bounding boxes, points and scribbles.
These forms of weak supervision come with different assumptions and state-of-the-art methods tackle them differently.
Weak supervision can be roughly categorized into two families: Coarse and Sparse supervision. Coarse annotations,
including image tags and bounding boxes, lack precise pixel localization and rely on Class Activation Map (CAM) to
localize coarse semantic cues and generate pseudo pixel labels. Sparse annotations, such as points and scribbles,
only label a small subset of pixels and Conditional Random Fields (CRF) are often used to propagate labels to unlabeled
pixels. However, it is frustrating to develop individual methods for each form of weak supervision. This problem motivates
us to develop a single method to deal with universal weakly supervised segmentation problems. In fact, weakly supervised
segmentation problems can be regarded as semi-supervised pixel classification problems. And the key is how to propagate and
refine annotations from coarsely and sparsely labeled pixels to unlabeled pixels?
Metric Learning and Contrastive Loss Formulation
To solve the semi-supervised learning problem, we take the viewpoint of feature representation learning. We aim at learning
the most optimal pixel-wise feature mapping to group (separate) pixels of the same (different) category. For every pixel in
the image, we generate corresponding embeddings (or feature representations) using a segmentation CNN. We thus can propagate
the semantic labels from labeled pixels to neighboring unlabeled ones in this latent feature space.
We adopt a metric learning framework and contrastive loss formulation to learn the optimal pixel-wise feature mapping.
More specifically, we break down an image into several segments and compute the representative features for each segment
(by averaging pixel embeddings within each segment). For each pixel, we collect same-category segments as the positive set,
and vice versa. As shown in the following figure, we then train the network to increase (decrease) the distance between the pixel
and its positive (negative) set of segments.
Grouping Relationships from Weak Supervision
Here, we see a problem immediately emerging in the metric learning framework. How do we deal with unlabeled pixels and segments
in the metric learning framework? Under the supervised setting, unlabeled pixels and segments are ignored in the contrastive loss
formulation. In the case of point annotations, as most pixels are unlabeled, the supervision signal will be too sparse to learn a
good feature mapping.
Instead, our key insight is to integrate them into discriminative feature learning to strengthen the supervision. We explore four
grouping relationships derived from visual cues and semantic information in images. According to these grouping relationships, we
can define corresponding positive and negative sets for every pixel in the image. As shown in the following figure, the grouping
relationships are based on (a) low-level image similarity, (b) semantic annotations, (c) semantic co-occurrence and (d) feature affinity.
In fact, each grouping relationship corresponds to a specific prior, which is introduced as one of the learning objectives for the
pixel-wise feature mapping. (a) low-level image similarity correlates with a spatial smoothness prior in visually coherent regions.
The intuition is that pixels of similar appearance are more likely to be in the same category. (b) semantic annotations are the
localized semantic cues in the image, such as points / scribbles / CAMs. (c) semantic co-occurrence reflects scene-context similarity.
Objects in the same scene should be more semantically similar than the ones in different scenes. For example, wildlife animals are always
outdoor, but furniture is usually indoor. We consider two images sharing any of semantic classes as similar-context, and vice versa.
(d) feature affinity considers smoothness prior in the latent feature space.
As shown in the figure above, we can define corresponding positive and negative sets of segments, and derive four contrastive losses w.r.t
each grouping relationship. By training the segmentation CNN with these losses jointly, we can find the most optimal feature mapping.
Across-the-board Improvements
As demonstrated in the following figures, our approach outperforms other methods by large margin for every form of weak supervision.
Contextual Retrievals
To demonstrate the semantic information encoded by pixel-wise feature mapping, we perform nearest neighbor retrievals using the image
segments and their features. As shown in the following figure, given the query segment (left), we observe our retrievals (top right) are in
a more similar scene context than baseline method (bottom right). For example, our retrieved horses are jumping over hurdles, which matches
the context of the query horse.
A Solution for Universal Weakly Supervised Segmentation
In this work, we propose a single method to tackle all forms of weak supervision, even if they carry different assumptions. Our core idea
is to learn the pixel-wise feature mapping, which respects various types of grouping relationships. These grouping relationships can be easily
derived from low-level visual cues and semantic information in images. Lastly, we demonstrate superior performance over other baseline methods
given every form of weak annotations.
We thank all co-authors of the paper “Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning” for
their contributions and insights for preparing this blog. The paper is presented at ICLR 2021. You
can see results on our website, and we provide code to to reproduce
our experiments.