research.khws.io
Multiformer

Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation

Eindhoven University of Technology

On this page

A self-driving system has to answer three questions about a street scene at once: what is in it, how far away each thing is, and which things are the same from one frame to the next. Multiformer answers all three in a single model, and does each one better than models built for a single task.

Contributions
  1. Multiformera state-of-the-art DVPS model balancing shared and task-specific representations — task-specific branches inside each decoder block, fused into a shared representation at the block interfaces
  2. A design-space explorationfive query-decoder designs evaluated under one training setup, including reimplementations of previous state-of-the-art methods

Summary

Frames from Cityscapes-DVPS, the benchmark this task is measured on.

Depth prediction on a street scene, near surfaces bright and far surfaces dark
Depth
Panoptic segmentation of the same scene, each object in its own colour
Panoptic segmentation

Panoptic segmentation labels every pixel and separates every object instance — one map instead of semantic and instance passes.

Context & background

Depth-aware video panoptic segmentation matters for autonomous driving and robotics, and existing approaches split on one question: whether the tasks should share an object representation or keep their own. Shared representations are efficient and learn their interactions during training, but can struggle to balance what each task needs; task-specific ones give control at the cost of the synergies. Multiformer is the hybrid — mostly shared, with task-specific branching where it pays.

Mask transformer. Built on Mask2Former, with depth and tracking added.

Multiformer model architecture. Numbered callouts one to seven mark the backbone, pixel decoder, feature pyramid head, context adapter, decoder blocks, depth head and object tracker.
1Backbone2Pixel decoder3Feature pyramid head4Context adapter5Decoder block6Depth head7Object tracker
Figure 1Multiformer architecture

Query. A learned vector that claims a region of the image; the decoder refines it into a mask, a depth and an identity.

The hybrid query decoder

Inside every decoder block the shared queries are split into a mask branch and a depth branch, each refined separately; at the block interface the two are fused back into one query. Task nuance is learned within a block, cross-task information is exchanged between blocks.

Diagram of the context adapter, responsible for initializing the shared multi-purpose queries.
(a)Context adapter
Branched decoder block of the hybrid query decoder system.
(b)Branched block
Figure 2Hybrid query decoder

Metric depth is depth in real units, not a value relative to the rest of the frame.

Metric depth without dataset hyperparameters

Previous methods predict a normalised depth and rescale it with per-dataset constants. Multiformer estimates scale and shift from the image itself, predicts unnormalised log depth with a query-wise affine transform per object, and merges the per-query depth maps by classification confidence rather than pasting them into the panoptic masks.

Diagram of the naive min-max denormalized depth head.
(a)Previous methods
Diagram of the metric depth head, predicting log-scale depths through query-wise affine transforms.
(b)Ours
Figure 3Depth estimation heads

The derivation, the training setup, the ablations and the full results are in the paper.

Citation

Citation downloads

If you find this work useful, please consider citing the paper.

@InProceedings{Stolle2025Balancing, title = {Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation}, author = {Stolle, Kurt H. W.}, booktitle = {WACV}, year = {2025} }

This publication is part of the NEON project with file number 17628 of the Crossover research program, which is (partly) financed by the Dutch Research Council (NWO). The Dutch national compute infrastructure was used with the support of the SURF Cooperative using grant EINF-5438.