Nirkin Yuval ; Wolf Lior ; Hassner Tal

Abstract

We present a novel, real-time, semantic segmentation network in which theencoder both encodes and generates the parameters (weights) of the decoder.Furthermore, to allow maximal adaptivity, the weights at each decoder blockvary spatially. For this purpose, we design a new type of hypernetwork,composed of a nested U-Net for drawing higher level context features, amulti-headed weight generating module which generates the weights of each blockin the decoder immediately before they are consumed, for efficient memoryutilization, and a primary network that is composed of novel dynamic patch-wiseconvolutions. Despite the usage of less-conventional blocks, our architectureobtains real-time performance. In terms of the runtime vs. accuracy trade-off,we surpass state of the art (SotA) results on popular semantic segmentationbenchmarks: PASCAL VOC 2012 (val. set) and real-time semantic segmentation onCityscapes, and CamVid. The code is available: https://nirkin.com/hyperseg.