Lidar Panoptic Segmentation in an Open World

Abstract

Addressing Lidar Panoptic Segmentation (LPS) is crucial for safe deployment of autnomous vehicles. LPS aims to recognize and segment lidar points w.r.t. a pre-defined vocabulary of semantic classes, including thing classes of countable objects (e.g., pedestrians and vehicles) and stuff classes of amorphous regions (e.g., vegetation and road). Importantly, LPS requires segmenting individual thing instances (e.g., every single vehicle). Current LPS methods make an unrealistic assumption that the semantic class vocabulary is fixed in the real open world, but in fact, class ontologies usually evolve over time as robots encounter instances of novel classes that are considered to be unknowns w.r.t. thepre-defined class vocabulary. To address this unrealistic assumption, we study LPS in the Open World (LiPSOW): we train models on a dataset with a pre-defined semantic class vocabulary and study their generalization to a larger dataset where novel instances of thing and stuff classes can appear. This experimental setting leads to interesting conclusions. While prior art train class-specific instance segmentation methods and obtain state-of-the-art results on known classes, methods based on class-agnostic bottom-up grouping perform favorably on classes outside of the initial class vocabulary (i.e., unknown classes). Unfortunately, these methods do not perform on-par with fully data-driven methods on known classes. Our work suggests a middle ground: we perform class-agnostic point clustering and over-segment the input cloud in a hierarchical fashion, followed by binary point segment classification, akin to Region Proposal Network (Ren et al. NeurIPS, 2015). We obtain the final point cloud segmentation by computing a cut in the weighted hierarchical tree of point segments, independently of semantic classification. Remarkably, this unified approach leads to strong performance on both known and unknown classes.

Appendices

LiDAR Panoptic Segmentation in Open-World

1.1 Vocabulary Splits

We detail our vocabulary split for SemanticKITTI (Behley et al., 2019, 2021) and KITTI360 (Liao et al., 2021) in Table 5. The categorization of stuff and thing follows from the KITTI360 ontology. Vocabulary 1 is constructed by sorting SemanticKITTI superclasses by the number of instances in a superclass and holding-out tail classes as other. Vocabulary 2 is constructed by holding out only the rarest object instances that do not correspond to any semantic class, labeled in SemanticKITTI (e.g., other-vehicle, other-object, etc.)

1.2 Vocabulary Consistency

Inconsistent labeling policies across datasets cause label shifts. For this reason, we base our evaluation set-up on SemanticKITTI and KITTI360. The two datasets adopt largely consistent labeling policies and the same sensor; therefore, the shift in data distribution can be thought of as a result of new classes emerging across datasets. To ensure we have a consistent class vocabulary, we perform the following measures:

• We merge rider and bicyclist (SemanticKITTI) with human and rider (KITTI360) into a single human class to ensure consistency.

• Classes pole and traffic sign are commonly treated as thing classes. However, in SemanticKITTI, they are treated as stuff classes because we do not have instance-level annotations for them. As instance labels for these classes are available in KITTI360, we treat them as other thing classes in KITTI360. Therefore, individual instances of these classes must be segmented. This is consistent with the overall goal of LiPSOW: methods must segment all instances, including those not labeled in SemanticKITTI.

• We treat building as a stuff class in SemanticKITTI and KITTI360 (i.e., we do not treat the building class as an object, a thing class).

Table 5 LiPSOW class ontology

1.3 Model Training

We train all LiPSOW methods using instance labels for known-things and semantic labels for known- things and known-stuff. The instance labels for the other classes are held out, i.e., not available during training. The LiPSOW methods should classify these points as other instead of performing a fine-grained semantic classification of these points. Additionally, LiPSOW methods need to segment novel instances from other, e.g., segment bicycles in SemanticKITTI (Vocabulary 1) and vending machines in KITTI360 (Vocabulary 1 & Vocabulary 2).

Fig. 7

Semantic and instance labels for KITTI360 evaluation, retrieved from the dense accumulated labeled point clouds. Grey color denotes points for which we could not retrieve labeled points within a 10 cm radius to transfer labels (Color figure online)

1.4 KITTI360 Ground-Truth

To evaluate methods on the KITTI360 dataset, we require per-scan semantic and instance labels for Velodyne Lidar scans. However, KITTI360 only provides multiple accumulated point clouds (accumulated over approximately 200 m), recorded by the SICK Lidar sensor. We use these dense accumulated point clouds to retrieve per-scan labels for individual Velodyne point clouds. Concretely, we use publicly available scripts (Mosig, 2022) to align individual Velodyne scans with the accumulated point cloud based on known vehicle odometry. Once aligned, we perform a nearest neighbor search for each Velodyne point in the corresponding accumulated point cloud. In case a match is not found within a 10 cm radius, we mark this point as unlabeled (ignored during the evaluation). We visualize the retrieved labels for Velodyne point clouds in Fig. 7.

Implementation Details

1.1 4DPLS$^\dagger$

Our method and several baselines are based on 4D-PLS (Aygün et al., 2021) that employs an encoder-decoder point-based KPConv (Thomas et al., 2019) backbone for point classification and instance segmentation. The instance segmentation branch consists of three network heads. The objectness head predicts for all points how likely they are to represent a (modal) instance center. The embedding and variance heads are used to associate points with their respective instance centers. During the inference, we select the point $p_i$ with the highest objectness, evaluate all points under a Gaussian (parameterized by the predicted mean and variances for $p_i$), and assign points to this cluster if the point-to-center association probability is higher than a threshold, i.e., $> 0.5$. This process is repeated until the maximum objectness is below a certain threshold (0.1 in 4D-PLS). To ensure high-quality segments, 4D-PLS also enforces that the highest objectness should be $> 0.7$.

Since the objectness head is trained independently of the semantic head in a class-agnostic fashion, we hypothesize it should be able to learn a general notion of objectness from geometric cues. Therefore, we evaluate whether it can segment instances of novel classes with lower confidence. Therefore, we adapt the inference procedure in 4D-PLS to allow additional instances to be segmented. We achieve this by reducing the minimum objectness threshold from 0.7 to 0.3 for other, while maintaining the same threshold for known things. Our experimental evaluation confirms that this baseline can segment a larger number of other instances compared to 4D-PLS.

1.2 OSeg (Cen et al., 2022)

OSeg (Cen et al., 2022) introduces a novel strategy for open-world semantic segmentation of LiDAR Point Clouds. The proposed framework consists of two stages: (i) Open-Set semantic segmentation (OSeg) and (ii) Incremental Learning. For a fair comparison, we benchmark OSeg against our baselines.

OSeg introduces redundancy classifiers on top of a closed-set model to output scores for the unknown class. In addition, OSeg uses unknown object synthesis to generate pseudo-unknown objects based on real novel objects. The OSeg formulation considers other vehicle as a novel category for SemanticKITTI. To benchmark under our proposed LiPSOW formulation, we modify OSeg to allow for more classes in other (based on our vocabulary splits) and train from scratch. We use the default set of hyper-parameters, and use 3 redundancy classifiers.

OWL: Instance Segmentation

1.1 Segmentation Tree Generation

Given an input point cloud, we first make a network pass and classify points into thing, stuff, and other classes. Then, we construct a hierarchical tree segmentation tree T by applying HDBSCAN (McInnes et al., 2017) on points. Concretely, at each level of the hierarchy, we reduce the distance threshold $\epsilon$ (HDBSCAN connectivity hyperparameter) to obtain finer point segments. Therefore, the nodes in T contain strictly smaller and finer-grained instances in child nodes. We follow Hu et al. (2020) and use distance thresholds $\epsilon \in [1.2488, 0.8136, 0.6952, 0.594, 0.4353, 0.3221]$.

1.2 Learning Objectness-Scoring Function

Given a hierarchical tree T over the input point cloud, we need to find a node partitioning such that each point is assigned to a unique instance. Naturally, some nodes in the tree would contain high-quality segments, while others would consist of a soup of segments or overly segmented instances. To associate a metric quality for each node, we follow Hu et al. (2020) and learn a function to score each segment in the node.

1.2.1 Network Architecture

Each node in the tree consists of a segment (i.e., a group of points), where the number of points may vary. We concatenate all the points in a segment to get a $N \times 3$ dimensional tensor, where N is the number of points in the segment. There are several ways how to learn such a function $f(p) \rightarrow [0,1]$ that estimates how likely a subset of points represents an object. One approach is to estimate a per-point objectness score. Following Aygün et al. (2021), this can be learned by regressing a truncated distance $O \in \mathbb {R}^{N \times 1}$ to the nearest point center of a labeled instance (Aygün et al., 2021) on-top of decoder features $F \in \mathbb {R}^{N \times D}$. The objectness value can then be averaged over the segment $p \subset P$.

Alternatively, we can train a holistic classifier as a second-stage network. The network comprises three major components: a) input projection layer, b) segment embedding layer, and c) objectness head. In the input projection layer, we project the input point cloud to a higher dimension of $N \times 256$ by passing through two fully-connected layers. Then, we compute a per-segment embedding of dimension 512 using the embedding layer. This consists of set abstraction layers inspired by PointNet++ (Qi et al., 2017), followed by a reduction over the points.

1.2.2 Network Training

The objectness head predicts per-segment objectness, using three fully-connected layers with a hidden layer of size 256. To obtain training supervision, we pre-built hierarchical segmentation trees $T_i$ for each point cloud i in the training set and minimize the training loss based on the signal we obtain from matched segments between the segmentation trees and set of labeled instances, $GT_i$. As described in Sec 4 of the main paper, for each node in the segmentation tree, the regressor predicts the objectness value which is supervised by the intersection-over-union of the segment with the maximal matching ground-truth instance.

Alternatively, we can also formulate the network as a classifier, where the post-softmax outputs from the network can be viewed as the quality of each segment (i.e., how good or bad a segment is), and this is trained using a binary cross-entropy loss. We observe that the regression formulation empirically results in a better tree cut compared to the classifier formulation. We attribute this to the over-confident and peaky distributions resulting from the classifier. The regressor formulation benefits from a smoother distribution over objectness scores, resulting in a better tree cut. We evaluate the aforementioned variants in Table 1 in the main paper.

1.3 Inference

With a score assigned to each segment, we now need to find a global segmentation, i.e., the optimal instance segmentation from an exponentially large space of possible segments. The global segmentation score is defined as the worst objectness among the individual segments in a tree cut. The optimal partition is the one that maximizes the global segmentation score. We outline this inference algorithm in Algorithm 1. Each node in the tree T constitutes a segment proposal for an object instance. For each proposal S, we score its objectness using the learned objectness function f. The segment S is deemed as an optimal node to perform a tree-cut if its objectness is greater than any of its child nodes. By design, this tree-cut algorithm ensures that each segment is assigned to a unique instance. For details about the algorithm and optimality guarantees, we refer the reader to Hu et al. (2020).

Algorithm 1

Node Partitioning given a Hierarchical Segmentation Tree

Implementation Details

1.1 Training Encoder–Decoder Network

.To train the point-classification network on each vocabulary, we follow the training procedure from Aygün et al. (2021). We train the network for 1000 epochs with a batch size of 8. We use the SGD optimizer with a learning rate of $1e-3$ and a linear decay schedule.

1.2 Second-Stage Training

.In contrast to the encoder-decoder network, which takes an entire point cloud as input, the second-stage network requires positive and negative training instances (i.e., examples of objects and non-objects) from the segmentation tree to evaluate the loss functions (regression or cross-entropy losses). To generate these instances, we first generate semantic predictions. Next, we use the points classified as thing or other to generate the segmentation tree using thresholds as described in Sect C.1. Each node in the segmentation tree is a training sample for the second-stage network. We use predictions from the encoder-decoder instead of ground-truth semantic labels, since during inference the second-stage network must be robust to misclassification errors within each node in the segmentation tree.

1.3 Training Objectness Regression Function

.To train the regressor, we need to generate the corresponding ground-truth for each segment in the generated training set. For a given segment, the target score is computed as the maximum intersection-over-union of the segment with all the ground-truth instances in the dataset. Finally, we train the network using a mean-squared error loss function with a learning rate of $2e-3$ and batch size of 512 for 200 epochs.

1.4 Training Objectness Classification Function

Alternatively, learning an objectness function can be posed as a classification problem, rather than a regression problem. In this case, we supervise the network via cross-entropy loss. The target labels for training this classifier are obtained by binarizing the regression targets using pre-defined intersection-over-union (IoU) thresholds. A regression target with an IoU greater than 0.7 is defined as a positive segment, and a regression target with an IoU less than 0.3 is treated as a negative segment. Since the ground-truth classification targets are generated from the hierarchical tree, which consists of predicted known-things or other, the generated training data is strongly biased towards positive samples. To elaborate, since the point classification network performs well, the segments in the tree most likely consist of known-things or other, barring misclassification error. Therefore, while training this network, we observe a disproportionate imbalance towards the positive class. To mitigate this, perform a weighted resampling; We resample instances (segments) of either positive or negative classes with a probability proportional to the inverse frequency of that class.

Complexity Analysis

OWL requires a two-stage training process. The first stage method, 4DPLS (Aygün et al., 2021) is not a real-time. In addition, the second stage requires the construction of a segmentation tree. Given N points, the algorithm’s (Hu et al., 2020) time complexity and space complexity are linear in N. In practice, we observe that N is quite large, often of the order of 100,000+ points per scan. Therefore, a limitation of OWL is that it cannot be run in real-time.