FLOSS: Free Lunch in Open-vocabulary Semantic Segmentation

Yasser Benigmim1, Mohammad Fahes1, Tuan-Hung Vu1,2, Andrei Bursuc1,2, Raoul de Charette1
1Inria 2Valeo.ai
Paper arXiv Code
FLOSS teaser showing template rankings across different classes

TL;DR

  • Focus: We challenge the conventional use of multiple templates in Open-Vocabulary Semantic Segmentation (OVSS) models.
  • Key Finding: For each class, there exist single-template classifiers that significantly outperform the conventional averaged classifier.
  • Our Approach: We propose FLOSS, a plug-and-play method that:
    • Identifies class-expert templates using prediction entropy
    • Requires no labels or additional training
    • Is complementary to existing OVSS methods
  • Results: FLOSS consistently improves state-of-the-art methods on various OVSS benchmarks and generalizes well across datasets with distribution shifts.

Method

Recent Open-Vocabulary Semantic Segmentation (OVSS) models extend the CLIP model to segmentation while maintaining the use of multiple templates for constructing class-wise averaged text embeddings. Our method, FLOSS, challenges this approach by:

  • Identifying single-template classifiers that outperform averaged classifiers
  • Estimating class-experts using prediction entropy on unlabeled images
  • Implementing a novel fusion method for more accurate OVSS predictions
  • Providing improvements without requiring labels or additional training
FLOSS main figure showing unsupervised identification of class-experts and expert fusion

Acknowledgements


This research was partially funded by the French Agence Nationale de la Recherche (ANR) with the project SIGHT (ANR-20-CE23-0016). We sincerely thank Telecom Paris for providing the resources necessary to run our experiments and Nacereddine Laddaoui for his invaluable help with infrastructure. We are also grateful to Ivan Lopes for proofreading.

BibTeX

@misc{benigmim2025flossfreelunchopenvocabulary,
      title={FLOSS: Free Lunch in Open-vocabulary Semantic Segmentation},
      author={Yasser Benigmim and Mohammad Fahes and Tuan-Hung Vu and Andrei Bursuc and Raoul de Charette},
      year={2025},
      eprint={2504.10487},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.10487},
}