Noise as a Knob: An Inference Time Noise Scheduling Strategy to Optimize Diffusion Based Vision Tasks

ICLR 2026 Conference Submission13224 Authors

Keywords: Diffusion Noise Schedule, Panoptic Segmentation, Text to Image Generation, Weather Robustness

TL;DR: My paper introduces an inference-time scheduling strategy that uses noise as a control knob to optimize diffusion models for diverse vision tasks, from panoptic segmentation to text-to-image generation.

Abstract: Diffusion models, originally developed for generative tasks, are increasingly showing promise in discriminative vision tasks like segmentation. Several studies have showcased their adaptability, with diffusion-based generalized frameworks simplifying complex architectures by unifying various components. Despite their architectural elegance, these models often face performance gaps when compared to established GAN and transformer-based methods. This paper delves into the limitations of diffusion models, particularly observing their tendency to prioritize recall over precision. To address this, we introduce a novel inference-time noise scheduling strategy that dynamically adjusts noise during the reverse diffusion process. Crucially, this method requires no additional training of the diffusion model. Our strategy significantly enhances precision with minimal recall reduction for pre-trained models. This leads to an improved Panoptic Quality (PQ) of 52.7 on the COCO validation dataset. While still trailing top performing transformer-based methods, our approach improves the panoptic segmentation benchmark among generalized diffusion-based frameworks by 1.5%. We also show our approach enhances panoptic segmentation in adverse weather. Furthermore, we validate its versatility in text-to-image generation, achieving an X-IQE image-text alignment score of 4.6 on DrawBench, improving the baseline score of 3.6. Our method provides a flexible and effective tool for optimizing task-specific performance and enhancing the utility of diffusion models across both generative and discriminative applications, all without requiring retraining.

Primary Area: generative models

Submission Number: 13224