Depth-aware guidance with self-estimated depth representations of diffusion models

Gyeongnyeon Kim, Wooseok Jang, Gyuseong Lee, Susung Hong, Junyoung Seo, Seungryong Kim

Abstract: Highlights•Prove that depth information is in learned U-Net diffusion models.•Train depth predictors in U-Net efficiently using pretrained models•A novel framework injects depth awareness on images from diffusion model.•New guidance methods apply consistency regularization and learned from depth map prior.