Optimizing GPT for Video Understanding: Zero-Shot Performance and Prompt Engineering

Mark Beliaev, Victor Yang, Madhura Raju, Jiachen Sun, Xinghai Hu

Track: long paper (up to 8 pages)

Keywords: VLM, GPT-4o, Video classification, Generative Multi-modality Models

Abstract: In this study, we tackle industry challenges in video content classification by exploring and optimizing GPT-based models for zero-shot classification across seven critical categories of video quality. We contribute a novel approach to improving GPT’s performance through prompt optimization and policy refinement, demonstrating that simplifying complex policies significantly reduces false negatives. Additionally, we introduce a new decomposition-aggregation-based prompt engineering technique, which outperforms traditional single-prompt methods. These experiments, conducted on real industry problems, show that thoughtful prompt design can substantially enhance GPT’s performance without additional finetuning, offering an effective and scalable solution for improving video classification.

Submission Number: 105