Video Annotator: a framework for efficiently building video classifiers using vision-language models and active learning

High-quality and consistent annotations are essential for developing robust machine learning models, but traditional annotation methods are often inefficient and rely on non-expert third-party annotators.

Hard samples, crucial for model training, are difficult to label accurately without domain expertise, leading to unpredictable expenses and time commitments.

To address these challenges, we propose a human-in-the-loop system called Video Annotator (VA) that involves domain experts directly in the annotation process. VA enhances the efficiency, usability, and effectiveness of video classifiers through a continuous annotation process that integrates data collection and model training.

By leveraging zero-shot vision-language models and active learning techniques, VA demonstrates significant improvements in model quality, achieving a median 8.3 point increase in Average Precision compared to competitive baselines. We provide a dataset with 153k labels across 56 video understanding tasks and release the code for replication at github.com/netflix/videoannotator.