Video highlight detection algorithm based on Transformer

• 1、Computer School(National Demonstration Software School), Beijing University of Posts and Telecommunications, Beijing 100876

Abstract：The goal of video highlight detection is to automatically find attractive clips in the video, thereby reducing the human resources consumed by video editing. But so far, there are still some problems that can\'t be properly solved in this field, such as narrow application scenarios, insufficient use of information and low detection accuracy. To solve the above problems, a video highlight detection algorithm based on Transformer is proposed. First, in order to capture global information and maintain fragment independence, both long-term and short-term Transformer modules are used. To blend visual and sound features and filter noise data, hybrid Transformer modules are introduced. Then, weakly supervised and supervised loss functions are designed respectively based on video category and highlight segment annotation to improve the universality of the algorithm. The effectiveness of the algorithm is demonstrated by the experimental results on two public datasets, YouTube Highlights and TVSum. In addition, the performance of the algorithm will decrease significantly after replacing its main components, which justifies the design of the algorithm.

Keywords： artificial intelligence video highlight detection deep neural network self-attention weakly supervised learning