Computational Performance Optionmization Algorithm for Multi-table Joins Based on Flink

• 1、State Key Laboratory of Networking and Switching Technology,Beijing University of Posts and Telecommunications

Abstract：Flink has been widely used in large-scale data analysis processing. Multi-table joins are one of the most common operations for data analysis, so improving the performance of Flink multi-table joins can speed up data processing and analysis. However, applying the existing multi-table join algorithm directly to Flink brings two problems: (1) The existing algorithm can not fully exploit the performance advantages of Flink\'s thread-based lightweight distributed computing. (2) The connection algorithm will cause great pressure on the network. Firstly, the Multi Bushy Tree algorithm for optimizing the parallelism of connections is proposed to improve the parallelism of multi-table join calculations. Secondly, the Semi Join algorithm for optimizing star connections is proposed, which can greatly reduce the amount of data that needs to be shuffled. In this paper, a lot of experiments are carried out on the large-scale dataset of TPC-H. The experimental results show that the proposed algorithm can effectively improve the parallelism of multi-table join calculation. Reduce job run time and reduce network IO cost in star connections.

Keywords： Distributed processing system Flink Multi-table join Parallelism of connections Data shuffle Connection order