Design and Implementation of Distributed Log Streams Processing System Based on Flume/Kafka/Spark

• 1、School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876

Abstract：With the development and popularization of mobile Internet technology, daily production and trading activities of enterprises bring about massive amounts of log data. How to deal with these logs which can't be processed by traditional log system with a new method and how to extract relevant business information has become an issue that needs to be addressed urgently in many industries. The emergence of the distributed computing framework provides a train of thought to solve this problem. Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. Kafka is a high-throughput, distributed publish-subscribe messaging system. Spark is a new generation distributed big data computing framework after Hadoop. Spark Streaming is a framework specially for stream-oriented computation. We have designed and implemented a distributed log streams processing system based on Flume, Kafka and Spark. With this system, enterprises can fetch and analyze log streams data and obtain the decision making information for enterprises business efficiently, real-timely and reliably, thereby the service quality and competitiveness of enterprise can be improved.

Keywords： distributed system log stream Kafka Flume Spark