Design of Distributed Timing Job Scheduling System for Data Analysis Platform

The data analysis platform provides the ability of real-time analysis of big data and accelerates the application of big data in production. In order to solve a series of timing problems existing in the data analysis platform, this paper designs a distributed timing job scheduling system oriented to the data analysis platform. The system adopts centerless Master-Worker architecture. It includes five functional modules: job management, job scheduling, job execution, log management, and worker node monitoring. The system supports scheduled jobs and broadcast jobs in segments. It supports a variety of job scheduling strategies and collects job execution logs in real time. Applying it to the data analysis platform can greatly facilitate job management and reduce job execution time.