Key technologies of big data processing platform construction for coal mining

This paper puts forward a proposal regarding the construction of a big data processing platform for coal mining in view of the facts that China's coal mining technology has stepped into the stage of mechanized, automated and intellectualized unmanned mining, as well as the construction of intelligent mines, and that the coal mining, like other industrial fields, has gradually entered a new era of relying on data production, resulting in massive data processing problems.After analyzing the characteristics of large amount of data, diversity, strong timeliness, high possibility of data distortion, high requirement of predictability and low density of data value generated in coal mining production in China, the paper proposes to construct a platform for both hardware and software based on the theory and technology of big data.In terms of hardware part, based on the selected servers of the original informatization construction, the server cluster technology is applied to build the server cluster, which is upgraded and reallocated.The inadequacies are continuously allocated and adjusted according to the operation needs.According to the number of management files, the size of file blocks, the number of management servers and the storage capacity of each service data, the memory configuration of the node name server in the cluster server is calculated according to the number of virtual cores and hyperthreads of the server CPU.For the CPU of the server, a multi-core and multi-threaded CPU is proposed for the main node server.For cluster storage system, after making comparison, it is proposed to separate the storage of server application software from the storage of mass data.Solid state disk is used to store application software on the server itself.The integration mode of network access storage and storage area network is adopted in mass data storage system to realize data unification, centralized management, easy expansion and fault tolerance, to ensure no single-point failure of network, and to improve cluster I/O speed.In terms of software part, according to the analysis, platform construction needs to meet the requirements of batch processing, flow calculation and transparency, incremental computing, distributed memory parallel computing, high available and scalable memory computing, counting, summing and averaging of various data in coal mining production, and real-time calculation of variance and standard deviation in fusion decision-making of a large number of real-time data acquisition sensor data and to meet the needs of multi-dimensional, long-term and multiple recalculations.This paper proposes to adopt a distributed big data processing system based on Hadoop and Storm.CentOS is adopted as the server operating system.Flume software is used for log message processing.Kafka software and other key technologies are used for data access buffer.The platform data visualization software can be chosen by users according to their needs, without affecting the platform's data processing.