From data upstream to data downstream, it can be roughly divided into: data collection -> data cleaning -> data storage -> data analysis and statistics -> data visualization.

Security is becoming an issue that must be considered when selecting a system. Kafka’s lack of security mechanisms also leads to serious security risks in its deployment in data-sensitive industries. This article will focus on Kafka, first introducing its overall architecture and key concepts, then in-depth analysis of the security issues existing in its architecture, and finally sharing Transwarp's work on Kafka security and how to use it.

Applicable scenarios:

hive is built on Hadoop based on static batch processing. Hadoop usually has high latency and requires a lot of overhead when submitting and scheduling jobs. Therefore, hive is not able to implement low-latency and fast queries on large-scale data sets. For example, hive generally has a minute-level delay when executing queries on data sets of several hundred MB. ?

Therefore, hive is not suitable for applications that require high real-time performance, such as online transaction processing (OLTP). The hive query operation process strictly follows the Hadoop MapReduce job execution model. hive converts the user's hiveSQL statement into a MapReduce job through the interpreter and submits it to the Hadoop cluster.