Analysis of Ten kinds of Big Data Open Source Technology

With development of science and technology, big data has become one of the most popular technologies. Open source allows more and more projects to be analyzed through big data. The following is an analysis of today's ten popular big data open source technologies.

1．Spark

It’s easy to use and supports all important big data languages (Scala, Python, Java, R). It has a strong ecosystem and grows rapidly. And it can support for microbatching/batching/SQL. Spark can better carry out data mining and machine learning. Spark is pretty suitable for MapReduce algorithms that need to be iterated.

2. NiFi

The design goal of Apache NiFi is to automate the data flow between systems. Based on its workflow programming philosophy, NiFi is very easy to use. The two most important features are its powerful user interface and good data backtracking tools. It can be called the Swiss Army knife in big data's toolbox.

3. Hadoop

It is efficient, reliable and scalable. And it can provide the YARN, HDFS and infrastructure you need for your data storage project. As well as running major big data services and applications.

4. Apache Hive

Hive is a data warehouse infrastructure built on Hadoop. It can provide a range of tools. It also can be used for data extraction and transformation to load (ETL). Storage and query are also its function. You could also analyze large-scale data stored in Hadoop. With the release of the latest version, the performance and functionality have been improved in an all-round way. Hive has become the best solution for SQL on big data.

5. Kafka

Kafka is a high-throughput distributed publish and subscribe messaging system. It can handle all the action flow data on the website that consumers need. It also has become the best choice for big data system between asynchronous and distributed messages. And Kafka is more like a bridge between Spark, NiFi, Java, Scala and third-party plug-in tools.

6. Phoenix

It is the SQL driver of HBase. At present, a large number of companies adopt it and expand its scale. NoSQL, supported by HDFS,can integrate all the tools well. The Phoenix query engine converts the SQL query into one or more HBase scan. The execution is then choreographed to generate a standard JDBC result set.

7. Zeppelin

Zeppelin is a Web-based notepad that provides interactive data analysis. It is convenient for people to make beautiful documents. And it can make them data-driven, interactive and collaborative. It also supports multiple languages. Including Scala, Python, SparkSQL, Hive, Markdown, Shell and so on.

8.Sparkling Water

H2O fills the gap in Spark Machine Learning. It can satisfy all your machine learning.

9.Apache Beam

Apache Beam can provide unified data process pipeline development in Java. And it can support Spark and Flink very well. With Providing a lot of online frameworks, developers do not need to learn too many frameworks.

10.Stanford CoreNLP

Natural language processing has great room for growth. And Stanford is trying to improve their framework.

The above ten big data open source technologies have provided great help in people's work and study. It can deal with all kinds of project data. As well as solving the problems encountered in the work. Therefore, it is welcomed by many open source enthusiasts.