General Myths about Big Data

Even after the existence of Big Data in the technology world for more than 10 years, I have seen many people having misunderstanding and confusions about the usage of Big Data.


Myth 1: Big Data is one thing-  Hadoop, and Hadoop is just HDFS

• Big data is a concept of handling large data sets whereas Hadoop is a framework. 
• Hadoop is an open source distributed processing framework that manages data processing and storage for big data applications running in a clustered system. 
• Hadoop Architecture consists of more than one application other than HDFS (Hadoop Distributed Filesystem).  In the image below, Flume is a streaming application whereas Sqoop is a batch loading application. Similarly, HBase is a Columnar NoSQL Database and Hive is a SQL Query Engine. Other applications like Oozie, Pig, Mahout, Zookeeper, Ambari, Yarn/MapReduce, R are also present in the Hadoop ecosystem.
• Similar to Hadoop, there are many other NoSQL databases like Cassandra and MongoDB that can also scale in multiple nodes are also considered as Big Data systems.
• Also, there are many cloud systems like Amazon EMR, S3, Oracle Object Storage, Azure Data Lake systems that are also considered as big data systems.

Myth 2. Big Data will replace my Relational Database

• Many database administrators are scared that Big Data or Hadoop systems would replace their Relational Database systems. For understanding the difference between RDBMS and other Big Data systems, we need to understand the CAP (Consistency, Availability, and Partitioning Tolerance) theorem. 
• According to the CAP theorem, data systems are optimized for any of the two features instead of all three. So it is almost virtually impossible to have a data store to have all the three as of today. Refer to the image below:

Myth 3: Big Data can be used in a similar way like a Relational Database

• Relational Database systems are built around the ACID rules. In computer science, ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties of database transactions intended to guarantee validity even in the event of errors, power failures, etc. 
• Big data systems are not built with ACID Rules in mind. They are built around functionalities such as scalability, performance, large clustered deployments of data stores and use-cases such as Business Analytics, Log Analytics, Data warehouse offloading, Recommendation Engines, 360-degree view of the customer, Sentiment Analysis, Fraud detection etc.
• Some of the Big Data systems believe in an eventual consistency across all the deployed nodes, which means that data is never consistent based on where the query happens. These data stores typically follow BASE protocol - Basically Available Soft State Eventually Consistent
• Majority of Big Data systems were also designed for non-transactional data, such as user session data, chat messages, videos, images, documents etc
• If you move data from RDBMS to the Big Data system, you cannot expect to see the same way as you see in a database due to the above-mentioned reasons.
  For example, you are moving change data from Oracle into HDFS. What you will see is a set of files that has incremental data that are captured from the Oracle database in file formats. To get a similar experience that of running a SELECT * from MYTABLE, you would need to create a Hive view and run a Hive query which generates a MapReduce that runs in the background. This might not be the most efficient method of making a query since Hive queries are optimized to run with very large data sets instead of a small dataset.

Myth 4:  Big Data Systems are cheap

• Typically Big Data systems are running on commodity hardware and hence this hardware is considered to be cheap compared to engineered commercial systems. 
• If you looking for a similar computing power like a big data warehousing appliance, it might comparable considering the space utilization, power utilization, number of people that are required to manage and many other factors such as software support/warranty etc.
• It also depends on where you are running it, whether it is on-premises, Hadoop as a Service, Cloud hosted or speciality services like Amazon EMR.

Myth 5: Big Data can solve problems faster than other conventional IT systems

Big Data systems can be usually slower when compared to the conventional system for smaller datasets:

• Big Data is just a concept and Big Data was not originally designed for speed. It has been designed for handling very large datasets that are spread across different systems.
• For example, a Hive query can take a long time by creating a MapReduce program which is a batch type of query.
• Data could be distributed across multiple systems which might take long time access is across networks
• Most of the storage use cheap disks since it runs on commodity hardware. This means that it could be really slow in responding.

Myth 6: Big Data projects can be really quick than a conventional 2 tier or 3 tier system design 

Implementing Big Data systems can take a really long time considering the following:

• Most of the commercial applications or systems are ready to use like a packaged software application, whereas the Big Data system needs to be designed from scratch.
• Often biggest challenge is identifying the right system and application for the use-case. 
• Next biggest challenge is changing requirements which require you to completely replace the system, for example, you started the project in Hive and then you realized that you would need to move to HBase or started with Cassandra and then had to move to another NoSQL Database.
• One major concern is a data security concern in Big Data applications since most of the applications are relatively nascent and might not have solved all the security loopholes.

References:

1. https://searchdatamanagement.techtarget.com/definition/Hadoop 
2. https://www.quora.com/Why-are-ACID-properties-not-followed-in-Big-Data