Is 'Big Data Analytics' based on Hadoop, HDFS & Hive/Hbase adopted as 100% integrated solution in a complex & heavy data processing environment?

The Apache Hadoop ecosystem (set of tools that consume / publish data with HDFS and integrate with other Apache tools), has one option for each one of the activities you have presented in your list. It might not be a 1 to 1 match, though.

Sometimes you have several tools that do the same in a different way, or using a different language, like Pig and Spark.

I like the (very comprehensive) table in the link below. If you go through it, you will find what you need, and how to make the tools, to the activities in your list.
https://hadoopecosystemtable.github.io/

Keep in mind that Hadoop comes in flavours, like Cloudera, Teradata and Huawei, and there are others running on Azure and AWS.

If you want to play with a sandbox, I like the Hortonworks (Cloudera) sandbox.

Hope this helps.
Take care and keep safe.
- Rui

Hi Hafiz

Apache Hadoop is a set of tools, not one tool off-the-shelf to make it all happen. Some of the tools are better supported than others, and integrated better than others, and you have to take it, use case by use case.

There are limitations that have to be “navigated”, but the technology is ripe and can be consumed. It works.

You see “big” IT players like Oracle and Teradata developing integration with Hadoop and selling the idea that their database + Hadoop is the way to go. You have other players like Cloudera and IBM, extending the Hadoop ecosystem and creating “polished” versions of that ecosystem with their own tools.

Hi Hafiz

Apache Hadoop is a set of tools. Some of the tools are better supported than others, and integrated better than others, and you have to design and implement it, use case by use case.
There are limitations that have to be “navigated”, but the technology is ripe and can be consumed. It works.

You see “big” IT players like Oracle and Teradata developing integration with Hadoop and selling the idea that their database + Hadoop is the way to go. You have other players like Cloudera and IBM, extending the Apache Hadoop ecosystem and creating “polished” versions of that ecosystem with their own tools.

The secret sauce here is in the delivery of the technology: the services.

You need to choose an SI (System Integrator) with experience and references in your specific market. They will define your Data Architecture and show you the way.

If you have a in-house team of talented professionals, they can even procure Hadoop support (ex: Cloudera), and develop by themselves.

There are pros and cons to both models, depending on scope, market, budget, etc.

Hope this helps.
Take care and keep safe.

Hi Hafiz, Rui has set out the core components of the apache environment for big data analytics, however i’ll add some additional observations for my own experience:
1. The data visualisation and creation of the Self service analytics (e.g. timeseries analysis expansions) are not brilliant or intuitive (INMO).
2. for operational intelligence and near real time analysis you need to spend a lot of time configuring and can be cumbersome.
3. When dealing with a wide variety of data sources (e.g. econometric modelling, smart cities, IOT, environmental assessment) to be analysed, the tools are restricted in the number of analytic factors that they can take on board. other tools are available.
4 when you need to have different Architectural deployments for HPC and HTC processing this requires stronger release management and version controls to make sure code is released for the different environments.
5. Depending on your compliance and legislation requirements you may find that security set-up and data protection requires multiple product thinking and integration. 

please do not mis understand, I have implemented Hadoop in £1bn companies delivering enterprise insight, it is dependable and part of the data and analytics landscape for the organisation.

However, analyse your requirements, data science methods and approaches before deciding on the technology.
I have stopped too many projects that have gone for a “if we build a datalake/data repository/data warehouse/fraud warehouse then people will come”  as they failed to meet some early core principles.
For your specific needs you may find it meets your needs and the commercial vaibility you seek.