Enterprise Data Warehousing (EDW) has been a mainstay of many major corporations for the last 20 years. However, with the tremendous growth of data (doubling every two years), the capacities of enterprise data warehouses are being exhausted. Load processing windows are similarly being maxed out, adversely affecting service and threatening the delivery of critical business insights. The need to store infrequently used, “just in case” data exacerbates this challenge by taking up (expensive) capacity.
So how can organizations tackle this challenge? The answer lies in Hadoop. Continue reading Why You Should Offload Your Data Warehouse to Hadoop
As discussed in my previous blog, data in Hadoop is mainly accessed via programming languages such as MapReduce or Python, a scripting language like Pig, or an SQL-like language (Hive). However, the skillset for data analysis most prevalent in IT shops is SQL, which means that Hadoop will have to support a SQL interface in some capacity to appeal to these people and to the widespread BI tools in existence. However, Hadoop was created to process data in a “batch” mode – you submit jobs to analyze massive datasets stored in HDFS. A number of initiatives and solutions are underway that are focused on marrying SQL with Hadoop. This convergence is truly in progress.
Continue reading Taking Hadoop Mainstream: The Convergence of SQL and Hadoop
The four current biggest trends in IT are Social, Mobile, Cloud, and Big Data. In this post, I will discuss the intersection of Mobile and Big Data. Without going into too much detail, I’ll first review what Big Data is. The first aspect of Big Data is gathering and storing extreme volumes (often in petabytes) of data from databases, log files, social media, connected devices, external data marketplaces and internal systems. Data is “deep,” meaning it is transactional in nature and nothing is thrown away. Continue reading How Mobile Enables Big Data
A number of people I have spoken to have a hard time differentiating between various solutions for traditional data warehousing and the upcoming big data solutions. At Knowledgent, we have a very simple way of looking at them.
Continue reading Data Warehouses, Analytic Databases and Hadoop – How are they different?