Big Data Integration: Where are we heading to?

I happen to hear that Splunk.com was ranked for the 4th best innovative company on planet earth in 2013 by an agency couple of weeks back. So what do they do? They provide a software platform for real-time operational intelligence. In simple terms they are the enterprise's Google equivalent search tool to search for information from the machine generated data such as log files and alerts from operational monitoring tools. So a first word, what is so big about it. The key is the answer "big data" where the volumes are really big and someone is trying to look for a needle in the haystack.


What does Big Data mean to you? Here is an interpretation. Big Data is something big that you cannot handle in your standard databases or data warehouses where majority of data does not make sense directly for you, but there is a possibility that you could find something interesting from it if you run the expected analytic rules on it. Being said that, we have been assuming that the data is coming in a structured format just like how Splunk handles. What about unstructured data or format/schema less data? How would your machine read and interpret such a data?

Immediately you think about using some fuzzy neural logic that could work in natural language processing (NLP) across various languages. Is that it? There is another think dimension between the NLP and structured data and it is data that comes with its own metadata such as an XML without the schema or a JSON without a schema or No-SQL database entry where every entry has its own structure. This is the next immediate topic to be resolved where the structure is coming in a free flowing format which carry's its on metadata or descriptors along with it. We need to solve this format before moving to NLP.

-still in the works...keep watching

Social Media Trends in 2013

These are the Social Media trends in 2013:

• Big data will get Social
• Big data will augment data from transactional records to customer behavior and their social graph behavior on the web along with location and device/mobile generated information.
• Social CRM
• Social data will be added to the CRM and marketing tools to find trends in sentiment, behavior and individual preferences of customers.
• Social media integration with other marketing
• People spend increasing amounts of time on social networking sites and marketers want to tap into the social media distribution channel
• Social media monitoring tools
• Marketing ROI counted by measuring results
• Social media monitoring tools combined with business metrics will lead to better understanding of the value of social interaction
• Social media budgets will be much bigger
• By finding new ways to interact and engage consumers.
• Expecting the spending to be double by the end of 2016.

Sybase acquisition - A winner for SAP

I believe that SAP did a very strategic acquisition by buying Sybase. When I look at the Sybase products, they were very much technologically advanced than many of the equivalent software that was available in the market such as Sybase ASE, Sybase Unwired Platform, Sybase CEP etc.

As I hear (since I was not tracking the company personally), Sybase has historically very poor sales team and hence could not really move forward in the competitive market. With the SAP acquisition and the SAP's war with Oracle, Sybase can add a lot of mileage to taking on the Oracle competition with respect to the Database market. Even though SAP sells the HANA as the next generation appliance to compete in the big data market against the Oracle, I think practically I would see Sybase products really taking on the Oracle products on a head2head competition. In addition, all the Oracle database market within SAP would be replaced by Sybase. This means that nearly 80% of the 65,000 of the SAP ERP installations which consists of Oracle database would go away for Oracle in the next few years.

OnPremises Software to come back

According to Gartner, the 30% organizations who have been using cloud applications would switch back to on-premises software by 2014.

The other major trends are Hybrid Cloud services which includes private and public cloud along with on-premises applications


http://www.forbes.com/sites/ericsavitz/2012/10/22/gartner-10-critical-tech-trends-for-the-next-five-years/

To augment this information with the data processing capabilities, when we look at many of the cloud application service providers, they either restrict you from accessing their cloud applications with their data processing capabilities
a) Number of API calls per day (for eg. Salesforce.com 5000, 25000, 100,000,000 calls)
b) the total data volume that you can process per day (for example: 500MB per day)
c) total data storage that you can work (eg. Salesforce.com 1GB, 10GB or 100GB)
d)  Data Processing Units per month (eg. max 250,000 DPU from Datasift)

BigData & NoSQL databases- The market trend

During the past couple of years there have been a lot of technology news around the NoSQL databases especially with the Big Data Story revving up. Some of them are MongoDB, Cassandra, Hbase, Couch DB, Redis, membase, Neo4J, CouchDB, Accumulo, TripleStore, membase, DynamoDB etc. Most of these NoSQL databases can be categorized into few groups, they are as follows:

1. Key-Value Stores:
   This technology uses a hash table where there is a unique key and a pointer to a particular item of data. The Key/value model is the simplest and easiest to implement. But it is inefficient when you are only interested in querying or updating part of a value
   Examples: Tokyo Cabinet/Tyrant, Redis, Voldemort, Oracle BDB, Amazon SimpleDB, Riak
   Typical Applications: Content caching (Focus on scaling to huge amounts of data, designed to handle massive load), logging, etc.
   Strengths: Fast lookups
   Weakness: Stored data has no schema
2. Column family store:
   This type was created to store and process very large amounts of data distributed over many machines. There are still keys but they point to multiple columns. The columns are arranged by column family.
   Examples: Cassandra, HBase, Riak
   Typical Applications: Distributed file systems
   Strengths: Fast lookups, good distributed storage of data
   Weakness: Very low-level API
3. Document store:
   These were inspired by Lotus Notes and are similar to key-value stores. The model is basically versioned documents that are collections of other key-value collections. The semi-structured documents are stored in formats like JSON. Document databases are essentially the next level of Key/value, allowing nested values associated with each key.  Document databases support querying more efficiently.
   Examples: CouchDB, MongoDb, Elastic Search
   Typical Applications: Web applications, Content Management systems
   Strengths: Tolerant of incomplete data
   Weakness: Query performance, no standard query syntax
4. Graph Databases:This model follows the flexible graph model which can scale across multiple machines. This does not have the tables of rows and columns and the rigid structure of SQL, . NoSQL databases do not provide a high-level declarative query language like SQL to avoid overtime in processing. Rather, querying these databases is data-model specific. Many of the NoSQL platforms allow for RESTful interfaces to the data, while other offer query APIs.
   Examples: Neo4J, InfoGrid, Infinite Graph
   Typical Applications: Social networking, Recommendations etc
   Strengths: Graph algorithms e.g. shortest path, connectedness, n degree relationships, etc.
   Weakness: Has to traverse the entire graph to achieve a definitive answer. Not easy to cluster.

 Eventhough there are several of these NoSQL databases in the market, I am not sure whether they can be compared as apples to apples and consider most of them specialized for very specific horizontal usecases or vertical needs. Hence there is one single NoSQL database that can solve your whole enterprise wide problem and hence the implementation of these NoSQL DB would be decided by different departments within the organization.

NoSQL databases are not a replacement for the conventional RDBMS. It is expected to supplement or   augment the data for other business needs. It is also expected that in the coming years some of the established Database vendors such as Oracle, Sybase, Microsoft, IBM would take over some of the active NoSQL databases and merge them into their portfolio.


The worldwide NoSQL software market is expected to reach $3.4 Billion by 2018. It is expected to grow  at a CAGR of 21% between 2013 and 2018. NoSQL market is expected to generate $14 Billion in revenues for the period 2013 – 2018.

The NoSQL market has been very active in the past 1 year especially with several venture capital funding, mergers & acquisitions and new product offerings:

• September 2012 – In-Q-Tel, the venture investment arm of the U.S. Intelligence Community, invests in 10gen, developer of the MongoDB open source database;
• August 2012 – Sqrrl, a National Security Agency’s spin-off startup, raised $2Mln to develop NoSQL database Accumulo
• July 2012 – NuoDB raises $10Mln to develop cloud NoSQL database that behaves like traditional SQL
• June 2012 – Cloudant launches NoSQL data layer service for Windows Azure;
• May 2012 – 10gen secures $42 million in venture funding;
• January 2012 – Amazon launches DynamoDB, a new NoSQL data service;
• January 2012 – Oracle announces the availability of Oracle Big Data Appliance and partners with Cloudera to provide an Apache Hadoop distribution and tools for the Big Data Appliance;
• November 2011 – Cloudera Inc., the provider of Apache Hadoop-based data management software and services, raises $40 million.
• November 2011 – Basho, the company behind Riak, raises $5 Mln.  

Some Usecases of When to Use NoSQL: 

• Logging/Archiving. Log-mining tools are handy because they can access logs across servers, relate them and analyze them.
• Social Computing Insight. Many enterprises today have provided their users with the ability to do social computing through message forums, blogs etc.
• External Data Feed Integration. Many companies need to integrate data coming from business partners. Even if the two parties conduct numerous discussions and negotiations, enterprises have little control over the format of the data coming to them. Also, there are many situations where those formats change very frequently – based on the changes in the business needs of partners.
• Front-end order processing systems. Today, the volume of orders, applications and service requests flowing through different channels to retailers, bankers and Insurance providers, entertainment service providers, logistic providers, etc. is enormous. These requests need to be captured without any interruption whenever an end user makes a transaction from anywhere in the world. After, a reconciliation system typically updates them to back-end systems as well as updates the end user on his/her order status.
• Enterprise Content Management Service. Content Management is now used across companies’ different functional groups, for instance, HR or Sales. The challenge is bringing together different groups using different meta data structures in a common content management service.
• Real-time stats/analytics. Sometimes it is necessary to use the database as a way to track real-time performance metrics for websites (page views, unique visits, etc.)  Tools like Google Analytics are great but not real-time — sometimes it is useful to build a secondary system that provides basic real-time stats. Other alternatives, such as 24/7 monitoring of web traffic, are a good way to go, too.

What Type of Storage Should you use?

Here’s a short summary that might help you make your decision:

NoSQL	Storage should be able to deal with very high load You do many write operations on the storage You want storage that is horizontally scalable Simplicity is good, as in a very simple query language (without joins)
RDBMS	Storage is expected to be high-load, too, but it mainly consists of read operations You want performance over a more sophisticated data structure You need powerful SQL query language

The other big news is around the Cloud databases such as Google BigQuery, Amazon Redshift and Cloudera Impala.

This big data space would be hotter by the end of 2013  as many they would solve many of the big-data and analytical problems for the businesses. We would also see several investments happening from the established vendors in this space and also some consolidation by M&A by the end of 2014/early 2015. A nice space to watch!

News from Feb 2014: IBM buys NoSQL cloud provider Cloudant
http://www.computerworld.com/s/article/9246543/IBM_buys_NoSQL_cloud_provider_Cloudant 

OLAP-OLTP-InMemory Convergence

Many vendors had started designing and selling the concept of OLAP(On-Line Analytical Processing) systems in the late 90's to provide better reporting of the interpretation of the data. The OLAP tools stored per-aggregated results unlike the pure database or the OLTP (On-Line Transacational Processing) systems.Tools like Hyperion Essbase, Cognos PowerPlay, were some of the prominent players in the market during the 90s.  In the late 2000's SAP introduced SAP Business Warehouse to compete in the OLAP and Datawarehousing space.

There were 3 type of OLAP systems

1. Multi-dimensional OLAP(M-OLAP): These systems had real multidimensional cubes defined within them. eg: Cognos Powerplay, Hyperion Essbase, Microsoft SSAS, SAS/MDDB, Oracle Express..etc
2.  Relational OLAP(R-OLAP): These systems were mainly modeled on relational star schema. eg: SAP BW, Microstrategy etc
3. Hybrid-OLAP(H-OLAP): Most of the M-OLAP systems support this configuration. This is done by using relational star schema's for detailed level and multidimensional for the aggregation representation

OLAP tools were highly successful since it was an easy way to implement the business requirement with better results unlike the complex SQL query written on a relational database system. It was mandatory to define the KPI's and dimensions for modelling a cube and the models were not standardized then. Over the years, the data started growing in the OLAP systems and there was a constant need to improve the performance of such systems. SAP has introduced an boosted appliance hardware called BIA (Business Intelligence Accelerator) to improve the OLAP processing over the SAP BW system.

However, with the recent explosion of data in Terabytes and Petabytes, most of the above mentioned systems cannot scale to process the data and arrive at the business expected solutions. This has triggered few of the leaders in this space to work on in-memory databases which would be a true convergence of the OLAP and the OLTP system.

SAP has introduced the SAP HANA in 2011 for which Oracle tried to answer with Exadata. However, Exadata could not meet the bench marks provided by HANA. This has triggered Oracle to resurrect an acquisition product named "TimesTen" as an answer to the SAP's HANA solution. According to SAP, they would actually move all the processing capability from the application layer in SAP BW to the in-memory database layer like the conventional DB since the DB can process faster than the application layer. Many of the SAP customers were very much confused by this move from SAP especially considering that SAP did lot of marketing for the BW application level processing capabilities and the customers spend a lot of time and energy writing those models on the BW application layer.

On the SAP business side, It seems SAP has acquired nearly 200 customers with an average deal size of USD 600-800K for every license of which 10-20 customers have gone on production in the last 8 months.

Future of Data Integration- Roadmap

Some one said that data in a data warehouse is like an old cloth in your closet, some time you might not pick it for the next two years. Many of us  have at least 1TB of pocket hard drive that contains lot of data including musics, photos and videos. How often have you deleted your old data in your personal hard drive or throw away an old data CD/DVD. I believe most of us never throw the data considering that the old data would be useful for us some day. How often has the 'someday' come to you, may be never till now.

Similar is the case with your business. In many cases, you are forced to keep the historical data due to legal compliance and you never use it. What would you do if you want to use it, how organized is your data for searching, filtering, sorting and presenting it? During the last few months, I was looking into the new demands that were being considered in the Data Integration space and how can it be projected for this decade. The key spaces that needs a good data integration system to take care has been observed as follows:

• Big Data
• Social Media
• Mobile integration
• Zero downtime
• Demand based real time data synchronization
• Cloud data integration

What does this mean, there is new and more data that is being produced for various business analytical applications to help the business managers to decide or run an automation system. There has been a severe competition in this space in the IT products and a lot of consolidation has happened during the 2000 decade. Many of the BI and analytics companies have been bought out by software giants and have been building mammoth products to help customers to help with business decision. One of the most interesting advancement in this space was the in-memory based database launched by SAP AG called "HANA". The in-memory systems are supposed to work hundred times faster than the conventional relational databases. The need for such a system was that many of the SAP based reports used to run for hours as SAP calls "lunch time processing".

In short, the data is being captured from every event, action unlike the earlier manual system where most of the data used to be on the books. The volume of data is exploding everywhere and the time to process is reducing (actually the time is constant but the requirement for processing time is low). The data needs needs to be synchronized and made available to all the decision support systems in real time which means real-time data streaming is required to meet the business demand. The data could be residing on the cloud systems such as Salesforce.com or Microsoft Dynamics CRM cloud based systems which are hosted outside of the enterprise. For this to be efficient, this means that the data filtering and interchange should be minimal so that the data interchange time is used most efficiently. The demand for the enterprise IT integration software to select, filter, sort, compress, encrypt and process the data is having new challenges in this era. In simple terms, the integration software is expected to perform at a higher scale that it used to perform earlier.

Technology Trends for 2012

I too believe that the  top three things for 2012 is going to be the following:

• Big Data
• Social Media 
• Agile and Pervasive BI solution

Among the top 3, I believe the last one (Agile and pervasive BI solution) has the biggest market share in the top 3 items. Some of the revolutions would probably happen combined with the cloud solutions.

HANA has been the magical word from SAP for 2011. However, here is one of the quotes from the Redbull implementation "Redbull did not see a big improvement in query performance because they were already using BWA.  Moreover, they found that HANA does not have all the features that BWA gave them. But they think this is ok, because SAP is committed to working with them to put in features in future revisions of HANA." I did not find this comment quite impressive after all with the in-memory processing (Ref: http://andvijaysays.wordpress.com/2011/11/09/redbull-migrates-bw-to-hana-i-am-suitably-impressed/)


Read more from the analysts here:
No SQL Databases
Business Intelligence 2012
Data Virtualization

Software Product Development - What you should not do

What are the cardinal sins in Software Development Life Cycle (SDLC)?

Rule 1: Never forget to support and maintain your existing active products. The first priority should go for supporting your existing product over the new committed release. Supporting your existing products should have much importance even though your company strategy is to move to the new product and technology. If you cannot service and support your existing products, how would you expect the same customer to buy your new product and use it.

Rule 2: Never create any regression bugs between the old version and the new versions. Customers remember and never forgets the bugs that you have introduced in the new version. 

Rule 3: Never forget to merge the hot-fixes or bug-fixes or patches into the next immediate main release. Customers do not like to face the same issue that they faced in the previous version in the new version also.

Rule 4: Never add additional features without the permission of Product Owner or Product Manager. Never over complicate the code base to create a fancy or cool feature. Oftentimes, such code is overworked immediately after the release and not removed. It is easy to add code, but very difficult to remove code as you never know if any customer is using that feature. Remember that you can never release a  software bug free product in your life. This is mainly due to the fact that many software products are used in various other use cases other than it is designed for. So write simple code which is easy to scale and maintain.


All the above mentioned rules are very important to have a successful product satisfaction for the customer. Do always remember to view or empathize from the view of your customer.

SAP HANA -Lunchtime processing

SAP's open war with Oracle is peaking out, see the comment from one of the SAP executives on the Oracle ambition
http://blogs.sap.com/bobevans/2011/06/23/the-top-10-reasons-sap-hana-is-disrupting-larry-ellison%E2%80%99s-grand-plans/

When I read the statement from the blog "Oracle has said that the best way to reduce complexity in your IT systems and across your IT infrastructure is to buy more Oracle products and technology."; I feel no different that SAP is behaving differently with HANA. One has to buy a separate new appliance box which has to be replaced. What happens to the BWA at the customer site, even that is a waste. Once the HANA support is launched for BW and ERP, then the customer has to buy more and more SAP licenses instead of continuing the with Oracle maintenance support. Similarly, the data loading is currently propriety using SAP BOBJ DS which is not exposed to third party software integrators. What does this mean, for loading and extracting data, one needs to buy more and more SAP software. So where is SAP behaving differently from Oracle. Imagine what happens if SAP exposes the interfaces to third party integrators, the existing integrators would provide an additional integration without additional cost. This way the cost of SAP end customers would be less, but see the SAP strategy to squeeze and extract the customers more and more with their proprietary software.


I was glad to see such a nice blog where an employee speaks the truth about their own products and the customer experience.

LunchTime Processing

I do not think this employee would have told this in the public if there was no alternative available currently. However,

Initially SAP reports used to run overnight and then for improved analytics, SAP has released a product called Business Warehouse Accelerator (BWA) few years back to speedup the business analytics reporting. However, the BWA appliance came in a big storm but just died away on the way as it really did not help the quantum of the data processing. It just stayed as a lunch time processing. Now SAP is storming with its new HANA and showcases as 4 hour processing would be done in 4 secs. I would be very glad to see such an innovation improvement, the only thing that I do not know what does it cost to make this change. HANA 1.0 stands as a separate appliance and would try to duplicate all the data that is sitting on a standard SAP database. Would it mean adding tera bytes of memory in line with the actual hard disk capacity since the HANA processing is all about in-memory processing which they claim to be cheaper than adding other hardware parts.  I never heard in any of the forums anything about fault-tolerance for this appliance and how it can be configured for high availability.

Even though such PoC would show up good results, I am not really sure whether the Lunch time processing would become as a Coffee time processing with the limitations to the hardware addition to especially an appliance which does not really let us scale the hardware. How many times were you forced to change your laptop either the device does not let you scale to the additional system requirement where as a typical assembled PC could let you scale for over a decade or so like attaching a Blue-ray disc or a better LED monitor or adding additional memory that are easily available to be attached. 

Background on Transaction Processing

TP systems that existed during 1990

1. IMS http://www-306.ibm.com/software/data/ims/

2. CICS and LU 6.2

3. Guardian 90( Tandem Guardian, TMF, Pathway and Nonstop SQL and Expand) Now part of HP

4. DECdta (Digital's Distributed Transaction Processing Architecture) Now part of HP
http://www.hpl.hp.com/hpjournal/dtj/vol3num1/vol3num1art1.pdf

5. X/Open DTP, OSI-TP, CCR(http://pavg.stanford.edu/rapide/examples/xopen/)

6, Universal Transaction Monitor(UTM) on Siemens Nixdorf main frames with BS2000 Operating Systems

7. ADABAS TPF( Software AG)http://www1.softwareag.com/corporate/products/adabas/default.asp(now a pure DB)

8. Encina( Transarc Corporation, a subsidary of IBM)Encina OTS is part of IBM’s TXSeries, which includes Encina and Distributed CICS for secure, high-volume transaction processing. In addition to TXSeries, Transarc develops and markets DFS, a global file system and TXSeries

9. Tuxedo (AT&T now with BEA)
http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/tux/