OLAP-OLTP-InMemory Convergence

Many vendors had started designing and selling the concept of OLAP(On-Line Analytical Processing) systems in the late 90's to provide better reporting of the interpretation of the data. The OLAP tools stored per-aggregated results unlike the pure database or the OLTP (On-Line Transacational Processing) systems.Tools like Hyperion Essbase, Cognos PowerPlay, were some of the prominent players in the market during the 90s.  In the late 2000's SAP introduced SAP Business Warehouse to compete in the OLAP and Datawarehousing space.

There were 3 type of OLAP systems

1. Multi-dimensional OLAP(M-OLAP): These systems had real multidimensional cubes defined within them. eg: Cognos Powerplay, Hyperion Essbase, Microsoft SSAS, SAS/MDDB, Oracle Express..etc
2.  Relational OLAP(R-OLAP): These systems were mainly modeled on relational star schema. eg: SAP BW, Microstrategy etc
3. Hybrid-OLAP(H-OLAP): Most of the M-OLAP systems support this configuration. This is done by using relational star schema's for detailed level and multidimensional for the aggregation representation

OLAP tools were highly successful since it was an easy way to implement the business requirement with better results unlike the complex SQL query written on a relational database system. It was mandatory to define the KPI's and dimensions for modelling a cube and the models were not standardized then. Over the years, the data started growing in the OLAP systems and there was a constant need to improve the performance of such systems. SAP has introduced an boosted appliance hardware called BIA (Business Intelligence Accelerator) to improve the OLAP processing over the SAP BW system.

However, with the recent explosion of data in Terabytes and Petabytes, most of the above mentioned systems cannot scale to process the data and arrive at the business expected solutions. This has triggered few of the leaders in this space to work on in-memory databases which would be a true convergence of the OLAP and the OLTP system.

SAP has introduced the SAP HANA in 2011 for which Oracle tried to answer with Exadata. However, Exadata could not meet the bench marks provided by HANA. This has triggered Oracle to resurrect an acquisition product named "TimesTen" as an answer to the SAP's HANA solution. According to SAP, they would actually move all the processing capability from the application layer in SAP BW to the in-memory database layer like the conventional DB since the DB can process faster than the application layer. Many of the SAP customers were very much confused by this move from SAP especially considering that SAP did lot of marketing for the BW application level processing capabilities and the customers spend a lot of time and energy writing those models on the BW application layer.

On the SAP business side, It seems SAP has acquired nearly 200 customers with an average deal size of USD 600-800K for every license of which 10-20 customers have gone on production in the last 8 months.

Future of Data Integration- Roadmap

Some one said that data in a data warehouse is like an old cloth in your closet, some time you might not pick it for the next two years. Many of us  have at least 1TB of pocket hard drive that contains lot of data including musics, photos and videos. How often have you deleted your old data in your personal hard drive or throw away an old data CD/DVD. I believe most of us never throw the data considering that the old data would be useful for us some day. How often has the 'someday' come to you, may be never till now.

Similar is the case with your business. In many cases, you are forced to keep the historical data due to legal compliance and you never use it. What would you do if you want to use it, how organized is your data for searching, filtering, sorting and presenting it? During the last few months, I was looking into the new demands that were being considered in the Data Integration space and how can it be projected for this decade. The key spaces that needs a good data integration system to take care has been observed as follows:

• Big Data
• Social Media
• Mobile integration
• Zero downtime
• Demand based real time data synchronization
• Cloud data integration

What does this mean, there is new and more data that is being produced for various business analytical applications to help the business managers to decide or run an automation system. There has been a severe competition in this space in the IT products and a lot of consolidation has happened during the 2000 decade. Many of the BI and analytics companies have been bought out by software giants and have been building mammoth products to help customers to help with business decision. One of the most interesting advancement in this space was the in-memory based database launched by SAP AG called "HANA". The in-memory systems are supposed to work hundred times faster than the conventional relational databases. The need for such a system was that many of the SAP based reports used to run for hours as SAP calls "lunch time processing".

In short, the data is being captured from every event, action unlike the earlier manual system where most of the data used to be on the books. The volume of data is exploding everywhere and the time to process is reducing (actually the time is constant but the requirement for processing time is low). The data needs needs to be synchronized and made available to all the decision support systems in real time which means real-time data streaming is required to meet the business demand. The data could be residing on the cloud systems such as Salesforce.com or Microsoft Dynamics CRM cloud based systems which are hosted outside of the enterprise. For this to be efficient, this means that the data filtering and interchange should be minimal so that the data interchange time is used most efficiently. The demand for the enterprise IT integration software to select, filter, sort, compress, encrypt and process the data is having new challenges in this era. In simple terms, the integration software is expected to perform at a higher scale that it used to perform earlier.

Technology Trends for 2012

I too believe that the  top three things for 2012 is going to be the following:

• Big Data
• Social Media 
• Agile and Pervasive BI solution

Among the top 3, I believe the last one (Agile and pervasive BI solution) has the biggest market share in the top 3 items. Some of the revolutions would probably happen combined with the cloud solutions.

HANA has been the magical word from SAP for 2011. However, here is one of the quotes from the Redbull implementation "Redbull did not see a big improvement in query performance because they were already using BWA.  Moreover, they found that HANA does not have all the features that BWA gave them. But they think this is ok, because SAP is committed to working with them to put in features in future revisions of HANA." I did not find this comment quite impressive after all with the in-memory processing (Ref: http://andvijaysays.wordpress.com/2011/11/09/redbull-migrates-bw-to-hana-i-am-suitably-impressed/)


Read more from the analysts here:
No SQL Databases
Business Intelligence 2012
Data Virtualization

Software Product Development - What you should not do

What are the cardinal sins in Software Development Life Cycle (SDLC)?

Rule 1: Never forget to support and maintain your existing active products. The first priority should go for supporting your existing product over the new committed release. Supporting your existing products should have much importance even though your company strategy is to move to the new product and technology. If you cannot service and support your existing products, how would you expect the same customer to buy your new product and use it.

Rule 2: Never create any regression bugs between the old version and the new versions. Customers remember and never forgets the bugs that you have introduced in the new version. 

Rule 3: Never forget to merge the hot-fixes or bug-fixes or patches into the next immediate main release. Customers do not like to face the same issue that they faced in the previous version in the new version also.

Rule 4: Never add additional features without the permission of Product Owner or Product Manager. Never over complicate the code base to create a fancy or cool feature. Oftentimes, such code is overworked immediately after the release and not removed. It is easy to add code, but very difficult to remove code as you never know if any customer is using that feature. Remember that you can never release a  software bug free product in your life. This is mainly due to the fact that many software products are used in various other use cases other than it is designed for. So write simple code which is easy to scale and maintain.


All the above mentioned rules are very important to have a successful product satisfaction for the customer. Do always remember to view or empathize from the view of your customer.

SAP HANA -Lunchtime processing

SAP's open war with Oracle is peaking out, see the comment from one of the SAP executives on the Oracle ambition
http://blogs.sap.com/bobevans/2011/06/23/the-top-10-reasons-sap-hana-is-disrupting-larry-ellison%E2%80%99s-grand-plans/

When I read the statement from the blog "Oracle has said that the best way to reduce complexity in your IT systems and across your IT infrastructure is to buy more Oracle products and technology."; I feel no different that SAP is behaving differently with HANA. One has to buy a separate new appliance box which has to be replaced. What happens to the BWA at the customer site, even that is a waste. Once the HANA support is launched for BW and ERP, then the customer has to buy more and more SAP licenses instead of continuing the with Oracle maintenance support. Similarly, the data loading is currently propriety using SAP BOBJ DS which is not exposed to third party software integrators. What does this mean, for loading and extracting data, one needs to buy more and more SAP software. So where is SAP behaving differently from Oracle. Imagine what happens if SAP exposes the interfaces to third party integrators, the existing integrators would provide an additional integration without additional cost. This way the cost of SAP end customers would be less, but see the SAP strategy to squeeze and extract the customers more and more with their proprietary software.


I was glad to see such a nice blog where an employee speaks the truth about their own products and the customer experience.

LunchTime Processing

I do not think this employee would have told this in the public if there was no alternative available currently. However,

Initially SAP reports used to run overnight and then for improved analytics, SAP has released a product called Business Warehouse Accelerator (BWA) few years back to speedup the business analytics reporting. However, the BWA appliance came in a big storm but just died away on the way as it really did not help the quantum of the data processing. It just stayed as a lunch time processing. Now SAP is storming with its new HANA and showcases as 4 hour processing would be done in 4 secs. I would be very glad to see such an innovation improvement, the only thing that I do not know what does it cost to make this change. HANA 1.0 stands as a separate appliance and would try to duplicate all the data that is sitting on a standard SAP database. Would it mean adding tera bytes of memory in line with the actual hard disk capacity since the HANA processing is all about in-memory processing which they claim to be cheaper than adding other hardware parts.  I never heard in any of the forums anything about fault-tolerance for this appliance and how it can be configured for high availability.

Even though such PoC would show up good results, I am not really sure whether the Lunch time processing would become as a Coffee time processing with the limitations to the hardware addition to especially an appliance which does not really let us scale the hardware. How many times were you forced to change your laptop either the device does not let you scale to the additional system requirement where as a typical assembled PC could let you scale for over a decade or so like attaching a Blue-ray disc or a better LED monitor or adding additional memory that are easily available to be attached. 

Background on Transaction Processing

TP systems that existed during 1990

1. IMS http://www-306.ibm.com/software/data/ims/

2. CICS and LU 6.2

3. Guardian 90( Tandem Guardian, TMF, Pathway and Nonstop SQL and Expand) Now part of HP

4. DECdta (Digital's Distributed Transaction Processing Architecture) Now part of HP
http://www.hpl.hp.com/hpjournal/dtj/vol3num1/vol3num1art1.pdf

5. X/Open DTP, OSI-TP, CCR(http://pavg.stanford.edu/rapide/examples/xopen/)

6, Universal Transaction Monitor(UTM) on Siemens Nixdorf main frames with BS2000 Operating Systems

7. ADABAS TPF( Software AG)http://www1.softwareag.com/corporate/products/adabas/default.asp(now a pure DB)

8. Encina( Transarc Corporation, a subsidary of IBM)Encina OTS is part of IBM’s TXSeries, which includes Encina and Distributed CICS for secure, high-volume transaction processing. In addition to TXSeries, Transarc develops and markets DFS, a global file system and TXSeries

9. Tuxedo (AT&T now with BEA)
http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/tux/