Literature Review - in Memory Technology

[pic 1]

Contents

Introduction        

Paper – 1 – In-Memory Databases in Business Information Systems        

Paper – 2 – Is ‘In-Memory’ always the right choice?        

Paper – 3 – Shifting the BI Paradigm with In Memory Database Technologies        

Research Gaps Identified        

Bibliography        

Introduction

        The primary focus for this literature review will be on an emerging technology with respect to databases. We are aware that traditional databases technologies have evolved over time and the need for faster data fetch from databases for analytical purposes has brought around techniques like query optimization, parallel processing of data sets, etc.

For this review, we are looking at the “In-Memory” technology which has taken database technologies to the next level in terms of response time and analytical capabilities. The following papers/articles have been utilized for the review:

1. “In-Memory Databases in Business Information Systems”, 2011

Authors:

Jens Lechtenbörger, Gottfried Vossen, Alexander Zeier, Jens Krüger, Jürgen Müller, Wolfgang Lehner, Donald Kossmann, Benjamin Fabian, Oliver Günther, Robert Winter

1. “Is ‘In-Memory’ always the right choice?”, 2013

Authors:

Katrina Read

1. “Shifting the BI Paradigm with In-Memory Database Technologies”,2007

Authors:

John J. Gill

Paper – 1 – In-Memory Databases in Business Information Systems

In – Memory databases reside on the main memory (RAM-Random Access Memory) within a computer system unlike normal databases that reside on a hard disk of a computer system. This helps to improve data access speeds to a great extent when compared to traditional hard disks which transfer data using I/O (Input/Output) mechanisms. (Loos et al., 2011)

In today’s market, the storage capacities are increasing massively and have become more affordable as well. This makes the In-Memory technology suitable for the implementation of Business Information Systems. In-Memory technology is intended to provide the flexibility for performing transactions and analytics on the same database thereby eliminating the need to maintain two separate databases for performing transactions (OLTP-Online Transaction Processing) and doing analytics (OLAP-Online Analytics Processing) respectively. (Loos et al., 2011)

Technology Potential(Loos et al., 2011):

1. Integration possible between transaction processing and analytical processing.
2. Can bring up new ways of structuring, modeling and programming techniques.

Consequences for OLTP and OLAP(Loos et al., 2011):

1. “Control Flow” driven architecture will shift to “Data-flow” driven architectures.
2. The concept of “data availability” will shift to “Need-to-know”.

On a brief note, in-memory technology relies on primary memory (RAM) instead of secondary memory (Hard Disk). Traditional databases store a minimal portion like frequently accessed data, data indexes, etc. in the main memory (RAM) and the remaining data is fetched as per the queries executed from the respective applications. On the other hand, the whole database is stored in the main memory in the case of in-memory technology. (Loos et al., 2011)

        The read/write operation from/to a physical hard disk takes more time than the read/write operation from/to the main memory (RAM). This explains the performance improvement that is achievable with the introduction of in-memory technology. (Loos et al., 2011)

        Apart from performance, there are some down sides such as the non-availability of conventional database features such as:

1. Data Recovery from Data Logs
2. Query Optimization allowing quick data fetch
3. Data Organization using Table Indexes

However, these shortcomings have been covered with the introduction of “Column Stores”. Hence, a huge potential of combining OLTP and OLAP systems is now becoming a possibility. (Loos et al., 2011)

There is a mentioning of the following in-memory databases:

1. MonetDB

1.  A column store residing on non-volatile RAM which has a tuned query execution engine.

1. HyPer

1. Main memory database capable of writing transactions and executing multiple analytical queries in a single instance or snapshot.
2. Uses hardware supported page shadowing controlled by the MMU (Memory Management Unit) of the processor
3. Transactions are written on separate DB partitions hence avoiding the need for DB locks.

The author talks also about increase of power and energy consumption with the introduction of larger DRAMs for in-memory technology and this could be controlled by using the low power modes of the DRAM devices with the help of throttling and scheduling techniques.  (Loos et al., 2011)

The migration to in-memory technology will require the rebuilding of existing applications to be compatible and utilize the full potential of the new technology. (Loos et al., 2011)

The author mentions that the point has been reached such that the availability of increasing storage density of main memory will alter the way application systems are built in the future which is separated into two phases:

1. Simplification :

1. Involves Rewriting of the database and application logic layers with simplified design.
2. The complex features such as decision support, tuning and storage hierarchy will not be required with the large availability of main memory.

1. Usage of Main Memory :

1. Even though main memory is fast, there will still be improvement in terms of optimization on main memory to reduce energy utilization thereby bringing about the need for lesser hardware.
2. This will also increase complexity for the database and application developers to bring about optimization techniques.

Thus, it is evident that after a stage of simplification, there will be a stage of increasing complexity. (Loos et al., 2011)

Factors influencing the adoption of In-Memory Technology:

1. OLAP is much quicker with the use of in-memory technology.
2. Reduction in cost and lower latency for complex queries.
3. Hardware required is highly affordable.
4. Will allow better data mining and data visualization.
5. Column Store method allows parallel processing of analytical OLAP queries.

Risks/drawbacks to be considered for the adoption of in-memory technology:

1. Rebuilding of database of and application logic layers for BI systems.
2. Investment required for procuring new hardware and RAM.
3. Software Licensing Costs and maintenance costs need to be factored as well.
4. New backup & recovery technologies have to be implemented.

Paper – 2 – Is ‘In-Memory’ always the right choice?

With the decrease in costs of DRAM required for the implementation of in-memory technology, companies are analyzing the effectiveness of using in-memory technology for their business information systems. (Read, 2013)

...