Large mission critical applications use Straight Through Processing, and these systems need to be highly scalable. So, when you apply for these high paying jobs, it really pays to prepare for your job interviews with the following questions and answers.
Q. What is Straight Through Processing (STP)?
A. This is the definition from INVESTOPEDIA.
“An initiative used by companies in the financial world to optimize the speed at which transactions are processed. This is performed by allowing information that has been electronically entered to be transferred from one party to another in the settlement process without manually re-entering the same pieces of information repeatedly over the entire sequence of events.”
Q. How will you go about designing a STP system?
A. Most conceptual architectures use a hybrid approach using a combination of different architectures based on the benefits of each approach and its pertinence to your situation. Here is a sample hybrid approach depicting an online trading system, which is a STP system.
STP – Straight Through Processing
The above system is designed for:
- Placing Buy/Sell online trades real time. The trades are validated first and then sent all the way to Stock Exchange system using the FIX (Financial Information eXchange) protocol.
- Once the trade is matched, the contract notes are asynchrnously issued via the SETTLEMENT queue, and processed by an ESB (Enterprise Service Bus) system like web Methods, Tibco, or Websophere MQ. Here are some useful links on asynchronous processing.
- MOLAP: Multidimensional OLAP – enabling OLAP by provding cubes.
- ROLAP: Relational OLAP – enabling OLAP using a relational database management system
- Oracle Enterprise BI Server, Oracle Hyperion System
- Microsoft BI & OLAP tools
- IBM Cognos Series 10
- SAS Enterprise BI Server
- JasperSoft (open source)
The above system is an operational OLTP (i.e. On-Line Transaction Processing) system. These systems are also known as STP (i.e. Straight Through Processing) system. This leads to another question.
Q. What is the difference between Data Warehousing and OLAP?
A. A data warehouse serves as a repository to store historical data that can be used for analysis. OLAP is Online Analytical processing that can be used to analyse and evaluate data in a warehouse. The warehouse has data coming from varied sources including OLTP sources.
Q. What is the difference between OLTP and OLAP?
A. OLTP stands for On-Line Transaction Processing and OLAP stands for On-Line Analytical Processing. OLAP contains a multidimensional or relational data store designed to provide quick access to pre-summarized data & multidimensional analysis.
OLTP vs OLAP
| OLTP | OLAP |
| Source data is operational data. This data is the source of truth. | Data comes from various OLTP data sources as shown in the above diagram |
| Transactional and normalized data is used for daily operational business activities. | Historical, de-normalized and aggregated multidimensional data is used for analysis and decision making (i.e. for business intelligence). |
| Data is inserted via short inserts and updates. The data is normally captured via user actions via web based applications. | Periodic (i.e. scheduled) and long running (i.e. during off-peak) batch jobs refresh the data. Also, known as ETL process as shown in the diagram. |
| The database design involves highly normalized tables. | The database design involves de-normalized tables for speed. Also, requires more indexes for the aggregated data. |
| Regular backup of data is required to prevent any loss of data, monetary loss, and legal liability. | Data can be reloaded from the OLTP systems if required. Hence, stringent backup is not required. |
| Transactional data older than certain period can be archived and purged based on the compliance requirements. | The volume of this data will be higher as well due to its requirement to maintain historical data. |
| The typical users are operational staff. | The typical users are management and executives to make business decisions. |
| The space requirement is relatively small if the historical data is archived. | The space requirement is larger due to the existence of aggregation structures and historical data. Also requires more indexes than OLTP. |
There are a number of commercial and open-source OLAP (aka Business Intelligence) tools like:
The OLAP tools are well known for their drill-down and slice-and-dice functionality. Also they enable users to very quickly analyze data by nesting the information in tabular or graphical formats. They generally provide good performance due to their highly indexed file structures (i.e. cubes) or in-memory technology.
Q. What is an OLAP cube?
A. An OLAP cube will connect to a data source to read and process the raw data to perform aggregations and calculations for its associated measures. Cubes are the core components of OLAP systems. They aggregate facts from every level in a dimension provided in a schema. For example, they could take data about products, units sold and sales value, then add them up by month, by store, by month and store and all other possible combinations. They’re called cubes because the end data structure resembles a cube.
OLAP Cube
