Tag Archives: rocket data replicate and sync

Treehouse High Availability Framework Service Ensures Minimal Downtime for Customers using Rocket Data Replicate and Sync on AWS

Posted on by

by Joseph Brady Director of Business Development at Treehouse Software, Inc. and Dan Vimont, Director of Innovation at Treehouse Software, Inc.

Treehouse Software customers are using Rocket Data Replicate and Sync (RDRS) to enable mission-critical Mainframe-to-AWS data replication pipelines.  Some of these production pipelines are providing vital near-real-time synchronization between source and target, and thus can’t afford any significant downtime in the event of failure.  So it’s only natural that a number of our customers have been asking for advice in setting up a high availability (HA) configuration for their RDRS components that run on AWS EC2 instances.  As a result, Treehouse Software provides an HA Framework Professional Services engagement, in which our expert Cloud engineers help customers with delivery, setup, rapid deployment, and customization of an RDRS HA framework.  The HA Framework seamlessly and quickly provides for a Failover EC2 instance to automatically pick up RDRS processing should the Primary instance (running in another Availability Zone) go down.

Setting Up Automatic Failover with EC2 Instances in Different Availability Zones

The core components of the RDRS HA Framework consist of two EC2 instances running in different Availability Zones:  1) a Primary EC2 instance and 2) a Failover EC2 instance.  Both identically-configured EC2 instances are attached to a shared working-storage file system (either an EFS or FSx volume), which allows the Failover instance to seamlessly and quickly pick up RDRS processing should the Primary instance suddenly become unavailable.

A Step Function Automates the Failover Process

In the event of failure of the Primary instance, the HA Framework calls for automatic triggering of a Step Function for reliable failover processing, with steps that include the following:

  • Verify that the Primary instance is unavailable (The RDRS service cannot be active on both instances simultaneously, so this verification is vital.).
  • Redirect all network traffic from the Primary instance to the Failover instance (via Route 53).
  • Start RDRS processing on the Failover instance.

Use a Step Function to Automate the Restoration Process

After operations personnel have completed recovery of the Primary EC2 instance, another Step Function may be manually triggered to reliably transfer RDRS processing back to the Primary instance.

AWS services utilized in the complete recommended framework include Step Functions, Lambda Functions, EventBridge rules, CloudWatch alarms, SNS topics, a Route 53 Private Hosted Zone, and more.  

For more information on Treehouse’s High Availability Framework Professional Service and our other offerings, visit Treehouse Software on the AWS Marketplace.

Interested in discussing your project? Contact us today…

Treehouse Software and Snowflake can help you build custom AI models, provide near-infinite scale data storage, or just have a chat with your enterprise data

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.; Dan Vimont, Director of Innovation at Treehouse Software, Inc.; and Eleonora Savova, Cloud Solutions Architect at Treehouse Software

Treehouse_Snowflake_001

Today’s infectious enthusiasm about Machine Learning (ML) and Artificial Intelligence (AI) has become one of the prime motivators for customers wanting to move enterprise data to the Cloud. Snowflake is the platform of choice for many of these enterprises looking for a Cloud platform onto which they can mobilize data at near-unlimited scale and performance, and tap into advanced ML/AI.

Snowflake’s unified platform is designed for secure development and deployment of Machine Learning (ML) and Large Language Models (LLMs). Here are some of the most exciting new Snowflake AI/ML technologies on offer… 

  • Snowflake Cortex AI: Efficiently process data at scale using cutting-edge models in Snowflake Cortex AI. Using serverless SQL and Python functions, it is easy to process data using fine-tuned models or foundation models such as Snowflake Arctic, Meta Llama 3, Mistral Large, Reka Core, and more. 
  • Snowflake Cortex AI Chat Services: Get answers from analytical tables such as sales transactions without writing any SQL with Cortex Analyst (public preview) text-to-answer service. Quickly find answers hidden among a large set of documents using Cortex Search (public preview) fully managed hybrid search and retrieval service.
  • Snowflake ML: Run distributed feature engineering and custom model training using popular python libraries. Manage features and models at scale with Snowflake Feature Store (public preview) and Model Registry. Democratize predictive model development by using SQL functions that abstract complexity of ML algorithms.

Snowflake also provides optimized storage that includes unstructured, semi-structured, and structured data together with near-infinite scale. The platform gives fast and efficient access, optimized compression, and secure data — all automated. Customers can work with data on-premises, or in open table formats, thus removing lock-in, which allows adaptation to any current or future architectural patterns.

Video: Using AI Within Snowflake For Everyday Analytics…

First things, first…

Before you can start using Snowflake’s AI, ML, and LLM capabilities, you must first load your data into Snowflake. This is where Treehouse Dataflow Toolkit (TDT) comes in with a state-of-the-art, fully automated offering. Working in tandem with Rocket Data Replicate and Sync (RDRS), TDT assures highly-available, auto-scalable, and event-driven data transfers from Kafka pipes to Snowflake, delivered as a set of proprietary microservices. TDT stands out with its strict adherence to Snowflake’s and AWS’s recommended best practices for massive data loading, making it MUCH more than just a “connector”.

____0_TDT_Snowflake_Diagram

The greatest “value added” our customers see among all TDT’s functions and features is our Snowflake connectivity. TDT’s innovative Lambda-based microservices approach loads data into Snowflake tables, which are architected to retain the entire history of source data ever since the source-to-target synchronization began (perfect for time-based trend/predictive/prescriptive analytics).  

Simply put, TDT is the self-contained, turn-key solution that gets your valuable data into Snowflake today, eliminating months, or even years of research and development time/costs. TDT’s high-speed, massive data movement to Snowflake only takes minutes to ramp up.

Further Reading… 

AWS TDT Product Brief

TDT: Much more than a mere “data connector” for Snowflake

Just What is the New Treehouse Dataflow Toolkit?

Treehouse Software and Confluent offer High-Speed Mainframe Dataflow for Cloud-based Advanced Analytics

Treehouse Dataflow Toolkit (TDT) is Copyright © 2024 Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges 

Contact Treehouse Software for a Demo Today!

Contact Treehouse Software today for more information or to schedule a product demonstration.

Getting Started with Rocket Data Replicate and Sync [RDRS] and Treehouse Dataflow Toolkit [TDT]

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc. and Dan Vimont, Director of Innovation at Treehouse Software, Inc.

Introduction
Treehouse Software offers customers and partners a hands-on opportunity to configure and execute data transfers using Rocket Data Replicate and Sync [RDRS] and the Treehouse Dataflow Toolkit (TDT) in the form of “Getting Started” CloudFormation templates in an AWS Virtual Private Cloud (VPC).

About RDRS
RDRS offers a multi-platform solution for real-time, continuous, and bidirectional data synchronization and replication. Transfer mainframe data to your AWS targets continuously and in real-time for Data Analytics, Business Intelligence, ERP, CRM, or for application modernization, mainframe offload initiatives, or mainframe migration. RDRS considerably simplifies and accelerates data exchange configurations through its intuitive Dashboard interface, which presents source, target, and mapping metadata in a straightforward, user-friendly format – even for users with no mainframe knowledge. Supported sources and targets include a wide range of relational and non-relational databases: DB2, IMS/DB, VSAM, sequential files, Adabas, Datacom DB, IDMS/DB, Oracle, SQL Server, PostgreSQL, and others. Additionally, RDRS can publish transactions in JSON and Avro format to pipelines such as Kafka and Kinesis.

About the Getting Started CloudFormation stack
Your Getting Started session covers the deployment and operation of a fully preinstalled RDRS implementation, generated as a CloudFormation stack via a template provided by Treehouse Software. As shown in the diagram below, the chief components of RDRS come preinstalled on two EC2 instances launched within a new or existing securely-provisioned VPC, with the main engine (the RDRS Agent) running on a Linux instance and the user interface (the RDRS Dashboard) running on a Windows instance.

The intent of the Getting Started stack is to allow new users to quickly skip past the complexities of designing and building the VPC shown above, and to bypass some of the initial installation/configuration steps of RDRS. The Getting Started stack allows the user to work with the product’s data transfer functionality using the sample pre-loaded databases within just minutes.

In addition to the RDRS product components, the stack includes three database instances for the purpose of the user’s initial experimentation with data transfers. They consist of a pre-loaded SOURCE database (the SQL Server AdventureWorks sample database), a TARGET database (PostgreSQL), and a REPOSITORY database for RDRS’s repository tables (PostgreSQL). The sample data in the SOURCE database tables may also be transferred to an MSK cluster, which is also created in the Getting Started stack.

TDT components provided in the Getting Started stack
The Getting Started stack includes components of TDT, which provide event-driven Lambda-based functions to automatically consume messages from MSK/Kafka (messages produced by RDRS, as shown in the diagram above), to land data in JSON format in S3 buckets, and to perform advanced crawling of the data to derive target table and view structures. Separate documentation is provided for configuration and operation of TDT for targeting Athena/S3.

Adding mainframe-based data sources
If you wish to add mainframe sources to the mix, you are encouraged to contact Treehouse Software to obtain RDRS’s mainframe components. Note that the usage of SQL Server as a source in this Getting Started stack very strongly resembles the usage of a mainframe source; the RDRS Dashboard is designed to provide a consistent interface, regardless of whether the data source is mainframe or non-mainframe based, and regardless of whether the source datastore is relational or nonrelational in nature.

____Treehouse_AWS_Badges 

Schedule your session today…

Contact us to schedule a Getting Started with RDRS and TDT session.

Copyright 2024 by Treehouse Software, Inc.

Quick Read: AWS Partner Solution Brief – Treehouse Dataflow Toolkit

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.

____0_TDT_Generic01

Treehouse Software and AWS are collaborating on several AWS-centric initiatives in the coming months. The focus of these efforts is to market our new Treehouse Dataflow Toolkit (TDT), a set of microservices that provides the turn-key solution for transferring data from Kafka into advanced Analytics/AI/ML-friendly targets, such as Amazon Redshift, Snowflake, Amazon Athena/S3, Amazon S3 Express One Zone Buckets, as well as Amazon Aurora PostgreSQL. We have worked with an AWS Marketing Manager to create the following TDT AWS Partner Solution Brief downloadable PDF that provides a one-minute overview of TDT, its benefits, and resource links for your team…

DOWNLOAD…AWS_TDT_Product_Brief_Thumb01

Treehouse Dataflow Toolkit (TDT) is Copyright © 2024 Treehouse Software, Inc. All rights reserved.

____Treehouse_AWS_Badges 

Contact Treehouse Software for a Demo Today!

Contact Treehouse Software today for more information or to schedule a product demonstration.

A Treehouse Software Proof of Concept is the low-risk approach to testing mainframe data replication on Cloud and Hybrid Cloud environments

Posted on by

by Joseph Brady, Director of Business Development / Cloud Alliance Leader at Treehouse Software, Inc.

____0_Mainframe_To_Cloud

Many Treehouse Software customers have discovered the value of saving weeks, or months in their mainframe modernization initiatives by engaging in a Rocket Data Replicate and Sync (RDRS) Proof of Concept (POC) for Mainframe-to-Cloud data replication. Depending on the complexity of the customer’s project, an RDRS POC generally lasts as little as 10 business days after the product is installed and all connectivity is set up between the mainframe and Cloud environments.

How does it work?

  1. Treehouse Software provides documentation beforehand that outlines all of the requirements and agenda for the POC, and Treehouse technicians assist in downloading and installing RDRS.
  2. The customer provides a representative subset of z/OS or z/VSE mainframe data (e.g., Db2, Adabas, VSAM, IMS/DB, CA IDMS, CA DATACOM, etc.), use case, and goals for the POC, and the Treehouse team mentors the customer’s technical team via remote screen sharing sessions.
  3. The application is executed on customer facilities, in a non-production environment, and a limited-scope implementation of RDRS is conducted to prove that the product meets the customer’s desired use case.

By the end of the POC, customers will have replicated mainframe data on their Cloud target, tested out product capabilities, and demonstrated a successful, repeatable data replication process, with documented results. After the POC, the customer has all the connectivity and processes in place to begin setting up the production phase of their mainframe data modernization project. The minimal cost and resources makes an RDRS POC a valuable ROI in the customer’s mainframe modernization journey.

About RDRS…

Many Cloud and Systems Integration partners are recommending RDRS for mainframe data modernization projects. RDRS focuses on changed data capture (CDC) when transferring information between mainframe data sources and Cloud targets. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of RDBMS and other targets.

RDRS utilizes a Windows-based GUI Control Board, which is ideal for non-mainframe programmers. While mainframe experts are required in the design/architecture phase during the POC and occasionally during implementation, the requirement for their involvement is limited. The RDRS Control Board acts as a single point of administration, data modeling and mapping, script generation, and monitoring. Comprehensive monitoring and logging of all data movements ensure transparency across all data exchange processes.

Additionally, once RDRS is up and running, the customer’s legacy mainframe environment can continue as long as needed, while they replicate data – in real time and bi-directionally – on the new Cloud platform. Now the enterprise can quickly take advantage of the latest Cloud services, such as advanced analytics, ML/AI, etc., as well as move data to a variety of highly available and secure databases and data stores.

__TSI_LOGO

Contact Treehouse Software Today…

Contact us to discuss how a Treehouse Software POC can accelerate your mainframe Cloud and hybrid Cloud data modernization journey.

3-Minute Video: Data Management and Processing with Rocket Data Replicate and Sync (formerly tcVISION)

Posted on by

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Treehouse Software is a worldwide distributor of Rocket Data Replicate and Sync (formerly tcVISION), the leading tool for using change data capture (CDC) for synchronizing mainframe data with real-time and bi-directional data replication. This video focuses on the product’s data management and use of “staged processing” to minimize its footprint on the mainframe system…

__TSI_LOGO

Contact us today for a live, online demo…

Simply fill out our Demonstration Request Form and a Treehouse representative will contact you to set up a time for your requested demonstration.