Tag Archives: education

Escape the complexity: Treehouse Software’s fully automated, Lambda-based solution accelerates highly scalable data delivery to AWS

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.

Since 1982, Treehouse Software has been serving enterprises worldwide with industry-leading mainframe software products and outstanding technical support. Today, Treehouse Software brings you Treehouse Dataflow Toolkit (TDT), a serverless (Lambda-based) application that goes beyond basic data transfer. It is a fully automated solution that prepares the full infrastructure needed to automatically prepare the staging infrastructure for the massive data loading to targets, such as Amazon Redshift, Snowflake, Amazon Athena/S3, Amazon S3 Express One Zone Buckets, and Amazon Aurora PostgreSQL. TDT supports data replication between mainframe and non-mainframe sources—without disrupting existing critical work on customers’ legacy systems.

The Treehouse solutions utilizes Rocket Data Replicate and Sync (RDRS) to pull data from the mainframe, where an agent (with a very small footprint) extracts data (either bulk-load or CDC processing). The raw data is then securely passed from the mainframe by RDRS, which transforms and publishes the data to a Kafka topic (in our example above, a topic in an Amazon MSK cluster). The TDT microservices consume the data from MSK/Kafka and land it in S3 buckets, where TDT’s proprietary crawler technology is used to automatically prepare landing tables, views, and additional infrastructure for various analytics friendly targets. Then the mainframe data is loaded into Redshift, Snowflake, S3, or PostgreSQL (all the while adhering to AWS’s and Snowflake’s recommended “best practices” for massive data loading, thus assuring shortest and surest loads). The inherent reliability and scalability of the entire pipeline infrastructure assures near-real-time synchronization between mainframe sources and the target tables, even with very large bulk-loads or transaction-heavy CDC processing.

What about non-mainframe data?

For customers who have non-mainframe data sources, Treehouse offers TDT-DIRECT which pulls data directly from PostgreSQL, SQL Server, Oracle, MySql, and Db2 for bulk-load and CDC into a variety of targets on AWS.

Instantaneous auto scaling…

For massive amounts of data, TDT takes advantage of the auto scaling and parallelizing of the Lambda framework. This allows many parallel selects to all run at once, thus loading large tables with minimal latency. Additionally, all TDT Lambda microservices are fully customizable (they will be YOUR Lambdas) to add extra monitoring capabilities, and any other functionalities for future needs.

TDT’s innovative Lambda-based microservices approach enables faster data flow than any conceivable ODBC-based solution, which is the standard tool used for most “roll your own” approaches, or “we have a connector for that” offerings. TDT offers several key differentiators from standard “connectors” on the market, including:

  • Automatic creation of target resources – TDT automatically prepares landing tables, views, and additional staging infrastructure for the target. Without TDT’s fully automated approach, a customer can spend months designing and creating target resources, such as delta tables, views, schemas, etc.
  • Ease of delivery/implementation – TDT is delivered via CloudFormation templates, which automate and accelerate the process of installing and configuring the complete TDT application (including AWS Lambda functions and numerous other AWS resources, all wrapped in a well-architected security framework) in your AWS account. This allows your site to be up and running with a fully preconfigured implementation of your new data transfer pipeline in minutes.
  • Adherence to best practices – TDTis built in alignment with AWS and Snowflake best practices, ensuring proper security and performance. The fault-tolerant design of the Cloud-native application provides for a robust, future-proof architecture.
  • Adaptability to evolving Cloud ecosystems – In today’s fast-evolving cloud world, TDT’s flexible design ensures lasting compatibility with emerging technologies. As AWS and Snowflake introduce new features, the application readily integrates them, staying ahead of the curve, keeping your data pipelines modern and efficient.

TDT and TDT-DIRECT are designed to deliver: 

  • rapid mainframe and non-mainframe data bulk-loading and CDC to Snowflake and AWS targets
  • access to the latest Analytics, AI, and ML tools and services
  • swift ROI

Contact us today to discuss your needs, or to book a free demo.

Visit Treehouse Software on the AWS Marketplace for all of our Cloud offerings…

Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright © Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

Contact us today to schedule a demo! 

Treehouse Software enables Adabas data replication on Snowflake and AWS without disrupting critical work on the mainframe

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.and Dan Vimont, Director of Innovation at Treehouse Software, Inc.

The Adabas mainframe database has been around since the early 1970s, and is still heavily used by government, banking, insurance, and other large enterprises worldwide. Most of these organizations have accumulated large volumes of mission critical and historical data stored in their legacy mainframe Adabas databases over the years. Their Adabas systems can support a broad range of services and programs, most of which require accurate, up-to-date, and secure data.

Having specialized in tools and services complementary to Adabas/Natural applications since 1982, Treehouse has encountered and successfully addressed countless unique situations in customers’ Adabas environments. In the mid-1990s, Treehouse Software responded to growing customer needs for Adabas-to-RDBMS data replication CDC technology with our renowned tRelational/DPS product set. Treehouse has always been a proponent of maintaining the coexistence of legacy mainframe Adabas systems and newer technologies, so our software is perfectly designed for those scenarios. As a matter of fact, many large enterprises are still using Treehouse data replication technology right alongside their mainframe Adabas databases after decades in production.

Entering the brave new world of Cloud Computing, Analytics, AI, and ML…

In recent years, Treehouse has been receiving a growing number of inquiries from mainframe Adabas customers wanting to quickly tap into today’s advanced Cloud-based Analytics/AI/ML technologies, such as Snowflake,  Amazon Redshift, and Amazon Athena/S3. The customers’ data science teams are eagerly awaiting the arrival of critical data from their mainframe databases to supercharge their predictive analytics and generative AI frameworks afforded by an ever-expanding array of AI/ML Cloud-based tools. Additionally, many of these enterprises require a hybrid architecture that allows legacy mainframe environments to continue concurrently with data engineering work in the Cloud. 

Treehouse Software answers the call… again!

Treehouse brings to market Treehouse Dataflow Toolkit Direct (TDT-DIRECT for Adabas), a self-service, fully automated offering that delivers bulk-loading and CDC of mainframe z/OS and VSE Adabas data to Snowflake and other AWS targets. And of course, this is all accomplished without disrupting the existing critical work on the legacy system. Unlike traditional tools that require extensive setup and orchestration, TDT-DIRECT for Adabas intelligently determines what needs to be configured and takes care of it automatically. From schema detection to the creation of all required target resources, TDT-DIRECT for Adabas ensures that everything is in place before the first row of data arrives.

TDT-DIRECT for Adabas is a data replication solution that leverages Treehouse Software’s renowned and rock-solid Adabas data replication technology for:

  • rapid Adabas data loading and CDC to AWS and Snowflake

  • AI-ready data delivery to the latest analytics, AI, and ML tools

  • swift ROI

  • speedy evaluation of various targets on AWS

Adabas connectivity to Snowflake and AWS

For customers who want a simple and inexpensive way to quickly move their Adabas data to Snowflake and AWS, Treehouse’s trusted and proven Adabas data replication capabilities provide complete data elements for automated transfer to AWS targets.

TDT-DIRECT for Adabas is a serverless (Lambda-based) application that goes beyond basic data transfer. It’s an automated, end-to-end solution that prepares the full infrastructure needed for data loading. Its advanced crawler functions automatically prepare all target resources requires for data transfer as seen in the following example:

Without TDT-DIRECT’s fully automated approach, customers could spend months designing and creating target resources like delta tables, views, and schemas. 

Fast and easy implementation…

Customers are happy to discover that Treehouse provides highly-detailed CloudFormation Templates which automate and accelerate the process of installing and configuring the complete TDT-DIRECT application (including AWS Lambda functions and a number of other AWS resources) in your AWS account(s). The TDT-DIRECT CloudFormation Templates create stacks consisting of all principal framework components, along with related IAM policies and roles which are carefully engineered to comply with “best practices” (such as a “least privileges” approach to permissions).

The TDT-DIRECT CloudFormation Templates also optionally provide for automatic creation of a VPC, its subnets, and all required standard VPC-oriented resources, as well as optional creation of a source database cluster (consisting of either a sample database provided by Treehouse for a quick trial/POC, or your own sample database data).

Simply put, TDT-DIRECT is a Cloud-native, turnkey solution that can eliminate months or years of research and development time and costs, and allow customers to be up and running in minutes.

Visit Treehouse Software on the AWS Marketplace for all of our Cloud offerings…

Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright © Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

Contact Treehouse Software for a TDT-DIRECT for Adabas Demo Today!

Contact us today to schedule your session! 

Ramp up your AI/ML game with AWS DMS and the TDT-DIRECT Plugin for Snowflake

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.

Many enterprise customers are using AWS Database Management Service (DMS) to simplify and accelerate migrations across different RDBMSs, such as PostgreSQL, SQL Server, Oracle, etc. and into AWS services like Amazon Redshift, and Amazon Athena/S3.

DMS delivers features for monitoring migration tasks, reviewing AWS CloudWatch metrics, inspecting logs, and validating data, making it a robust and cost effective solution.

Additionally, to ensure the tightest security, DMS implements a comprehensive security framework that safeguards data throughout the migration process using IAM policies, SSL/TLS encryption, and AWS Secrets Manager credential management. Network controls and monitoring tools provide access restriction and real-time visibility. 

High-level look at AWS DMS:

Treehouse Software introduces fully automated connectivity between DMS and Snowflake…

Today’s enthusiasm about AI and ML has become one of the prime motivators for customers wanting to move data to the Cloud, and Snowflake has become the platform of choice for many enterprises looking to mobilize data at near-unlimited scale and performance, while tapping into the most advanced AI/ML tools and services.

For DMS customers looking for the fastest and most straightforward way to connect to Snowflake, Treehouse Software brings you TDT-DIRECT, the ultimate DMS plugin for Snowflake. TDT-DIRECT leverages DMS to provide a turnkey approach that enables rapid bulk load and CDC data transfer directly from RDBMSs to Snowflake–AI-ready, with all target resources automatically created.

TDT-DIRECT is MUCH more than a mere “connector”—it is a serverless (Lambda-based), self-service, end-to-end solution that rapidly prepares the full infrastructure needed for loading data from DMS-supported RDBMSs into Snowflake. TDT-DIRECT’s advanced crawler functions automatically prepare all landing tables, views, and staging infrastructure for Snowflake as seen in this example: 

TDT-DIRECT is built in alignment with AWS’s and Snowflake’s best practices, ensuring proper security and performance. The fault-tolerant design of the AWS-native application provides for a robust, future-proof architecture. This adherence to best practices is a key differentiator of TDT-DIRECT from other “connector” offerings on the market.

Bonus points for fast and easy implementation…

Treehouse provides highly-detailed CloudFormation Templates which automate and accelerate the process of installing and configuring the complete TDT-DIRECT application (including AWS Lambda functions and a number of other AWS resources) in your AWS account(s). The TDT-DIRECT CloudFormation Templates create stacks consisting of all principal framework components, along with related IAM policies and roles which are carefully engineered to comply with “best practices” (such as a “least privileges” approach to permissions).

The TDT-DIRECT CloudFormation Templates also optionally provide for automatic creation of a VPC, its subnets, and all required standard VPC-oriented resources, as well as optional creation of a source database cluster (consisting of either a sample database provided by Treehouse for a quick trial/POC, or your own sample database data).

Simply put, TDT-DIRECT is a Cloud-native, turnkey solution that can eliminate months (or even years) of research and development time and costs, and allow customers to be up and running in minutes.

Visit Treehouse Software on the AWS Marketplace for all of our Cloud offerings…

Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright © Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

Contact Treehouse Software for a TDT-DIRECT for Adabas Demo Today!

Contact us today to schedule your session! 

See how Treehouse Software is helping an auto manufacturer replicate mainframe data to Snowflake on AWS without disrupting work on the legacy system

Posted on by

When Treehouse was approached by a large auto manufacturer to provide a solution to migrate their mainframe data from disparate source databases to Snowflake on AWS, the Treehouse Cloud engineering team was excited to take on the task. It wasn’t long before our experts drew upon their decades of mainframe expertise, along with deep skills and multiple AWS certifications, to come up with a prototype of the Treehouse Dataflow Toolkit (TDT). A quick proof of concept (POC) demonstrated that TDT worked exactly as expected and was the perfect tool for taking mainframe data that was pumped into Amazon MSK (Managed Streaming for Kafka) by Rocket Data Replicate and Sync (RDRS) and landing it into Snowflake on AWS.

TDT accelerated the customer’s move to Snowflake on AWS, because it is much more than a mere “connector” and goes beyond basic data transfer. It’s an automated, end-to-end solution that prepares the full infrastructure needed for Snowflake data loading. Its advanced crawler functions automatically prepare landing tables, views, and staging infrastructure for Snowflake. Additionally, TDT can generate optional archiving infrastructure and create Apache Iceberg tables for enhanced data management.

Treehouse Dataflow Toolkit (TDT) is Copyright © Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

For more information, contact Treehouse Software today!

Treehouse Software’s TDT-DIRECT for Adabas enables rapid data replication from mainframe z/OS and VSE Adabas to Snowflake and Analytics/AI/ML-friendly targets on AWS

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.and Dan Vimont, Director of Innovation at Treehouse Software, Inc.

 

Do you need a fast and easy way to replicate your mainframe z/OS and VSE Adabas data to Snowflake and Analytics/AI/ML-friendly targets on AWS? At Treehouse Software, we have always been proponents of maintaining the coexistence of legacy systems and newer technologies. This is why we support the idea of “Stay and go… at the same time!” and have developed time-tested solutions that allow bulk-loading and CDC of Adabas data into Snowflake and AWS, without disrupting the existing critical work on the legacy system. Our Treehouse Dataflow Toolkit Direct (TDT-DIRECT for Adabas) offers a straightforward and automated solution for rapid replication of Adabas data on Snowflake and AWS. Unlike traditional tools that require extensive setup and orchestration, TDT-DIRECT for Adabas intelligently determines what needs to be configured and takes care of it automatically. From schema detection to the creation of all required target resources, TDT-DIRECT for Adabas ensures that everything is in place before the first row of data arrives.

Treehouse Software has been in business since 1983, focusing on software that is complementary to  Software AG’s Adabas and Natural in the areas of data replication, security, control, auditing, performance enhancement, etc. Our Adabas data replication products have been used in many large enterprises worldwide for decades. With this deep experience, we are excited to offer TDT-DIRECT for Adabas, a data replication solution that leverages Treehouse Software’s renowned Adabas data replication technology for:

  • rapid Adabas data transfers (both bulk-loading and CDC) to AWS and Snowflake

  • access to the latest analytics, AI, and ML tools

  • swift ROI

Connectivity to Snowflake and AWS

For customers who want a simple and inexpensive way to quickly move their Adabas data to Snowflake and AWS, Treehouse’s trusted and proven Adabas data replication capabilities provide complete data elements for automated transfer to AWS targets.

The following example explains how TDT-DIRECT for Adabas accelerates a customer’s move to Snowflake on AWS. Snowflake’s unique architecture allows it to handle large-scale analytics workloads efficiently and reliably, while still offering a relational-style view of the data. The underlying framework enables businesses to run both bulk-load and CDC transactions with impressive scalability while maintaining the familiar relational format for users.

TDT-DIRECT for Adabas is a serverless (Lambda-based) application that goes beyond basic data transfer. It’s an automated, end-to-end solution that prepares the full infrastructure needed for Snowflake data loading. Its advanced crawler functions automatically prepare landing tables, views, and staging infrastructure for Snowflake. Additionally, TDT-DIRECT can generate optional archiving infrastructure and create Apache Iceberg tables for enhanced data management.

Without TDT-DIRECT’s fully automated approach, customers would spend months designing and creating target resources like delta tables, views, and schemas. Additionally, TDT-DIRECT for Adabas is delivered via CloudFormation templates, which can be deployed quickly and is preconfigured for immediate use — saving time, money, and configuration headaches.

Built for Scalability and Reliability

  • TDT-DIRECT for Adabas aligns with AWS’s and Snowflake’s best practices, ensuring security and performance. Its cloud-native, fault-tolerant design provides a future-proof solution that scales seamlessly with your growing data needs.

  • With TDT-DIRECT for Adabas, you can streamline data loading to Snowflake and AWS, enabling efficient, automated integration with minimal setup and maximum reliability.

  • TDT-DIRECT for Adabas is fully customizable to meet your organization’s specific needs (after all, TDT Lambdas are YOUR Lambdas).

Visit Treehouse Software on the AWS Marketplace for all of our Cloud offerings…

Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright © Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

Contact Treehouse Software for a TDT-DIRECT for Adabas Demo Today!

Contact us today to schedule your session! 

TREETIP: Auto scaling for massive data loading into Snowflake, Amazon Redshift, etc. with TDT-DIRECT

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.

Treehouse Dataflow Toolkit Direct (TDT-DIRECT) is a turn-key microservices-based offering that assures auto scalable, highly available, event driven bulk-load and Change Data Capture (CDC) transfers from legacy data sources to data analytics platforms like Snowflake, Amazon Redshift, etc.

This blog focuses on how TDT-DIRECT leverages the auto scaling capabilities of its Lambda microservices. These Lambdas are highly efficient compute services used to process TDT-DIRECT’s data transfer. There is no need to worry about throughput volume with TDT-DIRECT because the Lambdas scale automatically, with new instances spun up as needed  to handle increasing data transfer loads. 

Instantaneous auto scaling…

For massive amounts of data, TDT-DIRECT takes advantage of the auto scaling and parallelizing of the Lambda framework. This allows many parallel selects to all run at once, thus loading large tables with minimal latency.

And that’s not all! Here are TDT-DIRECT’s other key differentiators from standard “connectors” on the market:

  • Automatic creation of target resources – For example, TDT-DIRECT automatically prepares landing tables, views, and additional proprietary staging infrastructure for Snowflake. Without TDT-DIRECT’s fully automated approach, a customer can spend months designing and creating target resources, such as delta tables, views, schemas, etc.
  • Ease of delivery/implementation – TDT-DIRECT is delivered via CloudFormation templates, which automate and accelerate the process of installing and configuring the complete TDT-DIRECT application (including AWS Lambda functions and numerous other AWS resources, all wrapped in a well-architected security framework) in your AWS account. This allows your site to be up and running with a fully preconfigured implementation of your new data transfer pipeline in minutes.
  • Adherence to best practices – TDT-DIRECT is built in alignment with AWS and Snowflake best practices, ensuring proper security and performance. The fault-tolerant design of the Cloud-native application provides for a robust, future-proof architecture.
  • Adaptability to evolving Cloud ecosystems – In today’s fast-evolving cloud world, TDT-DIRECT’s flexible design ensures lasting compatibility with emerging technologies. As AWS and Snowflake introduce new features, the application readily integrates them, staying ahead of the curve, keeping your data pipelines modern and efficient.

Simply put, TDT-DIRECT is a Cloud-native, self-contained, turn-key solution that will eliminate months or years of development time and costs.

Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright © 2024 Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

Contact Treehouse Software for a TDT-DIRECT Demo Today!

Contact us today to schedule your session! 

Comprehensive Connectivity and Rapid Data Flow for Enterprise Customers with Treehouse Software and Confluent

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc., Dan Vimont, Director of Innovation at Treehouse Software, Inc., and Ram Dhakne, Staff Solutions Engineer at Confluent

Enterprise customers who are planning to modernize their data on Cloud environments are stating their needs clearly… “We want a way to unify and manage data from our applications, databases, data warehouses, etc., which have long operated in silos.”

These customers also have a crucial need to tap into today’s advanced data analytics platforms, such as Snowflake, Amazon Redshift, and Amazon Athena/S3, where an ever-expanding array of machine learning and artificial intelligence (ML/AI) tools are available to generate vital insights from their enterprise’s data.  Data science teams are eagerly awaiting the arrival of critical data from their enterprise’s data sources to supercharge their predictive analytics and generative AI frameworks.

Data Transfer + Unlimited Scaling and Storage

To address the need for rapid, high-volume data transfer from source DBs to Analytics/ML/AI-friendly platforms, Treehouse Software has recently gone to market with two powerful new offerings: Treehouse Dataflow Toolkit (TDT) for Mainframe Data Sources and TDT-DIRECT for Non-Mainframe Data Sources. These Cloud-native, fully automated, turn-key solutions work hand-in-hand with the premiere data streaming platform, Confluent to empower enterprise customers to rapidly migrate data – both bulk-load and change data capture (CDC) – to Snowflake, Amazon Redshift, Amazon Athena/S3, and Amazon S3 Express One Zone.

The TDT offerings are much more than mere “connectors”, providing an innovative and robust Lambda-based microservices infrastructure that automatically generates all target resources required for data transfer. Without TDT-DIRECT’s fully automated approach, a customer can spend months designing and creating target resources, such as delta tables, views, schemas, etc.

TDT-DIRECT extracts data directly from a source DB and loads it via Confluent into Snowflake’s “delta tables”, which inherently retain the entire history of source data ever since the source-to-target synchronization began (perfect for time-based trend/predictive/prescriptive analytics).

Figure 1: TDT-DIRECT automatically creates all Snowflake target structures (schemas, history tables, current views, user views, stages, and file formats), and Confluent delivers the data (e.g., insert, update, delete transactions) via bulk-load and CDC.

Leveraging AWS CloudFormation for ease of implementation…

For ease of implementation, TDT is delivered via CloudFormation templates, allowing customer sites to be up and running with a fully preconfigured implementation of a new data transfer pipeline in minutes. The TDT CloudFormation Templates create stacks consisting of all principal framework components, along with related IAM policies and roles which are carefully engineered to comply with “best practices” (such as a “least privileges” approach to permissions).

The TDT CloudFormation Templates also optionally provide for automatic creation of a VPC, its subnets, and all required standard VPC-oriented resources, as well as optional creation of a source database cluster (consisting of either a sample database provided by Treehouse for a quick trial/POC, or your own database and data).

The Confluent Advantage…

Treehouse Software’s TDT solutions fully support data transfers from mainframe and non-mainframe data sources to Confluent Cloud, which offers enhanced productivity, improved scalability, minimized downtime, and much more—all while reducing total cost of ownership. Confluent Cloud brings customers a Fully Managed Kafka Service and Complete Pre-Built Ecosystem that includes:

  • Elastic Scaling: Scale up and down quickly to meet fluctuating customer demand, without the ops burden that comes with scaling your data infrastructure.
  • Infinite Storage: Enable powerful use cases by never having to worry about Kafka retention limits again, while only paying for the storage used
  • Built-in Resiliency: Ensure high availability and offload Kafka ops with 99.99% uptime SLA, multi-AZ clusters, and no-touch Kafka patches
  • Serverless stream processing for Apache Flink®: Flink is the de facto industry standard for stream processing. Confluent Cloud for Apache Flink provides a cloud-native, serverless service for Flink that enables simple, scalable, and secure stream processing that integrates seamlessly with Apache Kafka®. Your Kafka topics appear automatically as queryable Flink tables, with schemas and metadata attached by Confluent Cloud.

A Powerful, Combined Solution…

Treehouse Software and Confluent provide a comprehensive framework that allows the target platform to constantly accrue the most current source data, which is ideally suited for data scientists looking to do trend analysis, predictive analytics, ML, and AI work. 

Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright ©Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

Contact Treehouse Software for a TDT Demo Today!

Treehouse Software offers SIs and consulting companies free “deep dive” learning sessions to educate your team on the value of bringing these turn-key data transfer solutions your customers.

Contact us today to schedule your session! 

Treehouse Dataflow Toolkit (TDT) Brings Added Value to Systems Integrators and Enterprise Consulting Companies

Posted on by

TDT_AI_ML

With decades of experience, Treehouse Software has helped systems integrators (SIs) and enterprise consulting companies streamline the migration of mainframe data to modern Cloud and Open Systems platforms—leveraging automation and innovation to accelerate time to value.

Treehouse Software is excited to introduce two powerful new offerings: Treehouse Dataflow Toolkit (TDT) for Mainframe Data Sources and TDT-DIRECT for Non-Mainframe Data Sources. These Cloud-native, fully automated, turn-key solutions empower enterprise customers to rapidly migrate data – both bulk-load and change data capture (CDC) – to advanced cloud and analytics targets such as Amazon Redshift, Snowflake, Amazon Athena/S3, Amazon S3 Express One Zone, and Amazon Aurora PostgreSQL.

TDT for Mainframe Data Sources…

01_Generic_MSK_TDT02

TDT-DIRECT for Non-Mainframe Data Sources…

TDT_DIRECT_03

  •  

With TDT and TDT-DIRECT, migrations take weeks – not months or years – supported by Treehouse Software’s 40+ years of leadership in data replication.

For SIs and consulting firms, TDT solutions act as critical accelerators – moving enterprise modernization initiatives swiftly into the value capture phase with Cloud and analytics platforms.

Substantial value of solutions that are more than merely “connectors”

  • TDT and TDT-DIRECT are ready to go: Customers can start pumping data into data analytics targets in days, rather than months, or years.
  • TDT and TDT-DIRECT are massively scalable through an efficient, event-driven AWS Lambda-based architecture.
  • TDT’s intelligent crawlers automatically generate JSON-based views and infrastructure – saving developers time and simplifying deployment to analytics environments where SQL-based handling is cumbersome.
  • TDT and TDT-DIRECT are delivered as robust CloudFormation Templates, automating the setup of the full TDT stack (including Lambda functions and other AWS components) within your AWS environment.
  • Treehouse Software provides dedicated technical expertise to ensure fast implementation and continuous support.
  • We say “NO!” to using only generic ODBC connections for data transmission, because:
    • To load large volumes of data, TDT and TDT-DIRECT use native bulk load utilities from target vendors – delivering superior scalability compared to ODBC, which relies on a narrow, transaction-based pipeline.
    • It is important to recognize that Snowflake and Redshift are analytical platforms – NOT OLTP systems—making ODBC-based CDC transfers both inefficient and misaligned with vendor best practices, often causing significant performance bottlenecks.
    • For Snowflake’s bulk-load functionality to operate effectively, proprietary objects beyond basic tables and views are required. TDT’s crawler automatically generates the necessary DDL to provision these components – saving time and preventing errors.

Challenges and impact of building a custom solution

A decision by an enterprise not to use TDT, but instead to build its own Kafka-to-Analytics/ML/AI-friendly targets solution, could result in any, or all, of the following:

  • accumulation of technical debt
  • extensive/unpredictable time to production (6 to 12 months of upfront development on average)
  • ongoing resource planning to maintain home-grown technologies (administrative and development)
  • vendor lock for maintenance of custom-made technologies designed and developed by consultants
  • managing a mix of manual and automated functions (requiring additional ongoing manpower)
  • difficulty in tracking cobbled together components created by multiple staff and consultants
  • limited agility for future customization and innovation (as technologies continue to rapidly evolve)
  • problems adhering to rapidly evolving best practices over time
  • high costs for future growth/scaling
  • potential lack of proper security/ongoing security updates
  • your organization, or your customer has now become an enterprise software development company, along with all of its associated costs!

Simply put, TDT and TDT-DIRECT are comprehensive, turn-key solutions that eliminate the need for months or even years of in-house development and associated costs.

Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright © 2024 Treehouse Software, Inc. All rights reserved.
____Treehouse_AWS_Badges

Contact Treehouse Software for a TDT Demo Today!

Treehouse Software offers SIs and consulting companies free “deep dive” learning sessions to educate your team on the value of bringing these turn-key data transfer solutions your customers.

Contact us today to schedule your session! 

Using AWS CloudFormation Templates for rapid solutions configuration and deployment

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc., and Dan Vimont, Director of Innovation at Treehouse Software, Inc.

TDT_CloudFormation_Template

This blog focuses on the value of delivering an AWS application offering via AWS CloudFormation Templates.  For those already familiar with CloudFormation, we invite you to skip to the next section of this blog entry (to the heading, “How TDT Leverages AWS CloudFormation”), where we describe how we use CloudFormation Templates for delivery and rapid deployment of the Treehouse Dataflow Toolkit (TDT), our turn-key solution for loading massive quantities of mainframe and non-mainframe data into Analytics/AI/ML-friendly targets on AWS.

Simply put, a CloudFormation Template is a formatted file (written in either JSON or YAML), that acts as a blueprint for automatically defining and deploying infrastructures in AWS by specifying the different resources needed, such as EC2 instances, databases, and security groups. A CloudFormation Template allows you to create and manage an entire application infrastructure as a single unit (called a “stack”) with a single command. Whenever you create a stack, you designate a template that CloudFormation uses to create the components described in that template.

CloudFormation Templates permit not only the building of complex sets of resources, but also the reuse of those templates in multiple contexts. To heighten flexibility and reusability, input parameters allow options to be specified when you create a CloudFormation stack. For example, you can specify an optional value like the type of an EC2 instance when you create a stack, making the template easier to reuse in different situations. For those familiar only with laboriously architecting and building applications one component at a time via the AWS console, the value of instead using CloudFormation Templates for more streamlined and assured deployment and management of your Cloud infrastructure cannot be overstated.

Advantages of using CloudFormation for managing infrastructure include…

  • Automating infrastructure provisioning
  • Defining infrastructure as code
  • Enabling reusability across environments
  • Facilitating easy deployment
  • Cost control through resource management
  • Scalability by quickly scaling up or down resources
  • Seamless integration with the AWS ecosystem
  • Reducing manual configuration errors

Video: Introduction to AWS CloudFormation…

How TDT leverages AWS CloudFormation

In the case of our TDT offering, Treehouse provides highly-detailed CloudFormation Templates which automate and accelerate the process of installing and configuring the complete TDT application (including AWS Lambda functions and a number of other AWS resources) in your AWS account(s). The TDT CloudFormation Templates create stacks consisting of all principal framework components, along with related IAM policies and roles which are carefully engineered to comply with “best practices” (such as a “least privileges” approach to permissions).

The TDT CloudFormation Templates also optionally provide for automatic creation of a VPC, its subnets, and all required standard VPC-oriented resources, as well as optional creation of a source database cluster (consisting of either a sample database provided by Treehouse for a quick trial/POC, or your own database and data).

____0_TDT_CloudFormation

Our customers and partners appreciate the delivery of a self-contained TDT solution via CloudFormation Templates. This eliminates weeks (potentially months) of engineering work and associated deployment time and costs, and allows a site to be up and running with TDT in minutes. Additionally, customers can optionally customize TDT’s CloudFormation Templates in order to adhere to enterprise architectural, security, and naming standards.

Download: TDT AWS Partner Solution Brief…

DOWNLOAD…AWS_TDT_Product_Brief_Thumb01

Treehouse Dataflow Toolkit (TDT) is Copyright © 2024 Treehouse Software, Inc. All rights reserved.

____Treehouse_AWS_Badges 

Contact Treehouse Software for a Demo Today!

Contact Treehouse Software today for more information or to schedule a product demonstration.

Enterprise Mainframe Customers Can Tap Into Today’s Most Advanced Data Analytics Platforms with Treehouse Software and Confluent

Posted on by

by Joseph Brady, Director of Business Development at Treehouse Software, Inc.; Dan Vimont, Director of Innovation at Treehouse Software, Inc.; and Ram Dhakne, Staff Solutions Engineer at Confluent

Treehouse_Confluent_001

Customers who are planning to modernize their enterprise mainframe systems on Cloud, Multi-Cloud, and Hybrid Cloud environments can be faced with decades of mission-critical and historical legacy mainframe data in disparate databases, as well as a variety of other data stores inherited through mergers, acquisitions, and other company growth scenarios.

Customers are stating their needs clearly…

“We want to modernize our mainframe data without disrupting the existing critical work on our legacy systems. We also want to bring together, view, and manage data from applications, databases, data warehouses, etc. that have been spread over many vastly different systems.”

Enterprises also want to tap into today’s advanced data analytics platforms, such as Amazon Redshift, Snowflake, and Amazon Athena/S3, where an ever-expanding array of machine learning and artificial intelligence (ML/AI) tools are available to generate vital insights from their enterprise’s data.  The customers’ data science teams are eagerly awaiting the arrival of critical data from their mainframes to supercharge their predictive analytics and generative AI frameworks.

The Solution = Mainframe CDC Data Replication + Unlimited Scaling and Storage

TRDRS_Logo

For those customers looking to move mainframe data to the Cloud, Rocket Data Replicate and Sync (RDRS) is the mainframe data replication tool that performs real-time synchronization of data sources, allowing for rapid data movement to newer data sinks/target platforms on AWS, Azure, Google Cloud, and other services. RDRS supports data replication from many mainframe data sources, including Db2 z/OS, Db2 z/VSE, VSAM, IMS/DB, IDMS, DATACOM, Adabas, or flat files. 

RDRS allows customers’ legacy mainframe environment to operate normally while replicating data on a variety of Cloud and Hybrid Cloud environments. The technology focuses on changed data capture (CDC) when transferring information between mainframe data sources and Cloud-based databases and applications. Through an innovative set of technologies, changes occurring in any mainframe datastore are tracked and captured, and published to various Cloud targets.

TDT_Logo

Additionally, Treehouse Software offers the Treehouse Dataflow Toolkit (TDT), a set of Lambda-based microservices that greatly enhances the architecture’s connectivity to high performance, non-relational, massive parallel processing data stores (Amazon Redshift, Snowflake, Amazon Athena/S3) that are primed to supply the most advanced ML/AI tools to data science teams.

To your data scientists, enterprise data history is GOLD…

TDT not only keeps things up to date faster than any conceivable ODBC-based solution, but the “delta tables” into which it loads data also inherently retain the entire history of source data ever since mainframe-to-target synchronization began.  So, for example, after TDT has been syncing a target table for 5 years, a data scientist now has 5 years’ worth of historical data to work with for trend analysis, predictive analytics, prescriptive analytics, ML, etc.

Confluent_logo_400x400

Confluent Cloud offers enhanced productivity, improved scalability, minimized downtime, and much more—all while reducing total cost of ownership. Confluent Cloud offers:

  • Elastic scaling: Scale up and down quickly to meet fluctuating customer demand, without the ops burden that comes with scaling your data infrastructure
  • Infinite Storage: Enable powerful use cases by never having to worry about Kafka retention limits again, while only paying for the storage used
  • Built-in Resiliency: Ensure high availability and offload Kafka ops with 99.99% uptime SLA, multi-AZ clusters, and no-touch Kafka patches

How does it all work?

Figure 1: An enterprise can now keep its options open by propagating data to the highly reliable, very scalable Confluent Cloud that can be “subscribed to” by any number of current or yet-to-be-invented ETL toolsets and target data stores.

____0_Confluent01

  1. We start at the source – the mainframe – where an agent (with a very small footprint) extracts data (in the context of either bulk-load or CDC processing).
  2. The raw data is securely passed from the mainframe to RDRS which speedily transforms mainframe-formatted data into Unicode/JSON and publishes the results to a Kafka topic in Confluent Cloud.
  3. TDT functions consume the data from Confluent and land it in S3 buckets, where Treehouse’s proprietary crawler technology is used to automatically prepare landing tables, views, and additional infrastructure for various analytics friendly targets. Then the mainframe data is loaded into Redshift, Snowflake, or S3 (all the while adhering to AWS’s and Snowflake’s recommended “best practices” for massive data loading, thus assuring shortest and surest loads).  The inherent reliability and scalability of the entire pipeline infrastructure assure near-real-time synchronization between mainframe sources and the target tables.

This Treehouse/Confluent framework allows data in staging tables to be constantly accruing the most current data, ideally suited for data scientists looking to do trend analysis, predictive analytics, ML, and AI work.  For business analysts and others who prefer structured data representations of potentially complex hierarchical data, this framework also automatically provides structured user-views, providing the look and feel of a SQL database.

Treehouse_AWS_Badge

Contact Treehouse Software today to schedule a product demonstration.