by Joseph Brady, Director of Business Development at Treehouse Software
Treehouse Software’s Treehouse Dataflow Toolkit (TDT) is currently in production at a large auto manufacturer as their key component for replicating dealership and vehicle order management data from multiple disparate mainframe databases to Snowflake on AWS.
The TDT solution, along with Treehouse’s decades of mainframe expertise, and our Cloud Engineers’ deep skills with multiple top-level AWS certifications accelerated the customer’s critical data move to Snowflake. Thanks to the Treehouse data delivery architecture, the customer’s data scientists and analysts can now access analytics-ready data through Snowflake. This enables sub-second query performance for rapid, intensive analytical tasks, and data sharing in real time, eliminating the need to move or copy data, thus enabling immediate insights across divisions and subsidiaries. The analytics teams can also make plans to easily add the latest AWS-based Analytics/AI/ML-friendly offerings, such as Amazon Redshift, Amazon Athena/S3, Amazon SageMaker AI, Amazon Bedrock, as well as any yet-to-be-developed Cloud services!
Since TDT is much more than a mere “connector,” the customer was able to eliminate months of development time and costs by using the tool to quickly and automatically prepared the full infrastructure needed for Snowflake data loading. As shown in the following architectural diagram, TDT is taking the customer’s mainframe data that was pumped into Amazon MSK (Managed Streaming for Kafka) by Rocket Data Replicate and Sync (RDRS) and lands it into Snowflake. TDT not only delivers the customer’s data, but its advanced crawler functions automatically prepare landing tables, views, and staging infrastructure for Snowflake. Additionally, when needed, TDT now stands ready to generate an archiving infrastructure and create Apache Iceberg tables for enhanced data management.
The Treehouse solution is enabling the customer to quickly move away from slow, on-premises, batch-oriented processes to their new scalable, highly available, and secure Cloud-native system. They are also better positioned to innovate for future data needs and strategies on a flexible and easily customizable architecture.
Customer Benefits
- The customer’s data scientists and analysts can now access analytics-ready data faster and ever before through Snowflake
- The new architecture easily allows testing and adding other Cloud-based Analytics/AI/ML-friendly targets and services.
- From a data scientist’s perspective, effective dataflow tooling with TDT has delivered tangible benefits:
- Fewer reconciliation issues with other divisions
- Greater confidence in analytical results
- Faster turnaround on urgent questions from leadership and multiple divisions
- TDT’s auto scaling and parallelizing Lambda framework allows many parallel selects to all run at once, thus loading large tables with minimal latency.
- Since TDT is built in alignment with AWS’s and Snowflake’s best practices, proper security and performance is ensured.
- TDT is delivered via CloudFormation Templates, which automated and accelerated the process of installing and configuring the complete TDT application (including AWS Lambda functions and numerous other AWS resources, all wrapped in a well-architected security framework) in their AWS account. This allowed the customer to be up and running with a fully preconfigured implementation of the new data transfer pipeline in minutes.
- The customer now has lasting compatibility with emerging Cloud technologies. As AWS and Snowflake introduce new features, TDT readily integrates them, staying ahead of the curve, keeping data pipelines modern and efficient.
In short: better data → better analytical judgment.
Visit Treehouse Software on the AWS Marketplace for all of our Cloud offerings…
Treehouse Dataflow Toolkit (TDT) and TDT-DIRECT are Copyright © Treehouse Software, Inc. All rights reserved.
Contact us today to discuss your project!
Treehouse Software's Blog 