TfGM opted for AWS due to its ability to handle the initial tasks and provide future solutions. They required fully scripted environments that would enable repeatable patterns, and Terraform was used to create all native AWS services and all database objects too.
To achieve maximum automation, flat file imports underwent verification of row counts, column counts, and data types using AWS Lambda functions; any failures would trigger help desk tickets with the supplier, without initiating ETL jobs on the TfGM side. Once verification had been completed, the ETL environment was activated using SQS, orchestrating the instantiation of Matillion and other tasks.
In addition to the native AWS tools, TfGM also leveraged the expertise of other AWS partners for the project. This included Matillion for ETL, Snowflake for the data warehouse, and Tableau Online for reporting and visualisation. All of these services were hosted on AWS, allowing for a seamless integration.