TfGM
Stories from synvert customers
Our success with...
TfGM
"We have lots of data and information on the transport network and the challenge is capturing all this information, the dependencies and getting insight from it. When we don’t have to worry about technology, infrastructure, storage or performance, it means we can concentrate and focus on getting insight to improve the transport network and travel of people in Greater Manchester. "
TfGM - Analysis of travel patterns and behaviour with Snowflake and Tableau
Transport for Greater Manchester (TfGM) is the local government body responsible for implementing Greater Manchester’s transport policies, set by the Greater Manchester Mayor and the Greater Manchester Combined Authority. They invest money in improving transport services and facilities, thus supporting the regional economy, and they own Metrolink (the UK’s largest light rail network), Greater Manchester’s bus stations, bus stops and shelters, and transport interchanges. They subsidise fares to help the elderly, children and disabled people get around, as well as providing bus services when and where there are no commercial services. They promote and invest in walking and cycling as safe, healthy and sustainable ways to travel, and work closely with bus, tram and train operators to help improve the full journey experience.
With the launch of contactless on Greater Manchester's Metrolink network in summer 2019, TfGM needed to use network data to better understand and enhance the services and products provided to customers. They required a solution that would ingest data from multiple service providers, transform it into useful insights, and automate all tasks.
They also wanted the solution to be expandable as a service; another key requirement was to automate data quality verification before any ETL processes were run. And of course, the solution had to be cost-effective, scaling to several hundred users without any changes to the architecture to support the new load.
TfGM opted for AWS due to its ability to handle the initial tasks and provide future solutions. They required fully scripted environments that would enable repeatable patterns, and Terraform was used to create all native AWS services and all database objects too.
To achieve maximum automation, flat file imports underwent verification of row counts, column counts, and data types using AWS Lambda functions; any failures would trigger help desk tickets with the supplier, without initiating ETL jobs on the TfGM side. Once verification had been completed, the ETL environment was activated using SQS, orchestrating the instantiation of Matillion and other tasks.
In addition to the native AWS tools, TfGM also leveraged the expertise of other AWS partners for the project. This included Matillion for ETL, Snowflake for the data warehouse, and Tableau Online for reporting and visualisation. All of these services were hosted on AWS, allowing for a seamless integration.
synvert, formerly Crimson Macaw, provided TfGM with a highly efficient solution that enabled them to bootstrap a full production, test and development environment in just a matter of weeks. With the ability to ingest data feeds, transform the data, and create reports, TfGM was able to streamline their data operations and speed up delivery times while still maintaining the highest level of certainty over their environments and testing. The full scripting of these environments allowed for the quick creation of new end-to-end environments in a matter of hours.
Thanks to the implementation of Snowflake and Tableau, TfGM was able to gain new insights into travel patterns and behaviours that enabled them to adapt their network and make improvements to their contactless services. The ability to store and analyse any volume of data in the new Snowflake EDW played a significant role in these new insights, and the new visualisation capability provided by Tableau gave TfGM the ability to see and understand the data in new ways, allowing for better decision-making and improved customer experiences.
