Reducing data processing time from 10+ hours to minutes for a pricing consultancy

Impact

Minutes

data processing time reduced from 10+ hours to seconds or minutes with the new ELT pipeline

Faster Iteration

clients can now experiment with pricing configurations in real time via a Streamlit interface

Scalable Foundation

architecture supports seamless onboarding of additional customers with strict data isolation per client

Challenge

Unlocking faster, scalable pricing insights

A strategic pricing consultancy faced challenges in processing large-scale customer pricing data. Their existing solution relied on Python scripts and Excel, which struggled to handle the increased data volume from a new customer. Processing times ballooned to over 10 hours, making iterative analysis impractical and hindering their ability to provide timely pricing insights.

The consultancy needed a flexible, scalable, cloud-based solution that could process data rapidly, support iterative experimentation, and lay the groundwork for a pricing platform that could accommodate additional customers while maintaining data security and isolation.

Solution

A modern ELT pipeline to optimize data processing

OneSix designed and implemented a modern ELT (Extract, Load, Transform) data processing pipeline leveraging Azure Data Factory and Snowflake. The architecture streamlined data ingestion, transformation, and analytics, delivering rapid performance and scalability. Key elements of the solution included:

Automated Data Ingestion: Azure Data Factory ingested raw customer data on a scheduled basis into the client’s Snowflake environment.
Medallion Architecture: The pipeline followed a medallion architecture (bronze, silver, gold layers) to clean, enhance, and transform raw data into actionable pricing insights and recommendations.
Data Processing: Processing was conducted with a combination of Snowflake SQL and Python for flexible and powerful transformations.
Interactive Analytics: A Snowflake Streamlit application allows customers to view insights, experiment with pricing configurations, and analyze results in real time.
Seamless Data Export: Azure Data Factory exported processed data products into the customer’s SQL database for further analysis and reporting.
CI/CD Integration: The entire Snowflake environment was managed using CI/CD pipelines via GitHub workflows and Snowflake’s Git integration tools for declarative schema management.

Technologies Implemented

Snowflake
Python
Azure Data Factory
Streamlit

Results

Achieving fast performance, enhanced agility, and seamless scalability

The solution drastically improved data processing efficiency, reducing execution time from over 10 hours to just minutes or seconds. This rapid performance enabled:

Faster Iterative Analysis: Clients could quickly experiment with changes to the pricing engine and configurations, facilitating a more iterative and agile approach to pricing strategy.
Enhanced Confidence: The ability to rapidly analyze data and refine pricing models helped build confidence in the pricing recommendations and insights provided.
Scalability and Flexibility: The architecture laid the foundation for onboarding additional customers seamlessly, with reusable components and strict data isolation for security.

The consultancy praised the performance and adaptability of the new solution, highlighting how the architecture enabled rapid, iterative development and delivered consistent, high-quality pricing insights.

Optimizing data pipeline efficiency to reduce cloud costs for an amusement park

By analyzing and adjusting data processing schedules, OneSix enabled a seaside amusement park to significantly reduce unnecessary cloud expenses while maintaining scalability during peak demand. This optimization improved cost efficiency and ensured the pipeline aligned with real-time business needs.