Reducing data processing time from 10+ hours to minutes for a pricing consultancy

A modern ELT pipeline to optimize data processing

OneSix designed and implemented a modern ELT (Extract, Load, Transform) data processing pipeline leveraging Azure Data Factory and Snowflake. The architecture streamlined data ingestion, transformation, and analytics, delivering rapid performance and scalability. Key elements of the solution included:

• Automated Data Ingestion: Azure Data Factory ingested raw customer data on a scheduled basis into the client’s Snowflake environment.
• Medallion Architecture: The pipeline followed a medallion architecture (bronze, silver, gold layers) to clean, enhance, and transform raw data into actionable pricing insights and recommendations.
• Data Processing: Processing was conducted with a combination of Snowflake SQL and Python for flexible and powerful transformations.
• Interactive Analytics: A Snowflake Streamlit application allows customers to view insights, experiment with pricing configurations, and analyze results in real time.
• Seamless Data Export: Azure Data Factory exported processed data products into the customer’s SQL database for further analysis and reporting.
• CI/CD Integration: The entire Snowflake environment was managed using CI/CD pipelines via GitHub workflows and Snowflake’s Git integration tools for declarative schema management.

Technologies Implemented

• Snowflake
• Python
• Azure Data Factory
• Streamlit

Results

Achieving fast performance, enhanced agility, and seamless scalability

The solution drastically improved data processing efficiency, reducing execution time from over 10 hours to just minutes or seconds. This rapid performance enabled:

• Faster Iterative Analysis: Clients could quickly experiment with changes to the pricing engine and configurations, facilitating a more iterative and agile approach to pricing strategy.
• Enhanced Confidence: The ability to rapidly analyze data and refine pricing models helped build confidence in the pricing recommendations and insights provided.
• Scalability and Flexibility: The architecture laid the foundation for onboarding additional customers seamlessly, with reusable components and strict data isolation for security.

The consultancy praised the performance and adaptability of the new solution, highlighting how the architecture enabled rapid, iterative development and delivered consistent, high-quality pricing insights.