Smart Order Solution
Fractal delivered a smart-order solution that runs on R studio at the store level for three regions in Pakistan. The solution was scaled to each of the regions of Pakistan, India and Indonesia, with all outputs generated simultaneously.
The smart-order solution was structured to be moved to Databricks, with all processes parallelized through the pipeline structure in Databricks.
- Using Databricks, files were stored in Blob and PySpark was used to get data from the SQL Server database.
- DBFS (Databricks File System) used to write the file back into the Blob. DBFS also worked on the distributed computing.
- Incorporated a code snippet (Master Script) which internally called all other child snippets by passing relevant parameters for each script.
Databricks notebook migration from Development to Production environment. The production setup was delivered with Github integration for code version maintenance and Azure Data Factory to enable pipeline creation for execution of R snippets across different countries in parallel.