Transforming Public Transport Data Management with Fractal’s DevOps Framework
Summary
A major public transport provider sought to eliminate their reliance on a third-party platform, NetBI, for obtaining transport and GIS (Geographic Information System) data from vendors and running analytics.
Fractal proposed a solution that provided tools for data quality, cleaning, and cataloging services. This new platform managed data quality, streamlined data pipelines, and provided robust data catalogs for comprehensive analytics.
Leveraging AWS services, the solution improved data governance, enhanced operational efficiency, and reduced the client’s dependency on external systems. This offered greater control and flexibility in their data management processes.
Business challenge
After the initial proof of concept (PoC), it became clear that while the solution effectively addressed the problem from a data engineering perspective, it lacked the agility and efficiency a robust DevOps framework provides. The main challenges included:
- Lack of Automated Deployments: The initial setup did not support automated build and deployment processes, leading to slower release cycles.
- Insufficient Monitoring and Logging: The existing setup did not have adequate monitoring and logging mechanisms to quickly identify, analyze, and resolve issues. This resulted in longer downtimes and delayed responses to system failures, negatively impacting operational efficiency and reliability.
- Security Vulnerabilities: The existing process did not include vulnerability detection during development. Vulnerabilities were typically identified post-deployment through VAPT (Vulnerability Assessment and Penetration Testing) by external tools and vendors, delaying feedback and compromising the overall security posture.
- Scalability Issues: The platform needed to support seamless scaling to handle varying workloads efficiently.
- Infrastructure Management: The process lacked infrastructure as code (IaC) practices, leading to potential errors in resource management.
Solution / Approach
To address these challenges, Fractal implemented a comprehensive DevOps framework alongside their data management solutions. This approach ensured that the platform not only managed data quality, streamlined data pipelines, and provided robust data catalogs but also incorporated DevOps practices to enhance agility, security, and efficiency.
The solution was deployed on AWS ECS with Fargate, using Docker container images stored in AWS ECR. The CI/CD pipeline was extended with GitHub Actions for building and deploying processes, securely connecting to the AWS account via an OIDC-based trusted role.
Key DevOps components of the solution
Infrastructure Deployment
- AWS ECS with Fargate: Enabled containerized deployment without the need to manage underlying servers. ECS (Elastic Container Service) provides the orchestration for containerized applications, while Fargate abstracts the underlying infrastructure, allowing for a serverless compute engine to run containers. This ensures that the client does not need to manage the EC2 instances, enhancing scalability and reducing operational overhead.
- Docker Container Images: Stored in AWS ECR for seamless integration and deployment across different environments.
- AWS Cloud Formation: Used to define and provision the infrastructure. Templates were version-controlled, and changes were reviewed through pull requests (PRs). Build and deployment scripts displayed infrastructure changes in PR comments, preventing unknown changes and accidental resource removal. This also facilitated rollback capabilities in case of issues.
CI/CD Pipeline
- GitHub Actions: Automated the build and deployment processes, ensuring consistent and reliable delivery of updates. Workflows were defined to automate the entire lifecycle from code commit to deployment.
- OIDC-based Trusted Role: Enabled secure connectivity between GitHub Actions and AWS, ensuring secure and seamless deployment processes.
Code Quality and Security
- Code Quality Testing: Ensured high standards in code development, catching bugs and code smells early in the development process.
- Vulnerability Scanning: We implemented Trivy to scan the build pipeline for vulnerabilities in both the code and Docker container images, ensuring secure deployments during development rather than post-deployment.
- Secret Detection: Tools like GitHub Advanced Security were used to scan the codebase for leaked secrets, ensuring no sensitive information was exposed.
Logging and Monitoring
- CloudWatch: Centralized log storage enabled easy access and analysis. CloudWatch Logs provided detailed log data and CloudWatch Alarms were configured for real-time alerting.
- Enhanced Monitoring with Grafana: It was integrated with CloudWatch for real-time monitoring and alerting. Custom dashboards were created to visualize system performance and health. This allowed proactive management and rapid issue resolution, reducing downtime and improving reliability. Alerts and notifications were set up to notify the operations team of anomalies, ensuring quick response times.
ETL Orchestration
AWS Managed Apache Airflow – Orchestrated ETL workflows, managing dependencies and scheduling jobs. Airflow DAGs (Directed Acyclic Graphs) were used to define workflows, ensuring streamlined data processing.
Seamless Deployment Across Multiple Environments
Leveraging IaC with AWS CloudFormation and GitHub Actions enabled the seamless deployment of applications across multiple environments, ensuring consistency and reducing the risk of configuration drift.
Results
- Increased Deployment Velocity: The automated CI/CD pipeline with GitHub Actions enabled faster and more frequent deployments, reducing the time-to-market for new features and updates.
- Enhanced Reliability and Stability: Continuous monitoring with CloudWatch and Grafana ensured real-time visibility into system performance, leading to quicker issue detection and resolution.
- Improved Security Posture: The implementation of Trivy for vulnerability scanning during development and secret detection enhanced the overall security of the platform, reducing the risk of security breaches.
- Scalability and Flexibility: Utilizing AWS ECS with Fargate and IaC with CloudFormation allowed the platform to scale seamlessly according to workload demands, ensuring high performance and availability.
- Error Prevention: Reviewing infrastructure changes in PRs with CloudFormation minimized the risk of accidental resource deletions or misconfigurations, ensuring a stable environment.
- Cost Efficiency: Automated processes and enhanced resource management have led to significant cost savings by optimizing resource usage and reducing manual intervention.
Conclusion
The robust DevOps framework and processes implemented with the Fractal DMS platform fixed the issues, enhanced operational efficiency, reduced costs, and provided the client with greater control over their data management processes.