Serverless Data Pipeline
Data Engineering
- Enables data to decision journey: Raw data to dashboard
- ELT: Raw Data, Ingestions, Transformations, Aggregations
Google Cloud services
Cloud Storage
Cloud Functions
Big Query
Cloud Build
Deployment Manager
Cloud Monitoring
Cloud Logging
Cloud Repositories
Features
- Serverless – No Infra provisioning
- Auto Scalable – Scales up and down as per data size
- Deployment - Managed and automated
- Security – Access-controlled & Encryption-enabled
- Performance – Can process GBs of data in seconds
- File types supported: CSV, JS
- Low to medium complexity of data movements
- Medium data volumes
- Incoming data per file up to 5 GB
- Transformation Query run time is not more than 5 minutes
- Less number of interdependent source files
- <= 200 source files on daily basis
Adoption
Operationalization
Deployment -
Automated through Deployment Manager.
Ingestion -
Automated for CSV, JSON. Schema is auto detected and auto updated as per files.
Transformation -
Uses BigQuery SQL. They are then orchestrated through Composer DAGs
Customization
Code can be extended to support other file formats.
Any specific generalization can be switched off or enhanced to meet specific requirements.
Can be extended to accommodate CMEK (customer managed encryption keys) related requirements.
Benefits
- Faster onboarding on Google Cloud means faster time to market
- Decreases ramp up time by 4–8 weeks
- Standardization of solutions leads to ease of maintenance
- Configuration driven allows businesses to deploy changes faster
- Out-of-box solutions for common tasks means reduced efforts
- Better risk management leads to more predictable outcomes
Use cases
