Serverless Data Pipeline
Data Engineering
- Enables data to decision journey: Raw data to dashboard
- ELT: Raw Data, Ingestions, Transformations, Aggregations
Google Cloud services

Cloud Storage

Cloud Functions

Big Query

Cloud Build

Deployment Manager

Cloud Monitoring

Cloud Logging

Cloud Repositories
Features

- Serverless – No Infra provisioning
- Auto Scalable – Scales up and down as per data size
- Deployment - Managed and automated
- Security – Access-controlled & Encryption-enabled
- Performance – Can process GBs of data in seconds
- File types supported: CSV, JS
- Low to medium complexity of data movements
- Medium data volumes
- Incoming data per file up to 5 GB
- Transformation Query run time is not more than 5 minutes
- Less number of interdependent source files
- <= 200 source files on daily basis

Adoption
Operationalization

Deployment -
Automated through Deployment Manager.

Ingestion -
Automated for CSV, JSON. Schema is auto detected and auto updated as per files.

Transformation -
Uses BigQuery SQL. They are then orchestrated through Composer DAGs
Customization
Code can be extended to support other file formats.
Any specific generalization can be switched off or enhanced to meet specific requirements.
Can be extended to accommodate CMEK (customer managed encryption keys) related requirements.
Benefits
- Faster onboarding on Google Cloud means faster time to market
- Decreases ramp up time by 4–8 weeks
- Standardization of solutions leads to ease of maintenance
- Configuration driven allows businesses to deploy changes faster
- Out-of-box solutions for common tasks means reduced efforts
- Better risk management leads to more predictable outcomes
Use cases
