Data Engineer with expertise in Snowflake and Kubernetes, skilled in developing automated deployment pipelines and managing DWH infrastructures. Proficient in ELT processes, data modeling, and API integrations, with a strong ability to enhance data efficiency and collaboration within engineering teams. Committed to optimizing data workflows and delivering impactful business insights.
1. Maintained and improved DWH infrastructure uptime to 99.9% through effective management of Airflow, Kafka Connect, and Airbyte on Kubernetes.
2. Developed and optimized dbt models and Jinja macros for efficient data transformation, supporting 20+ stakeholders in making informed business decisions. 3. Created and maintained ER diagrams to ensure alignment with Data Vault 2.0 standards, fostering better collaboration across teams.
4. Implemented GitOps-based CI/CD pipelines with Gitlab CI/CD, facilitating seamless automated deployments and improving workflow efficiency.
5. Built Python pipelines for seamless API data ingestion into Snowflake, improving data accessibility and reliability.
Data Pipeline Scalability: Built Python-based API ingestion framework for Snowflake, integrating 10+ public APIs (Google Adv, Meta, TikTok etc) with automated retries and deduplication.
Data Quality: Implemented dbt tests and Snowflake alerts for Data Vault 2.0, reducing data pipeline failures by 45%.
Infrastructure Optimization: Automated CI/CD pipelines using Helm and Gitlab CI/CD, reducing deployment time for a several services from 1 hour to 5 minutes
Cost Efficiency: Migrated legacy ETL workflows to Kubernetes-managed Airflow, cutting cloud storage costs by 30% through optimized task scheduling.
Title: Data Engineer