Data Engineer

Took leadership and single handedly managed an entire Data Analytics for a client, handling client communications, taking up enhancements, managing project infrastructure and providing constant support.
Designed and coded a custom SCD Type 1 and SCD Type 2 refresh logic with change tracking functionality for dataframes in Spark Scala. This improved the performance of a number of existing data pipelines considerably.
Deployed and tested several internal microservices and Apache Airflow instance on an EKS cluster using Kubernetes and Helm.
Developed several new components and services for company’s internal packages to provide new integrations and enhanced performance. All the developement with done in Scala and Python.
Wrote Spark Scala scripts to build efficient and robust ETL pipelines.