top of page

Aditya Goyal

Data Architect | Senior Data Engineer

Building scalable data platforms and analytics systems for global consulting clients.

EXPERIENCE

9+ Years

PLATFORM

Azure · Databricks · Spark

RESPONSIBILITIES

Platform Architecture
ETL Frameworks

Skills & Expertise

Data Engineering

  • Azure Databricks

  • PySpark & Spark SQL

  • ​ETL Design & Orchestration

  • Data Modeling​​

Analytics & Optimization

  • RGM & MMM

  • Forecasting Models

  • Optimization Techniques

  • Business Analytics

Cloud & Quality

  • Azure (ADF, ADLS, CI/CD)

  • Data Quality Frameworks

  • Automated Testing

  • Performance Tuning

Experience

Pratham Software

Senior Data Engineer | 2017 – Present

McKinsey & Company

Marketing & Revenue Analytics

What I do

​

  • Build and maintain scalable ETL pipelines on Azure Databricks for multi-client analytics

  • Ingest and process POS, P&L, and promo data from ADLS into curated datasets

  • Manage governance and publishing through Unity Catalog and Delta Lake layers

  • Orchestrate pipelines using Databricks Workflows/Jobs for scheduled runs

 

Key Contributions

​

  • Built a config-driven ETL framework to standardize ingestion and transformations

  • Implemented Bronze/Silver/Gold Delta Lake pipelines with reusable PySpark modules

  • Automated onboarding/orchestration via Databricks Jobs/Workflows APIs for scale

BCG

Analytics Platform Engineering

What I do

​

  • Built and scaled Marketing Catalyst, a marketing investment planning & optimization platform

  • Developed optimization workflows for annual allocation and weekly portfolio planning

  • Designed service patterns for sync vs async execution based on workload complexity

  • Persisted inputs and outputs in PostgreSQL (RDS) for downstream application usage

​​

Key Contributions

​

  • Implemented annual & weekly portfolio optimization with constraint-aware outputs

  • Designed async portfolio optimization using SQS to control throughput for compute-heavy runs

  • Improved optimization runtime by 8× (R redesign) + 4× (migration to Python)

Projects

End-to-end data platforms and pipelines built for real-world scale and reliability.

Enterprise ETL Platform for Retail Analytics

Designed and implemented a Databricks-based ETL platform integrating POS, P&L, and promotional data into analytics-ready datasets for enterprise clients.

Databricks · PySpark · Databricks ·  ETL · Data Engineering

Go to Project

Marketing Investment Optimization Platform

Annual market allocation + weekly portfolio optimization with constraint-aware planning, response curves, and scalable microservice execution.

Decision Support · Optimization  ·  AWS ·  Python · R

Go to Project

Engineering Stories

Real-world design decisions from building and operating data platforms at scale.

Migrating Databricks Pipeline from Hive to Unity Catalog

A production migration focused on governance, compatibility across clusters, and unexpected performance regressions.

Databricks  ·  Unity Catalog  ·  Platform Migration

Go to Story

Building a Layered Testing Strategy for a Production ETL Pipeline

A multi-layered approach to ensure development confidence, catch regressions early, and track downstream impact.

Nutter  ·  Unit Tests  ·  E2E  ·  Data bricks  ·  Data Pipeline

Go to Story

How I Work

Problem First

​I start by understanding the business problem and decision context before choosing tools or architecture.

Build for Scale

​I design platforms that are configurable, maintainable, and production-ready from day one.

Own Outcomes

​I take ownership beyond delivery, focusing on performance, reliability, and long-term usability.

bottom of page