Replaced ad-hoc SQL scripts with a version-controlled, tested, and documented transformation layer. Business analysts can now trust the data because every metric has a lineage trail and automated test coverage.
Analysts were maintaining dozens of overlapping SQL views with no version control, no testing, and no documentation. The same metric was calculated differently in three different dashboards. Data trust was at an all-time low.
The transformation layer had no CI/CD, no automated testing, and no lineage tracking. Deploying a model change required manual execution and prayer. There was no separation between dev, staging, and production environments.
dbt sources defined with freshness checks and schema documentation. Raw tables in Snowflake LANDING schema.
dbt profiles are parameterized by environment. Azure DevOps pipelines run dbt build on feature branches (dev), merge to main triggers staging, and a manual gate promotes to production. Each environment targets a separate Snowflake database.
ivy_dataTransformation/deployment.ymldbt models are fully idempotent by design — incremental models use unique_key for safe re-runs.
Azure DevOps pipeline retries on transient Snowflake connection errors. dbt --retry-all-errors flag enabled.
dbt run results and test results captured as JSON artifacts in Azure DevOps for audit trail.
Separate Snowflake roles for dbt service account per environment. Analysts have read-only access to mart layer.
Azure DevOps: feature branch → dev build → PR → staging build → manual gate → production deploy.
Strict branch strategy: feature/* → main → release tags. Each merge triggers the appropriate environment pipeline.
ivy_dataTransformation/deployment.ymlAzure DevOps pipeline for dbt build, test, and docs generation across all environments.
ivy_streamlit_apps/templates/deploy-streamlit-template.ymlReusable Azure DevOps template for deploying Streamlit apps to Snowflake.
ivy_streamlit_apps/azure-pipelines-streamlit.ymlTop-level pipeline that orchestrates multi-app Streamlit deployments.
"This is the dbt transformation platform I built on top of Snowflake. Let me give you the context first. When I joined, analysts were maintaining dozens of overlapping SQL views with no version control and no testing. The same metric — say, 'active policies' — was calculated differently in three different dashboards. Data trust was at an all-time low. My solution was to introduce dbt Core as the transformation layer. Here's what that looks like in practice. The architecture follows a strict layered approach: sources → staging → intermediate → marts. Each layer has a clear contract. Staging models are one-to-one with source tables — no business logic, just cleaning and type casting. Business logic lives in intermediate models. Mart models are the consumption layer for BI tools and Streamlit apps. Every model has automated tests — not-null, unique key, referential integrity. These run in CI before any merge is allowed. If a test fails, the deployment stops. For deployment, I built a multi-environment Azure DevOps pipeline. Feature branches deploy to dev, merge to main triggers staging, and a manual approval gate promotes to production. Each environment is a separate Snowflake database. The Streamlit apps are deployed directly on Snowflake — no external hosting. I built a reusable Azure DevOps template that packages and deploys any Streamlit app in under 5 minutes. The result: 100% test coverage on mart models, deployment time under 8 minutes, and — most importantly — analysts now trust the data because every metric has a lineage trail they can follow."
"I built a dbt Core transformation platform on Snowflake with multi-environment CI/CD via Azure DevOps. The key outcomes were 100% automated test coverage on mart models, a reusable Streamlit deployment template, and eliminating conflicting metric definitions that had eroded data trust."
How this solution demonstrates enterprise data platform delivery values:
Researched and introduced dbt to an organization that had never used it, including building internal training materials.
Established a PR review process for dbt models that made it safe for junior analysts to contribute without fear of breaking production.
Owned the full stack: dbt models, CI/CD pipelines, Streamlit apps, and analyst enablement.
Migrated 40+ existing SQL views to dbt models while maintaining zero downtime for downstream dashboards.