Rules in SQL, steps in Python
Why I keep business rules in SQL, next to the data, and use Python only to run the steps around them.
Senior Data Engineer. Migrated. Reconciled. On time.
I move banking and insurance data to the cloud and prove it arrived intact.
Greater Toronto Area · SQL · Oracle PL/SQL · Python · Azure · Google Cloud · English and Spanish
$ python migrate.py --source oracle --target gcp[info] profiling source tables…[info] null keys: 0 · schema drift: 2 columns[warn] drifted columns mapped in the standardization layer[info] landing → standardization → serving[info] loading partitions 1/12 … 12/12[info] reconciling row counts and amounts…[ OK ] rows: source = target[ OK ] totals match to the cent[ OK ] nightly window met✓ migration complete$
About
Data that has to add up, for banks and insurers.
I’m Manuel Pérez, a Senior Data Engineer in the Greater Toronto Area. Since 2009 I have built data warehouses and ETL for a bank, moved its pipelines to Azure and migrated a health insurer to a new platform in two countries. Today I work on Scotiabank’s International Banking data migration to Google Cloud.
17Years working with data
400ETL processes moved to Azure Data Factory
−50%Less load time after that move
55KInsured members migrated to a new platform
Blog
See all articles
Notes on data migration
How I check that a data migration arrived intact
A pipeline that finishes without errors has only proved that it ran. Here is how I prove the data is the same on both sides.
Let’s talk
Hiring a data engineer?
Tell me about the role or the project. I read every message and reply personally.