From Database Developer to Data Engineer
After more than two decades as a database and application developer at my university, I find myself at an exciting crossroads. Our institution is embarking on a significant digital transformation, gradually migrating from PeopleSoft to a modern Microsoft Azure-based data warehouse and data lake solution. For someone like me, with over 20 years of database experience, this isn't just an institutional change—it's a golden opportunity to evolve my career into data engineering.
Why This Transition Makes Sense
The shift from traditional database development to data engineering might seem daunting, but the reality is quite different. My two decades of experience aren't becoming obsolete; they're becoming the foundation for something bigger. Data engineering is essentially database work at scale, dealing with the same fundamental concepts I've worked with for years—data modeling, SQL optimization, ETL processes, data integrity, and application integration—just applied in new, cloud-native ways.
What makes this particularly exciting is the timing. Being present during an active migration to Azure means I can learn by doing, gaining hands-on experience with cutting-edge technology while applying my deep institutional knowledge. I understand our institution's data landscape, the quirks of our systems, and how different departments interact with data. This domain knowledge is incredibly valuable and nearly impossible for an outsider to replicate quickly.
Understanding the Technology Shift
The move from PeopleSoft to Microsoft's cloud ecosystem represents a fundamental architectural change. Instead of monolithic on-premises systems, we're moving toward a distributed, cloud-native architecture built on Azure's data platform services.
At the heart of this transition are two key components: the data lake and the data warehouse. The data lake (likely Azure Data Lake Storage) will serve as our repository for raw data extracted from PeopleSoft and other sources, stored in its native format. Meanwhile, the data warehouse (probably Azure Synapse Analytics) will contain structured, cleaned, and organized data ready for reporting and analytics. Together, these form the "single source of truth" that will gradually replace PeopleSoft's role.
The strategy is deliberate and gradual. Rather than a risky "big bang" migration, we'll keep PeopleSoft running while slowly moving functionality to modern cloud applications. Departments across campus will access their data through Power BI dashboards, built on top of this new infrastructure. It's a smart approach that minimizes disruption while modernizing our technology stack.
What's Different in Data Engineering
While my database background provides an excellent foundation, data engineering does involve some new concepts and skills. The scale is different—we're talking about massive datasets across distributed systems rather than single databases. The variety of data types expands beyond structured database tables to include semi-structured formats like JSON and XML, and even unstructured data like documents and logs.
The toolset is also evolving. Instead of on-premises servers, I'll be working with cloud-native services like Azure Data Factory for building data pipelines, Azure Databricks for large-scale data processing, and Azure Synapse Analytics for our data warehouse. While SQL remains crucial, Python becomes increasingly important, particularly with libraries like pandas and PySpark for data processing. Pipeline orchestration—building automated data workflows that run on schedules—becomes a core part of the job.
My Learning Roadmap
I've developed a practical learning path that balances formal education with hands-on experience.
In the immediate term (next 1-3 months), I plan to earn the Microsoft Azure Fundamentals (AZ-900) certification. With my background, this should be straightforward and will give me the big picture of Azure's ecosystem. I'll also start experimenting with Azure Data Factory through Microsoft's free learning platform and, if possible, get some hands-on time in Azure's portal.
For the short term (3-6 months), my focus shifts to the Azure Data Engineer Associate (DP-203) certification—this is the key credential for my target role. I'll deepen my Python skills, particularly focusing on pandas and PySpark, and study data lake architecture concepts like the medallion architecture (raw/bronze, curated/silver, and processed/gold zones). These patterns are becoming industry standards for organizing data lakes.
Over the medium term (6-12 months), I'll work on real projects in the PeopleSoft migration, learn Azure Databricks and Apache Spark for distributed data processing, and study modern data warehouse modeling techniques like star schemas and slowly changing dimensions. The goal is to become proficient enough to take on significant responsibilities in our migration project.
Leveraging My Experience
What excites me most is how my existing knowledge translates into immediate value. I know PeopleSoft's data structures intimately—where the critical data lives, which tables are clean versus messy, and where the proverbial "bodies are buried." This knowledge is gold during a migration. I can bridge the gap between our legacy systems and the new architecture, helping ensure that nothing important gets lost in translation.
I'm actively seeking ways to get involved in the migration project: expressing my interest to management, volunteering for pilot projects, attending internal training sessions, and connecting with the team leading the Azure implementation. My goal is to position myself as someone who understands both the old and new worlds.
The Broader Context
This transition isn't happening in a vacuum. Azure skills are in high demand across Canada and beyond. Universities, government agencies, healthcare organizations, and various industry sectors are all moving to the cloud, with many choosing Microsoft due to existing relationships and integration needs.
For our institution specifically, this migration makes strategic sense. Our existing Microsoft infrastructure—Active Directory, Office 365, Teams, SharePoint—integrates seamlessly with Azure, simplifying authentication and data flows. Microsoft's strong hybrid cloud capabilities also mean we can maintain some on-premises systems while gradually moving to the cloud, reducing risk during the transition.
Looking Ahead
What's remarkable about this career evolution is that I'm not starting from scratch—I'm building on a solid foundation. My 20 years of database experience, combined with deep institutional knowledge of our systems, positions me uniquely for this transition. The cloud migration isn't making my skills obsolete; it's creating an opportunity to apply them in more powerful and modern ways.
The next few years will be transformative, both for my institution and for my career. As we move from PeopleSoft to a modern, cloud-based data platform, I have the chance to be at the forefront of that change, helping shape how our organization manages and leverages data in the decades to come.
For anyone else in a similar position—a database professional watching their organization move to the cloud—my advice is simple: embrace the change. Your existing skills are more valuable than you might think. The fundamental concepts haven't changed; we're just applying them at a larger scale with more powerful tools. The best time to start learning these new technologies is now, while the migration is still in progress and opportunities for hands-on learning abound.
The future of data is in the cloud, and for those of us willing to grow with it, the opportunities are boundless.
