Secrets to Data Engineering
Top secret information. 🤫
Introduction
About four years ago, I was ice fishing on one of the remote lakes at the Arctic Circle. The sunshine felt good after a long and dark winter. While I was trying to catch some monster fish, I had lots of time to consider my future plans. At the time, I was working in the tourism industry as a northern lights guide. The pandemic had disrupted the whole tourism industry, and I needed to create a new plan for myself.
It was there, at that remote lake, that I got an idea. My dream of becoming a data engineer was born. The new dream felt exciting but also terrifying. How could I make the career shift from being a wilderness guide to being a data engineer? The skill sets for these positions were really distant. One big challenge for me was that I didn't really know what I needed to do in order to reach this dream.
Recently, I conducted a survey with data professionals (10 participants) where I asked them a bunch of questions relating to landing the first job in the data world.
The Traditional Pathway
In late 2020, I made the decision to pursue a degree that would help my transition into the field of data. As I explored various Bachelor's programs across Finland, I discovered that Lapland University of Applied Sciences was launching a new degree program in Machine Learning and Data Engineering. Upon learning about this new program, I applied and was fortunate enough to be accepted. The study program was international, and the teaching language was English. I found it incredibly beneficial that most of my classmate came from other countries, as it provided me with the opportunity to further develop my English language skills.
Currently, I am a third-year student in this program, and it has greatly helped me on my journey towards a career in data engineering. Studying topics related to machine learning has also been a fantastic experience, opening up the possibility of pursuing a career as a Machine Learning Engineer as well.
Some years ago, there weren’t any specific study programs for becoming a data engineer. The conventional route to entering this field typically involved pursuing degrees in computer science, data science, or following a pathway specific to a particular domain. For example, one of my friends received his Master’s degree from Economics and now he works as a data engineer in a finance sector. Data engineering itself is still an emerging field and the traditional pathway is a huge ship with a slowly changing course.
Having a degree can be a great asset. It helps you with learning, teamwork and networking.
Self-learning from Online Courses
We are currently living in an era of self-learning, where there are thousands of courses available to teach you almost any technical skill. Many people already have degrees, but they lack technical skills. For them, it doesn't make sense to spend years and a lot of money just to get a technical degree. Some people only need to learn a single important technical skill, and for this, online courses are great. Popular online course platforms like Udemy and Coursera offer excellent learning resources at affordable prices. Certifications are also valuable and highly valued by recruiters. In the data community, there are also data-specific learning platforms such as DataQuest, DataCamp, and Udacity.
I have spent hundreds of hours practicing data-related topics on both DataQuest and DataCamp. They offer rich learning resources that have greatly enhanced my learning.
Online courses can help you learn <insert the hottest tech skill here>, which will set you apart from others!
Fast Track into a Data Career
Bootcamps are becoming increasingly popular as they offer a fast track into the field, providing highly intensive training geared towards acquiring essential skills for a career. In Finland, many bootcamps offer "free" training, after which participants are placed into partner companies through negotiated deals.
Some of the biggest data bootcamp providers in Finland include Academic Work, Saranen Consulting, and Skillio.
Despite the appeal of bootcamps, I personally haven't considered them as I chose for the longer path of obtaining a degree. I hope that my degree, along with my other work, will help me land my dream job.
Landing the first data job can be a real challenge.
Experience from Personal Projects
There is a huge demand for data engineers. The main challenge is that there is a small demand for junior data engineers. All the companies desire those senior engineers with +5 years of experience and impressive tech stack. How would one be able to become one?
In addition to getting a degree or self-learning, one should spend time doing projects. Projects are fantastic because they are usually difficult to complete, require deeper understanding of topics and result into something useful.
What should your projects be about? I have been trying to do projects that I’m curious about. For example, I have a Numpy project where I do data analysis on golf scores on my home club. I try to analyze why one particular player is constantly winning.
If you were to apply to work as a taxi driver, they wouldn’t be impressed about your skills of changing gears. They would much rather just see that you are able to drive. And that you’re good with customers (magical soft skills).
Utilizing Previous Experiences
Sometimes, individuals transition into data-related fields from other positions within the same company. Having specific knowledge about an industry can be a significant advantage in such cases. Additionally, there's a notion that previous success in other positions increases the likelihood of success again.
The Power of Networking
Networking is essential for landing the first job in the data field. Establishing connections provides valuable insights, guidance, and potential job opportunities. Attending events, conferences, and online platforms like LinkedIn facilitates interaction with experts, expands knowledge, and offers access to a broader network.
Active participation in networking activities showcases skills and passion for the data field, demonstrating expertise to potential employers. Networking also unveils hidden job openings, giving individuals a competitive advantage. Moreover, networking can lead to mentorship opportunities, where experienced professionals provide guidance, feedback, and support, aiding in securing the first job in the data field.
For me, LinkedIn has helped me to see what successful data engineers have done in the past. I also found DataTribe from LinkedIn, which has already helped me to network and provide me with opportunities.
Gold Nuggets from Industry Experts
Imagine a person with no experience in the data world. They want to become a data engineer within next 5 years. What advice would you give them?
Gold Nuggets:
Learn batch methods really thoroughly (SQL, spark, orchestrators). Then build a few streaming pipelines ( Flink, beam/cloud dataflow, etc). It also helps a lot if you understand data and analyst work.
Data engineer is basically a python developer with a domain in data. So try to get a baseline grasp of wide set of areas, like python and SWE principles, understanding cli, testing, linting, CI/CD. And develop that over years, trying to practice or working in the companies with data that is interesting to you, this will keep the spark in you when you will be debugging issues.
Do personal hobby projects where you can test new and emerging tech, you can run a lot of stuff in AWS or GCP almost without a cost, also do some online courses. Learning SQL and data modeling is a must. Python is good to know too.
entry level jobs in data (and any software role) is all about potential and learning. Be curious, explore languages, know the basics, and show your passion :)
Study and do your best to partake in actual business-related projects during your academic years
Keep working on your skills and you will get there! End-to-end cloud platform projects, python, SQL. Learn to use serverless functions and fetch data from APIs to transform and store on cloud storage. Parallel processing with Apache Spark on Databricks. Keep grinding. Resilience, passion and true motivation are very important to show to future employers.
I'd probably suggest they follow what folks like Joe Reis, Ben Rogojan (Seattle Data Guy) and others are saying, who they're talking to, and find a few content creators who "click" for you and read what they write, listen to their podcasts, etc. From there you'll be introduced to other interesting thinkers and educators. See what subjects grab your attention and lean into those. There is a pretty comprehensive list of resources here: https://github.com/DataEngineer-io/data-engineer-handbook - the trick is finding some focus. Find ways to practice, if you can find a job that gives you a chance to apply some data skills (eg. SQL, Tableau) then use it to hone those skills and think about the work challenges from a DE perspective as much as you are able. Learn to think about data from a business perspective, practice speaking about data engineering comfortably in plain English (or plain spoken language in whatever language suits you), not just tech terms. Find a person or a community to be a mentor to you on this journey.
Learn the theory, and understand the process.
Learn SQL, Cloud and databases
Get volunteer experience. So something for free to build trust.
Written by Juhani Vanhatapio, Machine Learning and Data Engineering Student
Connect with me: https://www.linkedin.com/in/juhani-vanhatapio/