Data Engineering

Data engineers build the systems that move data from where it is created to where it is used. Analysts, data scientists and applications depend on that data being correct, fresh and easy to query. If data is water, the data engineer buil…

• What a data engineer does day to day
• How data engineers differ from nearby roles
• The data engineering lifecycle
• How to use this roadmap

Every data tool you will use is limited by four resources: CPU, memory, disk and network. Knowing which one is the bottleneck tells you how to make a job faster.

• 1.1 How a computer works
• 1.2 The Linux command line
• 1.3 Users and file permissions
• 1.4 Processes and environment variables
• 1.5 Shell scripting
• 1.6 Scheduling with cron
• 1.7 Networking basics
• 1.8 SSH and remote servers
• 1.9 Git
• 1.10 GitHub and collaboration

Basics: variables, data types, if / else , loops, functions

• 2.1 Python basics
• 2.2 Data structures
• 2.3 Working with files
• 2.4 Errors, exceptions and logging
• 2.5 Modules, packages and virtual environments
• 2.6 Calling REST APIs
• 2.7 Talking to databases from Python
• 2.8 pandas
• 2.9 Object-oriented basics
• 2.10 Testing with pytest
• 2.11 Code quality habits

• 3.1 Querying and the order SQL runs in
• 3.2 Aggregation

• 4.1 Normalization
• 4.2 Denormalization

• 5.1 Batch vs streaming, full vs incremental loads
• 5.2 Change data capture (CDC)

• 6.1 Why distribute: scaling up vs scaling out
• 6.2 The Hadoop legacy

• 7.1 Events and event-driven design
• 7.2 Message queues vs event logs

• 8.1 Cloud basics
• 8.2 Identity and access management (IAM)

• 9.1 Why plain Parquet files are not enough
• 9.2 How table formats work

• 10.1 Writing data tests
• 10.2 Data contracts

• 11.1 Docker
• 11.2 Docker Compose

• What makes a project stand out

• A typical study week
• If you have more or less time

• Books
• Free courses and practice