Why Data Engineers Care About S3
One of the first questions you should ask when designing a data platform is:
Where is the data going to live?
When you're working locally, the answer is usually simple. Your files are somewhere on your laptop, an external drive, or perhaps
1
10
329
◉ Lifecycle: How long should the data remain in its current storage class?
◉ Naming: Can another engineer understand what raw/orders/2026/ contains?
📌Key Points
◉ S3 is object storage.
◉ It is commonly used as a foundation for data lakes.
1
10
CI/CD for Airflow Pipelines
Your Airflow DAG is code. So treat it like code.
📌Instead of changing production DAGs manually, you can have a workflow like:
Code
↓
Git
↓
Pull Request
↓
Tests
↓
Review
↓
Merge
↓
Deploy
“I can maintain Airflow pipelines in a real engineering environment.”
CI/CD is part of the second skill.
📌If another engineer had to maintain your DAG tomorrow, would they know how it's tested and deployed?
1
11
Data Engineering Community (DEC) retweeted
This sounds like a simple question. But there is a trap.
The interviewer already knows you know what these two are. That is not what they are testing.
They want to know if you understand the architecture behind each one.
Here🧵
◉ Concurrency needs to be managed
📌Why it matters for data engineers
A schedule tells your pipeline when to start. It doesn't automatically tell you whether the data is ready or whether another run is already in progress.
📌Take one of your pipelines and ask:
1
6





