Data Engineering Community (DEC) (@data_dec) header
Data Engineering Community (DEC) (@data_dec) profile picture

A non-profit organisation providing Data Engineers a supportive and collaborative platform.

Nigeria
Joined April 2024
Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)Photo by Data Engineering Community (DEC) (@data_dec)
Why Data Engineers Care About S3 One of the first questions you should ask when designing a data platform is: Where is the data going to live? When you're working locally, the answer is usually simple. Your files are somewhere on your laptop, an external drive, or perhaps
1
10
329
◉ Lifecycle: How long should the data remain in its current storage class? ◉ Naming: Can another engineer understand what raw/orders/2026/ contains? 📌Key Points ◉ S3 is object storage. ◉ It is commonly used as a foundation for data lakes.
1
10
◉ Storage and compute can be separated. ◉ S3 can store many different types of data. ◉ File format, organization, access and lifecycle all matter.
7
CI/CD for Airflow Pipelines Your Airflow DAG is code. So treat it like code. 📌Instead of changing production DAGs manually, you can have a workflow like: Code ↓ Git ↓ Pull Request ↓ Tests ↓ Review ↓ Merge ↓ Deploy
1
1
8
245
“I can maintain Airflow pipelines in a real engineering environment.” CI/CD is part of the second skill. 📌If another engineer had to maintain your DAG tomorrow, would they know how it's tested and deployed?
1
11
📌Resource of the Week: Astronomer — Airflow CI/CD and Testing Resources
11
Data Engineering Community (DEC) retweeted
This sounds like a simple question. But there is a trap. The interviewer already knows you know what these two are. That is not what they are testing. They want to know if you understand the architecture behind each one. Here🧵
2
4
4
113
Scheduling Is More Than @daily 📌One of the easiest things to do in Airflow is schedule a DAG: >>> schedule="@daily" But production scheduling is more complicated than that.
2
1
6
202
◉ Concurrency needs to be managed 📌Why it matters for data engineers A schedule tells your pipeline when to start. It doesn't automatically tell you whether the data is ready or whether another run is already in progress. 📌Take one of your pipelines and ask:
1
6
What happens if the source data arrives two hours late? 📌Resource of the Week: Apache Airflow Documentation — Scheduling
4