Data Engineering Community (DEC) (@data_dec) header
Data Engineering Community (DEC) (@data_dec) profile picture

A non-profit organisation providing Data Engineers a supportive and collaborative platform.

Nigeria
Joined April 2024
Why Data Engineers Care About S3 One of the first questions you should ask when designing a data platform is: Where is the data going to live? When you're working locally, the answer is usually simple. Your files are somewhere on your laptop, an external drive, or perhaps
1
10
330
CI/CD for Airflow Pipelines Your Airflow DAG is code. So treat it like code. 📌Instead of changing production DAGs manually, you can have a workflow like: Code ↓ Git ↓ Pull Request ↓ Tests ↓ Review ↓ Merge ↓ Deploy
1
1
8
245
Scheduling Is More Than @daily 📌One of the easiest things to do in Airflow is schedule a DAG: >>> schedule="@daily" But production scheduling is more complicated than that.
2
1
6
202
Retries Are Not a Magic Fix Your pipeline fails. Airflow says: >>> Retrying... Great. But should every failure be retried? Consider an API timeout. That's potentially a temporary problem, so retrying could make sense. Now consider invalid SQL.
2
2
132
Your Pipeline Failed. Now What? Your pipeline has been running successfully for two weeks. 📌Then one morning: FAILED. What do you do? First, don't immediately rerun everything. Find the failed task and check its logs. 📌You might discover: ◉ An API timeout
1
1
174
Airflow vs Prefect vs Dagster Once you start learning orchestration, you'll quickly discover that Airflow isn't the only option. You'll come across tools like Prefect and Dagster too. 📌So what's the difference?
1
3
4
164
Writing Your First Airflow DAG Enough theory. Let's build something. Imagine we're pulling data from an API, transforming it with Python and loading it into PostgreSQL. 📌Our DAG needs to handle three main tasks: ◉ Extract → Transform → Load
1
1
125
Airflow Tasks and Operators Now that we understand DAGs, let's talk about the actual work happening inside them. A task represents a unit of work that Airflow needs to execute. 📌That work could be: ◉ Running a Python function ◉ Executing SQL ◉ Running a Bash command
1
2
5
157
What Exactly Is a DAG? You've probably heard: “Create a DAG in Airflow.” But what exactly is a DAG? DAG stands for Directed Acyclic Graph. The name sounds more complicated than the idea. A DAG is simply a way of representing tasks and the relationships between those tasks.
1
4
165
🎉 Happy Birthday, Chizoba Eze ! 🎂 Today, we celebrate you, not just for the amazing person you are, but also for the creativity and dedication you bring to the Data Engineering Community as our graphics designer and volunteer. Thank you for bringing our ideas to life
2
7
248
Why Data Engineers Need Orchestration A Python pipeline can work perfectly when you run it yourself. But what happens when it needs to run every morning? What if the API fails? What if one task depends on another? What if the database is unavailable? What if you need
2
6
16
370
🎉 Celebrating Nancy Amandi and Chidera Ozigbo! 🎂 Today, we celebrate two amazing members of the Data Engineering Community, Nancy Amandi, our Hackathon Coordinator, and Chidera Ozigbo, our Webmaster! Beyond their official roles, Nancy and Chidera have consistently
1
3
180
🚀 From Single-AZ to Multi-AZ: How We Migrated a Production Kafka Cluster on Confluent Cloud What does it actually take to migrate a production Kafka cluster from Single-AZ to Multi-AZ without disrupting producers, consumers, and critical workloads?
2
1
2
8
332
🎉 Happy Birthday, @musili_adebayo ! 🎉 Today, we celebrate our amazing Community Manager, Musili Adebayo! 🥳 Musili has been an incredible part of the Data Engineering Community , helping to build connections, create opportunities for learning, and bring our community
1
2
11
236
Docker and CI/CD You've written your pipeline. You've Dockerized it. It works on your machine. 💡Now what? 📌Eventually, your pipeline needs to move through environments such as: Development ↓ Testing ↓ Production
3
5
177
Docker Best Practices for Data Engineers Getting a Docker container to run is one thing. Building Docker images that are secure, maintainable, and efficient is another.
1
3
9
340
Docker Compose for Data Pipelines A real data engineering project rarely consists of one service. You might have: ◉ Python ETL ◉ PostgreSQL ◉ Redis ◉ Airflow ◉ dbt Running each service manually can quickly become frustrating.
1
4
152
Writing Your First Dockerfile So far, we've talked about what Docker does. Now let's build something. A Dockerfile is a text file containing instructions Docker uses to build an image. 📌Imagine you have a simple Python ETL project: etl-project/
1
3
103
Happy Birthday, Abdulqudus Ayoola On behalf of everyone at the Data Engineering Community (DEC), we celebrate you today and appreciate your contributions as a valued member of our volunteer team, especially your support with partnerships and community growth.
1
2
8
218
Docker Images vs Containers One of the first things you encounter when learning Docker is the relationship between images and containers. 📌 They are related, but they are not the same thing. A Docker image is a packaged blueprint containing everything needed to create
1
4
135