Freemanš¼ retweeted
Just built my first end-to-end data pipeline š
DefiLlama API ā pandas ā PostgreSQL, fully containerized with Docker. Hereās how it came together:
1ļøā£ Extract: pulled @monarq_mgmt's daily TVL history from the DefiLlama API with Pythonās requests
2ļøā£ Transform: used pandas to keep date and totalLiquidityUSD and convert Unix timestamps to UTC datetimes
3ļøā£ Load: wrote the DataFrame to a Postgres table with SQLAlchemy
4ļøā£ Containerize: packaged the pipeline in a Dockerfile, with dependencies managed by uv
5ļøā£ Verify: queried the table with pgcli to confirm the data landed
Big thanks to @AnalyticSages and @Freemandayly for the structure and guidance.
Repo: github.com/tobeydev/assignmeā¦
Freemanš¼ retweeted
If you are a data analyst, the best thing you can do, as you work with an AI, is make a skill out of Kimball's Dimensional modelling book.
I think AI is actually quite bad at making data models. IMO, there is something very human in making and accessing data models that AI can't capture.
If you read (well, the first few chapters) of the book, Kimball is really, really particular about designing the table to be intuitive to the user and never prioritizing hard data modelling rules.
My personal rule to the AI when making tables is that it has to be so intuitive that the user don't even need docs and just get cracking and exploring the tables
Likewise, in this new weird paradigm, also consider designing tables for the AI for the easiest possible access.
Idk what I've cooked with my Cursor, but it keeps creating new data marts out of my gold tables instead of writing queries to get some dataš
I get what she is saying. Historically, you have
- Human slop cannon
- People who think all code should be Good Code, who are very useful because of talent but requiring management because of the constant desire to boil the ocean / move at the speed of a snail
- People who think important (seams, data modeling, rules engines, stuff like that) code should be Good Code, and its fine if other code is slop
The most valuable persona is the third. The first will destroy a business due to overrunning tech debt, the second will tank a business due to neverending refinement. The third understands those forces need to be kept in balance.
I have a huge amount of respect for Lauren, I know folks who have worked with her and say she is absolute top tier. I know she spent a huge amount of time and effort working through Dune for example. There was probably a non trivial amount of code involved there, and it probably is the kind of thing that needed to be Good Code.
"Slop is a slur" I think is cope. "Most code in the before-times was slop and that was ok" I think is reasonable and defensible. "All code is slop" I think is an overreach, and dismisses a very important part of this conversation.
Freemanš¼ retweeted
Blockchain data is becoming more important.
But working with it takes more than just understanding blockchain.
You need to know how to collect, process, store and analyze on-chain data.
Thatās exactly what weāre exploring in our Blockchain Data Engineering Program.
Registration ends October 3rd.
Secure your spot now!!
analyticsages.io/programs/blā¦
See you in class.
#BlockchainData #DataEngineering #Web3 #Blockchain #AnalyticSages
Freemanš¼ retweeted
14 days until we kick off Blockchain Data Engineering on 6th October 2026.
This isnāt another theory-heavy course.
Itās a practical journey into building production-ready data pipelines for the blockchain space powered by the tools that actually matter:
Python | PostgreSQL | Web3.py | dbt
Apache Airflow | ClickHouse | Docker | Git
If youāre serious about becoming a Blockchain Data Engineer, this is your moment.
Spots are limited.
Register now š
analyticsages.io/programs/blā¦
#Blockchain #Web3 #DataEngineering
Freemanš¼ retweeted
Everyone wants to build on blockchain.
But hereās the question:
Who builds the data infrastructure that makes it all possible?
Blockchain is more than smart contracts, protocols, and dApps. Behind every meaningful on-chain insight is data infrastructure that helps us collect, process, analyze, and make sense of blockchain data.
Join us tomorrow as we unpack the people, systems, and skills powering the data layer of Web3.
TOPIC:
Everyone Wants to Build on Blockchain. Who Builds the Data Infrastructure?
Date: September 4, 2026
Time: 7:00 PM WAT
Space: xitter.cf/i/spaces/1rGmqpbā¦
Speakers
⢠@apostleoffin ā Quant/Data Engineer
⢠@oladeeayo ā Data Analyst
⢠@Freemandayly ā Blockchain Data Engineer.
Whether you're a data analyst, engineer, researcher, developer, or simply curious about blockchain data, this conversation is for you.
Come learn, Come ask questions, Come discover what happens behind the blockchain data.
Set your reminder, you don't want to miss this.
Freemanš¼ retweeted
I designed a pipeline that models cross-chain transactions , reconciling the txn from source chain to destination chain.
Agents & analysts can then know exactly what a txn is.
github.com/Freemandaily/crosā¦
Freemanš¼ retweeted
Working on Chainpull ,a no-code indexing platform that abstracts away the complexity of setting up indexing engines.
Blockchain data analysts can simply request the data they need and get it back as clean, ready-to-use output.
What feature would you want to see added?
Freemanš¼ retweeted
We are building a tool that helps detect anomalies in token distribution and gives you a clear view of how tokens are allocated among the top holders.
The days of hidden token distributions are over. With greater transparency, you can identify concentration risks
Freemanš¼ retweeted
Continuation.
In this article, I cover how to design an ELT pipeline. It provides a complete understanding of how to build and run your data warehouse efficiently.
Read here : open.substack.com/pub/freemaā¦
I designed an end-to-end ETL Pipeline for Ethereum Blockchain Data
Check out my latest article where I break down the entire architecture from extraction to storage
Read the full deep-dive here: open.substack.com/pub/freemaā¦
Freemanš¼ retweeted
In this project, I leveraged DuckDB to analyze token holder distribution.
DuckDB efficiently handles millions of rows on a local machine, making it a powerful and cost-effective choice for analytical workloads.
token-intelligence.onrender.ā¦
Freemanš¼ retweeted
Itās amazing what you can build when you have access to clean, structured, in-house data.
Iām building a blockchain forensics tool to explore token distribution, analyze holder clustering, and visually trace wallet connections. This is the power of proper data modeling
Freemanš¼ retweeted
Data engineering is the backbone of every data-driven application, and the modeling approach is central to how the product performs.
At Tradeslens, I used dbt for data modeling, which enables efficient and reliable querying.
product live at intel-tradeslens.onrender.coā¦.
Freemanš¼ retweeted
Kelp DAO exploit
I built ChainTrace Forensic, a reproducible forensic pipeline that
⢠Extracts the attacker's complete fund flow on-chain
⢠Visualizes the trace of every transfer
⢠Simulates fund movement step-by-step
Live here chaintrace-forensic.onrenderā¦
A young and smart Nigerian guy messaged me last night. Hopped on a call. He was like "Pashov Audit Group is one of a kind and all other security companies suck, I want to work with you"
There definitely are other good security companies. Already working with the guy thoughš«”
Freemanš¼ retweeted
I started blockchain investigation to provide assistance to the community and help victims of hacks and exploits.
Since then, Iāve offered pro bono support on several cases involving stolen funds and on-chain investigations.
If youāve found any of my work valuable, Iād appreciate your support in the giveth ongoing quadratic funding round.
Even a $1 donation can make a difference š¤
qf.giveth.io/project/specterā¦
Freemanš¼ retweeted
Just shipped a major upgrade to TradeLens .
A real-time DeFi data pipeline ingesting live Uniswap pool data across Ethereum, Arbitrum & Optimism.
This isn't a dashboard. It's a data engineering system. Here's the stack
#dataengineering
Freemanš¼ retweeted
I built a full end-to-end data engineering pipeline to analyze what the Ethereum blockchain looked like from the inside during the October 10, 2025 crypto crash.
Tech stacks
PySpark ā GCS ā BigQuery ā dbt ā Looker Studio
Here's the dashboard and what the data reveals
Freemanš¼ retweeted
Building TradesLens, a bird's-eye analytics layer for DEX trading across protocols.
If you've ever wanted a unified view of on-chain swap activity across protocol.. that's exactly what I'm building.
Here's how the stack works
#DataEngineering
























