Joined January 2019
- Tweets799
- Following145
- Followers1.2K
- Likes2.6K
Weekend is here and so are my hypervisor updates.
Was pretty occupied the entire week but managed to debug my `Invalid OpCode` issue with KVM.
Root cause? - I wasn't loading all of the segments of the ELF properly 🤦♂️.
Now finally kernel is loading properly and has a simple serial driver.
Off to the real work now, a hypervisor, deterministic hypervisor!
Not much of an update today. Spent almost the entire day debugging "KVM internal error". Happens only if I call `panic!` in my Kernel (not in bootloader).
The paging is set, stack is set, ELF sections are loaded properly. Will continue debugging this week.
And my bootloader can finally load my kernel! That completes step 1 of many other steps in my attempt create a hypervisor!
Bootloader can now properly parse the EFI_SYSTEM_TABLE and can extract RSDP related data to pass onto the kernel.
It might have been a mistake to write all the parsers on my own but it forces me to carefully go through both ACPI as well as UEFI specs (for copy-pasta).
After a busy week, finally started working on the hypervisor!
The goal is to have a simple bootloader + kernel (capable of creating simple VMs) up and running by the end of this weekend.
This weekend's read was "DiffKV". It was kinda dense read as it is an amalgamation of quite a few ideas:
- Inline KVs for small sized vals.
- Separate KVs to a vTree (LSM like structure) for mid sized vals.
- Separate KVs to a vLog (ref: WiscKey) for large vals.
- Use "hotness aware" vLog.
- Reduce GC overhead by running them "lazily"
- Several other optimizations!
I really liked the paper for the various ideas it shared but I think it definitely is more complex for my needs right now :).
ALT Screenshot of the first page of the paper "Differentiated Key-Value Storage Management for Balanced I/O Performance".
I am reviewing the literature around LSM based storages as I am planning to write a toy etcd compatible DB.
Today's read was "WiscKey". The basic ideas of the paper are:
- Store values in a separate vLog file to reduce amplifications.
- Replace regular log file with vLog file.
- To reduce the cost of randomized IO on the vLog file use prefetching (exploit SSD parallelism).
I am not sure how GC on vLog will affect performance when the system is under high write workload. My concern is not the algorithms or locks but GC saturating the Disk bandwidth.
ALT Screen short of the first page of the "WiscKey: Separating Keys from Values in SSD-Conscious Storage" paper.
And my copy of “Hacker’s delight” is finally here!
Thanks to #1BRC!
ALT Image of the book “Hacker’s Delight” - 2nd edition!
Yesterday I decided to give @zeddotdev a shot and saw "CRDT" (on their site) which piqued my interest in them again.
"A Gentle Introduction to CRDTs" was a fun little read!
I am considering to read a few foundation papers on CRDTs as they are still a huge black box for me.
ALT Screenshot of the first 2 paragraphs of the blog "A Gentle Introduction to CRDTs".
"Slashing Data Transfer Costs in AWS by 99%" was a fun little read.
The idea is to use S3 for data transfer relying on:
- S3 does replication across AZs in the region
- No cost to download and upload to S3
- Minimizes cost of storage is cleared < 1 hour
Cons: High latency
ALT Screenshot of the blog "Slashing Data Transfer Costs in AWS by 99%".
This was an interesting read! They faced primarily 2 issues on baremetal:
1. GC using too much CPU when there were many allocations - Fixed by tuning GC.
2. Making too many `EpollWait` syscalls due to limited NetPoll "ready sockets" buffer size - Fixed by spawning > 1 processes.
ALT Screenshot of the blog "Scaling Golang to 192 Cores with Heavy I/O" written on 10th Jan 2024. The screenshot shows first few paragraphs of the blog.
Zomato's blog on migration from TiDB to Dynamo DB was a confusing read for me. It seems they had 2 primary issues:
1. Slow queries bcz of their data model
2. Over provisioning due to self-hosting
Why not move to TiDB cloud and change the data model (they did anyway for dyanamo)?
Today's not so mini-read "Locking in Webkit".
The article talks about building a userspace adaptive lock focussed on minimum memory usage (<= 1 byte) and throughput maximization and contention.
Note: They build lock on top of their pre-existing thread parking framework.
ALT Screenshot of the first 2 paragraphs of the article "Locking in WebKit".
Absolutely enjoyed reading "The Myth of RAM"!
It's a 4 part article talking about how random memory access is not O(1) rather is O(sqrt(n)) and then goes on to talk about its implications in part 3.
PS: I don't really understand the "theoretical limit" in part 2.
ALT Screenshot of the article "The Myth of RAM, part 1". Shows the "Preface" and "Intro" sections.
I enjoy reading Swiggy's (Food Delivery Company) "BehindTheBug" articles where they share their learnings from an outage.
"Breaking the Event Pipeline" is a similar article which investigates the outage they experienced in Nov 2022.
My takeaway: Always have regression suite!
ALT Screenshot of the first paragraph of the article shared by Swiggy on Medium. The article is titled "#BehindTheBug: Breaking the Event Pipeline: A Migration Gone Wrong".