Wathon Aung (@Wathon_Aung) header
Wathon Aung (@Wathon_Aung) profile picture

Thrilled to make #data and #software_engineering

Myanmar
Joined May 2014
Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)Photo by Wathon Aung (@Wathon_Aung)
Run a full virtual desktop inside a Docker container, accessible via WebRTC, right from your browser.
77
473
44
5,418
415,852
coding on a friday with my last 2 brain cells
22
192
21
3,242
254,894
Using the combined power of CLI, Streamlit, and Ollama, Stefan Pietrusky presents an end-to-end workflow that allows us to "talk" to PDF files without using proprietary models. buff.ly/4dKnWk8
1
5
22
3,161
Wathon Aung retweeted
Linux grep command cheatsheet
6
310
5
1,851
236,377
AWS Glue federated datasets are available in BigQuery, which connects #BigQuery and an external data source at the dataset level. Federated dataset tables are populated from the tables in AWS Glue, and updates are reflected in BigQuery. Check it out → goo.gle/45v76TB
1
2
25
6,356
Do you remember when you joined X? I do! #MyXAnniversary
20
Wathon Aung retweeted
Scaling Database: When and How to Shard Database sharding refers to splitting data across multiple database servers and is commonly used for scaling. However, sharding introduces major operational and infrastructure complexity that should be 𝗮𝘃𝗼𝗶𝗱𝗲𝗱 𝘂𝗻𝗹𝗲𝘀𝘀 𝗮𝗯𝘀𝗼𝗹𝘂𝘁𝗲𝗹𝘆 𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆. 𝗔𝗹𝘁𝗲𝗿𝗻𝗮𝘁𝗶𝘃𝗲 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵𝗲𝘀 Vertical Scaling: Use more powerful single database servers with more CPUs, memory, storage and I/O bandwidth. Much simpler to manage than sharding. SQL Optimization: Tune SQL queries and database schema to maximize performance on a single server using proper indexes, efficient SQL, etc. Caching: Use in-memory caches like Redis to reduce database load by serving common queries from the cache instead of hitting the database every time. Read Replicas + Load Balancer: Add horizontal read scaleability without full complexity of sharding. Directs reads across replicas. These optimization approaches should be exhausted before considering sharding. 𝗦𝗵𝗮𝗿𝗱𝗶𝗻𝗴 𝗠𝗲𝘁𝗵𝗼𝗱𝘀 There are two high-level approaches: Vertical Sharding: Split database into columnar tables or sections vs rows. For example, having one table for names and another table for emails. Horizontal Sharding: Split database into row partitions distributed evenly across multiple servers. Methods include range based, directory based, and hash based sharding. 𝗪𝗵𝗲𝗻 𝘀𝗵𝗮𝗿𝗱𝗶𝗻𝗴, 𝘂𝘀𝗲 𝘁𝗵𝗲 𝘀𝗶𝗺𝗽𝗹𝗲𝘀𝘁 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 that meets requirements to minimize complexity. Seek to avoid sharding until necessary despite the scaling benefits. The infrastructure and operational overheads often outweigh gains. – Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/496keA7
8
168
4
623
31,212
Wathon Aung retweeted
If you’re reading this post, it’s because our servers are working
53,764
115,802
34,765
1,385,763
106,812,456
Wathon Aung retweeted
we know why you’re all here rn
15,286
42,519
17,855
323,327
37,243,114
Wathon Aung retweeted
Why is Kafka fast? There are many design decisions that contributed to Kafka’s performance. In this post, we’ll focus on two. We think these two carried the most weight. 1️. The first one is Kafka’s reliance on Sequential I/O. 2️. The second design choice that gives Kafka its performance advantage is its focus on efficiency: zero copy principle. The diagram below illustrates how the data is transmitted between producer and consumer, and what zero-copy means. 🔹Step 1.1 - 1.3: Producer writes data to the disk 🔹Step 2: Consumer reads data without zero-copy 2.1: The data is loaded from disk to OS cache 2.2 The data is copied from OS cache to Kafka application 2.3 Kafka application copies the data into the socket buffer 2.4 The data is copied from socket buffer to network card 2.5 The network card sends data out to the consumer 🔹Step 3: Consumer reads data with zero-copy 3.1: The data is loaded from disk to OS cache 3.2 OS cache directly copies the data to the network card via sendfile() command 3.3 The network card sends data out to the consumer Zero copy is a shortcut to save multiple data copies between the application context and kernel context. – Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/3KCnWXq
2
165
694
45,795
Wathon Aung retweeted
Why is Redis Fast? Redis is fast for in-memory data storage. Its speed has made it popular for caching, session storage, and real-time analytics. But what gives Redis its blazing speed? Let's explore: RAM-Based Storage At its core, Redis primarily uses main memory for storing data. Accessing data from RAM is orders of magnitude faster than from disk. This is a major reason for Redis's speed. However, RAM is volatile. To persist data, Redis supports disk snapshots and append-only file logging. This combines RAM's performance with disk's permanence. There is a tradeoff though - recovery from disk is slow. If a Redis instance fails, restarting from disk can be slow compared to failing over to a replica instance fully in memory. So while Redis offers durability via disk, it comes at the cost of slower recovery. A better solution is Redis replication. With a synchronized replica kept in memory, failover is instant with no rehydration. This maintains speed and near-instant recovery. IO Multiplexing & Single-threaded Read/Write Redis uses an event-driven, single-threaded model for its core operations. A main event loop handles all client requests and data operations sequentially. This single-threaded execution avoids context switching and synchronization overhead typical of multi-threaded systems. Redis uses non-blocking I/O to handle multiple connections asynchronously. This allows it to support many client connections with very low overhead, Redis does leverage threading in certain areas: - Background tasks like taking snapshots. - I/O threads are used for certain operations. - Modules can use threads. - Since Redis 6.0, it supports multi-threaded I/O for network communication, improving performance on multi-core systems. Redis also uses pipelining for high throughput. Clients pipeline commands without waiting for each response. This allows more efficient network round trips, boosting overall performance. Efficient Data Structures Redis supports various optimized data structures, from linked lists, zip lists, and skip lists to sets, hashes, and sorted sets, among others. Each is carefully designed for specific use cases for quick and efficient data access. Over to you: With Redis now supporting some multi-threading, how should we configure it to fully utilize all the CPU cores of modern hardware when deploying in production? – Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/496keA7
10
159
2
714
47,004
Wathon Aung retweeted
✨The Technology Behind Netflix Unveiled!✨
23
633
29
2,869
353,175
Harvard University is offering Statistics Course for FREE! Statistics 110: Probability is one of the best probability courses on the internet and it's FREE. Course link 🧵👇
1
178
687
105,951
𝗪𝗵𝗶𝗰𝗵 𝗰𝗹𝗼𝘂𝗱 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝗿 𝘀𝗵𝗼𝘂𝗹𝗱 𝗯𝗲 𝘂𝘀𝗲𝗱 𝘄𝗵𝗲𝗻 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗯𝗶𝗴 𝗱𝗮𝘁𝗮 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻? The diagram below illustrates the detailed comparison of AWS, Google Cloud, and Microsoft Azure, created by Satish Chandra Gupta. They all use similar parts of solutions: 1. 𝗗𝗮𝘁𝗮 𝗶𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 of structured or unstructured data. 2. 𝗥𝗮𝘄 𝗱𝗮𝘁𝗮 𝘀𝘁𝗼𝗿𝗮𝗴𝗲. 3. 𝗗𝗮𝘁𝗮 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴, including filtering, transformation, normalization, etc. 4. 𝗗𝗮𝘁𝗮 𝘄𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲, including key-value storage, relational database, OLAP database, etc. 5. 𝗣𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗹𝗮𝘆𝗲𝗿, with dashboards and real-time notifications. Hyperscalers use different names for the same service type, e.g., AWS "lambda" is Azure "function." Which services did you use, and for what kind of solutions? What are your experiences with these three providers? The full text is in the comments. -- If you like my posts, please follow me @milan_milanovic and hit the 🔔 on my profile to get a notification for all my new posts. Learn something new every day 🚀! #aws #technology #cloudomputing #data #softwareengineering
18
507
16
2,326
380,732
Wathon Aung retweeted
10 System design topics you must prepare for interviews 1. Caching 2. Sharding 3. load-balancing 4. replication 5. fault-tolerance 6. high-availability 7. Performance 8. Scalability 9. replication 10. indexing Learn more on Design Guru - bit.ly/3pMiO8g
6
269
1
1,060
152,660
Wathon Aung retweeted
Docker vs. Kubernetes. Which one should we use? What is Docker ? Docker is an open-source platform that allows you to package, distribute, and run applications in isolated containers. It focuses on containerization, providing lightweight environments that encapsulate applications and their dependencies. What is Kubernetes ? Kubernetes, often referred to as K8s, is an open-source container orchestration platform. It provides a framework for automating the deployment, scaling, and management of containerized applications across a cluster of nodes. How are both different from each other ? Docker: Docker operates at the individual container level on a single operating system host. You must manually manage each host and setting up networks, security policies, and storage for multiple related containers can be complex. Kubernetes: Kubernetes operates at the cluster level. It manages multiple containerized applications across multiple hosts, providing automation for tasks like load balancing, scaling, and ensuring the desired state of applications. In short, Docker focuses on containerization and running containers on individual hosts, while Kubernetes specializes in managing and orchestrating containers at scale across a cluster of hosts. Over to you: What challenges prompted you to switch from Docker to Kubernetes for managing containerized applications? – Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/3KCnWXq
9
231
6
1,009
171,817
Wathon Aung retweeted
Coding a side project? free-for.dev lists services with free tiers for developers.
40
890
31
3,975
430,034
Networking Basics Simplified! A thread of 20 fundamental Networking concepts 👇
23
327
12
1,074
230,903