Understanding Distributed Systems: What every developer should know about large distributed applications

Learning to build distributed systems is hard, especially if they are large scale. It's not that there is a lack of information out there. You can find academic papers, engineering blogs, and even books on the subject. The problem is that the available information is spread out all over the place, and if you were to put it on a spectrum from theory to practice, you would find a lot of material at the two ends, but not much in the middle.That is why I decided to write a book to teach the fundamentals of distributed systems so that you don’t have to spend countless hours scratching your head to understand how everything fits together. This is the guide I wished existed when I first started out, and it's based on my experience building large distributed systems that scale to millions of requests per second and billions of devices.If you develop the back-end of web or mobile applications (or would like to!), this book is for you. When building distributed systems, you need to be familiar with the network stack, data consistency models, scalability and reliability patterns, and much more. Although you can build applications without knowing any of that, you will end up spending hours debugging and re-designing their architecture, learning lessons that you could have acquired in a much faster and less painful way.Table of contents1 Introduction1.1 Communication1.2 Coordination1.3 Scalability1.4 Resiliency1.5 Operations1.6 Anatomy of a distributed systemCommunication2 Reliable links2.1 Reliability2.2 Connection lifecycle2.3 Flow control2.4 Congestion control2.5 Custom protocols3 Secure links3.1 Encryption3.2 Authentication3.3 Integrity3.4 Handshake4 Discovery5 APIs5.1 HTTP5.2 Resources5.3 Request methods5.4 Response status codes5.5 OpenAPI5.6 EvolutionCoordination6 System models7 Failure detection8 Time8.1 Physical clocks8.2 Logical clocks8.3 Vector clocks9 Leader election9.1 Raft leader election9.2 Practical considerations10 Replication10.1 State machine replication10.2 Consensus10.3 Consistency models10.4 Chain replication10.5 Solving the CAP theorem10.6 Coordination avoidance11 Transactions11.1 ACID11.2 Isolation11.3 Atomicity11.4 Asynchronous transactionsScalability12 Functional decomposition12.1 Microservices12.2 API gateway12.3 CQRS12.4 Messaging13 Partitioning13.1 Sharding strategies13.2 Rebalancing14 Duplication14.1 Network load balancing14.2 Replication14.3 CachingResiliency15 Common failure causes15.1 Single point of failure15.2 Unreliable network15.3 Slow processes15.4 Unexpected load15.5 Cascading failures15.6 Risk management16 Downstream resiliency16.1 Timeout16.2 Retry16.3 Circuit breaker17 Upstream resiliency17.1 Load shedding17.2 Load leveling17.3 Rate-limiting17.4 Bulkhead17.5 Health endpoint17.6 WatchdogTesting and operations18 Testing18.1 Scope18.2 Size18.3 Practical considerations19 Continuous delivery and deployment19.1 Review and build19.2 Pre-production19.3 Production19.4 Rollbacks20 Monitoring20.1 Metrics20.2 Service-level indicators20.3 Service-level objectives20.4 Alerts20.5 Dashboards20.6 On-call21 Observability21.1 Logs21.2 Traces21.3 Putting it all together22 Final words Read more