Build Your Own Web Server From Scratch In Node.JS : Learn network programming, HTTP, and WebSocket by coding a Web Server

Learn by doing: code a Web server in Node.js Most people use HTTP daily, but few understand its inner workings. This "Build Your Own X" book dives deep, teaching basics from scratch for a clearer understanding of the tools and tech we rely on. Network programming. Protocols & communication. HTTP in detail. WebSocket & concurrency. The project uses Node.js and TypeScript without any dependencies, but many concepts are language-agnostic, so it’s valuable for learners of any language. Beyond coding exercises At the end of each chapter, there are discussions about What’s missing from the code? The gap between toys and the real thing, such as optimizations and applications. Important concepts beyond coding, such as event loops and backpressure. These are what you are likely to overlook. Design choices. Why stuff has to work that way? You can learn from both the good ones and the bad ones. Alternative routes. Where you can deviate from this book. Build your own X Why take on a build-your-own-X challenge? A few scenarios to consider Students: Solidify learning, build portfolio, stand out in future careers. Developers: Master fundamentals beyond frameworks and tools. Hobbyists: Explore interests with flexible, extensible projects. This is part of the “Build Your Own X” book series, which includes books on building your own Redis, database, and compiler. https://build-your-own.org Cover Contents 01. Introduction 1.1 Why Code a Web Server? Network Programming Protocols & Communication HTTP in Detail 1.2 Build Your Own X From Scratch 1.3 The Book Project-Centric Discussions at the End of Chapter Node.js and TypeScript Book Series 02. HTTP Overview 2.1 Overview 2.2 HTTP by Example 2.3 The Evolution of HTTP HTTP/1.0: The Prototype HTTP/1.1: Production-Ready & Easy-to-Understand HTTP/2: New Capacities HTTP/3: More Ambition 2.4 Command Line Tools 03. Code A TCP Server 3.1 TCP Quick Review Layers of Protocols TCP Byte Stream vs. UDP Packet Byte Stream vs. Packet: DNS as an Example TCP Start with a Handshake TCP is Bidirectional & Full-Duplex TCP End with 2 Handshakes 3.2 Socket Primitives Applications Refer to Sockets by Opaque OS Handles Listening Socket & Connection Socket End of Transmission List of Socket Primitives 3.3 Socket API in Node.js Step 1: Create A Listening Socket Step 2: Accept New Connections Step 3: Error Handling Step 4: Read and Write Step 5: Close The Connection Step 6: Test It 3.4 Discussion: Half-Open Connections 3.5 Discussion: The Event Loop & Concurrency JS Code Runs Within the Event Loop JS Code and Runtime Share a Single OS Thread Concurrency in Node.JS is Event-Based 3.6 Discussion: Asynchronous vs. Synchronous Blocking & Non-Blocking IO IO in Node.js is Asynchronous Event-Based Programming Beyond Networking 3.7 Discussion: Promise-Based IO 04. Promises and Events 4.1 Introduction to `async` and `await` Using Callbacks Using Promises and `await` Creating Promises 4.2 Understanding `async` and `await` Normal Functions Yield to the Runtime by `return` `async` Functions Yield to the Runtime by `await` Calling `async` Functions Start New Tasks 4.3 From Events To Promises Step 1: Analyze The Solution Step 2: Handle the 'data' Event Step 3: Handle the 'end' and 'error' Event Step 4: Write to Socket 4.3 Using `async` and `await` 4.5 Discussion: Backpressure Waiting for Socket Writes to Complete? Producers are Bottlenecked by Consumers Backpressure in TCP: Flow Control Backpressure Between Applications & OS Unbounded Queues are Footguns 4.6 Discussion: Events and Ordered Execution 4.7 Conclusion: Promise vs. Callback 05. A Simple Network Protocol 5.1 Message Echo Server 5.2 Dynamic Buffers The Need for Dynamic Buffers Coding a Dynamic Buffer 5.3 Implementing a Message Protocol Step 1: The Server Loop Step 2: Split Messages Step 3: Handle Requests 5.4 Discussion: Pipelined Requests Reduce Latency by Pipelining Requests Support Pipelined Requests Deadlock by Pipelining 5.5 Discussion: Smarter Buffers Removing Data from the Front Using Fixed Sized Buffer without Reallocations 5.6 Conclusion: Network Programming Basics 06. HTTP Semantics and Syntax 6.1 High-Level Structures 6.2 Content-Length The Length of the HTTP Header The Length of the HTTP Body 6.3 Chunked Transfer Encoding Generate and Send Data on the Fly Another Layer of Protocol Chunks Are Not Messages 6.4 Ambiguities in HTTP The Happy Cases of Body Length Mind the Nasty Cases 6.5 HTTP Message Format Read the BNF Language HTTP Header Fields 6.6 Common Header Fields 6.7 HTTP Methods Read-Only Methods Cacheability CRUD and Resources Idempotence Comparison of HTTP Methods 6.8 Discussion: Text vs. Binary Text is Often Ambiguous Text is More Work & Error-Prone 6.9 Discussion: Delimiters Serialization Errors in Delimited Data Length-Prefixed Data in Binary Protocols 07. Code A Basic HTTP Server 7.1 Start Coding Step 1: Types and Structures Step 2: The Server Loop Step 3: Split the Header Step 4: Parse the Header Step 5: Read the Body Step 6: The Request Handler Step 7: Send the Response Step 8: Review the Server Loop 7.2 Testing Large HTTP Body Connection Reuse & Pipelining 7.3 Discussion: Nagle's Algorithm Optimization: Combining Small Writes Premature Optimization What People Actually Do in Practice 7.4 Discussion: Buffered Writer Alternative: Make Buffering Semi-Transparent Alternative: Add a Buffered Wrapper 08. Dynamic Content and Streaming 8.1 Chunked Transfer Encoding The Chunked Message Format Obscure HTTP Features 8.2 Generating Chunked Responses 8.3 JS Generators Producers, Consumers and Queues JS Generators as Producers Consume from JS Generators 8.4 Reading Chunked Requests Chunk Parser as a Producer Decode the Chunk Format Chunked Encoding Produces a Byte Stream 8.5 Discussion: Debugging with Packet Capture Tools Capture & Inspect Data with `tcpdump` Analyze Packets with Wireshark Search Packet Data with `ngrep` 8.6 Discussion: WebSocket Polling for Updates Real-Time Server Push WebSocket is Message-Based WebSocket is NOT Request-Response 09. File IO & Resource Management 9.1 File IO in Node.JS 9.2 Serving Disk Files Step 0: Add a Handler for Static Files Step 1: Open and Close a File Step 2: `stat()` a File Step 3: Construct the `BodyReader` Step 4: Make Sure the File is Closed 9.3 Discussion: Manual Resource Management Resources Are Owned by Something Resources Owned by Functions or Scopes Ownerships Can be Transferred Owners Are Chained How to Do Cleanups in a Generator 9.4 Discussion: Reusing Buffers Buffer Allocations Have Costs Using Uninitialized Buffers Reusing Large Buffers Pooled Buffers Mind the Lifetime When Reusing Buffers 10. Range Requests 10.1 How Range Requests Work The `Range` Header Field Syntax The Effective Range with Examples Range Requests Are Optional Range Responses Are Indicated by 206 `Content-Range` Returns the Effective Range Announcing `Range` Support Probing with the `HEAD` Method Multiple Ranges with the Multipart Message Discussion: Is Multipart Message Technically Necessary? Possible Cases for Clients 10.2 Implementing Range Requests Step 1: Parsing the Range Header Field Step 2: Reading a Portion of a File Step 3: Generating Range Responses Step 4: Adding the `HEAD` Method 11. HTTP Caching 11.1 Cache Validator Validate by Timestamp Validate by ETag Validate for Range Requests Summary of Validation Headers Implementing the Timestamp Validator 11.2 Discussion: Server-Side Cache Control When to Revalidate Cache Control the TTL Make the Client Always Check the Server Prevent Caching Reduce Fresh Validations Heuristic Caching Non-Client Caching Shared and Private Caches Caching in the Cloud 12. Compression & the Stream API 12.1 How HTTP Compression Works Negotiating with `Accept-Encoding` Compress with `Content-Encoding` Compression and `Content-Length` Compression and Caching Compressed Uploads Compression and Range Requests Compress with `Transfer-Encoding` 12.2 Data Processing with Pipes Data Processing and IO Unix Pipes for Composable Data Processors Abstract Pipes 12.3 Exploring the Stream API in Node.JS List of Stream Interfaces Using Pipes on Stream Interfaces 12.4 Implementing HTTP Compression Step 1: Check Header Fields and Enable Compression Step 2: Understanding the Gotchas without Pipes Step 3: Implementing `stream.Readable` Step 4: Creating a Pipe Step 5: Reading from `stream.Readable` Step 6: Test It Step 7: Flush the Compressor Buffer 12.5 Discussion: Refactoring to Stream 12.6 Discussion: High Water Mark and Backpressure One-Item Queue Backpressure by High Water Mark Batching Data with Queues High Water Mark in Node.JS 13. WebSocket & Concurrency 13.1 Establishing WebSockets High Level Overview Upgrade From HTTP/1.1 WebSocket Handshake The `Connection` Header Field 13.2 WebSocket Protocol Frame Format Types of Frames Fragmented Messages 13.3 Introduction to Concurrent Programming Race Conditions Atomic Operations Mutual Exclusion with Mutexes Multiplexing with Queues Flow Control with Blocking Queues Cancellation with Closeable Queues The Blocking Queue as a Synchronization Primitive 13.4 Coding a Blocking Queue Step 1: Analyze the Problem Step 2: Push and Pop Step 3: Close the Queue Step 4: Add a Buffer 13.5 WebSocket Server Step 1: Design the Message Interface Step 2: Design the Server Application Interface Step 3: Design the Server Tasks Step 4: Start Server Tasks Step 5: Write Messages to the Socket Step 6: Parse Frames From the Socket Step 7: Integrate with the HTTP Server Step 8: Test and Debug 13.6 Discussion: WebSocket in the Browser No Backpressure for WebSocket in the Browser Application Level Backpressure Message Size is Limited 13.7 Conclusion: What We Have Learned