Varnish 6 by Example

foreword what-is-varnish-1 what-is-vcl varnish-cache-and-varnish-enterprise version-numbers product-vs-project which-features-does-varnish-cache-have X58585f412a0759febe7dfb9fc97d8dce6804b1f which-use-cases-does-varnish-address video-streaming-acceleration web-application-firewalling under-the-hood the-manager-process the-vcl-compiler-process compilation-steps the-child-process threads the-cache-main-thread the-thread-pool-herder-thread the-acceptor-threads the-waiter-thread the-expiry-thread the-backend-poller-thread the-ban-lurker-thread worker-threads transports disembarking the-waiting-list serialization workspaces streaming varnish-fetch-and-delivery-processors chapter-summary varnish-6 why-varnish-6 a-lot-of-old-content-out-there varnish-versions-vs-vcl-syntax-versions encouraging-upgrades its-the-way-forward whats-new-in-varnish-6 whats-new-in-varnish-6.0 unix-domain-sockets-uds http2-support-considered-stable other-features-in-varnish-6.0 whats-new-in-varnish-6.1 whats-new-in-varnish-6.2 whats-new-in-varnish-6.3 explicitly-trigger-vcl_backend_error vmod-import-changes X6d447d32895815043889dd1bddb6a3198be4980 std.ip-accepts-optional-port-argument querying-changes-in-vsl-tools whats-new-in-varnish-6.4 if-range-support import-vmod_cookie-from-varnish_modules defining-none-backends other-vcl-changes whats-new-in-varnish-6.5 strict-cidr-checks-on-acls vcc__acl__pedantic-parameter obj.can_esi a-new-.resolve-method closing-the-connection blob-literal-syntax std.blobread X45cf4602cbd1c51c27afb3b331f9ba128d71d1e help-screen-in-varnishstat whats-new-in-varnish-6.6 start-varnish-without-a-backend header-validation vary-notices checking-ban-errors modulus-operator new-notation-for-long-strings new-built-in-vcl vcl-variable-changes backports-to-6.0-lts varnish-enterprise-6 the-origin-story new-features-in-varnish-enterprise-6 total-encryption-and-vmod_crypto encoding hashing encryption total-encryption vmod_urlplus the-return-of-req.grace vmod_synthbackend mse3 vmod_ykey varnish-high-availability-6 vmod_mmdb vmod_utils explicitly-return-errors json-formatting-support-in-varnishncsa vmod_str vmod_mse set-weighting-algorithm select-stores-by-tag last-byte-timeout if-range-support-1 built-in-tls-support memory-governor vmod_jwt vmod_stale vmod_sqlite3 vmod_tls vmod_headerplus vmod_resolver veribot vmod_brotli vmod_format scoreboard X8df9af27084e87155b099d167d630fa15fcf063 Xeb2b2f5e6838b611e94d7ba7ca44b08fbbacf82 where-to-get-it the-official-package-repositories installing-from-source official-docker-image official-cloud-images varnish-enterprise-features-in-the-cloud licensing-and-billing chapter-summary-1 its-all-about-http http-as-the-go-to-protocol the-strengths-of-http the-limitations-of-http newer-versions-of-the-http-protocol http1.1 http2 http3.0 what-about-varnish http2-in-varnish http3-in-varnish the-expires-header max-age-vs-s-maxage public-vs-private deciding-not-to-cache revalidation how-varnish-deals-with-cache-control surrogates the-surrogate-capability-header the-surrogate-control-header surrogate-caching surrogate-targeting surrogate-support-in-varnish ttl-header-precedence-in-varnish cacheable-request-methods cacheable-status-codes cache-variations the-vary-header accept-language-variation-example hit-rate-considerations sanitizing-user-input varying-on-custom-headers varnish-built-in-vcl-behavior when-is-a-request-cacheable cacheable-request-methods-1 invalid-request-methods state-getting-in-the-way X07e52c5f24f2c56014ca6dc5504948f0ca048b8 dealing-with-stale-content Xf5cba23f1cc2d00d2ca8b00b4ff710a1cd4905c X6526f93c28aabd34238d03664c57db689a0761e range-requests accept-ranges-response-header range-request-header content-range-response-header what-if-the-range-request-fails range-request-support-in-varnish impact-on-the-origin backend-range-requests-using-vcl conditional-requests not-modified etag-the-fingerprint if-none-match the-workflow strong-vs-weak-validation conditional-request-support-in-varnish conditional-request-workflow-in-varnish grace-vs-keep X5da31c8f1cb9dcd4de32de85ebbb717482feeaf some-context exit-early leveraging-varnish X03112052bc044c8800bdeef732f768cbe3df296 conditional-range-requests compression content-negotiation gzip-compression-in-varnish gzip-and-vcl brotli-compression-in-varnish content-streaming chunked-transfer-encoding streaming-support-in-varnish summary the-varnish-configuration-language what-is-vcl-again the-finite-state-machine the-client-side-flow the-backend-flow hooks-subroutines-and-built-in-vcl vcl_recv error-cases to-pipe-or-not-to-pipe only-get-and-head stateless anything-else-gets-cached vcl_hash vcl_hit a-dirty-little-secret-about-vcl_hit vcl_miss vcl_purge vcl_pass vcl_pipe vcl_synth vcl_deliver vcl_backend_fetch vcl_backend_response uncacheable zero-ttl a-cookie-was-set surrogate-control cache-control-says-no vary-all-the-things vcl_backend_error vcl_init vcl_fini vcl-syntax vcl-version-declaration assigning-values strings conditionals operators comments numbers booleans time-durations time duration regular-expressions backends the-basics optional-values probes default-values extending-values customizing-the-entire-http-request assigning-the-probe-to-a-backend tcp-only-probes unix-domain-sockets overriding-the-host-header access-control-lists functions ban hash_data synthetic regsub a-practical-example a-practical-example-1 subroutines include import vcl-objects-and-variables connection-variables proxy-vs-no-proxy the-ip-type local-variables identities request-variables a-request-example X2cc1151edb0723691ac001b012f4aa6335c7199 backend-request-variables backend-response-variables vfp-related-backend-response-variables X2cbdc4e51a8d96a540d4ca021d01e976e5a2ab0 other-backend-response-variables object-variables response-variables storage-variables making-changes excluding-url-patterns sanitizing-the-url alphabetic-sorting X6d61b5a3522df1c95ff67298c52473fc93a66bf removing-url-hashes removing-trailing-question-marks stripping-off-cookies removing-select-cookies removing-all-but-some-cookies using-vmod_cookie using-vmod_cookieplus sanitizing-content-negotiation-headers overriding-ttls static-data-example overriding-the-default-ttl zero-ttls-are-evil dealing-with-websockets enabling-esi-support inspect-the-url inspect-the-content-type-header surrogate-headers protocol-detection using-vmod_proxy using-vmod_tls vcl-cache-variations protocol-cache-variations language-cache-variations language-cookie-cache-variation using-vmod_cookie-1 using-vmod_cookieplus-1 custom-error-messages the-current-built-in-vcl-implementation customize-error-messages-using-templates caching-objects-on-the-second-miss validation-and-testing syntax-validation testing built-in-vcl-test a-failing-test looking-at-varnishs-tests a-vcl-test summary-1 varnish-modules-vmods whats-a-vmod scope-and-purpose vmod-api vcl-usage vmod-initialization installing-a-vmod Xa51c11c7c4305be167d66d38c59fb7a48b57e3d vmod_blob vmod_cookie vmod_directors vmod_proxy vmod_std logging string-manipulation environment-variables reading-a-file server-ports vmod_unix X7848bf78716e6bd3b12d94da8a5306b4931074b vmod_accept vmod_aclplus advanced-acls a-key-value-store-example vmod_cookieplus set-cookie-logic vmod_crypto hashing-encoding encryption-1 vmod_deviceatlas vmod_edgestash vmod_file file-backends command-line-execution vmod_format-1 vmod_json the-dns-backend the-dns-director extra-options dynamic-backends-example vmod_headerplus-1 vmod_http vmod_jwt-1 vmod_kvstore rewrite-rules-in-vcl vmod_rewrite-rulesets rulesets-as-a-string matching-url-patterns extracting-ruleset-fields where-can-you-find-other-vmods some-third-party-vmods-i-like vmod_basicauth vmod_redis the-varnish-software-vmod-collection vmod_bodyaccess vmod_header vmod_tcp vmod_var vmod_vsthrottle vmod_xkey how-to-install-these-vmods compiling-from-source debian-and-ubuntu-distro-packages writing-your-own-vmods vmod_example turning-vmod_example-into-vmod_os dependencies getting-the-code looking-at-the-vmod_os.c looking-at-the-vmod_os.vcc building-the-vmod testing-the-vmod using-the-vmod summary-2 purging purge-vcl-code triggering-a-purge vmod_purge hard-purge soft-purge Xd90636b4f77fbc0a2ef674ae0d6316196112832 banning ban-expressions expression-format expression-examples executing-a-ban-from-the-command-line ban-vcl-code purge-replacement invalidate-url-patterns complete-flexibility the-best-of-both-worlds the-ban-list there-is-always-an-item-on-the-list adding-a-first-ban adding-multiple-bans the-ban-lurker runtime-parameters ban-lurker-workflow ban-lurker-scope enforcing-asynchronous-bans tag-based-invalidation integrating-bans-in-your-application ban-limitations secondary-keys vmod_xkey-1 initializing-vmod_xkey registering-keys invalidating-content vmod_xkey-limitations locking old-objects-arent-processed Xa2e98b95eb4683e95c8172c1970637b0e42e3c4 vmod_ykey-2 why-ykey vmod_ykey-performance-improvements registering-keys-1 invalidating-content-1 a-vmod_xkey-replica multiple-keys-soft-purging native-support-for-headers namespacing forcing-a-miss Xaa91438a7688016b77ed19f9bbe07127d969d9b varnish-broadcaster varnish-inventory issuing-a-purge bans-and-secondary-keys broadcast-groups summary-3 varnish-for-operations install-and-configure packages official-packages varnish-enterprise-packages distro-packages cloud-images amazon-web-services microsoft-azure google-cloud-platform oracle-cloud-infrastructure digitalocean official-docker-container kubernetes config-map-definition service-definition deployment-definition deploying-varnish-to-kubernetes configuring-varnish systemd editing-via-systemctl-edit docker port-configuration object-storage naming-storage-backends transient-storage file-storage mse not-using-a-vcl-file varnish-cli-configuration runtime-parameters-1 tls historically hitch installing-hitch configuring-hitch networking-settings certificate-settings protocol-settings tls-protocols proxy-protocol alpn-protocols cipher-settings ocsp-stapling what-is-ocsp-stapling ocsp-support-in-hitch mutual-tls vmod_proxy-1 native-tls-in-varnish-enterprise enabling-native-tls configuring-native-tls when-to-use-native-tls vmod_tls-2 backend-tls end-to-end massive-storage-engine history the-file-stevedore the-persistence-stevedore early-versions-of-mse architecture memory-vs-disk books stores the-danger-of-disk-fragmentation X05e877b891eb80292805ef0858768aa7fd2f04d making-sure-there-is-room-for-more X866480020770fe8a1d712b0a77946c4a718a43b memory-governor-1 debt-collection lucky-loser configuration memory-configuration persistence book-configuration store-configuration store-selection tagging-stores tagging-books setting-the-default-stores vmod_mse-2 monitoring memory-counters book-counters store-counters cache-warming load-balancing directors round-robin-director random-director fallback-director hash-director routing-through-two-layers-of-varnish self-routing-varnish-cluster key-remapping shard-director hash-selection warmup-and-rampup key-mapping-and-remapping least-connections-director dynamic-backends high-availability keeping-the-caches-hot vha leveraging-the-broadcaster architecture-1 workflow efficient-replication when-does-replication-take-place security installing-vha nodes.conf vcl configuring-vha broadcaster-settings origin-settings tls-1 limits skipping-replication forcing-an-update monitoring-1 logging-1 not-using-the-broadcaster discovery the-varnish-discovery-program installing-varnish-discovery configuring-varnish-discovery dns aws azure kubernetes-1 monitoring-2 varnishstat varnishstat-options other-output-formats curses-mode varnish-counters main-counters management-process-counters malloc-stevedore-counters backend-counters mse-counters kvstore-counters prometheus varnish-exporter telegraf setting-up-prometheus grafana varnish-custom-statistics vcs-metrics defining-keys the-vcs-agent the-vcs-server the-vcs-api the-vcs-user-interface when-things-go-wrong counters-we-want-as-low-as-possible debugging varnish-scoreboard logging-2 varnish-shared-memory-log transactions transaction-hierarchy transaction-grouping tags transaction-tags session-tags request-tags response-tags backend-tags backend-request-tags backend-response-tags object-tags vcl-tags the-timestamp-tag the-ttl-tag output-filtering tag-inclusion tag-exclusion tag-inclusion-by-regular-expression tag-exclusion-by-regular-expression filtering-by-request-type the-all-in-one-example vsl-queries record-selection-criteria operators-1 operands chaining-queries other-vsl-options processing-the-entire-buffer rate-limiting storing-and-replaying-logs varnishncsa logging-modes modifying-the-log-format extended-variables vsl-queries-1 other-varnishncsa-options log-rotation varnishtop running-varnishncsa-as-a-service why-wasnt-this-page-served-from-cache because-it-was-a-post-request because-the-request-contained-a-cookie X0c40acfe148ea3737d93fd1b13e4dad609aeeea Xc2c9011376e3e38d8d3ab6863c1217f8ab6786f why-wasnt-this-page-stored-in-cache zero-ttl-1 private-no-cache-no-store surrogate-control-no-store setting-a-cookie wildcard-variations the-significance-of-vsl security-1 firewalling cache-encryption encrypting-persisted-cache-objects performance-impact skipping-encryption choosing-an-alternate-encryption-cipher header-encryption jailing making-runtime-parameters-read-only vcl-security tls-2 cache-busting query-string-filtering max-connections backend-throttling slowloris-attacks web-application-firewall installing-the-varnish-waf tuning-varnish threading-settings growing-the-thread-pools shrinking-the-thread-pools client-side-timeouts backend-timeouts workspace-settings http-limits http-request-limit-examples http-response-limit-examples Xd1ad1c24dfb0eaa023d0852f829d4a8fe0c898f limiting-io-with-tmpfs other-settings listen-depth nuke-limit shortlived logging-cli-traffic-in-syslog the-varnish-cli backend-commands banning-1 parameter-management displaying-parameters setting-parameter-values vcl-management vcl-inspection loading-vcl vcl-labels vcl-temperature configuring-remote-cli-access the-cli-protocol the-cli-command-file quoting-pitfalls expansion heredoc the-varnish-controller architecture-2 domain vcl-1 deployment vcl-group agent setup authentication-authorization the-api the-cli the-gui summary-4 decision-making-on-the-edge dealing-with-state body-access request-body-access vmod_bodyaccess-1 xbody json.parse_req_body response-body-access xbody-revisited edgestash json-endpoint advanced-mustache-templating an-e-commerce-example sessions cacheability the-caching-solution the-vcl-code the-end-result http-calls prefetching link-prefetching video-prefetching api-calls authentication database-access sqlite key-value-storage-kvstore memcached redis a-shopping-cart-example geo-features vmod_geoip2 vmod_mmdb-2 lookup-filters backend-geotargeting-example synthetic-responses synthetic-output-and-no-backend loading-an-html-template creating-a-simple-api synthetic-backends authentication-1 basic-authentication ensuring-cacheability vmod_basicauth-1 hashed-passwords-inside-vmod_kvstore digest-authentication digest-authentication-exchange X63ea4b4c6052c14cb7df66231e67329e42851b0 json-web-tokens jwt-header jwt-payload jwt-signature vmod_jwt-2 oauth google-oauth-in-varnish summary-5 what-is-a-cdn network-connectivity caching request-routing why-build-your-own-cdn why-varnish request-coalescing backend-request-routing performance-and-throughput horizontal-scalability transparency varnish-cache-or-varnish-enterprise varnish-cdn-architecture edge-tier hardware-considerations vcl-example storage-tier hardware-considerations-1 vcl-example-1 origin-shield-tier caching-video ott-protocols hls mpeg-dash cmaf varnish-and-video vcl-tricks controlling-the-ttl prefetching-segments no-origin ad-injection request-routing-1 powerdns aws-route53 anycast varnish-traffic-router varnish-and-5g multi-access-edge-computing use-cases varnish-edge-cloud summary-6 closing-notes thank-you what-does-the-future-bring _GoBack more-information Chapter 1: What is Varnish? 1.1 What is Varnish? 1.2 What is VCL? 1.3 Varnish Cache and Varnish Enterprise 1.3.1 Version numbers 1.3.2 Product vs project 1.3.3 Which features does Varnish Cache have? 1.3.4 Which features does Varnish Enterprise have? 1.4 Which use cases does Varnish address? 1.4.1 API acceleration 1.4.2 Web acceleration 1.4.3 Private CDN 1.4.4 Video streaming acceleration 1.4.5 Web application firewalling 1.5 Under the hood 1.5.1 The manager process 1.5.2 The VCL compiler process Compilation steps 1.5.3 The child process 1.5.4 Threads The cache-main thread The thread pool herder thread The acceptor threads The waiter thread The expiry thread The backend-poller thread The ban-lurker thread Worker threads 1.5.5 Transports 1.5.6 Disembarking 1.5.7 The waiting list 1.5.8 Serialization 1.5.9 Workspaces 1.5.10 Backend fetches Streaming Varnish Fetch and Delivery Processors 1.6 Chapter summary Chapter 2: Varnish 6 2.1 Why Varnish 6? 2.1.1 A lot of old content out there 2.1.2 Varnish versions vs VCL syntax versions 2.1.2 Encouraging upgrades 2.1.3 It’s the way forward 2.2 What’s new in Varnish 6? 2.2.1 What’s new in Varnish 6.0? UNIX domain sockets (UDS) HTTP/2 support considered stable Other features in Varnish 6.0 2.2.2 What’s new in Varnish 6.1? 2.2.3 What’s new in Varnish 6.2? 2.2.4 What’s new in Varnish 6.3? Explicitly trigger vcl_backend_error VMOD import changes Behavior change in auto VCL temperature state Querying changes in VSL tools 2.2.5 What’s new in Varnish 6.4? if-range support Import vmod_cookie from varnish_modules Defining none backends Other VCL changes 2.2.6 What’s new in Varnish 6.5? Strict CIDR checks on ACLs vcc_acl_pedantic parameter obj.can_esi A new .resolve() method Closing the connection BLOB literal syntax std.blobread() No connection is made to a backend administratively set as unhealthy Help screen in varnishstat 2.2.7 What’s new in Varnish 6.6? Start Varnish without a backend Header validation Vary notices Checking ban errors Modulus operator New notation for long strings New built-in VCL VCL variable changes 2.2.8 Backports to 6.0 LTS 2.3 Varnish Enterprise 6 2.3.1 The origin story 2.3.3 New features in Varnish Enterprise 6 Total encryption and vmod_crypto vmod_urlplus The return of req.grace vmod_synthbackend MSE3 vmod_ykey Varnish High Availability 6 vmod_mmdb vmod_utils Explicitly return errors JSON formatting support in varnishncsa 2.3.3 vmod_str vmod_mse Last byte timeout If-Range support Built-in TLS support Memory governor vmod_jwt vmod_stale vmod_sqlite3 2.3.4 vmod_tls vmod_headerplus vmod_resolver Veribot 2.3.5 vmod_brotli 2.3.6 vmod_format 2.3.8 Features ported from Varnish Cache Plus 4.1 2.3.9 What happens when a new Varnish Cache version is released? 2.4 Where to get it 2.4.1 The official package repositories 2.4.2 Installing from source 2.4.3 Official Docker image 2.4.4 Official cloud images Varnish Enterprise features in the cloud Licensing and billing 2.5 Chapter summary Chapter 3: It’s all about HTTP 3.1 HTTP as the go-to protocol 3.1.1 The strengths of HTTP 3.1.2 The limitations of HTTP 3.1.3 Newer versions of the HTTP protocol HTTP/1.1 HTTP/2 HTTP/3.0 3.1.4 What about Varnish? HTTP/2 in Varnish HTTP/3 in Varnish 3.2 HTTP caching 3.2.1 The Expires header 3.2.2 The Cache-Control header max-age vs s-maxage Public vs private Deciding not to cache Revalidation How Varnish deals with Cache-Control 3.2.3 Surrogates The Surrogate-Capability header The Surrogate-Control header Surrogate caching Surrogate targeting Surrogate support in Varnish 3.2.4 TTL header precedence in Varnish 3.2.5 Cacheable request methods 3.3.6 Cacheable status codes 3.2.7 Cache variations The vary header Accept-Language variation example Hit-rate considerations Sanitizing user input Varying on custom headers 3.3 Varnish built-in VCL behavior 3.3.1 When is a request cacheable? Cacheable request methods Invalid request methods State getting in the way 3.3.2 How does Varnish identify objects in cache? 3.3.3 Dealing with stale content 3.3.4 When does Varnish store a response in cache? 3.3.5 What happens if the response couldn’t be stored in cache? 3.4 Range requests 3.4.1 Accept-Ranges response header 3.4.2 Range request header 3.4.3 Content-Range response header 3.4.4 What if the range request fails? 3.4.5 Range request support in Varnish Impact on the origin Backend range requests using VCL 3.5 Conditional requests 3.5.1 304 Not Modified 3.5.2 Etag: the fingerprint 3.5.3 If-None-Match 3.5.4 The workflow 3.5.5 Strong vs weak validation 3.5.6 Conditional request support in Varnish Conditional request workflow in Varnish Grace vs keep 3.5.7 Optimizing the origin for conditional requests Some context Exit early Leveraging Varnish 3.5.8 Last-Modified and If-Modified-Since as your backup plan 3.5.9 Conditional range requests 3.6 Compression 3.6.1 Content negotiation 3.6.2 Gzip compression in Varnish 3.6.3 Gzip and VCL 3.6.4 Brotli compression in Varnish 3.7 Content streaming 3.7.1 Chunked transfer encoding 3.7.2 Streaming support in Varnish 3.8 Summary Chapter 4: The Varnish Configuration Language 4.1 What is VCL again? 4.2 The finite state machine 4.2.1 The client-side flow 4.2.2 The backend flow 4.3 Hooks, subroutines, and built-in VCL 4.3.1 vcl_recv Error cases To pipe or not to pipe Only GET and HEAD Stateless Anything else gets cached 4.3.2 vcl_hash 4.3.3 vcl_hit A dirty little secret about vcl_hit 4.3.4 vcl_miss 4.3.5 vcl_purge 4.3.6 vcl_pass 4.3.7 vcl_pipe 4.3.8 vcl_synth 4.3.9 vcl_deliver 4.3.10 vcl_backend_fetch 4.3.11 vcl_backend_response Uncacheable Zero TTL A cookie was set Surrogate control Cache control says no Vary all the things 4.3.12 vcl_backend_error 4.3.13 vcl_init 4.3.14 vcl_fini 4.4 VCL syntax 4.4.1 VCL version declaration 4.4.2 Assigning values 4.4.3 Strings 4.4.4 Conditionals 4.4.5 Operators 4.4.6 Comments 4.4.7 Numbers 4.4.8 Booleans 4.4.9 Time & durations Time Duration 4.4.10 Regular expressions 4.4.11 Backends The basics Probes UNIX domain sockets Overriding the host header 4.4.12 Access control lists 4.4.13 Functions ban() hash_data() synthetic() regsub() A practical example A practical example 4.4.14 Subroutines 4.4.15 Include 4.4.16 Import 4.5 VCL objects and variables 4.5.1 Connection variables PROXY vs no PROXY The IP type Local variables Identities 4.5.2 Request variables A request example Top-level requests and Edge Side Includes 4.5.3 Backend request variables 4.5.3 Backend response variables VFP-related backend response variables Timing-related backend response variables Other backend response variables 4.5.4 Object variables 4.5.5 Response variables 4.5.1 Storage variables 4.6 Making changes 4.6.1 Excluding URL patterns 4.6.2 Sanitizing the URL Alphabetic sorting Removing tracking query string parameters Removing URL hashes Removing trailing question marks 4.6.3 Stripping off cookies Removing select cookies Removing all but some cookies Using vmod_cookie Using vmod_cookieplus 4.6.4 Sanitizing content negotiation headers 4.6.5 Overriding TTLs Static data example Overriding the default TTL Zero TTLs are evil 4.6.6 Dealing with websockets 4.6.7 Enabling ESI support Inspect the URL Inspect the Content-Type header Surrogate headers 4.6.8 Protocol detection Using vmod_proxy Using vmod_tls 4.6.9 VCL cache variations Protocol cache variations Language cache variations 4.6.10 Language cookie cache variation Using vmod_cookie Using vmod_cookieplus 4.6.11 Custom error messages The current built-in VCL implementation Customize error messages using templates 4.6.12 Caching objects on the second miss 4.7 Validation and testing 4.7.1 Syntax validation 4.7.2 Testing Built-in VCL test A failing test Looking at Varnish’s tests A VCL test 4.8 Summary Chapter 5: Varnish Modules (VMODs) 5.1 What’s a VMOD? 5.1.1 Scope and purpose 5.1.2 VMOD API 5.1.3 VCL usage 5.1.4 VMOD initialization 5.1.5 Installing a VMOD 5.2 Which VMODs are shipped with Varnish Cache? 5.2.1 vmod_blob 5.2.2 vmod_cookie 5.2.3 vmod_directors 5.2.4 vmod_proxy 5.2.5 vmod_std Logging String manipulation Environment variables Reading a file Server ports 5.2.6 vmod_unix 5.3 Which VMODs are shipped with Varnish Enterprise? 5.3.1 vmod_accept 5.3.2 vmod_aclplus Advanced ACLs A key-value store example 5.3.3 vmod_cookieplus Set-Cookie logic 5.3.4 vmod_crypto Hashing & encoding Encryption 5.3.5 vmod_deviceatlas 5.3.5 vmod_edgestash 5.3.7 vmod_file File backends Command line execution 5.3.8 vmod_format 5.3.9 vmod_json 5.3.10 vmod_goto The DNS backend The DNS director Extra options Dynamic backends example 5.3.11 vmod_headerplus 5.3.12 vmod_http 5.3.13 vmod_jwt 5.3.14 vmod_kvstore 5.3.15 vmod_mmdb 5.3.16 vmod_mse 5.3.17 vmod_resolver 5.3.18 vmod_rewrite Rewrite rules in VCL vmod_rewrite rulesets Rulesets as a string Matching URL patterns Extracting ruleset fields 5.3.19 vmod_sqlite3 5.3.20 vmod_stale 5.3.21 vmod_synthbackend 5.3.22 vmod_tls 5.3.23 vmod_urlplus 5.3.24 vmod_xbody 5.3.25 vmod_ykey 5.4 Where can you find other VMODs? 5.4.1 Third-party VMODs vmod_basicauth vmod_redis 5.4.2 The Varnish Software VMOD collection vmod_bodyaccess vmod_header vmod_tcp vmod_var vmod_vsthrottle vmod_xkey 5.4.3 How to install these VMODs Compiling from source Debian and Ubuntu distro packages 5.5 Writing your own VMODs 5.5.1 vmod_example 5.5.2 Turning vmod_example into vmod_os Dependencies Getting the code 5.5.3 Looking at the vmod_os.c Looking at the vmod_os.vcc Building the VMOD Testing the VMOD Using the VMOD 5.6 Summary Chapter 6: Invalidating the cache 6.1 Purging 6.1.1 Purge VCL code 6.1.2 Triggering a purge 6.1.3 vmod_purge Hard purge Soft purge 6.1.4 Integrating purge calls in your application 6.2 Banning 6.2.1 Ban expressions Expression format Expression examples 6.2.2 Executing a ban from the command line 6.2.3 Ban VCL code Purge replacement Invalidate URL patterns Complete flexibility The best of both worlds 6.2.4 The ban list There is always an item on the list Adding a first ban Adding multiple bans 6.2.5 The ban lurker Runtime parameters Ban lurker workflow Ban lurker scope 6.2.6 Enforcing asynchronous bans 6.2.7 Tag-based invalidation 6.2.8 Integrating bans in your application 6.2.9 Ban limitations 6.3 Secondary keys 6.3.1 vmod_xkey Initializing vmod_xkey Registering keys Invalidating content vmod_xkey limitations 6.3.2 vmod_ykey Why Ykey? vmod_ykey performance improvements Registering keys Invalidating content 6.4 Forcing a miss 6.5 Distributed invalidation with Varnish Broadcaster 6.5.1 Varnish Broadcaster 6.5.2 Varnish inventory 6.5.3 Issuing a purge 6.5.4 Bans and secondary keys 6.5.5 Broadcast groups 6.6 Summary Chapter 7: Varnish for operations 7.1 Install and configure 7.1.1 Packages Official packages Varnish Enterprise packages Distro packages 7.1.2 Cloud images Amazon Web Services Microsoft Azure Google Cloud Platform Oracle Cloud Infrastructure DigitalOcean 7.1.3 Official Docker container 7.1.4 Kubernetes Config map definition Service definition Deployment definition Deploying Varnish to Kubernetes 7.2 Configuring Varnish 7.2.1 Systemd Editing via systemctl edit 7.2.2 Docker 7.2.3 Port configuration 7.2.4 Object storage Naming storage backends Transient storage File storage MSE 7.2.5 Not using a VCL file 7.2.6 Varnish CLI configuration 7.2.7 Runtime parameters 7.3 TLS 7.3.1 Historically 7.3.2 Hitch Installing Hitch Configuring Hitch Networking settings Certificate settings Protocol settings Cipher settings OCSP stapling Mutual TLS 7.3.3 vmod_proxy 7.3.4 Native TLS in Varnish Enterprise Enabling native TLS Configuring native TLS When to use native TLS 7.3.5 vmod_tls 7.3.6 Backend TLS 7.3.7 End-to-end 7.4 Massive Storage Engine 7.4.1 History The file stevedore The persistence stevedore Early versions of MSE 7.4.2 Architecture Memory vs disk Books Stores The danger of disk fragmentation Problems with the traditional memory allocator Memory governor Lucky loser 7.4.3 Configuration Memory configuration Persistence Book configuration Store configuration 7.4.4 Store selection Tagging stores Tagging books Setting the default stores vmod_mse 7.4.5 Monitoring Memory counters Book counters Store counters 7.4.6 Cache warming 7.5 Load balancing 7.5.1 Directors 7.5.1 Round-robin director 7.5.1 Random director 7.5.1 Fallback director 7.5.1 Hash director Routing through two layers of Varnish Self-routing Varnish cluster Key remapping 7.5.1 Shard director Hash selection Warmup and ramp-up Key mapping and remapping 7.5.1 Least connections director 7.5.1 Dynamic backends 7.6 High Availability 7.6.1 Keeping the caches hot 7.6.1 VHA 7.6.1 Leveraging the broadcaster 7.6.1 Architecture Workflow Efficient replication When does replication take place? Security 7.6.1 Installing VHA nodes.conf VCL 7.6.1 Configuring VHA Broadcaster settings Origin settings TLS Limits Skipping replication Forcing an update 7.6.1 Monitoring 7.6.1 Logging 7.6.1 Not using the broadcaster 7.6.1 Discovery The varnish-discovery program Installing varnish-discovery Configuring varnish-discovery DNS AWS Azure Kubernetes 7.7 Monitoring 7.7.1 Varnishstat Varnishstat options Other output formats Curses mode 7.7.2 Varnish counters Main counters Management process counters Malloc stevedore counters Backend counters MSE counters KVStore counters 7.7.3 Prometheus Varnish Exporter Telegraf Setting up Prometheus Grafana 7.7.4 Varnish Custom Statistics VCS metrics Defining keys The VCS agent The VCS server The VCS API The VCS user interface 7.7.5 When things go wrong Counters we want as low as possible Debugging 7.7.6 Varnish scoreboard 7.8 Logging 7.8.1 Varnish Shared Memory Log 7.8.2 Transactions Transaction hierarchy Transaction grouping 7.8.3 Tags Transaction tags Session tags Request tags Response tags Backend tags Backend request tags Backend response tags Object tags VCL tags The timestamp tag The TTL tag 7.8.4 Output filtering Tag inclusion Tag exclusion Tag inclusion by regular expression Tag exclusion by regular expression Filtering by request type The all-in-one exa