Getting started with Varnish 2.0

Welcome!

Welcome to the wonders of Varnish 2.0. Hopefully you will be up and running in no-time after a quick walkthrough of this document.

Installing Varnish

You can download Varnish packages for the major Linux distributions, such as Redhat, Suse and Debian from the Varnish website. For other Linux distributions, FreeBSD and other supported platforms, Varnish must be compiled and installed from Sourceforge..

The sample installation in this walkthrough is based on a FreeBSD installation from source.

For installation instructions on other operating systems, please visit the website, where we currently have the following available:

Prerequisites

Make sure you have the following accessible when you are installing your copy of Varnish.

A working command-line knowledge of your chosen OS
A recent version of GCC (3.3.x or newer should be fine, 4.2.1 or newer recommended)
A POSIX-compatible make (GNU make is fine)
A release tarball from Sourceforge

Installing Varnish from source on FreeBSD

First , run configure. In most cases, the defaults are correct and you do not need to specify any command-line options, except perhaps—prefix. If you plan on hacking the Varnish sources, however, you will most likely want to turn on stricter error checks and dependency tracking:

$ ./configure --enable-debugging-symbols --enable-developer-warnings --enable-dependency-tracking

If you have gcc 4.2.0 or newer, add --enable-extra-warnings to get additional compile-time warnings
If you have a version of gcc with stack protection, add --enable-stack-protector to enable run-time stack smashing detection
If you’re trying to track down an elusive bug or a race condition, --enable-diagnostics may help, but it will reduce performance and increase the amount of log data generated
If configure completes without any errors, simply run make to compile Varnish and make install to install it.
Start Varnish by typing: varnishd -f /etc/varnish/default.vcl

Configuring your Varnish installation

The Varnish daemon is configured by using command line options and the powerful Varnish Configuration Language (VCL). The location of the default VCL configuration file and the command line options varies, depending on which platform you have installed Varnish.

Command line options

The command line options configure the basics of the Varnish daemon, like the listen address and port, which VCL script to use and the working directory. The following options can be set:

Command	Function	Comment
-a address:port	HTTP listen address and port	—
-b address:port	backend address and port	—
—	—	-b <hostname_or_ip>
—	—	-b '<hostname_or_ip>:<port_or_service>'
-d	debug	—
-f file	VCL script	The -f option points to the VCL script to use. If it is omitted, the -b option must be used to define a backend to use with the default configuration.
-F	Run in foreground	—
-h kind[,hashoptions]	Hash specification	—
—	—	-h simple_list. Do not use – for debug only
—	—	-h classic [default]
—	—	-h classic,<buckets>
-l bytesize	Size of shared memory log	—
-n dir	varnishd working directory	—
-P file	PID file	—
-p param=value	set parameter	—
-s kind[,storageoptions]	Backend storage specification	—
—	—	-s malloc
—	—	-s file [default: use /tmp]
—	—	-s file,<dir_or_file>
—	—	-s file,<dir_or_file>,<size>
—	—	-s file,<dir_or_file>,%lt;size>,<granularity>
-t	Default TTL	—
-T address:port	Telnet listen address and port	To enable the command-line management interface, the -T option must be used. This will enable telneting to the defined port to pass commands to the running Varnish daemon either using varnishadm or use telnet directly to the port to access the command-line interface.
-V	version	—
-w int[,int[,int]]	Number of worker threads	—
—	—	-w <fixed_count>
—	—	-w min,max
—	—	-w min,max,timeout [default: -w2,500,300]
-u user	Priviledge separation user id	—

VCL

VCL is a small, domain-specific language used to define request handling and document caching policies for the Varnish HTTP accelerator. Some of its features are:

Simple syntax, similar to that of Perl and C
Access to request-manipulation
Regular expressions for matching
User defined sub-routines
Access control lists

The VCL configuration mainly consists of the backend and ACL definitions, in addition to a number of special sub-routines which hook into the Varnish workflow. These sub-routines may inspect and manipulate HTTP headers and various other aspects of each request, and to a certain extent decide how the request should be handled.

The direction in the workflow is determined in each sub-routine by a given keyword.

Function	Description	Possible keywords
vcl_recv	Called after receiving a request. Decides how to serve the request	error, pass, pipe
vcl_pipe	Called after entering pipe mode.Creates a direct connection between the client and the backend, bypassing Varnish all together.	error, pipe
vcl_pass	Called after entering pass mode. Unlike pipe mode, only the current request bypasses Varnish. Subsequent requests for the same connection are handled normally.	error, pass
vcl_hash	Called when computing the hash key for an object.	hash
vcl_hit	Called after a cache hit.	error, pass, deliver
vcl_miss	Called after a cache miss.	error, pass, fetch
vcl_fetch	Called after a successful retrieval from the backend. An ‘insert’ will add the retrieved object in the cache and then continue to vcl_deliver	error, pass, insert
vcl_deliver	Called before the cached object is delivered to the client.	error, deliver
vcl_timeout	Called by the reaper thread shortly before an object expires in the cache ‘discard’ will discard the object and ‘fetch’ will retrieve a fresh copy	discard, fetch
vcl_discard	Called by the reaper thread when a cached object is about to be discarded due to expiration or space is running low	discard, keep

Varnish ships with a default configuration ( default.vcl in /etc/varnish ), so you won’t have to define all the subroutines yourself. This default file is set up to work straight out of the box after you have defined a backend. Please note that default subroutines will be invoked should the custom configuration not terminate with a keyword

Defining a backend and the use of directors

A backend can either be set by using the -b command line option, or by defining it in the VCL configuration file. A backend declaration in VCL is defined like this:

backend www {
	.host = "www.example.com";
	.port = "http";
}

The backend object can later be used to select a backend at request time:

if (req.http.host ~ "^example.com$") {
	set req.backend = www;
}

If there are several backends delivering the same content, they can be grouped together using a director declaration:

director www-director round-robin {
	{ .backend = www; }
	{ .backend = { .host = "www2.example.com"; .port = "http"; } }
}

A director will choose one of the defined backend depending on its policy. A ‘random’ director will choose a random backend, biased by a weight for each backend, and a ‘round-robin’ backend will choose a backend in a round robin fashion. The director object can be used in the same way as the backend object for selecting a backend:

if (req.http.host ~ "^(www.)\.example\.com$") {
	set req.backend = www-director;
}

Access Control Lists

An Access Control List (ACL) declaration creates and initializes a named access control list which can later be used to match client addresses:

acl local 
	{ 
		"localhost"; /* myself */ 
		"192.0.2.0"/24; /* and everyone on the local network */
		! "192.0.2.23"; /* except for the dialin router */ 
	}

To match an IP address against an ACL, use the match operator:

if (client.ip ~ local) { pipe; }

Examples

As mentioned, Varnish ships ships with a default set of subroutines which hook themselves up to what you define.These subroutines will be processed subsequently, provided they are not explicitly terminated in the custom VCL. Tuning is an important part of implementing a cache-solution, and a custom configuration will probably be needed. Here are some examples to get you going - also note that irc.linpro.no #varnish is a great place to get help and to discuss all things Varnish:

Selecting a backend based on the type of document

- this can be done with the regular expression matching operator.

sub vcl_recv {
	if (req.url ~ "\.(gif|jpg|swf|css|j)$")	{
		unset req.http.cookie;
		unset req.http.authenticate;
		set req.backend = b1;
	} else {
		set req.backend = b2;
	}
}

h3. Retrying with another backend if one backend reports a non-200 response.

sub vcl_recv {
	if (req.restarts == 0) {
		set req.backend = b1;
	} else {
		set req.backend = b2;
	}
}

sub vcl_fetch {
	if (obj.status != 200) {
		restart;
	}
}

Preventing search engines from populating the cache with old documents

This can be done by checking the user-agent header in the HTTP request.

sub vcl_miss {
	if (req.http.user-agent ~ "spider") {
		error 503 "Not presently in cache";
	}
}

For more examples, please visit our wiki at http://www.varnish-cache.org.

Varnish tools

Varnish has a set of command line tools and utilities to monitor and administer Varnish. These are:

varnishncsa: Displays the varnishd shared memory logs in Apache / NCSA combined log format
varnishlog: Reads and presents varnishd shared memory logs.
varnishstat: Displays statistics from a running varnishd instance.
varnishadm: Sends a command to the running varnishd instance.
varnishhist: Reads varnishd shared memory logs and presents a continuously updated histogram showing the distribution of the last N requests by their processing.
varnishtop: Reads varnishd shared memory logs and presents a continuously updated list of the most commonly occurring log entries.
varnishreplay: Parses varnish logs and attempts to reproduce the traffic.

For further information and example of usage, please refer to the man-pages.

Further documentation

The project web site is a good source for information about Varnish, with updated news, mailings lists, how-to's, troubleshooting tips, and more. The web site is located at http://www.varnish-cache.org. For on-line reference manual, man-pages exists for both the VCL language as well as all the Varnish command line tools.