Tag Archives: varnish cache

Varnish config for wordpress with ngx_pagespeed and wp-touch

This is the Varnish config I am using currently. It is working with wp-touch, pagespeed and wordpress and (bonus) deals with the pagespeed not allowing pages to cache. No time for pretty comments and explanations, here’s the code. I will answer questions, or come back and explain the code in comments – but it is pretty self explanatory.

backend default {
.host = "127.0.0.1";
.port = "80";
.first_byte_timeout = 300s;
}

sub generate_user_agent_based_key {
set req.http.default_ps_capability_list_for_large_screens = "LargeScreen.SkipUADependentOptimizations:";
set req.http.default_ps_capability_list_for_small_screens = "TinyScreen.SkipUADependentOptimizations:";

set req.http.PS-CapabilityList = req.http.default_ps_capability_list_for_large_screens;

# Lazyload
if (req.http.User-Agent ~ “(?i)Chrome/|Firefox/|MSIE |Safari”) {
set req.http.PS-CapabilityList = “ll,ii,dj:”;
}
# lazyload_images (ll), inline_images (ii), defer_javascript (dj), webp (jw) and lossless_webp (ws).
if (req.http.User-Agent ~
“(?i)Chrome/[2][3-9]+\.|Chrome/[[3-9][0-9]+\.|Chrome/[0-9]{3,}\.”) {
set req.http.PS-CapabilityList = “ll,ii,dj,jw,ws:”;
}
# odd ones
if (req.http.User-Agent ~ “(?i)Firefox/[1-2]\.|MSIE [5-8]\.|bot|Yahoo!|Ruby|RPT-HTTPClient|(Google \(\+https\:\/\/developers\.google\.com\/\+\/web\/snippet\/\))|Android|iPad|TouchPad|Silk-Accelerated|Kindle Fire”) {
set req.http.PS-CapabilityList = req.http.default_ps_capability_list_for_large_screens;
}
# mobile
if (req.http.User-Agent ~ “(?i)Mozilla.*Android.*Mobile*|iPhone|BlackBerry|Opera Mobi|Opera Mini|SymbianOS|UP.Browser|J-PHONE|Profile/MIDP|portalmmm|DoCoMo|Obigo|Galaxy Nexus|GT-I9300|GT-N7100|HTC One|Nexus [4|7|S]|Xoom|XT907”) {
set req.http.PS-CapabilityList = req.http.default_ps_capability_list_for_small_screens;
}
# Remove placeholder header values.
remove req.http.default_ps_capability_list_for_large_screens;
remove req.http.default_ps_capability_list_for_large_screens;
}

sub vcl_hash {
# Block 3: Use the PS-CapabilityList value for computing the hash.
hash_data(req.http.PS-CapabilityList);
}
# Block 3a: Define ACL for purge requests
acl purge {
# Purge requests are only allowed from localhost.
“localhost”;
“127.0.0.1”;
#Add your server IP to this list
}
# Block 3b: Issue purge when there is a cache hit for the purge request.
sub vcl_hit {
if (req.request == “PURGE”) {
purge;
error 200 “Purged.”;
}
}

# Block 3c: Issue a no-op purge when there is a cache miss for the purge
# request.
sub vcl_miss {
if (req.request == “PURGE”) {
purge;
error 200 “Purged.”;
}
}

sub vcl_recv {
call generate_user_agent_based_key;

set req.http.X-Forwarded-For = client.ip;
set req.http.Host = regsub(req.http.Host, “:[0-9]+”, “”);

# Block 3d: Verify the ACL for an incoming purge request and handle it.
if (req.request == “PURGE”) {
if (!client.ip ~ purge) {
error 405 “Not allowed.”;
}
return (lookup);
}
# Blocks which decide whether cache should be bypassed or not go here.

# Did not cache the admin and login pages
if (req.url ~ “/wp-(login|admin)”) {
return (pass);
}
// server1 must handle file uploads
if (req.url ~ “media-upload.php” || req.url ~ “file.php” || req.url ~ “async-upload.php”) {
return(pass);
}

// do not cache xmlrpc.php
if (req.url ~ “xmlrpc.php”) {
return(pass);
}

// strip cookies from xmlrpc
if (req.request == “GET” && req.url ~ “xmlrpc.php”){
remove req.http.cookie;return(pass);
}

# Remove the “has_js” cookie
set req.http.Cookie = regsuball(req.http.Cookie, “has_js=[^;]+(; )?”, “”);

# Remove any Google Analytics based cookies
set req.http.Cookie = regsuball(req.http.Cookie, “__utm.=[^;]+(; )?”, “”);

# Remove the Quant Capital cookies (added by some plugin, all __qca)
set req.http.Cookie = regsuball(req.http.Cookie, “__qc.=[^;]+(; )?”, “”);

# Remove the wp-settings-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, “wp-settings-1=[^;]+(; )?”, “”);

# Remove the wp-settings-time-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, “wp-settings-time-1=[^;]+(; )?”, “”);

# Remove the wp test cookie
set req.http.Cookie = regsuball(req.http.Cookie, “wordpress_test_cookie=[^;]+(; )?”, “”);

# Are there cookies left with only spaces or that are empty?
if (req.http.cookie ~ “^ *$”) {
unset req.http.cookie;
}

if (req.http.Accept-Encoding) {
# Do no compress compressed files…
if (req.url ~ “\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$”) {
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ “gzip”) {
set req.http.Accept-Encoding = “gzip”;
} elsif (req.http.Accept-Encoding ~ “deflate”) {
set req.http.Accept-Encoding = “deflate”;
} else {
remove req.http.Accept-Encoding;
}
}

# Cache the following files extensions
if (req.url ~ “\.(css|js|png|gif|jp(e)?g)”) {
unset req.http.cookie;
}

# Check the cookies for wordpress-specific items
if (req.http.Cookie ~ “wordpress_” || req.http.Cookie ~ “comment_”) {
return (pass);
}
if (!req.http.cookie) {
unset req.http.cookie;
}

# — End of WordPress specific configuration

# Did not cache HTTP authentication and HTTP Cookie
if (req.http.Authorization || req.http.Cookie) {
# Not cacheable by default
return (pass);
}

# Cache all others requests
return (lookup);

}

# Block 5b: Only cache responses to clients that support gzip. Most clients
# do, and the cache holds much more if it stores gzipped responses.
if (req.http.Accept-Encoding !~ “gzip”) {
return (pass);
}

# Block 6: Mark HTML uncacheable by caches beyond our control.
sub vcl_fetch {
# For static content related to the theme, strip all backend cookies
if (req.url ~ “\.(css|js|png|gif|jp(e?)g)”) {
unset beresp.http.cookie;
}

# A TTL of 30 minutes
set beresp.ttl = 1800s;

return (deliver);
}
# Block 7: Add a header for identifying cache hits/misses.
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = “HIT”;
} else {
set resp.http.X-Cache = “MISS”;
}
}

Configuring Varnish Cache for WordPress

Oh, so you installed Varnish, what good will it do, if most of your content is not cached? I went for the “Preparing Varnish/Wordpress? for a Slashdotting in 60 seconds or less… ” code provided on the Varnish site. Its rather ruthless, but I’m not particularly attached to seeing the logged in version of my websites that I want Varnished anyway. Few log in to them. Ruthless works for me. So here’s how to do it. Please note that there is a change of code since that sample was provided, which I have corrected below. Feel free to plug and play.

Edit your /etc/default/varnish file and edit the port and cache size in.

DAEMON_OPTS=”-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1gb”

Port is at 6081, which you change to 80. Set the storage size as per your needs. Don’t obsess over it. You can always edit if needed later.

Then, in /etc/varnish/default.vcl paste the following code.

backend default {    .host = "localhost";    .port = "8080";    .max_connections = 30;    .connect_timeout = 4.0s;    .first_byte_timeout = 600s;    .between_bytes_timeout = 600s;
}

# Drop any cookies sent to WordPress.        sub vcl_recv {                if (!(req.url ~ "homeschoolingindia.in|phpmyadmin|wp-(login|admin)")) {                        unset req.http.cookie;                }        }
# Drop any cookies WordPress tries to send back to the client.        sub vcl_fetch {                if (!(req.url ~ "phpmyadmin|wp-(login|admin)")) {                        unset beresp.http.set-cookie;                }        }

You don’t have to configure everything. What you don’t configure falls back on pretty decent defaults. What is being done here is that all cookies are dropped so that the page becomes cachable, unless you are accessing login or admin, where you need cookies to be able to access. I had some trouble with interaction on the front page. No admin option was available, since this was the production version I was seeing. Making comments was a problem.

The solution was rather simple. I installed the Discus plugin to handle comments. It formats them rather nicely, magages efficiently, integrates reasonably well with wordpress, adds features like likes and shares along with the oh so fabulous lists of mentions. However, the bestest part is that it is delivered through javascript, so it is totally functioning when the page is cached.

Other problems likely may be using any analytics software from the server end. Since most requests will not reach the server at all, there is no way for the server to record hits and so on. Again, javascript to the rescue. What do I say. In my opinion, google analytics works best for my needs anyway.

So, you understand the theme of the matter, basically, you are not going to be pulling any customized pages. Javascript being rendered in the browser, couldn’t care less if it were served from a html page or php. Your adsense will work, so will analytics. Some “link selling” plugins that rely on php may not register as active with your providers, but then you shouldn’t be selling I’ll not comment on the ethics of that…. This site, AamJanataWide Aware and Nisarga run like that.

However, this brings us to the site that won’t work. A site that users login and use. You got that right. BuddyPress. Homeschoolingindia.in is not getting the varnish treatment. I guess you could do it so that you use varnish with non logged in users, if you have a lot of non-member visitors. I found it simpler to leave it out.

Two possibilities. The first is to exclude it through varnish. Either by passing requests through for target domain or configuring Varnish per domain, and not for this one. Please to also remember to exclude it in the cookie killing settings. Many possibilities depending on what you want.

The second option is what I did, because it was simple and I had an extra IP from my provider. Configure all sites to be cached to answer on one IP on your backend port 8080 in this example and those not to be cached to answer on port 80 (the regular port) on the second IP. In the default.vcl, in the backend configuration, replace localhost with the IP address serving sites to be cached on port 8080. Done. Your buddypress is now happily guzzling scandalous amounts of resources, while your other wordpresses are playing static html. ;)

There is more, much more, but I find that this is adequate for basic configuration. Later, as you get used to the cache, you will be able to analyze what is happening, and filter in more and more hits to the cache and reduce server load further, but that is another how to in itself, if at all I have the competency to write it.

This should keep your server from crashing under whatever it is that guzzles up memory.

Please note that these are my learnings as I struggle to find out things. I am not a professional. Only a  person who wanted to build a website. I am cutting through the massive finding out missions I had to take and providing the results of that learning. No guarantees, though whatever I say here is working according to my server.

If you would like something more tweakable (and complicated), with load balancing and multiple servers for one site, etc. Try here

Note: This post is old. Soon, there is one more coming up with more nuanced settings, and alternative vcls.