Category: varnish

First of all, check what ports nginx is currently listening on, you can do that with any of the following commands

netstat -tlpn| grep nginx
ss -tlpn| grep nginx

So, you probably found nginx listening to port 443 for SSL connections, and on port 80 for plain http….

On many occasions, you may want other application (Such as varnish or apache) to be listening on port 80, So nginx needs to move to another port, in this example, I am moving it to port 8080

Step 1: Go to sites available, there is the default site, and there are any other sites you added to nginX, open those config files that you will find in /etc/nginx/sites-available, search for 80, and replace it wherever it may occure with 8080 or any port of your choice, restrictions are the following, port numbers under 1024 will requier root privilages (So keep it above 1024), and the maximum port number is 65535, Also port 0 can not be used for http (Relevant to UDP though)

Here is my problem, I have a website, and in a directory in that website, I have a wordpress installation, and that installation opens correctly, loads all the images, css, js and any other files for a proper experience, only problem is, when you put this behind a varnish reverse proxy, and an nginx reverse proxy for SSL (https), the website design (theme) does not load, you only see the actual html page that was loaded, but all other elements are never fetched from the server, i actually sniffed the data and found that css, javascript, and images are never even requested !

So the short of this story, if you are having problems with page loading without the theme or design, and you have a similar setup, odds are the problem is with wordpress settings not with nginx or with varnish !

So a closer look at the page source reveals that the page was loaded with https, but the links to all the page resources are in HTTP ! why is that ? simple

when you open the website in SSL, your browser creates a secure connection with nginx (termination), nginx requests the page from varnish, which relays the page again to the server.

As far as the web server serving wordpress is concerned, this request came in http, not https ! so all the page resources should be in http right ? yes, this is what is happening, but what is the solution

I tried a few solutions, for example, i changed the wordpress address and site address to httpS, but wordpress is smart enough to use whatever protocol the user accessed and use it for all resources !

there are many solutions programmatically, which is something i avoid because i update wordpress, and don’t want to fix it every time i upgrade, so whatever solution i need has got to be in the only file that is never modified when upgrading wordpress, the config file

wordpress knows it is on SSL from the following two entries in the environment, the entries $_SERVER[‘HTTPS’] and $_SERVER[‘SERVER_PORT’], the proxy sends a hint that the user used https with the variable $_SERVER[‘HTTP_X_FORWARDED_PROTO’] in the request header, hence, adding the following code snippet somewhere in the beginning of the config file should deceive wordpress into thinking it has been accessed over https !

if($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https'){
   $_SERVER['HTTPS'] = 'on';
   $_SERVER['SERVER_PORT'] = 443;
 }

Hope this all works okay for you, if not please let me know in the comments and i would be more than glad to help

So, before we start, I assume you have already installed Varnish the casual way, and that you have made sure you do not have something else occupying port 80, if nginx for example is listening on port 80, try this post to switch it to a different port (Changing the default port 80 on nginx), if it is a different app, just follow that app’s instructions to change it’s port before asking varnish to listen on it.

To check that port 80 is not occupied and free to use, try the following command

netstat -anpe | grep "80" | grep "LISTEN"

If for example you want to know what ports nginx is listening to, try one of the following commands

netstat -tlpn| grep nginx
or
ss -tlpn| grep nginx

Within the results, check if any are using port 80, mind you, a service using port 8083 for example will show up, you need to see if anything is using port 80 , Now, install varnish with the following command.

apt-get install varnish

As you may have noticed, and probably the reason why you are here, varnish will not work !

this is somewhat of an old problem, since Debian moved to systemD back with the Debian 8 release (Jessie), instead of editing the file in /etc/default/varnish, you will need to create a file in /etc/systemd/system/ and name it varnish.service, the contents of such a file should look like the following, note that xxx.xxx.xxx.xxx is the IP varnish should be listening on, one of the IPs assigned to the machine running varnish.

So to run the following command

systemctl edit varnish.service

[Unit]
Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/6.1/ man:varnishd

[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a xxx.xxx.xxx.xxx:80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
ExecReload=/usr/share/varnish/varnishreload
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

[Install]
WantedBy=multi-user.target

Once you have added the file execute the following

systemctl daemon-reload
systemctl restart varnish

Should i let varnish cache in RAM or switch to disk ?

Note that the configuration file above uses RAM to cache the content, My recommendation is to use DISK (Disk is cached in ram in a more dynamic and more useful way utilizing all the ram you are not using, while keeping it available to any app that needs it), but that is just me…

To switch from RAM to File system, replace the following in the above as follows

"-s malloc,512m" becomes "-s file,/path/to/cahce/file.bin,100G"

I have a certain setup with a Varnish5 caching reverse proxy, with nginx to terminate SSL connections.

The problem with this setup is that wordpress can’t detect https, hence, it can not enforce it, nor will it link the CSS accordingly etc…, and if your blog’s address starts with https, you have a problem.

there is more than one solution that i will enumerate here, I chose to force all connections to come through https (The first), but there are others, the one i will present here (Which i did not use) resolves the problem and allows wordpress to detect whether we are on a secure connection or not.

And even though this post-article talks about wordpress, everything here can apply to any PHP application (Or even other applications written in different programming languages)

1- Enforce https for the whole website

The simplest way to solve the problem (Which i chose) is to redirect all traffic to https.

in the varnish script, implement the following

in the sub vcl_recv, implement the following

if (req.http.host ~ "^(www\.)?example\.com$") 
{
	if (req.http.host ~ "^(?i)example.com" || req.http.X-Forwarded-Proto !~ "(?i)https") {
		return (synth(750, ""));
	}
}

Now, the following section is the sub vcl_synth

sub vcl_synth {
    if (resp.status == 750) {
        set resp.status = 301;
        set resp.http.Location = "https://www.example.com" + req.url;
        return(deliver);
    }
}

Once the above is in place in the varnish script, You will need to tell wordpress that it’s all HTTPS (SSL)

We do that in the config file, the only file we know that does not change when we update

define('FORCE_SSL_ADMIN', true);
$_SERVER['HTTPS']='on';

Now, the above should do it if you don’t mind that your website only works in https, if you do want http to remain an option, then here is another solution

Start by creating a phpinfo() page to check whether you have the $_SERVER[‘HTTP_X_FORWARDED_PROTO‘] variable , if you do, your work is partially done, but you need to mind too things

the varnish server needs to only cache the https copy (because most browsers won’t allow mixed content these days, and if the css is linked to as http, it won’t display correctly), you can do that easily with something like the following in the vcl_hash area

if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }

And then, also in the wp-config file, add the following (probably almost anywhere)

if ( isset( $_SERVER['HTTP_X_FORWARDED_PROTO'] ) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
        $_SERVER['HTTPS']='on';

In a previous post, i posted how you can put varnish on a virtual machine to cache content for your web server, But there was one small thing that we did not deal with, The IP address in the logs and that the website can see is simply that of the varnish server, not the visitor’s IP, and if you write PHP like me, I use the IP in everything, from securing logins to limiting page views to limiting comments. So, here is how we can work around this problem

First, the mission is as follows, Varnish needs to append something in the request header, that apache needs to consider the Visitor’s IP

To make apache do that, there is already a module (mod-remoteip) to do this

1- Tool to check the IP of the visitor

Create a PHP script to find out if what we are doing actually worked.

<?php
print "Your IP is: {$_SERVER['REMOTE_ADDR']} <br />\n\n";
$headers = apache_request_headers();
foreach($headers as $xheader => $xvalue)
{
print "$xheader : $xvalue <br />\n"; }
?>

and upload it to the website, when accessing this from the internet, you should see the IP as the IP of the varnish server, not your IP (Like it should)

1- Playing with cpanel

First, we must download mod_remoteip from the cpanel website (See here)

Once downloaded, we will upload it to the cpanel server at

/var/cpanel/easy/apache/custom_opt_mods/

Once done, we will now extract it as we would extract any tar.gz file

cd /var/cpanel/easy/apache/custom_opt_mods

tar -C /var/cpanel/easy/apache/custom_opt_mods -xvf custom_opt_mod-remoteip.tgz

Now, this mod should appear in easy apache when rebuilding apache, so go ahead, visit

Home »Software »EasyApache 3

and on the Exhaustive Options List you should be able to see mod_remoteip, be sure to check it.

Now, before coming to configure it, We need to inform Varnish to send us those headers. So add this at the very beginning of the Varnish sub vcl_recv section

NOTE: It seems that for the latest varnish (VCL 4), you no longer need this, if you add this, you will get the header as ( X-Forwarded-For : xxx.105.60.194, xxx.105.60.194) meaning the same IP twice

if (req.restarts == 0) {
  if (req.http.X-Forwarded-For) {
    set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
  } else {
    set req.http.X-Forwarded-For = client.ip;
  }
}

Now, we need to tell apache, what the extra header’s name is (X-Forwarded-For) and what servers this header should be honored for (the varnish server IPs, currently displaying in the script we wrote right after Your IP is:) So start by editing

/usr/local/apache/conf/includes/pre_virtualhost_global.conf

And add this, the IP here is the varnish server IP

<IfModule mod_remoteip.c>
   RemoteIPHeader X-Forwarded-For
   RemoteIPInternalProxy xxx.172.13.208
</IfModule>

Or, Probably the more suitable

/usr/local/apache/conf/includes/pre_virtualhost_global.conf

And add this, the IP here is the varnish server IP

<IfModule mod_remoteip.c>
   RemoteIPHeader X-Forwarded-For
   RemoteIPTrustedProxy xxx.172.13.0/24
   RemoteIPTrustedProxy xxx.172.14.0/24
   RemoteIPTrustedProxy xxx.172.19.0/24
</IfModule>

Now, visit Home >> Restart Services on your cpanel server, and restart apache

There you have it, refresh that PHP script, and your IP address should appear.

Now, the apache logs will still log the Varnish server IP, to fix that, you need to modify the log section in your apache config, changing %h with %a, like so

#LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined