nginx + a backend with a dynamic IP (e.g. AWS ELB)

Recently, I wrote about the dynamic resolution of upstream servers in nginx which was achieved by quite an intrusive patch to the core nginx module. The patch was invented a while ago and was working very well up until recent nginx versions were released.

With the release of nginx 1.10 it was noticed that the patch crashes some workers under heavy load and this was unacceptable for the production load, hence a new approach was implemented.

The beauty of the new solution is that it is non-intrusive and works with any services that communicate via sockets.

In a nutshell, I just looked at the problem from a little bit different angle after I defined the requirements:

Since we are not allowed to patch the application the only place we can meddle with the communication between nginx and the FastCGI server is the socket nginx connects to. Therefore, we need some kind of a proxy that would take requests from nginx, determine the FastCGI endpoint, and forward the requests to that endpoint.

Initially, I thought that I would use something like netcat or a similar tool for this, but then I found that systemd provides systemd-socket-proxyd binary which fits the purpose perfectly and could be configured to be socket activated.

Before we dive into the implementation details of this solution, let’s describe what we are going to do and how:

The nginx part is easy – we just need to replace the FastCGI server endpoint address (in our example below it was remote.php.backend.domain.tld.:9000) with our local socket (we are using a unix socket at /run/systemd-socket-proxyd/fastcgi.sock in this example):

upstream php {

    #server remote.php.backend.domain.tld.:9000;

    server  unix:/run/systemd-socket-proxyd/fastcgi.sock;

}

location ~ \.php$ {

    try_files   $uri = 404;

    fastcgi_pass    php;

    fastcgi_index   index.php;

    include     fastcgi.conf;

}

The next step is to define the socket activated proxy service. This generally requires creating two files in /etc/system/systemd directory: one for the socket and the other for the proxy itself. However, in this article we will go an extra mile and will define template units so the same configuration could be reused to launch multiple proxies using the same base templates.

The first template file is for the systemd service which will provide the proxy capability:

[Unit]

Description="Generic Socket Proxy (%I)"

Documentation=https://www.freedesktop.org/software/systemd/man/systemd-socket-proxyd.html

After=network.service



[Service]

EnvironmentFile=-/etc/sysconfig/systemd-socket-proxyd.%i

User=nobody

Group=nobody

OOMScoreAdjust=-1000

UMask=077

ExecStart=/usr/lib/systemd/systemd-socket-proxyd $TARGET

Restart=on-failure

PrivateTmp=true

PrivateDevices=true

#PrivateUsers=true

#ProtectSystem=strict

ProtectSystem=full

#ProtectKernelTunables=true

#ProtectControlGroups=true

#NoNewPrivileges=true

#ProtectKernelModules=true

#MemoryDenyWriteExecute=true

Depending on the version of your systemd manager you may be able to uncomment more lines than was shown in this example, which was tested on CentOS 7.3.1611.

Also, in the example template given above we are using OOMScoreAdjust=-1000 to protect this proxy service from being killed in the event the system is starving for memory – this may be something you do not need.

The second template file is for the socket unit that would trigger the activation of the proxy service when a request arrives on the socket:

[Unit]

Description="Socket for Generic Socket Proxy (%I)"

Documentation=https://www.freedesktop.org/software/systemd/man/systemd-socket-proxyd.html



[Socket]

ListenStream=/run/systemd-socket-proxyd/%i.sock

SocketUser=root

SocketGroup=root

SocketMode=0660

DirectoryMode=0711



[Install]

WantedBy=sockets.target

You may notice that the defaults are pretty strict:

These are safe and sane defaults and can be tweaked per instance that was instantiated using the template as shown later in this article.

Since our goal is to connect nginx to the backend FastCGI service we need to ensure that the proxy socket is read/write accessible to nginx’es workers, so we need to tweak the settings of the socket unit:

[Socket]

SocketGroup=nginx

Note how we extended the template for a specific named instance of the socket unit: we defined the systemd-socket-proxyd@fastcgi.socket.d sub-directory and put a drop-in configuration snippet there.

Now, to achieve our goal we need to specify the target endpoint for our proxy service (the named instance we spawn using the template):

[Service]

# Reset the ExecStart, so we could override it

ExecStart=

ExecStart=/usr/lib/systemd/systemd-socket-proxyd remote.php.backend.domain.tld.:9000

The mechanics behind extending the configuration is the same as for the socket unit, but here we overrode the ExecStart command to specify the target endpoint for the proxy.

OK, we are done with the configuration of systemd, so it would be a good time to reload the systemd daemon:

[root@localhost ~]# systemctl daemon-reload

If you run a system where SELinux is disabled (why?!) you don’t need to do anything additional and should be good, but if you are security conscious and want to ensure that you follow the least privilege principle, then read on :)

Unfortunately, systemd-socket-proxyd seems not to be used a lot by the community (most likely people are just unaware of it) so the tool has no dedicated policy attached to it in the targeted SELinux policy. I plan to push the change into the SELinux reference policy, but before I do we are going to use a custom loadable policy module.

To build a module you need the SELinux reference policy development framework installed on the instance you are building your policies (it can be the same instance you are running your proxy on, but I would advise to use a temporary VM for building/compiling policies since the only time you need that development stuff is when you are compiling the module from sources).

On CentOS, you can install all the necessary bits to build a loadable SELinux module by installing the selinux-policy-devel package using yum:

[root@localhost ~]# yum -y install selinux-policy-devel

Note, I skipped the output of the command since it does not provide any useful information for the purposes of this article.

Once the SELinux development framework is installed we can start designing our loadable policy for the systemd-socket-proxyd service.

Our policy will consist of two files:

Eventually, we will need to introduce the corresponding module interface support file too, but for the purposes of this article we should be fine with the automatically generated one.

The content of the systemd-socket-proxy.te file is listed below:

policy_module(systemd-socket-proxyd, 1.0)



## <desc>

##  <p>

##  Allow systemd-socket-proxyd to bind any port instead of one labelled

## with systemd_socket_proxyd_port_t.

##  </p>

## </desc>

gen_tunable(systemd_socket_proxyd_bind_any, false)



## <desc>

## <p>

## Allow systemd-socket-proxyd to connect to any port instead of

## labelled ones.

## </p>

## </desc>

gen_tunable(systemd_socket_proxyd_connect_any, false)



systemd_domain_template(systemd_socket_proxyd)



type systemd_socket_proxyd_unit_file_t;

systemd_unit_file(systemd_socket_proxyd_unit_file_t)



sysnet_dns_name_resolve(systemd_socket_proxyd_t)



# resolver

allow systemd_socket_proxyd_t self:unix_dgram_socket { create getopt setopt sendto read write };



# listener

type systemd_socket_proxyd_port_t;

corenet_port(systemd_socket_proxyd_port_t);

allow systemd_socket_proxyd_t self:tcp_socket accept;



tunable_policy(`!systemd_socket_proxyd_bind_any',`

 allow systemd_socket_proxyd_t systemd_socket_proxyd_port_t:tcp_socket name_bind;

')



tunable_policy(`systemd_socket_proxyd_bind_any',`

 corenet_tcp_bind_all_ports(systemd_socket_proxyd_t)

')



# target

tunable_policy(`!systemd_socket_proxyd_connect_any',`

 allow systemd_socket_proxyd_t port_type:tcp_socket name_connect;

')



tunable_policy(`systemd_socket_proxyd_connect_any',`

 corenet_tcp_connect_all_ports(systemd_socket_proxyd_t)

')



# consumer

allow daemon systemd_socket_proxyd_t:unix_stream_socket connectto;

This is still work in progress, but so far it works at least for my projects. Note that there are two SELinux booleans defined which affect the behaviour of the policy:

Now, to allow the proper transition into the systemd_socket_proxyd_t domain we need to label the systemd-socket-proxyd binary with the systemd_socket_proxyd_exec_t label.

The content of the systemd-socket-proxy.fc file that implements this behavior is as follows:

/(usr/lib|etc)/systemd/system/systemd-socket-proxyd\.service  gen_context(system_u:object_r:systemd_socket_proxyd_unit_file_t,s0)

/usr/lib/systemd/systemd-socket-proxyd -- gen_context(system_u:object_r:systemd_socket_proxyd_exec_t,s0)

At this point, we have everything we need to compile a policy module, so let’s just do it now:

[root@localhost ~]# ls -l

total 8

-rw-r--r--. 1 root root  235 Jan  5 05:21 systemd-socket-proxyd.fc

-rw-r--r--. 1 root root 1467 Jan  6 00:10 systemd-socket-proxyd.te

[root@localhost ~]# make -f /usr/share/selinux/devel/Makefile

Compiling targeted systemd-socket-proxyd module

/usr/bin/checkmodule:  loading policy configuration from tmp/systemd-socket-proxyd.tmp

/usr/bin/checkmodule:  policy configuration loaded

/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/systemd-socket-proxyd.mod

Creating targeted systemd-socket-proxyd.pp policy package

rm tmp/systemd-socket-proxyd.mod tmp/systemd-socket-proxyd.mod.fc

[root@localhost ~]# semodule -i systemd-socket-proxyd.pp

[root@localhost ~]# restorecon -v /usr/lib/systemd/systemd-socket-proxyd

restorecon reset /usr/lib/systemd/systemd-socket-proxyd context system_u:object_r:init_exec_t:s0->system_u:object_r:systemd_socket_proxyd_exec_t:s0

The semodule -i systemd-socket-proxyd.pp command has actually installed the module into the system, but if you were building the module on a different instance, then instead of installing the module you just need to grab the resulting systemd-socket-proxyd.pp file and transfer it to the target instance where proxy is going to be running and only apply the last two commands (semodule -i ... and restorecon).

We approached the time when we need to perform the pre-flight checks before launching our new service :). First, we need to check whether the file context was applied to the systemd-socket-proxyd binary:

[root@localhost ~]# ls -ldZ /usr/lib/systemd/systemd-socket-proxyd 

-rwxr-xr-x. root root system_u:object_r:systemd_socket_proxyd_exec_t:s0 /usr/lib/systemd/systemd-socket-proxyd

Looks good! Let’s start the socket unit and check the permissions set on the socket file:

[root@localhost ~]# systemctl start systemd-socket-proxyd@fastcgi.socket

[root@localhost ~]# ls -ldZ /run/systemd-socket-proxyd{,/fastcgi.sock}

drwx--x--x. root root  system_u:object_r:var_run_t:s0   /run/systemd-socket-proxyd

srw-rw----. root nginx system_u:object_r:var_run_t:s0   /run/systemd-socket-proxyd/fastcgi.sock

This also looks as expected. The next test would be to check that our proxy service is running with the desired set of permissions and in the correct SELinux domain:

[root@localhost ~]# socat -v unix-client:/run/systemd-socket-proxyd/fastcgi.sock stdin



< 2017/01/13 00:59:22.987083  length=1 from=0 to=0



[root@localhost ~]# ps uZ -C systemd-socket-proxyd

LABEL                           USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

system_u:system_r:systemd_socket_proxyd_t:s0 nobody 28643 0.0  0.0 86628 756 ? Ssl  00:59   0:00 /usr/lib/systemd/systemd-socket-proxyd 127.0.0.1:9000

[root@localhost ~]# cat /proc/28643/status 

Name: systemd-socket-

State: S (sleeping)

Tgid: 28643

Ngid: 0

Pid: 28643

PPid: 1

TracerPid: 0

Uid: 99 99 99 99

Gid: 99 99 99 99

FDSize: 64

Groups: 99 

[truncated]

The first socat command was needed to trigger the socket activation that resulted in the systemd-socket-proxyd.service being launched.

The second command confirmed that the service is running under the nobody user and within the systemd_socket_proxyd_t domain.

Finally, the third command confirmed that the privileges were properly dropped and there is no way the service could regain the escalated privileges back.

Looks like we are all, so let’s make our changes persistent. To achieve this we just need to enable the systemd-socket-proxyd@fastcgi.socket unit:

[root@localhost ~]# systemctl enable systemd-socket-proxyd@fastcgi.socket

Created symlink from /etc/systemd/system/sockets.target.wants/systemd-socket-proxyd@fastcgi.socket to /etc/systemd/system/systemd-socket-proxyd@.socket.

[root@localhost ~]# systemctl status systemd-socket-proxyd@fastcgi.socket | fgrep Loaded:

   Loaded: loaded (/etc/systemd/system/systemd-socket-proxyd@.socket; enabled; vendor preset: disabled)

Just to show that the configuration is working I set a simple lab up in a VM and followed this article with the only exception of the PHP/FPM location which I am running on the same VM:

[root@localhost ~]# cat /usr/share/nginx/html/test.php

<?php echo "This is the output from PHP\n" ?>

[root@localhost ~]# cat /etc/nginx/conf.d/php-upstream.conf 

upstream php {

    #server        remote.php.backend.domain.tld.:9000

    server         unix:/run/systemd-socket-proxyd/fastcgi.sock;

}

[root@localhost ~]# cat /etc/nginx/default.d/php.conf

location ~ \.php$ {

    try_files      $uri = 404;

    fastcgi_pass   php;

    fastcgi_index  index.php;

    include        fastcgi.conf;

}

[root@localhost ~]# cat /etc/systemd/system/systemd-socket-proxyd@fastcgi.service.d/fastcgi.conf

[Service]

# Reset the ExecStart, so we could override it

ExecStart=

ExecStart=/usr/lib/systemd/systemd-socket-proxyd 127.0.0.1:9000

[root@localhost ~]# telnet 0 80

Trying 0.0.0.0...

Connected to 0.

Escape character is '^]'.

GET /test.php HTTP/1.0



HTTP/1.1 200 OK

Server: nginx/1.10.2

Date: Fri, 13 Jan 2017 03:01:16 GMT

Content-Type: text/html

Connection: close

X-Powered-By: PHP/5.4.16



This is the output from PHP

Connection closed by foreign host.

Works as expected :).

There is one thing one needs to be aware of: when I started to work on this I discovered that systemd-socket-proxyd had a hard-coded limit for the number of connections set to 256 (I introduced the -c parameter to systemd-socket-proxyd, so one could dynamically set the limit, however it would take some time until this change is propagated to all major distros).

Also, it is worth it to mention that the provided configuration is not efficient if you are using a domain name for the target endpoint for the proxy, so if this is the case I would advise to run a local DNS caching service (e.g. dnsmasq) so you would not spend time on the DNS queries.

As always, I would appreciate any feedback you may have.