Recently, I wrote about the dynamic resolution of upstream servers in nginx which was achieved by quite an intrusive patch to the core nginx module. The patch was invented a while ago and was working very well up until recent nginx versions were released.
With the release of nginx 1.10 it was noticed that the patch crashes some workers under heavy load and this was unacceptable for the production load, hence a new approach was implemented.
The beauty of the new solution is that it is non-intrusive and works with any services that communicate via sockets.
In a nutshell, I just looked at the problem from a little bit different angle after I defined the requirements:
- nginx needs to delegate the requests to a FastCGI server over a socket
- we want to work with the standard packages provided by the distribution
- the FastCGI server could be on a dynamic IP address
Since we are not allowed to patch the application the only place we can meddle with the communication between nginx and the FastCGI server is the socket nginx connects to. Therefore, we need some kind of a proxy that would take requests from nginx, determine the FastCGI endpoint, and forward the requests to that endpoint.
Initially, I thought that I would use something like netcat or a similar tool
for this, but then I found that systemd provides
which fits the purpose perfectly and could be configured to be socket
Before we dive into the implementation details of this solution, let’s describe what we are going to do and how:
- nginx will be configured to talk to the locally bound socket - there are two options, actually: either a local TCP socket or a Unix socket
- a proxy service will be socket activated by a request coming from nginx to that local socket
- a proxy service should resolve the target and forward the request there
The nginx part is easy – we just need to replace the FastCGI server endpoint
address (in our example below it was
with our local socket (we are using a unix socket at
/run/systemd-socket-proxyd/fastcgi.sock in this example):
The next step is to define the socket activated proxy service. This generally
requires creating two files in
/etc/system/systemd directory: one for the
socket and the other for the proxy itself. However, in this article we will go
an extra mile and will define template units so the same configuration could be
reused to launch multiple proxies using the same base templates.
The first template file is for the systemd service which will provide the proxy capability:
Depending on the version of your
systemd manager you may be able to uncomment
more lines than was shown in this example, which was tested on CentOS 7.3.1611.
Also, in the example template given above we are using
protect this proxy service from being killed in the event the system is
starving for memory – this may be something you do not need.
The second template file is for the socket unit that would trigger the activation of the proxy service when a request arrives on the socket:
You may notice that the defaults are pretty strict:
- the socket is owned by root and only root is allowed to work with the socket;
- the directory permissions are set in such a way that the
/run/systemd-socket-proxyddirectory file list is not readable by anyone except root.
These are safe and sane defaults and can be tweaked per instance that was instantiated using the template as shown later in this article.
Since our goal is to connect nginx to the backend FastCGI service we need to ensure that the proxy socket is read/write accessible to nginx’es workers, so we need to tweak the settings of the socket unit:
Note how we extended the template for a specific named instance of the socket
unit: we defined the
email@example.com sub-directory and
put a drop-in configuration snippet there.
Now, to achieve our goal we need to specify the target endpoint for our proxy service (the named instance we spawn using the template):
The mechanics behind extending the configuration is the same as for the socket
unit, but here we overrode the
ExecStart command to specify the target endpoint
for the proxy.
OK, we are done with the configuration of
systemd, so it would be a good time
to reload the
If you run a system where SELinux is disabled (why?!) you don’t need to do anything additional and should be good, but if you are security conscious and want to ensure that you follow the least privilege principle, then read on :)
systemd-socket-proxyd seems not to be used a lot by the
community (most likely people are just unaware of it) so the tool has no
dedicated policy attached to it in the targeted SELinux policy. I plan to push
the change into the SELinux reference policy, but before I do we are going to
use a custom loadable policy module.
To build a module you need the SELinux reference policy development framework installed on the instance you are building your policies (it can be the same instance you are running your proxy on, but I would advise to use a temporary VM for building/compiling policies since the only time you need that development stuff is when you are compiling the module from sources).
On CentOS, you can install all the necessary bits to build a loadable SELinux
module by installing the
selinux-policy-devel package using
Note, I skipped the output of the command since it does not provide any useful information for the purposes of this article.
Once the SELinux development framework is installed we can start designing our
loadable policy for the
Our policy will consist of two files:
systemd-socket-proxyd.te– the type enforcement ruleset; and
systemd-socket-proxyd.fc– the file context ruleset.
Eventually, we will need to introduce the corresponding module interface support file too, but for the purposes of this article we should be fine with the automatically generated one.
The content of the systemd-socket-proxy.te file is listed below:
This is still work in progress, but so far it works at least for my projects. Note that there are two SELinux booleans defined which affect the behaviour of the policy:
allows to bind proxy to any TCP socket if set to true, otherwise the proxy would be able to connect to TCP ports labelled with
allows proxy to connect to any target TCP ports if set to true, otherwise the target is limited by the ports labelled with
Now, to allow the proper transition into the
systemd_socket_proxyd_t domain we
need to label the
systemd-socket-proxyd binary with the
The content of the systemd-socket-proxy.fc file that implements this behavior is as follows:
At this point, we have everything we need to compile a policy module, so let’s just do it now:
semodule -i systemd-socket-proxyd.pp command has actually installed the
module into the system, but if you were building the module on a different
instance, then instead of installing the module you just need to grab the
systemd-socket-proxyd.pp file and transfer it to the target instance
where proxy is going to be running and only apply the last two commands
semodule -i ... and
We approached the time when we need to perform the pre-flight checks before
launching our new service :). First, we need to check whether the file context
was applied to the
Looks good! Let’s start the socket unit and check the permissions set on the socket file:
This also looks as expected. The next test would be to check that our proxy service is running with the desired set of permissions and in the correct SELinux domain:
system_u:system_r:systemd_socket_proxyd_t:s0 nobody 28643 0.0 0.0 86628 756 ? Ssl 00:59 0:00 /usr/lib/systemd/systemd-socket-proxyd 127.0.0.1:9000
socat command was needed to trigger the socket activation that
resulted in the
systemd-socket-proxyd.service being launched.
The second command confirmed that the service is running under the nobody user
and within the
Finally, the third command confirmed that the privileges were properly dropped and there is no way the service could regain the escalated privileges back.
Looks like we are all, so let’s make our changes persistent. To achieve this we
just need to enable the
Created symlink from /firstname.lastname@example.org to /etc/systemd/system/systemd-socket-proxyd@.socket.
Just to show that the configuration is working I set a simple lab up in a VM and followed this article with the only exception of the PHP/FPM location which I am running on the same VM:
Works as expected :).
There is one thing one needs to be aware of: when I started to work on this I
systemd-socket-proxyd had a hard-coded limit for the number
of connections set to 256 (I introduced the
-c parameter to
systemd-socket-proxyd, so one could dynamically set the limit, however it
would take some time until this change is propagated to all major distros).
Also, it is worth it to mention that the provided configuration is not
efficient if you are using a domain name for the target endpoint for the proxy,
so if this is the case I would advise to run a local DNS caching service (e.g.
dnsmasq) so you would not spend time on the DNS queries.
As always, I would appreciate any feedback you may have.