Mind Drops

NordicTrack Commercial 1750 Treadmill

2026-07-17T14:00:00+10:00

My wife has convinced me to purchase on a treadmill. Since I am quite picky when it comes to purchasing anything that contains some kind of technology, so I went into a rabbit hole of researching on the state of modern treadmills. After reading a lot of articles, watching reviews on YouTube, and looking into the issues people are encountering with their treadmills, I chose to go ahead and order NordicTrack Commercial 1750.

What I found in my research is that no matter how much a treadmill costs, it basically consists of the following components:

A motor (we are only discussing the electric treadmills here). There is a variety of motors used in treadmills, but basically the industry standard seems to be a brushless motor with two to four horse power with more powerful being used on more premium models;
A deck that supports the user and on top of which the belt is running. Surprisingly, it turned out to be the cheapest and weakest part of even the most premium treadmill models. For example, the $4,000 treadmill such as NordicTrack Commercial 1750 uses an Medium Density Fibreboard (MDF) deck, which is basically compressed paper. For this reason, NordicTrack refuses to provide warranty for outdoor use: MDF absorbs water as a sponge;
A belt that is the primary working surface of the treadmill. This is where the user is running on. It seems that all existing treadmills settled on a layered vinyl-based belts with some fabric underneath that absorbs lubrication agent. The idea is that the belt is slides on top of the deck, so the lubrication in the fabric layer reduces the friction when there is user’s step that presses the belt against the deck;
A controller which is the “brain” of the machine. Usually, located next to the motor, it is quite a simple piece of electronics and usually controls the speed of the motor (or motors, if the model supports changing the treadmill inclination);
And finally, the control interface which allows the user to operate the machine. In low-end models, the interface is usually quite simplistic and consists of segmented LED displays and buttons for changing the speed, inclination, and program selectors. In more premium models, it is usually implemented as a touch screen with the addition of some physical buttons for controlling speed and inclination.

There is a myriad of quality of life options that can be present on a particular model. For example, the rails may have integrated heartbeat sensors or there could be cooling fans installed.

Anyhow, when I was choosing the treadmill, my key criteria were to have more premium materials for the construction of the frame and to have a convenient interface for the user. I also wanted to ensure that I am not purchasing some quite expensive equipment just to be then locked into the manufacturer’s subscription: it seems both NordicTrack and Pelaton are trying their best to hook you up on their respective memberships which are quite costly (around AUD$50 per month).

So, the NordicTrack Commercial 1750 has a touch screen that runs a custom Android build. My initial analysis shows that one can get access to the underlying system by following a procedure known publicly: on a welcome screen tap ten times on a non-interactive background, wait approximately 7 seconds, and tap ten times again. The interface will challenge you with a code to enter the privileged mode. The code can be acquired by visiting https://getresponsecode.com (you may want to download the page locally, in case the person who is hosting that page decides to shut it down – the logic of acquiring the code is implemented as the client Javascript, so you can have a local copy and it will work).

I was able to access the underlying system, so at least I should be able to do stuff other users have successfully achieved already, e.g. there is a nice Wiki on Reddit for maintaining the privileged mode on the device.

It seems that Nordic Unchained was a one off attempt by somebody to “own” their device and the archive does not have a definitive home. I downloaded it from the Reddit Wiki and looked at the installation script. Basically, all it does is: disables the system iFit apps using adb and then installs a bunch of Android Package Kits (APKs) which the author of the script wanted to have on their device (like Chrome, Hulu, etc.). I think, the most important part the script is doing can be summarised as follows:

adb shell pm disable-user --user 0 com.ifit.eru
adb shell pm disable-user --user 0 com.ifit.launcher
adb shell pm disable-user --user 0 com.ifit.standalone
...
adb shell cmd package set-home-activity com.teslacoilsw.launcher/com.teslacoilsw.launcher.NovaLauncher

So, the first three lines disable the system provided iFit apps, then a bunch of APKs (including iFit) are installed as a user. One of the APKs is for the alternative launcher (Nova Launcher in this case) so the last line above is registering that launcher as the default.

So, what’s next? I will most likely start the way Nordic Unchained did it, since my wife is using the treadmill every day and I can’t just disassemble and experiment with the stuff until I am happy with the results.

The ultimate goal is to have my own build of the firmware and to have the full control of the device. I also want to reverse-engineer how the metadata is incorporated in iFit’s training courses: I suspect they just add an additional track (like subtitles track) to their video containers and that track encodes speed and inclination, so when you are following the course the machine adjusts accordingly. If I can figure that out, I would be able to create my own “exercises”, e.g. using OpenStreetMap, etc.

There is a camera in the touch screen device, but at least in my machine it was attached upside down. I am still deciding whether I want to physically rotate it or do it virtually. In any case, it would be nice to have it for video calls while you are exercising :).

I will try to update my blog with the progress on this topic. Stay tuned!

Samsung CSC Change Using Modem Commands

2025-10-09T15:30:00+11:00

This is a brief brain dump, so I can recall it when needed.

Each time I get a new Samsung phone I have to figure out a way to change the Country Service Code (CSC) to a country that allows automated call recordings, for example, Indonesia (XID). Previously, approximately 5 years ago, it was quite straight forward: you needed to download the HOME_CSC image for the region you are interested in, ensure that it was for the same bootloader version, and flash it to the CSC partition.

However, something changed in recent years (and I was not paying attention to the developments in Samsung world). I just recently purchased Samsung ZFold 7 and the trick with flashing did not work, so I started to investigate. Firstly, I discovered that the modern Samsung devices come with a multi-CSC configuration, so everything is already pre-installed on the device – one just needs to change the setting to set the desired CSC for the device. Secondly, the articles on the net on the change of CSC were all pointing to the SamFW tool, but I do have an issue with using that tool: for starters, it is a Windows x86_64 tool and I don’t have any Intel based devices to run Windows x86_64. So, I started to research on how SamFW was doing the change of the CSC. According to what I gathered, the tool was using the Serial-over-USB to perform the change.

Below are the steps I took to successfuly change the CSC on my phone from XSA (Australia) to XID (Indonesia).

You need to enable the “Developer options” menu on your Samsung device: go to Settings -> About phone -> Software information and tap on the “Build number” line 7 times.
In the “Developer options” menu make sure that you enable the following options:

a) “Stay awake” – this is optional, but you need to ensure that the phone is awake and not locked while you are going to communicate with it via USB, so it is easier to turn this option on and once the changes are done to turn it back off;

b) “USB debugging” – I am not really sure whether this is strictly needed, but in my session it was enabled;

c) “3GPP AT commands” – This will enable the interface we are going to use to issue modem commands;

d) “Default USB configuration”, select “Transferring files” – this is important since we are going to land on the debug screen that would not allow us to access notifications, so there is no way to change the USB configuration on the fly. You may want to change to your preferred setting after the CSC change is applied, though.

Launch the dialler application and enter *#0*#, the screen will change to a white background with lots of square buttons for diagnostic purposes. I believe this step is essential to enable some of the proprietary commands in the modem interface.
Connect your phone to your computer. Mine is Apple Macbook M4, so it runs macOS. Yours may run Linux or Windows. It does not really matter, except with Windows you will need to install serial drivers for Samsung (e.g. if you install any official Samsung tools like KIES, you already should have the necessary drivers installed). For macOS and Linux, it is much easier – you don’t need to install anything, the Serial-over-USB should be supported out of the box.
Locate the modem endpoint on your operating system. On macOS, use ls -l /dev/cu.usbmodem* to locate it. On Linux, I would suggest to do dmesg | tail -30 after connecting the device, you should be able to see which /dev/tty* has been assigned for the Serial-over-USB. For Windows, check your Device Manager to see which COM* port was assigned.
Use your favourite communication program to connect to the endpoint. On macOS, I prefer to use cu, which is included with the operating system and my examples below will use it. On Linux, you have myriad of options, but I would use either screen or minicom. On Windows, I think, HyperTerminal is still included with the operating system by default, but last time I have seen Windows up close was years ago. Anyhow, you need to connect your communication program to the located endpoint, set the baudrate to 115200, and you should be all set.
Now is the magic :). Below is my terminal session on macOS with my comments after the snippet:
```
galaxy@galaxy ~ $ sudo -i            
Password:

root@galaxy ~ # ls -la /dev/cu.*                    
crw-rw-rw-  1 root  wheel  0x9000003 Aug 26 10:39 /dev/cu.Bluetooth-Incoming-Port
crw-rw-rw-  1 root  wheel  0x9000001 Aug 26 10:38 /dev/cu.debug-console
crw-rw-rw-  1 root  wheel  0x9000005 Oct  9 14:03 /dev/cu.usbmodemXXXXXXXXXXXX
Eris:~ root# cu -l /dev/cu.usbmodemXXXXXXXXXXXX -s 115200
Connected.
AT
OK
ATI
Manufacturer: SAMSUNG ELECTRONICS CORPORATION
Model: SM-F966B
Revision: F966BXXU4AYI1
SVN: 01
IMEI: IIIIIIIIIIIIIII
+GCAP: +CGSM

OK
AT+DUMPCTRL=1,0

+CME Error:PACM(AP),PROTECTED_NO_TOK

OK
AT+SWATD=0
[DR]: CHANGE TO DDEXE[ATD]: CHANGE TO DDEXE

OK
AT+ACTIVATE=0,0,0

+CME Error:PACM(AP),UNREGISTED

OK
AT+SWATD=1
[DR]: CHANGE TO ATD[ATD]: CHANGE TO ATD

OK
AT+PRECONFG=2,XID
+PRECONFG:2,OK

OK
AT+CFUN=1,1
+CFUN=1,1:NA

OK
~.

Disconnected.
root@galaxy ~ #
```
I used AT at the beginning of the session to confirm that the modem interface is active (I got an OK, so it was active). Then I used ATI to ensure that I am actually talking to the desired device (it was indeed SM-F966B).

From this point on, all commands I am using are AT+ and, moreover, they all come from the Application Processor (AP) module.

AT+DUMPCTRL=1,0 is needed to enable the following AT+SWATD= commands to work.

AT+ACTIVATE=0,0,0 is a weird one. There is not too much information on the net regarding this command, but it seems that every single occurrence is related to enabling something in the firmware to allow the privileged settings to be changed, like we do with the very next command.

AT+PRECONF=2,nnn is the primary command we want to execute. The command sets the CSC of the device, where nnn is a three letter CSC used by Samsung. In my case it was XID for Indonesia. You can get a list of all available/known CSC from SamFW (the site is really heavy on ads, though, so beware).

AT+CFUN=1,1 is supposed to reboot the device. In my case, it did not, so I rebooted it manually.
Once you rebooted your device, go to Settings -> About phone -> Software information and look at the “Service provider software version” section. Mine looks like the following (after the change):
```
XID/XID,XID/XSA,OPS
```
Somewhere on the net you could find the format of this string, but basically it means that I have a dual-SIM enabled device with XID (Indonesia) CSC enabled for both SIMs, that the custom CSC is XID, but factory one is XSA (Australia). In your case, you should see your selected CSC in that line, if everything worked out correctly.

This is basically it. Enjoy! :)

macOS builtin VPN client tricks

2022-11-25T13:33:00+11:00

I think everyone would agree that when it comes to the VPN configuration macOS builtin VPN client lacks any flexibility and its GUI exposes a fraction of supported features. This is one of the reasons why there are so many third party VPN clients for macOS in the world, I believe. However, I found that the majority of the time I can get away by using the standard macOS VPN client to connect to the desired VPN endpoint.

In this article I am going to describe my journey of configuring macOS VPN client to support Windscribe VPN, but the described approach applies to vast majority of the VPN service providers who support IKEv2.

XXX: This article is still work in progress, but I thought an example published is better than nothing at all.

I happen to be a customer of Windscribe VPN service.

The macOS client is terrible The native builtin VPN client is good enough for me. GUI configuration produces unworkable result.

Apple Configuration Profile Reference VPN Payload

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>PayloadContent</key>
    <array>
        <dict>
            <key>IKEv2</key>
            <dict>
                <key>AuthName</key>
                <string>@@LOGIN@@</string>
                <key>AuthPassword</key>
                <string>@@PASSWORD@@</string>
                <key>AuthenticationMethod</key>
                <string>None</string>
                <key>DeadPeerDetectionRate</key>
                <string>High</string>
                <key>ExtendedAuthEnabled</key>
                <integer>1</integer>
                <key>IKESecurityAssociationParameters</key>
                <dict>
                    <key>DiffieHellmanGroup</key>
                    <integer>20</integer>
                    <key>EncryptionAlgorithm</key>
                    <string>AES-256-GCM</string>
                </dict>
                <key>LocalIdentifier</key>
                <string></string>
                <key>RemoteAddress</key>
                <string>@@REMOTE_SERVER@@</string>
                <key>RemoteIdentifier</key>
                <string>@@REMOTE_SERVER@@</string>
            </dict>
            <key>PayloadDisplayName</key>
            <string>@@DISPLAY_NAME@@</string>
            <key>PayloadIdentifier</key>
            <string>com.apple.vpn.managed.@@UUID1@@</string>
            <key>PayloadType</key>
            <string>com.apple.vpn.managed</string>
            <key>PayloadUUID</key>
            <string>@@UUID1@@</string>
            <key>PayloadVersion</key>
            <integer>1</integer>
            <key>UserDefinedName</key>
            <string>@@DISPLAY_NAME@@</string>
            <key>VPNType</key>
            <string>IKEv2</string>
        </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>@@DISPLAY_NAME@@</string>
    <key>PayloadIdentifier</key>
    <string>VPN.@@UUID2@@</string>
    <key>PayloadType</key>
    <string>Configuration</string>
    <key>PayloadUUID</key>
    <string>@@UUID2@@</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict>
</plist>

Fixing "Content wider than screen"

2021-09-03T13:00:00+10:00

Recently, I started to get notifications from my Google Search Console that some pages have their content wider than screen and that impacts Mobile Usability. However, when I went to the affected pages and performed a test through “TEST LIVE PAGE” no Mobile Usability issues were found. Yet, the re-validation requests through Google Search Console were returning the error again and again.

I searched for possible root cause and suggested fixes on forums, but did not find any satisfactory approaches for investigating and fixing the issue. Below, is my process of hunting the root cause down, which resulted in resolving the issue of “Content wider than screen” in my Google Search Console.

Let me start by saying that the website the error was reported against does not have any visual issues on mobile devices: it renders fine, achieves 90+ in PageSpeed Insights, and, overall, is very standards aligned.

So, how one can find what Google Search does not like about their seemingly perfect website? In this particular case, I found Chrome DevTools to be an invaluable tool to investigate the issue, specifically its insight into how layers are composed. To open the DevTools one need to simply right-click anywhere on the page they are about to analyse and choose the “Inspect” item at the bottom of the context menu. Once we are inside the DevTools window, we need to the mobile view (by clicking on the “Toggle device toolbar” icon in the upper left corner of the DevTools window – all in all, we are investigating the Mobile Usability issue.

If you never used the Layers view of DevTools before, you can find that view by clicking on three vertical dots in the upper right corner of the DevTools window, then expand the “More tools” sub-menu, and select “Layers”. This will switch your DevTools to a view representing how different layers are rendered by the web-browser engine. The view is usually split in three distinctive panes: the layer hierarchy, the visual representation of layers, and the details pane for the currently selected layer. Depending on whether you have the WebGL support or not, the representation of layers may be unavailable. If it is unavailable, it would be a bit inconvenient, but not a big issue, since our primary working horse is the layer hierarchy pane.

We should start with expanding the top level layer container and check the dimensions next to the first child, which is #document. In my example, the #document layer has the size of 720 by 8811 pixels. These dimensions represent the size of the viewport (assuming that your website visually fits into the mobile view with no horizontal scroll bars). Looking at my example, we can immediately see an issue with the rendering: instead of a rectangle representing our page we only see a single line across the pane. The pane is using auto scaling to fit all rendered layers inside the pane. This means that some element (or elements) in out page are so wide or rendered so far away from the primary content that to represent all the layers Chrome needed to scale it down significantly.

Now that we know what is wrong with the page, how do we locate the offending element(s)? First, we need to look at other layers in the layer hierarchy pane: any layer that is bigger horizontally (or vertically) than our #document layer is the likely suspect for triggering the Mobile Usability error of content being wider than screen. In my case, the header layer is extremely wide: 3200060 pixels wide, to be exact – so, something inside the header container is responsible for this. It is time to switch to the Elements view of DevTools and interrogate the DOM tree. To make the search easier, once I located the header container, I recursively expanded all children of that container.

This allows me to select the topmost container (in my case, it is “header”), select “Computed” tab to see the position and dimensions of the selected element, and then by pressing the down arrow key on my keyboard walk the tree until I find the offending element. We start with the “header” element and, as you can see from the screenshot, its dimensions are reasonable and are smaller than the dimensions of the “body” element which represents our document.

Basically, any child element of the “body” container should be equal or smaller than the dimensions of the “body” container. In some extreme cases, it would require one to do some tedious maths each time you move down the tree, but in most cases skimming through will reveal a standout element like the label element I stumbled upon.

Despite that the size of this label element is just 40 by 40 pixels, its position is way to the left (by 1599980 pixels, to be precise). To understand why this element behaves like this we need to switch to the “Styles” tab to look at the associated CSS rules for the offending element. Sure enough, after a bit of reading through the styles, I located the corresponding rule that was pushing the element to the far left side of the universe.

Once you find the offending element, you need to understand why it was implemented like that and what goal was pursued by the author. When this is clear, you should be able to come up with an appropriate fix addressing the Mobile Usability issue while preserving the intended behaviour of the element. In my case, the author was me and this particular element was part of the UI. Since my website is responsive and tries to cater for different audiences (including people who are using screen readers), I needed to hide some text from the visual audiences (they are fine with icons) and present textual label to people who are using assistive technology like screen readers. Unfortunately, I followed one of the articles I found on the web, which suggested to hide elements by pushing them off the screen. Therefore, my fix in this instance was to cut the overflowing label off using overflow: hidden and white-space: nowrap. I also used opacity: 0 for the “contextual” class, which was used by some static, non-interactive elements on the page.

I hope this small tutorial helped you to hunt down the source of your Google Search Console errors related to the content being outside the viewport.

Truly Responsive Pagination

2020-06-14T20:07:00+10:00

Searching the web for an example of a fully responsive pagination widget I found that either people consider “responsive” to be something else or that it is a non-trivial problem to solve. In any case, I decided to have a crack at it myself.

My definition of a responsive pagination widget is pretty simple: such a widget should maintain its look and feel when the viewport (or a window) changes the size, yet the excessive information such as page numbers in the middle should be omitted if there is no space for them.

In this post I am going to describe my journey on the topic and the solution I ended up with.

My starting point was pretty simple – I had the following HTML markup structure that was representing the page numbers in a pagination widget:

<span class="paginator">
    <ul>
        <li><a href="#">&lt;</a></li>
        <li><a href="#">1</a></li>
        <li><a href="#">2</a></li>
        <li><a href="#">3</a></li>
        <li><a href="#">4</a></li>
        <li><a href="#">5</a></li>
        <li><a href="#">6</a></li>
        <li><a href="#">7</a></li>
        <li><a href="#">8</a></li>
        <li><a href="#">9</a></li>
        <li><a href="#">10</a></li>
        <li><a href="#">&gt;</a></li>
    </ul>
</span>

The goal was to create a fully responsive widget using pure CSS and HTML and to try to avoid messing up the HTML structure unnecessarily, since I needed to preserve the “cleanness” of navigation for screen readers (i.e. accessibility is important!).

After a bit of brainstorming I ended up with two candidates for the implementation of my idea:

the “old-school” approach of presenting a list as a line of inline-block elements floating next to each other;
the recently new trend in web design to use the flexbox model.

Since I did not use display: flex before, I decided to try the flexbox layout first. The idea was that I could use two flex rows which wrap when the width was shrinking. The challenge was that I wanted to implement two visual effects:

the currently selected page should be in the middle of the pagination widget and the wrapping of the rows should be adjacent to that selected page number;
it should be visually unambiguous that the ranges of off-screen pages were omitted.

After thinking a bit I realised that the above description demands to have three logically separate components: a part before the selected page number (I call it “head”), a part containing the selected page number (I call it “middle”), and a part following the selected page number (I call it “tail”). Hence, my HTML markup became:

<span class="paginator">
    <ul class="head">
        <li><a href="#">&lt;</a></li>
        <li><a href="#">1</a></li>
        <li><a href="#">2</a></li>
        <li><a href="#">3</a></li>
        <li><a href="#">4</a></li>
    </ul>
    <ul class="middle">
        <li><a href="#">5</a></li>
    </ul>
    <ul class="tail">
        <li><a href="#">6</a></li>
        <li><a href="#">7</a></li>
        <li><a href="#">8</a></li>
        <li><a href="#">9</a></li>
        <li><a href="#">10</a></li>
        <li><a href="#">&gt;</a></li>
    </ul>
</span>

Initially, I just laid out the lists in a line just to see how it looks and whether I like the mechanics of it, so I started with the following style definitions:

.paginator {
    /* width: 50%; */
    /* margin: 0 auto; */
    display: flex;
    white-space: nowrap;
    overflow: hidden;
    min-width: 15em;    /* 5 items 3em each */
}

ul {
    display: flex;
    position: relative;
    flex-flow: row wrap;
    list-style: none;
    margin: 0;
    padding: 0;
    line-height: 1.2;
    height: 3.2em;
    overflow: hidden;
    min-width: 6em;
}

ul li {
    display: block;
    width: 3em;
    height: 3em;
    line-height: 3;
}

ul li a {
    display: block;
    text-align: center;
    text-decoration: none;
}

ul.middle {
    min-width: 3em;
}

The above produced the following result (the example is wrapped in a resizable <div>, so if you are using a browser that supports it, like Chrome, you should be able to play with it right away):

There are two issues which are immediately manifesting themselves when you resize the widget:

The “tail” section wraps at the end truncating the list of the available pages and making it impossible to jump to the last page number in the list;
There is no visual guidance whatsoever that the page number were omitted.

The first issue is a difficult one since in the flexbox model we cannot easily control which side of the row is used for wrapping. However, if we adjust our HTML markup and leverage not so often used row-reverse value of the flexbox, then achieve the desired effect.

So, in the HTML markup I reversed the “tail” list:

<ul class="tail">
        <li><a href="#">&gt;</a></li>
        <li><a href="#">10</a></li>
        <li><a href="#">9</a></li>
        <li><a href="#">8</a></li>
        <li><a href="#">7</a></li>
        <li><a href="#">6</a></li>
    </ul>

and I adjusted the style of the “tail” list to apply row-reverse (which basically visually presents the list in the reverse order):

ul.tail {
    flex-flow: row-reverse wrap;
}

With the above changes applied the result looks much more closer to what we want:

Now, to tackle with the visual guidance for the omitted page numbers we can do something like the following: we can display dots in space when the page number is being wrapped. To do so we could use pseudo-elements for the “middle” list:

ul.middle::before {
    content: '';
    width: 300%;
    height: 1em;
    position: absolute;
    z-index: -1;
    border-top: dotted;
    left: -100%;
    top: 1.5em;
}

A few things to note: we are creating a pseudo-element which is three times wider than its parent, we position it absolutely, shift it 1/3 to the left and half way down. With the only border defined to be the top border we are getting a dotted line that goes through the middle of our “middle” list.

However, to make this work we also need to ensure that the border we created can extend beyond the box defined by the “middle” list, hence we need to set overflow to visible and let the border overflow the boundaries:

ul.middle {
    overflow: visible;  /* show the background dot line */
}

Finally, some additional cosmetic touches are required:

the list items have transparent background and the border line is visible behind the page numbers, so to address this we need to explicitly set the background property for list items;
to overlay our explicitly defined background on top of the border line we need to bump our list items up in the rendering order by setting z-index to be higher;
we need to handle a situation when the widget is compressed to its minimal size so there are just three page numbers: the first, the current, and the last page number – in this situation we would not have any space between list items, yet we need to indicate that the page ranges were omitted – we can do this by playing with min-width for the “head” and “tail” lists.

The above translates to the following styling rules:

ul li {
    z-index: 1;
    background: whitesmoke;
}

ul.head,
ul.tail {
    min-width: 6.5em;
}

OK, it is time to see the result of all these changes:

Although the result looks promising and is a workable solution, there is still a couple of issues left unsolved:

The biggest concern I have with this solution is that it requires reordering of the “tail” list and it is messing up the screen readers, therefore making it much harder for people with assistive technologies to use it;
I also discovered an edge case if the current page was adjacent to “head” (e.g. the second page) or to “tail” (e.g. the second to last page) the range dots would be still present in that case since they are regulated by that .5em space I left for the “head” and “tail” lists in the min-width attribute.

The first issue may be solved by introducing a shadow copy of the pagination widget that is designed specifically for the screen readers while making this visually pleasing widget hidden from the screen readers.

The second issue is much more puzzling and so far I could not find a pure HTML + CSS solution for it. It can be addressed by a bit of JavaScript, but I did not want to go that route for my blog. Most likely, since my blog is statically generated I will address it using the logic of the generator itself.

Still, I think the techniques I demonstrated here may be quite useful to some people who are designing pure HTML and CSS adaptive widgets.

Automating Static Website Deployment

2020-05-17T21:32:00+10:00

Table of Contents

Setting up GitHub Pages
Setting up the private, code repository
Configuring the GitHub Action for publishing

In this post I am going to document the steps I took to implement a fully automated deployment of my blog using GitHub Actions and GitHub Pages.

As always, I started my journey with the definition of what I really wanted to get at the end:

The website is published on GitHub pages

Since the website is static and all of its content can be easily downloaded using a web crawler (like wget --mirror https://website.tld) I was OK with exposing the structure in the public repository, which is what GitHub offers on a free plan.
The code to generate the website should be private

I do a lot of work on the SSG (which is Pelican in my case) itself: extend it with plug-ins that may contain API tokens to reach out to some third party APIs, hack the core code when I want to quickly test stuff, etc. – so, I really did not have any desire to publish publicly all the commotions I did in the background (sometimes I do more than a hundred commits per day just to experiment with different ideas I have).
There should be a valid history of changes in both repositories

Well, I would get the history on my private repository for free, since it is the core value of maintaining a repository in the VCS, but I also wanted to have clean history of changes to the content I publish publicly.

It would be a pleasant bonus if the changes in the public repository could refer back to the corresponding commit in the private repository.

One may say that to do what I set out to do I would need to subscribe for a paid account with GitHub since according to their help page GitHub Pages for private repositories are only available on the paid plans.

However, as I pointed out above, it does not make sense to hide the content of the actual static website, hence all I needed to do is to find a way how to “publish” the resulting artefact to the GitHub Pages repository, and, preferably, that “publishing” should happen on GitHub’s side.

Luckily for me, GitHub started to support GitHub Actions on the free plan some time ago and as long as it is not abused according to their terms and conditions, it is a perfect vehicle for what I am trying to do, in my opinion.

Setting up GitHub Pages

There are multiple howtos and tutorials on the Internet regarding how to set GitHub Pages up, including the official help section on this topic, so I will only elaborate on details where I did something specific for the purposes of achieving my goals.

There are different types of GitHub Pages:

user or organisation
per-project

The difference between two is subtle (the former requires a dedicated repository for your website, while the latter allows you to keep it in a branch of the existing repository), but for the purposes of this article I am assuming that we are working with the user level GitHub pages which are residing in the repository named “<username>.github.io” (where <username> is your GitHub user name) as per the official documentation.

A few caveats I found and spent some time solving after following the official documentation are listed below:

GitHub’s documentation assumes use of Jekyll for the site generation.

It is not obvious how to use a different SSG (like Pelican). As far as I understand, there are multiple triggers for GitHub to consider that the web site is in a “published” state, so just ignore any references to Jekyll in the documentation: you will trip one of the triggers sooner or later, for example by pushing HTML files into your repository.
Configure your DNS before setting the custom domain name in GitHub Pages.

Pushing the CNAME file with the name of your custom domain within will trigger a DNS check from GitHub to see that your custom domain name is pointing back to GitHub Pages.

DNS heavily relies on caching and is depending on the TTL settings in your zone: if a negative check is performed (that is, when GitHub fails to retrieve the corresponding record) you will likely need to wait for quite a while for GitHub to retry.

Setting up the CNAME record in advance and then verifying it with a query before you commit the CNAME file to your repository ensures that you will get the quickest validation response from GitHub, e.g. I set up my CNAME records and then verified it from the command line (before) submitting the request to GitHub:
```
[user@localhost ~]$ host -t cname dmitry.khlebnikov.net 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

dmitry.khlebnikov.net is an alias for galaxy4public.github.io.
```
There are some shenanigans with the “Enforce HTTPS” option.

It is not obvious from the documentation, but the enforcement of HTTPS for custom domains on GitHub’s side is dependent on the several things:
- before the checkbox is enabled your custom domain name should be confirmed by GitHub (your CNAME file is in place and the repository settings show that the name was recognised);
- the CNAME record should point to your “<username>.github.io.” DNS record (or, you can point it directly to GitHub Pages IP addresses if you want to conceal the repository name in the DNS output);
- if GitHub did not like something and you adjusted anything in the above dot points the only way to trigger the enforcement of HTTPS is to re-submit the CNAME file to the repository (yes, you read it right: you need to delete the file and push it to the repository again);
- Removing the CNAME file from the repository is a disruptive action – the site will not be accessible for the duration of the file being missing.

OK, you have your public repository configured the way you want, so let’s look at the settings we need to be able to publish our code to this public repository.

When I try to automate something, I usually start with writing down manual steps I would do to achieve the results. This helps me to see patterns and to understand what I can easily automate and what will require some brain-storming to resolve.

In the case of updating the repository it is quite trivial: if I were to push updates manually, all I need is a private SSH key with the corresponding public SSH key configured with write privileges for the repository and I could push with git push from my local copy of the repository.

… private keys are called “private” for a reason – they are not supposed to leave the device under any circumstances. […] please pay attention when you read of hear somebody advising you to upload your private keys somewhere, it is usually bad advice.

My private keys are called “private” for a reason – they are not supposed to leave my device(s) under any circumstances (except for backup purposes such as storing them in a safe). So, please pay attention when you read or hear somebody advising you to upload your private keys somewhere, it is usually bad advice.

For the integration purposes, GitHub provides so-called “Deploy keys” and “Personal access tokens”. The former is just an SSH key pair associated with a particular repository (you can configure it in repository’s setting) while the latter is an OAuth access token associated with your account.

While you can successfully use both, I would recommend to use the “Deploy keys” only: despite that you can try to scope access down for a personal token, it would not be good enough and the actions performed using that token will look like you are executing them.

To configure a “Deploy key” we need two things:

Generate an SSH key pair, e.g.:

[user@localhost ~]$ ssh-keygen -t ed25519 -N '' -C 'Updating the blog from GH Action' -f ~/gh-action
Generating public/private ed25519 key pair.
Your identification has been saved in /home/user/gh-action
Your public key has been saved in /home/user/gh-action.pub
The key fingerprint is:
SHA256:7W9pUV5IlrRVE0RAVkjLgKJz4RdtVbH7GKQu8AfYITw Updating the blog from GH Action
The key's randomart image is:
+--[ED25519 256]--+
|          o.+BOX*|
|       + o o+.=oo|
|      o E +  =oo |
|     o o B . oo o|
|      o S + .o.o |
|         + o. .o.|
|          + oo. .|
|           ++    |
|           o.    |
+----[SHA256]-----+

Here, I chose the ed25519 key type since it is the shortest from the GitHub supported key types at the moment, yet it is strong enough.

I also made the key pair passphrase-less (-N '') since the purpose of the key pair is to automate things in the unattended fashion and there will be nobody to type in the passphrase.

The key pair comment just makes it easier to maintain your keys, but is optional.

Finally, the -f ~/gh-action option specifies where the generated private key is going to be stored. The public counterpart will use the same path with the .pub suffix appended to it.

Set the newly generated public key up as the “Deploy key”:

All you need to do is to go to the repository settings page for the public repository you created for GitHub Pages, click on “Deploy keys” in the left side menu, then click on the “Add deploy key” button in the upper right corner.

On the next page, provide a sensible description for the deploy key (I used the same text as I put into the keys comment, i.e. “Updating the blog from GH Action”) and copy and paste the recently generated public key. GitHub does not allow you to upload files over there, so you need to copy the content of the public key file and paste it into the form, e.g.:
```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGHCy+stVCBjsrVO2ld1DwKCwcKL9+i1sjxcZu4u4lFQ Updating the blog from GH Action
```
NOTE: You need to ensure that you tick the “Allow write access” checkbox, otherwise it would not be possible to push to the repository with the corresponding private key.

This, actually, concludes the configuration of the GitHub Pages repository for now – in later articles I will document how one could leverage the repository Issues for managing comments on the web site and maintain the counters for likes on the pages, but it would be a completely separate post :).

Setting up the private, code repository

A typical Pelican repository layout is quite simple and comprises one mandatory directory, one semi-mandatory file, and everything else is optional, but could be used to enhance your experience.

The mandatory directory is the so-called “content” directory (in Pelican’s terms). The name of the directory can be anything you want, but it is better be reflected in the PATH = directive of the setting file.

I am saying “better be” since Pelican can operate without any configuration files, but the result will be limited, hence I call the pelicanconf.py file (which is the default name for the configuration file) to be “semi-mandatory”. The name of the configuration file can be also anything you like, however, I suggest to stick with the default for now.

Basically, you can quickly start by following the Pelican documentation and doing something as follows:

[user@localhost ~]$ virtualenv ~/venv/pelican
created virtual environment CPython3.8.2.final.0-64 in 477ms
  creator CPython3Posix(dest=/home/user/venv/pelican, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/user/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[user@localhost ~]$ . ~/venv/pelican/bin/activate
(pelican) $ mkdir ~/blog
(pelican) $ cd ~/blog
(pelican) $ git init
Initialized empty Git repository in /home/user/blog/.git/
(pelican) $ git config --local user.email "your@github-email.here"
(pelican) $ git config --local user.name "Joe Happy"
(pelican) $ pelican-quickstart 
Welcome to pelican-quickstart v4.2.0.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.


> Where do you want to create your new web site? [.] 
> What will be the title of this web site? My Awesome Blog
> Who will be the author of this web site? Joe Happy
> What will be the default language of this web site? [en] 
> Do you want to specify a URL prefix? e.g., https://example.com   (Y/n) n
> Do you want to enable article pagination? (Y/n) 
> How many articles per page do you want? [10] 
> What is your time zone? [Europe/Paris] Australia/Melbourne
> Do you want to generate a tasks.py/Makefile to automate generation and publishing? (Y/n) n
Done. Your new project is available at /home/user/blog
(pelican) $ ls -l
total 16
drwxr-xr-x 2   user   user 4096 May 10 00:31 content
drwxr-xr-x 2   user   user 4096 May 10 00:31 output
-rw-r--r-- 1   user   user  869 May 10 00:31 pelicanconf.py
-rw-r--r-- 1   user   user  589 May 10 00:31 publishconf.py
(pelican) $ rm -rf output publishconf.py
(pelican) $ git add content pelicanconf.py 
(pelican) $ git commit -m 'Initial commit'
[master (root-commit) f077002] Initial commit
 1 file changed, 35 insertions(+)
 create mode 100644 pelicanconf.py

A short break down of the above session snippet is:

On line 1 we create a virtual Python environment, so we could install Pelican locally;
We enter the newly created virtual environment on line 2, which makes Pelican available to us;
We create an empty repository (~/blog) and initialise it using Pelican’s quickstart;
Since we are not using the default publishing capabilities and we are not interested in storing the generated pages in our code repository, we clean things up a bit;
Finally, we commit the generated skeleton to Git.

A good test at this stage would be to ensure that Pelican is working and likes our structure:

[user@localhost ~]$ pelican
WARNING: No valid files found in content for the active readers:
  | BaseReader (static)
  | HTMLReader (htm, html)
  | MarkdownReader (md, markdown, mkd, mdown)
  | RstReader (rst)
Done: Processed 0 articles, 0 drafts, 0 pages, 0 hidden pages and 0 draft pages in 0.07 seconds.

So far so good, but it is not a real test, since there are no source files to generate something from, so let’s give Pelican something to work on:

[user@localhost ~]$ printf 'Title: First Post\nDate: 2020-05-10\n\n#First post\nPelican is awesome!' > content/first.md
[user@localhost ~]$ pelican
Done: Processed 1 article, 0 drafts, 0 pages, 0 hidden pages and 0 draft pages in 0.12 seconds.
[user@localhost ~]$ elinks -dump output/index.html 
                               [1]My Awesome Blog

     • [2]misc

                                 [3]First Post

   Published: Sun 10 May 2020

    By [4]Joe Happy

   In [5]misc.

                                   First post

   Pelican is awesome!

The output from elinks was truncated on purpose since I just wanted to showcase that Pelican has indeed generated the structure for a static website from just one article file we created.

Before we push our local repository to GitHub we may want to do some house keeping first, e.g. create the .gitignore file and list the temporary things we do not want Git to track. A good enough version of the .gitignore file I am using for my code repository is the following:

*~
*.pyc
.*.swp

**/__pycache__

/output

Do not forget to actually commit that .gitignore file to your local repository using the git add .gitignore && git commit -m 'Added .gitignore', by the way.

Now, we need to create a private repository on GitHub, so jump into your browser, go to your GitHub account, press the “+” icon in the upper right corner (right next to your profile icon), and select “New repository”.

On the “Create repository” page put whatever you desire as the name and the description of the repository you are about to create. Ensure that the “Private” radio button is selected and uncheck the “Initialize this repository with a README” if it was checked.

Once the repository is created, you will be presented with a page that enumerates your options for the next step, but I will just go ahead and show a session dump of what you will need to do. In the following session snippet blog is the repository name I chose for my private code repository and you will need to replace it with your private repository name (the working directory is our newly created local repository):

[user@localhost ~]$ git remote add origin git@github.com:galaxy4public/blog.git
[user@localhost ~]$ git push -u origin master
Enter passphrase for key '/home/user/.ssh/keys/github': 
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 4 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 1008 bytes | 1008.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:galaxy4public/blog.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

Do you remember how we generated a deploy key pair earlier and installed the public key part into the public blog repository, so GitHub would allow the bearer of the private key to authenticate and deploy changes to the public blog repository? Well, since the purpose of this article is to introduce the full automation, the bearer of the key would be the GitHub Action associated with the private repository, hence we need to provide the action with the private key somehow.

GitHub has a feature called “repository secrets” and it is a perfect candidate to pass the private key to the GitHub Action. We need to follow the official documentation for the feature and create a secret called “DEPLOY_KEY” with the content of the private part of the deploy key. This will be used in the last step of the GitHub Action we are about to define.

Configuring the GitHub Action for publishing

Everything is well and good, but “where is the automation?” you may ask. After all, I suspect this was the primary reason you are reading this post. Well, we are about to start to look into the automation part and it is rather short in comparison to all the steps we did to set repositories up.

Our automation relies on the GitHub Action feature of GitHub. In plain terms, GitHub Action is a free compute resource provided by GitHub (there are some limits, but for the purposes of a personal blog it is unlikely that you will ever hit these limits).

Each GitHub Action is associated with a specific repository and is defined using quite a simple YAML configuration file which instructs GitHub on how to provision a required compute environment and what to run inside that environment. The YAML file can be arbitrarily named and resides in the .github/workflows/ subdirectory (starting from the root of the corresponding repository).

The GitHub Action I am using for my blog web site is stored in .github/workflows/pelican.yml and contains the following (we will dissect it further down the post):

name: Static Website Generator

on:
  push:
    branches: [ master ]

env:
  LANG: en_AU.UTF-8

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - name: Initialise locale
      run: |
        if [ "$LANG" != 'C' -a "{$LANG:0:2}" != 'C.' ]; then
          CP="${LANG#*.}" && [ -z "$CP" ] && CP=UTF-8 ||:
          sudo sed -i -E "/^\s*$LANG(\s|\$)/{:a;n;ba;q};\$a$LANG $CP" /etc/locale.gen
          sudo locale-gen
          sudo localectl set-locale LANG="${LANG:-C.UTF-8}"
        fi
        locale -a
    - name: Checkout the primary repo
      uses: actions/checkout@v2
      with:
        fetch-depth: 0
        submodules: recursive
    - name: Restore modification times for content
      run: |
        git log --pretty=tformat:"%at" --name-status --no-merges -- \
            content \
            themes/mind-drops/content \
        | sed -nE '
            /^\s*$/d;/^[[:digit:]]+$/{h;d};
            /^[UXB]/d;
            /^[AMT]/{s,^\S\s+,,;G;s,^(.+)\n(.+),\2 \1,;p};
            /^[DR]/{
              s,^(D|[CR][[:digit:]]+)\s+,\1 ,;G;
              s,^(\S+) ([^[=\t=]]+[[=\t=]])?(.*)\n(.+),\4\1 \3,;
              p
            }' \
        | LC_ALL=C sort -k2 -k1rn | uniq -f1 \
        | sed -E '/^[[:digit:]]+D /d' \
        | while read TSTAMP FILE; do
            if [ -f "$FILE" ]; then
                echo "$FILE => $(date -d @$TSTAMP)"
                touch -m -d "@$TSTAMP" -- "$FILE"
            fi
        done
    - name: Checkout Pages repo
      uses: actions/checkout@v2
      with:
        repository: galaxy4public/galaxy4public.github.io
        path: output
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: 3.x
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Generate the website
      run: |
        ls -laR content/
        rm -rf output/*
        ls -la output/
        TIMEZONE=$(sed -nE 's|^\s*TIMEZONE\s*=\s*['"'"'"]([^'"'"'"]+)['"'"'"].*|\1|;T;p' pelicanconf.py)
        TZ="${TIMEZONE:-UTC}" pelican
    - name: Publish to GitHub Pages
      env:
        DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
      run: |
        cd output
        git config --local user.email "action@github.com"
        git config --local user.name "GitHub Action"
        if output=$(git status --porcelain) && [ -z "$output" ]; then
          echo "No new content was generated, exiting gracefully"
        else 
          git add -A
          git commit -m "Updated content on $(date)"
          eval $(ssh-agent)
          echo "$DEPLOY_KEY" | ssh-add -t 5m /dev/stdin
          ssh-add -l
          git push git@github.com:galaxy4public/galaxy4public.github.io.git
          ssh-add -D
          ssh-agent -k
        fi
        cd ..
        echo Completed

This is a copy of my live GitHub Action for deploying my blog that you are most likely reading right now and I decided not to edit anything, so if you just want to re-use it you will need to replace a few things, namely:

en_AU.UTF-8 => to a locale you are using (you can run locale -a if you are running Linux to see the list of locales available on your system);
content => you may need to change that to the name of your content directory (if you did not use the default name);
themes/mind-drops/content => you will need to drop this line since it is my theme’s content directory and you would not have it;
galaxy4public/galaxy4public.github.io => to <your_username/your_blog_repo_name>, obviously :)

Let’s look a bit more closely to understand how this GitHub Action is structured and what each step is doing.

It all starts with the definition of the action itself, the conditions of how it is triggered, and how it runs: you can get a formal description of the YAML structure of this configuration file in the official GitHub documentation on Workflows.

Here, we are only going to focus on steps defined under the “jobs:” section of the file since these steps are defining the logic we are after.

The “Initialise locale” step is quite important for Pelican since with a misconfigured locale Pelican tends to produce incorrect output (which is kind of expected). So in this step we are trying to determine whether the user (us :) ) has supplied the LANG variable and if they did we update /etc/locale.gen file, run the locale-gen command to update the corresponding files, and set the locale of the container to the requested locale.

The “Checkout the primary repo” step is leveraging the official “Checkout V2” Action and checks out a full copy of the source code repository of our blog and all the linked submodules. Initially, I was using a shallow copy using fetch-depth: 1, but the next step was requiring the full repository history to do its job reliably and I changed it to be a full history clone.

Since git does not store timestamps for the files and directories under its control, yet Pelican relies on timestamps to populate the modification time of the artefacts – we need to find a way to reconstruct at least file timestamps after the tree was checked out. One of the possible approaches would be to create a plugin that could determine whether we are inside a git working tree or not and depending on that apply different timestamp extraction policies, but I thought that a much easier way would be to prepare the checked out tree, hence making it compatible with the way Pelican expects things to be.

The “Restore modification times for content” step is my variant of how one could reconstruct the timestamps for files close enough to make it possible to use with Pelican. The approach relies on the fact that git records the timestamp of each commit including adding, updating, and deleting files. We create a list of all these file events using git log for file trees under “content” (where our blog content lives) and “themes/mind-drops/content” (where my custom theme injects some content such as the Web service worker script), then we use sed to filter and to re-arrange the output a bit, followed by reverse sorting to help to remove the entries that were introduced and later deleted. In the end, we have a list of file names with timestamps, so we go through the list in a loop and set the timestamps to files using touch.

The “Checkout Pages repo” step is cloning the public blog repository into the “output” directory where Pelican will put the generated files. This is needed to ensure that we can track the changes to the public repository, since Pelican is careful enough (if not instructed otherwise) only to update the files it generates and leave everything else in place as is. We use this later to determine whether any new content has been generated or not.

The “Set up Python” and the “Install dependencies” steps are pretty generic: the former is using the official GitHub Action to install and configure the latest available version of Python 3.x and the latter is leveraging pip to install all blog’s dependencies (including Pelican itself).

The “Generate the website” step is running pelican to process our articles and pages and to generate the result in the “output” directory. There are a couple of tricks with this step, though.

The first trick, which is not that obvious, is that we are removing the content of the “output” directory. It seems a bit weird since we just checked it out several steps before, does not it? Well, we are removing everything except hidden files and directories which happen to contain the “.git” subdirectory with all the actual data about the repository. Why do we do it? It is simple, this helps us to determine a situation if some file or directory was removed, so we could propagate that knowledge to the public blog repository. If we did not clean up the content of the “output” directory we would only append new changes and would never remove anything – this is how it was before I stumbled upon the problem, by the way. :)

The second trick of the “Generate the website” step is the extraction of the time zone information from the configuration files and is setting the TZ variable correctly just before we call pelican. Without this either Pelican may fail or if it does not it will produce UTC based date and times, which would be undesirable (at least for me, since my time zone is in Australia).

The final step is to push the updated content to the public blog repository, which will make it visible via GitHub Pages. Several things to notice there are:

In the env: section we are setting up the DEPLOY_KEY variable – this syntax is used to retrieve a named secret value from the associated secret key for a repository. We store the private part of the deploy key we specifically generated for this purpose at the beginning of this article in the private repository secret.
git status is used to determine whether there are any changes between what we have in the working tree and the repository index. If no changes were detected we just exit gracefully.
If any change to the generated content was detected, we temporarily load the private part of the deploy key into the ssh-agent (for 5 minutes), push changes to the public blog repository, then clean up the key from the agent and kill the agent itself.

From this point on, any push to the private codebase repository will trigger the GitHub Action and if the change has resulted in any updated content such content will be published to GitHub Pages!

There are quite a few things we could improve: such as introducing a broken links check, doing some sanity checks, etc. – but this would be for another article, I guess. :)

Wrap indicator in

 blocks

2020-05-10T21:00:00+10:00

I am not a front-end developer, not a UI designer, nor a UX guru, but I am an engineer, so when I face a puzzle worth solving my brain switches on and I cannot let it go until I find a satisfactory solution to the puzzle.

My blog heavily relies on me sharing session dumps and file excerpts using the code blocks. I am using PrismJS to highlight syntax in these blocks. However, after a while I found that there is one thing that really irritates me: these code blocks are not designed to be truly responsive, for instance – when a line wraps there is no easy to spot indication that such a wrap happened.

I quick search on the topic highlighted that this is kind of an unresolved issue and I found just one blog post from 2012 by Ian Yang after I already implemented my solution.

The approaches are quite close, but I like mine better since I think it is more semantically correct (I am using <div> instead of <span> and I did a bit deeper research into how to make it universal).

So, below I present you with my version of the solution and I would appreciate any feedback you may have, especially if it could help to find a nicer solution in the end.

Before we dive into details on how I come up with the solution, feel free to play with the basic demo – try to resize the box by dragging the bottom right corner and see how the text inside responds (if you are using anything than the Chrome browser you may be out of luck with the resizable <div>s, sorry):

This is quite a long line.  It is long enough to ensure that it will wrap no matter how big your screen is.  Well, it is possible that in 20 years from the moment I type this humanity will invent a medium that could display the whole line with no breaks, but until then it should be good enough for the demonstration purposes.
A very short line.
The short line above is used to demonstrate that it stays clear of additional info when its neighbour lines are wrapped.

I hope that demo gods were not angry at me and the demo above worked for you as expected, so if you want to understand how it works let’s dive under the hood of this solution :)

Preparation

First of all, we need to define what we have and what we want to achieve – it helps to stay on course, to understand when we reached a solution, and to assess how good the solution is:

we have one or more code blocks expressed through the <pre> HTML element;
these code blocks may have continuous white space that we want to preserve;
our page layout is flexible and there are no guarantees that the width of a particular viewport (a window through which a browser renders the visible part of the page) is enough to display the whole line of code;
we want to ensure that layout does not break when a very long code line is encountered, so we expect that line to be wrapped to the next line upon hitting width of the containing box;
we need to present the reader (a user who is consuming information) a visual indicator that the line was wrapped;
we also want the syntax highlighter (such as PrismJS in my case) not to be affected by whatever we come up with.

I think the above quite a solid definition of the goals and requirements for our little project. It is time to assess the artefacts we have and what we can do with them.

Analysis

Our primary artefact is the <pre> element. It contains the information we want to share with the world and we want to do this in the most easy to digest way.

By default, the <pre> element is quite a simplistic container. Its semantic meaning dictates that the content preserves the original formatting (such as amount and type of white space on the lines). The element also provides some basic functionality: you can define the size of the container and browsers are supposed to present you with controls to scroll the content if it does not fit the size of the container.

The following is just such bare almost not styled <pre> block wrapped into a resizable <div> container, so you could play with the dimensions of the container (in order to preserve the layout I am keeping the wrapping styles, so the box would fit into the layout):

This is an example of a basic <pre> container.  You should
be able to resize it by dragging the bottom right corner (but if you are
using anything other than Chrome you may be out of luck with the resizing).

This line was made specifically long to trigger the line wrapping, so even those who are using a browser that cannot resize should be able to play with it :)

However, if you played with the above container you may have noticed that at some point all lines are so mixed up together and it is really hard to tell that the original text comprised of five separate lines.

Unfortunately, the CSS standards do not provide any selectors or pseudo-elements to anchor the soft breaks. All we can do with the CSS selectors on the contained text is to choose the first letter (with :first-letter) and the first line of the text inside the <pre> element (with :first-line).

This is not good enough if you ask me. I would love to see some CSS selectors for soft breaks in the future iterations of the standard, but we are not there yet, so we have to work with what we have got.

And all we have is the standard CSS basic box model presentation focused selectors such as ::before and ::after.

So, if we look how text is rendered inside the <pre> block (browser’s DevTools is a really good tool to do the observations, by the way) we will notice that the browser considers the entire text as a single entity – there are no lines or anything, just a blob of text.

Therefore, the first piece of the puzzle would be to introduce something that could make the distinction between different lines of text possible.

Assuming that we can solve the above issue with introducing lines, the next step would be to display a marker on the right side of the line. That marker should follow the height of the line box as the line wraps.

Finally, we will need to figure out how not to display the marker on the last line of the wrapped multiline text to denote the end of wrapped line.

Implementation

Since the definition of the <pre> element only stipulates that the content of the element is pre-formatted with white space we can use other tags inside, e.g. we can wrap each physical line (a line that is terminated by a newline character) in a block element such as <div>.

Let’s see how it looks on our sample <pre> block when we wrap each physical line in <div class="line">…</div>. Keep in mind that <div> is a block element, so in order not to introduce unintended line breaks we need to ensure that every newline character in the original text except the very first one is replaced with the combination of the closing and opening tags such as </div><div class="line">, below is the result of such a change on our sample:

This is an example of a basic <pre> container.  You should
be able to resize it by dragging the bottom right corner (but if you are
using anything than Chrome you may be out of luck with the resizing).
This line was made specifically long to trigger the line wrapping, so even those who are using a browser that cannot resize should be able to play with it :)

Visually, it is almost exactly the same as the original sample we presented in the “Analysis” section, however there is a couple of differences.

The first one is visual and is obvious: the empty line (the fourth line of the sample) has collapsed into nothing. This is an undesired side effect and the reason for that is that the corresponding <div> block has no content, hence it has zero height. We can fix it by defining the min-height attribute for the <div>s defining line boundaries.

The second difference is not so obvious, but if you inspect any line of this <pre> block using your browser’s DevTools you will see that now we can differentiate between lines (when you walk through the DOM in your DevTools your browser will highlight the corresponding line inside the <pre> block and this is what we wanted to achieve at this step!

The next step is to display a marker on the right side of the line and that marker should occupy the exactly the same height as the line block we are attaching it to. Well, this can be easily done with the ::after pseudo-element:

pre div.line {
    position: relative; /* so we could position the child */
    min-height: 1em;    /* to avoid collapsing empty lines */
}

pre div.line::after {
    content: '';        /* we need content for pseudo elements */
    position: absolute;
    display: block;
    width: 1em;     /* without width it will be invisible */
    height: 100%;       /* to match the height of the parent */
    top: 0;
    right: -1em;
    background-color: blue; /* temporary, to make it visible */
}

Note that we also introduced position: relative to our <div>’s. It is needed since if we want to position our pseudo-element relative to its parent, the parent needs to have the position attribute set to anything but static. In our context, relative is the desired positioning for the <div> element. The corresponding rendered sample with the above stylesheet applied follows:

This is an example of a basic <pre> container.  You should
be able to resize it by dragging the bottom right corner (but if you are
using anything than Chrome you may be out of luck with the resizing).
This line was made specifically long to trigger the line wrapping, so even those who are using a browser that cannot resize should be able to play with it :)

So far so good! We have a blue coloured strip hanging on the right side just next to our text. Let’s style it a bit so it looks more appealing.

We have a couple of options here:

we can just use a transparent bitmap image with the desired marker;
we can leverage SVG as the source of the image.

The issue I have with transparent bitmap images is that they are not very responsive: to accommodate to all the possible resolutions I would end up with creating many bitmaps of different sizes to ensure that the marker looks good no matter what device is rendering the page.

On the other hand, the SVG graphics was not well supported on some browsers like Internet Explorer 6, etc. – The support is much better now, hence I would choose SVG every time.

Ideally, I did not want to use an image, but I could not figure out how I could put a repeating text block next to my lines, hence I found the following compromise (this is the content of the SVG file I created):

<svg xmlns="http://www.w3.org/2000/svg"
        version="1.1"
        viewBox="0 0 24 24">
    <text x="0" y="19">&#x23ce;</text>
</svg>

The first two lines are mandatory and describe to a browser that it works with an SVG image file.

The third line (viewBox=) is really important since it allows the SVG image to be scaleable, basically it says that the dimensions of the image are 25 by 25 units (the units are abstract, but you may think of them as pixels if it makes it easier).

Finally, the <text> element puts a text box with just a single Unicode character for the return symbol (⏎), since I think it is quite an adequate symbol to represent a line wrap point.

There are different ways how you can incorporate an SVG image into CSS rule, but since the content of the SVG file is so small and I did not want to host it externally (it would be an additional fetch request and more maintenance), I just did the following:

[user@localhost ~]$ cat nl.svg 
<svg xmlns="http://www.w3.org/2000/svg"
        version="1.1"
        viewBox="0 0 24 24">
    <text x="0" y="19">&#x23ce;</text>
</svg>
[user@localhost ~]$ sed -n '1{h;d};$!H;${H;x;s,\s\+, ,g;s,>\s*<,><,g;p}' nl.svg | base64 -w 0 ; echo
PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZlcnNpb249IjEuMSIgdmlld0JveD0iMCAwIDI0IDI0Ij48dGV4dCB4PSIwIiB5PSIxOSI+JiN4MjNjZTs8L3RleHQ+PC9zdmc+Cg==

In other words, I removed all the white space, then deleted all line breaks, so I got a single line SVG body, and I converted it to Base64. The resulting line is usable as the inline definition of an image. Let’s update our stylesheet with the new values (we just replaced the last line to change the background to background-image) and added dimensions for the background itself:

pre div.line {
    position: relative; /* so we could position the child */
    min-height: 1em;    /* to avoid collapsing empty lines */
}

pre div.line::after {
    content: '';        /* we need content for pseudo elements */
    position: absolute;
    display: block;
    width: 1em;         /* without width it will be invisible */
    height: 100%;       /* to match the height of the parent */
    top: 0;
    right: -1em;
    background-image: url(data:image/svg+xml;charset=utf-8;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZlcnNpb249IjEuMSIgdmlld0JveD0iMCAwIDI0IDI0Ij48dGV4dCB4PSIwIiB5PSIxOSI+JiN4MjNjZTs8L3RleHQ+PC9zdmc+Cg==);
    background-size: 1em 1em;
}

This is an example of a basic <pre> container.  You should
be able to resize it by dragging the bottom right corner (but if you are
using anything than Chrome you may be out of luck with the resizing).
This line was made specifically long to trigger the line wrapping, so even those who are using a browser that cannot resize should be able to play with it :)

Look at that, it is almost shaping into what we want :). However, something is not right – the alignment of the markers to the corresponding lines is nowhere to be seen. It is, like, they are of different sizes despite that the font-size property for both of them is the same. Hmm.

If we recall that CSS basic model document we will discover in the very last paragraph that:

note that for non-replaced inline elements, the amount of space taken up (the contribution to the height of the line) is determined by the line-height property

Is this our hint? Let’s try to explicitly set the line-height property for our lines (reading documentation you may find that the default value is normal and you can also find that normal equals 1.2 for the majority of browsers, but relying on that, I think is a bad idea). We also need to adjust our markers with the new information we have got:

pre div.line {
    position: relative; /* so we could position the child */
    min-height: 1em;    /* to avoid collapsing empty lines */
    line-height: 1.2;   /* set the height explicitly */
}

pre div.line::after {
    content: '';        /* we need content for pseudo elements */
    position: absolute;
    display: block;
    width: 1.2em;       /* without width it will be invisible */
    height: 100%;       /* to match the height of the parent */
    top: 0;
    right: -1.2em;
    background-image: url(data:image/svg+xml;charset=utf-8;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZlcnNpb249IjEuMSIgdmlld0JveD0iMCAwIDI0IDI0Ij48dGV4dCB4PSIwIiB5PSIxOSI+JiN4MjNjZTs8L3RleHQ+PC9zdmc+Cg==);
    background-size: 1.2em 1.2em;
    background-repeat: repeat-y;    /* optional, but nice to have */
    background-position: top;   /* optional, but better to be explicit */
}

This is an example of a basic <pre> container.  You should
be able to resize it by dragging the bottom right corner (but if you are
using anything than Chrome you may be out of luck with the resizing).
This line was made specifically long to trigger the line wrapping, so even those who are using a browser that cannot resize should be able to play with it :)

This is an improvement, is not it? So we just need some final touches to call it done:

As it stands, we are showing the marker for the last part of the wrapped line, but we should not;
The colour of the marker is too bright and matches the text colour of the line wrapping of which we are highlighting.

To address the latter a simple play with the opacity attribute will suffice, e.g. I think 50% transparency should make the marker less distracting.

The former, however, requires some thinking. Given that the post is already too long, I will just state that out of multiple options I had I chose the following: raise the background image by one line-height from the bottom, and clip a square 1x1 line-height at the top to compensate, as follows:

pre div.line {
    position: relative; /* so we could position the child */
    min-height: 1em;    /* to avoid collapsing empty lines */
    line-height: 1.2;   /* set the height explicitly */
}

pre div.line::after {
    content: '';        /* we need content for pseudo elements */
    position: absolute;
    display: block;
    width: 1.2em;       /* without width it will be invisible */
    height: 100%;       /* to match the height of the parent */
    bottom: 0;
    right: -1.2em;
    background-image: url(data:image/svg+xml;charset=utf-8;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZlcnNpb249IjEuMSIgdmlld0JveD0iMCAwIDI0IDI0Ij48dGV4dCB4PSIwIiB5PSIxOSI+JiN4MjNjZTs8L3RleHQ+PC9zdmc+Cg==);
    background-size: 1.2em 1.2em;
    background-repeat: repeat-y;    /* optional, but nice to have */
    background-position: top;   /* optional, but better to be explicit */
    margin-bottom: 0;
    clip: rect(1.2em 1.2em 100vh 0);
    opacity: .5;
}

The only tricky part in the added lines is the clip: rect(1.2em 1.2em 100vh 0) line. The clip property defines the visible rectangle the browser should preserve, but there is a conundrum: the height of the wrapped line is variable, so we cannot deterministically set this. The trick here is that it is highly unlikely (on the border of completely impossible :) ) that the visible height of the wrapped line would be bigger that the height of the viewport the user is looking through.

Therefore, we are setting the height of the clipped area to the height of the viewport (100vh), but this will be a rare edge case to actually utilise this – in the majority of cases the height of the clipping rectangle will be limited by the height of the block containing the wrapped line. Basically, this is exactly what we want :).

Let’s see the final result in action:

This is an example of a basic <pre> container.  You should
be able to resize it by dragging the bottom right corner (but if you are
using anything than Chrome you may be out of luck with the resizing).
This line was made specifically long to trigger the line wrapping, so even those who are using a browser that cannot resize should be able to play with it :)

Bonus Round

It should be mentioned, it is also possible to apply the same approach to the left side of the lines with a bit of a twist.

pre div.line {
    position: relative;     /* so we could position the child */
    min-height: 1em;        /* to avoid collapsing empty lines */
    line-height: 1.2;       /* set the height explicitly */
    text-indent: -1.2em;    /* first line indent */
    padding-left: 1.2em;    /* padding for the line */
}

pre div.line::before {
    content: '';            /* we need content for pseudo elements */
    position: absolute;
    display: block;
    width: 1.2em;           /* without width it will be invisible */
    height: 100%;           /* to match the height of the parent */
    top: 0;
    left: 0;
    background-image: url(data:image/svg+xml;charset=utf-8;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZlcnNpb249IjEuMSIgdmlld0JveD0iMCAwIDI0IDI0Ij48dGV4dCB4PSIwIiB5PSIxOSI+JiN4MjNjZTs8L3RleHQ+PC9zdmc+Cg==);
    background-size: 1.2em 1.2em;
    background-repeat: repeat-y;    /* optional, but nice to have */
    background-position: top;   /* optional, but better to be explicit */
    clip: rect(1.2em 1.2em 100vh 0);
    opacity: .5;
    transform: rotateY(180deg); /* mirror the arrow */
    transform-origin: left;
}

The trick here is the play between padding-left and text-indent for the <div> element since text-indent will be applied to the first line only (effectively negating the padding set for the whole element), thus the rest of wrapped line will be padded to make space for the marker, which we place as usual with the clipping region to suppress it for the very first part of the wrapped line.

The result looks as follows (and I am actually torn which side I like better, so, maybe, for my blog I may just go with the left-side solution):

This is an example of a basic <pre> container.  You should
be able to resize it by dragging the bottom right corner (but if you are
using anything than Chrome you may be out of luck with the resizing).
This line was made specifically long to trigger the line wrapping, so even those who are using a browser that cannot resize should be able to play with it :)

This idea for the padding-left and text-indent trick came from C. Shaun “Kainaw” Wagner when he answered a question about the wrap indicators on Stack Overflow (I am sure that it was known before, but this is where I learnt it).

Assessment of the result

We started with the following requirements:

these code blocks may have continuous white space that we want to preserve;

Achieved: we did some changes to ensure that even empty lines are preserved.
our page layout is flexible and there are no guarantees that the width of a particular viewport (a window through which a browser renders the visible part of the page) is enough to display the whole line of code

Achieved: our blocks are flexible and if a line is too long it is wrapped in an easy to understand way. We proved it by ensuring that any block can be dynamically resized.
we want to ensure that layout does not break when a very long code line is encountered, so we expect that line to be wrapped to the next line upon hitting width of the containing box;

Achieved: ditto as the previous item.
we need to present the reader (a user who is consuming information) a visual indicator that the line was wrapped;

Achieved: we got our dynamic markers showing when the line was wrapped.
we also want the syntax highlighter (such as PrismJS in my case) not to be affected by whatever we come up with.

~~Unknown: I think it would be the material for another post, but I do not expect a lot of issues since we only replaced the newline characters and preserved the rest of lines intact.i~~

It turned out, that it was quite a hassle since none of plugins PrismJS have for working with lines expect semantic markup of the lines inside the block. Therefore, I spent some time and came up with a plugin that does it “right”.

~~It is highly likely, that my plugin will get merged into the PrismJS tree (see the corresponding Pull Request).~~ After a discussion with upstream, it seems they want to rework a large chunk of the project to use the proposed functionality and make it a library of some sort. I do not have time to actively see it through, so one of PrismJS developers will continue the integration of my idea into PrismJS and something comparable will be implemented in PrismJS in near future. However, if for some reason it would not be or you need the functionality “here & now”, you can always get it from my fork of PrismJS (I will keep the plugin-lines branch until ~~it is merged upstream~~ upstream decides how to incorporate my idea).

By the way, my Lines plugin is the primary working horse for all syntax highlighted blocks on this site (so you have already seen it in action).

I hope this helped you and you learnt something :). If you have any constructive feedback, I would appreciate it, specifically if you have ideas on how to improve the presented solution.

Migrating blog to Pelican

2020-05-01T20:00:00+10:00

I had my blog site for more than a decade now, but until now I was not putting any effort or thoughts into maintaining my audience or promoting the site. It was dormant for nearly a decade so I decided to rejuvenate it and start using it as a platform I could leverage to share some ideas I think are worth sharing.

Before I embarked on the journey of renovating the blog site I needed to set some goals and requirements up, so I would be able to assess my progress and estimate how much effort is required. The primary goal is already known at this point: I need a tool that would allow me to easily share my ideas, grow and nurture the audience, and accumulate knowledge in one place over time.

I started to be increasingly unhappy with the Blogger platform: they were changing things and as the result parts of my blog became defunct, e.g.

in 2017, I lost the comment capability and did not have enough time to investigate and fix it,
in 2018, my theme became incompatible with the new trends of the platform,
and so on.

Therefore, the first requirement became to self-host my blog site, so I would have full control over the software that is supporting the site. This would give the flexibility and determinism of how the site is configured and operates.

The downside for hosting the site myself is that it would require ongoing maintenance cost (primarily calculated in the time I spend on the maintenance) and with my quite busy schedule I was not sure that I could afford it on the ongoing basis. So the second requirement became the low maintenance cost of the solution. Preferably, I wanted to make changes only when I needed to change site’s functionality.

I did some research and figured that a static website generator could be a solution that addressed both aforementioned requirements assuming that I could find a place where all underlying maintenance of the infrastructure and application layer handled by somebody else (and, preferably, free of charge).

There are plenty of static site generators (SSG). According to some articles, there are more than 400 different SSG at the moment, so the process of selecting one that is right for you could be quite challenging. I decided to define what I would like to see in the generator I could use and be happy about it before I start shopping around:

should be written in one of the scripting languages that are widely adopted

This requirement comes from my desire of running that software on a platform that is fully managed by somebody else (my initial thought was to host the site in an AWS S3 bucket and generate the site using an AWS Lambda function)
should be lean with minimum dependencies for the generator itself

The more dependencies software has the more likely it is to get an unexpected behaviour once one of the upstream dependencies make an undesirable breaking change. I really wanted stability and to touch the configuration in rare occasions only when I needed to change the behaviour of my site and not in an adhoc response to an upstream breaking change.
should be extensible and should support a plug-in mechanism

I have a really high bar on what the final result should be, e.g. valid and standard compliant HTML and CSS documents, semantic structuring, etc. In order not to be limited by the generator I needed a way on how I could influence the generation process without patching or tweaking the core of the generator itself, since doing it this way would mean maintaining a fork of the generator and as I mentioned above, it does not fit the second requirement.
the template language should be flexible and should allow conditional logic

Over the years I worked with multiple template engines and I found that the most comfortable engines are those that potentially allow you to break the concept of separating design from code. Not that I recommend doing so, but having such power gives you yet another flexible interface to express yourself.
the architecture of the generator should feel clean, stable, and thought out

I do not know how to easily describe this, but my understanding of the well engineered software is when its layout is simple to understand, structured, yet easily extendable without introducing any invasive changes to the structure, e.g. callbacks/hooks in the key transition points, so you could hook the external code up and influence the logic, etc.

A bit more research brought me to the realisation that there is a trend in the community and it has a name: Javascript, API, and Markup Stack(Jamstack). The ideas behind Jamstack resonate with my vision on how I wanted to run my blog, so if you want to understand the reasoning behind many choices I made their website would be a good place to start reading about the approach.

After a lot of consideration the following three SSGs were the primary contenders to become the engine of my blog site:

All of these three were aligned to the requirements, however I have an indescribable allergy to NodeJS ecosystem and I prefer Python over Ruby, hence Pelican was the first generator to spike with and to see whether we are a match or not.

The more I worked with Pelican the more I was falling in love with the software: with less than 600K of source code spread across just 15 files (with a couple being redundant in my configuration) it is packed with features and is very extensible through a plug-in system.

The next step was to create the templates (the collection of templates used to generate the output is called a theme in Pelican). While I was spiking I found a theme I somewhat liked (Brutalist by Matt McManus), then I found a free theme that I liked a lot (Striped by AJ), but that theme was just an HTML/CSS/JavaScript concept which was unrelated to Pelican in any way.

Initially, I started to port the Striped theme to Pelican using the Brutalist theme as the wireframe, but in the middle of that work I realised that despite the Striped theme was posed as “a free, fully responsive HTML5 site template” it was generated in a haste and was not that responsive as the claim would make you think. There were lots of inconsistencies and no overarching conventions on how things were structured within.

After a while, struggling to fix everything I was not happy with in Striped theme’s code, I decided to take a different approach: I decided to re-create Striped-like Pelican theme from scratch using Pelican’s default theme called “simple” as the foundation.

The result (which is still an ongoing process) as you can see has some resemblance to the theme I was taking inspiration from, but on the other hand it is quite different and standalone in its own right. Up until some point I was even retaining “Design by HTML5 UP” at the bottom of the left hand side menu, but when I realised that almost nothing left of the original design and layout, I removed it, the theme evolved much further that the inspirational theme I started with.

This post is already getting too long, so I will conclude with the following: I am going to publish a series of posts covering the development of this website, the tips and tricks I learnt over the course of creating it, and the infrastructure supporting the site and its deployment process in detail.

All blog posts related to these topics will be tagged with the “blog” tag, so you should be able to find them easily (or even subscribe to the blog RSS feed (you may need to copy the link into your RSS Reader) to get the latest updates as soon as I publish them).

nginx + a backend with a dynamic IP (e.g. AWS ELB)

2017-01-13T10:00:00+11:00

Recently, I wrote about the dynamic resolution of upstream servers in nginx which was achieved by quite an intrusive patch to the core nginx module. The patch was invented a while ago and was working very well up until recent nginx versions were released.

With the release of nginx 1.10 it was noticed that the patch crashes some workers under heavy load and this was unacceptable for the production load, hence a new approach was implemented.

The beauty of the new solution is that it is non-intrusive and works with any services that communicate via sockets.

In a nutshell, I just looked at the problem from a little bit different angle after I defined the requirements:

nginx needs to delegate the requests to a FastCGI server over a socket
we want to work with the standard packages provided by the distribution
the FastCGI server could be on a dynamic IP address

Since we are not allowed to patch the application the only place we can meddle with the communication between nginx and the FastCGI server is the socket nginx connects to. Therefore, we need some kind of a proxy that would take requests from nginx, determine the FastCGI endpoint, and forward the requests to that endpoint.

Initially, I thought that I would use something like netcat or a similar tool for this, but then I found that systemd provides systemd-socket-proxyd binary which fits the purpose perfectly and could be configured to be socket activated.

Before we dive into the implementation details of this solution, let’s describe what we are going to do and how:

nginx will be configured to talk to the locally bound socket - there are two options, actually: either a local TCP socket or a Unix socket
a proxy service will be socket activated by a request coming from nginx to that local socket
a proxy service should resolve the target and forward the request there

The nginx part is easy – we just need to replace the FastCGI server endpoint address (in our example below it was remote.php.backend.domain.tld.:9000) with our local socket (we are using a unix socket at /run/systemd-socket-proxyd/fastcgi.sock in this example):

upstream php {
    #server remote.php.backend.domain.tld.:9000;
    server  unix:/run/systemd-socket-proxyd/fastcgi.sock;
}
location ~ \.php$ {
    try_files   $uri = 404;
    fastcgi_pass    php;
    fastcgi_index   index.php;
    include     fastcgi.conf;
}

The next step is to define the socket activated proxy service. This generally requires creating two files in /etc/system/systemd directory: one for the socket and the other for the proxy itself. However, in this article we will go an extra mile and will define template units so the same configuration could be reused to launch multiple proxies using the same base templates.

The first template file is for the systemd service which will provide the proxy capability:

[Unit]
Description="Generic Socket Proxy (%I)"
Documentation=https://www.freedesktop.org/software/systemd/man/systemd-socket-proxyd.html
After=network.service

[Service]
EnvironmentFile=-/etc/sysconfig/systemd-socket-proxyd.%i
User=nobody
Group=nobody
OOMScoreAdjust=-1000
UMask=077
ExecStart=/usr/lib/systemd/systemd-socket-proxyd $TARGET
Restart=on-failure
PrivateTmp=true
PrivateDevices=true
#PrivateUsers=true
#ProtectSystem=strict
ProtectSystem=full
#ProtectKernelTunables=true
#ProtectControlGroups=true
#NoNewPrivileges=true
#ProtectKernelModules=true
#MemoryDenyWriteExecute=true

Depending on the version of your systemd manager you may be able to uncomment more lines than was shown in this example, which was tested on CentOS 7.3.1611.

Also, in the example template given above we are using OOMScoreAdjust=-1000 to protect this proxy service from being killed in the event the system is starving for memory – this may be something you do not need.

The second template file is for the socket unit that would trigger the activation of the proxy service when a request arrives on the socket:

[Unit]
Description="Socket for Generic Socket Proxy (%I)"
Documentation=https://www.freedesktop.org/software/systemd/man/systemd-socket-proxyd.html

[Socket]
ListenStream=/run/systemd-socket-proxyd/%i.sock
SocketUser=root
SocketGroup=root
SocketMode=0660
DirectoryMode=0711

[Install]
WantedBy=sockets.target

You may notice that the defaults are pretty strict:

the socket is owned by root and only root is allowed to work with the socket;
the directory permissions are set in such a way that the /run/systemd-socket-proxyd directory file list is not readable by anyone except root.

These are safe and sane defaults and can be tweaked per instance that was instantiated using the template as shown later in this article.

Since our goal is to connect nginx to the backend FastCGI service we need to ensure that the proxy socket is read/write accessible to nginx’es workers, so we need to tweak the settings of the socket unit:

[Socket]
SocketGroup=nginx

Note how we extended the template for a specific named instance of the socket unit: we defined the systemd-socket-proxyd@fastcgi.socket.d sub-directory and put a drop-in configuration snippet there.

Now, to achieve our goal we need to specify the target endpoint for our proxy service (the named instance we spawn using the template):

[Service]
# Reset the ExecStart, so we could override it
ExecStart=
ExecStart=/usr/lib/systemd/systemd-socket-proxyd remote.php.backend.domain.tld.:9000

The mechanics behind extending the configuration is the same as for the socket unit, but here we overrode the ExecStart command to specify the target endpoint for the proxy.

OK, we are done with the configuration of systemd, so it would be a good time to reload the systemd daemon:

[root@localhost ~]# systemctl daemon-reload

If you run a system where SELinux is disabled (why?!) you don’t need to do anything additional and should be good, but if you are security conscious and want to ensure that you follow the least privilege principle, then read on :)

Unfortunately, systemd-socket-proxyd seems not to be used a lot by the community (most likely people are just unaware of it) so the tool has no dedicated policy attached to it in the targeted SELinux policy. I plan to push the change into the SELinux reference policy, but before I do we are going to use a custom loadable policy module.

To build a module you need the SELinux reference policy development framework installed on the instance you are building your policies (it can be the same instance you are running your proxy on, but I would advise to use a temporary VM for building/compiling policies since the only time you need that development stuff is when you are compiling the module from sources).

On CentOS, you can install all the necessary bits to build a loadable SELinux module by installing the selinux-policy-devel package using yum:

[root@localhost ~]# yum -y install selinux-policy-devel

Note, I skipped the output of the command since it does not provide any useful information for the purposes of this article.

Once the SELinux development framework is installed we can start designing our loadable policy for the systemd-socket-proxyd service.

Our policy will consist of two files:

systemd-socket-proxyd.te – the type enforcement ruleset; and
systemd-socket-proxyd.fc – the file context ruleset.

Eventually, we will need to introduce the corresponding module interface support file too, but for the purposes of this article we should be fine with the automatically generated one.

The content of the systemd-socket-proxy.te file is listed below:

policy_module(systemd-socket-proxyd, 1.0)

## <desc>
##  <p>
##  Allow systemd-socket-proxyd to bind any port instead of one labelled
## with systemd_socket_proxyd_port_t.
##  </p>
## </desc>
gen_tunable(systemd_socket_proxyd_bind_any, false)

## <desc>
## <p>
## Allow systemd-socket-proxyd to connect to any port instead of
## labelled ones.
## </p>
## </desc>
gen_tunable(systemd_socket_proxyd_connect_any, false)

systemd_domain_template(systemd_socket_proxyd)

type systemd_socket_proxyd_unit_file_t;
systemd_unit_file(systemd_socket_proxyd_unit_file_t)

sysnet_dns_name_resolve(systemd_socket_proxyd_t)

# resolver
allow systemd_socket_proxyd_t self:unix_dgram_socket { create getopt setopt sendto read write };

# listener
type systemd_socket_proxyd_port_t;
corenet_port(systemd_socket_proxyd_port_t);
allow systemd_socket_proxyd_t self:tcp_socket accept;

tunable_policy(`!systemd_socket_proxyd_bind_any',`
 allow systemd_socket_proxyd_t systemd_socket_proxyd_port_t:tcp_socket name_bind;
')

tunable_policy(`systemd_socket_proxyd_bind_any',`
 corenet_tcp_bind_all_ports(systemd_socket_proxyd_t)
')

# target
tunable_policy(`!systemd_socket_proxyd_connect_any',`
 allow systemd_socket_proxyd_t port_type:tcp_socket name_connect;
')

tunable_policy(`systemd_socket_proxyd_connect_any',`
 corenet_tcp_connect_all_ports(systemd_socket_proxyd_t)
')

# consumer
allow daemon systemd_socket_proxyd_t:unix_stream_socket connectto;

This is still work in progress, but so far it works at least for my projects. Note that there are two SELinux booleans defined which affect the behaviour of the policy:

systemd_socket_proxyd_bind_any

allows to bind proxy to any TCP socket if set to true, otherwise the proxy would be able to connect to TCP ports labelled with systemd_socket_proxyd_port_t;
systemd_socket_proxyd_connect_any

allows proxy to connect to any target TCP ports if set to true, otherwise the target is limited by the ports labelled with systemd_socket_proxyd_port_t.

Now, to allow the proper transition into the systemd_socket_proxyd_t domain we need to label the systemd-socket-proxyd binary with the systemd_socket_proxyd_exec_t label.

The content of the systemd-socket-proxy.fc file that implements this behavior is as follows:

/(usr/lib|etc)/systemd/system/systemd-socket-proxyd\.service  gen_context(system_u:object_r:systemd_socket_proxyd_unit_file_t,s0)
/usr/lib/systemd/systemd-socket-proxyd -- gen_context(system_u:object_r:systemd_socket_proxyd_exec_t,s0)

At this point, we have everything we need to compile a policy module, so let’s just do it now:

[root@localhost ~]# ls -l
total 8
-rw-r--r--. 1 root root  235 Jan  5 05:21 systemd-socket-proxyd.fc
-rw-r--r--. 1 root root 1467 Jan  6 00:10 systemd-socket-proxyd.te
[root@localhost ~]# make -f /usr/share/selinux/devel/Makefile
Compiling targeted systemd-socket-proxyd module
/usr/bin/checkmodule:  loading policy configuration from tmp/systemd-socket-proxyd.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/systemd-socket-proxyd.mod
Creating targeted systemd-socket-proxyd.pp policy package
rm tmp/systemd-socket-proxyd.mod tmp/systemd-socket-proxyd.mod.fc
[root@localhost ~]# semodule -i systemd-socket-proxyd.pp
[root@localhost ~]# restorecon -v /usr/lib/systemd/systemd-socket-proxyd
restorecon reset /usr/lib/systemd/systemd-socket-proxyd context system_u:object_r:init_exec_t:s0->system_u:object_r:systemd_socket_proxyd_exec_t:s0

The semodule -i systemd-socket-proxyd.pp command has actually installed the module into the system, but if you were building the module on a different instance, then instead of installing the module you just need to grab the resulting systemd-socket-proxyd.pp file and transfer it to the target instance where proxy is going to be running and only apply the last two commands (semodule -i ... and restorecon).

We approached the time when we need to perform the pre-flight checks before launching our new service :). First, we need to check whether the file context was applied to the systemd-socket-proxyd binary:

[root@localhost ~]# ls -ldZ /usr/lib/systemd/systemd-socket-proxyd 
-rwxr-xr-x. root root system_u:object_r:systemd_socket_proxyd_exec_t:s0 /usr/lib/systemd/systemd-socket-proxyd

Looks good! Let’s start the socket unit and check the permissions set on the socket file:

[root@localhost ~]# systemctl start systemd-socket-proxyd@fastcgi.socket
[root@localhost ~]# ls -ldZ /run/systemd-socket-proxyd{,/fastcgi.sock}
drwx--x--x. root root  system_u:object_r:var_run_t:s0   /run/systemd-socket-proxyd
srw-rw----. root nginx system_u:object_r:var_run_t:s0   /run/systemd-socket-proxyd/fastcgi.sock

This also looks as expected. The next test would be to check that our proxy service is running with the desired set of permissions and in the correct SELinux domain:

[root@localhost ~]# socat -v unix-client:/run/systemd-socket-proxyd/fastcgi.sock stdin

< 2017/01/13 00:59:22.987083  length=1 from=0 to=0

[root@localhost ~]# ps uZ -C systemd-socket-proxyd
LABEL                           USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
system_u:system_r:systemd_socket_proxyd_t:s0 nobody 28643 0.0  0.0 86628 756 ? Ssl  00:59   0:00 /usr/lib/systemd/systemd-socket-proxyd 127.0.0.1:9000
[root@localhost ~]# cat /proc/28643/status 
Name: systemd-socket-
State: S (sleeping)
Tgid: 28643
Ngid: 0
Pid: 28643
PPid: 1
TracerPid: 0
Uid: 99 99 99 99
Gid: 99 99 99 99
FDSize: 64
Groups: 99 
[truncated]

The first socat command was needed to trigger the socket activation that resulted in the systemd-socket-proxyd.service being launched.

The second command confirmed that the service is running under the nobody user and within the systemd_socket_proxyd_t domain.

Finally, the third command confirmed that the privileges were properly dropped and there is no way the service could regain the escalated privileges back.

Looks like we are all, so let’s make our changes persistent. To achieve this we just need to enable the systemd-socket-proxyd@fastcgi.socket unit:

[root@localhost ~]# systemctl enable systemd-socket-proxyd@fastcgi.socket
Created symlink from /etc/systemd/system/sockets.target.wants/systemd-socket-proxyd@fastcgi.socket to /etc/systemd/system/systemd-socket-proxyd@.socket.
[root@localhost ~]# systemctl status systemd-socket-proxyd@fastcgi.socket | fgrep Loaded:
   Loaded: loaded (/etc/systemd/system/systemd-socket-proxyd@.socket; enabled; vendor preset: disabled)

Just to show that the configuration is working I set a simple lab up in a VM and followed this article with the only exception of the PHP/FPM location which I am running on the same VM:

[root@localhost ~]# cat /usr/share/nginx/html/test.php
<?php echo "This is the output from PHP\n" ?>
[root@localhost ~]# cat /etc/nginx/conf.d/php-upstream.conf 
upstream php {
    #server        remote.php.backend.domain.tld.:9000
    server         unix:/run/systemd-socket-proxyd/fastcgi.sock;
}
[root@localhost ~]# cat /etc/nginx/default.d/php.conf
location ~ \.php$ {
    try_files      $uri = 404;
    fastcgi_pass   php;
    fastcgi_index  index.php;
    include        fastcgi.conf;
}
[root@localhost ~]# cat /etc/systemd/system/systemd-socket-proxyd@fastcgi.service.d/fastcgi.conf
[Service]
# Reset the ExecStart, so we could override it
ExecStart=
ExecStart=/usr/lib/systemd/systemd-socket-proxyd 127.0.0.1:9000
[root@localhost ~]# telnet 0 80
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
GET /test.php HTTP/1.0

HTTP/1.1 200 OK
Server: nginx/1.10.2
Date: Fri, 13 Jan 2017 03:01:16 GMT
Content-Type: text/html
Connection: close
X-Powered-By: PHP/5.4.16

This is the output from PHP
Connection closed by foreign host.

Works as expected :).

There is one thing one needs to be aware of: when I started to work on this I discovered that systemd-socket-proxyd had a hard-coded limit for the number of connections set to 256 (I introduced the -c parameter to systemd-socket-proxyd, so one could dynamically set the limit, however it would take some time until this change is propagated to all major distros).

Also, it is worth it to mention that the provided configuration is not efficient if you are using a domain name for the target endpoint for the proxy, so if this is the case I would advise to run a local DNS caching service (e.g. dnsmasq) so you would not spend time on the DNS queries.

As always, I would appreciate any feedback you may have.

Dynamic resolution of upstream servers in nginx

2016-09-20T10:00:00+10:00

UPDATE: This approach was superseded by the proxying through systemd-socket-proxyd approach.

Many of my clients are running application stacks consisting of nginx plus some kind of scripting engine behind it (be it PHP, Ruby, or something else).

The architecture I designed for this kind of workload involves at least two load balancers:

an external, frontend load balancer that serves the web requests from visitors; and
an internal, backend load balancer that distributes load between the backends.

Everything looks great when you implement this using “in-house” infrastructure where you control most of the networking aspects.

However, the tendency is that most enterprises are moving to the cloud providers and with that we lose some control.

Specifically, often the cloud providers define their load-balancers as auto-scaling entities that change their IP addresses depending on the scale-in/out activity.

Unfortunately, the community version of nginx does not know how to dynamically resolve the specified upstream servers (such a functionality is available from the nginx commercial subscription only), so I spent a couple of evenings to implement the desired functionality as a patch.

The patch implements the dynamic DNS resolution of the specified upstream servers in the upstream compatible way: we are re-using the very same “resolve” keyword on the server line as the commercial version of nginx does ensuring that if you ever decide to switch to the commercial subscription you would not need to change your configs.

The patch was originally created for nginx 0.8.6 and was used in production for the last couple of years. The work on the patch was sponsored by Openwall (Australia) and Data Solutions Group.

Enjoy! :)

Transparent SSH host-jumping (Expert)

2016-07-26T10:00:00+10:00

A while ago in the Transparent SSH host-jumping (Advanced) post I described a technique on how one could jump quite effortlessly through a chain of intermediate hosts. However, there was a catch: the user names and ports across the whole chain should be the same and there was no easy way to change that.

Given that I recently paid quite a lot of attention to the ProxyCommand directive I decided to look into the implementation of the helper script that will allow one to tweak parameters for the hosts in the chain.

You can read the original post for the details of how this host-jumping technique works, here I am only going to provide the proxy script and the corresponding ssh config parameter block to use the script.

The goal was to support the following syntax:

[user@localhost ~]$ ssh default_user@userA^hostA/userB^hostB:port/hostC

It was a little challenge to come up with the character for identifying the user part for intermediate hosts:

@ - cannot be used since SSH parses the command line before providing the string to the ProxyCommand script
% - cannot be used since SSH was thinking that I’m trying to do an expansion of the internal SSH variable
!, $, (, ), and & - are all shell unfriendly

So I decided on the ^ character as a delimiter.

In the proposed command above the “default_user” is the user name we ultimately want to use for logging into the last host in the chain (it happens that this user name will be used for any host in the chain where no alternative name is provided).

Each host in the chain could also be provided with the relevant port or, if the port is omitted, it will use the global port configuration (usually 22/tcp but can be changed with the -p argument to ssh). The script is a bit not optimised (bash is really slow on string processing, but I decided to stick with pure bash where it was possible):

#!/bin/bash
set -eu -o pipefail
exec 10<&0 11>&1 0<&2 1>&2
DEFAULT_USER="$1"
DEFAULT_PORT="$3"
HOST_CHAIN="$2"
HOST_NEXT="${HOST_CHAIN%%/*}"
HOST_USER="${HOST_NEXT%%^*}"
[ "$HOST_USER" == "$HOST_NEXT" ] && HOST_USER="$DEFAULT_USER" ||:
HOST_PORT="${HOST_NEXT##*:}"
[ "$HOST_PORT" == "$HOST_NEXT" ] && HOST_PORT="$DEFAULT_PORT" ||:
TARGET_HOST="${HOST_CHAIN##*/}"
TARGET_PORT="${TARGET_HOST##*:}"
TARGET_HOST="${TARGET_HOST%:*}"
[ "$TARGET_PORT" == "$TARGET_HOST" ] && TARGET_PORT="$DEFAULT_PORT" ||:
TARGET_HOST="${TARGET_HOST#*^}"
HOST_NEXT="${HOST_NEXT#*^}"
HOST_NEXT="${HOST_NEXT%:*}"
HOST_CHAIN="${HOST_CHAIN%/*}"
[ "$HOST_CHAIN" == "${HOST_CHAIN#*/}" ] && HOST_CHAIN= || HOST_CHAIN="${HOST_CHAIN#*/}"
HOST_CHAIN="$HOST_NEXT${HOST_CHAIN:+/$HOST_CHAIN}"

if [ ! -d "$HOME/.ssh/.sessions" ]; then
    echo "Creating the sessions directory" >&2
    mkdir -m700 "$HOME/.ssh/.sessions"
fi

CONTROL_SOCKET=$(printf "$HOST_USER@HOST_CHAIN:$HOST_PORT\n" | shasum | cut -f1 -d' ')

exec 0<&10 1>&11
exec ssh \
    -o "ControlMaster auto" \
    -o "ControlPath ~/.ssh/.sessions/$CONTROL_SOCKET" \
    -o "ControlPersist 120s" \
    -l "$HOST_USER" \
    -p "$HOST_PORT" \
    "$HOST_CHAIN" \
    -W "$TARGET_HOST":"$TARGET_PORT"

The above is a proof of the concept and it “works for me” :). I am using it every day when I need to access boxes behind a bastion host. Your mileage may vary and you are free to create your own version of the script that would perform better (I would be really glad if a version of such a script could be shared with me).

The corresponding configuration block in the ssh config file looks as follows:

Host */*
    # if you uncomment ControlPath you also need to uncomment ControlMaster
    #ControlMaster auto
    # For OpenSSH < 6.7 you may uncomment the following, but long chains will fail:
    #ControlPath   ~/.ssh/.sessions/%r@%h:%p
    # For OpenSSH >= 6.7 you should uncomment the following:
    #ControlPath   ~/.ssh/.sessions/%C
    ProxyCommand ~/bin/ssh-helper.sh %r %h %p

Well, this is a bit messy since OpenSSH introduced the %C macro in version 6.7 and without %C the ControlPath string gets too long for OpenSSH to create a socket on the filesystem for long chains of hosts.

SSH: Interactive ProxyCommand

2016-07-25T10:00:00+10:00

I was involved in the creation of the sshephalopod project, which was an attempt to build an enterprise level authentication framework for SSH authentication using the SSH CA feature.

The project is based on a wrapper script that signs a user via a SAML identity provider and gets user’s public key signed for the further usage.

In one of the discussions I pointed out that such a wrapper script is not good for the end user experience and I proposed to provide the users with an excerpt for their ssh config file, so the functionality of sshephalopod would be transparent to the general usage scenario of the ssh tool.

The response was that ProxyCommand do not support interactivity. Well, as they say: The challenge is accepted :)

The following is my train of thoughts before I came up with a general solution on how to allow an interactive command to be used as the ProxyCommand in the ssh config file.

Before we start solving the problem at hand we need to create a test environment, so we would be able to confirm when we reached success. The task itself was very simple: we needed a host we could ssh into (an sshd daemon running on the local host would be sufficient), then we needed an interactive script, and a configuration block for the connection.

The configuration block is pretty simple (%h expands to localhost and %p expands to the port specified on the command line or to “22” otherwise):

[user@localhost ~]$ fgrep -A1 'Host localhost' ~/.ssh/config
Host localhost
 ProxyCommand ~/bin/interactive.script.sh %h %p

Since most of our research is going to be inside the interactive script you will see several incarnations of script’s body. The very first one was the following:

#!/bin/bash
exec nc "$1" "$2"

At this point we just need to confirm that our test environment works as expected – the ssh session should be proxied through the nc command and we should be able to login under our own account via ssh to the localhost (my private key was added to the key manager with ssh-add, hence no password prompt was displayed):

[user@localhost ~]$ ssh galaxy@localhost
Last login: Thu Jul 21 01:30:21 2016
$

OK, we confirmed that we can establish a proxied connection and tunnel our ssh session through it.

Each interactive script or program relies on the communication channel with the user otherwise it could not be interactive. This channel comprises at least of two file descriptors: one for standard input and the other for standard output, so let’s check what descriptors are available for our script:

#!/bin/bash
# on Linux the following line would be much simpler: ls -l /proc/$$/fd/, but
# on OS X they do not expose the open file descriptors through /proc, so I
# used "lsof" instead.
lsof -p $$ >&2
exec nc "$1" "$2"

If we try to connect now we should see something like the following (I am writing this article on an OS X machine so I provide the output from OS X, however this also works for Linux):

[user@localhost ~]$ ssh galaxy@localhost
COMMAND   PID   USER   FD   TYPE             DEVICE  SIZE/OFF     NODE NAME
bash    45225   user  cwd    DIR                1,2       612   893854 /Users/user/.ssh
bash    45225   user  txt    REG                1,2    628640  2329236 /bin/bash
bash    45225   user  txt    REG                1,2    625712 13892061 /usr/lib/dyld
bash    45225   user  txt    REG                1,2 385393734 13894121 /private/var/run/dyld_shared_cache_x86_64
bash    45225   user    0   PIPE 0x44d71099589485df     16384          ->0x44d7109951223d0f
bash    45225   user    1   PIPE 0x44d710995122454f     16384          ->0x44d71099512246af
bash    45225   user    2u   CHR               16,1  0t631830     9705 /dev/ttys001
bash    45225   user  255r   REG                1,2       219 15455259 /Users/user/bin/interactive.script.sh
Last login: Mon Jul 25 17:52:40 2016 from localhost
$

We are interested in file descriptors 0 (standard input), 1 (standard output), and 2 (standard error). As you can see the standard input and output are part of the pipes (presumably linking them to the parent ssh process) and standard error is pointing to our terminal session.

I could have occupied a bit more of the page space showcasing that if you try to communicate on standard input and/or output the ssh client will terminate since you will be messing with the SSH protocol flow, but I believe you will trust me on this :).

What can we do to interact with the user, yet to preserve the channel with the parent ssh process? Well, the answer is quite obvious:

we have a pointer to the terminal session (file descriptor 2, the standard error, points to the terminal),
so we just need to save pointers to the pipes’ ends,
re-open standard input and output with the terminal before we interact with the user,
and restore these file descriptors back once we are ready to hand over the ssh session.

The following version of the script demonstrates the implementation of the above logic:

#!/bin/bash
exec 10<&0 11>&1 0<&2 1>&2
# start of the interactive behaviour
lsof -p $$
read -p "Type something: " I
echo "You typed: $I"
# finish of the interactive behaviour
exec 0<&10 1>&11
exec nc "$1" "$2"

I think it is time to test it :) :

[user@localhost ~]$ ssh galaxy@localhost
COMMAND   PID   USER   FD   TYPE             DEVICE  SIZE/OFF     NODE NAME
bash    45238   user  cwd    DIR                1,2       612   893854 /Users/user/.ssh
bash    45238   user  txt    REG                1,2    628640  2329236 /bin/bash
bash    45238   user  txt    REG                1,2    625712 13892061 /usr/lib/dyld
bash    45238   user  txt    REG                1,2 385393734 13894121 /private/var/run/dyld_shared_cache_x86_64
bash    45238   user    0u   CHR               16,1  0t640407     9705 /dev/ttys001
bash    45238   user    1u   CHR               16,1  0t640407     9705 /dev/ttys001
bash    45238   user    2u   CHR               16,1  0t640407     9705 /dev/ttys001
bash    45238   user   10   PIPE 0x44d710995122378f     16384          ->0x44d71099589494ff
bash    45238   user   11   PIPE 0x44d7109951223d0f     16384          ->0x44d710995122454f
bash    45238   user  255r   REG                1,2       135 15455285 /Users/user/bin/interactive.script.sh
Type something: This is a test
You typed: This is a test
Last login: Mon Jul 25 17:57:05 2016 from localhost
$ logout
Connection to localhost closed.

Mission accomplished! :)

I hope this small article would help somebody to design better wrappers around SSH. Keep in mind that you could always optimise it further. For example, recent versions of OpenSSH support passing of a file descriptor from the ProxyCommand script, so if you have a decent netcat tool that supports the “-F” option (fdpass) you could get native performance for the ssh communication link with no proxy process hanging around.

P.S. if you have any questions do not hesitate to comment.

Raspberry Pi 3 toolchain on CentOS 7

2016-05-22T12:00:00+10:00

I heard a lot about Raspberry Pi boards but until now I had no need nor time to work with one.

However, recently I purchased a Dodge Journey R/T and found that although I love the car I am so disappointed with its software and hard-wired logic that I decided to experiment a bit and fix the most annoying things.

Since almost everything inside the car is talking over the CAN bus I needed some kind of a enclave inside the car where I could run my code and inject/intercept CAN messages.

I looked around and found that I can build the desired appliance using Raspberry Pi 3 (Model B) + PiCAN 2 HAT board.

Once the hardware was delivered to my home the time came to start building the software side of things. My distribution of choice for this project became CentOS 7 (userland), however, building stuff on the Raspberry Pi itself was a painful and long process, so I needed a proper toolchain to be able to utilise much more powerful hardware and do builds quicker.

The following is a session dump (with some notes) on how I built my toolchain on an AWS EC2 instance which was running a minimal CentOS 7 as its OS.

I spawned an EC2 instance (you need at least 8GB of free space there to be able to build the toolchain), logged in, and ensured that my system is up to date (and update it if it was not):

[root@localhost ~]# yum -y update
Loaded plugins: at-exit, fastestmirror, post-transaction-actions
Loading mirror speeds from cached hostfile
 * base: mirror.domain.tld
 * extras: mirror.domain.tld
 * updates: mirror.domain.tld
No packages marked for update

The easiest way to build a toolchain is to use the crosstool-ng project, so I went ahead and downloaded the latest version with the corresponding GPG signature, verified the signature (although the verification was very superficial since Bryan Hundven is not in my Web of Trust and the crosstool-ng website was not providing an HTTPS page to confirm the fingerprint of the key), and unpacked the sources:

[root@localhost ~]# yum -y install bison bzip2 flex gcc gcc-c++ glibc-static gperf help2man libstdc++-static libtool make ncurses-devel texinfo wget
[root@localhost ~]# useradd -m build
[root@localhost ~]# su - build
[build@localhost ~]$ wget -q 'http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.22.0.tar.xz'
[build@localhost ~]$ wget -q 'http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.22.0.tar.xz.sig'
[build@localhost ~]$ gpg --recv-keys 35B871D1 --keyserver pgp.surfnet.nl
gpg: "--keyserver" not a key ID: skipping
gpg: "pgp.surfnet.nl" not a key ID: skipping
gpg: requesting key 35B871D1 from hkp server keys.gnupg.net
gpg: /home/build/.gnupg/trustdb.gpg: trustdb created
gpg: key 35B871D1: public key "Bryan Hundven " imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
[build@localhost ~]$ gpg --verify crosstool-ng-1.22.0.tar.xz.sig
gpg: Signature made Fri Nov 20 13:09:26 2015 UTC using RSA key ID 35B871D1
gpg: Good signature from "Bryan Hundven "
gpg:                 aka "Bryan Hundven "
gpg:                 aka "Bryan Hundven "
gpg:                 aka "Bryan Hundven "
gpg:                 aka "[jpeg image of size 3080]"
gpg:                 aka "[jpeg image of size 16246]"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 561E D9B6 2095 88ED 23C6  8329 CAD7 C8FC 35B8 71D1
[build@localhost ~]$ tar xJSf crosstool-ng-1.22.0.tar.xz
[build@localhost ~]$ rm crosstool-ng-1.22.0.tar.xz*

The next step was to prepare the environment for the crosstool-ng build process:

[build@localhost ~]$ cd crosstool-ng
[build@localhost ~]$ /configure --prefix=$HOME/ct-ng
checking build system type... x86_64-pc-linux-gnu
[skipped]
configure: creating ./config.status
config.status: creating Makefile
[build@localhost ~]$ make
  SED    'ct-ng'
[skipped]
  SED    'docs/ct-ng.1'
  GZIP   'docs/ct-ng.1.gz'
[build@localhost ~]$ make install
  GEN    'config/configure.in'
[skipped]

For auto-completion, do not forget to install ct-ng.comp into your bash completion directory (usually /etc/bash_completion.d/)

[build@localhost ~]$ export PATH=$HOME/ct-ng/bin:$PATH
[build@localhost ~]$ cd

Once the above was done we are ready to actually configure and build our toolchain:

[build@localhost ~]$ mkdir ~/builddir
[build@localhost ~]$ cd ~/builddir
[build@localhost ~]$ ct-ng menuconfig

I was quite happy with the defaults and the only changes I made were the following:

Paths and misc options -> Try features marked as EXPERIMENTAL -> check

You may skip this but I prefer to see all available options when I go through the list of configurable parameters, hence I enabled this;
Target options -> Target Architecture -> arm

My board is Raspberry Pi 3 (Model B) which runs on ARMv8 Cortex-A53, yours may be different but most likely it is also ARM based :)
Target options -> Bitness -> 64-bit

There is actually almost no point in building a 64-bit version when you are running on such a limited device as Raspberry, but at the time I was writing this blog post I was building the AArch64 version of the toolchain. I would recommend to stick with the 32-bit version of the toolchain if you do not have any specific requirements for the 64-bit one;
Target options -> Architecture level -> armv8-a

Since my only target was Raspberry Pi 3 (Model B) and the toolchain was created for the personal use I did additional optimisation for my board.

If you are not sure or if you want a toolchain that would produce more compatible code for a range of ARM devices, then leave this setting at its default, i.e. blank
Target options -> Emit assembly for CPU -> cortex-a53

Actually, this supersedes the previous option since it is more explicit. If you are not optimising for a specific hardware, then leave it at the default setting (which is blank).

Also, there are more optimisations, e.g. you can specify that FPU for this board is neon-fp-armv8, etc. but to keep it simple I decided not to showcase that.
Toolchain options -> Tuple’s vendor string -> rpi3

This is optional but I like my toolchain to be as descriptive as possible, e.g. with this setting binaries will be prefixed with aarch64-rpi3-linux-gnueabi-
Operating System -> Target OS -> linux

It is essential to set this to “linux” otherwise crosstool-ng will assume that you are building a toolchain for a bare-bone hardware and the produced binaries would not be properly linked to work with the OS on the board

Once we are comfortable with the configuration of the future toolchain we exit the configuration menu (saving the changes) and launch the build process (it would take approximately an hour on AWS t2.small instance):

configuration written to .config

*** End of the configuration.
*** Execute 'ct-ng build' to start the build or try 'ct-ng help'.

[build@localhost ~]$ ct-ng build
[INFO ]  Performing some trivial sanity checks
[INFO ]  Build started 20160522.070915
[skipped]
[INFO ]  Build completed at 20160522.080629
[INFO ]  (elapsed: 57:14.18)
[INFO ]  Finishing installation (may take a few seconds)...
[57:15] /

Seems that our Raspberry Pi 3 toolchain is ready to be used. Let’s compile something and see how it goes:

[build@localhost ~]$ export PATH=$HOME/x-tools/aarch64-rpi3-linux-gnueabi/bin:$PATH
[build@localhost ~]$ aarch64-rpi3-linux-gnueabi-gcc --version
aarch64-rpi3-linux-gnueabi-gcc (crosstool-NG crosstool-ng-1.22.0) 5.2.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[build@localhost ~]$ printf '#include \nint main(){printf("Hello\\n"); return 0;}\n' > sample.c
[build@localhost ~]$ aarch64-rpi3-linux-gnueabi-gcc -o sample sample.c
[build@localhost ~]$ ls -l sample
-rwxr-xr-x. 1 build build 7672 May 22 08:19 sample
[build@localhost ~]$ file sample
sample: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 4.3.0, not stripped

Well, it is time to build some awesome stuff for Raspberry Pi 3, I guess, but this will be in another post. :)

Building a firewall? Simple and easy!

2016-05-22T10:00:00+10:00

I strive for simplicity since I am a strong believer that achieving a goal with the most simplest solution looks elegant, proves that you have deep knowledge on the subject, and overall is beautiful by itself. Additionally to this, a simple solution is easier to comprehend and to audit, hence it is much easier to ensure the security of such a solution.

Over the last decade, I stumbled upon numerous complicated firewalls erected on the NAT boxes with tens (sometimes, hundreds!) of rules describing the traffic flows and punched holes for some edge cases. Every time I wondered: what kind of a bug has bitten the person who composed such a convoluted ruleset that is a nightmare to manage?

In 99% of the cases, I was able to come up with a ruleset of usually less than 20 rules for the whole firewall to achieve exactly the same result. So, in this article I will explain my approach on building firewalls that are easy to support and to understand.

Before you dive into optimising (or creating) your firewall, there are some things you need to have a clear understanding of:

Networking

You do not need to be an expert or a guru, but you should know fundamentals about the protocols, standards, and internals of the networking related to the area you are going to secure with the firewall.

When you are in doubt, do not rely on an assumption: use search engines and locate the corresponding RFC, standard, and/or any other authoritative source of information documenting the protocol you are working with.

It is impossible to know everything, but the common fallacy is that people often create things based on wrong assumptions when it was a couple of clicks away to research and understand the subject. I have seen so many times when people are “designing” their firewalls without a slightest clue in regard to the standards covering interoperability of the hosts in the IP network.

For example, you can easily spot these “creations” if you look at the ruleset and see that there is a rule that drops ICMP packets unconditionally (or do not define any rules dealing with ICMP when the default policy is “deny”)
Traffic flows

Think of a firewall as a filter that manages traffic flows (Figure 1 shows a good visual representation of how traffic passes through the Linux netfilter subsystem).

Before you can make a decision in regard to whether to allow or deny something you must know what your are dealing with and what outcome you want to achieve.

It would tremendously help you later if you could simply draw a diagram of your network and overlay it with a layer documenting the inbound and outbound flows of each node on the diagram. Such a “data flow” diagram is supposed to be a part of any solution documentation involving network infrastructure, but in reality most of the enterprises (at least the ones I worked with) simply forget to generate one.

On such a diagram you would be able to see straight away what kind of legitimate traffic is going to cross your firewall and what traffic is not supposed to reach your firewall at all.
Goals

This may sound obvious, but one of the most important things is to know exactly what you are trying to achieve. You are not building a firewall for the sake of building a firewall, are you?

Frankly speaking, there are just a handful number of scenarios for creating a firewall (I fail to come up with more than five at this point) and all of them are very simple.

If your network provides any services to the external network (e.g. Internet) there are just two options I can think of:
1. Allow external entities to access services on defined public endpoints (an inbound flow) and allow unrestricted outbound traffic from your network to the external one (an outbound flow);
2. Allow external entities to access services on defined public endpoints and restrict outbound traffic from your network to a defined set of external endpoints.
If your network does not provide any services to the external network and only consumes resources from it, I see only two options:
1. Disallow any access from the external entities to your resources, but allow your network to access the external endpoints and receive responses from there;
2. The same as above, but restrict the outbound traffic to a selected set of the external endpoints.
The fifth scenario is the one I personally do not use and is the blacklist approach:
1. allow everything in each direction and block communication for the specific endpoints only (be they external and/or internal).
Tools

Last, but not least, is the requirement to know the corresponding toolset you are going to utilise to achieve your goals.

There are numerous high-level frameworks which are supposed to make systems administrator’s life easier (e.g. RHEL/CentOS/Fedora are using firewalld on top of iptables), however, personally I prefer to work with iptables directly (thus, my systems do not have firewalld installed).

Before you start implementing the rules it often helps to describe your firewall ruleset in simple sentences, e.g.:

deny any incoming traffic to our network through the firewall unless it is explicitly allowed by the rules;
allow any outbound traffic from our network through the firewall unless it is explicitly denied by the rules;
deny any forwarding of traffic through the firewall unless it is explicitly allowed by the rules;
allow anyone on the Internet to connect to the webserver on ports 80/tcp and 443/tcp;
allow our office to connect to the firewall via SSH on port 22/tcp;
allow traffic from our network to pass the firewall toward the Internet;
allow the back channel traffic from the resources on the Internet to our network;
…

And then research how each of these sentences can be implemented using the configuration language of the tool. The point here is that you need to take one step at a time and translate the logic exactly so you do not deviate from your goals – you will be able to optimise the result at a later stage.

Well, it is time for a real world example, I guess. The following snippet is used as the skeleton for the iptables ruleset on systems I manage (this snippet assumes that 192.168.0.0/16 is the internal network and that the ruleset is installed on the NAT instance which is a gateway to the Internet):

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state INVALID -j DROP
# If it was already established pass bits through
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# Allow for the local traffic
-A INPUT -i lo -j ACCEPT
# Let's MTU discovery and other network management to work properly
-A INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
# Allow remote SSH logins on the specified port (should be <1024 and would require sshd re-configuration if the port is not standard)
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
# Allow DHCP client to get its information
-A INPUT -i eth0 -p udp -m state --state NEW -m udp --sport 67:68 --dport 67:68 -j ACCEPT

# Reject anything that we did not specifically allow above
-A INPUT -j REJECT --reject-with icmp-host-prohibited

# This is a NAT box, so we accept packets from our network designated to the world, but reject any other forwarding attempts.
-A FORWARD -i eth0 -s 192.168.0.0/22 ! -d 192.168.0.0/22 -j ACCEPT
-A FORWARD -i eth0 ! -s 192.168.0.0/22 -d 192.168.0.0/22 -j ACCEPT
# If there is a MTU mismatch between the NAT box and hosts behind it we want to allow ICMP for the MTU discovery
-A FORWARD -i eth0 -p icmp -m icmp --icmp-type any -s 192.168.0.0/22 -d 192.168.0.0/22 -j ACCEPT
# We may be interested in anything that is rejected in this chain, so let's log it
-A FORWARD -j LOG

# We do not allow to forward anything else
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 192.168.0.0/22 ! -d 192.168.0.0/22 -j MASQUERADE
COMMIT

There are a few things to note:

default policies for INPUT and FORWARD are set to ACCEPT while the last rule in each chain is set to REJECT. The reason for this is that the default policy can be either ACCEPT or DROP with the latter silently dropping packets. Dropping packets makes it hard to investigate network issues, hence I prefer REJECT instead.
if you need to open more ports the best place to insert your rules would be between SSH and DHCP rules in the INPUT chain
if you are setting up a firewall on the server (so no traffic forwarding is needed) you may want to either drop all rules from the FORWARD chain and set the default policy for that chain to DROP, or you can simply leave the last two rules (LOG and REJECT) in the chain removing the rest. Also, the nat table is not needed on the servers, so should be omitted.
the above ruleset is not strict enough in regard to filtering ICMP traffic and allows all types of ICMP, perhaps it could be further tightened to allow only ICMP types 0 (reply), 3 (destination unreachable), 8 (request), 11 (time exceeded), and 30 (traceroute) to pass through the firewall.

So, here you have it a solid firewall ruleset consisting of just 13 rules.

Transparent SSH host-jumping (Advanced)

2015-08-06T10:00:00+10:00

In this brief article I am going to describe how I resolved a nagging issue I had with setting up access to hosts which are not directly reachable, but where you need to forward your connection through an intermediate host.

Previously, I was using the local SSH port-forwarding technique (although I was configuring hosts I connect to in the ~/.ssh/config file instead of using the command-line options). However, this approach turned out to be quite inconvenient since every time I wanted to connect to a new host (and, possibly, through a new intermediate host) I had to edit my SSH configuration file and add something like the following:

Host intermediate
    HostName 192.168.1.1
    HostKeyAlias intermediate
    LocalForward 10001 target:22

Host target
    HostName 127.0.0.1
    HostKeyAlias target
    Port 10001

Upon closer examination of my day-to-day routine I found two things that frustrated me the most:

My ~/.ssh/config file was growing uncontrollably and became hard to navigate;
Each time I needed to connect to the target host through the intermediate host I had to open two sessions with one of them being idle most of the time.

After a while I stumbled upon an article describing quite a generic way to tunnel through an intermediate host and found the approach quite convenient for the day-to-day use. So, I have added the following block into my ~/.ssh/config file just before the Host * section:

Host */*
    ProxyCommand ssh $(dirname %h) -W $(basename %h):%p

From that point on, I could connect to a target host via an intermediate one by simply executing the following command:

[user@localhost ~]$ ssh user@intemediate/target

The configuration with the ProxyCommand directive was spawning two ssh processes with one connected to the intermediate host in the background and the other proxied through the intermediate host and connected to the target running in the foreground, so from my point of view I had just one terminal session open. The configuration allowed to chain as many hosts as I wanted, e.g.:

[user@localhost ~]$ ssh user@hostA/hostB/hostC/hostD

The above would result in three ssh processes running in the background (the first connected to hostA, the second connected to hostB proxied through hostA, and the third connected to hostC proxied through hostB) and one foreground process which was connected to hostD proxied via hostC. This is great and quite flexible to use, however, this approach has a number of limitations:

you cannot specify different ports for different hosts in a chain;
neither can you use different login names for different hosts in the chain;
establishing connection to different chains sharing a part of the chain would not reuse already established connections, i.e. slow connection times.

Personally, I am using the same login name and the same ports on hosts I am accessing, so the first two items were not an issue for me, but the last one was irritating enough and I decided to figure out whether it is possible to optimise it. After a bit of reading the documentation and a few attempts I came up with the following configuration block in my ~/.ssh/config file (remember, this block should be placed before the Host * one):

Host */*
    ControlMaster auto
    ControlPath   ~/.ssh/.sessions/%r@%h:%p
    ProxyCommand /bin/sh -c 'mkdir -p -m700 ~/.ssh/.sessions/"%r@$(dirname %h)" && exec ssh -o "ControlMaster auto" -o "ControlPath   ~/.ssh/.sessions/%r@$(dirname %h):%p" -o "ControlPersist 120s" -l %r -p %p $(dirname %h) -W $(basename %h):%p'

Let’s review it line by line, so the logic is clear:

Host */*

This host definition block would catch any host specified on the ssh command line when the host name matches the */* pattern, so ssh hostA/hostB/hostC will be matched as hostA/hostB being the first part before / and hostC as the second part after /. Due to a recursive call to ssh (see below) this block will be recursively applied to all hosts in the specified chain.

ControlMaster auto

This directive instructs ssh to try to reuse an existing control channel to communicate with the remote host, and if such a channel does not exist it will be created, so further connections to the same remote host would benefit from a speedup provided by the already established connection.

ControlPath ~/.ssh/.sessions/%r@%h:%p

This directive provides ssh with the location of the control channel socket file. The socket file should be unique for each remote host. Since we are reusing the existing connection and skipping the authentication the socket file should be tagged with the corresponding login name, this is why we are using %r (remote login name), %h (the remote host name), and %p (the remote port) as part of the file name. Please note that due to our usage of “/” as a host separator in the chain the path constructed here will have a subdirectory defined in the middle of the %h expansion. ssh would not automatically create that subdirectory, so it is something we need to address (see below)

ProxyCommand …

This is the heart of the whole block. I am starting this proxy command with /bin/sh -c '…' since ssh is exec()uting the specified command (this replaces the spawned shell and makes it impossible to conditionally chain commands), therefore I am using the shell binary as the proxy command to get the ability to script my logic. Then I am creating the required directory structure for the control channels under ~/.ssh/.sessions (note the -p argument to mkdir, this will create all the missing parts of the specified tree, but also would silence mkdir in case all of the directories already exist). It is worth to mention that with this mkdir command I am creating the subdirectory for the ControlPath defined for the enclosing Host */* block.

The second part of the command line is conditionally executing ssh if mkdir did not report any issues. It is good to execute ssh here since we do not need a redundant shell hanging around in the process tree. In this recursive ssh call we explicitly specify that we also need multiplexing of the control channels created by the parent connections (they are “parent” since this is the connection that established first and which enables access to the hosts further down the specified chain) as well as we explicitly specify the location of the control channel (note that since it is a parent connection we are stripping the rest of host names from the %h macro using dirname.

Finally, the third explicitly specified directive is ControlPersist which is set to 120s. This directive instructs ssh to stay in the background and maintain the control channel in case we decide to reuse it, but if no activity on the control channel is detected for 2 minutes the ssh process would terminate. Without this directive the moment you close the connection which was the master connection all dependent connections would also be closed, e.g. if you have two sessions: one to hostA/hostB and the other to hostA/hostC, the moment you closed the first connection the second one will be immediately terminated if you do not have the ControlPersist configured.

The rest of the ssh arguments is obvious: we connect to the first host in the provided host chain (we are extracting that part with dirname %h) and we are proxying stdin/stdout to the last host in the supplied chain with the -W option.

Basically, the control flow when you do ssh user@hostA/hostB/hostC is the following:

ssh matches the */* pattern against the provided host name (hostA/hostB/hostC)
ssh tries to reuse the control channel by attempting to open the ~/.ssh/.sessions/user@hostA/hostB/hostC:22 socket, if successful the connection is established and the command prompt is displayed to the calling user, otherwise the execution continues
ssh executes the defined ProxyCommand command
the first part of the command creates ~/.ssh/.sessions/hostA/hostB if it is not there
the second part executes
```
ssh … -o "ControlPath ~/.ssh/.sessions/user@hostA/hostB:22" … hostA/hostB -W hostC:22
```
This will initiate another round of the above steps, but with a shorter chain and it will be recursive until there is just a single host left, e.g. when we ascend to hostA as the host to connect to.
now, with connected stdin/stdout to port 22 on hostC (in the last iteration) ssh performs the authentication against hostC
if authentication is successful ssh creates the ~/.ssh/.sessions/user@hostA/hostB/hostC:22 control channel socket and becomes the master of that control channel
a command prompt is displayed to the calling user

I hope this little trick will save you some time and will make your life easier. :)

Should we use "sudo" for day-to-day activities?

2015-07-18T20:04:00+10:00

Table of Contents

History
People are saying “sudo” is good. Is it true?
What is the problem with the “sudo” approach?
The light side of “sudo”. Is there one?
How can we improve the security of our systems in relation to the “sudo” usage?

None of the systems I administer or supervise have sudo installed with the SUID bit set.

Every time I answer a question on how to do privileged work on these systems (i.e. do tasks that require administrator privileges) with a proposal to SSH under a privileged account directly to do such a work, whoever asked the question starts to blabber about how insecure that is, that one should use sudo, and that nobody should ever login directly as root.

I have spent quite some time explaining the misconception behind so-called “secure way to access systems through sudo”, so I decided to write up an article that describes the issues of using that approach and why using sudo is actually less secure than a direct SSH access approach.

History

The following is based on my personal recollection of the history around early 90’s with some references to documents I could quickly find. Unfortunately, this is one of the topics that is not so easy to reconstruct in full detail. Although there may be minor inaccuracies, the outlined view on the history should be pretty close to the true events that took place.

Many years ago (but not that far in the past) system administrators were using telnet and rsh to access and administer their servers. Networks were simpler and traffic sniffing techniques were common. Therefore, it was very easy to eavesdrop on a root account login and to automate the task of gathering the credentials. To mitigate the issue to some extent the following approach was proposed:

login as a non-privileged account first;
do some stuff;
if you need privileged account access (e.g. root) switch to it using su.

The idea behind this proposal was that such a sequence makes it harder for a traffic listener to catch root’s credentials on the wire due to the login sequence mixed in to the stream of other activities the systems administrator was doing before escalating their account to the privileged user.

At approximately the same time discussions started in regard to the dangers of working as root, that people were tending to work as root for prolonged periods of time performing tasks that did not require escalated privileges without proper justification for that behaviour. As the result of these discussions the sudo utility was born. The utility allowed to bind root privileges to a restricted set of commands and maintained access controls through its configuration file in order to allow or deny access to the defined functionality to specific users/groups. This allowed systems administrators to delegate some of the privileged routines to less privileged user/groups. (See more on the history of sudo if you are interested).

Somewhere in early 90’s the two approaches were merged, so the approach became:

login as a non-privileged account and do your every day tasks;
once a privileged operation is required execute it through sudo.

Although it seems like a minor optimisation, in fact, this introduced a major security issue: previously, the attacker should have needed to listen for traffic in attempt to figure out when the victim is executing su and is providing root’s password, but now they did not need to do anything except of capturing user’s password at the beginning of the session (the same problem the “su-after-normal-login” approach was trying to solve just reappeared :) ) since once you know user’s password you can use sudo which authenticates you with user’s password!

In 1995, Tatu Ylönen as a response to the issues related to exchanging the credentials over non-protected, easily sniffable networks released his first implementation of the SSH protocol as freeware to the public. Over the next five years SSH was adopted worldwide and it eventually replaced telnet/rlogin/rsh for the remote access and management activities in most places around the globe.

However, due to inertia the recommended approach of logging in as a non-privileged account and escalating privileges later somehow survived, is still followed, and often people don’t even try to analyse and see the flaws of the approach.

People are saying “sudo” is good. Is it true?

^{(or looking critically into some common misconceptions re: “sudo”)}

So let’s look at the most commonly used “pros” in favour of using sudo. For example, Ubuntu’s community help page provides a nice, aggregated list of benefits provided by sudo usage in their default installation. Let’s walk through all of them and see if they are real benefits at all:

The installer has fewer questions to ask.

This is a very questionable “benefit” to the end user since it implies that the system would do more stuff with escalated privileges behind the scenes without the user even knowing it. From the security standpoint it just silently expands the attack surface.
Users don’t have to remember an extra password for occasional use (i.e. the root password). If they did, they’d be likely to forget it (or record it unsafely, allowing anyone to easily crack into their system).

This is another hard to comprehend “benefit”. Given that we now have SSH and the recommended best practice is to use SSH keys instead of passwords, sudo requires users to memorise yet another password.

Moreover, the argument in regard to users’ likely behaviour is purely speculative and assumptive — to the same extent we can assume that password policies make it hard for users to memorise their passwords and they would be likely to forget the password (or record it unsafely, …).

The truth is, the key based authentication solves the issue of the requirement to have multiple passwords for multiple accounts.

There is one place, however, where the discussed “benefit” is actually applicable — local console access (no key authentication there, usually), but even there the best practice would be to login directly as root on the virtual console if there is such a need (the reason for that is quite complicated and in short could be described as the following: there are multiple checks and assumptions in the kernel code and the accompanying C library on allocating a terminal, spawning a process, etc. for root over the same actions performed for a non-privileged user).
It avoids the “I can do anything” interactive login by default. You will be prompted for a password before major changes can happen, which should make you think about the consequences of what you are doing.

This statement also assumes that for some reason people would prefer to always login as root and do all of their work under that account.

In reality users are so used to prefix almost any failed command with sudo that this “benefit” can be considered as the quite opposite item, against sudo.

In a properly configured environment you explicitly need to login as the privileged account to do functions that require privileges.
sudo adds a log entry of the command(s) run (in /var/log/auth.log). If you mess up, you can go back and see what commands were run.

This statement is also somewhat true, but it does not defend the sudo usage. A proper auditing subsystem is what keeps audit logs no matter how activity was performed. Logging of executed commands for user’s history reasons is the job for the shell.

The ability of logging is so limited in sudo that it cannot be used for anything except a substitute for the shell history. Just imagine the following scenario — a user executes:
```
[user@localhost ~]$ sudo less /var/log/messages
```
and then types “!” followed by the Enter/Return key — the user effectively now in the root shell and what sudo will log into its logs has nothing to do with what user actually did.

Instead of relying on sudo’s logging abilities, one should configure auditd and send events to a centralised log aggregator to get audit logs that can be trusted.
On a server, every cracker trying to brute-force their way in will know it has an account named root and will try that first. What they don’t know is what the usernames of your other users are. Since the root account password is locked, this attack becomes essentially meaningless, since there is no password to crack or guess in the first place.

This is such a weak attempt to bring host security into the play that it is hard to comment on it without dropping a tear :).

First, use SSH keys and disable the password authentication on the server — this (and not some security through obscurity) will defend the system from the brute force attacks.

Second, protect your remote access entry points with properly configured firewall and allow remote access from a defined list of locations only (i.e. whitelist authorised locations) — this will shrink the possible attack surface of the SSH service considerably.

Third, nobody actually prevents you from having a different user name with uid 0. The root account is just an accepted default, the system does not really care since it works with numerical UIDs, not with names. See the next point for an option how to achieve that.
Allows easy transfer for admin rights by adding and removing users from groups. When you use a single root password, the only way to de-authorize users is to change the root password.

This is an interesting one: half of the statement is true, another is misleading.

The true part is that you can easily delegate privileged operations using sudo. All in all, it was the primary goal and requirement of the tool creation to provide access delegation.

The misleading part is that the alternative is to use a single root account/password. The truth is that nobody is limited by a single root account: you can create as many as you want and each of them could have their own distinct password, e.g.
```
[root@localhost ~]# useradd -om -u 0 -g 0 -s /bin/sh new_root
[root@localhost ~]# passwd new_root
```
Better yet, do not set the password and lock the account instead with usermod -L new_root since we are using keys, remember?

This approach also provides additional accountability since users will have separate shell histories, their login attempts will be clearly logged under separate names, etc.
sudo can be setup with a much more fine-grained security policy.

Another half-truth in the list of “benefits”. The statement is incomplete and lacks the part it is comparing the functionality to. If we are comparing a legacy Unix access control system with sudo, then yes sudo is much more configurable. If we compare sudo with, say, SELinux or GRsecurity’s RBAC – sudo will lose since both have much more fine-grained security controls than sudo.
The root account password does not need to be shared with everybody who needs to perform some type of administrative task(s) on the system (see the previous bullet).

As with item #6 this statement assumes for some obscure reason that there can be just a single root account in the system. Therefore, the result of such a logical exercise is also questionable since it is based on a wrong assumption.
The authentication automatically expires after a short time (which can be set to as little as desired or 0); so if you walk away from the terminal after running commands as root using sudo, you will not be leaving a root terminal open indefinitely.

This statement also compares sudo with something reader cannot compare to. Moreover, it mixes up two logically unrelated things: the credentials expiration and the terminal security. While it is great that sudo implements the former, the latter is usually addressed by entirely different means: starting with auto-logout functionality, lock screen, physical security, etc.

Well, we can continue to critically assess other common statements in regard to sudo made mostly by people who do not have any strong Information Security background, but it would be a waste of time for the readers.

You can always raise a question regarding some particular claim and/or assumption related to sudo and if it is interesting I would add it (and the corresponding response) to this article.

What is the problem with the “sudo” approach?

Well, there are several in fact. The most pressing issue is that the usage of sudo (or su, or any other utility that has its SUID bit set) is crossing the security boundary from the less privileged account to the more privileged account. This opens doors (or widens the attack surface) to privilege escalation techniques. In plain English it makes the non-privileged account to be essentially equal to the privileged one, let me explain by a fairly simple example:

imagine if you are a developer and you work on a server under your non-privileged account;
as a part of your daily routine you need to download some third party package and install it (we are going to leave out all the security complexities involved with such an activity like verifying signatures, using a separate instance to prepare a package for deployment, etc.);
the installation of the package usually requires executing some third party code under your non-privileged account. This code was not written by you and there is a high chance that you did not read/verify the foreign code line by line in order to ensure that it does not do anything malicious since this would be quite time consuming, would require a lot of effort, and your team has more important priorities than this (remember, this is an example based on situations you would encounter in the real world, which is by no means perfect);
it happens, that that particular third party was compromised and some malicious code has been injected into the package installation routines;
after the execution of the installation routine (and the malicious code for that matter) your ~/.bashrc (for example) is modified in such a way that each time you login it starts up a key logger or some other kind of remotely controllable piece of software that talks back to its master;

So far we just got an issue localised to this non-privileged account only (with a possibility to spread across the entire fleet of servers that non-privileged account has access to in case of the NFS mounted home directories). Is this bad? Yes, it is since it may disrupt this particular project, steal reachable sensitive information, could be used as a trampoline to jump start further research and exploitation of other vulnerable resources. Is it critically bad? Not necessarily. If the systems are built properly with host-based security in mind, if the proper privilege separation techniques are used throughput the company infrastructure, etc. the impact is localised and with proper monitoring systems it would be detected eventually and investigated (keep in mind that it is really hard for a non-privileged account to hide their activities from more privileged processes).

Now, let’s add sudo to the mix, e.g. suddenly the developer decided to install an additional library package into the system. So, what would happen next? You guessed it: the developer would need to type in their password to convince sudo that they are “allowed” to do such a privileged thing as installing a system package, the malicious software installed by the attacker would happily intercept that and send it back to its master.

From this point on, the attacker has the account password of the account where their software runs and which they control. The attacker can now utilise sudo powers at their will. The security impact would be ranging from “high” to “extreme” depending on how committed the attackers are.

Following the logic, why would we want to introduce an additional complexity that does not address the issue it was supposed to address, which is “to limit exposure of the root account”?

The light side of “sudo”. Is there one?

So, is sudo any good for anything? Actually, yes, it is. All in all, sudo is a tool that attracts a lot of attention from the security researchers and auditors, its codebase got numerous peer reviews and the functionality the tool provides can be used for good. The following scenarios come to my mind right away:

On a SELinux enabled system it seems that sudo is the only properly implemented and reliable way to assume a different SELinux role. All other mechanisms are either lacking in the functionality or just do it half way leaving some artefacts behind;
Unless some effort is put into tweaking the way PAM (Pluggable Authentication Modules, an authentication framework used on Linux, Solaris, and some other Unix-like systems) authenticates users on LDAP enabled systems there is no easy way to have two separate accounts (a privileged and a non-privileged account) for the same LDAP user. This could be addressed administratively (e.g. by defining additional privileged users in LDAP) or technically by implementing account names’ prefixes. However, if there are budgetary and/or time constraints to implement the proper security framework sudo with a quite restricted configuration would be an acceptable compromise.

Unfortunately, personally, I failed to find any other justified applications of sudo in a secure environment and would be happy to get some feedback if you have something in mind worth including into the list above.

It is worth to mention that in the first scenario (SELinux roles), sudo should not have the SUID bit set since normal SELinux users could assume roles with newrole. The difficulty arises when an administrator tries to assume a user role (e.g. such a role transition is not prescribed in the policy). One could argue that either the SELinux policy should be amended and, then, newrole could be used or such a behaviour of the administrator account should not be allowed at all. However, in real life, administrators need to see the world through users’ eyes from time to time to expedite the troubleshooting process, in these cases sudo will help, but since it is going to be executed by an administrator (a privileged account) to switch to a user (a low privileged account) there is no need for any other permissions as 0100 (execute for the owner) on the sudo binary.

How can we improve the security of our systems in relation to the “sudo” usage?

OK, so we got to the point where we are standing at a crossroad and we kind of established that excessive usage of sudo is a bad thing (for security-paranoid readers - read: “almost any usage of sudo except for changing SELinux roles on a SELinux-enabled system is a bad thing”). So, what is the alternative way of doing things? Well, there is a complex approach on addressing and minimising the privilege escalation risks and roughly it can be summarised as follows:

apply the least privilege principle to everything

For example, if you need to work on content, the account you use to log into the remote server should be allowed to do just that. This sounds a bit extreme, so we may re-phrase it as follows:

Each account should be provided the least number of required privileges to do specific tasks under that account and should not be used for anything else outside the defined set of activities.

Again, if you include “becoming root” into the list of defined activities it would kind of defeat the purpose, so please resist this temptation. :)
there should be a defined list of accounts which are allowed to do system-wide modifications (e.g. privileged accounts)

In line with the previous bullet point, these accounts must be used for these activities only (system updates, installing new software, modifying system-wide configuration). The expectation is that the privileged accounts are used on an occasion only;
privileged accounts during their activities should not cross the security boundary with the less privileged accounts.

This is needed to avoid attacks from the less privileged accounts toward more privileged (e.g. process hijacking, file races, etc.). Unfortunately, this point is a bit confusing without a proper explanation and that may take several blog posts to be fully covered, but in a nutshell it means that root should not touch anything writable by the less privileged account. Some examples of the bad and insecure behaviour include:
- changing directory to a directory writable by non-root,
- executing scripts from a directory writable by non-root accounts or scripts that are writable by non-root,
- copying a file/directory from/to a directory writable by non-root accounts,
- and so on.
if possible (and this is highly recommended) privileged accounts should be accessed from the trusted and secure machines.

It is really hard to define what the “trusted and secure machine” is, but generally it should be assumed
- that it is a laptop or a desktop station that is used entirely for the work purposes (no free time surfing on the leisure sites and stuff :) ),
- that proper firewall rules and protection techniques were used to secure the machine, and
- that the operator is security-minded person and does everything they can to ensure integrity and security of their machine and software installed on it.

I would be really happy to answer any question in regard to this article and provide any possible help in making our everyday system level activities more secure, so do not hesitate to comment or contact me.

SSH port-forwarding (Intermediate)

2010-12-10T10:00:00+11:00

In my previous blog entry I described some basic functionality of SSH in terms of port-forwarding. Now it’s time for a little bit more complex stuff.

In this article I will highlight:

(forward) piercing of a firewall (getting access to resources behind it);
dynamic port-forwarding (AKA proxy);
(reverse) piercing of a firewall (exposing your local services on the remote side).

Forward firewall piercing

Let’s start with the forward firewall piercing, since it is the easiest and was somewhat already described in my previous blog entry on this topic. Now, imagine that you already have SSH access to some host which is multi-home connected (e.g. the host is connected to more than one network). Let’s also assume that the host is a firewall and is masquerading other hosts in the internal network and is translating just a handful set of ports to the servers (looks familiar, doesn’t it? :) ). In other words, we are speaking of a standard firewall/NAT router.

Now, how can you access port 12345 on host behind the firewall given that this port is not “exported” by the NAT on the firewall? This is quite simple. Open a terminal window on your local computer and type the following:

[user@localhost ~]$ ssh -L12345:192.168.1.2:12345 joe@firewall.domain.tld

From now on, as long as your SSH session is up and running you will be able to reach 192.168.1.2:12345 by connecting to localhost:12345 (i.e. the 12345 port on your local computer). Indeed, for this to work you need SSH access anywhere inside the protected network (not necessarily on the firewall itself) and if the firewall blocks any SSH access, you are out of luck.

Dynamic port-forwarding

There are at least two usage patterns where I find SSH’s ability to forward requests to many ports useful:

I need to connect to different services (possibly on different hosts) inside the protected network (as per the configuration described above) and I don’t want to specify all of them on the command line;
I need to access some resource which is Geo-protected (e.g. allows access from a particular part of the world), e.g. want to watch US Netflix being physically in Australia

In both cases, you use the following SSH command in your terminal window:

[user@localhost ~]$ ssh -D3128 joe@relay.domain.tld

As long as you SSH session is active, you can use port 3128 on your local machine as SOCKS4/SOCKS5 proxy (e.g. you can configure proxy settings in your browser to use localhost:3128) and browse the Net through your SSH connection, and all your requests will look like they are coming from relay.domain.tld.

There are many uses for this: for example, some payment processors won’t allow you to pay for goods if you are trying to pay through them from some countries even if you are a legitimate user, another use case is when you are concerned re: your privacy – you can conceal your actual location by building a chain of SSH tunnels and access the desired web site through this chain :) .

Reverse firewall piercing

Finally, what if you are behind a very strict firewall that limits almost everything, but you need to provide some services to the outside world from your computer (e.g. sharing your access to company’s confidential information to folks from WikiLeaks… just kidding :) )?

To achieve this you need to have an SSH account somewhere in the Net – just Google for “free ssh account” and you will surely find one for yourself).

Now, when you have the account, you can execute the following on your local computer (which is inside that highly secure network :) ):

[user@localhost ~]$ ssh -R:60000:127.0.0.1:22 joe@friendly.domain.tld

The above command will setup such a configuration that connecting on port 60000 at friendly.domain.tld will forward traffic to your local machine’s port 22 (which is behind a firewall) – this will work as long as your SSH session to the friendly.domain.tld is active. Unfortunately, there are several pitfalls in this approach, but they are all resolvable.

Firstly, you need to ensure that friendly.domain.tld is using a recent version of SSH daemon, otherwise you will be limited to bind only to the loopback interface on the remote host.

Secondly, even if they are using a recent version of the SSH daemon, they can disallow such binding (e.g. setting “GatewayPorts no” in /etc/ssh/sshd_config), and, again, you will be restricted to the loopback interface only.

Finally, you need to find such a friendly host which allows you to connect to the bound ports from the outside (many public ones have a firewall rule preventing such access in order to prevent abuses of their services).

All in all, the best option to try this is to have your own host somewhere (e.g. purchase a small virtual environment from some hosting provider or create an instance on AWS EC2), then you will be able to configure the remote side the way you want it!

Anyway, even if you fail to find a host that allows you to expose your service to the public, you still should be able to access it yourself – using the forward firewall piercing technique described at the beginning of this post:

# from your home computer
[user@localhost ~]$ ssh -L60000:127.0.0.1:60000 joe@friendly.domain.tld

Once it is done your computer in the highly secure network should be reachable via SSH on your localhost:60000, e.g.

[user@localhost ~] ssh -p6000 my_account@localhost

Where my_account is a user allowed to SSH into your computer in the highly secure network.

SSH port-forwarding (Basic)

2010-11-29T10:00:00+11:00

I think all of you are using SSH in your daily routines. However, do you use its full potential? Today’s topic is the SSH port-forwarding feature and how it can be use to achieve some interesting configurations.

I’m sure most of you are aware of the feature, but how many of you are using it? Personally, I’m a bit obsessed with it and have found numerous cases where this feature of SSH is a real life saver.

Let’s start with simple things and imagine that you have a server where you are running MySQL (as a backend for your website) and that you are not comfortable to work with MySQL neither through the MySQL command line interface nor through a web-interace like phpMyAdmin. Instead you prefer to use, say, MySQL WorkBench or something similar running on your local computer.

How one could do this? Well, I’ve seen a lot that people tend to configure the MySQL server to listen on a public interface (i.e. an interface that is reachable from the Internet), then access it from their computer directly. It’s not that bad if you have a static IP assigned to your computer and you added a firewall rule protecting the MySQL port on the server to be reachable from your IP address only, but what if you are behind some kind of NAT and the same IP address is shared among others in your network? What if you don’t have a static IP and the dynamic IP range assigned by your ISP is too broad? Overall, I personally don’t like running MySQL on a public interface since there were publicly known security vulnerabilities in the MySQL binary protocol, and I can assure you there will be more discovered over the time.

So, what should we do to access our remote database securely and conveniently? Run the MySQL server on 127.0.0.1:3306 (well, if your web server is located on another machine, you may want to bind MySQL to some internal interface – the network that links your servers only, e.g. 192.168.1.1:3306), and then use the SSH port-forwarding feature to setup a tunnel between your remote database and your local computer:

[user@localhost ~]$ ssh -L3306:127.0.0.1:3306 user@server

Here, -L3306:127.0.0.1:3306 says that we want 127.0.0.1:3306 to be “mapped” to our local host to port 3306. While the SSH session is active any packet sent to localhost:3306 (on your computer) will be forwarded over the SSH encrypted channel to the remote server and then will be fed to remote’s 127.0.0.1:3306. This means that as long as your SSH session is alive you will be able to work with the remote MySQL server like it’s running on your local computer and you can use any MySQL tools locally.

Well, the above command was given for Linux, macOS, and other Unix-based systems. For Windows, you can use PuTTY to archive the same. I found the following instructions for PuTTY.

OK, this is not a new topic and it was described many times here and there on the Internet, but I decided to write this article as a foundation for my further articles on this topic which will cover more advanced usage of the SSH port-forwarding feature. For example, I intend to document a configuration of the MySQL replication setup between two servers located at two different data centres where each server is a part of a server farm protected by a firewall.

HOWTO: VMware Player as a remote console (VNC)

2010-11-24T10:00:00+11:00

VMware VMs from a Linux-based PC. Tags: howto, hack Alias: /2010/11/howto-vmware-player-as-remote-console.html

Since I am doing a lot of remote systems administration tasks due to the nature of my IT consulting work and since I am also running Linux on all my computers I was looking for a native way how to get a remote console to VMware VMs from linux.

After some searching I found that VMware Player (which has native binaries for Linux) can be used as a VNC client to get to VMs consoles. However, once I have downloaded VMware Player’s bundle and was faced with its requirement to run the installation script as root I became quite unhappy with an idea of running some proprietary software on my machine as root, especially after looking into the bundle and the way the installation script was written. Moreover, there was no need for other parts of VMware Player – I just wanted to have a small tool to be able to hook the remote consoles up under my lovely Linux environment. Therefore, I decided to take a challenge and to tweak the installation so it will be possible to install the whole thing as a non-privileged user. Another sub-goal was to strip the installation further down and prepare a small package with only components needed for remote console sessions.

If you are not concerned about security (and integrity) of your system, e.g. you are fine with the re-installation of the whole system, then it will be cheaper to just install the VMware Player under the root account. In this case you don’t need to read any further since what I am describing below is for those brave hearts who value their systems and who do not want to give a chance to mess their systems up by running low-quality custom installation scripts as root.

Well, if you are still reading, then I hope that my research on this topic and the how-to I have spent considerable time to come up with is worth something and will be of some help to you.

Our starting point is a Linux-based system (it does not matter what distribution you are running, but I did everything on a customised ALT Linux’s RPM-based distribution) running on an x86 compatible hardware (mine was 32-bit, but I see no issues with 64-bit ones).

The first step is to download VMware Player for your architecture, set proper permissions on the downloaded file, and then extract the payload as follows (you need to ensure that you have at least 40MB of free space on /tmp, BTW):

[user@localhost ~]$ chmod 0700 ./VMware-Player-3.1.3.324285.i386.bundle
[user@localhost ~]$ ./VMware-Player-3.1.3-324285.i386.bundle --console -x $(pwd)/vmplayer
Extracting VMware Installer...done.
No protocol specified
No protocol specified
[user@localhost ~]$ ls -l vmplayer
total 20
drwxr-xr-x  8 vmware vmware 4096 Nov 23 09:01 vmware-installer
drwxr-xr-x  4 vmware vmware 4096 Nov 23 09:01 vmware-ovftool
drwxr-xr-x  5 vmware vmware 4096 Nov 23 09:01 vmware-player
drwxr-xr-x 11 vmware vmware 4096 Nov 23 09:01 vmware-player-app
drwxr-xr-x  3 vmware vmware 4096 Nov 23 09:01 vmware-player-setup

So far so good. We now have the whole bundle unpacked into the specified directory and we are interested in just two subdirectories: vmware-player and vmware-player-app, the rest is not related to the functionality we are looking for.

Now, let’s pick up all parts from which we will build our future “VMware remote console” tool. To make it easier create a dedicated subdirectory, e.g. vmrconsole, with the following structure:

[user@localhost ~]$ mkdir -m700 ~/vmrconsole
[user@localhost ~]$ mkdir -m700 ~/vmrconsole/{bin,etc,lib,share}
[user@localhost ~]$ ls -l ~/vmrconsole
total 12
drwx------ 2 vmware vmware 4096 Nov 24 01:21 bin
drwx------ 2 vmware vmware 4096 Nov 24 01:21 etc
drwx------ 2 vmware vmware 4096 Nov 24 01:21 lib
drwx------ 2 vmware vmware 4096 Nov 24 01:21 share

From now on we are going to populate these directories with files from the unpacked bundle.

The first file we are interested in is appLoader – this is the primary executable by the way, we need to copy it to our bin directory and then try to run it:

[user@localhost ~]$ cp ~/vmplayer/vmware-player-app/lib/bin/appLoader ~/vmrconsole/bin/
[user@localhost ~]$ chmod 0700 ~/vmrconsole/bin/appLoader
[user@localhost ~]$ ln -s appLoader ~/vmrconsole/bin/vmplayer
[user@localhost ~]$ ~/vmrconsole/bin/vmplayer 
[user@localhost ~]$ echo $?
255
[user@localhost ~]$

Huh, this is not very informative, is it? The binary silently exits with error code of 255. Well, you may get other errors at this stage if you don’t have all the required shared libraries installed on you system – however I doubt it since the requirements of this binary are pretty reasonable: glibc and zlib.

OK, let’s take a peek inside and figure out what is going on (originally I used strace with logging to a file, but to keep this article reasonable short I am highlighting important things only):

[user@localhost ~]$ strace -f -eopen ~/vmrconsole/bin/vmplayer 2>&1 | tail -2
open("/etc/localtime", O_RDONLY)        = 6
open("/etc/vmware/config", O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied)

It looks that vmplayer wants to access a global config file and does not try to look for an alternative, home directory based one. Well, this is understandable since VMware folks did not expect it to be run as a non-privileged process, but we need to deal with this somehow. What are our options here? The simplest option I could think of at the moment is to substitute the hardcoded absolute path inside the binary with something relative:

[user@localhost ~]$ strings ~/vmrconsole/bin/appLoader | fgrep /etc/vmware | uniq -c
      1 /etc/vmware/config
      1 /etc/vmware/icu
      1 /etc/vmware/ssl/rui.crt
      1 /etc/vmware/ssl/rui.key
      1 /etc/vmware/ssl/dh512.pem
      1 /etc/vmware/ssl/dh1024.pem
[user@localhost ~]$ sed -i 's,/etc/vmware,..//////etc,g' ~/vmrconsole/bin/appLoader

The trick here is to substitute one string with another of the same length (we are modifying a binary so we do not want to mess offsets up) and luckily enough we can use whatever number of slashes we want – they all are considered as a single separator nevertheless. OK, we could have used a hex editor and could have terminated strings with a NULL byte, but the point is that the approach I took is the quickest and is working well. Let’s run the modified binary through strace again, but this time we need to be prepared for the changed behaviour:

[user@localhost ~]$ cd ~/vmrconsole/bin/
[user@localhost ~/vmrconsole/bin]$ touch ../etc/config
[user@localhost ~/vmrconsole/bin]$ strace -f -emkdir,lstat,open,write ./vmplayer 2>&1 | tail -12
mkdir("/tmp/.private/vmware/vmware-vmplayer", 0700) = -1 EEXIST (File exists)
open("/tmp/.private/vmware/vmware-vmplayer/appLoader-9245.log", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE, 0644) = 5
open("/etc/localtime", O_RDONLY)        = 6
write(5, "Nov 24 01:53:09.824: app-3077760"..., 121) = 121
write(5, "Nov 24 01:53:09.824: app-3077760"..., 60) = 60
write(5, "Nov 24 01:53:09.824: app-3077760"..., 71) = 71
write(5, "Nov 24 01:53:09.824: app-3077760"..., 57) = 57
write(5, "\"\n", 2)                     = 2
write(5, "Nov 24 01:53:09.825: app-3077760"..., 82) = 82
open("..//////etc/config", O_RDONLY|O_LARGEFILE) = 6
write(5, "Nov 24 01:53:09.825: app-3077760"..., 89) = 89
write(5, "Nov 24 01:53:09.825: app-3077760"..., 73) = 73

Looks better, does not it? Our binary found the config file and was able to open it, however it still produces no output, but it reports something to a log file:

[user@localhost ~/vmrconsole/bin]$ tail -7 /tmp/.private/vmware/vmware-vmplayer/appLoader-9245.log
Nov 24 01:53:09.824: app-3077760704| Log for VMware Workstation pid=9245 version=7.1.3 build=build-324285 option=Release
Nov 24 01:53:09.824: app-3077760704| The process is 32-bit.
Nov 24 01:53:09.824: app-3077760704| Host codepage=utf8 encoding=UTF-8
Nov 24 01:53:09.824: app-3077760704| Calling: "./vmplayer"
Nov 24 01:53:09.825: app-3077760704| Using configuration file ..//////etc/config.
Nov 24 01:53:09.825: app-3077760704| libdir entry was not present in ..//////etc/config.
Nov 24 01:53:09.825: app-3077760704| Unable to lookup library directory.

I do not know much about VMware Players config file but according to the log message it wants some variable called libdir and this variable should point to the library directory, so let’s introduce such a variable and try to execute the binary again:

[user@localhost ~/vmrconsole/bin]$ echo 'libdir = ..' >> ../etc/config
[user@localhost ~/vmrconsole/bin]$ strace -f -emkdir,lstat,open,write ./vmplayer 2>&1 | tail -4
write(5, "LOG NOT INITIALIZED | LoadLibrar"..., 73) = 73
open("../lib/libvmplayer.so/libvmplayer.so", O_RDONLY) = -1 ENOENT (No such file or directory)
write(5, "LOG NOT INITIALIZED | Error load"..., 153) = 153
write(5, "LOG NOT INITIALIZED | Could not "..., 74) = 74

I hope you have noticed that I have used a relative path for the library directory in the config file and this means that we always should run the binary with its directory being the current working directory. This is a bit inconvenient, but we will solve this with a wrapper script later. Right now, we need to get it working and we see that it tried to dynamically load some library from the library directory. OK, let’s search for this library in the unpacked bundle directory and copy the library file over to our tree:

[user@localhost ~/vmrconsole/bin]$ find ~/vmplayer -name libvmplayer.so
/home/vmware/vmplayer/vmware-player-app/lib/lib/libvmplayer.so
/home/vmware/vmplayer/vmware-player-app/lib/lib/libvmplayer.so/libvmplayer.so
[user@localhost ~]$ cp -a ~/vmplayer/vmware-player-app/lib/lib/libvmplayer.so ~/vmrconsole/lib/
[user@localhost ~]$ ldd ~/vmrconsole/lib/libvmplayer.so/libvmplayer.so | fgrep 'not found'
 libview.so.2 => not found
 libcds.so => not found
 libvmwarebase.so.0 => not found
 libvmwareui.so.0 => not found
 libgvmomi.so.0 => not found

The last command showed that libvmplayer.so depends on some libraries and that their locations are currently unknown to the system. In order to solve this there are two things we need to do:

We need to tell the system where it should search for the libraries;
We need to locate these libraries and put them into a directory where the system will find them.

To accomplish the first thing we need to create a wrapper script around vmplayer and use this script for fine-tuning later. Here is the very basic script for this purpose (created as ~/vmrconsole/bin/loader.sh):

#!/bin/bash

# We must run vmplayer from the directory it resides in since all
# relative paths are solved from there.
ORIG_NAME="${BASH_SOURCE##*/}"
ABS_NAME=$(readlink -e $BASH_SOURCE) || exit 1
VMW_BINDIR="${ABS_NAME%/*}"
VMW_LIBDIR="$VMW_BINDIR/../lib"

# check that directories exist, if not notify the user
if [ ! -d "$VMW_BINDIR" ]; then
    echo "ERROR: cannot determine the directory where this script resides!" >&2
    exit 1
fi

# preserve the current working directory
OLD_PWD="$PWD"

if ! cd "$VMW_LIBDIR" >/dev/null 2>&1 ; then
    echo "ERROR: the '$VMW_LIBDIR' directory does not exist!" >&2
    exit 1
fi

# resolve the library directory path (to get rid off ../ inside of it)
VMW_LIBDIR=$(pwd -P 2>/dev/null)
if [ $? -ne 0 -o ! -d "$VMW_LIBDIR" ]; then
    echo "ERROR: could not resolve the '$VMW_LIBDIR' directory path!" >&2
    exit 1
fi

# return back since it possible that we were called as ./vmrconsole
if ! cd "$OLD_PWD" >/dev/null 2>&1 ; then
    echo "ERROR: could not return to the original '$OLD_PWD' directory!" >&2
    exit 1
fi

# we don't need the following variable anymore
unset OLD_PWD

# change the current directory to $VMW_BINDIR
if ! cd "$VMW_BINDIR" >/dev/null 2>&1 ; then
    echo "ERROR: failed to change directory to '$VMW_BINDIR'!" >&2
    exit 1
fi

# set the library search path so the dynamic linker will be able
# to locate locally installed libraries.
LD_LIBRARY_PATH="$VMW_LIBDIR${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
export LD_LIBRARY_PATH

# execute the real binary and pass the supplied arguments to it
exec -a "$ORIG_NAME" ./appLoader "$@"

This wrapper script should not be called directly, instead we need to create a symbolic link to this wrapper, e.g. for vmplayer the following should be performed:

[user@localhost ~]$ ln -sf loader.sh ~/vmrconsole/bin/vmplayer

Since our vmplayer is not a full scale VMware Player I suggest to create another small wrapper script and name it vmrconsole:

[user@localhost ~]$ cat ~/vmrconsole/bin/vmrconsole
#!/bin/bash
ABS_NAME=$(readlink -e $BASH_SOURCE) || exit 1
BINDIR="${ABS_NAME%/*}"
exec "$BINDIR/vmplayer" -h "$@"
[user@localhost ~]$ chmod 0700 ~/vmrconsole/bin/vmrconsole

Now we need to populate our library directory with the needed libraries. It is a bit tricky to describe since on different systems you will likely end up with different sets of libraries inside our local directory. For example, on my system I have quite a few of the libraries installed from the distribution repositories and these versions of libraries are fresher and with many bug fixes in comparison to the VMWare provided ones.

Anyway, the general approach to install missing libraries is the following – we start with the libraries we determined as missing during our ldd ~/vmrconsole/lib/libvmplayer.so/libvmplayer.so | fgrep 'not found' step (we need to locate and copy them over to our library directory):

[user@localhost ~]$ ldd ~/vmrconsole/lib/libvmplayer.so/libvmplayer.so | fgrep 'not found' | awk '{ print $1; }' | xargs -i find ~/vmplayer -type f -name '{}' -execdir cp -avL '{}' ~/vmrconsole/lib/ \;
`libview.so.2' -> `/home/vmware/vmrconsole/lib/libview.so.2'
`libcds.so' -> `/home/vmware/vmrconsole/lib/libcds.so'
`libvmwarebase.so.0' -> `/home/vmware/vmrconsole/lib/libvmwarebase.so.0'
`libvmwareui.so.0' -> `/home/vmware/vmrconsole/lib/libvmwareui.so.0'
`libgvmomi.so.0' -> `/home/vmware/vmrconsole/lib/libgvmomi.so.0'

Once this is done we need to follow the following loop until there is no output from the command listed below (in fact, on my system this step was not needed since I had all dependencies in place already, but it is harmless to execute this command anyway):

[user@localhost ~]$ LD_LIBRARY_PATH=~/vmrconsole/lib ldd ~/vmrconsole/lib/* 2>/dev/null | fgrep 'not found' | awk '{ print $1; }' | xargs -i find ~/vmplayer -type f -name '{}' -execdir cp -avL '{}' ~/vmrconsole/lib/ \;

If we try to access any remote VM’s console it will be clear that some parts are still missing:

[user@localhost ~]$ vmrconsole 192.168.70.31
Failed to open file '/usr/lib/vmware/share/pixmaps/progress.png': No such file or directory
Failed to open file '/usr/lib/vmware/share/pixmaps/eula.png': No such file or directory
Failed to open file '/usr/lib/vmware/share/pixmaps/stream-spinner.png': No such file or directory
Failed to open file '/usr/lib/vmware/share/pixmaps/stream-spinner-stopped.png': No such file or directory
SSLLoadSharedLibrary: Failed to load library libcrypto.so.0.9.8:/home/vmplayer/vmrconsole/bin/libdir/lib/libcrypto.so.0.9.8/libcrypto.so.0.9.8: cannot open shared object file: No such file or directory

strace is our best friend here, just run the command through strace, examine the log file, and fix stuff properly:

[user@localhost ~]$ strace -f -eopen -o ~/strace.log vmrconsole 192.168.70.31 >/dev/null 2>&1
[user@localhost ~]$ less -n ~/strace.log
[user@localhost ~]$ cd ~/vmrconsole/lib
[user@localhost ~/vmrconsole/lib]$ fgrep -lr /vmware *
libcds.so
libgvmomi.so.0
libvmplayer.so/libvmplayer.so
libvmwarebase.so.0
libvmwareui.so.0
[user@localhost ~/vmrconsole/lib]$ strings libcds.so | fgrep /vmware
/etc/vmware/bootstrap
/etc/vmware-installer/bootstrap
/etc/vmware-installer/database
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/etc/vmware/,..//////etc/,' libcds.so 
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/etc/vmware-installer/,..//////etc/installer/,' libcds.so 
[user@localhost ~/vmrconsole/lib]$ strings libgvmomi.so.0 | fgrep /vmware
/etc/vmware/hostd/proxy.xml
/etc/vmware/icu
/etc/vmware/ssl/rui.crt
/etc/vmware/ssl/rui.key
/etc/vmware/ssl/dh512.pem
/etc/vmware/ssl/dh1024.pem
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/etc/vmware/,..//////etc/,g' libgvmomi.so.0
[user@localhost ~/vmrconsole/lib]$ strings libvmplayer.so/libvmplayer.so | fgrep /vmware
/usr/lib/vmware
bin/vmware-vmrc-daemon
[user@localhost ~/vmrconsole/lib]$ strings libvmwarebase.so.0 | fgrep /vmware
/etc/vmware/config
/usr/lib/vmware
/etc/vmware/icu
/etc/vmware/CVP
vmauthd/vmware-authd
/usr/sbin/vmware-authd
/var/run/vmware/authd_
/var/run/vmware
/etc/vmware/ssl/rui.crt
/etc/vmware/ssl/rui.key
/etc/vmware/ssl/dh512.pem
/etc/vmware/ssl/dh1024.pem
/var/run/vmware/usbarbitrator-socket
/usr/lib/vmware/settings
/usr/lib/vmware/config
bin/vmware-vmx
bin/vmware-vmx-debug
bin/vmware-vmx-stats
%s/vmware-speedtest-%d.tmp
/etc/vmware/vmware-ace/host.vmpl
/etc/vmware/vmware-ace/host-update.vmpl
/etc/vmware
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/etc/vmware/,..//////etc/,g' libvmwarebase.so.0 
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/etc/vmware,..//////etc,g' libvmwarebase.so.0 
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/usr/lib/vmware/,..//////////etc/,g' libvmwarebase.so.0 
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/var/run/vmware,..//////var/run,g' libvmwarebase.so.0 
[user@localhost ~/vmrconsole/lib]$ strings libvmwareui.so.0 | fgrep /usr/lib/vmware
/usr/lib/vmware
[user@localhost ~/vmrconsole/lib]$ strings libvmwareui.so.0 | fgrep /var/run/vmware
/var/run/vmware/fuse/%Lu.info
[user@localhost ~/vmrconsole/lib]$ sed -i 's,/var/run/vmware,..//////var/run,g' libvmwareui.so.0

Ufff, we did a lot of binary patching – luckily, VMware binaries and libraries are not calculating their checksums. Now, we need to put all this stuff we have seen in the strings output and during the execution of the program in place (in accordance to our new relative paths):

[user@localhost ~]$ find ~/vmplayer -name icu
/home/vmware/vmplayer/vmware-player-app/lib/icu
[user@localhost ~]$ cp -aL /home/vmware/vmplayer/vmware-player-app/lib/icu ~/vmrconsole/etc/
[user@localhost ~]$ mkdir -m700 ~/vmrconsole/etc/ssl
[user@localhost ~]$ find ~/vmplayer -name CVP
[user@localhost ~]$ mkdir -m700 ~/vmrconsole/var
[user@localhost ~]$ mkdir -m700 ~/vmrconsole/var/run
[user@localhost ~]$ find ~/vmplayer -name pixmaps
/home/vmware/vmplayer/vmware-player/lib/share/pixmaps
/home/vmware/vmplayer/vmware-player-app/lib/share/pixmaps
[user@localhost ~]$ mkdir -m700 ~/vmrconsole/share/pixmaps
[user@localhost ~]$ cp -aL /home/vmware/vmplayer/vmware-player-app/lib/share/pixmaps/* ~/vmrconsole/share/pixmaps/
[user@localhost ~]$ cp -aL /home/vmware/vmplayer/vmware-player-app/lib/share/EULA.txt ~/vmrconsole/share/
[user@localhost ~]$ cp -aL /home/vmware/vmplayer/vmware-player-app/lib/share/*.ui ~/vmrconsole/share/
[user@localhost ~]$ cp -aL /home/vmware/vmplayer/vmware-player-app/lib/share/icons ~/vmrconsole/share/

At this stage we should be able to launch the vmplayer program via our wrapper script (I am launching it on a remote machine through SSH, but here I am describing how it should look on the local console):

[user@localhost ~]$ ~/vmrconsole/bin/vmplayer

So far so good, but we need to resolve the issue with the OpenSSL library dependency. To resolve the issue we need to create symbolic links from our local library directory to the system-wide version of the OpenSSL library:

[user@localhost ~]$ ln -s /usr/lib/libcrypto.so.0.9.8 ~/vmrconsole/lib/
[user@localhost ~]$ ln -s /usr/lib/libssl.so.0.9.8 ~/vmrconsole/lib/
[user@localhost ~]$ ls -ld ~/vmrconsole/lib/lib*.so.0.9.8 
lrwxrwxrwx 1 vmware vmware 27 Nov 24 05:27 /home/vmware/vmrconsole/lib/libcrypto.so.0.9.8 -> /usr/lib/libcrypto.so.0.9.8
lrwxrwxrwx 1 vmware vmware 27 Nov 24 05:27 /home/vmware/vmrconsole/lib/libssl.so.0.9.8 -> /usr/lib/libssl.so.0.9.8

Let’s try to connect to remote VM’s console again:

[user@localhost ~]$ vmrconsole 192.168.70.31

Well, this error message does not say much except that we are not getting our console :). If we check the log file directory we would see there is a file called player-XXXXX.log (where XXXXX is the PID of the VMware Player that produced this log file). Inside the log file we may see some interesting parts like the following (I have included only those messages which are related to our task):

Nov 24 03:40:42.002: player| CDS error: Cannot locate VMIS, bootstrap file /etc/vmware-installer/bootstrap unavailable!
Nov 24 03:40:42.072: player| Unable to launch vmplayer-daemon: File does not exist.
Nov 24 03:40:42.072: player| Unable to find /home/vmplayer/bin/vmware-unity-helper in attempt to launch daemon.
Nov 24 03:41:04.630: player| Player dispatch: Opening VM while not connected to the daemon.
Nov 24 03:41:05.437: player| ConnectMksClient - calling VMClient_ConnectMksClientEx
Nov 24 03:41:05.437: player| VMClient_ConnectMksClientEx - trying local socket connection
Nov 24 03:41:05.444: player| Cnx_Connect: Returning false because CnxConnectAuthd failed
Nov 24 03:41:05.444: player| Cnx_Connect: Error message: Connection terminated by server
Nov 24 03:41:05.444: player| VMClient_ConnectMksClientEx - trying remote socket connection
Nov 24 03:41:05.611: player| VMClient_ConnectMksClientEx - connecting the MKS client
Nov 24 03:41:05.616: player| vmdbPipe_Streams: Couldn't read
Nov 24 03:41:15.941: player| Gdk: losing last reference to undestroyed window

From the messages quoted above it is quite clear that the program is trying to launch some helper binaries/daemons in the background and to delegate the actual remote console task to them. You can strace/ltrace the whole thing, review the resulting logs and you will find that it tries to execute two helper binaries: vmware-authd and vmware-remotemks, the latter is the remote console engine, while the former is some kind of an authentication daemon and I do not think it is needed for our purposes. I am not going to describe in details how I arrived at the following (it is all clear once you have studied the strace/ltrace log files):

[user@localhost ~]$ find ~/vmplayer -name vmware-authd -o -name vmware-remotemks
/home/vmware/vmplayer/vmware-player-app/lib/bin/vmware-remotemks
/home/vmware/vmplayer/vmware-player-app/sbin/vmware-authd
[user@localhost ~]$ mkdir -m700 ~/vmrconsole/vmauthd
[user@localhost ~]$ ln -s `which true` ~/vmrconsole/vmauthd/vmware-authd
[user@localhost ~]$ ls -ld ~/vmrconsole/vmauthd/vmware-authd
lrwxrwxrwx 1 vmware vmware 9 Nov 24 04:15 /home/vmware/vmrconsole/vmauthd/vmware-authd -> /bin/true
[user@localhost ~]$ cp -aL ~/vmplayer/vmware-player-app/lib/bin/vmware-remotemks ~/vmrconsole/bin/
[user@localhost ~]$ chmod 0700 ~/vmrconsole/bin/vmware-remotemks
[user@localhost ~]$ strings ~/vmrconsole/bin/vmware-remotemks | fgrep /vmware
/usr/lib/vmware
/etc/vmware/config
/etc/vmware/icu
/etc/vmware/ssl/rui.crt
/etc/vmware/ssl/rui.key
/etc/vmware/ssl/dh512.pem
/etc/vmware/ssl/dh1024.pem
/etc/vmware/CVP
vmauthd/vmware-authd
/usr/sbin/vmware-authd
/var/run/vmware/authd_
/var/run/vmware
[user@localhost ~]$ sed -i 's,/var/run/vmware,..//////var/run,g' ~/vmrconsole/bin/vmware-remotemks
[user@localhost ~]$ sed -i 's,/etc/vmware/,..//////etc/,g' ~/vmrconsole/bin/vmware-remotemks

Another attempt:

[user@localhost ~]$ vmrconsole 192.168.70.31

Ouch, we are presented with a bunch of hints – your mileage may vary since it depends on the environment, but you surely will see the hint/error message presented on the above screenshot.

This is not a showstopper if you are an English-only user - just click on the OK button and you should be able to work with the console, however I think it is just a right thing to do to fix this little bugger:

[user@localhost ~]$ find ~/vmplayer -iname xkeymap
/home/vmware/vmplayer/vmware-player-app/lib/xkeymap
[user@localhost ~]$ cp -aL /home/vmware/vmplayer/vmware-player-app/lib/xkeymap ~/vmrconsole/
[user@localhost ~]$ cp -aL /home/vmware/vmplayer/vmware-player-app/lib/vnckeymap ~/vmrconsole/
[user@localhost ~]$ vmrconsole 192.168.70.31

We are almost done! At least we achieved the goal we set at the beginning of this article.

Now, it is time to ensure that all file/directory permissions are strict enough and to create a tarball, then place it somewhere so we will be able to use our new tool when the right time comes:

[user@localhost ~]$ cd ~/vmrconsole
[user@localhost ~/vmrconsole]$ find . -type d -execdir chmod 0700 '{}' \;
[user@localhost ~/vmrconsole]$ find . -type f -execdir chmod 0600 '{}' \;
[user@localhost ~/vmrconsole]$ chmod 0700 bin/*
[user@localhost ~]$ cd
[user@localhost ~]$ mv vmrconsole vmrconsole-3.1.3.324285
[user@localhost ~]$ tar cjpf ~/vmrconsole-3.1.3.324285.i386.tar.bz2 vmrconsole-3.1.3.324285
[user@localhost ~]$ ls -lh vmrconsole-3.1.3.324285.i386.tar.bz2 
-rw-r--r-- 1 vmware vmware 13M Nov 24 07:06 vmrconsole-3.1.3.324285.i386.tar.bz2

P.S. Oh, boy, it takes 2 hours to investigate and to come up with a solution, then 8 hours to write an article to describe steps to reproduce! I hope that somebody has found this article useful and I would appreciate any comments.