Should we use ‘sudo’ for day-to-day activities?

None of the systems I administer or supervise have ‘sudo’ installed and every time I answer a question on how to do privileged work on these systems (i.e. do tasks that require administrator privileges) with a proposal to SSH under the privileged account directly to do such a work whoever asked the question start to blabber how insecure that is, that one should use ‘sudo’ and that nobody should ever login directly as root.

I've spent quite some time explaining the misconception behind so-called "secure way to access systems through sudo", so I decided to write up an article that describes the issues of using that approach and why using ‘sudo’ is actually less secure than a direct SSH access.

History

The following is based on my personal recollection of the history around early 90’s with some reference to documents I could quickly find. Unfortunately, this is one of the topics that is not so easy to reconstruct in full detail. Although there may be minor inaccuracies the outlined view on the history should be very close to the true events that took place.

Many years ago (but not that far in the past) system administrators were using telnet and rsh to access and administer their servers. Networks were simpler and traffic sniffing techniques were common. Therefore, it was very easy to eavesdrop on a root account login and to automate the task of gathering the credentials. To mitigate the issue to some extent the following approach was proposed:

  • login as a non-privileged account first;
  • do some stuff;
  • if you need privileged account access (e.g. root) switch to it using ‘su’.

The idea behind this proposal was that such a sequence makes it harder for a traffic listener to catch root’s credentials on the wire due to the login sequence mixed in to the stream of other activities the systems administrator was doing before escalating their account to the privileged user.

At approximately the same time discussions started in regard to the dangers working as root, that people were tending to work as root for a prolonged periods of time performing tasks that do not require escalated privileges without proper justification for that behaviour. As the result of these discussions the sudo utility was born. The utility allowed to bind root privileges to a restricted set of commands and maintained access controls through its configuration file in order to allow or deny access to the defined functionality to specific users/groups. This allowed systems administrators to delegate some of the privileged routines to less privileged user/groups. (See more on the history of sudo if you are interested).

Somewhere in early 90’s the two approaches were merged, so the approach became:

  • login a a non-privileged account and do your every day tasks;
  • once a privileged operation is required execute it through ‘sudo’.

Although it seems like a minor optimisation, in fact, this introduced a major security issue: previously, the attacker should have needed to listen for traffic in attempt to figure out when the victim is executing `su` and providing root's password, but now they did not need to do anything except of capturing user's password at the beginning of the session (the same problem the "su-after-normal-login" approach was trying to solve just reappeared :) ) since once you know user's password you can use 'sudo' which authenticates you with _user's_ password!

In 1995, Tatu Ylönen as a response to the issues related to exchanging the credentials over non-protected, easily sniffable networks released his first implementation of the SSH protocol as freeware to the public. Over the next five years SSH was adopted worldwide and it eventually replaced telnet/rlogin/rsh for the remote access and management activities in the most places.

However, due to inertia the recommended approach of logging in as a non-privileged account and escalating privileges later somehow survived, is still followed, and often people don’t even try to analyse and see the flaws of the approach.

People are saying ‘sudo’ is good. Is it true?
(or looking critically into some common misconceptions re: ‘sudo’)

So let’s look at the most commonly used “pros” in favour of using ‘sudo’. For example, Ubuntu’s community help page provides a nice, aggregated list of benefits provided by ‘sudo’ usage in their default installation. Let’s walk through all of them and see if they are real benefits at all:

  1. The installer has fewer questions to ask.
    This is a very questionable “benefit” to the end user since this implies that the system would do more stuff with escalated privileges behind the scenes without the user even knowing it. From the security standpoint it just silently expands the attack surface.
  2. Users don't have to remember an extra password for occasional use (i.e. the root password). If they did, they'd be likely to forget it (or record it unsafely, allowing anyone to easily crack into their system).
    This is another hard to comprehend “benefit”. Given that we now have SSH and the recommended best practice is to use SSH keys instead of passwords ‘sudo’ requires users to memorise yet another password. Moreover, the argument in regard to users’ likely behaviour is purely speculative and assumptive — to the same extent we can assume that password policies make it hard for users to memorise their passwords and they would be likely to forget the password (or record it unsafely, …). The truth is that the key based authentication solves the issue of the requirement to have multiple passwords for multiple accounts. There is one place where the discussed “benefit” is actually applicable — local console access (no key authentication there usually), however even there the best practice would be to login directly as root on the virtual console if there is such a need (the reason for that is quite complicated and in short could be described as the following: there are multiple checks and assumptions in the kernel code and the accompanying C library on allocating a terminal, spawning a process, etc. for root over the same actions performed for a non-privileged user).
  3. It avoids the "I can do anything" interactive login by default. You will be prompted for a password before major changes can happen, which should make you think about the consequences of what you are doing.
    This statement also assumes that for some reason people would prefer to always login as root and do all of their work under that account. In the reality users are so used to prefix almost any failed command with ‘sudo’ that this “benefit” can be considered as the quite opposite item against ‘sudo’. On a properly configured environment you explicitly need to login as the privileged account to do functions that require privileges.
  4. sudo adds a log entry of the command(s) run (in /var/log/auth.log). If you mess up, you can go back and see what commands were run.
    This statement is also somewhat true, but it does not defend the ‘sudo’ usage. A proper auditing subsystem is what keeps audit logs no matter how activity was performed. Logging of executed commands for user's history reasons is the job for the shell. The ability of logging is so limited in ‘sudo’ that it cannot be used for anything except a substitute for the shell history (just imagine the following scenario: a user executes “sudo less /var/log/messages”, then types “!” followed by the Enter/Return key — the user effectively now in the root shell and what ‘sudo’ will log into its logs has nothing to do with what user will actually do).
  5. On a server, every cracker trying to brute-force their way in will know it has an account named root and will try that first. What they don't know is what the usernames of your other users are. Since the root account password is locked, this attack becomes essentially meaningless, since there is no password to crack or guess in the first place.
    This is such a weak attempt to bring host security into the play that it’s hard to comment on it. First, use SSH keys and disable the password authentication on the server — this (and not some security through obscurity) will defend the system from the brute force attacks. Second, protect your remote access entry points with properly configured firewall and allow remote access from a defined list of locations (i.e. whitelist) — this will shrink the possible attack surface on the SSH service.
  6. Allows easy transfer for admin rights by adding and removing users from groups. When you use a single root password, the only way to de-authorize users is to change the root password.
    This is an interesting one: half of the statement is true, another is misleading. The true part is that you can easily delegate privileged operations using sudo. The misleading part is that the alternative is to use a single root password. The truth is that nobody is limited by a single root account: you can create as many as you want and each of them could have their own distinct password (e.g. “useradd -om -o 0 -g 0 -s /bin/sh new_root” and “passwd new_root”, but please don’t set password and lock the account instead with "usermod -L new_root" since we are using keys, remember?). This approach also provides additional accountability since users will have separate shell histories, their login attempts will be clearly logged under separate names, etc.
  7. sudo can be setup with a much more fine-grained security policy.
    Another half-truth in the list of “benefits”. The statement is incomplete and lacks the part it is comparing the functionality to. If we are comparing a legacy Unix access control system with ‘sudo’, then yes ‘sudo’ is much more configurable. If we compare ‘sudo’ with, say, SELinux or GRsecurity’s RBAC - ‘sudo’ will lose since both have much more fine-grained security controls than ‘sudo’.
  8. The root account password does not need to be shared with everybody who needs to perform some type of administrative task(s) on the system (see the previous bullet).
    As with item #6 this statement assumes for some obscure reason that there can be just a single root account in the system. Therefore, the result of such a logical exercise is also questionable since it is based on a wrong assumption.
  9. The authentication automatically expires after a short time (which can be set to as little as desired or 0); so if you walk away from the terminal after running commands as root using sudo, you will not be leaving a root terminal open indefinitely.
    This statement also compares ‘sudo’ with something reader cannot compare to. Moreover, it mixes up two logically unrelated things: the credentials expiration and terminal security. While it is great that ‘sudo’ implements the former, the latter is usually addressed by entirely different means: starting with auto-logout functionality, lock screen, physical security, etc.

Well, we can continue to critically assess other common statements in regard to sudo made mostly by people who do not have any strong InfoSec background, but it would be a waste of time for the readers. You can always raise a question regarding some particular claim and/or assumption related to ‘sudo’ and if it is interesting I would add it (and the corresponding response) to this article.

What is the problem with the ‘sudo’ approach?

Well, there are several in fact. The most pressing issue is that the usage of ‘sudo’ (or ‘su’, or any other utility that has its SUID bit set) is crossing the security boundary from the less privileged account to the more privileged account. This opens doors (or widens the attack surface) to privilege escalation techniques. In plain English it makes the non-privileged account to be essentially equal to the privileged one, let me explain by a fairly simple example:

  • imagine if you are a developer and you work on a server under your non-privileged account;
  • as a part of your daily routine you need to download some third party package and install it (we are going to leave out all the security complexities involved with such an activity like verifying signatures, using a separate instance to prepare a package for deployment, etc.);
  • the installation of the package usually requires executing some third party code under your non-privileged account. This code was not written by you and there is a high chance that you did not read/verify the foreign code line by line in order to ensure that it does not do anything malicious since this would be quite time consuming, would require a lot of effort, and your team has more important priorities than this (remember, this is an example based on situations you would encounter in the real world, which is by no means perfect);
  • it happens, that that particular third party was compromised and some malicious code has been injected into the package installation routines;
  • after the execution of the installation routine (and the malicious code for that matter) your ~/.bashrc (for example) is modified in such a way that each time you login it starts up a key logger or some other kind of remotely controllable piece of software that talks back to its master;

So far we just got an issue localised to this non-privileged account only (with a possibility to spread across the entire fleet of servers that non-privileged account has access to in case of the NFS mounted home directories). Is this bad? Yes, it is since it may disrupt this particular project, steal reachable sensitive information, could be used as a trampoline to jump start further research and exploitation of other vulnerable resources. Is it critically bad? Not necessarily. If the systems are built properly with host-based security in mind, if the proper privilege separation techniques are used throughput the company infrastructure, etc. the impact is localised and with proper monitoring systems it would be detected eventually and investigated (keep in mind that it is really hard for a non-privileged account to hide their activities from more privileged processes).

Now, let’s add ‘sudo’ to the mix, e.g. suddenly the developer decided to install an additional library package into the system. So, what would happen next? You guessed it: the developer would need to type in their password to convince ‘sudo’ that they are “allowed” to do such a privileged thing as installing a system package, the malicious software installed by the attacker would happily intercept that and send it back to its master.

From this point on, the attacker has the account password of the account where their software runs and which they control. The attacker can now utilise ‘sudo’ powers at their will. The security impact would be ranging from “high” to “extreme” depending on how committed the attackers are.

Following the logic, why would we want to introduce an additional complexity that does not address the issue it was supposed to address, this is “to limit exposure of the root account”?

The “light” side of ‘sudo’. Is there one?

So, is ‘sudo’ any good for anything? Actually, yes, it is. All in all, ‘sudo’ is a tool that attracts a lot of attention from the security researchers and auditors, its codebase got numerous peer reviews and the functionality the tool provides can be used for good. The following scenarios come to my mind right away:

  • On a SELinux enabled system it seems that ‘sudo’ is the only properly implemented and reliable way to assume a different SELinux role. All other mechanisms are either lacking in the functionality or just do it half way leaving some artefacts behind;
  • Unless some effort is put into tweaking the way PAM (Pluggable Authentication Modules, an authentication framework used on Linux, Solaris, and some other Unix-like systems) authenticates users on LDAP enabled systems there is no easy way to have two separate accounts (a privileged and a non-privileged account) for the same LDAP user. This could be addressed administratively (e.g. by defining additional privileged users in LDAP) or technically by implementing account names’ prefixes. However, if there are budgetary and/or time constraints to implement the proper security framework ‘sudo’ with a quite restricted configuration would be an acceptable compromise.

Unfortunately, personally, I failed to find any other justified applications of ‘sudo’ in a secure environment and would be happy to get some feedback if you have something in mind worth including into the list above.

How can we improve the security of our systems in relation to the ‘sudo’ usage?

OK, so we got to the point where we are standing near a crossroad and we kind of established that excessive usage of ‘sudo’ is a bad thing (for security-paranoid readers - read: “almost any usage of ‘sudo’ except for changing SELinux roles on a SELinux-enabled system is a bad thing”). So, what is the alternative way of doing things? Well, there is a complex approach on addressing and minimising the privilege escalation risks and roughly it can be summarised as follows:

  • apply the least privilege principle to everything, e.g. if you need to work on the content the account you use to log into the remote server should be allowed to do just that. This sounds a bit extreme, so we may re-phrase it as follows: each account should be provided the least number of required privileges to do the day-to-day tasks under that account and should not be used for any activities outside the defined set of activities. Again, if you include “becoming root” into the list of defined activities it would kind of defeat the purpose.
  • there should be a defined list of accounts which are allowed to do system-wide modifications (e.g. privileged accounts), these accounts must be used for these activities only (system updates, installing new software, modifying system-wide configuration);
  • privileged accounts during their activities should not cross the security boundary with the less privileged accounts. This is needed to avoid attacks from the less privileged accounts toward more privileged (e.g. process hijacking, file races, etc.). Unfortunately, this point is a bit confusing without a proper explanation, but in a nutshell it means that root should not touch anything writable by the less privileged account (examples of the bad and insecure behaviour include: changing directory to a directory writable by non-root, executing scripts from a directory writable by non-root accounts or scripts that are writable by non-root, copying a file/directory from/to a directory writable by non-root accounts, etc.);
  • if possible (and this is highly recommended) privileged accounts should be accessed from the trusted and secure machines. It is really hard to define what the “trusted and secure machine” is, but generally it should be assumed that it is a laptop or a desktop station that is used entirely for the work purposes (no free time surfing on the leisure sites and stuff :) ), that proper firewall rules and protection techniques were used to secure the machine, and that the operator is security-minded and does everything they can to ensure integrity and security of their machine and software installed on it.

I would be really happy to answer any question in regard to this article and provide any possible help in making our everyday system level activities more secure, so do not hesitate to contact me.

Comments

  1. Can we set a claymation death match between you and Mr. Sudo (Michael W Lucas)? On the serious side, what about following the PLP by using unprivileged service accounts, e.g. the ntp account runs the ntpd daemon, and that's *all* -- no password, no login.

    ReplyDelete
    Replies
    1. WirelessBen,

      I watched some of the presentations by Michael Lucas and he does have a point and proper usage scenarios for advanced use of sudo. However, general public is abusing the tool and this was the point of my article. I also mentioned that there are cases when sudo is the right tool for the job (e.g. switching SELinux roles), but the application of the tool is not that wide as people tend to use the tool for.

      Your example of ntpd won't work unfortunately as you described it since ntpd needs privileges to adjust the system clock and to listen on the privileged port. You can lower the privileges if you use capabilities (man 7 capabilities) and set CAP_SYS_TIME (to allow the process to adjust the clock) and CAP_NET_BIND_SERVICE (if you run it as a time server, to allow binding to the privileged port), then you can run it under non-privileged account. Well, it will be still a bit more privileged than a regular account due to capabilities, but this will follow the principle of least privilege.

      Delete
  2. Lucas reluctantly recommends using an oracle user to install oracle software, yet I couldn't even use yum:

    [me@someserver ~]$ sudo -u oracle yum install xclock
    eDir Password:
    Loaded plugins: rhnplugin
    *Note* Spacewalk repositories are not listed below. You must run this command as root to access Spacewalk repositories.
    You need to be root to perform this command.

    So sudo doesn't work as billed, and you seem to suggest using

    root
    admindmitry (install apps, remove apps, reboot)
    adminjane (backup)
    adminyuri (add volumes)
    adminjane (view logs)

    How do you do that without sudoers? SELinux voodoo?

    ReplyDelete
    Replies
    1. I think what he meant (I can't pretend that I know what's going on in Lucas'es head, but he seems to be a logical and reasonable person) is that you install custom Oracle software that was not packaged/provided by your distribution and or through yum repositories. For example, if you need to install a specific version of Java you create a dedicated, non-privileged user account, download the tarball from Oracle's website under that account, and install that under that non-privileged account. This way you contain that proprietary thing inside that non-privileged user's home.

      What I'm proposing is that when you need to install software through yum, edit system-wide configuration, manage the disk subsystem - you directly ssh into the server using the corresponding privileged account (e.g. root, or if you have multiple people managing the same system you can create multiple privileged accounts using "useradd -om -u 0 -g 0 -d /root/.users/ben r_ben" (the /root/.users directory should be precreated before executing that command)).

      If you need to delegate a specific task to a specific user/group you may create a privileged account (say r_service) and create an SSH key that allows a user with the corresponding private key execute just that task (via command="" prepended to the public key in ~r_service/.ssh/authorized_keys).

      Now, backup does not require root privileges at all, the only thing backup process needs is CAP_DAC_READ_SEARCH (see man 7 capabilities).

      With logs there are several approaches and it highly depends on the distribution of choice (e.g. if systemd is used it would be enough to grant membership in journald's group to be able to see system logs), but, generally, it possible to grant access to logs via group membership.

      I hope I answered your questions :)

      Delete
  3. You did, sir. Thank you! This is a monster article that has changed the way I look at Linux security. You are hereby followed.

    ReplyDelete
  4. nice to find another geek in the net. thanks for the work you do. I would like to read in your blog your opinion about new cloud technologies for business deals or maybe some kind of a review of ideals cloud. Is this your sphere of interest? thanks in advance for your reply!

    ReplyDelete
    Replies
    1. Rick, thanks for your comment. I do have some material on the security in the cloud which I will publish in the near future, but I don't have any experience with the mentioned vendor (and it is actually out of my interest zone to dive into that, so do not expect any reviews of that particular vendor).

      Delete
  5. I'm trying to configure sshd on an Ubuntu 16.04 LTS, following your recommendation. But when I lock the user with 'usermod -L', then sshd is denying a login, since that user has been locked (according to auth.log).
    Am I missing something here, or have things changed?

    ReplyDelete
    Replies
    1. Okay, I found out, that usermod -L puts an exclamation mark in front of the password hash inside the shadow file. This seems to mark the account completely locked.
      But, if you replace the hash with an asterisk, still the password is not usable, since that is no valid crypt hash, but SSH-login is possible.
      Strange, but seems to work.

      Delete
  6. A few thoughts I have,
    1. Good sudo use would not allow non-admin users to install software unless it was very specific approved software. I.e. I would never permit the use of zypper/yum/apt-get through sudo.

    2. I use sudo to allow sap basis admins to change user to without knowing the password. Sudo allows me to audit who is using that account and for non-repudiation. I don't think using a key with a command would ever work in this situation, unless you have any suggestions?

    3. I would never permit direct ssh login with root, service accounts or general admin accounts by users. This doesn't easily allow for tracking unless you have a good logging solution on the rest of the infrastructure that can access the servers. Most places do not have that in place, but it can be easily done on linux.

    I think its easy to make mistakes with sudo and permit more access then you intend. Any use of sudo should be heavily scrutinised and only experienced admins should be putting in the rules.

    My last thought is that any company or admin that is blindly putting in sudo rules is probably also permitting direct root access, sharing passwords and probably just does not do security well.

    ReplyDelete
  7. correction: I use sudo to allow sap basis admins to change user to ""

    ReplyDelete
  8. oh seems like it does not like certain characters.

    Change user to sidadm

    ReplyDelete

Post a Comment

Popular posts from this blog

SSH: Interactive ProxyCommand

Raspberry Pi 3 toolchain on CentOS 7