Testers wanted for Jailkit package

Developers
1

Howdy all,

If you’ve ever worked with Jailkit, or have some general experience with chroot jails, and you use CentOS 7, please have a look at the package I just published in our bleeding edge repo here:

http://software.virtualmin.com/bleed/centos/7/x86_64/jailkit-2.19-1.el7.centos.vm.3.x86_64.rpm

This package should, theoretically, work for pretty much everything documented on the Jailkit home page, without needing to make package-level changes (i.e. config files need to change, as documented, but it should Just Work, once the config is right).

Those docs are here: http://olivier.sessink.nl/jailkit/

Once we’ve worked out any issues with the CentOS packages, we’ll get it packaged for Debian and Ubuntu, as well. Note that this is not recommended for folks who aren’t familiar with chroot jails, or don’t understand the security risks inherent in their use. Chroot jails are dangerous tools. So, either poke at this on a development server, or know what you’re doing, or don’t mess with it at all!

Something to note: The chrootsh binary in this package uses capabilities, rather than being setuid; this is safer, but, I haven’t tested every use case to know if everything that needs to have elevated privileges is setup right.

2

Is there any particular reason why you decided to go with chroot jail (old way) vs what most are using today which is namespaces?

Namespaces is the new approach build into the Linux kernel and as opposed to chroot that only isolates the file system, namespaces does it all levels. This means that while chroot still protects one users from accessing another users files, they can still tap into their process. Chroot was broken before several times and many control panels had some sort of chroot setup that didn’t solve the security issue which I guess you want to solve here.

There was even a time when cPanel had to change their documentation because they where saying their SSH jail chroot was secure and it was proven wrong, so they change it to inform that while it provides some security it can be escaped by more experienced users. Most script kiddies can potentially escape a chroot implementation today.

Namespaces on the other side is so popular and used by major projects like Docker or Linux Containers and backed up by companies like Red Hat that its getting more eyeballs on the code. If your intention is to make data safe (isolation) between users in a multi tenant environment like shared hosting, then a better approach would be to use the new Linux namespaces because that way you can even provide SSH access to users and they will only see their own process as well.

Also, I want to mention that making everything work under chroot is a pain in the butt. You will be able to make some things work but it would require a whole rewrite change on how most of the things and services are created in Virtualmin. I’m not saying that its not the case with Namespaces but at least if so much work is going to be done, go with something that will be bullet proof in the future.

A nice read here:

http://uwsgi-docs.readthedocs.io/en/latest/Namespaces.html

https://www.toptal.com/linux/separation-anxiety-isolating-your-system-with-linux-namespaces

Either way. I’m happy to test the jailkit implementation if this is the plan forwards.

3

“what most are using today which is namespaces?”

Who is this “most” you speak of? I was unable to find anyone doing jail type configurations for shared hosting with namespaces/cgroups, and believe me I searched!

I’ve been working on namespaces/cgroups stuff for a while now, as well. But, as far as I can tell it’s not quite ready for use as a user isolation tool (it’s getting very close, though, but it’ll be maybe years before stock kernels in our most popular distros have the necessary bits). There’s process isolation, network isolation, etc. but no user isolation in older kernels (and we have to support older kernels). I did a bunch of experiments before settling on Jailkit as our near-term way forward on this feature, but couldn’t figure out how to make it work with namespaces and cgroups.

The issues with chroot jails have mostly been resolved. Our jailkit build uses kernel capabilities (rather than suid 0), the Jailkit commands protect users against the most common mistakes people make when building jails, and we’ll be bind mounting homes so that the various other parts of the system don’t need to change or have additional privileges. And chroot no longer breaks privilege separation in openssh, which was a major factor in my always saying no in the past to chroot jails. Historically, the risk with jails was not just that someone might break out of the jail, but that breaking the jail potentially meant escalating to root. I was never comfortable with that state of affairs, but those issues have been resolved for a couple of years now (long enough for all of the bits to make it into all of the operating systems we support).

But, I should be clear that I have never once recommended a chroot jail as a security feature (you’ll find me ranting about it all over these forums), but there are tons of users who want it. It’s not hard to implement, so we’re adding it. I will still never claim that a chroot jail is more secure. But, many users want that belief of isolation among their users.

The reasons we aren’t using namespaces (yet):

The tools available for managing them are in their infancy, and none (that I’ve found) are focused on isolating human users. They are mostly targeted toward service-based architectures (which is another useful area of development for Virtualmin, but it’ll come late). Containers do not yet contain, very well, is what I’m trying to say, especially in older kernels.

So, the reasons for Jailkit support are:

  1. Users have been asking for it for years. The primary reasons I always said “no” have been resolved.
  2. Namespaces/cgroups do not have tooling for isolating users; there is nothing comparable to Jailkit for namespaces (Docker is close, but having a user login shell inside a docker is almost antithetical to their intended model, so it doesn’t get a lot of attention).
  3. Namespaces/cgroups support in the kernels for the older operating systems we support lacks a number of the features needed to actually be safe for users to have access to. Again, we’re back to the “breaking the jail might allow more damage than just escaping the jail” problem that chroot jails had for many years. I think the current kernel has fixed all that, but it’ll be a while before we can rely on it being available on our supported distros. Probably need to wait until CentOS 6 and Debian 7 and Ubuntu 14.04 are EOL.

All that said, if you’re enthusiastic about namespaces/cgroups for user isolation, and you want to help accelerate that development, I’d be happy to have someone else involved. We can make it happen during the Virtualmin 6 cycle, as a beta plugin, if the kernel support is there on CentOS 7 and Ubuntu 16.04 (I’m not sure it is, but we could sort it out).

I don’t like chroot jails. Never have. They’re mostly snake oil security. But, many people really want them and think Virtualmin is insecure because of its lack of them (I’ve seen the “Virtualmin is insecure because users can see outside of their home” argument way too many times now by folks who have no idea how chroot works or its history of exploit risk). So, we will support them in a way that doesn’t alter the experience for users who don’t turn them on, and only moderately effects users who do turn them on. We’ll work on namespaces/cgroups for the future.

4

Note the first doc you linked has this to say about user namespaces:

user -> CLONE_NEWUSER, still complex to manage (and has differences in behaviours between kernel versions) use with caution

That’s what I’m referring to. It’s very poorly documented, very poorly understood, and the tools for managing user namespaces is practically non-existent. We’ll have to write our own, if something doesn’t arrive soon.

5

CloudLinux is moving towards, and they also use cgroups, granted they use OpenVZ parts as well for their file layer system if memory serves me right.

I don’t honestly know any reputable hosting company that is offering shared hosting without CloudLinux because it’s just too insecure. You can’t properly provide SSH access to users, and the Apache symlink race condition is just staggering, not to mention other exploits because process are shared. You may be right about nobody using namespaces with shared hosting as now, but that doesn’t mean it should be discarded, Docker does and at this point Docker is a far better solution than any shared hosting solution (unless you plan to use CloudLinux). And while I don’t believe it will kill shared hosting, tons of people I talked with say it will for the simple reason that it solves the isolation issues that shared hosting does not. Each account is properly isolated which gives you security, stability and individual resource metrics.

Does CloudLinux work with Virtualmin? Yes. But it requires manually work and compilation. And you have to change settings from the command line, there are no fancy web panels and officially they don’t support it. They told me it’s going to costs them over $100,000 and when I mentioned them that they are supporting a less used software like ISP Manager they told me they did the integration on their own.

Of course, this means extra costs (around $10/month) per server but I would honestly not consider giving untrusted users access to a server without CL today. If you can implement something similar, including cgroups (in some future) that can potentially limit computing resources per account, then this would be selling point for the PRO and it would sell like hotcakes. Why? Because nobody else has it today. All other control panels, including cPanel and Plesk have a bundled edition with CloudLinux but it costs extra. The security problem is a real problem to solve and namespaces are the way to go…

The cgroups feature can potentially limit CPU, or RAM and measure them, could also be an extremely premium selling point as people move towards cloud computing because it could potentially allow providers to charge based on consumed resources. These things where developed by Google for a reason, they want to provide users as much control on a low level as possible.

If you think chroot is a better option today, then you are probably right. I will not argue but personally I would still pay extra and put CloudLinux into the server just to be sure, because it just works. The problem with any approach like this is that Virtualmin is multi OS and this is extremely hard to get working on all operating systems. Hence you mentioned the kernel differences. Most companies are mono distro, they support RHEL/CentOS or that and Ubuntu or Debian. That includes Plesk, cPanel, CloudLinux, the rest. So I understand your position here.

Do I want something like CloudLinux on Virtualmin? Absolutely. I’m working on that already. Would I love to drop them and just build it natively into Virtualmin? (at least some futures)

Of course, but that is way over my head today. Now between paying them $100K and then have to pay them again recurring licenses per server, I would rather invest in doing this directly on Virtualmin instead and just pay one license (the PRO). I have several people interested already that also want to use Virtualmin in a commercial environment but don’t for some of the reasons mentioned.

6

“Docker does and at this point Docker is a far better solution than any shared hosting solution”

I think one of us has a wrong idea about what Docker does. :wink:

Docker is not at all a multitenancy solution. It isolates processes not people/users. I’d love to use Docker or similar for isolation instead of chroot jails, but it’s just not a feature that exists yet. Docker is not virtualization in the traditional sense.

LXC/LXD might currently be usable for this with a bind mount, but I couldn’t find good existing tooling for it, or any examples of people using it that way.

We already have CPU, memory, and some other limits built into Virtualmin Pro, but newer kernel features definitely permit a lot of stuff we aren’t taking advantage of yet. I’d like to spend more time on that once VM6 is out the door.

7

I know how Docker works and I know it’s not virtualization. Not at least traditional…in the sense of bare metal but more a file layer but I disagree with you on the multi tenancy part. There was a time Docker was not secure as it was running things as root inside a container. This is not the case anymore today. Only the external daemon does. Docker is very similar to LXC which you mentioned and the security features are similar. Basically, namespaces and cgroups.

The fact you can isolate process means it’s a multi-tenant solution and while it needs a dozen extra things to make it secure after that (and it’s still on development) tons of companies are offering Docker services on production, including companies like Amazon or Google. You just combine it together with traditional virtualization like Xen or KVM and launch the containers for that customers in the same VM so they complement each other in terms of security and you don’t risk a container compromising the host OS.

I say this because you are not supposed to run different process in Docker in the first place. At least that is not the best use for it. You run one for Apache, one for MySQL, one for your web app, etc. You run one container for single service which makes it even more secure in some sense as different process belong to specific container and different containers can’t talk to each other (unless you allow it). If you add things like AppArmor or just the default SeLinux in CentOS, its quite secure actually.

This is not true with something like Apache where every user runs on the same shared user and usergroup that the service is running. Isolating data is just one process of the equation. The end result should also to try to isolate the running scripts (PHP, Perl, Ruby, etc.) process. Files alone is great, but process is amazing.

The CPU, memory, etc. settings in Virtualmin today are not exactly for measurement or stopping abusing users (terminating process when limits are hit). I think you answered how they work already to Diabolico here:

https://www.virtualmin.com/comment/767819#comment-767819

Your words:

Limits are not based on cores (that’s more sensible in a virtual machine or container environment, where Cloudmin is the right tool for the job). Limits are “Maximum number of processes”, “Maximum size per process”, “Maximum CPU time per process”.

Trying to devote a core to a user in a virtual hosting environment would either be impossible or would require every service they use to be dedicated to them”

Except this is exactly what you can do with cgroups today!

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu_and_memory-use_case.html

Just switch the departments for users and the concept is the same…

8

Except this is exactly what you can do with cgroups today!

No you misunderstand me.

cgroups can’t determine which user is being served by, say, Postix, Dovecot, SpamAssassin, ClamAV, MySQL/MariaDB, etc. at this moment, because these services all run as their own user…not the user we’re trying to limit. The only part of a shared virtual hosting system like this that can be controlled in the way you’re talking about is web apps running under suexec, and any processes that are actually owned by the user. And, there’s no way to devote a core to one user, when every user on the system shares all of the services.

Now, to be fair, in many cases, the web application (which can be restricted) may contribute a significant amount of usage to the system, and it may make sense to manage them in some way. But, you still probably shouldn’t expect to use cgroups to push a users web apps into one core; I’m not sure how that would play out with Apache being shared and it being the parent of all of the forks that are serving the web app (though PHP-FPM runs independently, and so could be isolated…we didn’t have PHP-FPM in Virtualmin when that was written). We’d need to experiment to see what that does. But, I’m not convinced that dedicating a core to a user is a good idea in a shared hosting system. Use limits, sure. But, a whole core locked down for one user? That doesn’t make a lot of sense, IMHO.

In short: If you fully virtualize (with, e.g. Cloudmin and KVM or whatever), you can control your users with that kind of granularity. But, if everyone is sharing all of the services, it becomes a lot murkier who gets what resources. We could put knobs on them, but it wouldn’t necessarily accurately reflect what actually happens.

9

No, of course not. Not a full core, but maybe 10% of that. I think you should really read or even test how CloudLinux works to see what I mean:
https://www.youtube.com/watch?v=sXoFCaFVSQA

What about PAM? Services like MySQL can run under PAM and CloudLinux can measure or take actions on basically everything that works already under PAM, Apache, FTP, SSH. They could not for mail for this and other reasons but something like this is not expected to be able to work in everything. I think if it can for scripts in general that is great already as MySQL already has plenty of limits per user or database you can set.

10

Just commenting to get notifications, I didn’t see a way to do that without commenting.

11

Good to know you adopted Jailkit, it is a good package to implement what we asked for many years. I am interested doing tests on Debian/Ubuntu servers. Only RPM version is available as I see.

12

Up for any testing …