Distributed Virtualmin on a LAN

Virtualmin
1

I’ve got a multi-node rackmount server.

These are hooked up to a network switch and form their own little “cabinet” LAN. There will be a UPS keeping it all alive and there will be NAS drives too.

I’ve already got Virtualmin running on a single standalone server just fine, but it’s time to scale things up, add redundancy and high availability, and so forth.

What puzzles me here is how to get Virtualmin working in a distributed fashion.

That is, traffic would come in and I’d use something like HAProxy to load balance things. Request 1 can be sent on to server A, request 2 can be sent onto server B or whatever.

The thing is, that both server A and server B need to work with the same data. Whichever website is being served, either server A or server B is capable of serving it up. Also, if “website1.com” is changed on server A, then a request for “website1.com” on server B shortly afterwards should serve up the newly changed data.

In other words, they need to be working with the same data. So I was thinking that I could create a shared “/home” directory on the NAS drive. An NFS share that’s mounted to the “/home” directory on each server. So that they are both working from the same “/home” directory and, thus, the same data.

But, well, I need more than just that. They need to also share, for example, the “/etc/apache2” directory, so that they also both have the same “sites-available” and all the same configuration details. This naturally leads to the idea of also just sharing “/etc” on the NAS drive too. Then all the servers are working from the same configurations too (which could also simplify administration, as I only need to change it in one place and, by merit of it being a shared directory on the LAN, the configuration changes for them all).

What puzzles me is whether this is even possible. You can share “/etc/apache2” but not everything in “/etc” is necessarily shareable. For example, with “/etc/hosts”, I could list all the host names and IP address for all the machines on the LAN. But what about “localhost” (127.0.0.1)? Each server needs its own IP address and hostname on the LAN. Some of “/etc” is shareable, some of it isn’t.

Also, are there any other directory or files to be worried about? Webmin / Virtualmin has its own files. Could that / should that be shared?

Or, alternatively, instead of a network share, is there a way to keep the servers constantly synchronised with the NAS drive somehow?

How would I make this work?

2

Nobody is going to be able to reply here because scalability is more an art than anything else. Virtualmin is just a control panel so when you say you want to scale Virtualmin you really are saying you want to scale Apache. What else? PHP? MySQL as well? How about email? All of them are individual services or servers.

So you should not be looking into how to scale Virtualmin but the single service first. Some services are easy, Apache certainly is. Some are harder, (databases) and some are even more difficult (PHP with sessions).

The problem is that your applications have to scale as well. It doesn’t work by just putting everything in a shared NAS. Nope. It will not. Not only that will be a complete mess but slow as hell. Without mentioning, all your hardware has to be scale ready as well.

And you are not really achieving redundancy here. If your shared storage fails, now all your servers and services are down. That is actually a worse solution than just running individual servers. NFS and most shared storage will always be slower than local drives because the network roundtrips are slower than local access. So you have a slower, complex and prompt to fail solution. Using an enterprise SAN would probably be a better idea, unless you only plan to run one or two sites on your NFS storage.

Regarding your questions, the way this is done is by having Apache (and everything else) in sync. That means that when you create a user on server A, the same apache configs will be on Server B. You can do this in several ways, like setting a master server and then syncs to slave servers and configuring Virtualmin to run post script that will activate the sync after an account was created/terminated, etc.

Or you could use Ansible or something else for automation. Or just do it manually. Again, there are several ways to do the same. But that is the least of your problems. Your users will have different Linux ID’s and usergroups that you also need to sync or you can use a shared system like LDAP or similar. But if you are running them from the NAS, then the user/groups and even space quota has to work on the shared storage device, not the end servers (quotas may not even work, Virtualmin can’t manage them unless you notify it from the storage). And finally, you also need to make sure PHP sessions are properly in sync with HAproxy.

With all this. Now you realized that your solution sucks, because your shared storage has a limited I/0 and you can’t scale it to a certain limit. And now you also have a single point of failure (your share storage), your switch, etc.

Can it be done? Yes, but if you are asking this questions here, it means this is way over your head in terms of configuration and management. And in the end, even your applications have to be aware of the whole setup. You can’t just expect to put a WordPress site on a shared database cluster and shared storage and expect it work. Ok, for WordPress you can because of its scale ready, but not all applications are or even work in a cluster. So if you are just doing basic static sites, the solution is easy. If you are doing PHP, Perl, or any other dynamic coding environment and you are also doing databases, then it’s a whole other story.

You should Google how to scale Drupal, WordPress, etc. And see that most require code modifications as well. You should also Google scale Apache, MySQL, etc. Then you need to put all together, and then you need to also be able to scale the hardware and storage. How all of them work and play together has nothing to do with Virtualmin. Actually, Virtualmin will be pretty much useless on those cases for managing your sites, (unless you also modified Virtualmin to work with your whole platform).

3

Thank you for your reply. Even if, oddly, you seem to be telling me off for trying to understand.

Of course I don’t know how to do this properly, as that’s obviously why I came here to ask how to do it properly. If I already knew the answers, then I wouldn’t be asking the questions, right?

And, to answer your question, in saying that I wanted to scale up Virtualmin, that really was a shorthand way of saying that, yes, I want to scale up Apache, PHP, mySQL, emails, etc… To put it another way, I want to scale up Virtualmin’s overall functionality (well, minus git repositories, as I’m not using that at the moment). As mentioned, I’ve already got Virtualmin up and running on a single standalone server - but the whole operation will need to be scaled up and that’s what I’m looking into. How I’d go about doing that. This is the beginnings of my research.

Yes, my proposed solution sucks and has a single point of failure. I do realise that. That’s why I’m coming here to ask the questions. I realise that it’s not very good. So I must be doing something wrong. So I’ve come here to ask “how do I do it properly?”. That was more or less literally the main question I trailed that post with. How do I do this properly, as my current ideas clearly aren’t doing it properly.

I did, you’ll notice, give a nod to the idea of synchronising the servers with each other. But I couldn’t see how to do that easily. Hence the network share.

I am surprised, though, to hear that it “requires code modifications”. That’s fine. Coding is really what I do but I’ve been roped into handling the networking here. Yes, I know. Hardly ideal. But it is what it is, unfortunately. I did too well in setting up the single standalone server with Virtualmin that we’ve already got that I’m being happily asked to just scale it up. Which I recognise is a tall order, but everyone else doesn’t. They think you just buy better hardware and it just works.

Am I in over my head here? Totally. That’s why I’m seeking assistance.

Thank you for the reply. Including the rebuke. To be assured that, yes, this is all a bit slightly insane is a reassurance to me.

4

I’m just trying to save you from a bad experience, money but more important time wasted. If you plan to do this for your own site it may not be that much of a work. If you plan to host others people sites. Do you expect to modify the code on every single PHP or web app they upload to make it work with your clustering/platform?

There are better solutions today. Like virtualization that can fail over. That would be much easier and simpler than trying to scale individual services.

The problem with asking this on a forum is that it would require a full forum on its own. It would require pages and pages to explain how to do it properly and this assuming we know exactly all your hardware and network specs + configurations. Like I said before, you actually need to work together with the hardware here. This may sound surprising but its really true. How are you provisioning the IPs in the network and then servers/sites? Do you plan to switch IP’s on a site that is failing over? Do you run BGP sessions? Or how do you plan to fail over the Haproxy otherwise? If you don’t, then you actually have a huge bottle neck in your setup. The load balancer + the storage. So you need to fail over and scale both as well and for that you need to be able to change IP’s dynamically, plus Haproxy alone, assuming you are using the simple hearbeats is too simple. You probably need an external monitoring system to detect when to fail over properly and when to revert it back.

Its ok to ask. Sure. But its not really a question for Virtualmin. Virtualmin does not actually manage any services. Its just a GUI, a Perl software with a web interface to let you click and point manage everything. You can do absolutely everything Virtualmin does by just running command line commands or editing files manually. My point is that Virtualmin can’t scale this for you and there is no easy click and point solution. Virtualmin was never designed for that, and while it does have some features that can help you here, you will need to get dirty by creating your own scripts if you plan for this work. Once you do all this, Virtualmin can’t really manage anything, so its very much useless for that type of environment unless you want it messing up your config files.

Consultants are paid thosuands of dollars for such solutions. And companies may spend even more money for their hardware alone. If you want to this right, you probably need a $200K SAN storage in the first place. If you want to this even better, you need 2 datacenters or 2 installation sites. Using NFS is a home solution. Will it work? Yes. Will it be fast? Nope. Will it have problems with file locks and writes? Sure, once you send anything with a bit of traffic. Can it scale? Nope. Unless you plan different NFS storage to different accounts.

If you plan to offer a simple HA service that can scale as a hosting service. You are looking into a $1 million dollar or more investment. I’m not kidding. That is more or less what you will need in terms of time (people do have to get paid) and hardware in 2 separated facilities. Assuming you plan to do everything yourself and use all open source, you are still looking at $100K in hardware alone for anything you can consider a start up and months of works. Weeks if you know how to build everything but you still need alot of testing. You don’t want to send traffic without load testing everything. In your case, if you plan to do this alone, you actually need to master how Apache, DNS, MySQL, Postfix and other softwares work first. By mastering I mean you need to completely understand how those softwares work and process data. If you don’t you will never be able to fix issues in your platform and not even Virtualmin developers have such extensive knowledge in everything. That assuming you already know everything about networking layers because you need those as well.

You would need to read a book itself just for 1 single service. Example, reading a full book on Apache, is still not enough. Because you need Apache to talk with PHP and MySQL, which means 3 books. And you need DNS sync as well, which means BIND now, and then you need this and that… Assuming you read and know everything, all that is still not real world experience. People that set up these things have years of experience, not months.

What you are basically trying to achieve is not simple. Its very, very complicated. It’s the holy grale of services and I know million dollar companies that did it wrong (and some closed down because of that vs others that did simple things and succeeded). Let me make it clear. There is a reason why Google, Amazon and Microsoft are not offering that. There is no point. Today? Not anymore. You should look into failing complete servers instead of individual services. That is my point. I’m your friend here and you will save time money and frustation. What you are trying to achieve is old school and while I know how do it, I would probably not go that route today.

If you really want a click and point solution that can scale basically unlimited and fail over web applications (without any code changes). Look at Rancher (free open source) or Docker Enterprise Edition (commercial with support).

You will be surprised how easy it is, because doing full containers where the application is not aware of the running services is much easier. Same with servers. (separate the application from the services and the services from the hardware, then separate the data itself). Doing this with full servers, physical or virtual is much easier than doing it with individual services. Trying to scale every single services is useless with most scenarios. And most services will have a limit as well. Example, Apache may have a limit on the number of vhosts it can boot up. It may work with 1000, it may fail with 5,000. Same is true for MySQL and other softwares, you are going to hit a vertical scaling limit (CPU or RAM eventually) and then you can’t scale anymore. You see the problem here? Even if you manage to create such a complex solution, MySQL may be very unstable with lets say 512 GB RAM and 1000 databases running. All softwares are known to memory leaks and bugs. At scale you are going to start hitting those issues like pancake. So its much easier just to manage 100 MySQL servers, than one that does everything.

This also avoids putting all your eggs in the same basket. Sure, failing individual services is still great and not that hard for your own site or a bunch of sites. Assuming they are not extreme high traffic, but scaling a single site or web application is actually an art. Facebook had to develop its own systems because PHP didn’t scale properly or MySQL had problems. So did every single major website with high traffic today. Once you hit a certain point, no software will help you without heavy modifications. Now, if you want to do this for alot of sites, (with a service) that is a nightmare scenario. Let your users do that. And secondly, assuming you even need that. Servers today are so horrible powerful, monster servers, that any server today can handle million of unique visitors with a properly optimized site (Mr. Cache is your friend, turn dynamic pages to static ones).

Even a single monster hardware is enough today to handle most traffics for a site. So scaling horizontally (adding more and more servers) is not a requirement as it was in the past years. Hardly any websites unless you are Amazon or Google needs this. Scaling verticially (adding bigger CPU and RAM) can really serve even huge sites. Huge with millions of visitors a month, in particular when you can buy servers easily with over 512 GB RAM today. My server motherboards, all of them support between 256 GB and 512 GB and they are not even new ones.

I’m your friend here and for your own sanity forget what you are trying to do. Just run individual Virtuamin servers. Master that, and if for some reason you still want this, look into virtualization or containers like I told you. If your concern is uptime then I can inform you that if you are using with a proper datacenter and running quality hardware (server hardware, not desktop hardware) that is not a problem either. Is not worth doing all that for just a few minutes of downtime a year or a 5 minutes a month to run some updates. If you are having outages frequently it means something is not right with your network, servers or Internet providers.

Example, I saw people trying to discover the mystery of why their services crash every 2 weeks because they were so cheap that used standard computing systems with standard RAM sticks on it when they don’t realize that servers use ECC registered Ram for a reason. (error correction). Or having high load because they throw everything into a single drive instead of using a proper RAID hardware card with multiple disks (I/O has a limit). Or people buying budget servers with some vendors that offer desktop machines as servers and then having all sorts of horrible performance and service crashes. That hardware was never designed to run 24/7…Sure you can do that if you are Google because they treat machines as cattle, it doesn’t matter if you burn ten today the data is resilient and lives in multiple systems. If you want that, you may look into CoreOS and KuberDock and but more important Mesos which is another story.

I could go on here for pages and pages, but then again, I have to put food on my table :slight_smile:

Trust my advice. If you want to play with this around to learn, you are welcome. If you want to offer a commercial service, the fact you asked this here means you are not ready, and you would just provide a horrible service to your users or let’s say disservice.

You can trust my word and save time and money and lots of frustrations or just experience the pain on your own. Some people may have time or money to burn. But never both. People that don’t have time, usually have money. People that don’t have money tend to have a lot of free time. If you are the on the last option you can learn everything yourself, if you are the first one, just pay someone to do it for you. In the end, time is money, and that is exactly how you should take it, regardless of the time it took to implement something.

Either way, good luck with your project :slight_smile: