I don’t know anything about CageFS, but Virtualmin Pro has resource limits, wherein you can limit number of processes, memory per process, and CPU time per process. The online help for these options is:
This page allows you to define limits on CPU and memory use that apply to this virtual server and all sub-servers. Limits apply to both PHP scripts, and commands run via an SSH login. They can be used to prevent any one domain owner from overloading the system with a fork bomb or commands that use an excessive amount of RAM.
The available limit types are :
Maximum number of processes
Attempting to launch more processes running as the server’s Unix user than the limit set here will fail. This applies to concurrent PHP scripts too.
Maximum size per process
Any single process that tries to allocate more memory that this limit will be terminated. However, the sum total of all processes owned by the server can still exceed the limit.
Maximum CPU time per process
Any process whose time spent actually using the CPU (not waiting for IO or sleeping) exceeds this limit will be killed.
These options are in addition to the obvious disk quotas, domain/account/database limits, etc. There’s also a variety of monitoring and reporting features in Webmin and Virtualmin.