I’d be willing to bet that the reason things weren’t working on the old system was that spinning up the spamassassin processor was failing, due to memory allocation errors. The standalone program would use no memory when no processing mail, but it would have to allocate a new (rather large) block of memory whenever mail was received. Given that the system was running out of memory when simply installing packages, I’m sure the big SpamAssassin process would occasionally (maybe always) fail, too, and for the same reason.
Processing mail is the most memory-intensive function on many Virtualmin systems. SpamAssassin is quite large, and ClamAV is huge. You need about 130MB free (I think, though it’s been a while since I did the math) in order to start both. On a 286MB system with all the other services running, that may just not be possible.