Multiple Scripts Installed on a Virtual Server

What is a good common practice for installing multiple scripts? Ex: I install Wordpress first to make he “base site” and say i want to add a shop with Magento, and a wiki with MEdiaWiki…

Do i make Magento in a sub directory called Magento, or shop…and install the Wiki under a sub directory called MediaWiki, or wiki_name?

Or is it better to make them as sub domains?

If security is a priority then create a separate virtual server for each domain. Then if one of your domains, say the WordPress one, has a vulnerability and a hacker gains access, it will be limited to the files and database(es) of the WordPress domain and not all your files and all databases of all your domains. If you put everything under one domain / virtual server, then when a hacker gains access it will be to all your files and all databases.

If billing / resource allocation is important then stick to your original idea: create a top-level server for WordPress and install the rest of the scripts in their own directories / sub servers. You can then control how much disk space and CPU cycles the user / virtual server can consume on your shared hosting service.

I recommend you give security a priority, considering you are using WordPress, so create a seperate user / top level virtual server and domain / subdomain for each script. You could then tweak disk quota and PHP processes manually to control the overall resources consumed by all the domians / sub domains if that is important.

I will probably stick with scripts as subdomains, so if I install magneto, I can install it at shop.mydomain.com

I tried playing with installing in sub folders/directories but the installers never start(for Magento or mediawiki) and that was my biggest issue

Yup, sub domains on top level virtual servers would be best.