Looking for some planning/rebuilding guidance

Hi All,

The more I work with Cloudmin the more i love it but I feel like I set everything up wrong. Recently I have been having problems with my Ubuntu VMs where they cannot apt-get update (Cannot connect) and other such networking troubles (this is related to virtualization and networking as opposed to inaccesible hosts at ubuntu) . My systems are unorthodox so I recognize there are lots of potential things wrong, now I just need to figure out how to rebuild my infrastructure correctly in situ. with three hosts I should be able to fix one at a time and move my vms around until I have all three set up correctly.

So, generally speaking, I have 3 Host systems, each with 2 nics, This was originally done to have two separate but secure networks that can access each system (and the VMs they host) from two separate networks but never have the networks talk to each other. This was done to satisfy the security folks here. (we are a govt operation so making them happy comes before practicality or functionality) So on the internet side, I have a DSL modem that connects to the public side of a firewall (IPcop). IPcop acts as the primary internet firewall and provides DHCP for a private network 192.168.80.0 (255.255.255.0)

So each of my 3 host systems (HP Proliant DL 360 G6 running 64 bit Ubuntu 12.04.1 LTS) has a nic on the local lan, 216.xxx.xxx.0 (255.255.255.128) - inward facing - Intranet if you will, and one on the private network to the internet

Now on each host I have configured the network as such:

The loopback network interface

auto lo
iface lo inet loopback

The CORP network interface

auto eth0
iface eth0 inet manual

CORP Bridge for VMs on this host

auto br0
iface br0 inet static
address 216.xxx.xxx.10
netmask 255.255.255.128
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
dns-nameservers 192.168.80.1 192.235.200.134

The INTERNET network interface

auto eth1
iface eth1 inet manual

INTERNAT Bridge for VMs on this host

auto br1
iface br1 inet dhcp
bridge_ports eth1
bridge_stp off
bridge_fd 0
bridge_maxwait 0
gateway 192.168.80.1

I have installed, Cloudmin on one of these servers, and added the other two as KVM hosts.

Further, I have set up logical groups and volumes on each host as such:
groups: lg_storage (130gb) lg_system (68 gb)
volumes on lg_storage: lv_backups(50GB)
volumes on lg_system lv_root (18gb) lv_swap (9gb)

The Host OS is installed on lv_root and uses lv_swap as swap.
Cloudmin uses free space on lv_storage on each host to create logical volumes for each VM.

automated cloudmin backups use lv_snapshots and save to lv_backups on each server

most virtual machines have two nics so they can be accessed from the internet and the local lan both. but I have been systematically trying to move away from this and simply have intranet users access the internet version but some systems need direct access to secure systems on the govt lan, so it cant be nixed entirely.
an example of this networking inside the vm (/etc/network/interfaces)looks like this:

The loopback network interface

auto lo eth0 eth1
iface lo inet loopback

The primary network interface CORP network

iface eth0 inet static
address 216.xxx.xxx.59
netmask 255.255.255.128
up ip route add 216.xxx.xxx.0/25 via 216.xxx.xxx.1

Secondary is INTERNET

iface eth1 inet static
address 192.168.80.245
netmask 255.255.255.0
broadcast 192.168.80.255
network 192.168.80.0
gateway 192.168.80.1

while this works for most things including access to this VM over the internet and this VM connecting to private secure servers on the Corp lan, it cannot seem to

sudo apt-get update

It seems my older VMs (ubuntu 10.10) which were migrated from virtmanager and converted into cloudmin VMs seem to have strange hangovers and the result is mostly:

Err http://security.ubuntu.com maverick-security/restricted Sources
404 Not Found [IP: 91.189.92.190 80]

or they hang like this:

0% [Connecting to ca.archive.ubuntu.com] [Connecting to security.ubuntu.com]…

I have read as much and as fast I can to understand the nuances of Cloudmin and for the most part I love it, I just dont understand it enough to troubleshoot these strange networking issues I have.

My VMs have to talk to each other, and servers on the local lan, and to websites on the internet, and this system needs to be mission critical before the spring.

So if you can see something that I have done wrong, or even if I have done it the “less than desirable” way, I would love to hear your thoughts and make this work better.

Thanks For any and all advice/discussion
Franco Nogarin

Howdy,

Hmm, well, it sounds like you’re saying that it mostly works… that it’s accessible from the Internet.

It’s just that you’re seeing some errors during the “apt-get update”.

One thing I see is that it looks like you’re running Ubuntu 10.10 – and that’s not actually supported any longer, it reached it’s EOL back in April 2012… so the one error you’re seeing is actually expected – that’s just saying that the repository it’s trying to contact no longer exists.

That may actually explain all the issues you’re seeing.

Are your more recent VM’s working as expected, and able to run an “apt-get update” without a problem?

-Eric

Hi Eric, ya thats the ubuntu issue for sure, my solution is to rebuild the VMs from scratch using LTS versions and to take advantage of the imaging system in cloudmin to keep them updated. so thanks, other than that can you see any issues about the way this is built? I still have problems getting certain vms to talk to each other… eg unable to ping each other on the same subnet.

Ah, I think I mis-read your initial post.

I was initially thinking you said your VM’s could communicate with each other, but re-reading it, that’s not what you said :slight_smile:

Okay, so a few thoughts –

Regarding Ubuntu – you can upgrade Ubuntu to an LTS release. Or you could pull down Ubuntu 10.04 or 12.04 from the Cloudmin images that are available.

Regarding the VM’s – just to clarify the problem that you’re seeing, I had a few questions –

Can the VM’s ping the host that’s running Cloudmin?

Although the VM’s can’t contact each other, can they contact other IP’s on your LAN (I believe you said they could, but I wanted to make sure I understood that correctly)?

What IP addresses are your VM’s using, are they using 216.xxx.xxx.xxx IP’s, or 192.168.xxx.xxx IP’s?

Thanks!

-Eric

thanks for looking at this Eric, the vms use IPs on both subnets, some have an ip in CORP have an ip in PRIVATE still others have both.

No, the vm and host cannot ping each other on the problem vms.

One example is vm-gs-64 which has both interfaces. the CORP IPs work buth the PRIVATE ones do not.
the config looks like this in the vm:

sparcsadmin@vm-gs-64:~$ cat /etc/network/interfaces

This file describes the network interfaces available on your system

and how to activate them. For more information, see interfaces(5).

The loopback network interface

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 216.xxx.xxx.116
netmask 255.255.255.128
hwaddr ether 54:52:00:76:d0:99

auto eth1
iface eth1 inet static
address 192.168.80.140
netmask 255.255.255.0
hwaddr ether 02:54:00:af:9c:2d
gateway 192.168.80.1

routing in the vm looks like this:

sparcsadmin@vm-gs-64:~$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 216.xxx.xxx.0 * 255.255.255.128 U 0 0 0 eth0 192.168.80.0 * 255.255.255.0 U 0 0 0 eth1 default 192.168.80.1 0.0.0.0 UG 100 0 0 eth1

The host the VM lives on SMVMBACKUP64 has this config:

sparcsadmin@smvmbackup64:~$ cat /etc/network/interfaces

This file describes the network interfaces available on your system

and how to activate them. For more information, see interfaces(5).

setup by FN on Nov 6, 2012

The loopback network interface

auto lo
iface lo inet loopback

The CORP network interface

auto eth0
iface eth0 inet manual
auto br0
iface br0 inet static
address 216.xxx.xxx.12
netmask 255.255.255.128
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_maxwait 0

The INTERNET network interface

auto eth1
iface eth1 inet manual
auto br1
iface br1 inet dhcp
gateway 192.168.80.1
bridge_ports eth1
bridge_stp off
bridge_fd 0
bridge_maxwait 0

and the routing on the host looks like this:

sparcsadmin@smvmbackup64:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ipcop-wxnet.spa 0.0.0.0 UG 100 0 0 br1
192.168.80.0 * 255.255.255.0 U 0 0 0 br1
216.xxx.xxx.0 * 255.255.255.128 U 0 0 0 br0

pinging from the vm to host looks like this:

sparcsadmin@vm-gs-64:~$ ping 216.xxx.xxx.12
PING 216.xxx.xxx.12 (216.xxx.xxx.12) 56(84) bytes of data.
64 bytes from 216.xxx.xxx.12: icmp_req=1 ttl=64 time=11.2 ms
64 bytes from 216.xxx.xxx.12: icmp_req=2 ttl=64 time=0.119 ms
64 bytes from 216.xxx.xxx.12: icmp_req=3 ttl=64 time=0.111 ms
^C
— 216.xxx.xxx.12 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.111/3.828/11.254/5.250 ms

sparcsadmin@vm-gs-64:~$ ping 192.168.80.130
PING 192.168.80.130 (192.168.80.130) 56(84) bytes of data.
From 192.168.80.140 icmp_seq=1 Destination Host Unreachable
From 192.168.80.140 icmp_seq=2 Destination Host Unreachable
From 192.168.80.140 icmp_seq=3 Destination Host Unreachable
^C
— 192.168.80.130 ping statistics —
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3013ms

pinging from the host looks like this:

sparcsadmin@smvmbackup64:~$ ping 192.168.80.140 PING 192.168.80.140 (192.168.80.140) 56(84) bytes of data. From 192.168.80.130 icmp_seq=1 Destination Host Unreachable From 192.168.80.130 icmp_seq=2 Destination Host Unreachable From 192.168.80.130 icmp_seq=3 Destination Host Unreachable ^C --- 192.168.80.140 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3015ms pipe 3 sparcsadmin@smvmbackup64:~$ ping 216.xxx.xxx.116 PING 216.xxx.xxx.116 (216.xxx.xxx.116) 56(84) bytes of data. 64 bytes from 216.xxx.xxx.116: icmp_req=1 ttl=64 time=0.145 ms 64 bytes from 216.xxx.xxx.116: icmp_req=2 ttl=64 time=0.088 ms 64 bytes from 216.xxx.xxx.116: icmp_req=3 ttl=64 time=0.077 ms ^C --- 216.xxx.xxx.116 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.077/0.103/0.145/0.030 ms