Bug 1088 - Add an ability to set netmasks for --ipadd
Add an ability to set netmasks for --ipadd
Status: CLOSED FIXED
Product: OpenVZ
Classification: Unclassified
Component: vzctl
zzz-rhel5-2.6.18_028stabXXX
x86_64 (AMD64) RHEL/CentOS 5
: P2 major
Assigned To: Kir Kolyshkin
: 1293 1671 1858 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-23 22:23 EST by Kevin Heatwole
Modified: 2015-04-01 14:45 EDT (History)
11 users (show)

See Also:


Attachments
Patch for debian-add_ip script, which allows to pass netmask in /cidr notation (1.52 KB, patch)
2010-09-27 09:38 EDT, Guido
Details | Diff
if-up.d script to have the right source address when there are 2 or more IPs in the VE. (320 bytes, application/octet-stream)
2011-06-04 16:19 EDT, Simon Deziel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Kevin Heatwole 2008-11-23 22:23:25 EST
My HN has eth0 with ISP's Private VLAN and eth1 with Public WAN.  The server has 8 Public IPs (5 usable) and 8 Private IPs (5 usable).

If I create a VE and set a unique Public IP and a unique Private IP for the CT, within the VE, only the network for the first IP in the CT forwards to that network's gateway.  If I reverse the order that the IP addresses are assigned, still only the first network goes through the gateway as defined on the HN.

So, if the first address added to the CT is the Public WAN IP, then the VE in the CT can see the internet (e.g., ping google.com) but cannot see the Private VLAN (e.g., ping 10.0.0.1).

Conversely, if the first address added to the CT is the Private VLAN IP, then the VE in the CT can see the Private VLAN (e.g., ping 10.0.0.1) but cannot see the Public WAN (e.g., ping google.com).

I assume there is a route missing but when I check the routes, they are the same on both the HN and in the VE regardless of the order that the Private and Public IPs were assigned in the VE.

Here are the routes in the VE:

[root@vps102p /]# ip route
192.0.2.0/24 dev venet0  scope host 
169.254.0.0/16 dev venet0  scope link 
default via 192.0.2.1 dev venet0 

Here are the routes in the HN

[root@node1 ~]# ip route
63.248.94.19 dev venet0  scope link 
63.248.94.21 dev venet0  scope link 
63.248.94.20 dev venet0  scope link 
10.0.15.52 dev venet0  scope link 
10.0.15.51 dev venet0  scope link 
63.248.94.16/29 dev eth1  proto kernel  scope link  src 63.248.94.18 
10.0.15.48/29 dev eth0  proto kernel  scope link  src 10.0.15.50 
10.0.0.0/16 via 10.0.15.49 dev eth0 
169.254.0.0/16 dev eth1  scope link 
default via 63.248.94.17 dev eth1 

(I changed my public IPs above just to keep from posting them here).

If this isn't a bug, but a feature (only first IP in a VE routes through the gateway of the HN, then I think it probably should be implemented so that both networks are accessible.

This should be fairly common set up and everything is so close to working as if by magic with the venet0 interface.  Maybe it is just me.  I just downloaded OpenVZ and I am a bit of a newbie to all this networking stuff.
Comment 1 Pavel Emelyanov 2008-11-24 04:50:22 EST
Vitaly, yours.
Comment 2 Kevin Heatwole 2008-11-24 11:43:34 EST
Doing a little more investigation here, it appears that within a VE, the traceroutes all go through the Private VLAN interface of eth0 even for the internet.  Apparently, the gateway IP has a route to the Public Internet.

So, I guess there is no routing done on the HN to switch over to eth1 (specifically, the default routing rules to send google.com ip over eth1).

I'm not real sure why 10.0.0.1 pings okay if the first IP is the Private VLAN IP for the CT and doesn't ping okay if first is Public VLAN IP.  It must be that somehow the route on eth0 on the HN is being honored if first IP is Private VLAN but is ignored and it goes directly out eth0 which has this IP masked.

Anyway, I'm a newbie at networking, but I can kind of see what is going on now.  I really wish there was a wiki entry that explained exactly what I want to do.  That is, obey the routing rules of the HN when the HN has private network on eth0 and public network on eth1.
Comment 3 Den Lunev 2008-11-25 04:47:38 EST
pls provide the following information for a container:
- ip r l t local
- ip a l
- ip r g <private ip>
- ip r g <public ip>
Comment 4 Den Lunev 2008-11-25 04:58:56 EST
the problem is that both IPs have /32 mask. Unfortunately, it can't be changed as 'vzctl' overwrites it on each VE reboot

So, you may use veth as a temporary solution.

Igor, it seems that vzctl should understand syntax like --ipadd 1.1.1.1/24 (with default to /32)
Comment 5 Vitaliy Gusev 2008-11-25 05:04:20 EST
No vzctl doesn't understand --ipadd IPADDR/24

Really, "vzctl set VEID --ipadd IPADDR" adds ipaddr to venet0 with netmask 255.255.255.255, so for any dst address from non-local net, src is a first addr from venet0 (global scope)

[root@suse10ve2 /]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/void
    inet 127.0.0.1/32 scope host venet0
    inet 10.30.40.41/32 brd 10.30.40.41 scope global venet0:0
    inet 192.168.1.15/32 brd 192.168.1.15 scope global venet0:1

vzctl doesn't provide tools to manipulate netmask for venet.

Thus, temporary solution is reset by hand ip-addr in venet0 with netmask 255.255.255.0 (or which you want)



Comment 6 Vitaliy Gusev 2008-11-25 05:05:24 EST
Igor, can we add this functionality to vzctl?
Comment 7 Kevin Heatwole 2008-11-25 09:05:13 EST
I think I found a workaround to my problem.

The problem, from my point of view, was that when you add two ips to a VE, only the HN routes for the ethernet interface that goes with the first IP is honored (for traffic originating within the VE).  So, if you define the ips in a different order, you get different routing.

The workaround was to always put the private ip first (for the VE) and then use iptables to route private traffic to the public interface for the HN:

iptables -t nat -A POSTROUTING -s 10.0.15.64/27 -o eth1 -j SNAT --to 63.248.94.18

This seems to have fixed my problem.

I don't quite understand why the venet0 inside the VE can route traffic to all IPs on the subnets of eth0 and eth1 regardless of order, but doesn't pick up all routes defined on the HN (specifically, the routes defined for the second IP).

This should be more seamless, but the workaround seems to work for me.
Comment 8 Kevin Heatwole 2008-11-25 09:17:55 EST
Another thought...  maybe this is just the way it is and what I've run across isn't a bug, but a feature.

I only have 1 server on my two subnets, so when I say that the VE can route to the other subnet, I'm not sure that is correct.  It really can route to all the VEs on the HN.  Maybe that is all that works.

And, maybe expecting the HN to bridge the two networks for the VEs without something like SNAT, is unrealistic. 

Maybe the resolution is to just clearly document how to set up VEs with multiple IPs and put an emphasis on the order defined is important and add that SNAT is needed to bridge the two networks on the HN.
Comment 9 Kir Kolyshkin 2009-10-26 13:15:28 EDT
Allright, we need to add functionality of specifying a netmask while adding an IP.

I am working on that
Comment 10 Kir Kolyshkin 2009-10-26 13:16:13 EDT
*** Bug 1293 has been marked as a duplicate of this bug. ***
Comment 11 Kir Kolyshkin 2010-03-03 11:50:51 EST
Roman has emailed me he can't work on this right now, so changing back to NEW and the default owner (poor me).
Comment 12 Kir Kolyshkin 2010-03-03 12:02:35 EST
This one is pretty important
Comment 13 Guido 2010-09-27 09:38:15 EDT
Created attachment 1275 [details]
Patch for debian-add_ip script, which allows to pass netmask in /cidr notation

Hi Kir,
I've made this patch, that checks if the IP_ADDR env variable contains a netmask in 0.0.0.0/24 notation, and sets the netmask accordingly, or defaults to /32 (as it is now).

Sadly, vzctl won't let me pass this kind of notation, spitting: 

    # vzctl set 100 --ipadd 10.0.0.1/24 --save
    Bad parameter for --ipadd: 10.0.0.1/24

But at least half of the work is done :) If someone could patch the vzctl utility, making it accept this kind of notation for --ipadd parameter, and passing it as-is to the ip_add.sh script, it would be great.
In that case, I offer "as a reward" my time to patch the rest of the distro scripts (gentoo-ip_add.sh , redhat-ip_add.sh , etc) :)

Cheers!

Guido
Comment 14 Roman Ustyugov 2010-09-30 02:14:51 EDT
(In reply to comment #13)
> Created attachment 1275 [details]
> Patch for debian-add_ip script, which allows to pass netmask in /cidr notation
> 
> Hi Kir,
> I've made this patch, that checks if the IP_ADDR env variable contains a
> netmask in 0.0.0.0/24 notation, and sets the netmask accordingly, or defaults
> to /32 (as it is now).
> 
> Sadly, vzctl won't let me pass this kind of notation, spitting: 
> 
>     # vzctl set 100 --ipadd 10.0.0.1/24 --save
>     Bad parameter for --ipadd: 10.0.0.1/24
> 
> But at least half of the work is done :) If someone could patch the vzctl
> utility, making it accept this kind of notation for --ipadd parameter, and
> passing it as-is to the ip_add.sh script, it would be great.
> In that case, I offer "as a reward" my time to patch the rest of the distro
> scripts (gentoo-ip_add.sh , redhat-ip_add.sh , etc) :)
> 
> Cheers!
> 
> Guido

Hi, Guido!
I've already done the ipaddress/mask parsing in vzctl. I will send that patches to Kir in few days. So the final part of this work will be changing of distro scripts. And if I need your help for this part of work I will let you know. :)
Comment 15 Kir Kolyshkin 2011-05-04 10:15:57 EDT
*** Bug 1858 has been marked as a duplicate of this bug. ***
Comment 16 Ulrich Goettlich 2011-05-04 10:54:36 EDT
My Bug 1858 was closed, as a duplicate of this bug. Can you tell me the state of this (1088) Bug? 
I thought this bug is only about defining subnet masks. I need the ability to configure the routing table for a VE. We got 2 Interfaces and 2 default routes on our VE0 so I have to choose the routing table to get the traffic out of the right interface. Is this also included in this bug?
Comment 17 Simon Deziel 2011-06-04 16:19:10 EDT
Created attachment 1502 [details]
if-up.d script to have the right source address when there are 2 or more IPs in the VE.
Comment 18 Simon Deziel 2011-06-04 16:20:25 EDT
On Debian there is an easy workaround. The following (executable) files (also attached here) need to be copied to the VE.

/etc/network/if-up.d/internal-routes:

#!/bin/sh

# This route is required to have outgoing packets coming from 
# the private IP instead of the first IP assigned to the VZ
# This allow packets from 10.1.30.0/24 to be normally routed
# by the host

if echo $IF_ADDRESS | grep -q "^10\.1\.30\." ; then
  ip route add 10.1.30.0/24 dev venet0 src $IF_ADDRESS
fi
Comment 19 Ulrich Goettlich 2011-06-06 17:24:37 EDT
(In reply to comment #18)
> On Debian there is an easy workaround. The following (executable) files (also
> attached here) need to be copied to the VE.
> 
> /etc/network/if-up.d/internal-routes:
> 
> #!/bin/sh
> 
> # This route is required to have outgoing packets coming from 
> # the private IP instead of the first IP assigned to the VZ
> # This allow packets from 10.1.30.0/24 to be normally routed
> # by the host
> 
> if echo $IF_ADDRESS | grep -q "^10\.1\.30\." ; then
>   ip route add 10.1.30.0/24 dev venet0 src $IF_ADDRESS
> fi

This does not fix my Problem. I don't want to use more IP-Adresses on the Guest, I got 2 Network Interfaces on VE0 (Host). Each of this Interfaces is able to route to the internet (2 routing tables each with its own default route). I need a functionality to set the interfaces used for outcomming traffic for each guest.
In other words.
VEID 100 i.E. uses eth0 on host for outgoing (and incomming) traffic
VEID 101 i.E. uses eth1 on host for outgoing (and incomming) traffic
Comment 20 Kir Kolyshkin 2011-06-28 16:44:11 EDT
*** Bug 1671 has been marked as a duplicate of this bug. ***
Comment 21 Kir Kolyshkin 2011-06-28 16:46:25 EDT
OK last git commit makes vzctl set --ipadd/--ipdel able to parse netmasks in CIDR notation:
http://git.openvz.org/?p=vzctl;a=commitdiff;h=f018af5faa10fbe2fa56363c5b393485372c246f

Next step is to work on making distscripts understand (or at least ignore) the netmask
Comment 23 Kir Kolyshkin 2011-07-05 18:04:52 EDT
Fixed in GIT, will be available in vzctl >= 3.0.29

Also, I have made a nightly build available for those who want to test it.

http://download.openvz.org/utils/nightlies/vzctl/3.0.28.3-git.57.30d0cf8/

PLEASE TEST and report bugs to bugzilla.
Comment 24 Kir Kolyshkin 2011-07-06 10:35:32 EDT
A newer build (version fixed to allow upgrade from 3.0.28.3):

http://download.openvz.org/utils/nightlies/vzctl/3.0.28.3-58.git.3502d2b/

please test
Comment 25 Kir Kolyshkin 2011-07-07 04:12:10 EDT
This stuff is not working yet. Will fix.
Comment 26 Kir Kolyshkin 2011-07-07 04:56:33 EDT
http://download.openvz.org/utils/nightlies/vzctl/3.0.28.3-62.git.4a0dd88/
should work better, but changing the mask of IP address doesn't work as expected. Will fix.
Comment 27 Kir Kolyshkin 2011-07-22 03:00:32 EDT
Pity that no one tests it...
Comment 29 Kir Kolyshkin 2011-08-24 12:53:38 EDT
http://download.openvz.org/utils/nightlies/vzctl/current/

Please test vzctl-3.0.28.3-79.git.ba6d944 wrt IP netmask functionality.
Comment 30 Kir Kolyshkin 2011-09-15 09:06:20 EDT
Fixed in vzctl >= 3.0.29
Comment 31 Mario Kleinsasser 2011-09-19 15:55:04 EDT
(In reply to comment #30)
> Fixed in vzctl >= 3.0.29

Tested on latest Debian 6 base installation with custom compiled kernel.
Kernel: Vanilla 2.6.32
Patch: feoktistov

vzctl: 3.0.29.1-1
vzctl-lib: 3.0.29.1-1
vzdump: 3.0.12-1

All OK!
Comment 32 Hristo 2011-09-20 12:41:18 EDT
I tested on CentOS 6 x64 HN and CentOS 6 VE template and it doesn't work 100%. vzctl understands masks in CIDR just fine, the configured mask is set in the VE properly, so this part it ok, but for some reason there is no network connectivity as soon as I add a mask to the IP. This is also true when I explicitly set /24 as mask (which seems to be the default when there is no mask configured)

When I have no mask in the configuration then the mask inside the VE default to /24. In this case all networking in the VE works just fine. However, as soon as I try to add mask (any mask including /24) and restart the VE then there is no networking inside the VE. Again, the configured mask is set properly (checked with ifconfig inside the VE), but there is no network.

I even tried to capture some packets on the HN using wireshark, but there were no packets coming from the VE IP.

I have tested on two HNs with two different VEs:
- VE with a single IP (described above) and
- VE with two IPs from two different subnets (on a HN which has two interfaces).

The result is that as soon as I add a mask, the IP communication for this IP/interface in the VE is lost. The other IP/interface works just fine (in the case with two IPs/subnets). Also even when I add the default mask (/24) the problem is still present.

Tested with:
kernel: 2.6.32-042stab037.1
vzctl: 3.0.29.1
NH: centos6-x64
VE: centos6-x64 (template downloaded today)
Comment 33 Mario Kleinsasser 2011-09-20 14:58:25 EDT
Well I'am sorry, but it seems that I made a mistake yesterday. What Hristo posted is correct. Sad but true.

(In reply to comment #32)
> I tested on CentOS 6 x64 HN and CentOS 6 VE template and it doesn't work 100%.
> vzctl understands masks in CIDR just fine, the configured mask is set in the VE
> properly, so this part it ok, but for some reason there is no network
> connectivity as soon as I add a mask to the IP. This is also true when I
> explicitly set /24 as mask (which seems to be the default when there is no mask
> configured)
> 

Right. The setting is set correctly. Today I double checked it with vzlist -a. 

> When I have no mask in the configuration then the mask inside the VE default to
> /24. In this case all networking in the VE works just fine. However, as soon as
> I try to add mask (any mask including /24) and restart the VE then there is no
> networking inside the VE. Again, the configured mask is set properly (checked
> with ifconfig inside the VE), but there is no network.

I tested it without restart and had the same behavior today.

> 
> I even tried to capture some packets on the HN using wireshark, but there were
> no packets coming from the VE IP.
> 
> I have tested on two HNs with two different VEs:
> - VE with a single IP (described above) and
> - VE with two IPs from two different subnets (on a HN which has two
> interfaces).
> 
> The result is that as soon as I add a mask, the IP communication for this
> IP/interface in the VE is lost. The other IP/interface works just fine (in the
> case with two IPs/subnets). Also even when I add the default mask (/24) the
> problem is still present.
> 

Not tested at this detail level.

Sorry once again, it seems that yesterday I wasn't careful enough.
Comment 34 Kir Kolyshkin 2011-09-21 05:35:58 EDT
Got you guys, thanks for the comments, the problem with the networking is a subject of bug #2001
Comment 35 Sergey Bronnikov 2015-04-01 14:45:26 EDT
Bug was fixed more than one year ago and there were no complains from reporter after fix. We believe bug fix helped and mark bug as closed.