Bug 1732 - half-configured container starts
half-configured container starts
Status: CLOSED FIXED
Product: OpenVZ
Classification: Unclassified
Component: vzctl
unspecified
All All
: P2 major
Assigned To: Andrey Vagin
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-12-23 13:28 EST by Andrew Moore
Modified: 2015-04-01 14:44 EDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Moore 2010-12-23 13:28:10 EST
vzctl 3.0.25 gives containers the amount of memory on the hostnode upon reboot, It also gives 'fake' swap which makes the OpenVZ VE appear to have the amount of swap on the main node.

A workaround for the time being is downgrading to 3.0.24.

Hi,

You had me worried for a minute... Looks like there a sefualt in either the kernel or vzctl.

[root@13 ~]# vzctl restart 1034
Removing stale lock file /vz/lock/1034.lck
Restarting container
Stopping container ...
Container was stopped
Container is unmounted
Starting container ...
Container is mounted
Adding IP address(es): 184.154.13.109
Setting CPU limit: 200
Setting CPU units: 1000
Setting CPUs: 2
Setting devices
Segmentation fault

2010-12-23T10:58:11-0500 vzeventd : Child 19264 failed with exit code 79
2010-12-23T11:03:25-0500 vzctl : CT 1034 : Removing stale lock file /vz/lock/1034.lck
2010-12-23T11:03:25-0500 vzctl : CT 1034 : Restarting container
2010-12-23T11:03:25-0500 vzctl : CT 1034 : Stopping container ...
2010-12-23T11:03:34-0500 vzctl : CT 1034 : Container was stopped
2010-12-23T11:03:34-0500 vzctl : CT 1034 : Container is unmounted
2010-12-23T11:03:34-0500 vzctl : CT 1034 : Starting container ...
2010-12-23T11:03:34-0500 vzctl : CT 1034 : Container is mounted
2010-12-23T11:03:35-0500 vzctl : CT 1034 : Adding IP address(es): 184.154.13.109
2010-12-23T11:03:35-0500 vzctl : CT 1034 : Setting CPU limit: 200
2010-12-23T11:03:35-0500 vzctl : CT 1034 : Setting CPU units: 1000
2010-12-23T11:03:35-0500 vzctl : CT 1034 : Setting CPUs: 2
2010-12-23T11:03:35-0500 vzctl : CT 1034 : Setting devices
2010-12-23T11:04:29-0500 vzeventd : Child 2261 failed with exit code 79


vzctl restart 1034
Removing stale lock file /vz/lock/1034.lck
Restarting container
Stopping container ...
Container was stopped
Container is unmounted
Starting container ...
Container is mounted
Adding IP address(es): *.*.*.*
Setting CPU limit: 200
Setting CPU units: 1000
Setting CPUs: 2
Setting devices
Segmentation fault
Comment 1 Phill 2010-12-23 13:50:00 EST
Confirmed.
Comment 2 Phill 2010-12-23 13:57:54 EST
This has also been tested with the latest CentOS 5 kernel from your repo, all with the same results.

I think we need a little more documentation on this. If a yum upgrade is all it takes to cause these issues then theres going to be a lot of upset people out there.

Incase anyone needs a fix, heres the instructions:

Check the version of vzctl you have installed: 

rpm -qa | grep vzctl

Remove version 3.0.25-1: 

yum remove vzctl vzctl-libDownload and install version 3.0.24-1: 

For a 64bit host: 

cd /tmp
wget http://download.openvz.org/utils/vzctl/3.0.24/vzctl-3.0.24-1.x86_64.rpm
wget http://download.openvz.org/utils/vzctl/3.0.24/vzctl-lib-3.0.24-1.x86_64.rpm
rpm -ihv vzctl-3.0.24-1.x86_64.rpm vzctl-lib-3.0.24-1.x86_64.rpm

For a 32bit host: 

cd /tmp
wget http://download.openvz.org/utils/vzctl/3.0.24/vzctl-3.0.24-1.i386.rpm
wget http://download.openvz.org/utils/vzctl/3.0.24/vzctl-lib-3.0.24-1.i386.rpm
rpm -ihv vzctl-3.0.24-1i386.rpm vzctl-lib-3.0.24-1.i386.rpm

Reboot the containers after.
Comment 3 Phill 2010-12-23 14:13:50 EST
Actually can you roll it back in the openvz repo for RHEL ? i'm getting a hell of a lot of tickets about this and it's getting very close to Christmas for comfort.
Comment 4 Kir Kolyshkin 2010-12-24 07:03:38 EST
It looks like you are describing two issues in one bug.

The segfault is a dup of #1729.

The "memory issue" is something that you need to emphasize on. I have just checked container's memory and swap using both vzctl-3.0.24 and vzctl-3.0.25.1 -- the output of 'cat /proc/user_beancounters' and 'free' are about the same for both.

Feel free to reopen and explain what do you mean by 'memory issues'.

*** This bug has been marked as a duplicate of bug 1729 ***
Comment 5 Andrew Moore 2010-12-24 07:07:40 EST
(In reply to comment #4)
> It looks like you are describing two issues in one bug.
> 
> The segfault is a dup of #1729.
> 
> The "memory issue" is something that you need to emphasize on. I have just
> checked container's memory and swap using both vzctl-3.0.24 and vzctl-3.0.25.1
> -- the output of 'cat /proc/user_beancounters' and 'free' are about the same
> for both.
> 
> Feel free to reopen and explain what do you mean by 'memory issues'.
> 
> *** This bug has been marked as a duplicate of bug 1729 ***

The issue with 3.0.25 is when the container is rebooted it gives the container 'fake' memory, Reboot a container with vzctl 3.0.25 and then vzctl into the container and free -m, It will show Swap & the amount of memory on the HN.
Comment 6 Phill 2010-12-24 07:27:40 EST
(In reply to comment #4)
> It looks like you are describing two issues in one bug.
> The segfault is a dup of #1729.
> The "memory issue" is something that you need to emphasize on. I have just
> checked container's memory and swap using both vzctl-3.0.24 and vzctl-3.0.25.1
> -- the output of 'cat /proc/user_beancounters' and 'free' are about the same
> for both.
> Feel free to reopen and explain what do you mean by 'memory issues'.
> *** This bug has been marked as a duplicate of bug 1729 ***

Basically vzctl is not reading the .conf for the vps. When it hits the segfault it continues to boot but does not set any settings from the config after that point. 

I agree it's the same bug but it's fairly serious.
Comment 7 Kir Kolyshkin 2010-12-24 10:53:52 EST
(In reply to comment #6)
> Basically vzctl is not reading the .conf for the vps. When it hits the segfault
> it continues to boot but does not set any settings from the config after that
> point. 
> 
> I agree it's the same bug but it's fairly serious.

OK thanks for the explanation, got it.

I have just checked it. All the user beancounter limits are applied, the only problem is what 'free' (and /proc/meminfo) shows. This is so-called '/proc/meminfo virtualization' feature, search vzctl(8) man page for --meminfo description.

So, yes, I confirm in vzctl-3.0.25 meminfo virtualization is not applied to a container which config file contains "DEVICES=" string because of the bug leading to segfault. But this has nothing to do with actually giving a container all the ram and swap available on the host system -- it is still limited by beancounters.

So, it's a bug, but not a major one.

Here's my test:

# vzctl restart 101
Restarting container
Stopping container ...
Container was stopped
Container is unmounted
Starting container ...
Container is mounted
Setting CPU units: 1000
Setting devices
Segmentation fault
[root@dhcp-10-30-18-157 ~]# vzctl enter 101
entered into CT 101
ca[root@dhcp-10-30-18-157 /]# cat /proc/user_beancounters 
Version: 2.5
       uid  resource           held    maxheld    barrier      limit    failcnt
      101:  kmemsize         803584    1165697   11055923   11377049          0
            lockedpages           0          0        256        256          0
            privvmpages        1548       7914      65536      69632          0
            shmpages              4          4      21504      21504          0
            dummy                 0          0          0          0          0
            numproc              11         19        240        240          0
            physpages           991       1174          0 2147483647          0
            vmguarpages           0          0      33792 2147483647          0
            oomguarpages        991       1174      26112 2147483647          0
            numtcpsock            1          2        360        360          0
            numflock              0          7        188        206          0
            numpty                1          1         16         16          0
            numsiginfo            0          2        256        256          0
            tcpsndbuf          8960          0    1720320    2703360          0
            tcprcvbuf         16384          0    1720320    2703360          0
            othersockbuf      13632      17280    1126080    2097152          0
            dgramrcvbuf           0       8384     262144     262144          0
            numothersock         12         14        360        360          0
            dcachesize            0          0    3409920    3624960          0
            numfile             228        312       9312       9312          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            numiptent            10         10        128        128          0


# free
             total       used       free     shared    buffers     cached
Mem:       1033876     822184     211692          0     295564     383156
-/+ buffers/cache:     143464     890412
Swap:      2096472          0    2096472


# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/simfs            1.1G  451M  574M  45% /
none                  505M   16K  505M   1% /dev
Comment 8 Kir Kolyshkin 2010-12-24 11:41:13 EST
The relevant code portion is vps_setup_res() function in src/lib/res.c, here's a snippet:

        if ((ret = vps_set_cpu(h, veid, &res->cpu)))
                return ret;
        if ((ret = vps_set_devperm(h, veid, fs->root, &res->dev)))
                return ret;
        if ((ret = vps_set_pci(h, veid, ADD, fs->root, &res->pci)))
                return ret;
        if ((ret = vps_set_pci(h, veid, DEL, fs->root, &param->del_res.pci)))
                return ret;
        if ((ret = vps_set_fs(fs, &res->fs)))
                return ret;
        if((ret = vps_meminfo_set(h, veid, &res->meminfo, param, vps_state)))
                return ret;
        if ((ret = ve_ioprio_set(h, veid, &res->io)))
                return ret;

It segfaulted in vps_set_devperm(), therefore vps_meminfo_set() was never called.
Comment 9 Kir Kolyshkin 2010-12-24 11:44:06 EST
vzctl forks a child who waits for the parent to do all the configuration and signal that the environment (ie CT) is fully configured. Then the child execs init.

The problem is that signal is closing a specific file descriptor. If parent segfaults, kernel closes this fd and so child thinks that everything is fine and runs init.

We need to rework that scheme to send something in the "all right" case, too.
Comment 10 Kir Kolyshkin 2010-12-24 12:26:19 EST
OK the nature of the bug (bad logic of determining OK/fail) is in comment #9. Bug #1729 just helped to discover this one, so removing dependency.
Comment 11 Kir Kolyshkin 2011-01-12 08:31:22 EST
Fixing the subject according to latest findings.
Comment 12 Kir Kolyshkin 2011-01-12 09:48:37 EST
Should be fixed by this commit:

http://git.openvz.org/?p=vzctl;a=commit;h=fd657abf122caee09189b115f116c9d089695991

Will appear in vzctl >= 3.0.26
Comment 13 Kir Kolyshkin 2011-01-12 09:48:58 EST
fixed in git
Comment 14 Kir Kolyshkin 2011-02-07 16:18:08 EST
Unfortunately fixing this bug lead to another bug:

dionysos ~ # vzctl restore 401
Restoring container ...
Starting container ...
Container is mounted
	undump...
Setting CPU units: 1000
	resume...
Error: resume failed: No such file or directory
Restoring failed:
Error: CPT: lock fd is closed incorrectly: 1
Container start failed
Stopping container ...
Container was stopped
Container is unmounted

When restoring, vzctl passes wait_p to the kernel (as LOCKFD) and then kernel expects it to close w/o any data coming. Since we changed this scheme kernel now complains.

Need to figure out why this is needed by the kernel and how to fix the bug without breaking compatibility.
Comment 15 Kir Kolyshkin 2011-02-11 05:23:45 EST
Andrey,

Yours, as discussed with Pavel, it needs some kernel changes
Comment 16 Kir Kolyshkin 2011-02-17 08:23:24 EST
vzctl restore should be fixed by the following GIT commit:
http://git.openvz.org/?p=vzctl;a=commit;h=c4fd52f7301201cb9e439fad0e23b600e058a2ab

Will appear in vzctl >= 3.0.26
Comment 17 Sergey Bronnikov 2015-04-01 14:44:58 EDT
Bug was fixed more than one year ago and there were no complains from reporter after fix. We believe bug fix helped and mark bug as closed.