Running bcfg2s-server for a few days results in 'Too many open files' crash

[[email protected]…]

After upgrading from v0.9.6 to v1.0pre2, bcfg2-server crashes every few days with 'too many open files'.

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 8192
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8192
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

May 18 18:07:04 tauon bcfg2-server[7059]: Processed 1 gamin events in 0.000 seconds. 0 collapsed
May 18 18:07:04 tauon bcfg2-server[7059]: error in service loop Traceback (most recent call last):   File "/usr/sbin/bcfg2-server", line 243, in ?     BSERV.serve_forever()   File "/usr/lib/python2.4/site-packages/Bcfg2/Component.py", line 317, in serve_forever     self.handle_request()   File "/usr/lib64/python2.4/SocketServer.py", line 217, in handle_request     request, client_address = self.get_request()   File "/usr/sbin/bcfg2-server", line 102, in get_request     self.Core.Service()   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Core.py", line 222, in Service     self.revision = plugin.get_revision()   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Plugins/Svn.py", line 37, in get_revision     stdout=PIPE).stdout.readlines()   File "/usr/lib64/python2.4/subprocess.py", line 542, in __init__     errread, errwrite)   File "/usr/lib64/python2.4/subprocess.py", line 899, in _execute_child     errpipe_read, errpipe_write = os.pipe() OSError: [Errno 24] Too many open files

May 11 07:40:59 tauon bcfg2-server[0f07fb09115d71990bcf3844782ce4174f2158ff] (SVN r5584): error in service loop Traceback (most recent call last):   File "/usr/sbin/bcfg2-server", line 243, in ?     BSERV.serve_forever()   File "/usr/lib/python2.4/site-packages/Bcfg2/Component.py", line 317, in serve_forever     self.handle_request()   File "/usr/lib64/python2.4/SocketServer.py", line 217, in handle_request     request, client_address = self.get_request()   File "/usr/sbin/bcfg2-server", line 102, in get_request     self.Core.Service()   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Core.py", line 222, in Service     self.revision = plugin.get_revision()   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Plugins/Svn.py", line 37, in get_revision     stdout=PIPE).stdout.readlines()   File "/usr/lib64/python2.4/subprocess.py", line 542, in __init__     errread, errwrite)   File "/usr/lib64/python2.4/subprocess.py", line 899, in _execute_child     errpipe_read, errpipe_write = os.pipe() OSError: [Errno 24] Too many open files

[…]

lsof output of a freshly started instance.

[anonymous]

Same instance, 24 hours later: $ sudo /usr/sbin/lsof -n -p $(pgrep bcfg2-server) | wc -l 873 $ sudo /usr/sbin/lsof -n -p $(pgrep bcfg2-server) | grep pipe | wc -l 798

[desai]

Fixed in [ec937861bea0bba0a82073c9de752fd0fdd862f9] (SVN r5251)