bcfg2-server SSHbase fails to generate key pairs

[https://www.google.com/accounts/o8/id?id=AItOawlo86Mt9tqt2k06humTEpjOdekfWJoTv4M]

When a client connects to the bcfg2 server, the server fails to generate new SSL certificate pairs for a client that does not already have valid certs in the SSHbase directory.

logging from /var/log/messages:

Mar 22 11:02:02 rhel bcfg2-server[19580]: Unexpected failure in BindStructure: ConfigFile /etc/ssh/ssh_host_rsa_key Traceback (most recent call last):   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Core.py", line 179, in BindStructure     self.Bind(entry, metadata)   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Core.py", line 212, in Bind     return glist[0].Entries[entry.tag][entry.get('name')](entry, metadata)   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Plugins/SSHbase.py", line 179, in build_hk     self.GenerateHostKeys(client)   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Plugins/SSHbase.py", line 215, in GenerateHostKeys     open(fileloc, 'w').write(open(temploc).read()) IOError: [Errno 2] No such file or directory: '/tmp/tmpdfu5E0/ssh_host_dsa_key.H_rhel'
Mar 22 11:02:02 rhel bcfg2-server[19580]: Unexpected failure in BindStructure: ConfigFile /etc/ssh/ssh_host_rsa_key.pub Traceback (most recent call last):   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Core.py", line 179, in BindStructure     self.Bind(entry, metadata)   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Core.py", line 212, in Bind     return glist[0].Entries[entry.tag][entry.get('name')](entry, metadata)   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Plugins/SSHbase.py", line 179, in build_hk     self.GenerateHostKeys(client)   File "/usr/lib/python2.4/site-packages/Bcfg2/Server/Plugins/SSHbase.py", line 215, in GenerateHostKeys     open(fileloc, 'w').write(open(temploc).read()) IOError: [Errno 2] No such file or directory: '/tmp/tmpz6Mnzl/ssh_host_dsa_key.H_rhel'
Mar 22 11:02:02 rhel bcfg2-server[19580]: Generated config for rhel in 0.520s
Mar 22 11:02:03 rhel bcfg2-server[19580]: Client rhel reported state dirty

From what I can tell, there is an os.system call (line #214 of Bcfg2/Server/Plugins/SSHbase.py) that is supposed to fire off ssh-keygen to generate the keys in a temporary directory. It seems that this call isn't generating the keys correctly. Adding in some debug os.listdir(tempdir) calls before and after the os.system call, shows that there are no files before and after the os.system call. Running the command manually generates valid key pairs.

Has anyone else seen this problem?

[[email protected] SSHbase]# uname -a
Linux rhel 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[[email protected] SSHbase]# rpm -q bcfg2-server
bcfg2-server-1.0.1-1
[[email protected] SSHbase]# rpm -q bcfg2
bcfg2-1.0.1-1
[[email protected] SSHbase]# pwd
/var/lib/bcfg2/SSHbase
[[email protected] SSHbase]# ls -l
total 4
-rw-r----- 1 root root 0 Mar 22 11:02 ssh_host_dsa_key.H_rhel
[[email protected] SSHbase]# which ssh-keygen
/usr/bin/ssh-keygen
directory)
[[email protected] SSHbase]# file /usr/bin/ssh-keygen
/usr/bin/ssh-keygen: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, stripped
[[email protected] SSHbase]# ls -l /tmp/tmpz6Mnzl
/tmp/tmpz6Mnzl:
total 0

[[email protected] SSHbase]# ssh-keygen -q -f /tmp/tmpz6Mnzl/ssh_host_rsa_key.H_rhel -N
"" -t rsa -C [email protected] < /dev/null
[[email protected] SSHbase]# ls -l /tmp/tmpz6Mnzl/
/tmp/tmpz6Mnzl:
total 16
-rw------- 1 root root 1675 Mar 22 11:13 ssh_host_rsa_key.H_rhel
-rw------- 1 root root  391 Mar 22 11:13 ssh_host_rsa_key.H_rhel.pub

[https://www.google.com/accounts/o8/id?id=AItOawlo86Mt9tqt2k06humTEpjOdekfWJoTv4M]

So there seems to be some problem with the os.system call to ssh-keygen. (I'm still working on figuring that out.) This results in the temporary key pair files not getting created. Then on line #215 and #216 the open.read is failing ungracefully causing the open.write to create an empty file in the /var/lib/bcfg2/SSHbase/ directory.

It seems that either bcfg2 should use something like shutil.copy or something like the following patch to SSHbase.py

215,216c215,230
<                 open(fileloc, 'w').write(open(temploc).read())
<                 open(publoc, 'w').write(open("%s.pub" % temploc).read())
---
>                 # separate the opens and the reads so that we don't
>                 # create an empty key file in /var/lib/bcfg2/SSHbase/
>                 # if the read fails
>                 try:
>                     temp_key = open(temploc)
>                     open(fileloc, 'w').write(temp_key.read())
>                 except IOError, ioe:
>                     self.logger.error('Error reading %s (%s)' % (
>                         ioe.filename, ioe.strerror))
> 
>                 try:
>                     temp_key = open("%s.pub" % temploc)
>                     open(publoc, 'w').write(temp_key.read())
>                 except IOError, ioe:
>                     self.logger.error('Error reading %s (%s)' % (
>                         ioe.filename, ioe.strerror))

This way, when/if the read fails, the write doesn't get called and there isn't an empty host/rsa/dsa key files created for a host.

[solj]

Changed in [8999c249ccc3d93f08060fbfb0b7598e7e15cfda] (SVN r5794) to us shutil as per your suggestion. I will leave this ticket open so that we can continue to track the os.system call.

[solj]

There are some SSHbase changes that were made upstream which might resolve this issue. The code was updated to use the normal FileMonitor? events instead of having SSHbase explicitly call AddEntry?.

[solj]

Closing for now as we have been unable to reproduce and a lot of the SSHbase code has been rewritten. Please reopen if you see this again.