Failed to upload configuration statistics

[[email protected]…]

I think this is a scaling problem with the server. It occurs after I have run bcfg2 on a few clients. I am currently running 4 clients at a time. After about 10 clients have run, the clients start showing this message:

Failed to upload configuration statistics

I found this traceback in the server's log at the same time. It is probably truncated:

Aug 27 15:12:01 ibm13 bcfg2-server[27436]: Unexpected handler failure Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/Bcfg2/Component.py", line 220, in _cobalt_marshalled_dispatch response = self._dispatch(method, (address, ) + params) File "/usr/lib/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch return func(*params) File "/usr/sbin/bcfg2-server", line 181, in Bcfg2RecvStats self.Core.stats.StoreStatistics?(meta, sdata) File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 116, in StoreStatistics? self.data.updateStats(xdata, client.hostname) File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 96, in updateStats self.WriteBack?() File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 33, in WriteBack? fout.write(lxml.etree.tostring(self.element, encoding='UTF-8', xml_declaration=True)) File "etree.pyx", line 2012, in etree.tostring File "serializer.pxi", line 41, in etree._

When I was running with 10 clients at a time I noticed that after a while the clients were not getting probes. I am nopt sure if there is a traceback for that issue. All of the tracebacks in the log look like the one above.

The server I am running bcfg2 on has 2 dual core Xeon's and 2 GB of RAM. How many simultaneous clients should I be able to run?

[solj]

You can use wiki formatting in tickets to produce better output on these tracebacks with the 3 opening and closing braces {}. It'll make them easier to read.

[desai]

We handle 200-300 clients with a server less beefy than that.

This traceback still looks truncated. Can you try running the bcfg2-server process interactively to get a full traceback? I will figure out how these are getting truncated in the logs.

I'm not able to replicate the problem here. Are you using the reporting system? As a general rule, it should scale better than this mechanism does, in terms of client scalability.

[…]

I am running the server overnight with "-v -d" and capturing the output. My 180 clients will connect at various times tonight and will probably trigger this problem.

[…]

It looks like none of my clients were able to send back statistics.

Here is one of the tracebacks from the server:

Unexpected handler failure
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/Bcfg2/Component.py", line 220, in _cobalt_marshalled_dispatch
    response = self._dispatch(method, (address, ) + params)
  File "/usr/lib/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch
    return func(*params)
  File "/usr/sbin/bcfg2-server", line 181, in Bcfg2RecvStats
    self.Core.stats.StoreStatistics(meta, sdata)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 116, in StoreStatistics
    self.data.updateStats(xdata, client.hostname)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 96, in updateStats
    self.WriteBack()
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 33, in WriteBack
    fout.write(lxml.etree.tostring(self.element, encoding='UTF-8', xml_declaration=True))
  File "etree.pyx", line 2012, in etree.tostring
  File "serializer.pxi", line 41, in etree._tostring
MemoryError

[…]

As an update, I haven't seen this problem since I started using mysql reporting.

[desai]

This looks like it was a problem with statistics that are too large. I'm not sure there is much we can do about it.