Modify

Ticket #598 (closed defect: worksforme)

Opened 12 years ago

Last modified 11 years ago

Failed to upload configuration statistics

Reported by: jstrunk@… Owned by: desai
Priority: major Milestone: Bcfg2 1.0.0 Release
Component: bcfg2-server Version: 0.9.x
Keywords: Cc:

Description

I think this is a scaling problem with the server. It occurs after I have run bcfg2 on a few clients. I am currently running 4 clients at a time. After about 10 clients have run, the clients start showing this message:

Failed to upload configuration statistics

I found this traceback in the server's log at the same time. It is probably truncated:

Aug 27 15:12:01 ibm13 bcfg2-server[27436]: Unexpected handler failure Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/Bcfg2/Component.py", line 220, in _cobalt_marshalled_dispatch response = self._dispatch(method, (address, ) + params) File "/usr/lib/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch return func(*params) File "/usr/sbin/bcfg2-server", line 181, in Bcfg2RecvStats self.Core.stats.StoreStatistics?(meta, sdata) File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 116, in StoreStatistics? self.data.updateStats(xdata, client.hostname) File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 96, in updateStats self.WriteBack?() File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 33, in WriteBack? fout.write(lxml.etree.tostring(self.element, encoding='UTF-8', xml_declaration=True)) File "etree.pyx", line 2012, in etree.tostring File "serializer.pxi", line 41, in etree._

When I was running with 10 clients at a time I noticed that after a while the clients were not getting probes. I am nopt sure if there is a traceback for that issue. All of the tracebacks in the log look like the one above.

The server I am running bcfg2 on has 2 dual core Xeon's and 2 GB of RAM. How many simultaneous clients should I be able to run?

Attachments

Change History

comment:1 Changed 12 years ago by solj

You can use wiki formatting in tickets to produce better output on these tracebacks with the 3 opening and closing braces {}. It'll make them easier to read.

comment:2 Changed 12 years ago by desai

  • Milestone set to Bcfg2 0.9.6 Release

We handle 200-300 clients with a server less beefy than that.

This traceback still looks truncated. Can you try running the bcfg2-server process interactively to get a full traceback? I will figure out how these are getting truncated in the logs.

I'm not able to replicate the problem here. Are you using the reporting system? As a general rule, it should scale better than this mechanism does, in terms of client scalability.

comment:3 Changed 12 years ago by jstrunk@…

I am running the server overnight with "-v -d" and capturing the output. My 180 clients will connect at various times tonight and will probably trigger this problem.

comment:4 Changed 12 years ago by jstrunk@…

It looks like none of my clients were able to send back statistics.

Here is one of the tracebacks from the server:

Unexpected handler failure
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/Bcfg2/Component.py", line 220, in _cobalt_marshalled_dispatch
    response = self._dispatch(method, (address, ) + params)
  File "/usr/lib/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch
    return func(*params)
  File "/usr/sbin/bcfg2-server", line 181, in Bcfg2RecvStats
    self.Core.stats.StoreStatistics(meta, sdata)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 116, in StoreStatistics
    self.data.updateStats(xdata, client.hostname)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 96, in updateStats
    self.WriteBack()
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Statistics.py", line 33, in WriteBack
    fout.write(lxml.etree.tostring(self.element, encoding='UTF-8', xml_declaration=True))
  File "etree.pyx", line 2012, in etree.tostring
  File "serializer.pxi", line 41, in etree._tostring
MemoryError

comment:5 Changed 12 years ago by jstrunk@…

As an update, I haven't seen this problem since I started using mysql reporting.

comment:6 Changed 12 years ago by solj

  • Milestone changed from Bcfg2 0.9.6 Release to Bcfg2 1.0 Release

comment:7 Changed 11 years ago by desai

  • Status changed from new to closed
  • Resolution set to worksforme

This looks like it was a problem with statistics that are too large. I'm not sure there is much we can do about it.

WARNING! You need to establish a session before you can create or edit tickets. Otherwise the ticket will get treated as spam.

View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.