Modify

Ticket #954 (closed defect: fixed)

Opened 12 years ago

Last modified 10 years ago

web_debug = true in bcfg2.conf causes memory over usage

Reported by: https://www.google.com/accounts/o8/id?id=AItOawkfar6gGD3PuUixQdEZQyRr2mpowLW1A70 Owned by: https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc
Priority: major Milestone: Bcfg2 1.2.1 Release (Bugfix)
Component: bcfg2-server Version: 1.0
Keywords: memory dbstats Cc:

Description

When running tests using guppy. Adding this into bcfg2 code somewhere:

from guppy.heapy import Remote Remote.on()

Then from another window ran:

$ python -c "from guppy import hpy;hpy().monitor()" <Monitor> * Connection 1 opened * <Monitor> lc CID PID ARGV

1 12075 ['/usr/sbin/bcfg2-server', '-d']

<Monitor> sc 1 Remote connection 1. To return to Monitor, type <Ctrl-C> or .<RETURN> <Annex> int Remote interactive console. To return to Annex, type '-'.

hp.setref()

At this point the memory counters are reset and I ran a bcfg2 client update on another server. Next taking a tip from a blog post about analysing django memory leaks[f5e4281313ff9004be488e420f90d5fe413bb5c1] (SVN r2) I run a method to extract information about the references hierarchy. This is as the post states magic!

hp.heap().get_rp(40)

Reference Pattern by <[dict of] class>.

0: _ --- [-] 1134 (Bcfg2.Server.Plugins.Metadata.ClientMetadata? |

Bcfg2.Serv...

1: a [-] 351 dict (no owner): 0xf083c64*2, 0xfcfc46c*2, 0xfe7dacc*2... 2: aa ---- [-] 2 list: 0x10066aac*4914, 0x1043a82c*8 3: a3 [-] 1 dict of

django.db.backends.mysql.base.DatabaseWrapper?: 0x8...

4: a4 ------ [-] 1 dict (no owner): 0x104992d4*3 5: a5 [R] 1 guppy.heapy.heapyc.RootStateType?: 0xf58f38acL 6: a3b ----- [-] 1 django.db.backends.mysql.base.DatabaseWrapper?: 0x8625194 7: a3ba [S] 12 dict of module: ..db..., ..importscript, ..models 8: a3bb ----- [-] 2 dict of

django.db.backends.mysql.creation.DatabaseCreati?...

9: a3bba [-] 2

django.db.backends.mysql.creation.DatabaseCreation?: 0x8... <Type e.g. '_.more' for more.>

"Django keeps track of all queries for debugging purposes (connection.queries). This list is reset at the end of HTTP request. But in standalone mode, there are no requests. So you need to manually reset to queries list after each working cycle."

Django is only caching these queries if it's in debug mode. I have web_debug = true in my /etc/bcfg2.conf. I've set this to False and have reran some of the tests and the memory usage is much different.

Memory increase between client runs with web_debug:

False: 118856 bytes True: 1303532 bytes

So that's 1.2Mb increase per client update.

Attachments

Change History

comment:1 Changed 12 years ago by laszlo

  • Owner changed from desai to laszlo
  • Status changed from new to assigned

comment:2 Changed 11 years ago by solj

  • Milestone changed from Bcfg2 1.2.0 Release to Bcfg2 1.2.1 Release (Bugfix)

comment:3 Changed 10 years ago by https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc

  • Owner changed from laszlo to https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc
  • Status changed from assigned to accepted

comment:4 Changed 10 years ago by https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc

  • Status changed from accepted to closed
  • Resolution set to fixed

As I understand it, the memory leak caused by web_debug is pretty unavoidable; lots of Django things warn about memory leaks if DEBUG is true. Consequently, I have:

  • Disabled web_debug in the default bcfg2.conf created by 'bcfg2-admin init';
  • Disabled web_debug in the sample bcfg2.conf files referenced in the setup guides; and
  • Added a warning if web_debug is enabled.

https://github.com/Bcfg2/bcfg2/commit/c31d8664f0c1bf78c085f0e6d83db44c81c81372

WARNING! You need to establish a session before you can create or edit tickets. Otherwise the ticket will get treated as spam.

View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.