Ticket #954 (closed defect: fixed)
web_debug = true in bcfg2.conf causes memory over usage
| Reported by: | https://www.google.com/accounts/o8/id?id=AItOawkfar6gGD3PuUixQdEZQyRr2mpowLW1A70 | Owned by: | https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc |
|---|---|---|---|
| Priority: | major | Milestone: | Bcfg2 1.2.1 Release (Bugfix) |
| Component: | bcfg2-server | Version: | 1.0 |
| Keywords: | memory dbstats | Cc: |
Description
When running tests using guppy. Adding this into bcfg2 code somewhere:
from guppy.heapy import Remote Remote.on()
Then from another window ran:
$ python -c "from guppy import hpy;hpy().monitor()" <Monitor> * Connection 1 opened * <Monitor> lc CID PID ARGV
1 12075 ['/usr/sbin/bcfg2-server', '-d']
<Monitor> sc 1 Remote connection 1. To return to Monitor, type <Ctrl-C> or .<RETURN> <Annex> int Remote interactive console. To return to Annex, type '-'.
hp.setref()
At this point the memory counters are reset and I ran a bcfg2 client update on another server. Next taking a tip from a blog post about analysing django memory leaks[f5e4281313ff9004be488e420f90d5fe413bb5c1] (SVN r2) I run a method to extract information about the references hierarchy. This is as the post states magic!
hp.heap().get_rp(40)
Reference Pattern by <[dict of] class>.
0: _ --- [-] 1134 (Bcfg2.Server.Plugins.Metadata.ClientMetadata? |
Bcfg2.Serv...
1: a [-] 351 dict (no owner): 0xf083c64*2, 0xfcfc46c*2, 0xfe7dacc*2... 2: aa ---- [-] 2 list: 0x10066aac*4914, 0x1043a82c*8 3: a3 [-] 1 dict of
django.db.backends.mysql.base.DatabaseWrapper?: 0x8...
4: a4 ------ [-] 1 dict (no owner): 0x104992d4*3 5: a5 [R] 1 guppy.heapy.heapyc.RootStateType?: 0xf58f38acL 6: a3b ----- [-] 1 django.db.backends.mysql.base.DatabaseWrapper?: 0x8625194 7: a3ba [S] 12 dict of module: ..db..., ..importscript, ..models 8: a3bb ----- [-] 2 dict of
django.db.backends.mysql.creation.DatabaseCreati?...
9: a3bba [-] 2
django.db.backends.mysql.creation.DatabaseCreation?: 0x8... <Type e.g. '_.more' for more.>
"Django keeps track of all queries for debugging purposes (connection.queries). This list is reset at the end of HTTP request. But in standalone mode, there are no requests. So you need to manually reset to queries list after each working cycle."
Django is only caching these queries if it's in debug mode. I have web_debug = true in my /etc/bcfg2.conf. I've set this to False and have reran some of the tests and the memory usage is much different.
Memory increase between client runs with web_debug:
False: 118856 bytes True: 1303532 bytes
So that's 1.2Mb increase per client update.
Attachments
Change History
comment:1 Changed 12 years ago by laszlo
- Owner changed from desai to laszlo
- Status changed from new to assigned
comment:2 Changed 12 years ago by solj
- Milestone changed from Bcfg2 1.2.0 Release to Bcfg2 1.2.1 Release (Bugfix)
comment:3 Changed 11 years ago by https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc
- Owner changed from laszlo to https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc
- Status changed from assigned to accepted
comment:4 Changed 11 years ago by https://www.google.com/accounts/o8/id?id=AItOawnSjgovXZr-_V3vGkvMSR0pc5LDykRc1Nc
- Status changed from accepted to closed
- Resolution set to fixed
As I understand it, the memory leak caused by web_debug is pretty unavoidable; lots of Django things warn about memory leaks if DEBUG is true. Consequently, I have:
- Disabled web_debug in the default bcfg2.conf created by 'bcfg2-admin init';
- Disabled web_debug in the sample bcfg2.conf files referenced in the setup guides; and
- Added a warning if web_debug is enabled.
https://github.com/Bcfg2/bcfg2/commit/c31d8664f0c1bf78c085f0e6d83db44c81c81372
