UnicodeDecodeError in Logger.py

[https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ]

Using python 2.6 Logger fails to report errors containing utf-8 encoded characters.

Version affected - git master branch: bd44478c06f65b7cb2a7f87ab8de862033dd80dc

Steps to reproduce:

1. create a folder Cfg/home/user/Рабочий стол/cyrillicfoldercontent.txt/cyrillicfoldercontent.txt
2. Do not provide info.xml
3. Run bcfg2-lint

Do note that error reported is UnicodeDecodeError? (not Encode one) The code failing should work with Python3 but fails with earlier versions due to the fact that internal strings are already encoded.

Fixing patch: diff --git a/src/lib/Bcfg2/Logger.py b/src/lib/Bcfg2/Logger.py index c2eac1e..2c2e148 100644 --- a/src/lib/Bcfg2/Logger.py +++ b/src/lib/Bcfg2/Logger.py @@ -105,7 +105,11 @@ class FragmentingSysLogHandler?(logging.handlers.SysLogHandler?):

(self.encodePriority(self.facility, newrec.levelname.lower()),

self.format(newrec))

try:

• self.socket.send(msg.encode('ascii'))

+ try: + encoded = msg.encode('utf_8') + except UnicodeDecodeError?: + encoded = msg + self.socket.send(encoded)

except socket.error:

for i in range(10): # pylint: disable=W0612

try:

$bcfg2-lint Could not process event exists for head; ignoring Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/etc/sudoers.d/bcfg2/:info: Use of info/:info files is deprecated Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/etc/cron.hourly/bcfg2-custom/:info: Use of info/:info files is deprecated Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/usr/local/bin/disableScreenLock.sh/:info: Use of info/:info files is deprecat ed Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/usr/local/bin/firefox.sh/:info: Use of info/:info files is deprecated Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/usr/local/sbin/bcfg2/:info: Use of info/:info files is deprecated WARNING: The following files are similar: /etc/shadow/shadow.G50_lucid, /etc/shadow/shadow.G50_quantal. Consider merging

them

into a single Genshi template.

WARNING: No info.xml found for /etc/sudoers.d/bcfg2 WARNING: Deprecated :info file found at /home/gulevich/admin/bcfg2.d/repo/Cfg/etc/sudoers.d/bcfg2/:info WARNING: No info.xml found for /usr/share/applications/firefox-shared.desktop WARNING: No info.xml found for /usr/local/home/guest/Рабочий стол/Робот.desktop Traceback (most recent call last):

File "src/sbin/bcfg2-lint", line 207, in <module>

sys.exit(main())

File "src/sbin/bcfg2-lint", line 195, in main

setup=setup, files=files)

File "src/sbin/bcfg2-lint", line 31, in run_server_plugins

errorhandler=errorhandler, setup=setup, files=files)

File "src/sbin/bcfg2-lint", line 55, in run_plugin

return plugin(*args, dict(files=files, errorhandler=errorhandler)).Run()

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/InfoXML.py", line 27, in Run

"No info.xml found for %s" % filename)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 81, in LintError?

self.errorhandler.dispatch(err, msg)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 140, in dispatch

self.errortypes[err](msg)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 155, in warn

self._log(msg, self.logger.warning, prefix="WARNING: ")

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 177, in _log

logfunc(prefix + line.lstrip())

File "/usr/lib/python2.6/logging/init.py", line 1068, in warning

self._log(WARNING, msg, args, kwargs)

File "/usr/lib/python2.6/logging/init.py", line 1173, in _log

self.handle(record)

File "/usr/lib/python2.6/logging/init.py", line 1183, in handle

self.callHandlers(record)

File "/usr/lib/python2.6/logging/init.py", line 1220, in callHandlers

hdlr.handle(record)

File "/usr/lib/python2.6/logging/init.py", line 679, in handle

self.emit(record)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Logger.py", line 108, in emit

self.socket.send(msg.encode('ascii'))

UnicodeDecodeError?: 'ascii' codec can't decode byte 0xd0 in position 76: ordinal not in range(128)

[https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ]

Patch that fixes problem symptoms

[https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ]

Stack trace illustrating the problem

[solj]

Can you try the following patch:

diff --git a/src/lib/Bcfg2/Logger.py b/src/lib/Bcfg2/Logger.py
index c2eac1e..284a40e 100644
--- a/src/lib/Bcfg2/Logger.py
+++ b/src/lib/Bcfg2/Logger.py
@@ -10,6 +10,8 @@ import struct
 import sys
 import termios
 
+import Bcfg2.Compat
+
 logging.raiseExceptions = 0
 
 
@@ -105,7 +107,7 @@ class FragmentingSysLogHandler(logging.handlers.SysLogHandler):
                 (self.encodePriority(self.facility, newrec.levelname.lower()),
                  self.format(newrec))
             try:
-                self.socket.send(msg.encode('ascii'))
+                self.socket.send(Bcfg2.Compat.u_str(msg))
             except socket.error:
                 for i in range(10):  # pylint: disable=W0612
                     try:

[https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ]

Replying to solj:

Can you try the following patch:

Still fails.

File "/usr/lib/python2.6/logging/init.py", line 679, in handle

self.emit(record)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Logger.py", line 109, in emit

self.socket.send(Bcfg2.Compat.u_str(msg))

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Compat.py", line 87, in u_str

return unicode(string)

UnicodeDecodeError?: 'ascii' codec can't decode byte 0xd0 in position 75: ordinal not in range(128)

This is understandable - I'm using UTF-8 encoding.

I've tried Bcfg2.Compat.u_str(msg, "utf8") and failed again with

File "/usr/lib/python2.6/logging/init.py", line 679, in handle

self.emit(record)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Logger.py", line 109, in emit

self.socket.send(Bcfg2.Compat.u_str(msg, "utf8"))

UnicodeEncodeError?: 'ascii' codec can't encode characters in position 75-81: ordinal not in range(128)

This can be explained by socket API nature - it expects bytes.

Then I tried: self.socket.send(Bcfg2.Compat.u_str(msg, "utf8").encode("utf8")) and this one worked just fine (no fatal exceptions).

How do we avoid explicit encoding here?

[solj]

I've gone ahead and applied your patch to the maint branch so that it will be included in 1.3.1. I'm going to keep this ticket open so that we can think about a better way to fix this. Thanks for the patch!