Modify

Ticket #1140 (closed defect: fixed)

Opened 9 years ago

Last modified 7 years ago

UnicodeDecodeError in Logger.py

Reported by: https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ Owned by: solj
Priority: major Milestone: Bcfg2 1.3.5 Release (Bugfix)
Component: bcfg2-client Version: 1.0
Keywords: Cc: [email protected]

Description

Using python 2.6 Logger fails to report errors containing utf-8 encoded characters.

Version affected - git master branch: bd44478c06f65b7cb2a7f87ab8de862033dd80dc

Steps to reproduce:

  1. create a folder Cfg/home/user/Рабочий стол/cyrillicfoldercontent.txt/cyrillicfoldercontent.txt
  2. Do not provide info.xml
  3. Run bcfg2-lint

Do note that error reported is UnicodeDecodeError? (not Encode one) The code failing should work with Python3 but fails with earlier versions due to the fact that internal strings are already encoded.

Fixing patch: diff --git a/src/lib/Bcfg2/Logger.py b/src/lib/Bcfg2/Logger.py index c2eac1e..2c2e148 100644 --- a/src/lib/Bcfg2/Logger.py +++ b/src/lib/Bcfg2/Logger.py @@ -105,7 +105,11 @@ class FragmentingSysLogHandler?(logging.handlers.SysLogHandler?):

(self.encodePriority(self.facility, newrec.levelname.lower()),

self.format(newrec))

try:

  • self.socket.send(msg.encode('ascii'))

+ try: + encoded = msg.encode('utf_8') + except UnicodeDecodeError?: + encoded = msg + self.socket.send(encoded)

except socket.error:

for i in range(10): # pylint: disable=W0612

try:

$bcfg2-lint Could not process event exists for head; ignoring Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/etc/sudoers.d/bcfg2/:info: Use of info/:info files is deprecated Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/etc/cron.hourly/bcfg2-custom/:info: Use of info/:info files is deprecated Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/usr/local/bin/disableScreenLock.sh/:info: Use of info/:info files is deprecat ed Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/usr/local/bin/firefox.sh/:info: Use of info/:info files is deprecated Cfg: /home/gulevich/admin/bcfg2.d/repo/Cfg/usr/local/sbin/bcfg2/:info: Use of info/:info files is deprecated WARNING: The following files are similar: /etc/shadow/shadow.G50_lucid, /etc/shadow/shadow.G50_quantal. Consider merging

them

into a single Genshi template.

WARNING: No info.xml found for /etc/sudoers.d/bcfg2 WARNING: Deprecated :info file found at /home/gulevich/admin/bcfg2.d/repo/Cfg/etc/sudoers.d/bcfg2/:info WARNING: No info.xml found for /usr/share/applications/firefox-shared.desktop WARNING: No info.xml found for /usr/local/home/guest/Рабочий стол/Робот.desktop Traceback (most recent call last):

File "src/sbin/bcfg2-lint", line 207, in <module>

sys.exit(main())

File "src/sbin/bcfg2-lint", line 195, in main

setup=setup, files=files)

File "src/sbin/bcfg2-lint", line 31, in run_server_plugins

errorhandler=errorhandler, setup=setup, files=files)

File "src/sbin/bcfg2-lint", line 55, in run_plugin

return plugin(*args, dict(files=files, errorhandler=errorhandler)).Run()

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/InfoXML.py", line 27, in Run

"No info.xml found for %s" % filename)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 81, in LintError?

self.errorhandler.dispatch(err, msg)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 140, in dispatch

self.errortypes[err](msg)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 155, in warn

self._log(msg, self.logger.warning, prefix="WARNING: ")

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Server/Lint/init.py", line 177, in _log

logfunc(prefix + line.lstrip())

File "/usr/lib/python2.6/logging/init.py", line 1068, in warning

self._log(WARNING, msg, args, kwargs)

File "/usr/lib/python2.6/logging/init.py", line 1173, in _log

self.handle(record)

File "/usr/lib/python2.6/logging/init.py", line 1183, in handle

self.callHandlers(record)

File "/usr/lib/python2.6/logging/init.py", line 1220, in callHandlers

hdlr.handle(record)

File "/usr/lib/python2.6/logging/init.py", line 679, in handle

self.emit(record)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Logger.py", line 108, in emit

self.socket.send(msg.encode('ascii'))

UnicodeDecodeError?: 'ascii' codec can't decode byte 0xd0 in position 76: ordinal not in range(128)

Attachments

patch.diff (747 bytes) - added by https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ 9 years ago.
Patch that fixes problem symptoms
backtrace.txt (2.9 KB) - added by https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ 9 years ago.
Stack trace illustrating the problem

Change History

Changed 9 years ago by https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ

Patch that fixes problem symptoms

Changed 9 years ago by https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ

Stack trace illustrating the problem

comment:1 follow-up: ↓ 3 Changed 9 years ago by solj

  • Owner changed from desai to solj
  • Status changed from new to accepted

Can you try the following patch:

diff --git a/src/lib/Bcfg2/Logger.py b/src/lib/Bcfg2/Logger.py
index c2eac1e..284a40e 100644
--- a/src/lib/Bcfg2/Logger.py
+++ b/src/lib/Bcfg2/Logger.py
@@ -10,6 +10,8 @@ import struct
 import sys
 import termios
 
+import Bcfg2.Compat
+
 logging.raiseExceptions = 0
 
 
@@ -105,7 +107,7 @@ class FragmentingSysLogHandler(logging.handlers.SysLogHandler):
                 (self.encodePriority(self.facility, newrec.levelname.lower()),
                  self.format(newrec))
             try:
-                self.socket.send(msg.encode('ascii'))
+                self.socket.send(Bcfg2.Compat.u_str(msg))
             except socket.error:
                 for i in range(10):  # pylint: disable=W0612
                     try:

comment:2 Changed 9 years ago by solj

comment:3 in reply to: ↑ 1 Changed 9 years ago by https://www.google.com/accounts/o8/id?id=AItOawlPKYcLUHnVh6m82dkD_mNpQLOwSI_RzeQ

Replying to solj:

Can you try the following patch:

Still fails.

File "/usr/lib/python2.6/logging/init.py", line 679, in handle

self.emit(record)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Logger.py", line 109, in emit

self.socket.send(Bcfg2.Compat.u_str(msg))

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Compat.py", line 87, in u_str

return unicode(string)

UnicodeDecodeError?: 'ascii' codec can't decode byte 0xd0 in position 75: ordinal not in range(128)

This is understandable - I'm using UTF-8 encoding.

I've tried Bcfg2.Compat.u_str(msg, "utf8") and failed again with

File "/usr/lib/python2.6/logging/init.py", line 679, in handle

self.emit(record)

File "/home/gulevich/admin/bcfg2/src/Bcfg2/Bcfg2/Logger.py", line 109, in emit

self.socket.send(Bcfg2.Compat.u_str(msg, "utf8"))

UnicodeEncodeError?: 'ascii' codec can't encode characters in position 75-81: ordinal not in range(128)

This can be explained by socket API nature - it expects bytes.

Then I tried: self.socket.send(Bcfg2.Compat.u_str(msg, "utf8").encode("utf8")) and this one worked just fine (no fatal exceptions).

How do we avoid explicit encoding here?

comment:4 Changed 9 years ago by solj

  • Milestone changed from Bcfg2 1.3.0 Release to Bcfg2 1.3.2 Release (Bugfix)

comment:5 Changed 9 years ago by solj

I've gone ahead and applied your patch to the maint branch so that it will be included in 1.3.1. I'm going to keep this ticket open so that we can think about a better way to fix this. Thanks for the patch!

comment:6 Changed 7 years ago by solj

  • Status changed from accepted to closed
  • Resolution set to fixed

WARNING! You need to establish a session before you can create or edit tickets. Otherwise the ticket will get treated as spam.

View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.