Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!newsreader4.netcologne.de!news.netcologne.de!feed.xsnews.nl!border-3.ams.xsnews.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: chris@rebertia.com
In-Reply-To: <310c83c4-cfa2-4425-b291-d1a3604b3e29@googlegroups.com>
References: <b1b99ce0-6088-4c80-9e43-ac12e7799b92@googlegroups.com> <mailman.1527.1357065016.29569.python-list@python.org> <90663ba3-2307-45ed-a1b7-c3dbe5130ebd@googlegroups.com> <mailman.1552.1357108357.29569.python-list@python.org> <310c83c4-cfa2-4425-b291-d1a3604b3e29@googlegroups.com>
Date: Wed, 2 Jan 2013 22:00:16 -0800
Subject: Re: Handling Special characters in python
From: Chris Rebert <clp2@rebertia.com>
To: anilkumar.dannina@gmail.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.20.1357192819.2939.python-list@python.org>
Lines: 53
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:36037

On Wed, Jan 2, 2013 at 5:39 AM,  <anilkumar.dannina@gmail.com> wrote:
> On Wednesday, January 2, 2013 12:02:34 PM UTC+5:30, Chris Rebert wrote:
>> On Jan 1, 2013 8:48 PM, <anilkuma...@gmail.com> wrote:
>> > On Wednesday, January 2, 2013 12:00:06 AM UTC+5:30, Chris Rebert wrote=
:
>> > > On Jan 1, 2013 3:41 AM, <anilkuma...@gmail.com> wrote:
>> > > > I am facing one issue in my module. I am gathering data from sql s=
erver database. In the data that I got from db contains special characters =
like "endash". Python was taking it as "\x96". I require the same character=
(endash). How can I perform that. Can you please help me in resolving this =
issue.
>>
>> > > 1. What library are you using to access the database?
>> > > 2. To confirm, it's a Microsoft SQL Server database?
>> > > 3. What OS are you on?
>>
>> > 1. I am using "pymssql" module to access the database.
>> > 2. Yes, It is a SQL server database.
>> > 3. I am on Ubuntu 11.10
>>
>> Did you set "client charset" (to "UTF-8", unless you have good reason to=
 choose otherwise) in freetds.conf? That should at least ensure that the dr=
iver itself is exchanging bytestrings via a well-defined encoding.
>> If you want to work in Unicode natively (Recommended), you'll probably n=
eed to ensure that the columns are of type NVARCHAR as opposed to VARCHAR. =
Unless you're using SQLAlchemy or similar (which I personally would recomme=
nd using), you may need to do the .encode() and .decode()-ing manually, usi=
ng the charset you specified in freetds.conf.
>>
>> Sorry my advice is a tad general. I went the alternative route of SQLAlc=
hemy + PyODBC + Microsoft's SQL Server ODBC driver for Linux (http://www.mi=
crosoft.com/en-us/download/details.aspx?id=3D28160 ) for my current project=
, which likewise needs to fetch data from MS SQL to an Ubuntu box. The driv=
er is intended for Red Hat and isn't packaged nicely (it installs via a she=
ll script), but after that was dealt with, things have gone smoothly. Unico=
de, in particular, seems to work properly.
>
> Thanks Chris Rebert for your suggestion, I tried with PyODBC module, But =
at the place of "en dash(-)", I am getting '?' symbol. How can I overcome t=
his.

I would recommend first trying the advice in the initial part of my
response rather than the latter part. The latter part was more for
completeness and for the sake of the archives, although I can give
more details on its approach if you insist.

Additionally, giving more information as to what exactly you tried
would be helpful. What config / connection settings did you use? Of
what datatype is the relevant  column of the table? What's your code
snippet look like? Etc..

Regards,
Chris