Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #29077 > unrolled thread

Least-lossy string.encode to us-ascii?

Back to article view | Back to comp.lang.python

Contents

  Least-lossy string.encode to us-ascii? Tim Chase <python.list@tim.thechases.com> - 2012-09-13 16:26 -0500
    Re: Least-lossy string.encode to us-ascii? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-09-14 03:49 +0000
    Re: Least-lossy string.encode to us-ascii? wxjmfauth@gmail.com - 2012-09-14 09:15 -0700
      Re: Least-lossy string.encode to us-ascii? Terry Reedy <tjreedy@udel.edu> - 2012-09-14 16:43 -0400
        Re: Least-lossy string.encode to us-ascii? wxjmfauth@gmail.com - 2012-09-15 01:58 -0700
        Re: Least-lossy string.encode to us-ascii? wxjmfauth@gmail.com - 2012-09-15 01:58 -0700
    Re: Least-lossy string.encode to us-ascii? wxjmfauth@gmail.com - 2012-09-14 09:15 -0700

#29077 — Least-lossy string.encode to us-ascii?

I've got a bunch of text in Portuguese and to transmit them, need to
have them in us-ascii (7-bit).  I'd like to keep as much information
as possible, just stripping accents, cedillas, tildes, etc.  So
"serviço móvil" becomes "servico movil".  Is there anything stock
that I've missed?  I can do mystring.encode('us-ascii', 'replace')
but that doesn't keep as much information as I'd hope.

-tkc

[toc] | [next] | [standalone]

#29109

On Thu, 13 Sep 2012 16:26:07 -0500, Tim Chase wrote:

> I've got a bunch of text in Portuguese and to transmit them, need to
> have them in us-ascii (7-bit).

That could mean two things:

1) "The receiver is incapable of dealing with Unicode in 2012, which is 
frankly appalling, but what can I do about it?"

2) "The transport mechanism I use to transmit the data is only capable of 
dealing with 7-bit ASCII strings, which is sad but pretty much standard."

In the case of 1), I suggest you look at the Unicode Hammer, a.k.a. "The 
Stupid American":

http://code.activestate.com/recipes/251871

and especially the very many useful comments.


In the case of 2), just binhex or uuencode your data for transport.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#29176

Le jeudi 13 septembre 2012 23:25:27 UTC+2, Tim Chase a écrit :
> I've got a bunch of text in Portuguese and to transmit them, need to
> 
> have them in us-ascii (7-bit).  I'd like to keep as much information
> 
> as possible, just stripping accents, cedillas, tildes, etc.  So
> 
> "serviço móvil" becomes "servico movil".  Is there anything stock
> 
> that I've missed?  I can do mystring.encode('us-ascii', 'replace')
> 
> but that doesn't keep as much information as I'd hope.
> 

Interesting case. It's where the coding of characters
meets characters usage, scripts, typography, linguistic
features.

I cann't discuss the Portugese case, but in French
and in German one way to achieve the task is to
convert the text in uppercases. It preserves a correct
text.

>>> s = 'Lætitia  cœur  éléphant  français  LUŸ  Stoß  Erklärung  stören'
>>> libfrancais.SpecMajuscules(s)
'LAETITIA  COEUR  ELEPHANT  FRANCAIS  LUY  STOSS  ERKLAERUNG 
STOEREN'

>>> r = 'LAETITIA  COEUR  ELEPHANT  FRANCAIS  LUY  STOSS  ERKLAERUNG STOEREN'
>>> r.encode('ascii', 'strict').decode('ascii', 'strict') == r
True

PS Avoid Py3.3 :-)

jmf

[toc] | [prev] | [next] | [standalone]

#29187

On 9/14/2012 12:15 PM, wxjmfauth@gmail.com wrote:

> PS Avoid Py3.3 :-)

pps Start using 3.3 as soon as possible. It has Python's first fully 
portable non-buggy Unicode implementation.  The second release candidate 
is already out.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#29212

Le vendredi 14 septembre 2012 22:45:05 UTC+2, Terry Reedy a écrit :
> On 9/14/2012 12:15 PM, wxjmfauth@gmail.com wrote:
> 
> 
> 
> > PS Avoid Py3.3 :-)
> 
> 
> 
> pps Start using 3.3 as soon as possible. It has Python's first fully 
> 
> portable non-buggy Unicode implementation.  The second release candidate 
> 
> is already out.
> 
> 
> 
> -- 
> 
> Terry Jan Reedy

- I will drop Python.
- No complaints.
- (OT, luckily one of the two Unicode TeX engines is called LuaTeX.)

jmf

[toc] | [prev] | [next] | [standalone]

#29213

Le vendredi 14 septembre 2012 22:45:05 UTC+2, Terry Reedy a écrit :
> On 9/14/2012 12:15 PM, wxjmfauth@gmail.com wrote:
> 
> 
> 
> > PS Avoid Py3.3 :-)
> 
> 
> 
> pps Start using 3.3 as soon as possible. It has Python's first fully 
> 
> portable non-buggy Unicode implementation.  The second release candidate 
> 
> is already out.
> 
> 
> 
> -- 
> 
> Terry Jan Reedy

- I will drop Python.
- No complaints.
- (OT, luckily one of the two Unicode TeX engines is called LuaTeX.)

jmf

[toc] | [prev] | [next] | [standalone]

#29177

Le jeudi 13 septembre 2012 23:25:27 UTC+2, Tim Chase a écrit :
> I've got a bunch of text in Portuguese and to transmit them, need to
> 
> have them in us-ascii (7-bit).  I'd like to keep as much information
> 
> as possible, just stripping accents, cedillas, tildes, etc.  So
> 
> "serviço móvil" becomes "servico movil".  Is there anything stock
> 
> that I've missed?  I can do mystring.encode('us-ascii', 'replace')
> 
> but that doesn't keep as much information as I'd hope.
> 

Interesting case. It's where the coding of characters
meets characters usage, scripts, typography, linguistic
features.

I cann't discuss the Portugese case, but in French
and in German one way to achieve the task is to
convert the text in uppercases. It preserves a correct
text.

>>> s = 'Lætitia  cœur  éléphant  français  LUŸ  Stoß  Erklärung  stören'
>>> libfrancais.SpecMajuscules(s)
'LAETITIA  COEUR  ELEPHANT  FRANCAIS  LUY  STOSS  ERKLAERUNG 
STOEREN'

>>> r = 'LAETITIA  COEUR  ELEPHANT  FRANCAIS  LUY  STOSS  ERKLAERUNG STOEREN'
>>> r.encode('ascii', 'strict').decode('ascii', 'strict') == r
True

PS Avoid Py3.3 :-)

jmf

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web