Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #49275

Re: re.finditer() skips unicode into selection

Show all headers | View raw

On 26/06/2013 20:18, akshay.ksth@gmail.com wrote:
> I am using the following Highlighter class for Spell Checking to work on my QTextEdit.
>
> class Highlighter(QSyntaxHighlighter):

In Python 2.7, the re module has a somewhat limited idea of what a
"word" character is. It recognises 'DEVANAGARI LETTER NA' as a letter,
but 'DEVANAGARI VOWEL SIGN E' as a diacritic. The pattern ur'(?u)\w+'
will therefore split "नेपाली" into 3 parts.

>      pattern = ur'\w+'
>      def __init__(self, *args):
>          QSyntaxHighlighter.__init__(self, *args)
>          self.dict = None
>
>      def setDict(self, dict):
>          self.dict = dict
>
>      def highlightBlock(self, text):
>          if not self.dict:
>              return
>          text = unicode(text)
>          format = QTextCharFormat()
>          format.setUnderlineColor(Qt.red)
>          format.setUnderlineStyle(QTextCharFormat.SpellCheckUnderline)

The LOCALE flag is for locale-sensitive 1-byte per character
bytestrings. It's rarely useful.

The UNICODE flag is for dealing with Unicode strings, which is what you
need here. You shouldn't be using both at the same time!

>          unicode_pattern=re.compile(self.pattern,re.UNICODE|re.LOCALE)
>
>          for word_object in unicode_pattern.finditer(text):
>              if not self.dict.spell(word_object.group()):
>                  print word_object.group()
>                  self.setFormat(word_object.start(), word_object.end() - word_object.start(), format)
>
> But whenever I pass unicode values into my QTextEdit the re.finditer() does not seem to collect it.
>
> When I pass "I am a नेपाली" into the QTextEdit. The output is like this:
>
>      I I I a I am I am I am a I am a I am a I am a I am a I am a I am a I am a
>
> It is completely ignoring the unicode. What might be the issue. I am new to PyQt and regex. Im using Python 2.7 and PyQt4.
>
There's an alternative regex implementation at:

http://pypi.python.org/pypi/regex

It's a drop-in replacement for the re module, but with a lot of
additions, including better handling of Unicode.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

re.finditer() skips unicode into selection akshay.ksth@gmail.com - 2013-06-26 12:18 -0700
  Re: re.finditer() skips unicode into selection Terry Reedy <tjreedy@udel.edu> - 2013-06-26 16:14 -0400
  Re: re.finditer() skips unicode into selection MRAB <python@mrabarnett.plus.com> - 2013-06-26 21:24 +0100
  Re: re.finditer() skips unicode into selection darpan6aya <akshay.ksth@gmail.com> - 2013-06-26 20:26 -0700
    Re: re.finditer() skips unicode into selection darpan6aya <akshay.ksth@gmail.com> - 2013-06-26 20:31 -0700
      Re: re.finditer() skips unicode into selection darpan6aya <akshay.ksth@gmail.com> - 2013-06-26 20:39 -0700
        Re: re.finditer() skips unicode into selection darpan6aya <akshay.ksth@gmail.com> - 2013-06-26 21:25 -0700

csiph-web