Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #79832 > unrolled thread

Regex substitution trouble

Back to article view | Back to comp.lang.python

Contents

  Regex substitution trouble massi_srb@msn.com - 2014-10-28 04:02 -0700
    Re: Regex substitution trouble Chris Angelico <rosuav@gmail.com> - 2014-10-28 22:24 +1100
      Re: Regex substitution trouble massi_srb@msn.com - 2014-10-28 05:28 -0700
        Re: Regex substitution trouble Chris Angelico <rosuav@gmail.com> - 2014-10-28 23:59 +1100
        Re: Regex substitution trouble MRAB <python@mrabarnett.plus.com> - 2014-10-28 17:52 +0000
    Re: Regex substitution trouble TP <wingusr@gmail.com> - 2014-10-28 06:25 -0700
    Re: Regex substitution trouble Tim <jtim.arnold@gmail.com> - 2014-10-28 09:38 -0700
    Re: Regex substitution trouble Cameron Simpson <cs@zip.com.au> - 2014-10-29 13:14 +1100
    Re: Regex substitution trouble Eugene <hexumg@gmail.com> - 2014-11-05 13:44 +0300

#79832 — Regex substitution trouble

Hi everyone,

I'm not really sure if this is the right place to ask about regular expressions, but since I'm usin python I thought I could give a try :-)
Here is the problem, I'm trying to write a regex in order to substitute all the occurences in the form $"somechars" with another string. This is what I wrote:

newstring = re.sub(ur"""(?u)(\$\"[\s\w]+\")""", subst, oldstring)

This works pretty well, but it has a problem, I would need it also to handle the case in which the internal string contains the double quotes, but only if preceeded by a backslash, that is something like $"somechars_with\\"doublequotes".
Can anyone help me to correct it?

Thanks in advance!

[toc] | [next] | [standalone]

#79835

On Tue, Oct 28, 2014 at 10:02 PM,  <massi_srb@msn.com> wrote:
> I'm not really sure if this is the right place to ask about regular expressions, but since I'm usin python I thought I could give a try :-)

Yeah, that sort of thing is perfectly welcome here. Same with
questions about networking in Python, or file I/O in Python, or
anything like that. Not a problem!

> Here is the problem, I'm trying to write a regex in order to substitute all the occurences in the form $"somechars" with another string. This is what I wrote:
>
> newstring = re.sub(ur"""(?u)(\$\"[\s\w]+\")""", subst, oldstring)
>
> This works pretty well, but it has a problem, I would need it also to handle the case in which the internal string contains the double quotes, but only if preceeded by a backslash, that is something like $"somechars_with\\"doublequotes".
> Can anyone help me to correct it?

But this is a problem. You can use look-ahead assertions and such to
allow the string \" inside your search string, but presumably the
backslash ought itself to be escapable, in order to make it possible
to have a loose backslash legal at the end of the string. I suggest
that, instead of a regex, you look for a different way of parsing.
What's the surrounding text like?

ChrisA

[toc] | [prev] | [next] | [standalone]

#79838

Hi Chris, thanks for the reply. I tried to use look ahead assertions, in particular I modified the regex this way:

newstring = re.sub(ur"""(?u)(\$\"[\s\w(?<=\\)\"]+\")""", subst, oldstring) 

but it does not work. I'm absolutely not a regex guru so I'm surely missing something. The strings I'm dealing with are similar to formulas, let's say something like:

'$["simple_input"]+$["messed_\\"_input"]+10'

Thanks for any help!

[toc] | [prev] | [next] | [standalone]

#79839

(Please quote enough of the previous text to provide context, and
write your replies underneath the quoted text - don't assume that
everyone's read the previous posts. Thanks!)

On Tue, Oct 28, 2014 at 11:28 PM,  <massi_srb@msn.com> wrote:
> Hi Chris, thanks for the reply. I tried to use look ahead assertions, in particular I modified the regex this way:
>
> newstring = re.sub(ur"""(?u)(\$\"[\s\w(?<=\\)\"]+\")""", subst, oldstring)
>
> but it does not work. I'm absolutely not a regex guru so I'm surely missing something.

Yeah, I'm not a high-flying regex programmer either, so I'll leave the
specifics for someone else to answer. Tip, though: Print out your
regex, to see if it's really what you think it is. When you get
backslashes and quotes coming through, sometimes you can get tangled,
even in a raw string literal; sometimes, one quick print(some_re) can
save hours of hair-pulling.

> The strings I'm dealing with are similar to formulas, let's say something like:
>
> '$["simple_input"]+$["messed_\\"_input"]+10'
>
> Thanks for any help!

Hmm. This looks like a job for ast.literal_eval with an actual
dictionary. All you'd have to do is replace every instance of $ with a
dict literal; it mightn't be efficient, but it would be safe. Using
Python 2.7.8 as you appear to be on 2.x:

>>> expr = '$["simple_input"]+$["messed_\\"_input"]+10'
>>> values = {"simple_input":123, "messed_\"_input":75}
>>> ast.literal_eval(expr.replace("$",repr(values)))
Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    ast.literal_eval(expr.replace("$",repr(values)))
  File "C:\Python27\lib\ast.py", line 80, in literal_eval
    return _convert(node_or_string)
  File "C:\Python27\lib\ast.py", line 79, in _convert
    raise ValueError('malformed string')
ValueError: malformed string

Unfortunately, it doesn't appear to work, as evidenced by the above
message. It works with the full (and dangerous) eval, though:

>>> eval(expr.replace("$",repr(values)))
208

Can someone who better knows ast.literal_eval() explain what's
malformed about this? The error message in 3.4 is a little more
informative, but not much more helpful:
ValueError: malformed node or string: <_ast.BinOp object at 0x0169BAF0>
My best theory is that subscripting isn't allowed, though this seems odd.

In any case, it ought in theory to be possible to use Python's own
operations on this. You might have to do some manipulation, but it'd
mean you can leverage a full expression evaluator that already exists.
I'd eyeball the source code for ast.literal_eval() and see about
making an extended version that allows the operations you want.

If you can use something other than a dollar sign - something that's
syntactically an identifier - you'll be able to skip the textual
replace() operation, which is risky (might change the wrong thing). Do
that, and you could have your own little evaluator that uses the ast
module for most of its work, and simply runs a little recursive walker
that deals with the nodes as it finds them.

ChrisA

[toc] | [prev] | [next] | [standalone]

#79849

On 2014-10-28 12:28, massi_srb@msn.com wrote:
> Hi Chris, thanks for the reply. I tried to use look ahead assertions, in particular I modified the regex this way:
>
> newstring = re.sub(ur"""(?u)(\$\"[\s\w(?<=\\)\"]+\")""", subst, oldstring)
>
> but it does not work. I'm absolutely not a regex guru so I'm surely missing something. The strings I'm dealing with are similar to formulas, let's say something like:
>
> '$["simple_input"]+$["messed_\\"_input"]+10'
>
> Thanks for any help!
>
Your original post said you wanted to match strings like:

$"somechars_with\\"doublequotes".

This regex will do that:

ur'\$"[^"\\]*(?:\\.[^"\\]*)*"'

However, now you say you want to match:

'$["simple_input"]'

This is different; it has '[' immediately after the '$' instead of '"'.

[toc] | [prev] | [next] | [standalone]

#79840

[Multipart message — attachments visible in raw view] — view raw

On Tue, Oct 28, 2014 at 4:02 AM, <massi_srb@msn.com> wrote:

> Hi everyone,
>
> I'm not really sure if this is the right place to ask about regular
> expressions, but since I'm usin python I thought I could give a try :-)
> Here is the problem, I'm trying to write a regex in order to substitute
> all the occurences in the form $"somechars" with another string. This is
> what I wrote:
>
> newstring = re.sub(ur"""(?u)(\$\"[\s\w]+\")""", subst, oldstring)
>
> This works pretty well, but it has a problem, I would need it also to
> handle the case in which the internal string contains the double quotes,
> but only if preceeded by a backslash, that is something like
> $"somechars_with\\"doublequotes".
> Can anyone help me to correct it?
>
> Thanks in advance!
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Carefully reading the Strings section of "Example Regexes to Match Common
Programming Language Constructs" [1] should (with a bit of effort), solve
your problem I think. Note the use of the negated character class for one
thing.

[1] http://www.regular-expressions.info/examplesprogrammer.html

[toc] | [prev] | [next] | [standalone]

#79848

On Tuesday, October 28, 2014 7:03:00 AM UTC-4, mass...@msn.com wrote:
> Hi everyone,
> I'm not really sure if this is the right place to ask about regular expressions, but since I'm usin python I thought I could give a try :-)
> Here is the problem, I'm trying to write a regex in order to substitute all the occurences in the form $"somechars" with another string. This is what I wrote:
> newstring = re.sub(ur"""(?u)(\$\"[\s\w]+\")""", subst, oldstring)
> 
> This works pretty well, but it has a problem, I would need it also to handle the case in which the internal string contains the double quotes, but only if preceeded by a backslash, that is something like $"somechars_with\\"doublequotes".
> Can anyone help me to correct it?
> Thanks in advance!

You have some good answers already, but I wanted to let you know about a tool you may already have which is useful for experimenting with regexps. On windows, the file `redemo.py` is in the Tools/Scripts folder. 

If you're on a Mac, see http://stackoverflow.com/questions/1811236/how-can-i-run-redemo-py-or-equivalent-on-a-mac

It has really helped me work on some tough regexps.
good luck,
--Tim

[toc] | [prev] | [next] | [standalone]

#79868

On 28Oct2014 04:02, massi_srb@msn.com <massi_srb@msn.com> wrote:
>I'm not really sure if this is the right place to ask about regular 
>expressions, but since I'm usin python I thought I could give a try :-)
>Here is the problem, I'm trying to write a regex in order to substitute all the occurences in the form $"somechars" with another string. This is what I wrote:
>
>newstring = re.sub(ur"""(?u)(\$\"[\s\w]+\")""", subst, oldstring)
>
>This works pretty well, but it has a problem, I would need it also to handle the case in which the internal string contains the double quotes, but only if preceeded by a backslash, that is something like $"somechars_with\\"doublequotes".
>Can anyone help me to correct it?

People seem to be making this harder than it should be.

I'd just be fixing up your definition of what's inside the quotes. There seem 
to be 3 kinds of things:

   - not a double quote or backslash
   - a backslash followed by a double quote
   - a backslash followed by not a double quote

Kind 3 is a policy call - take the following character or not? I would go with 
treating it like kind 2 myself.

So you have:

   1 [^\\"]
   2 \\"
   3 \\[^"]

and fold 2 and 3 into:

   2+3 \\.

So your regexp inner becomes:

   ([^\\"]|\\.)*

and the whole thing becomes:

   \$"(([^\\"]|\\.)*)"

and as a raw string:
   
   ur'\$"(([^\\"]|\\.)*)"'

choosing single quotes to be more readable given the double quotes in the 
regexp.

Cheers,
Cameron Simpson <cs@zip.com.au>
-- 
cat: /Users/cameron/rc/mail/signature.: No such file or directory

Language... has created the word "loneliness" to express the pain of
being alone. And it has created the word "solitude" to express the glory
of being alone. - Paul Johannes Tillich

[toc] | [prev] | [next] | [standalone]

#80256

massi_srb@msn.com wrote:

> Hi everyone,
> 
> I'm not really sure if this is the right place to ask about regular
> expressions, but since I'm usin python I thought I could give a try 
:-)
> Here is the problem, I'm trying to write a regex in order to 
substitute
> all the occurences in the form $"somechars" with another string. This 
is
> what I wrote:
> 
> newstring = re.sub(ur"""(?u)(\$\"[\s\w]+\")""", subst, oldstring)
> 
> This works pretty well, but it has a problem, I would need it also to
> handle the case in which the internal string contains the double 
quotes,
> but only if preceeded by a backslash, that is something like
> $"somechars_with\\"doublequotes". Can anyone help me to correct it?
> 
> Thanks in advance!
Hi!

Next snippet works for me:

re.sub(r'\$"([\s\w]+(\\")*[\s\w]+)+"', 'noop', r'$"te\"sts\"tri\"ng"')

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web