Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #5148 > unrolled thread

Escaping of line breaks for text replacement

Back to article view | Back to comp.unix.programmer

Contents

  Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-02 14:00 +0100
    Re: Escaping of line breaks for text replacement jt@toerring.de (Jens Thoms Toerring) - 2014-03-02 15:49 +0000
      Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-03 08:38 +0100
    Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-02 16:52 +0000
      Re: Escaping of line breaks for text replacement Kaz Kylheku <kaz@kylheku.com> - 2014-03-02 17:53 +0000
        Re: Escaping of line breaks for text replacement Kaz Kylheku <kaz@kylheku.com> - 2014-03-02 20:27 +0000
          Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-03 16:17 +0000
            Re: Escaping of line breaks for text replacement Kaz Kylheku <kaz@kylheku.com> - 2014-03-03 17:44 +0000
              Re: Escaping of line breaks for text replacement Geoff Clare <geoff@clare.See-My-Signature.invalid> - 2014-03-04 13:30 +0000
      Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-02 20:50 +0100
        Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-03 16:52 +0000
      Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-03 09:24 +0100
      Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-03 12:55 +0100
        Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-03 17:02 +0000
          Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-06 12:02 +0100
            Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-06 18:51 +0000
              Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-06 20:48 +0100
                Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-06 22:59 +0000
                  Re: Escaping of line breaks for text replacement Kaz Kylheku <kaz@kylheku.com> - 2014-03-07 06:38 +0000
                    Re: Escaping of line breaks for text replacement Ben Bacarisse <ben.usenet@bsb.me.uk> - 2014-03-07 13:33 +0000
                    Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-07 15:08 +0000
                  Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-07 11:23 +0100
                    Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-07 14:43 +0000
                      Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-07 17:23 +0100
                        Re: Escaping of line breaks for text replacement dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-07 19:53 +0000
                          Bash 4.3.0 escape expansion (was: Re: Escaping of line breaks for text replacement) dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-08 12:26 +0000
                            Re: Bash 4.3.0 escape expansion (was: Re: Escaping of line breaks for text replacement) Kaz Kylheku <kaz@kylheku.com> - 2014-03-08 20:05 +0000
                              Re: Bash 4.3.0 escape expansion Ben Bacarisse <ben.usenet@bsb.me.uk> - 2014-03-09 03:48 +0000
                              Re: Bash 4.3.0 escape expansion dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-09 13:21 +0000
    Re: Escaping of line breaks for text replacement Kaz Kylheku <kaz@kylheku.com> - 2014-03-02 18:11 +0000
      Re: Escaping of line breaks for text replacement Markus Elfring <Markus.Elfring@web.de> - 2014-03-02 20:08 +0100
        Re: Escaping of line breaks for text replacement Kaz Kylheku <kaz@kylheku.com> - 2014-03-02 20:35 +0000
          Re: Which content should be escaped? Markus Elfring <Markus.Elfring@web.de> - 2014-03-03 10:30 +0100
            Re: Which content should be escaped? dave.gma+news002@googlemail.com.invalid (Dave Sines) - 2014-03-04 14:04 +0000
              Re: Which content should be escaped? Kaz Kylheku <kaz@kylheku.com> - 2014-03-04 17:32 +0000
    Re: Escaping of line breaks for text replacement Siri Cruz <chine.bleu@yahoo.com> - 2014-03-07 02:47 -0800

Page 2 of 2 — ← Prev page 1 [2]

#5185

Kaz Kylheku <kaz@kylheku.com> wrote:
> On 2014-03-06, Dave Sines <dave.gma+news002@googlemail.com.invalid> wrote:
>> Here's a demo Makefile (it creates r.txt and t.txt in the working
>> directory)
>> 
>> #v+
> 
> What is this? Both my news-reader and editor recognize it and colorize
> the stuff between. Some de-facto standard delimiting "verbatim" blocks?

Yes.  I think it was originally introduced in slrn.

> Is this in some RFC?

Uh, pass.

[toc] | [prev] | [next] | [standalone]

#5181

>> Would you like to tell a bit more about the bash versions you try out?
> 
> 4.3.0, 4.2.37, 4.2.20, 4.1.9, 4.0.35, 3.2.48, 3.1.17, 2.05b.7.

How often were differences in the run time behaviour you show here discussed by
other forums or news groups already?

Regards,
Markus

[toc] | [prev] | [next] | [standalone]

#5184

Markus Elfring <Markus.Elfring@web.de> wrote:
>>> Would you like to tell a bit more about the bash versions you try out?
>> 
>> 4.3.0, 4.2.37, 4.2.20, 4.1.9, 4.0.35, 3.2.48, 3.1.17, 2.05b.7.
> 
> How often were differences in the run time behaviour you show here
> discussed by other forums or news groups already?

None that I'm aware of.

[toc] | [prev] | [next] | [standalone]

#5186

>> How often were differences in the run time behaviour you show here
>> discussed by other forums or news groups already?
> 
> None that I'm aware of.

Would it be useful to expand the clarification of corresponding implementation
details to other mailing lists or issue trackers?

Regards,
Markus

[toc] | [prev] | [next] | [standalone]

#5187

Markus Elfring <Markus.Elfring@web.de> wrote:
>>> How often were differences in the run time behaviour you show here
>>> discussed by other forums or news groups already?
>> 
>> None that I'm aware of.
> 
> Would it be useful to expand the clarification of corresponding
> implementation details to other mailing lists or issue trackers?

It may be a bug in bash 4.3, I've reported it to the bash bugs list.
The relevant part of the bug report follows:

Description:
        When invoked as sh, bash 4.3.0 doesn't interpret C-style escapes
within double-quoted substitution expansions ("${var/$'what'/ever}").

Repeat-By:

  bash -c "a=-@- ; echo \${a/\$'\\x40'/+} ; echo \"\${a/\$'\\x40'/+}\""
  -+-
  -+-
  sh   -c "a=-@- ; echo \${a/\$'\\x40'/+} ; echo \"\${a/\$'\\x40'/+}\""
  -+-
  -@-

[toc] | [prev] | [next] | [standalone]

#5188 — Bash 4.3.0 escape expansion (was: Re: Escaping of line breaks for text replacement)

Dave Sines <dave.gma+news002@googlemail.com.invalid> wrote:
> Markus Elfring <Markus.Elfring@web.de> wrote:

>> Would it be useful to expand the clarification of corresponding
>> implementation details to other mailing lists or issue trackers?
> 
> It may be a bug in bash 4.3, I've reported it to the bash bugs list.
> The relevant part of the bug report follows:
> 
> Description:
>        When invoked as sh, bash 4.3.0 doesn't interpret C-style escapes
> within double-quoted substitution expansions ("${var/$'what'/ever}").

There's a response (and a patch) available here:

  <http://article.gmane.org/gmane.comp.shells.bash.bugs/21208>
  <http://cache.gmane.org//gmane/comp/shells/bash/bugs/21208-001.bin>

[toc] | [prev] | [next] | [standalone]

#5189 — Re: Bash 4.3.0 escape expansion (was: Re: Escaping of line breaks for text replacement)

On 2014-03-08, Dave Sines <dave.gma+news002@googlemail.com.invalid> wrote:
> Dave Sines <dave.gma+news002@googlemail.com.invalid> wrote:
>> Markus Elfring <Markus.Elfring@web.de> wrote:
>
>>> Would it be useful to expand the clarification of corresponding
>>> implementation details to other mailing lists or issue trackers?
>> 
>> It may be a bug in bash 4.3, I've reported it to the bash bugs list.
>> The relevant part of the bug report follows:
>> 
>> Description:
>>        When invoked as sh, bash 4.3.0 doesn't interpret C-style escapes
>> within double-quoted substitution expansions ("${var/$'what'/ever}").
>
> There's a response (and a patch) available here:

What a mind-blowing coincidence!

I JUST used this Bash syntax (for the first time ever, I think!) merely
20 minutes ago, and ran into some issues with quotes.

I was trying to escape occurrences of ' with '\''.

I found two strange behaviors:

  $ echo $FOO
  a'b'c

Goal:   a'\''b'\''c

Not quite:

  $ echo "${FOO//\'/'\\\'\'}"
  a'\\'\'b'\\'\'c

Note how in the replacement pattern '\\\'\' the first quote is verbatim
and nicely goes through (and so it is not an activel quote), then there is an
escaped backslash \\ which However the two \' sequences each turn into \':
backslash is not a recognized escape for the quote.

Then, here is another kicker. Suppose we turn the final \' into just '

  $ echo "${FOO//\'/'\\\'\}"
  > '
  > "
  > }
  > what are you waiting for me to close?
  > ??
  > 
  > Ah screw this, Ctrl-C.
  > ^C

The syntax won't close no matter what you type. It does, however, evidently
espect the single quote (which came through the substitution verbatim!) to be
closed.

If we leave out a matching quote, Bash prompts us to close something:

  $ echo "${FOO//\'/'\\}"
  > 
  > ^C

If we add the closing quote, the opening and closing quotes are part of the
substitution!

So we have to obey proper quoting syntax, and yet that quoting is not
processed and stripped!

  $ echo "${FOO//\'/'\\'}"
  a'\'b'\'c

Or whatever ...

[toc] | [prev] | [next] | [standalone]

#5190 — Re: Bash 4.3.0 escape expansion

Kaz Kylheku <kaz@kylheku.com> writes:
<snip?
> I JUST used this Bash syntax (for the first time ever, I think!) merely
> 20 minutes ago, and ran into some issues with quotes.
<snip>
>   $ echo $FOO
>   a'b'c
>
> Goal:   a'\''b'\''c

<snip>
> Or whatever ...

You may have found this yourself but you can write:

  Q="'"
  echo "${FOO//\'/'\\'$Q}"

It's horridly asymmetric, but it illustrates a useful general trick --
put odd stuff in plain variables.  This is the minimal version (it only
deals with the unmatched quote), but for readability I'd write:

  Q="'\''"
  echo "${FOO//\'/$Q}"

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#5191 — Re: Bash 4.3.0 escape expansion

Kaz Kylheku <kaz@kylheku.com> wrote:
> On 2014-03-08, Dave Sines <dave.gma+news002@googlemail.com.invalid> wrote:

[bash substitution expansion oddness]

> I JUST used this Bash syntax (for the first time ever, I think!) merely
> 20 minutes ago, and ran into some issues with quotes.
> 
> I was trying to escape occurrences of ' with '\''.
> 
> I found two strange behaviors:
> 
>  $ echo $FOO
>  a'b'c
> 
> Goal:   a'\''b'\''c

[...]

> Then, here is another kicker. Suppose we turn the final \' into just '
> 
>  $ echo "${FOO//\'/'\\\'\}"
>  > '
>  > "
>  > }
>  > what are you waiting for me to close?

The final backslash escapes the closing brace so the trailing double
quote opens a new string.  Close the double-quote, close the braces,
close the outer double-quote:

  $ echo "${FOO//\'/'\\\'\}"
  > "}"

If you meant:

  $ echo "${FOO//\'/'\\\''}"

The final single-quote opens a new string.  Close that, provide the
closing brace and then the double-quote:

  $ echo "${FOO//\'/'\\\''}"
  > '}"

[...]

> So we have to obey proper quoting syntax, and yet that quoting is not
> processed and stripped!
> 
>  $ echo "${FOO//\'/'\\'}"
>  a'\'b'\'c

This was changed for bash 4.3 which now behaves like ksh and mksh.

     echo "${FOO//\'/\'\\\'\'}"

Or:

     echo "${FOO//"'"/"'\\''"}"

For bash versions prior to 4.3, either use the method posted by Ben
Bacarisse elsewhere in this thread or perform the substitution unquoted
(which requires assignment to an intermediate variable to avoid
problems with word splitting etc).

  a=${FOO//\'/\'\\\'\'}
  echo "$a"

[toc] | [prev] | [next] | [standalone]

#5153

["Followup-To:" header set to comp.unix.programmer.]
On 2014-03-02, Markus Elfring <Markus.Elfring@web.de> wrote:
> Hello,
>
> I get the following result if I try out such a command on my openSUSE system.
>
> elfring@Sonne:~> $SHELL --version && sed --version && LINE='echo ---' && $LINE \
> && DRAFT='template1.txt' && cat $DRAFT && $LINE \
> && INPUT='replacement1.txt' && cat $INPUT && XY=$(< $INPUT) && $LINE \
> && LANG=C sed "s/# placeholder/$XY/" $DRAFT
> GNU bash, Version 4.2.45(1)-release (x86_64-suse-linux-gnu)
>    ...
> sed (GNU sed) 4.2.2
>    ...
> ---
> +++++++++++++
> # placeholder
> *************
> ---
> ABC
> XYZ
> ---
> sed: -e expression #1, char 19: unterminated `s' command

Unix already has a mechanism for interpolating variables into multi-line
text, called a "here document".

Here, it behooves us to use it, rather than reach for sed tricks
to do variable substitution.

Live demo:

$ cat template.txt 
my terminal is $TERM
the number of lines is $LINES
the number of colums is $COLUMNS

Now: dynamically wrap the here-doc syntax around the contents of template.txt
and evaluate it:

$ eval "cat <<EOF
$(cat template.txt)
EOF"
my terminal is xterm
the number of lines is 44
the number of colums is 80

You can have newlines in the variables that are substituted.

If you put the here-doc prologue and epilogue "cat <<EOF" and "EOF" the
template itself, then it's just a straight eval "$(cat template.txt)".

Remember to escape dollar signs in the here doc. It has to follow the
rules for a here doc.

If you ever do use sed for variable substitution have a clear delimiter on the
variables.  For instance, call it @REPLACEMENT@, not replacement,
for reasons which should be obvious.

[toc] | [prev] | [next] | [standalone]

#5155

> Unix already has a mechanism for interpolating variables into multi-line
> text, called a "here document".

Thanks for your suggestion.


> You can have newlines in the variables that are substituted.

This detail is nice.


> Remember to escape dollar signs in the here doc. It has to follow the
> rules for a here doc.

This implementation detail does not fit to my expectations for the replacement
input file so far.
How would you achieve the corresponding escaping for dollar characters instead
of line breaks (and slashes eventually)?

Regards,
Markus

[toc] | [prev] | [next] | [standalone]

#5158

On 2014-03-02, Markus Elfring <Markus.Elfring@web.de> wrote:
>> Remember to escape dollar signs in the here doc. It has to follow the
>> rules for a here doc.
>
> This implementation detail does not fit to my expectations for the replacement
> input file so far.
> How would you achieve the corresponding escaping for dollar characters instead
> of line breaks (and slashes eventually)?

We have to trade one form escaping instead of another, but the escaping we have
to do now is in the body of the static template, to ensure it is a valid
here-doc, not in the variable being substituted, which has "wild and wooly"
run-time contents.

The template is more "static" relative to the contents of the wild and wooly
variable being substituted. We can just write the template correctly by hand,
and that's it.

(By the way, in a here doc template have to watch out not only for unintended
variable substitutions due to stray dollar signs and some other things like
backslashes or the coincidental occurence of the EOF marker.)

[toc] | [prev] | [next] | [standalone]

#5163 — Re: Which content should be escaped?

> We have to trade one form escaping instead of another, but the escaping we have
> to do now is in the body of the static template, to ensure it is a valid
> here-doc, not in the variable being substituted, which has "wild and wooly"
> run-time contents.
> 
> The template is more "static" relative to the contents of the wild and wooly
> variable being substituted. We can just write the template correctly by hand,
> and that's it.

I would agree to your advice in the general situation. The simple test data I
started the clarification with fits to your view.

But I find that the real template files have got such constraints that the
here-document rules are not useful there.


> (By the way, in a here doc template have to watch out not only for unintended
> variable substitutions due to stray dollar signs and some other things like
> backslashes or the coincidental occurence of the EOF marker.)

Your hint is very interesting because it points some implementation details out
for further consideration in the involved software design. I prefer to adjust
the replacement text for my concrete use case.

Regards,
Markus

[toc] | [prev] | [next] | [standalone]

#5174 — Re: Which content should be escaped?

Markus Elfring <Markus.Elfring@web.de> wrote:
>> We have to trade one form escaping instead of another, but the
>> escaping we have to do now is in the body of the static template, to
>> ensure it is a valid here-doc, not in the variable being substituted,
>> which has "wild and wooly" run-time contents.
>>
>> The template is more "static" relative to the contents of the wild
>> and wooly variable being substituted. We can just write the template
>> correctly by hand, and that's it.
> 
> I would agree to your advice in the general situation. The simple test
> data I started the clarification with fits to your view.
>
> But I find that the real template files have got such constraints that
> the here-document rules are not useful there.

Have you considered using cpp?

template1.txt:

  +++++++++++++
  #include REPL
  *************

replacement1.txt:

  ABC
  XYZ

  cpp -P -nostdinc -xc -DREPL="\"replacement1.txt\"" template1.txt

[toc] | [prev] | [next] | [standalone]

#5175 — Re: Which content should be escaped?

On 2014-03-04, Dave Sines <dave.gma+news002@googlemail.com.invalid> wrote:
> Markus Elfring <Markus.Elfring@web.de> wrote:
>>> We have to trade one form escaping instead of another, but the
>>> escaping we have to do now is in the body of the static template, to
>>> ensure it is a valid here-doc, not in the variable being substituted,
>>> which has "wild and wooly" run-time contents.
>>>
>>> The template is more "static" relative to the contents of the wild
>>> and wooly variable being substituted. We can just write the template
>>> correctly by hand, and that's it.
>> 
>> I would agree to your advice in the general situation. The simple test
>> data I started the clarification with fits to your view.
>>
>> But I find that the real template files have got such constraints that
>> the here-document rules are not useful there.
>
> Have you considered using cpp?

You have to be careful, because the C preprocessor decomposes the input
into C preprocessor tokens. (I can't think of a good example of how that
will screw up, off the top of my head.) It also assembles line continuations,
strips comments, and reacts to lines that begin with #.  You are severely
limited in what constitutes a macro name.

> template1.txt:
>
>   +++++++++++++
>   #include REPL
>   *************
>
> replacement1.txt:
>
>   ABC
>   XYZ
>
>   cpp -P -nostdinc -xc -DREPL="\"replacement1.txt\"" template1.txt

I once had a Makefile for generating text documents from templates, where
three suffixes were supported: .ct (C preprocessor template), .st (shell template) and .mt 
(m4 template).

For each one, the build rule would generate a command line with all
the variable definitions for the given language (cpp, shell here document
or m4) and any necessary wrapping (for the here doc).

In that project, I used the appopriate processor based on the situation.

I managed to find this. The relevant parts, which use GNU Make syntax,
are:

%: %.ct
        cpp -P $(shell env | sed -e "s/.*/-D_ENV_'&'/") $< > $@

%: %.st
        ( export __FILE__=$< ; echo "cat <<!" ; cat $< ; echo "!" ) | bash > $@

%: %.mt
        rm -f $@.err
        m4 -D__FILE__=$< $(shell env | sed -e "s/\([^=]*\)=\(.*\)/-D'_ENV_\\1=\\2'/") $< > $@ 2> $@.err
        if [ -s $@.err ] ; then cat $@.err ; exit 1 ; fi
        rm -f $@.err


In .ct and .mt files you can refer to environment variables as _ENV_<name>.

So for instance if we have a foo.st file containing

  the TERM variable is $TERM

and a bar.ct file containing 

  the TERM variable is __ENV_TERM

we can do "make foo bar" to create a foo and bar which contain, perhaps:

  the TERM variable is xterm

or whatever the value.

[toc] | [prev] | [next] | [standalone]

#5182

> I wonder why I do not get this approach to work so far with some additional
> backslashes here. How should newlines be escaped for this purpose?
> 
> Would you like to share any further ideas?

Don't use sed if sed doesn't do what you want. Use tclsh, perl, etc

@ cat <<':eof:'>/tmp/a
abc def pqr stu
ijk mno abc
def mno def
:eof:


@ tclsh /dev/stdin /tmp/a <<':eof:'

set ch [open [lindex $argv 0]]
set file [read $ch]
close $ch

#   Change
#       abc def pqr stu
#       ijk mno abc
#       def mno def
#   to
#       abc def pqr stu
#       ijk mno + mno def

regsub -all {abc\ndef} $file {+} file

#   Change
#       abc def pqr stu
#       ijk mno + mno def
#   to
#       X X pqr stu
#       ijk mno + mno X

regsub -all {[a-f]+} $file {X} file

set ch [open [lindex $argv 0] w]
puts $ch $file
close $ch

:eof:

-- 
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted.
'I desire mercy, not sacrifice.'

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.unix.programmer

csiph-web