Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #8511 > unrolled thread

rsync: exclude all, then include only specified, wildcard problems..

Back to article view | Back to comp.os.linux.misc

Contents

  rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-19 15:17 +0000
    Re: rsync: exclude all, then include only specified, wildcard problems.. Rikishi42 <skunkworks@rikishi42.net> - 2013-06-20 00:49 +0200
      Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 11:50 +0000
        Re: rsync: exclude all, then include only specified, wildcard problems.. Rikishi42 <skunkworks@rikishi42.net> - 2013-06-26 01:59 +0200
    Re: rsync: exclude all, then include only specified, wildcard problems.. Richard Kettlewell <rjk@greenend.org.uk> - 2013-06-20 10:16 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. The Natural Philosopher <tnp@invalid.invalid> - 2013-06-20 14:19 +0100
        Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 11:17 +0000
    Re: rsync: exclude all, then include only specified, wildcard problems.. Chris Davies <chris-usenet@roaima.co.uk> - 2013-06-20 10:57 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 11:40 +0000
        Re: rsync: exclude all, then include only specified, wildcard problems.. Chris Davies <chris-usenet@roaima.co.uk> - 2013-06-25 21:00 +0100
    Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-20 18:37 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 14:00 +0000
        Re: rsync: exclude all, then include only specified, wildcard problems.. Richard Kettlewell <rjk@greenend.org.uk> - 2013-06-25 15:24 +0100
        Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-25 19:23 +0100
    Re: rsync: exclude all, then include only specified, wildcard problems.. Chick Tower <c.tower@deadspam.com> - 2013-06-20 18:04 +0000
    Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-25 23:37 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-26 01:54 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-28 16:12 +0000
        Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-29 01:30 +0100
        Re: rsync: exclude all, then include only specified, wildcard problems.. Chris Davies <chris-usenet@roaima.co.uk> - 2013-07-01 09:51 +0100

#8511 — rsync: exclude all, then include only specified, wildcard problems..

I've been struggling with rsync (and its helpfully voluminous, but somewhat
confusing, man page) all day.. :-(

I am trying to write a shell script to use rsync to sync certain files to 
another location on the same computer. (The files are checked out from a
subversion repository into the source location and need to be transferred 
to their final location, and have ownership and permission changes.)

Just in case something has inadvertently found its way into the source
location that shouldn't have (eg, a file that was checked in in the wrong
place or some other human error), I want to essentially first of all 
exclude everything in the source folder from being synced, and then 
explicitly allow only the folder hierarchies that are supposed to be 
synced.

I'm having problems trying to work out what my exclude rule should be [1], 
and I think my problems are further compounded by trying to assemble my 
command in a shell script and having wildcards (*) misinterpreted 
somewhere, but I'm not sure what I need to do to ensure that they are
interpreted when they need to be, and passed through unscathed when not!

(I get different results when I run my script, versus running the command 
that is assembled in it, directly at the command line!)


[1] The more I re-read rsync's definitions of / and * in various places and 
quantities, the more confused I get.. ;-(  ]


My script is included below. Any help in trying to diagnose what's wrong
with it gratefully received! (And if you want to pick it apart for 
inelegance too, feel free..)

I've been juggling /s and *s all day and make no claims that what is in
the script below make any sense, that's just where they are at present!


Many thanks for any advice,

David.


::::


#!/bin/sh
#
# rsync-webs: rsync the checked-out svn files into the webserver files area


############################################################
# Config
############################################################

# The repository being used
REPONAME=web-test
#REPONAME=web


# Make sure src & dest "match" (ie, both paths end in 'data')
SRCROOT="/data/import/svn-test/svn-checkedout/$REPONAME/data"
#SRCROOT="/data/import/svn-holding/svn-checkedout/$REPONAME/data"

DSTROOT="/data/import/svn-test/data-test"
DSTROOT_LIVE="/data"


# DRYRUN is the rsync option for running in dryrun mode 
# (no files actually transferred)
DRYRUN="-n"
MODE='dryrun'


# Exclude *all* under "data", and include *only* the allowed folders/files 
# (just to be on the safe side and to prevent inadvertent overwrites,
# "shouldn't happen")
#
# Pattern rules: 
#   * = any path (but stops at slashes)
#   ** = anything (including slashes)
#   dir/*** = dir *and* everything in dir
#
# rsync incl/excl rules are highly confusing. The following web page helped:
# http://www.only10types.com/2012/03/rsync-only-specific-files-and.html
#
# Need to include first, *then* exclude, see:
# https://bugzilla.redhat.com/show_bug.cgi?id=10343
#
# /name = anchor the name to the root of the transfer hierarchy
RS_INCLUDE="
--include='/webs/web1/php_includes/'
--include='/webs/web1/php_snippets/'
--include='/webs/web2/php/'
"

RS_EXCLUDE="
--exclude='*'
"

RS_FILTER=" "
# How to use this:
# http://hintsforums.macworld.com/showthread.php?t=123523
# echo $RS_FILTER | rsync --exclude-from=-  (- = stdin)



# rsync options:
# v = verbose
# i = itemize-changes
# a = archive mode (-rlptgoD (no -H,-A,-X))
# z = use compression
#
# *First* matching pattern is used: ie, need to include first, *then* exclude, 
# see rsync manual, and:
# https://bugzilla.redhat.com/show_bug.cgi?id=10343
RS_OPTS="-vviaz --no-perms --chmod=u=rx,g=rx,o-rwx
$RS_INCLUDE
$RS_EXCLUDE
--cvs-exclude
"


############################################################
# Main
############################################################

if [ "$USER" != "root" ]; then
    echo 'You need to run this script as root, exiting..'
    exit
fi


# Add group perms to the src files first 
# (so that they can be svn update'd later by another dev)
echo 'Adding group permissions to source files..'
chmod -R g+rwx "$SRCROOT"

# chown the src files first 
# (so that the dest files will have the correct u+g ownership)
echo 'chowning source files with required ownership for destination..'
chown -R root:apache "$SRCROOT"


# run rsync in dry-run mode (no changes actually made) UNLESS 
# 'test' or 'commit' option explicitly specified

# test = copy to test destination
if [ "$1" == "-test" ]; then
    unset DRYRUN
    MODE='test'
fi

# commit = copy to real destination
if [ "$1" == "-commit" ]; then
    unset DRYRUN
    DSTROOT="$DSTROOT_LIVE"
    MODE='commit'
fi



# Set up the rsync command
# src/ = trailing slash means sync *contents* of src folder 
# (do not create 'src' folder on dest)
RS_CMD="rsync $DRYRUN  $RS_OPTS  $SRCROOT/  $DSTROOT"



# Sync the files..
echo '** Running in mode: ' "$MODE"
echo '** (specify: -test to sync to test dir, -commit to sync to live dir).'

echo 'Source:'  $SRCROOT
echo '  Dest:'  $DSTROOT
echo 'rsync command:'
echo $RS_CMD

echo
echo '** OK to sync files? [yN]'
read CONFIRM
if [ "$CONFIRM" == "y" ]; then
    echo 'Syncing files..'
    $RS_CMD
fi


echo 'chgrping source files back again..'
chgrp -R staff "$SRCROOT"
echo 'End of script.'

[toc] | [next] | [standalone]

#8515

On 2013-06-19, David <david@55952163-3189045.bogus.domain.invalid> wrote:
> I've been struggling with rsync (and its helpfully voluminous, but somewhat
> confusing, man page) all day.. :-(
>
> I am trying to write a shell script to use rsync to sync certain files to 
> another location on the same computer. (The files are checked out from a
> subversion repository into the source location and need to be transferred 
> to their final location, and have ownership and permission changes.)
>
> Just in case something has inadvertently found its way into the source
> location that shouldn't have (eg, a file that was checked in in the wrong
> place or some other human error), I want to essentially first of all 
> exclude everything in the source folder from being synced, and then 
> explicitly allow only the folder hierarchies that are supposed to be 
> synced.
>
> I'm having problems trying to work out what my exclude rule should be [1], 
> and I think my problems are further compounded by trying to assemble my 
> command in a shell script and having wildcards (*) misinterpreted 
> somewhere, but I'm not sure what I need to do to ensure that they are
> interpreted when they need to be, and passed through unscathed when not!
>
> (I get different results when I run my script, versus running the command 
> that is assembled in it, directly at the command line!)
>
>
> [1] The more I re-read rsync's definitions of / and * in various places and 
> quantities, the more confused I get.. ;-(  ]
>
>
> My script is included below. Any help in trying to diagnose what's wrong
> with it gratefully received! (And if you want to pick it apart for 
> inelegance too, feel free..)
>
> I've been juggling /s and *s all day and make no claims that what is in
> the script below make any sense, that's just where they are at present!

Not entirely certain, but you miht want to look into the --file-from option,
to define what has to be included rather then what has to be excluded.

No time to look into your case in details, hope it helps anyway.


-- 
When in doubt, use brute force.
                -- Ken Thompson

[toc] | [prev] | [next] | [standalone]

#8575

Rikishi42 wrote, on 2013-06-19:
>
> Not entirely certain, but you miht want to look into the --file-from option,
> to define what has to be included rather then what has to be excluded.

I did see --files-from in the manual, but I maybe interpreted the 
description "..specify the exact list of files to transfer" wrongly?

I certainly don't want to have to specify explicitly every single file
that I want to transfer, or does --files-from allow you to specify some 
folders and then, if you explicitly switch -r (recursive) on, it will
indeed "explore" within those folders and include all that it finds? If 
so, perhaps that is what I want to use?

Thanks,

David.

[toc] | [prev] | [next] | [standalone]

#8600

On 2013-06-25, David <david@55952163-3189045.bogus.domain.invalid> wrote:
> Rikishi42 wrote, on 2013-06-19:
>>
>> Not entirely certain, but you miht want to look into the --file-from option,
>> to define what has to be included rather then what has to be excluded.
>
> I did see --files-from in the manual, but I maybe interpreted the 
> description "..specify the exact list of files to transfer" wrongly?
>
> I certainly don't want to have to specify explicitly every single file
> that I want to transfer, or does --files-from allow you to specify some 
> folders and then, if you explicitly switch -r (recursive) on, it will
> indeed "explore" within those folders and include all that it finds? If 
> so, perhaps that is what I want to use?

Well, I can't say I have used it before. But I stumbled on it and it seemed
potentially usefull, so I thought I'd better mension it.
No time to test this myself now, since I'm allready knee deep in rsync
transfers for a project of mine and I'd rather not be sidetracked.

Good luck with it, hope it helped somehow.


-- 
When in doubt, use brute force.
                -- Ken Thompson

[toc] | [prev] | [next] | [standalone]

#8518

David <david@55952163-3189045.bogus.domain.invalid> writes:
> I am trying to write a shell script to use rsync to sync certain files to 
> another location on the same computer. (The files are checked out from a
> subversion repository into the source location and need to be transferred 
> to their final location, and have ownership and permission changes.)
>
> Just in case something has inadvertently found its way into the source
> location that shouldn't have (eg, a file that was checked in in the wrong
> place or some other human error), I want to essentially first of all 
> exclude everything in the source folder from being synced, and then 
> explicitly allow only the folder hierarchies that are supposed to be 
> synced.

That seems like a weird way to do it.  Why not just copy the directories
you’re interested in, each in a separate call to rsync (or cp)?

-- 
http://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#8522

On 20/06/13 10:16, Richard Kettlewell wrote:
> David <david@55952163-3189045.bogus.domain.invalid> writes:
>> I am trying to write a shell script to use rsync to sync certain files to
>> another location on the same computer. (The files are checked out from a
>> subversion repository into the source location and need to be transferred
>> to their final location, and have ownership and permission changes.)
>>
>> Just in case something has inadvertently found its way into the source
>> location that shouldn't have (eg, a file that was checked in in the wrong
>> place or some other human error), I want to essentially first of all
>> exclude everything in the source folder from being synced, and then
>> explicitly allow only the folder hierarchies that are supposed to be
>> synced.
> That seems like a weird way to do it.  Why not just copy the directories
> you’re interested in, each in a separate call to rsync (or cp)?
>
+1.

I too gave up trying to be clever, and produced about 15 calls to rsync 
to back up the bits of the computer that were non volatile.

if there is any serious data to be moved, the overhead is trivial.

-- 
Ineptocracy

(in-ep-toc’-ra-cy) – a system of government where the least capable to lead are elected by the least capable of producing, and where the members of society least likely to sustain themselves or succeed, are rewarded with goods and services paid for by the confiscated wealth of a diminishing number of producers.

[toc] | [prev] | [next] | [standalone]

#8571

The Natural Philosopher wrote, on 2013-06-20:
> On 20/06/13 10:16, Richard Kettlewell wrote:
>> David <david@55952163-3189045.bogus.domain.invalid> writes:
>>> I am trying to write a shell script to use rsync to sync certain files to
>>> another location on the same computer. (The files are checked out from a
>>> subversion repository into the source location and need to be transferred
>>> to their final location, and have ownership and permission changes.)
>>>
>>> Just in case something has inadvertently found its way into the source
>>> location that shouldn't have (eg, a file that was checked in in the wrong
>>> place or some other human error), I want to essentially first of all
>>> exclude everything in the source folder from being synced, and then
>>> explicitly allow only the folder hierarchies that are supposed to be
>>> synced.
>> That seems like a weird way to do it.  Why not just copy the directories
>> you’re interested in, each in a separate call to rsync (or cp)?
>>
> +1.
>
> I too gave up trying to be clever, and produced about 15 calls to rsync 
> to back up the bits of the computer that were non volatile.
>
> if there is any serious data to be moved, the overhead is trivial.

I guess you both maybe have a point! :-)

I suppose I was trying to follow my instincts and was thinking of how to 
assemble *one* rsync command that would Do What I Mean "in a one-er", but 
you may be right that a sequence of simpler commands may be a better way to 
get the job done instead..


David.

[toc] | [prev] | [next] | [standalone]

#8519

In uk.comp.os.linux David <david@55952163-3189045.bogus.domain.invalid> wrote:
> I've been struggling with rsync (and its helpfully voluminous, but
> somewhat confusing, man page) all day.. :-(

IME it's reasonably ok until you hit the FILTER RULES section...!


> I am trying to write a shell script to use rsync to sync certain files
> to another location on the same computer. [...]

> I want to [...] exclude everything in the source folder from being
> synced, and then explicitly allow only the folder hierarchies that
> are supposed to be synced. [...]

Do you even need to worry about filters?

rsync --delete -avHP dir1 dir2 dir3 dir4 /some/target/

Chris

[toc] | [prev] | [next] | [standalone]

#8573

Chris Davies wrote, on 2013-06-20:
> In uk.comp.os.linux David <david@55952163-3189045.bogus.domain.invalid> wrote:
>
>> I am trying to write a shell script to use rsync to sync certain files
>> to another location on the same computer. [...]
>
>> I want to [...] exclude everything in the source folder from being
>> synced, and then explicitly allow only the folder hierarchies that
>> are supposed to be synced. [...]
>
> Do you even need to worry about filters?
>
> rsync --delete -avHP dir1 dir2 dir3 dir4 /some/target/


Ah, I hadn't realised that you could specify several source folders in the
command, that could be useful..

But if the source folders are at different "levels" (which might happen
if we need to later update some existing inherited files which were put
in "less helpful" locations - and may be difficult to move elsewhere), eg:

/data/import/svn-test/svn-checkedout/web-test/data/folder_in_silly_place/
/data/import/svn-test/svn-checkedout/web-test/data/web2/php/
etc..

..would rsync be able to make the right target folders in the destination?

(Sorry, almost as soon as I wrote the above, I thought of the answer: as
long as you specify enough of the path that what you want is specified 
relatively, then it should work..?)


Unfortunately, I probably won't be able to use --delete, as the files that
are in "unhelpful" places are sufficiently far "up" the folder path that
--delete would delete whole swathes of the target file structure "below" 
that level that aren't in the repository / checked-out area. This is
one of the main reasons that I wanted to be very careful to try to specify 
carefully what to exclude and include, in order to prevent inadvertent
"damage" at the target end ..or can I work around this somehow?



Thanks,


David.

[toc] | [prev] | [next] | [standalone]

#8594

In uk.comp.os.linux David <david@55952163-3189045.bogus.domain.invalid> wrote:
> Chris Davies wrote, on 2013-06-20:
>> rsync --delete -avHP dir1 dir2 dir3 dir4 /some/target/


> But if the source folders are at different "levels" (which might happen
> if we need to later update some existing inherited files which were put
> in "less helpful" locations - and may be difficult to move elsewhere), eg:

> /data/import/svn-test/svn-checkedout/web-test/data/folder_in_silly_place/
> /data/import/svn-test/svn-checkedout/web-test/data/web2/php/
> etc..

> ..would rsync be able to make the right target folders in the destination?

The -R flag will keep relative paths intact. It's a surprisingly powerful
but oft-neglected option.


> Unfortunately, I probably won't be able to use --delete, as the files that
> are in "unhelpful" places are sufficiently far "up" the folder path that
> --delete would delete whole swathes of the target file structure "below" 
> that level that aren't in the repository / checked-out area.

Ah ok. You could specify deep paths and use -R to maintain those paths
on the target. Or, as you indicate, you're going to have to resort to
the complication of includes and excludes.

Chris

[toc] | [prev] | [next] | [standalone]

#8525

[ Followup-To: set to colm ]

In comp.os.linux.misc, David <david@55952163-3189045.bogus.domain.invalid>
> wrote:
> I've been struggling with rsync (and its helpfully voluminous, but
> somewhat confusing, man page) all day.. :-(

> explicitly allow only the folder hierarchies that are supposed to be 
> synced.

You want to rsync a few directories from a hierarchy while retaining
the directory structure?

> SRCROOT="/data/import/svn-test/svn-checkedout/$REPONAME/data"
> #SRCROOT="/data/import/svn-holding/svn-checkedout/$REPONAME/data"
> 
> DSTROOT="/data/import/svn-test/data-test"
> DSTROOT_LIVE="/data"

[snip]

> RS_INCLUDE="
> --include='/webs/web1/php_includes/'
> --include='/webs/web1/php_snippets/'
> --include='/webs/web2/php/'
> "

[snip: building up a command line in a string]

You have

 "$SRCROOT"/webs/web1/php_includes/
 "$SRCROOT"/webs/web1/php_snippets/
 "$SRCROOT"/webs/web1/irrelevant_things
 "$SRCROOT"/webs/web2/php/
 "$SRCROOT"/webs/web2/something_to_be_ignored
 "$SRCROOT"/webs/other/stuff

You want

 "$DSTROOT"/webs/web1/php_includes/
 "$DSTROOT"/webs/web1/php_snippets/
 "$DSTROOT"/webs/web2/php/

Right?  Use -R (note use of '/./' to strip preceding path elements).

rsync -av -R                            \
  "$SRCROOT/./webs/web1/php_includes/"  \
  "$SRCROOT/./webs/web1/php_snippets/"  \
  "$SRCROOT/./webs/web2/php/"           \
  "$DSTROOT"



Referring back to the attempt to have embedded quotes within a string,
when the shell expands RS_INCLUDE it will treat the embedded single
quotes as ordinary characters.  To have the behaviour you were aiming
for (are there spaces in there?) you'd have to use eval.  Don't use
eval.  Use arrays.

  RS_INCLUDE=(
    --include='/webs/we 1/php includes/'
    --include='/webs/we 1/php snippets/'
    --include='/webs/web2/php/'
  )
  printf '%s\n' "${RS_INCLUDE[@]}"

Contrast the output from that with the following

  RS_INCLUDE="
    --include='/webs/we 1/php includes/'
    --include='/webs/we 1/php snippets/'
    --include='/webs/web2/php/'
  "
  printf '%s\n' $RS_INCLUDE



The script would look something like this

include=(
  "$SRCROOT/./webs/web1/php_includes/"
  "$SRCROOT/./webs/web1/php_snippets/"
  "$SRCROOT/./webs/web2/php/"
 )

opts=(
  -av
  --relative
  $DRYRUN
 )

rsync "${opts[@]}" "${include[@]}" "$DSTROOT"

[toc] | [prev] | [next] | [standalone]

#8579

Dave Gibson wrote, on 2013-06-20:
> [ Followup-To: set to colm ]

[Noted, but I have un-set the followup, as I don't see any harm in 
continuing the thread in all of the small number of groups that it was 
posted to. The original crosspost was to try to maximise replies in 
case some people only regularly read one group or the other, and I 
believe that _limited_ sensible crossposting is not against the Spirit 
of Usenet [1] :-)  Apologies if we differ on this!]

[1] In true usenet fashion, an off-topic debate on this matter will then
ensue.. ;-D 



> In comp.os.linux.misc, David <david@55952163-3189045.bogus.domain.invalid>
>> wrote:
>> I've been struggling with rsync (and its helpfully voluminous, but
>> somewhat confusing, man page) all day.. :-(
>
>> explicitly allow only the folder hierarchies that are supposed to be 
>> synced.
>
> You want to rsync a few directories from a hierarchy while retaining
> the directory structure?

Basically, yes. :-)
If we incorporate further files into our repository in the future, then
there may be a few further folders that also are to be added to the list
of what-to-be-synced..



>> SRCROOT="/data/import/svn-test/svn-checkedout/$REPONAME/data"
>> #SRCROOT="/data/import/svn-holding/svn-checkedout/$REPONAME/data"
>> 
>> DSTROOT="/data/import/svn-test/data-test"
>> DSTROOT_LIVE="/data"
>
> [snip]
>
>> RS_INCLUDE="
>> --include='/webs/web1/php_includes/'
>> --include='/webs/web1/php_snippets/'
>> --include='/webs/web2/php/'
>> "
>
> [snip: building up a command line in a string]
>
> You have
>
>  "$SRCROOT"/webs/web1/php_includes/
>  "$SRCROOT"/webs/web1/php_snippets/
>  "$SRCROOT"/webs/web1/irrelevant_things
>  "$SRCROOT"/webs/web2/php/
>  "$SRCROOT"/webs/web2/something_to_be_ignored
>  "$SRCROOT"/webs/other/stuff
>
> You want
>
>  "$DSTROOT"/webs/web1/php_includes/
>  "$DSTROOT"/webs/web1/php_snippets/
>  "$DSTROOT"/webs/web2/php/


Sort of :-)

There should (for now), in $SRCROOT, only exist the three folders listed 
immediately above, and I want to be sure to sync *only* those into $DSTROOT.

There may (probably unlikely, but possibly), inadvertently, be other 
files/folders that have found their way into $SRCROOT, and if so, these must
somehow expressly *not* be synced.

However, in the future, there may/will also be (eg)
"$SRCROOT"/webs/web2/something_to_be_ignored/something_to_be_synced/

and so something_to_be_synced/ should be synced across, without "damaging"
anything else further "up" in "$DSTROOT"/webs/web2/something_to_be_ignored/


> Right?  Use -R (note use of '/./' to strip preceding path elements).
>
> rsync -av -R                            \
>   "$SRCROOT/./webs/web1/php_includes/"  \
>   "$SRCROOT/./webs/web1/php_snippets/"  \
>   "$SRCROOT/./webs/web2/php/"           \
>   "$DSTROOT"

I have to say, I didn't really understand the section of the manual about
-R!! My previous uses of rsync have been rather simpler, and I didn't 
need to worry about it, one way or another.. :-(


> Referring back to the attempt to have embedded quotes within a string,
> when the shell expands RS_INCLUDE it will treat the embedded single
> quotes as ordinary characters.  To have the behaviour you were aiming
> for 

..I'm not quite sure what I was aiming for! ;-/
I was getting thoroughly confused as to where a wildcard character would 
be being interpreted by rsync, and where by the shell. Not knowing how 
the respective command interpreters really do their stuff, I am probably 
guilty of trying to do something too clever for my own good.. :-(


> (are there spaces in there?) you'd have to use eval.  Don't use
> eval.  Use arrays.

There shouldn't be any spaces in the file paths that would be being 
specified explicitly in the lists of what to sync/not-sync (and 
hopefully not in any of the actual files in the repository, assuming 
sensible naming conventions by all of the handful of developers), but 
you are right that we should consider the possibility..


>
>   RS_INCLUDE=(
>     --include='/webs/we 1/php includes/'
>     --include='/webs/we 1/php snippets/'
>     --include='/webs/web2/php/'
>   )
>   printf '%s\n' "${RS_INCLUDE[@]}"
>
> Contrast the output from that with the following
>
>   RS_INCLUDE="
>     --include='/webs/we 1/php includes/'
>     --include='/webs/we 1/php snippets/'
>     --include='/webs/web2/php/'
>   "
>   printf '%s\n' $RS_INCLUDE


[goes away and tries the exercise..]

Thanks. I wasn't aware of shell arrays. Is this bash-specific or
applicable to sh as well? (I try to avoid bashisms just in case I should
ever need to write something portable (no, the need hasn't actually
arisen yet..))

So, if I understand that correctly, each line inside the ( ..... ) is
a separate entry in the array?



Thanks,


David.

[toc] | [prev] | [next] | [standalone]

#8581

David <david@55952163-3189045.bogus.domain.invalid> writes:
> Thanks. I wasn't aware of shell arrays. Is this bash-specific or
> applicable to sh as well? (I try to avoid bashisms just in case I should
> ever need to write something portable (no, the need hasn't actually
> arisen yet..))

They’re not part of POSIX sh.

-- 
http://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#8589

[ Followup-To: set to colm ]

In comp.os.linux.misc, David
<david@55952163-3189045.bogus.domain.invalid> wrote:
> Dave Gibson wrote, on 2013-06-20:
>> [ Followup-To: set to colm ]
> 
> [Noted, but I have un-set the followup,

Tin's quite naggy about setting the header when cross-posting.  I've
done it again, by the way.  Feel free to ignore it.

>>> --include='/webs/web1/php_includes/'
>>> --include='/webs/web1/php_snippets/'
>>> --include='/webs/web2/php/'

[ rsync -a --include=... --include=... src tgt ]

>> You have
>>
>>  "$SRCROOT"/webs/web1/php_includes/
>>  "$SRCROOT"/webs/web1/php_snippets/
>>  "$SRCROOT"/webs/web1/irrelevant_things
>>  "$SRCROOT"/webs/web2/php/
>>  "$SRCROOT"/webs/web2/something_to_be_ignored
>>  "$SRCROOT"/webs/other/stuff
>>
>> You want
>>
>>  "$DSTROOT"/webs/web1/php_includes/
>>  "$DSTROOT"/webs/web1/php_snippets/
>>  "$DSTROOT"/webs/web2/php/
> 
> 
> Sort of :-)
> 
> There should (for now), in $SRCROOT, only exist the three folders
> listed immediately above, and I want to be sure to sync *only* those
> into $DSTROOT.
>
> There may (probably unlikely, but possibly), inadvertently, be other
> files/folders that have found their way into $SRCROOT, and if so,
> these must somehow expressly *not* be synced.

If they're not named on rsync's command line they won't be synced.

My suggestion is to drop the use of --include and --exclude and to
specify the "include" subdirectories as sources to be sync'd (rather
than the whole $SRCROOT).

This

  rsync -a --include=/want/this --include=/and/thisone $SRCROOT/ $DSTROOT

Becomes

  rsync -a -R $SRCROOT/./want/this $SRCROOT/./and/thisone $DSTROOT


> I have to say, I didn't really understand the section of the manual
> about -R!! My previous uses of rsync have been rather simpler, and
> I didn't need to worry about it, one way or another.. :-(

When -R is used rsync will include the source's leading path in the
target rather than just the basename.

  rsync -a /a/b/c foo

copies "/a/b/c" to "foo/c"

  rsync -a -R /a/b/c foo

copies "/a/b/c" to "foo/a/b/c"

Using a './' in a source name tells rsync to retain only from that point
forward in the target name.

  rsync -R /a/./b/c foo

copies "/a/b/c" to "foo/b/c"

[snip]

>>   RS_INCLUDE=(
>>     --include='/webs/we 1/php includes/'
>>     --include='/webs/we 1/php snippets/'
>>     --include='/webs/web2/php/'
>>   )
>>   printf '%s\n' "${RS_INCLUDE[@]}"

> Thanks. I wasn't aware of shell arrays. Is this bash-specific or
> applicable to sh as well?

Not sh.  Bash, ksh93, zsh and mksh.  Possibly also openbsd's ksh.

> So, if I understand that correctly, each line inside the ( ..... ) is
> a separate entry in the array?

Each distinct string becomes an entry:

  a=( "first entry" "second entry"
      third )

  a[0]="first entry"
  a[1]="second entry"
  a[2]="third"

[toc] | [prev] | [next] | [standalone]

#8526

On 2013-06-19, David <david@55952163-3189045.bogus.domain.invalid> wrote:
> I've been struggling with rsync (and its helpfully voluminous, but somewhat
> confusing, man page) all day.. :-(
>
> ...
>
> # /name = anchor the name to the root of the transfer hierarchy
> RS_INCLUDE="
> --include='/webs/web1/php_includes/'
> --include='/webs/web1/php_snippets/'
> --include='/webs/web2/php/'
> "
>
> RS_EXCLUDE="
> --exclude='*'
> "

Rikishi42's suggestion of using the --include-from optiom is a good one.
It makes more sense to me to use that rather than multiple --include
options.

A cursory examination of your script (and I am by no means an expert)
indicates to me that you want to include everything in those three
directories and nothing else.  If so, why bother to use the --exclude
option?  You would then not have to worry about whether --exclude takes
precedence over --include, or if their order on the command line makes a
difference.
-- 
                                 Chick Tower

For e-mail:  colm DOT sent DOT towerboy AT xoxy DOT net

[toc] | [prev] | [next] | [standalone]

#8597

[ Followup-To set ]

In comp.os.linux.misc, David
> <david@55952163-3189045.bogus.domain.invalid> wrote:

> I am trying to write a shell script to use rsync to sync certain files to 
> another location on the same computer. (The files are checked out from a
> subversion repository into the source location and need to be transferred 
> to their final location, and have ownership and permission changes.)

> My script is included below. Any help in trying to diagnose what's wrong
> with it gratefully received! (And if you want to pick it apart for 
> inelegance too, feel free..)

> #!/bin/sh
> #
> # rsync-webs: rsync the checked-out svn files into the webserver files area
> 
> 
> ############################################################
> # Config
> ############################################################
> 
> # The repository being used
> REPONAME=web-test
> #REPONAME=web
> 
> 
> # Make sure src & dest "match" (ie, both paths end in 'data')
> SRCROOT="/data/import/svn-test/svn-checkedout/$REPONAME/data"
> #SRCROOT="/data/import/svn-holding/svn-checkedout/$REPONAME/data"
> 
> DSTROOT="/data/import/svn-test/data-test"
> DSTROOT_LIVE="/data"
> 
> 
> # DRYRUN is the rsync option for running in dryrun mode 
> # (no files actually transferred)
> DRYRUN="-n"
> MODE='dryrun'

[snip: --include=... --exclude=... ]

> # rsync options:

> RS_OPTS="-vviaz --no-perms --chmod=u=rx,g=rx,o-rwx
> $RS_INCLUDE
> $RS_EXCLUDE
> --cvs-exclude
> "

> if [ "$USER" != "root" ]; then
>    echo 'You need to run this script as root, exiting..'
>    exit
> fi

Use id -u rather than testing $USER.

> # Add group perms to the src files first 
> # (so that they can be svn update'd later by another dev)
> echo 'Adding group permissions to source files..'
> chmod -R g+rwx "$SRCROOT"

The comment gives the impression that the hierarchy having the
correct/desired permissions is a side-effect of this script.

If that's the case, the modes should be set in a separate script.

> 
> # chown the src files first 
> # (so that the dest files will have the correct u+g ownership)
> echo 'chowning source files with required ownership for destination..'
> chown -R root:apache "$SRCROOT"

Use newgrp to change root's primary group to apache, then run rsync
with --no-g.

> 
> 
> # run rsync in dry-run mode (no changes actually made) UNLESS 
> # 'test' or 'commit' option explicitly specified
> 
> # test = copy to test destination
> if [ "$1" == "-test" ]; then
>    unset DRYRUN
>    MODE='test'
> fi
> 
> # commit = copy to real destination
> if [ "$1" == "-commit" ]; then
>    unset DRYRUN
>    DSTROOT="$DSTROOT_LIVE"
>    MODE='commit'
> fi

"==" is (currently) non-standard.

When testing a single parameter against several strings, use case.

> 
> 
> # Set up the rsync command
> # src/ = trailing slash means sync *contents* of src folder 
> # (do not create 'src' folder on dest)
> RS_CMD="rsync $DRYRUN  $RS_OPTS  $SRCROOT/  $DSTROOT"

That approach often creates problems related to word splitting, quoting
and filename generation.

> 
> 
> 
> # Sync the files..
> echo '** Running in mode: ' "$MODE"
> echo '** (specify: -test to sync to test dir, -commit to sync to live dir).'
> 
> echo 'Source:'  $SRCROOT
> echo '  Dest:'  $DSTROOT
> echo 'rsync command:'
> echo $RS_CMD
> 
> echo
> echo '** OK to sync files? [yN]'
> read CONFIRM
> if [ "$CONFIRM" == "y" ]; then
>    echo 'Syncing files..'
>    $RS_CMD
> fi
> 
> 
> echo 'chgrping source files back again..'
> chgrp -R staff "$SRCROOT"
> echo 'End of script.'

You're dealing with the sort of task often run as a cron job.  The
interactive confirmation makes the current script unsuitable for
such use.


I *think* the following script does more or less what you want but
consider it untested.  You have known-good backups, right?



#! /bin/sh -

if [ `id -u` -ne 0 ]; then
  echo "$0: administrator privileges required." 1>&2
  exit 1
fi

REPONAME=web-test
#REPONAME=web

SRCROOT="/data/import/svn-test/svn-checkedout/$REPONAME/data"
#SRCROOT="/data/import/svn-holding/svn-checkedout/$REPONAME/data"

DSTROOT="/data/import/svn-test/data-test"
DSTROOT_LIVE="/data"

# DRYRUN is the rsync option for running in dryrun mode 
DRYRUN="-n"
MODE='dryrun'

synclist="
$SRCROOT/./webs/web1/php_includes
$SRCROOT/./webs/web1/php_snippets
$SRCROOT/./webs/web2/php
"

# destination group
dgrp=apache

debugging=no

# "yes" to prompt for confirmation.
pause=no

opts="
-vviaz
-R
--no-g
--no-p
--chmod=u=rx,g=rx,o-rwx
--cvs-exclude
"

while [ $# -gt 0 ]; do
  case $1 in
    -test )    unset DRYRUN ; MODE=test ;;
    -commit )  unset DRYRUN ; MODE=commit ; DSTROOT=$DSTROOT_LIVE ;;
    -debug )   debugging=yes ;;
    -auto )    pause=no ;;
    --help )   echo '***FIXME*** help not implemented' ; exit 0 ;;
    -- )       shift ; break ;;
    * )        break ;;
  esac
  shift
done

cmd="
rsync
$DRYRUN
$opts
$synclist
$DSTROOT
"

# Restrict word splitting and disable globbing.
# There is one newline between the single quotes -- "IFS='<newline>'"
IFS='
'
set -f

if [ "$debugging" = yes ] || [ "$pause" = yes ]; then
  printf '** Running in mode: %s\n' "$MODE"
  printf 'Source: %s\n  Dest: %s\n' "$SRCROOT" "$DSTROOT"
  printf 'rsync command: '
  printf ' %s' $cmd
  printf '\n'
fi

if [ "$pause" = yes ]; then
  printf '\n** OK to sync files? [yN] '
  read CONFIRM
  case $CONFIRM in
    [yY] | [yY][eE][sS] ) true ;;
    * )                   exit 0 ;;
  esac
fi

[ -n "$dgrp" ] && newgrp "$dgrp"

$cmd

[toc] | [prev] | [next] | [standalone]

#8601

In comp.os.linux.misc, Dave Gibson
<dave.gma+news002@googlemail.com.invalid> wrote:
> [ Followup-To set ]
> 
> In comp.os.linux.misc, David
>> <david@55952163-3189045.bogus.domain.invalid> wrote:
> 
>> I am trying to write a shell script to use rsync to sync certain files to 
>> another location on the same computer. (The files are checked out from a
>> subversion repository into the source location and need to be transferred 
>> to their final location, and have ownership and permission changes.)
> 
>> My script is included below. Any help in trying to diagnose what's wrong
>> with it gratefully received! (And if you want to pick it apart for 
>> inelegance too, feel free..)


>> # chown the src files first 
>> # (so that the dest files will have the correct u+g ownership)
>> echo 'chowning source files with required ownership for destination..'
>> chown -R root:apache "$SRCROOT"
> 
> Use newgrp to change root's primary group to apache, then run rsync
> with --no-g.

Bugger.  I was confusing sg with newgrp and getting them both wrong.
newgrp runs a shell so can't be used within the script and while sg
can run a command as a different group it accepts only a single argument
with -c so arguments with embedded spaces need special quoting which
pretty much puts you back to square one.

With newgrp:

  newgrp apache
  /path/to/script -commit
  exit

With sg:

  sg apache -c "/path/to/script -commit"


One possibility would be to use a wrapper script to run the main script
(which doesn't need to accept arguments with embedded spaces) with the
appropriate group.

#! /bin/sh -
mainscript=/path/to/script
if [ `id -u` -ne 0 ]; then
  echo "$0: administrator privileges required." 1>&2
  exit 1
fi
if [ ! -x "$mainscript" ]; then
  echo "$0: missing script: $mainscript" 1>&2
  exit 1
fi
if [ "`id -gn`" = apache ]; then
  "$mainscript" "$@"
else
  sg apache -c "$mainscript $*"
fi



> I *think* the following script does more or less what you want but
> consider it untested.

Well, you were warned.  =8-0

> #! /bin/sh -

> [ -n "$dgrp" ] && newgrp "$dgrp"

Remove that line, it's completely wrong.

Funny how things come back to you just after posting.

[toc] | [prev] | [next] | [standalone]

#8640

Dave Gibson wrote, on 2013-06-25:
> In comp.os.linux.misc, David
>> <david@55952163-3189045.bogus.domain.invalid> wrote:
>
>> if [ "$USER" != "root" ]; then
>>    echo 'You need to run this script as root, exiting..'
>>    exit
>> fi
>
> Use id -u rather than testing $USER.

Thanks for the tip, I didn't know about the 'id' command.
I'm guessing that this might be because the $USER variable could be faked?


>> # Add group perms to the src files first 
>> # (so that they can be svn update'd later by another dev)
>> echo 'Adding group permissions to source files..'
>> chmod -R g+rwx "$SRCROOT"
>
> The comment gives the impression that the hierarchy having the
> correct/desired permissions is a side-effect of this script.
>
> If that's the case, the modes should be set in a separate script.

The reason for that line was that when the files are checked out of the
repo (by an ordinary user), the user's umask doesn't give them group
permissions, and it might be another developer who does the next 
'svn up', so they need to be able to over-write the files.

You're right that it's a sort of separate activity from actually
syncing the files from where they are checked out, to their final
location, though.


>> # chown the src files first 
>> # (so that the dest files will have the correct u+g ownership)
>> echo 'chowning source files with required ownership for destination..'
>> chown -R root:apache "$SRCROOT"
>
> Use newgrp to change root's primary group to apache, then run rsync
> with --no-g.

I wasn't aware of the 'newgrp' command either(!).

I tried your revised script, but I think I have stumbled at this point.

Nothing gets synced when using newgrp, but does if I comment that out 
(ie, not over-riding(?) the existing root group permissions).

The source part of the filesystem (where the files were checked out to)
isn't readable by apache (I guess we'd need to make a new group 
(currently staff) and add apache and the developer users to it?), and
similarly, the $DSTROOT of the destination (...../data/ isn't, for
legitimate/essential reasons, writable by apache either (only certain
folders within it).

It's not entirely clear from the manual what newgrp does exactly, but 
if it gives the user *only* the permissions that the apache group has 
(rather than *adding* (as it were?) to the existing permissions), then
that might not work as a solution in this case.


>> if [ "$1" == "-commit" ]; then
>
> "==" is (currently) non-standard.

oops, too much of a rush-job, and thinking in another language ;-(

> When testing a single parameter against several strings, use case.

Agreed! I went for the quick and dirty copy/paste since I was only
checking a couple of values. But you're right, it could easily become
more and then 'if' would be inappropriate.


>> # Set up the rsync command
>> # src/ = trailing slash means sync *contents* of src folder 
>> # (do not create 'src' folder on dest)
>> RS_CMD="rsync $DRYRUN  $RS_OPTS  $SRCROOT/  $DSTROOT"
>
> That approach often creates problems related to word splitting, quoting
> and filename generation.

Yes, I suspect that may have been part of what got me tangled up..!


> You're dealing with the sort of task often run as a cron job.  The
> interactive confirmation makes the current script unsuitable for
> such use.

I made it intentionally interactive as the checking the files out of the
repository would also be interactive (ie, the developer would have to 
decide that they would be ready to deploy), and so wanted an "Are you sure"
as a sanity check or in case of butterfingers.. :-)


> I *think* the following script does more or less what you want but
> consider it untested.  You have known-good backups, right?

Yup. And I'm testing all this on the dev (not live) server first, and
into a test 'destination' first for extra paranoia :-)

Thanks for taking the time to review my script, it's genuinely much
appreciated.

> #! /bin/sh -

Is the "-" this?

 -      Expands  to  the  current option flags as specified upon invoca‐
        tion, by the set builtin command, or  those  set  by  the  shell
        itself (such as the -i option).

I'm afraid that man page excerpt doesn't really leave me any wiser as to
what that means?


> # DRYRUN is the rsync option for running in dryrun mode 
> DRYRUN="-n"
> MODE='dryrun'
>
> synclist="
> $SRCROOT/./webs/web1/php_includes
> $SRCROOT/./webs/web1/php_snippets
> $SRCROOT/./webs/web2/php
> "

I'm just idly curious why you have chosen to name some variables in 
upper case and some in lower case? Personal choice/preference, of 
course, but I would normally (not that I would claim to be a real 
shell expert) use upper case, having picked that up as a convention 
from somewhere..


> # Restrict word splitting and disable globbing.
> # There is one newline between the single quotes -- "IFS='<newline>'"
> IFS='
> '
> set -f

Ah, this is clearly what I needed to know to avoid tangling myself up
with filenames and wildcards, etc..

[Looks at manual..]

Could you maybe explain what the reason is for setting IFS to 
<newline> *only* (rather than the default)?

..and 'set -f' "[d]isable[s] pathname expansion", so that's what prevents
wildcards being interpreted?


> [ -n "$dgrp" ] && newgrp "$dgrp"
>
> $cmd

Thanks, your revised script (at least on initial testing!) seems to
work other than the 'newgrp' aspect. I suspect I may have to go back
to my less elegant chgrp "solution" because of the restrictions on 
the server as to what apache can read/write?


Again, many thanks for taking the time to help,


Cheers,


David.

[toc] | [prev] | [next] | [standalone]

#8645

In comp.os.linux.misc, David
 <david@55952163-3189045.bogus.domain.invalid> wrote:
> Dave Gibson wrote, on 2013-06-25:
>> In comp.os.linux.misc, David
>>> <david@55952163-3189045.bogus.domain.invalid> wrote:

>> Use id -u rather than testing $USER.
> 
> Thanks for the tip, I didn't know about the 'id' command.
> I'm guessing that this might be because the $USER variable could be
> faked?

Yes.

>>> # chown the src files first 
>>> # (so that the dest files will have the correct u+g ownership)
>>> echo 'chowning source files with required ownership for destination..'
>>> chown -R root:apache "$SRCROOT"
>>
>> Use newgrp to change root's primary group to apache, then run rsync
>> with --no-g.
> 
> I wasn't aware of the 'newgrp' command either(!).
> 
> I tried your revised script, but I think I have stumbled at this point.

Ah.  You found that banana skin I dropped.

> 
> Nothing gets synced when using newgrp, but does if I comment that out 
> (ie, not over-riding(?) the existing root group permissions).

Humble pie from comp.os.linux.misc
(Message-ID: <omqp9axbq2.ln2@perseus.wenlock-data.co.uk>)

------------------------------------------------------------------------
Bugger.  I was confusing sg with newgrp and getting them both wrong.
newgrp runs a shell so can't be used within the script and while sg
can run a command as a different group it accepts only a single argument
with -c so arguments with embedded spaces need special quoting which
pretty much puts you back to square one.

With newgrp:

  newgrp apache
  /path/to/script -commit
  exit

With sg:

  sg apache -c "/path/to/script -commit"


One possibility would be to use a wrapper script to run the main script
(which doesn't need to accept arguments with embedded spaces) with the
appropriate group.

#! /bin/sh -
mainscript=/path/to/script
if [ `id -u` -ne 0 ]; then
  echo "$0: administrator privileges required." 1>&2
  exit 1
fi
if [ ! -x "$mainscript" ]; then
  echo "$0: missing script: $mainscript" 1>&2
  exit 1
fi
if [ "`id -gn`" = apache ]; then
  "$mainscript" "$@"
else
  sg apache -c "$mainscript $*"
fi
------------------------------------------------------------------------


> 
> The source part of the filesystem (where the files were checked out to)
> isn't readable by apache (I guess we'd need to make a new group 
> (currently staff) and add apache and the developer users to it?),
> and similarly, the $DSTROOT of the destination (...../data/ isn't, for
> legitimate/essential reasons, writable by apache either (only certain
> folders within it).

As long as the user running the script is able to access the hierarchies
a change of primary group won't matter.

> 
> It's not entirely clear from the manual what newgrp does exactly, but 
> if it gives the user *only* the permissions that the apache group has 
> (rather than *adding* (as it were?) to the existing permissions), then
> that might not work as a solution in this case.

Added.

Users are often members of several groups with one of those groups
being the "current" group ID.  newgrp changes the current group ID but
the user remains a member of the other groups.

  id -gn
  id -Gn

>> #! /bin/sh -
> 
> Is the "-" this?
> 
> -      Expands  to  the  current option flags as specified upon invoca?
>        tion, by the set builtin command, or  those  set  by  the  shell
>        itself (such as the -i option).

That's referring to the expansion of "$-"

 $ echo "$-"
 himBH

See the set builtin section:

          -       Signal  the  end of options, cause all remaining args to
                  be assigned to the positional parameters.  The -x and -v
                  options are turned off.  If there are no args, the posi-
                  tional parameters remain unchanged.

It's roughly equivalent to:

  #! /bin/sh +vx -- "$@"

>> # DRYRUN is the rsync option for running in dryrun mode 
>> DRYRUN="-n"
>> MODE='dryrun'
>>
>> synclist="
>> $SRCROOT/./webs/web1/php_includes
>> $SRCROOT/./webs/web1/php_snippets
>> $SRCROOT/./webs/web2/php
>> "
> 
> I'm just idly curious why you have chosen to name some variables in 
> upper case

Yanked from the original script.

> and some in lower case?

allcaps aversion disorder.

> Personal choice/preference, of 
> course, but I would normally (not that I would claim to be a real 
> shell expert) use upper case, having picked that up as a convention 
> from somewhere..

I once spent FAR TOO LONG trying to figure out why /usr/bin/gzip would
compress itself whenever a particular script was run.  It turned out
that in the script $GZIP was used to refer to a compressor:

  GZIP=/usr/bin/gzip

then

  "$GZIP" /path/to/tarball

It's a good idea in general -- edit one line to change the compressor
everywhere in the script.

However, gzip examines its environment and essentially treats $GZIP as
an extension to the command line.  /usr/bin/gzip isn't a gzip option so
was taken as the name of a file to compress.

For extra annoyance, although GZIP wasn't exported in the script, it was
present in (and exported from) the script's parent's environment which
marked it for export within the script and thus visible to gzip.

>> # Restrict word splitting and disable globbing.
>> # There is one newline between the single quotes -- "IFS='<newline>'"
>> IFS='
>> '
>> set -f
> 
> Ah, this is clearly what I needed to know to avoid tangling myself up
> with filenames and wildcards, etc..
> 
> [Looks at manual..]
> 
> Could you maybe explain what the reason is for setting IFS to 
> <newline> *only* (rather than the default)?

When the shell expands an unquoted variable it splits the expansion
into individual fields ("words") delimited by any characters in $IFS.

Omitting the space and tab from $IFS allows fields to contain those
characters.

$ foo='how many
fields here'
$ printf '%s\n' $foo
how
many
fields
here
$ IFS='
'
$ printf '%s\n' $foo
how many 
fields here

> ..and 'set -f' "[d]isable[s] pathname expansion", so that's what prevents
> wildcards being interpreted?

Yes.  After word-splitting, each word is checked for wildcard characters,
if any are present the shell treats the word as a pattern to be replaced
with a list of matching filenames.

  foo=*
  echo $foo

> 
> 
>> [ -n "$dgrp" ] && newgrp "$dgrp"

It doesn't matter how many times I see it, that damn line still isn't
getting any less wrong.

>>
>> $cmd

[toc] | [prev] | [next] | [standalone]

#8673

In uk.comp.os.linux David <david@55952163-3189045.bogus.domain.invalid> wrote:
> Thanks for the tip, I didn't know about the 'id' command.
> I'm guessing that this might be because the $USER variable could be faked?

Yes. But this isn't really a security issue because the remainder of
the script cannot work without those root permissions anyway.

Chris

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.misc

csiph-web