Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #8511

rsync: exclude all, then include only specified, wildcard problems..

Cross-posted to 2 groups.

Show all headers | View raw

I've been struggling with rsync (and its helpfully voluminous, but somewhat
confusing, man page) all day.. :-(

I am trying to write a shell script to use rsync to sync certain files to 
another location on the same computer. (The files are checked out from a
subversion repository into the source location and need to be transferred 
to their final location, and have ownership and permission changes.)

Just in case something has inadvertently found its way into the source
location that shouldn't have (eg, a file that was checked in in the wrong
place or some other human error), I want to essentially first of all 
exclude everything in the source folder from being synced, and then 
explicitly allow only the folder hierarchies that are supposed to be 
synced.

I'm having problems trying to work out what my exclude rule should be [1], 
and I think my problems are further compounded by trying to assemble my 
command in a shell script and having wildcards (*) misinterpreted 
somewhere, but I'm not sure what I need to do to ensure that they are
interpreted when they need to be, and passed through unscathed when not!

(I get different results when I run my script, versus running the command 
that is assembled in it, directly at the command line!)


[1] The more I re-read rsync's definitions of / and * in various places and 
quantities, the more confused I get.. ;-(  ]


My script is included below. Any help in trying to diagnose what's wrong
with it gratefully received! (And if you want to pick it apart for 
inelegance too, feel free..)

I've been juggling /s and *s all day and make no claims that what is in
the script below make any sense, that's just where they are at present!


Many thanks for any advice,

David.


::::


#!/bin/sh
#
# rsync-webs: rsync the checked-out svn files into the webserver files area


############################################################
# Config
############################################################

# The repository being used
REPONAME=web-test
#REPONAME=web


# Make sure src & dest "match" (ie, both paths end in 'data')
SRCROOT="/data/import/svn-test/svn-checkedout/$REPONAME/data"
#SRCROOT="/data/import/svn-holding/svn-checkedout/$REPONAME/data"

DSTROOT="/data/import/svn-test/data-test"
DSTROOT_LIVE="/data"


# DRYRUN is the rsync option for running in dryrun mode 
# (no files actually transferred)
DRYRUN="-n"
MODE='dryrun'


# Exclude *all* under "data", and include *only* the allowed folders/files 
# (just to be on the safe side and to prevent inadvertent overwrites,
# "shouldn't happen")
#
# Pattern rules: 
#   * = any path (but stops at slashes)
#   ** = anything (including slashes)
#   dir/*** = dir *and* everything in dir
#
# rsync incl/excl rules are highly confusing. The following web page helped:
# http://www.only10types.com/2012/03/rsync-only-specific-files-and.html
#
# Need to include first, *then* exclude, see:
# https://bugzilla.redhat.com/show_bug.cgi?id=10343
#
# /name = anchor the name to the root of the transfer hierarchy
RS_INCLUDE="
--include='/webs/web1/php_includes/'
--include='/webs/web1/php_snippets/'
--include='/webs/web2/php/'
"

RS_EXCLUDE="
--exclude='*'
"

RS_FILTER=" "
# How to use this:
# http://hintsforums.macworld.com/showthread.php?t=123523
# echo $RS_FILTER | rsync --exclude-from=-  (- = stdin)



# rsync options:
# v = verbose
# i = itemize-changes
# a = archive mode (-rlptgoD (no -H,-A,-X))
# z = use compression
#
# *First* matching pattern is used: ie, need to include first, *then* exclude, 
# see rsync manual, and:
# https://bugzilla.redhat.com/show_bug.cgi?id=10343
RS_OPTS="-vviaz --no-perms --chmod=u=rx,g=rx,o-rwx
$RS_INCLUDE
$RS_EXCLUDE
--cvs-exclude
"


############################################################
# Main
############################################################

if [ "$USER" != "root" ]; then
    echo 'You need to run this script as root, exiting..'
    exit
fi


# Add group perms to the src files first 
# (so that they can be svn update'd later by another dev)
echo 'Adding group permissions to source files..'
chmod -R g+rwx "$SRCROOT"

# chown the src files first 
# (so that the dest files will have the correct u+g ownership)
echo 'chowning source files with required ownership for destination..'
chown -R root:apache "$SRCROOT"


# run rsync in dry-run mode (no changes actually made) UNLESS 
# 'test' or 'commit' option explicitly specified

# test = copy to test destination
if [ "$1" == "-test" ]; then
    unset DRYRUN
    MODE='test'
fi

# commit = copy to real destination
if [ "$1" == "-commit" ]; then
    unset DRYRUN
    DSTROOT="$DSTROOT_LIVE"
    MODE='commit'
fi



# Set up the rsync command
# src/ = trailing slash means sync *contents* of src folder 
# (do not create 'src' folder on dest)
RS_CMD="rsync $DRYRUN  $RS_OPTS  $SRCROOT/  $DSTROOT"



# Sync the files..
echo '** Running in mode: ' "$MODE"
echo '** (specify: -test to sync to test dir, -commit to sync to live dir).'

echo 'Source:'  $SRCROOT
echo '  Dest:'  $DSTROOT
echo 'rsync command:'
echo $RS_CMD

echo
echo '** OK to sync files? [yN]'
read CONFIRM
if [ "$CONFIRM" == "y" ]; then
    echo 'Syncing files..'
    $RS_CMD
fi


echo 'chgrping source files back again..'
chgrp -R staff "$SRCROOT"
echo 'End of script.'

Back to comp.os.linux.misc | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-19 15:17 +0000
  Re: rsync: exclude all, then include only specified, wildcard problems.. Rikishi42 <skunkworks@rikishi42.net> - 2013-06-20 00:49 +0200
    Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 11:50 +0000
      Re: rsync: exclude all, then include only specified, wildcard problems.. Rikishi42 <skunkworks@rikishi42.net> - 2013-06-26 01:59 +0200
  Re: rsync: exclude all, then include only specified, wildcard problems.. Richard Kettlewell <rjk@greenend.org.uk> - 2013-06-20 10:16 +0100
    Re: rsync: exclude all, then include only specified, wildcard problems.. The Natural Philosopher <tnp@invalid.invalid> - 2013-06-20 14:19 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 11:17 +0000
  Re: rsync: exclude all, then include only specified, wildcard problems.. Chris Davies <chris-usenet@roaima.co.uk> - 2013-06-20 10:57 +0100
    Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 11:40 +0000
      Re: rsync: exclude all, then include only specified, wildcard problems.. Chris Davies <chris-usenet@roaima.co.uk> - 2013-06-25 21:00 +0100
  Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-20 18:37 +0100
    Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-25 14:00 +0000
      Re: rsync: exclude all, then include only specified, wildcard problems.. Richard Kettlewell <rjk@greenend.org.uk> - 2013-06-25 15:24 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-25 19:23 +0100
  Re: rsync: exclude all, then include only specified, wildcard problems.. Chick Tower <c.tower@deadspam.com> - 2013-06-20 18:04 +0000
  Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-25 23:37 +0100
    Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-26 01:54 +0100
    Re: rsync: exclude all, then include only specified, wildcard problems.. David <david@55952163-3189045.bogus.domain.invalid> - 2013-06-28 16:12 +0000
      Re: rsync: exclude all, then include only specified, wildcard problems.. dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2013-06-29 01:30 +0100
      Re: rsync: exclude all, then include only specified, wildcard problems.. Chris Davies <chris-usenet@roaima.co.uk> - 2013-07-01 09:51 +0100

csiph-web