Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scripting question: how to use quotation marks inside a script
#1
Hello!

I'm working on my custom backup script with rsync, and I have stumbled upon something that is not obvious for me (I'm quite new to bash, so forgive me if I'm wrong here).

Before doing the sync, I define the options and other things in variables:

Code:
OPTS="-aAXi"
EXCLUDE='--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/*"}'
SRC="/"
BCKP="/media/data/backup"
LINK="--link-dest=$BCKP/$LAST/"
When it is done, I try to invoke the command by
Code:
rsync $OPTS $EXCLUDE $LINK $SRC $BCKP/$NAME/ >> "$LOG"
Which interestingly doesn't respect my --exclude array. NB, I also print the command into a file with
Code:
echo "Running: rsync $OPTS $EXCLUDE $LINK $SRC $BCKP/$NAME/" >> "$LOG"
and in that case, the string that's written to the log file is a valid rsync command, which respects the exclude array (tried it and ran it).

My strong suspicion is that there's some trickery regarding the " quotation marks I am not familiar with, as if I change the variables to
Code:
SRC="\"/\""
to contained escaped quotation marks, the whole rsync command falls apart, and starts reporting missing directories, while the rsync command written to the log file with the quotation marks present, work properly.

Anyone more familiar with bash, could you please give me a resource where to look? Thanks.
My top 10 reasons to still use Arch after 2 months on my main PC at home.
Reply
#2
(10-05-2019, 04:30 AM)TarsolyGer Wrote:
Code:
rsync $OPTS $EXCLUDE $LINK $SRC $BCKP/$NAME/ >> "$LOG"

The first thing I'd try it putting quotatin marks around the variables, like this:

Code:
rsync "$OPTS" "$EXCLUDE" "$LINK" "$SRC" "$BCKP/$NAME/" >> "$LOG"

Now you should be able to escape the quotation marks inside the variables.
Reply
#3
(10-05-2019, 10:09 PM)leon.p Wrote: The first thing I'd try it putting quotatin marks around the variables, like this:

Code:
rsync "$OPTS" "$EXCLUDE" "$LINK" "$SRC" "$BCKP/$NAME/" >> "$LOG"

Now you should be able to escape the quotation marks inside the variables.

What I have found that in case of the exclude={"/dev/*","/proc/*",...} case it's not rsync that understands this expression, but it's a bash curly braces expansion o what, that passes exclude=/dev/* exclude=/proc/* etc to rsync, and if I put it inside a variable, it doesn't get expanded, so that's why rsync doesn't know what to do with it.

Code:
VAR=exclude={"/dev/*","/proc/*"}
echo $VAR             #exclude={"/dev/*","/proc/*"}

VAR=$(echo exclude={"/dev/*","/proc/*"})
echo $VAR             #exclude=/dev/* exclude=/proc/*

Is there any more elegant way to do this?

In the other places, it seems like I shouldn't use escaped quotation marks at all in the variables as they will be sent to the command as literal quotation marks, so rsync will think those are part of the filename. Quoting around the variable names also has some weird behavior, but I haven't got around testing it to detail yet.


Thanks for the help!
My top 10 reasons to still use Arch after 2 months on my main PC at home.
Reply
#4
So I have ran into this:


Code:
EXCLUDE=(--exclude={"Some Files","Shared files"})  #makes an array with two elements, both containing spaces
echo ${EXCLUDE[@]}                                #prints all the elements separated by spaces
                                                 #--exclude=Some Files --exclude=Shared files
rsync $OPTS ${EXCLUDE[@]} $LINK $SRC $BCKP/$NAME/ #fails as the spaces inside the folder names stop being escaped and the second words are treated as separate arguments by rsync after that.

Provided I don't have any spaces inside the exclude array, this works, but I have no idea how to force bash to interpret the elements of the array as their own parameters. If I do ${EXCLUDE[0]} ${EXCLUDE[1]}, that also fails to use the whole variable as parameter to rsync. If I introduce quotation marks escaped inside the quotation marks around my folder names  
Code:
(--exclude={"\"Some Files\"","\"Shared files\""})
this on the other hand, makes the escaped quotation marks literal, meaning, when passed as an argument to rsync, rsync will fail to find the files with the quotation marks in the file name.

Does anyone have any idea what's going on?
My top 10 reasons to still use Arch after 2 months on my main PC at home.
Reply
#5
Okay, for future reference, if you have trouble with quoting stuff passing as parameters when you do curly braces expansion:


Code:
OPTS="-aAXi"
EXCLUDE=({"/Directory One/*","/Shared files/*"})  # Curly braces expansion into an array of folders
SRC="/home/username/Documents/"
BCKP="/media/bkp/Documents"
LOG="$BCKP/logs/$NAME.log" # some variables I define in other places of the script
LINK="--link-dest=$BCKP/$LAST/"

rsync $OPTS "${EXCLUDE[@]/#/--exclude=}" "$LINK" "$SRC" "$BCKP/$NAME/"


This will expand the EXCLUDE array based on the pattern I found here:



Code:
--exclude=/Directory One/* --exclude=/Shared files/*


And due to the whole stuff being inside the quotation marks, for some reason, each expanded element of the array will be passed as an individual parameter, not affected by white spaces.
My top 10 reasons to still use Arch after 2 months on my main PC at home.
Reply
#6
Joe wrote XBT I use it all the time. Try snooping at his code mabee ?
Dos 3.2 to Win 10.
Main - Cinnamon 19.2
   Resistance Is Not Futile!
       It's voltage divided by current
Reply
#7
(10-15-2019, 11:21 PM)rick forges Wrote: Joe wrote XBT  I use it all the time. Try snooping at his code mabee ?

Thanks Smile
I'm mostly trying to build my stuff up from the ground-up, it's more about learning bash than doing backups (I can do those manually in the meantime anyways), and what I'm trying to do is more like a replacement for timeshift to do system backups with hard links and stuff.
My top 10 reasons to still use Arch after 2 months on my main PC at home.
Reply
#8
(10-16-2019, 02:43 AM)TarsolyGer Wrote: I'm mostly trying to build my stuff up from the ground-up...

I appreciate your approach.  By intimately knowing your backup or snapshot solutions,  you are more likely able to perform successful maintenance on your backup/snapshot storage media as well as perform successful restoration tests.

I am curious why you chose to make your script more complex using the array to create the exclusion options.   The rsync utility will allow you to specify an exclusion file as an argument.  You can quickly add your exclusions to a file and use one exclusion option.  Also, this allows you to re-use the same script by re-configuring source, destination, link-dest, and exclusion_file parameters.  This is just a friendly suggestion, and I am not trying to be argumentative concerning your script. 

Also, the bash style is much more free form than if you are accustom to stanza-based or tab centric languages.  You can easily extend the rsync utility options on many lines to make it very readable.   In bash the line continuation syntax is the "\" character. 

This is an example of a more readable but lengthy rsync utility command in a script.


                rsync  -aHhh"${Verbose}${Dry_Run}" \
                        --stats \
                        --delete \
                        --exclude-from="${Exclusions_File}" \
                        --numeric-ids \
                        --one-file-system \
                        --sparse \
                        --link-dest="../${Current_Snapshot}"  \
                        "${Source_Dir}" "${Temp_Dir}" \
                        > "${Hidden_Rsync_Log_File}"

                Rec="${?}"
Idea Give a person a fish, and you feed them for a day. Teach a person how to fish, and you feed them for a lifetime. ✝️ Proverbs 4:7 Wisdom is the principal thing; therefore get wisdom: and with all thy getting get understanding.
Reply
#9
Thanks for the suggestions, even though I had to squint my eyes hard to read your choice of font Big Grin

My rationale for using the array for the ignore list is just keep everything in the same file for portability reasons, mostly. I do not expect to change the exclude list, and this way it's always there. And also, it annoyed the hell out of me that I couldn't do it for first, so I couldn't get rest until I solved this Smile

The suggestion about breaking stuff up into lines is a good idea though, a lot of my places my script started having extra long lines.

Thanks a lot!
My top 10 reasons to still use Arch after 2 months on my main PC at home.
Reply
#10
(10-12-2019, 02:35 AM)TarsolyGer Wrote: And due to the whole stuff being inside the quotation marks, for some reason, each expanded element of the array will be passed as an individual parameter, not affected by white spaces.


I would encourage you to read the bash man page to get a better understanding of word splitting.  See the EXPANSION, PARAMETERS, Special Parameters, and Word Splitting topics.



EXPANSION

       Only brace expansion, word splitting, and pathname expansion can change
       the number of words of the expansion; other expansions expand a  single
       word  to a single word.  The only exceptions to this are the expansions
       of "$@" and "${name[@]}" as explained above (see PARAMETERS).

...

PARAMETERS
    Special Parameters

       @      Expands to the positional parameters, starting from  one.   When
              the  expansion  occurs  within  double  quotes,  each  parameter
              expands to a separate word.  That is, "$@" is equivalent to "$1"
              "$2"  ...   If the double-quoted expansion occurs within a word,
              the expansion of the first parameter is joined with  the  begin-
              ning  part  of  the original word, and the expansion of the last
              parameter is joined with the last part  of  the  original  word.
              When  there  are no positional parameters, "$@" and $@ expand to
              nothing (i.e., they are removed).

...

   Word Splitting
       The shell scans the results of parameter expansion,  command  substitu-
       tion,  and arithmetic expansion that did not occur within double quotes
       for word splitting.

       The shell treats each character of IFS as a delimiter, and  splits  the
       results of the other expansions into words on these characters.  If IFS
       is unset, or its value is exactly <space><tab><newline>,  the  default,
       then  any  sequence  of IFS characters serves to delimit words.



The following C program has the basis for argument processing used in most unix-like utilities, and it might help you debug passing complex arguments.


$ cat > display_args.c <<EoF


/* 
The display_args.c is a simple program to display each argument(s) supplied to
the program.  Also, it might be useful to demonstrate how a file name with
space(s) effects the  argv[ ] arguments. 
*/

#include <stdio.h> 


int main(int argc, char *argv[])
{
   int Rec = 0 ; // return for bash exit code

   puts("") ; 
   if (argc == 1)
   {
      fprintf(stderr, "ERROR: You did not supply an argument.\n") ;
      Rec = 1 ;
    }
    else
    {
       for ( int i = 0 ; i < argc ; i++ )
          printf("argv[%d]: %s\n", i, argv[i]) ;
    }

    puts("") ;
    return Rec ; // return as bash exit code (zero is success)

}

EoF



$ gcc display_args.c -o display_args  # compile the C program
$
$ chmod 755 display_args
$
$ ./display_args

ERROR: You did not supply an argument.

$
$ ./display_args my file.txt

argv[0]: ./display_args
argv[1]: my
argv[2]: file.txt

$ ./display_args "my file.txt"

argv[0]: ./display_args
argv[1]: my file.txt

 
$
$
$ EXCLUDE=({"/Directory One/*","/Shared files/*"})
$
$ declare -p EXCLUDE
declare -a EXCLUDE='([0]="/Directory One/*" [1]="/Shared files/*")'

$ ./display_args ${EXCLUDE[@]/#/--exclude=}

argv[0]: ./display_args
argv[1]: --exclude=/Directory
argv[2]: One/*
argv[3]: --exclude=/Shared
argv[4]: files/*


$ ./display_args "${EXCLUDE[@]/#/--exclude=}"

argv[0]: ./display_args
argv[1]: --exclude=/Directory One/*
argv[2]: --exclude=/Shared files/*


$ # This example is based on the bash default IFS.
$
$ echo -e "${IFS}\c" | od -cb
0000000       \t  \n                                                    
          040 011 012                                                    
0000003
$
$
Idea Give a person a fish, and you feed them for a day. Teach a person how to fish, and you feed them for a lifetime. ✝️ Proverbs 4:7 Wisdom is the principal thing; therefore get wisdom: and with all thy getting get understanding.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)