Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Reading and writing to a file preserving formatting
#1
Hi,

This is my first post here, so please be kind if I make a faux pas Smile

I need to read a file into Bash that I'm using as a template, do some string substitution, and then write out the file to a different location, all while preserving the formatting. It's a plain text file and I only need to keep the whitespace, tabs, and newlines.

I've done some searching around and can accomplish most of what I need but seem to be unable to preserve the formatting when writing the file out again.

Is anyone able to show how to read and then write a file with formatting preserved?

I should then be able to achieve what I need if someone can help with the read and write process.

Many Thanks

Paul
Reply
#2
Hi phollyer, welcome to EzeeTalk. Don't worry, most of us are very kind most of the time.  Wink

I get the feeling, that you are trying to reinvent the wheel, making your problem seem much more complicated, than it really is.

A simple sequence, like
Code:
cat template | sed -e 's/string1/substitute1/' -e 's/string2/substitute2/' ... > newfile
should do absolutely nothing detrimental to your precious formatting.

Or is there any reason, you cannot use the fine tools given to you for free? If so, a little more information about what exactly your script is doing, would be nice.

HTH
Reply
#3
If I understand you correctly, you want to read the contents of a file, automatically edit it and then write it to another file.

This can easiely be accomplished with 'sed'.


Here is an example, reading from 'file_1', writing to 'file_2', while replacing "foo" with "bar":
Code:
sed "s,foo,bar,g" < file_1 > file_2



EDIT: Radolkin apparently posted at the same time as I did. I'll leave mine up anyway.
My website - My git repos

"Things are only impossible until they’re not." - Captain Jean-Luc Picard
Reply
#4
(08-18-2019, 03:09 PM)radolkin Wrote: I get the feeling, that you are trying to reinvent the wheel

Quite possibly.

I have a pretty solid (self taught) background with Ruby, Elixir and JS, so could use any of those to solve this, but after watching some of Joe's videos, which I found to be excellent, I ended up here. I've long needed to improve my bash skills, and Joe's videos have given me the impetus to do so, and I figured this problem would be a good place to start.

(08-18-2019, 03:09 PM)radolkin Wrote: If so, a little more information about what exactly your script is doing, would be nice.

Basically, I want to automate the creation of my development environment for new projects (be it Ruby or Elixir) with a single Bash command which takes just the app name on the CL and then builds my environment from there. Each project is created from the CL, so it seemed to make sense to use a custom bash command to create the project and then customise it as needed. Ordinarily, I would just create each project as is, and then customise by hand, but as each project will use the same basic tools and configuration, I've decided to try to automate the whole process, or as much of it as I can.

Each project utilises a bunch of tools that are installed on a per project basis, such as Cypress for E2E testing of web apps. Automating the installation of these additional tools is simple with bash. My underlying problem is that each project has it's own config files that detail DB configurations, for example, and application dependencies that need to be downloaded and compiled. It's the adjustment of these config files that I'm having problems with.

I need to maintain the whitespace chars in order for the config files to continue to be human readable so that I can adjust if need be as each project is developed further.

All of my projects use the same basic environment to start off with along with the same tools which need to be installed and configured. But this 'same basic environment' is not the standard environment that is created when beginning a new project from the CL.

My first approach was to use sed to stream the config files, make the string substitutions, and then write the output to file but I hit a few snags along the way. The main one being that I need to insert text into certain places of each config file, sometimes multiple lines of text, with predefined tab chars prior to the text.

So, for example, I might have to turn (1):
Code:
     {:phoenix, "~> 1.3.4"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_ecto, "~> 3.2"},
     {:postgrex, ">= 0.0.0"},
     {:phoenix_html, "~> 2.10"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0"}
into (2):
Code:
     {:phoenix, "~> 1.4.9"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_ecto, "~> 3.0"},
     {:phoenix_html, "~> 2.10"},
     {:mongo_ecto, "~> 0.2"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0"},
     {:plug_cowboy, "~> 1.0"},
     {:comeonin, "~> 4.1"},
     {:bcrypt_elixir, "~> 1.1"},
     {:canada, "~> 1.0.2"},
     {:canary, "1.1.1"},
     {:poison, "~> 2.1.0", override: true},
     {:httpoison, "~> 0.11.0", override: true},
     {:timex, "~> 3.1"},
     {:conform, "~> 2.5"},
     {:edeliver, "~> 1.4.5"},
     {:distillery, "~> 1.5", runtime: false},
     {:sendgrid, "~> 2.0"},
     {:arc, "~> 0.8.0"}
Which requires some lines to be changed, and some lines to be inserted.

After a lot of trial and error, I ended up deciding that the simplest approach would be to just create some template files that match what each app requires (2) and then just read in the contents, do some string substitution against 'tags' in the template files, and then write them out to where they should be for each app.

So my first step was to read a file and store its' contents in a variable:
Code:
var=$(cat file)
(then do some string substitution)

and then write it out seemingly unchanged:
Code:
echo $var >> newFile
but this loses all of the original whitespace chars.

Hence my question about reading and writing while maintaining the formatted whitespace chars. I figured once this was solved independently of performing any interim steps of string substitution, I could then look into my next problem of replacing the tags in the text with what I actually need to be there.

The tags will be something like
Code:
<AppName>
<app_name>

When the script is run, it requests the app name in the form
Code:
appName
- camel case with the first letter lowercase. This will need to be converted to both version above. The camel case version with the first char mutated to be uppercase, and the snake case version created from the camel case. I could ask for each version on the CL when the script runs, it wouldn't be the end of the world because it's only a script for myself, but if I can mutate the input with bash to the required format that would be my preferred approach. I guess going from snake case to camel case would probably be simpler but I haven't looked that far yet.

Hopefully that explains my problem a little fuller, without rambling on too much.

** edit ** I should probably also say I'm using Bash in OSX, so using sed with -i doesn't work. However, I can spin up a Linux VM should I need to.

Thanks

Paul
Reply
#5
(08-18-2019, 09:15 PM)phollyer Wrote: After a lot of trial and error, I ended up deciding that the simplest approach would be to just create some template files that match what each app requires (2) and then just read in the contents, do some string substitution against 'tags' in the template files, and then write them out to where they should be for each app.

So my first step was to read a file and store its' contents in a variable:
Code:
var=$(cat file)
(then do some string substitution)

and then write it out seemingly unchanged:
Code:
echo $var >> newFile
but this loses all of the original whitespace chars.

Hence my question about reading and writing while maintaining the formatted whitespace chars. I figured once this was solved independently of performing any interim steps of string substitution, I could then look into my next problem of replacing the tags in the text with what I actually need to be there.

 

Concerning "formatting" you are referring to the word splitting aspect of the bash shell. Bash will split words based on the IFS variable.  By default IFS will be a space, tab, and newline.  However, you can eliminate word splitting by surrounding the $var with double quotes.  See the following bash example:


  
Code:
$ cat example.txt
This is line one.
This is line two.
This line three with             tabs.
$
$ MyExample=$(cat example.txt)
$
$ echo $MyExample
This is line one. This is line two. This line three with tabs.
$
$ echo "$MyExample"
This is line one.
This is line two.
This line three with             tabs.
$

The bash example was created using the OS X 10.14 bash.  Smile
Idea  Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime.

Reply
#6
(08-18-2019, 10:37 PM)aetesaki Wrote: Just tested sed on my Mac, and found that sed -i does in deed work though the command on a Mac is 
Code:
sed -i "" "s/old/new/" file

Yes you're right, thanks. I'd seen a fair few comments on StackOverflow saying sed -i didn't work on OSX and after trying it myself without success I had given up on that approach. 

(08-18-2019, 10:37 PM)aetesaki Wrote: For your camel case problem, it's easily solved by telling the script where app name should be camel cased. The transformation can easily be done using the tr command. 

Running your script using the following format
Code:
newapp newapp 4 -phoenix 1.4.9
would, when you've made the script, change the default :phoenix variable to 1.4.9, and camel case the app name as newApp, and snake name to new_app. The 4 tells the script where the camel case is to be transformed. All of which can be done using sed -i "" and basic bash scripting on a Mac.

The only problem with that approach is that I won't know ahead of time where the app name will need to be transformed, each project will have its own name. However, now that you've shown me how to use sed on my Mac I can probably go back to my original way and just change the dependencies list etc in each config file while letting the project build tool create the snake and camel case in the way it normally would.

Thanks, just need to learn how to make the best use of sed now.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)