bulk string replacement in repository

[amsnyder]

I am starting a discussion thread to document helpful ways to search all the files in our repo for a particular string that we want to replace.

[amsnyder]

Since we made a conus404 intake catalog and removed all conus404 entries from the main hytest intake catalog, I wanted to update all notebooks to point to the new subcatalog.

I wanted to manually update the notebooks given that reading in an intake catalog and then a dataset from it usually takes a couple lines of code that may be formatted in different ways. I searched all ipynbs that contained text that looked like a read of the CONUS404 hourly dataset from any location (conus404-hourly-) with:

grep --include=\*.ipynb -r './' -e 'conus404-hourly-' -c | grep -v ':0$'

I also searched for the strings 'conus404-daily-' and 'conus404-monthly-'. I manually opened each notebook and found the string, then made a replacement where appropriate. Some notebooks did not need replacement because the string was contained in cell output. I chose not to update these cell outputs, though they will be different the next time a user runs the notebook.

[gzt5142]

nice use of grep there. You can see from my use of find to do something similar that I'm a little more stale/old-school. Did not know that grep had the -r.

[gzt5142]

You might have success in turning that into an alias... The only thing you need to change in the expression is the search term.

[amsnyder]

@gzt5142 also had some helpful suggestions for automatic replacement of the string here.

[gzt5142]

The 'automatic' replacement mechanism I suggested uses sed, the unix "stream editor". sed has been around unix since the earliest days, and it is super powerful, with complexity to match. A useful search pattern is:

sed -I.bak -E 's!somethingtosearchfor!whattoreplacewith!g' FILE.ipynb

Note that I am using the bang (!) as the command delimiter in this example... makes it easier to include slashes (/) in the search term. Historically, the slash is the delimiter, so you'll see most examples use something like s/find/replace/g. But that makes it hard to look for file paths:

s/\/home\/gtrantham\/hytest/~\/hytest/g

The slashes in the search pattern have to be escaped if the slash is the delimiter. Bangs are easier.

s!/home/gtrantham/hytest!~/hytest!g

All of the sed commands and capabilities are duplicated from vi. That same search command would work in command mode in vi to do the same substitution in the currently open file.

[gzt5142]

Using sed is mostly about writing good regex patterns for searching.

If you're not really careful, your searchstring will match in more places than you want (or fewer). Since regex is not sed-only, you can test your regex using grep to verify where it matches. once you're satisfied that it matches exactly what you want, move to sed.

[gzt5142]

Also... when I am testing a regex, I don't alter the input file or save it to .bak -- but rather let it write the edited file to the screen so I can see if it did what I wanted:

sed -E 's!searchstring!SUCCESS!g' FILE.ipynb | more

Then I look for "SUCCESS" and see if it is where I expect.