Dynamic Text Processing - Revision history

Dondi: Wrap up sed notes.

2017-09-11T05:44:00Z

Wrap up sed notes.

Dondi: Start transcribing sed information.

2017-09-11T05:10:02Z

Start transcribing sed information.

Dondi: Write up some examples.

2017-09-11T04:53:15Z

Write up some examples.

Dondi: Start transcribing text processing page.

2017-09-11T04:21:25Z

Start transcribing text processing page.

New page

As hinted in our [[Introduction to the Command Line]], we actually have more power at our fingertips than one might expect thanks to the command line’s ability to pass a coherent stream of data from one command to another. On this page, we cover two commands that lend themselves particularly well to this approach: <code>grep</code> and <code>sed</code>.

== Finding Text: grep ==

'''grep''' finds specific text within its input data according to some pattern. Unfortunately, explaining the name is too complicated for now, so let’s just leave it at '''grep''':
grep "<pattern>"
This will try to find the desired pattern in the lines that you type. If a line matches, it will repeat that line. If it doesn’t match, it will just wait for the next line until you hit <kbd>Control-d</kbd> to end your input.

Try this:
grep "Romance"

Then, type any number of lines. Include the word <code>Romance</code> in some but not others. Notice that the only lines that repeat are the ones with <code>Romance</code> in them. Notice also that the matching is ''case-sensitive''—i.e., <code>romance</code> will not match.

===== Non-Exact Matches =====

Exact matches are interesting, but most other everyday applications can do this without a problem. Note how we said that '''grep''' can match a ''pattern'' and not just ''search text''. It turns out that '''grep''' can “understand” a ''wide'' variety of symbols that represent different patterns of text.

A period ('''.''') represents ''any'' single character. Thus, this pattern:
grep "st..r"
...produces all lines that have “st” and “r” with ''any two'' symbols in between. So lines with <code>steer</code> or <code>Fred Astaire</code> will match, but <code>store</code> or <code>restart</code> will not.

Here are some other patterns that you’ll find useful. Needless to say, this is just the tip of the iceberg; as you get more comfortable with '''grep''', you can learn more and more variations for text patterns.
:{|
| style="width: 10em" | '''['''''<characters>''''']'''
| Matches lines that have ''any'' of the characters listed in ''<characters>''
|-
| '''^'''''pattern''
| Matches lines that ''start with'' the given pattern
|-
| ''pattern'''''$'''
| Matches lines that ''end with'' the given pattern
|-
| ''pattern'''''*'''
| Matches zero or more repetitions of the given pattern
|-
| '''[^'''''<characters>''''']'''*
| Matches lines that do ''not'' have the characters listed in ''<characters>''
|}
Note the dual use of '''^'''; when within brackets '''[ ]''' this means “do not match the characters” but when it is the first symbol of the pattern, it represents the start of a line.

As mentioned, there are many more, but this is a start.

← Older revision		Revision as of 05:44, 11 September 2017
Line 190:		Line 190:
	sed "y/bcdefghijklmnopqrstuvwxyza/abcdefghijklmnopqrstuvwxyz/"		sed "y/bcdefghijklmnopqrstuvwxyza/abcdefghijklmnopqrstuvwxyz/"
	Inclusion of uppercase letters, plus any other substitutions, are left to you for practice. Do note, however, how '''y/<original characters>/<new characters>/''' is materially different from '''s/<pattern>/<replacement>/g'''.		Inclusion of uppercase letters, plus any other substitutions, are left to you for practice. Do note, however, how '''y/<original characters>/<new characters>/''' is materially different from '''s/<pattern>/<replacement>/g'''.
		+
		+	=== Slash (/) in Your Text ===
		+
		+	The slash character (“/”) is used by '''sed''' as a ''separator'': in the '''s///g''' and '''y///''' rules, the slash separates the text/pattern to match from the text that will replace those matches. But what if you want to ''include'' the slash character for either matches or replacements? In this case, the slash isn’t a separator, but just another symbol in the pattern or replacement.
		+
		+	To tell '''sed''' that a particular slash should be interpreted as part of the text, and not a separator, precede it with a ''backslash'' (“\”). The slash that follows it is then treated as part of the pattern or replacement text:
		+	sed "s/\// slash /g"
		+	...will replace all “/” characters with the word “slash.”
		+	<pre>sed "s/Title/<h1>&<\/h1>/g"</pre>
		+	...will enclose all instances of the text “Title” in '''h1''' HTML tags: <nowiki><h1>Title</h1></nowiki>.
		+
		+	This technique for indicating that a symbol should be treated as “plain” text and not something with special meaning is usually called ''escaping'' that symbol. Escaping is necessary for any symbol that has special meaning in '''sed'''’s patterns. Thus, you can correctly infer that symbols like '''[''', ''']''', '''^''', '''.''', '''*''', and '''$''' also need to be escaped.
		+
		+	=== Line-Based Actions ===
		+
		+	You may or may not have noticed that '''sed''' is very line-centric: it operates on patterns ''within'' lines of text, with each line being treated separately. This can be used to your advantage thanks to a few line-oriented commands.
		+
		+	==== Picking Lines ====
		+
		+	You can precede your '''s///g''' or '''y///''' rules with one or two numbers — these numbers represent the ''lines'' which you would like the rule to affect. Two numbers represent a ''line range'' — they must be separated by a comma. The '''sed''' command will then only affect the lines within that range. For example:
		+	sed "5s/hello/goodbye/g"
		+	...will only replace “hello” with “goodbye” on the fifth line of the input text.
		+	sed "3,5s/LMU/UCLA/g"
		+	...will only affect lines 3, 4, and 5 of the input text.
		+
		+	==== Inserting Lines ====
		+
		+	A backslash followed by a lowercase ''n'' — '''\n''' — represents a new line. When used in the replacement section of either the '''s///g''' or '''y///''' rules, '''\n''' will “break” the incoming line, thus adding a line to the resulting output. Thus,
		+	sed "s/paragraph/&\n/g"
		+	...adds a line break after every instance of “paragraph” in the input text.
		+
		+	==== Combining Lines ====
		+
		+	Paradoxically, the reverse operation of ''eliminating'' lines is actually very tricky in '''sed'''; this is precisely due to '''sed'''’s use of lines as separate units of work. For input, the newlines are effectively ''invisible'' to '''sed''', requiring some trickery which, for now, will be left unexplained because what’s going on is just too involved.
		+
		+	Thus, for now, take on faith that this '''sed''' rule will combine all of the input text into a single line:
		+	sed ':a;N;$!ba;s/\n//g'
		+	A few notes on this this particular rule:
		+	* Needless to say, you must type this in ''exactly'' — note the ''single'' quotes as opposed to the usual double quotes.
		+	* The latter part of the rule should look familiar: '''s/\n//g''' does mean “substitute all newlines with nothing.” The trick is that the previous sequence of symbols is what makes these newline characters visible to '''sed''' in the first place. '''s/\n//g''' by itself does not work.
		+	* A variant of the command above is to replace newlines with something else. For example: <pre>sed ':a;N;$!ba;s/\n/ /g'</pre> ...still combines the input text into a single line, but inserts a space wherever the linebreaks used to be.
		+
		+	==== Deleting Lines ====
		+
		+	You can delete lines outright with a new “letter rule:” '''d'''. This invocation of '''sed''', then, pretty much deletes everything:
		+	sed "d"
		+	Thus, '''d''' is not very useful by itself. It does work with the “line picking” directive of preceding the letter with numbers, and this is typically how '''d''' is used:
		+	sed "2,4d"
		+	...deletes the second, third, and fourth line from the input text.
		+	sed "10d"
		+	...will delete only the tenth line of the input text.
		+
		+	There are other ways to tell '''sed''' which lines to delete, but for now we’ll stop with the above deletion by line number.
		+
		+	==== Quitting After a Specific Line ====
		+
		+	A special case for deleting lines is deleting from a certain line ''up to the very end''. This can be tricky especially if you don’t know how many more lines are coming in. For this, there is the '''q''' rule: '''q''' for “quit.” If you think about it, deleting from a certain line onward is actually the same as just stopping further processing. If '''sed''' stops output past a certain line, then that is equivalent to deleting those lines:
		+	sed "59q"
		+	…will effectively cut off the data stream from the 59th line onward.
		+
		+	=== Changing Only the ''n''th Match on a Line ===
		+
		+	At this point, we can expand the '''s///g''' substitution rule a little bit by giving you something other than '''g''' at the end of that rule. '''g''' stands for ''g''lobal, meaning that ''all'' matches of the given pattern should be replaced in the input text.
		+
		+	Instead of '''g''', you can place a ''number'' at the end of the '''s''' rule. This will replace only the ''number''-th occurrence of the given pattern on each line. Thus, replacing only the third “hello” in a line with “dolly” can be done as follows:
		+	sed "s/hello/dolly/3"
		+	This command will leave “hello” and “hello hello” unchanged, but will change “hello hello hello hello” into “hello hello dolly hello”.
		+
		+	=== Caveat: Pattern Matching is ''Greedy'' ===
		+
		+	We conclude this overview of '''sed''' with a note on how it matches text: '''sed''' is said to be ''greedy''. It is probably best to illustrate this with an example:
		+	sed "s/(.*)/term/g"
		+	Before trying this out yourself, try to anticipate what this will do. Ostensibly, this should take any set of characters between parentheses and replace that with the string <code>term</code>. If you agree with that reasoning, go ahead and try it out. What does a line of <code>(a + b) * (a - b) / 2</code> produce?
		+
		+	If you were surprised to see <code>term / 2</code> in the output, and not <code>term * term / 2</code> don’t feel bad—this catches almost everyone off guard. This is where being ''greedy'' comes in: the reason you see just <code>term / 2</code> is that '''sed''' does not match the closing ''')''' until ''the very last one that it finds''. Thus, it replaces the entire <code>'''('''a + b) * (a - b''')'''</code> match with <code>term</code>. Think of it as gobbling everything up until there’s no longer any match for the rest of the line.
		+
		+	The solution to this is to make sure '''sed''' does ''not'' count intervening ''')'''s (or whatever symbol you consider as the terminator) within the in-between text:
		+	sed "s/([^)]*)/term/g"
		+	Note how we have replaced the period '''.''' here with “anything that is not ''')'''”—this will effectively force '''sed''' to declare a match at the first closing parenthesis ''')''' that it finds:
		+	(a + b) * (a - b) / 2
		+	term * term / 2
		+	This approach isn’t perfect—for example it does not handle nested parentheses correctly—but for certain situations, this trick will be just what you need (nudge nudge hint hint).
		+
		+	== There’s More Where This Came From ==
		+
		+	Amazingly, there’s a lot more that '''grep''' and '''sed''' can do that has not been covered here. But what’s on this page should give you a broad enough palette of things to try when tackling any assignments involving the command line. You won’t necessarily need ''all'' of these features, but the features you ''do'' need for certain have definitely been covered.

@@ Line 34: / Line 34: @@
 | Matches zero or more repetitions of the given pattern
 |-
-| '''[^'''''<characters>''''']'''*
+| '''^[^'''''<characters>''''']*$'''
 | Matches lines that do ''not'' have the characters listed in ''<characters>''
 |}
@@ Line 90: / Line 90: @@
   zoundz!
 Remember again, though, that we are case-sensitive by default, so you need to include '''Q''' and '''Z''' in your pattern if you want to factor in capital letters.

@@ Line 12: / Line 12: @@
 Then, type any number of lines. Include the word <code>Romance</code> in some but not others. Notice that the only lines that repeat are the ones with <code>Romance</code> in them. Notice also that the matching is ''case-sensitive''—i.e., <code>romance</code> will not match.
-===== Non-Exact Matches =====
+=== Non-Exact Matches ===
 Exact matches are interesting, but most other everyday applications can do this without a problem.  Note how we said that '''grep''' can match a ''pattern'' and not just ''search text''.  It turns out that '''grep''' can “understand” a ''wide'' variety of symbols that represent different patterns of text.
@@ Line 40: / Line 40: @@
 As mentioned, there are many more, but this is a start.