The Web from the Command Line - Revision history

Dondi: Final note on scratching the surface.

2017-09-11T03:57:46Z

Final note on scratching the surface.

Dondi: Add POST example.

2017-09-11T03:53:40Z

Add POST example.

Dondi: More curl content.

2017-09-11T02:06:11Z

Dondi: Start writing out curl page.

2017-09-11T01:47:25Z

Start writing out curl page.

New page

Due to the “need-to-know” approach that we take in this course, our study of the command line necessarily takes a leap to a fairly powerful and advanced command. This command beings the web to the command line, and learning it helps one understand what is truly happening behind the scenes when we visit websites with our web browsers. That command is '''curl'''. Strictly speaking, it is spelled '''cURL''' because its name is intended to mean ''“see URL.”'' “See” and “c”—get it?

== Basic Usage ==

Put simply, '''curl''' performs single web requests and displays the response provided by the contacted web server without further processing (nor layout). In its simplest form, one can just give it a URL:
curl <nowiki>http://www.lmu.edu</nowiki>
For most URLs, invoking this command will produce a flood of text—a perfect use case for '''more'''/'''less''', or for output redirection in case you want to save the file to your computer:
curl <nowiki>http://www.lmu.edu</nowiki> > lmu.html
That’s it, really. Using '''curl''', you can get the content of a web page purely at the data level. No images are loaded, no layout is done, no visuals are rendered. This command is like the very first step that a web browser takes when visiting a website, except that it goes no further than that.

← Older revision		Revision as of 03:57, 11 September 2017
Line 25:		Line 25:
	Logging in is a whole other issue, but the point of this example is to show that you can simulate a web browser action that you wouldn’t otherwise be able to do without interacting with the web browser directly.		Logging in is a whole other issue, but the point of this example is to show that you can simulate a web browser action that you wouldn’t otherwise be able to do without interacting with the web browser directly.

−	If you can see a web request in the Network tab of a web browser’s developer tools, then you can simulate this request with '''curl'''.	+	If you can see a web request in the Network tab of a web browser’s developer tools, then you can simulate this request with '''curl'''. And to conclude on that note: if you look at all of the information in the Network tab, you will see that the web browser actually sends ''a lot'' of data and metadata to a web server with every request. Given the claim that '''curl''' can simulate these requests accurately, you can also infer that this page only scratches the surface of what '''curl''' can do. However, from the “need-to-know” perspective, the page does contain enough hints to help you figure out applicable assignments in this class, particularly [[Week 3]].

@@ Line 15: / Line 15: @@
 == curl Mimics Requests in the Network Developer Tools Tab ==
-One such use of '''curl''' is to trigger requests that you can’t perform just by typing a URL into a web browser. The location bar in web browsers only perform what are called '''GET''' requests—requests that are meant to retrieve content from a server. Some requests, however, have a different ''method'', such as '''POST''' or '''PUT'''—these are meant to ''submit'' data to a server. On web browsers, you do this implicitly by filling out forms and clicking on some ''Submit'' button. If you don’t want to go through a browser or would like to do this automatically, you can use '''curl''':
+One such use of '''curl''' is to trigger requests that you can’t perform just by typing a URL into a web browser. The location bar in web browsers only perform what are called '''GET''' requests—requests that are meant to retrieve content from a server. Some requests, however, have a different ''method'', such as '''POST''' or '''PUT'''—these are meant to ''submit'' data to a server. On web browsers, you do this implicitly by filling out forms and clicking on some ''Submit'' button. If you don’t want to go through a browser or would like to do this automatically, you can use '''curl'''.
-{{ Under Construction }}

← Older revision		Revision as of 02:06, 11 September 2017
Line 8:		Line 8:
	curl <nowiki>http://www.lmu.edu</nowiki> > lmu.html		curl <nowiki>http://www.lmu.edu</nowiki> > lmu.html
	That’s it, really. Using '''curl''', you can get the content of a web page purely at the data level. No images are loaded, no layout is done, no visuals are rendered. This command is like the very first step that a web browser takes when visiting a website, except that it goes no further than that.		That’s it, really. Using '''curl''', you can get the content of a web page purely at the data level. No images are loaded, no layout is done, no visuals are rendered. This command is like the very first step that a web browser takes when visiting a website, except that it goes no further than that.
		+
		+	== Why curl? ==
		+
		+	One look at this might beg the question of why the command exists at all, especially when we have web browsers that work perfectly fine for daily use. ''Daily use'' is the operative term here—of course '''curl''' is not meant to be a web browser replacement. Instead, as a command line program that performs a simple request/response cycle, '''curl''' can be used for scripting, automation, and other types of processing that go beyond the visual consumption of a website’s content.
		+
		+	== curl Mimics Requests in the Network Developer Tools Tab ==
		+
		+	One such use of '''curl''' is to trigger requests that you can’t perform just by typing a URL into a web browser. The location bar in web browsers only perform what are called '''GET''' requests—requests that are meant to retrieve content from a server. Some requests, however, have a different ''method'', such as '''POST''' or '''PUT'''—these are meant to ''submit'' data to a server. On web browsers, you do this implicitly by filling out forms and clicking on some ''Submit'' button. If you don’t want to go through a browser or would like to do this automatically, you can use '''curl''':
		+	{{ Under Construction }}