What is curl command?

The curl command is a powerful command line tool used to transfer data to or from a server.
It can use any one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE) for data transfer. In many aspects curl is similar to the wget command but curl is more feature rich as compared to wget. The cURL stands for “See URL“.

The curl command is generally available by default on UNIX systems.
Let’s verify that we have curl installed.

[root@linuxnix ~]# which curl
/usr/bin/curl
[root@linuxnix ~]# rpm -qf /usr/bin/curl
curl-7.19.7-52.el6.x86_64

In case your system does not have curl installed then you can install it manually.

If you are using a RedHat based system then type:

yum install curl

If you are using a Debian based system then type:

apt-get install curl

In this article, we’ll go through some examples to understand some of the features curl provides.

Example 1: Save a web page.

[root@linuxnix ~]# curl https://www.linuxnix.com/ >> linuxnix.html
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 206k 0 206k 0 0 29952 0 --:--:-- 0:00:07 --:--:-- 179k
[root@linuxnix ~]#

The above example will save the content of the web page https://www.linuxnix.com/ to a file named linuxnix.html.

Example 2: Specify output file name in curl command
By using the -o option you may specify the name of the output file in the curl command itself instead of using output redirection as we did in the previous example.

[root@linuxnix ~]# curl -o sahil.html https://www.linuxnix.com/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 206k 0 206k 0 0 27231 0 --:--:-- 0:00:07 --:--:-- 133k
[root@linuxnix ~]# ls -ltrh sahil.html
-rw-r--r--. 1 root root 207K Nov 20 20:32 sahil.html

The above command saves the content of the web page to a file named sahil.html.

Example 3: Save multiple files with a single curl command
We can specify multiple URLs to save content from and in a separate file using a single curl command line statement as shown below:

[root@linuxnix ~]# curl -o sahil.html https://www.linuxnix.com/7-mpstat-command-examples-in-linux/ -o sahil2.html https://www.linuxnix.com/awk-scripting-print-header-line-along-with-pattern-match/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 94230 0 94230 0 0 41826 0 --:--:-- 0:00:02 --:--:-- 85353
100 79251 0 79251 0 0 84024 0 --:--:-- --:--:-- --:--:-- 487k
[root@linuxnix ~]# ls -l *.html
-rw-r--r--. 1 root root 79251 Nov 20 20:39 sahil2.html
-rw-r--r--. 1 root root 94230 Nov 20 20:39 sahil.html

Example 4: Do not display progress bar
We can use the -s option with the curl command if we do not want it to display(silent) the default progress bar.

Here is an example.

[root@linuxnix ~]# curl -s -o sahil.html https://www.linuxnix.com/
[root@linuxnix ~]#

Example 5: Follow HTTP location headers
By default, the curl command does not follow location headers.
When a requested web page is moved, then an HTTP Location header will be sent as a response and it will mention the present location of the web page.

For example, when someone types google.com in the browser from India, it will be automatically redirected to ‘google.co.in’.
This is done based on the HTTP Location header as shown below.

[root@linuxnix ~]# curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.in/?gfe_rd=cr&amp;dcr=0&amp;ei=4KMSWtCLPPScX-rqvoAF">here</A>.
</BODY></HTML>
[root@linuxnix ~]#

We can instruct curl to follow the redirection by using -L option, as shown below.

[root@linuxnix ~]# curl -L google.com
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-IN">
<head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image">
<title>Google</title><script>(function(){window.google={----output truncated for brevity

Now curl will print the web page in HTML format.

Example 6: Fetch and display the HTTP-header only
To just verify that a website URL is accessible and to print it’s HTTP header, we use the -I option with the curl command.

[root@linuxnix ~]# curl -I https://www.linuxnix.com
HTTP/1.1 200 OK
Date: Mon, 20 Nov 2017 09:55:19 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=dd037dc84d9c6b26fc56c8e37c1be10f91511171718; expires=Tue, 20-Nov-18 09:55:18 GMT; path=/; domain=.linuxnix.com; HttpOnly
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Content-Type-Options: nosniff
Server: cloudflare-nginx
CF-RAY: 3c0a886a3b2270ce-SIN
[root@linuxnix ~]#

Example 7: Use curl command in verbose mode
We can use the -v option with curl to perform a verbose mode while using it to display more detailed information.

[root@linuxnix ~]# curl -v https://www.linuxnix.com
* About to connect() to www.linuxnix.com port 443 (#0)
* Trying 104.28.4.110... connected
* Connected to www.linuxnix.com (104.28.4.110) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=sni155736.cloudflaressl.com,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated
* start date: Nov 09 00:00:00 2017 GMT
* expire date: May 18 23:59:59 2018 GMT
* common name: sni155736.cloudflaressl.com
* issuer: CN=COMODO ECC Domain Validation Secure Server CA 2,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: www.linuxnix.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 20 Nov 2017 09:56:07 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
-----------------------------------------------output truncated for brevity

The * symbols represent information printed by the curl tool and ‘<‘ and ‘>’ display the data sent and received respectively by Curl

Example 8: Save the file with the same name as that in the URL
To save a file with the same name as that mentioned in the URL, we use the -O option as shown below:

[root@linuxnix ~]# curl -O http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
22 436k 22 98906 0 0 10952 0 0:00:40 0:00:09 0:00:31 27704

Example 9: Resume interrupted downloads
I had canceled the ongoing download shown in the previous example at 22%. By using the -C option we can actually resume the download from where we left off.

[root@linuxnix ~]# curl -C - -O http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
** Resuming transfer from byte position 151552
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 288k 100 288k 0 0 29356 0 0:00:10 0:00:10 --:--:-- 61218
[root@linuxnix ~]#

Note: You need to add an extra dash (-) in the command above. If you don’t curl seems to print the file on stdout which isn’t usable.

Example 10: Limit download speed
To limit the download speed while downloading a file we use the –limit-rate option.

[root@linuxnix ~]# curl --limit-rate 10K -O http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
18 436k 18 81146 0 0 9172 0 0:00:48 0:00:08 0:00:40 21949

In the above example, we’ve limited the download speed to 10Kb/s for the operation.

Example 11: Download files depending on the specified modified data
We can use the -z option with curl to download files only if they have been modified after or before a specified modified date.

[root@linuxnix ~]# curl -z 21-Nov-17 -O http://192.168.188.129/test.htm
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 9495 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
[root@linuxnix ~]# ls -l test.htm
ls: cannot access test.htm: No such file or directory
[root@linuxnix ~]# curl -z -21-Nov-17 -O http://192.168.188.129/test.htm
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 9495 101 9495 0 0 1773k 0 --:--:-- --:--:-- --:--:-- 9272k
[root@linuxnix ~]# ls -l test.htm
-rw-r--r--. 1 root root 9495 Nov 20 21:27 test.htm
[root@linuxnix ~]#

If we specify a hyphen or dash before the date then curl checks if the file has been modified before the mentioned date and if there is no dash then curl downloads the file if it has been modified after the specified date.

Example 12: Download file from using FTP
To download a file via FTP or HTTP we use the -u option to provide the connecting username and password details separated by a colon to be used to authenticate with the URL.
Given below is an example:

[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -O ftp://192.168.188.129/test.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 20 0 20 0 0 413 0 --:--:-- --:--:-- --:--:-- 20000
[root@linuxnix ~]#

In the above command, sahil is the username and $eCur1_p@$$w0rd is the password I’m using to authenticate with FTP server 192.168.188.129 and download the file test.txt

Example 13: Upload a file via FTP.
In order to upload a file to an FTP server, we use the -u option to specify the credentials and -T option to specify the file name to be uploaded.

[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -T test.htm ftp://192.168.188.129/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 9495 0 0 101 9495 0 188k --:--:-- --:--:-- --:--:-- 0
[root@linuxnix ~]#

Example 14: Upload multiple files
We can enclose multiple file names within parentheisis and seperated by commas to upload them in a single curl command.

[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -T "{test.htm,test.awk}" ftp://192.168.188.129/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 9495 0 0 101 9495 0 123k --:--:-- --:--:-- --:--:-- 0
109 109 0 0 109 109 0 2881 --:--:-- --:--:-- --:--:-- 2881

Example 15: Upload file with content fed from STDIN
An interesting feature of curl is that while uploading files, you could create a file on the fly by supplying input directly from STDIN and the content would be available in the file at the destination.
Here is a demonstration:

[root@linuxnix ~]# curl -u sahil:$eCur1_p@$$w0rd -T - ftp://192.168.188.129/sahil.txt
This is a test file
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 20 0 0 0 20 0 2 --:--:-- 0:00:06 --:--:-- 2To check upload from STDIN
0 47 0 0 0 47 0 2 --:--:-- 0:00:20 --:--:-- 2
[root@linuxnix ~]#

The above curl command will create a file named sahil.txt on the destination FTP server as soon as my credentials get authenticated.
After that, I can type content and press ctrl+d when done and this will add the content I wrote to the file on the destination.
Also, don’t mind the progress bar from curl interfering in the lines.
This would be there in the file updated at the destination location.

[root@linuxnix sahil]# cat sahil.txt
This is a test file
To check upload from STDIN
[root@linuxnix sahil]#

 

Example 16: Upload/Download files by supplying password non-interactively.
Thus far we’ve successfully downloaded and uploaded files via FTP. But one problem with this approach is that we are typing passwords in plain text which isn’t a good thing.
So, in this example, we will modify our approach and will not pass the password in the curl command but enter it non-interactively instead.

[root@linuxnix ~]# curl -u sahil -T - ftp://192.168.188.129/sahil.txt
Enter host password for user 'sahil':
test file % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 9 0 0 0 9 0 1 --:--:-- 0:00:07 --:--:-- 1
0 10 0 0 0 10 0 0 --:--:-- 0:00:10 --:--:-- 0

As you can see from the above command I did not type in my password in the curl command.
I was prompted for it and when I typed in the correct password and authenticated successfully, things moved further.

Example 17: Download files via SFTP
To download or upload files via sftp we simply need to replace ftp with sftp in the server URL as shown below:

[root@linuxnix ~]# curl -u sahil -O sftp://192.168.188.129/home/sahil/sahil.txt
Enter host password for user 'sahil':
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 47 0 47 0 0 113 0 --:--:-- --:--:-- --:--:-- 113

Example 18: Use key-based authentication to download a file.
In the given example we use public key based authentication to download a file using the as the desired SCP protocol.

[sahil@linuxnix ~]$ curl -k -u sahil: --pubkey ~/.ssh/id_rsa.pub -O scp://192.168.188.129/tmp/test.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 33 0 33 0 0 87 0 --:--:-- --:--:-- --:--:-- 87
[sahil@linuxnix ~]$ ls -l test.txt
-rw-rw-r--. 1 sahil sahil 33 Nov 20 22:17 test.txt

Conclusion:

This concludes our detailed exploration of the wide variety of features provided by the curl command.
Although we’ve covered a lot of features and options in this article we’ve not covered them all.
For example, you could use curl to access web content while being behind a proxy by using the -x option.
Also, you can use the curl command to send HTTP POST, PUT, DELETE requests in HTML.
We encourage you to explore these features and as always we welcome your suggestions and feedback.

Post Views: 14,220
The following two tabs change content below.

Sahil Suri

He started his career in IT in 2011 as a system administrator. He has since worked with HP-UX, Solaris and Linux operating systems along with exposure to high availability and virtualization solutions. He has a keen interest in shell, Python and Perl scripting and is learning the ropes on AWS cloud, DevOps tools, and methodologies. He enjoys sharing the knowledge he's gained over the years with the rest of the community.