What is wget command?

The ability to download content from the world wide web (the internet) and store it locally on your system is an important feature to have. The wget(Web get) command line utility is a freely available package which can be used to retrieve files from the web using HTTP, HTTPS, and FTP protocols. In addition to the mentioned protocols, wget can also be used to download files through HTTP proxies. This is similar to cURL command which we covered recently.

One very useful feature of wget is that it is non-interactive and therefore can be easily used in scripts and cron jobs.
This allows wget to be used to download files as part of triggering a specific action or retrieve files at a specific point in time.
It is also worth noting that wget is designed to work with unreliable network connections and has the ability to resume downloads interrupted due to network disruptions.
We can use wget to download files recursively and also set the number of times wget will try to resume an interrupted download.

In this article, we show you some useful examples implementing the features of the wget command we’ve mentioned thus far.

Before proceeding with the examples, let’s first verify that the command is available on the system.

[[email protected] ~]# which wget
/usr/bin/wget
[[email protected] ~]# rpm -qf /usr/bin/wget
wget-1.12-10.el6.x86_64
[[email protected] ~]#

If the command is not available, you may install it.

Install wget in RedHat/CentOS based systems

yum install wget

Install wget in Debian/Ubuntu based systems

apt-get install wget

You may also download the package from the URL http://ftp.gnu.org/gnu/wget/.

Now, let’s move on to the examples.

Example 1: Download file via HTTP

For our first example, we’ll download the Centos DVD via the HTTP protocol.

[[email protected] ~]# wget http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
--2017-11-20 17:03:18-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
Resolving centos.mirror.net.in... 103.240.105.131
Connecting to centos.mirror.net.in|103.240.105.131|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3972005888 (3.7G) [application/octet-stream]
Saving to: “CentOS-6.9-x86_64-bin-DVD1.iso”
0% [ ] 190,117 14.6K/s eta 3d 9h

How wget command works?

The wget command first resolves the hostname mentioned in the URL to its corresponding IP address. After that, a connection on port 80 is established. An HTTP request is sent and if the response to the request is 200 indicating success, then the size of the file to be downloaded is calculated and the file download is started.

As soon as the download starts wget creates a file with the same name as the file being downloaded in the current working directory.

[[email protected] ~]# ls -ltrh CentOS-6.9-x86_64-bin-DVD1.iso 
-rw-r--r--. 1 root root 373K Nov 20 17:04 CentOS-6.9-x86_64-bin-DVD1.iso

You may monitor the size of this file. It should increase as the download progresses.

Example 2: Download a file with different name from the source

In this example, we will download the same file but this time we will save it with a different name on our local system.
We use the -O option to specify the output file name.

[[email protected] ~]# wget -O CentOS-6_9.iso http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
--2017-11-20 17:13:43-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
Resolving centos.mirror.net.in... 103.240.105.131
Connecting to centos.mirror.net.in|103.240.105.131|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3972005888 (3.7G) [application/octet-stream]
Saving to: “CentOS-6_9.iso”
0% [ ] 1,158,917 46.5K/s eta 11h 50m

As soon the download begins the file CentOS-6_9.iso is created in the current working directory of my system from where I executed the wget command and the size of the file increases as the download progresses.

[[email protected] ~]# ls -ltrh CentOS-6_9.iso
-rw-r--r--. 1 root root 956K Nov 20 17:14 CentOS-6_9.iso

Example 3: Download a file in the background
In the examples, we’ve seen so far, when we execute the wget command we do not get our prompt back until the download is completed.
We could run type & after the command and put the process in the background.
But it’s useful to know that we can use -b option with the wget command to download the file in the background and get our command prompt back immediately after executing the command.

[[email protected] ~]# wget -b http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
Continuing in background, pid 3496.
Output will be written to “wget-log”.

As you may notice from the above output that when used with the -b option the wget command is pushed to the background.
We get our command prompt back immediately and the PID of the process running wget is displayed along with the name of a log file which we may query to see the progress of the download.
Here’s a snippet from the log file wget-log:

[[email protected] ~]# tail -f wget-log
Saving to: “CentOS-6.9-x86_64-bin-DVD1.iso”
0K .......... .......... .......... .......... .......... 0% 33.3K 32h20m
50K .......... .......... .......... .......... .......... 0% 41.4K 29h11m
100K .......... .......... .......... .......... .......... 0% 20.3K 37h7m
150K .......... .......... .......... .......... .......... 0% 46.7K 33h36m
200K .......... .......... .......... .......... .......... 0% 35.8K 32h54m
250K .......... .......... .......... .......... .......... 0% 5.98K 2d9h
300K .......... .......... .......... .......... .......... 0% 38.1K 2d5h
350K .......... .......... .......... .......... .......... 0% 24.9K 2d4h
400K .......... .......... .......... .......... .......... 0% 21.2K 2d3h

 

Related concept:   Linux Booting process explained

Example 4: Resume an interrupted download
I had terminated the download we started in one of our previous examples by pressing ctrl+c. We can use the -c option to resume this download from where we left off.
This holds true provided we have not removed the part of the file which had been downloaded before it got interrupted.
If we’ve removed the file part then the download even with -c option will be treated as a fresh download.

[[email protected] ~]# wget -c http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
--2017-11-20 17:28:16-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
Resolving centos.mirror.net.in... 103.240.105.131
Connecting to centos.mirror.net.in|103.240.105.131|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 3972005888 (3.7G), 3971332771 (3.7G) remaining [application/octet-stream]
Saving to: “CentOS-6.9-x86_64-bin-DVD1.iso”
0% [ ] 1,284,570 15.2K/s eta 35h 4m

Example 5: Change destination location

Up till now the files that we have downloaded were stored in the same location as the current directory from where we were executing the wget command. If needed we may modify the destination path location for the file download using the –directory-prefix option as shown below.

 
[[email protected] ~]# wget --directory-prefix=/tmp http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso 
--2017-11-20 17:37:51-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso 
Resolving centos.mirror.net.in... 103.240.105.131 
Connecting to centos.mirror.net.in|103.240.105.131|:80... connected. 
HTTP request sent, awaiting response... 200 OK Length: 3972005888 (3.7G) 
[application/octet-stream] 
Saving to: “/tmp/CentOS-6.9-x86_64-bin-DVD1.iso” 0% [ ] 5,317 1.15K/s eta 67d 10h

The above command will download the file CentOS-6.9-x86_64-bin-DVD1.iso in the /tmp directory.

Example 6: We can force wget to be quite when doing the work with quite mode(-q)
In the examples we’ve seen so far, wget printed verbose output regarding the HTTP connection being established and the progress of the download.
If we do not want to view this information then we can use the -q option to run the command in quite mode.

[[email protected] ~]# wget -q http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso

If you are downloading a large file you should certainly avoid using quite mode.

Related concept:   How Copying of files to remote servers in Linux works(a Tricky concept)?

Example 7: Download multiple files
We can specify more than one URL to download files from while using the wget command.
Here is an example where we use the wget command to fetch the wget package from the GNU website via HTTP as well as FTP protocols.

[[email protected] ~]# wget http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz
--2017-11-20 17:51:31-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: “wget-1.5.3.tar.gz”
100%[==========================================================================================>] 446,966 86.8K/s in 5.0s
2017-11-20 17:51:51 (86.8 KB/s) - “wget-1.5.3.tar.gz” saved [446966/446966]
--2017-11-20 17:51:51-- ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz
=> “wget-1.7.1.tar.gz”
Connecting to ftp.gnu.org|208.118.235.20|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /gnu/wget ... done.
==> SIZE wget-1.7.1.tar.gz ... 1035518
==> PASV ... done. ==> RETR wget-1.7.1.tar.gz ... done.
Length: 1035518 (1011K) (unauthoritative)
100%[==========================================================================================>] 1,035,518 60.3K/s in 14s
2017-11-20 17:52:13 (70.5 KB/s) - “wget-1.7.1.tar.gz” saved [1035518]
FINISHED --2017-11-20 17:52:13--
Downloaded: 2 files, 1.4M in 19s (74.7 KB/s)

I would like to point out here that the downloads are serial and not parallel i.e. one file download completes after which the second download starts.
Both files are not downloaded together.

Example 8: Download files from URL list contained in a text file
We can put the list of URLs we’d like to download files from within a text file and then feed that file to the wget command using the -i option.
We’ll use the two URLs we used in our previous example to demonstrate this.

I’ve added the URLs to a file named url.txt.

[[email protected] ~]# cat url.txt
http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz
[[email protected] ~]#

Now we’ll feed this file to the wget command with the -i option as shown below:

[[email protected] ~]# wget -i /root/url.txt
--2017-11-20 18:01:38-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: “wget-1.5.3.tar.gz”
100%[==========================================================================================>] 446,966 81.2K/s in 5.4s
2017-11-20 18:01:51 (81.2 KB/s) - “wget-1.5.3.tar.gz” saved [446966/446966]
--2017-11-20 18:01:51-- ftp://ftp.gnu.org/gnu/wget/wget-1.7.1.tar.gz
=> “wget-1.7.1.tar.gz”
Connecting to ftp.gnu.org|208.118.235.20|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /gnu/wget ... done.
==> SIZE wget-1.7.1.tar.gz ... 1035518
==> PASV ... done. ==> RETR wget-1.7.1.tar.gz ... done.
Length: 1035518 (1011K) (unauthoritative)
100%[==========================================================================================>] 1,035,518 62.9K/s in 17s
2017-11-20 18:02:16 (60.1 KB/s) - “wget-1.7.1.tar.gz” saved [1035518]
FINISHED --2017-11-20 18:02:16--
Downloaded: 2 files, 1.4M in 22s (65.2 KB/s)
[[email protected] ~]#

Example 9: Downloading files from sites requiring authentication
A very useful feature of the wget command is that it allows us to download files from password protected sites.

In this example, we’ll download a file via FTP from my FTP server by providing credentials for my username.

[[email protected] ~]# wget --user=sahil [email protected][email protected]$$w0rd ftp://192.168.188.129/test.txt
--2017-11-20 18:53:58-- ftp://192.168.188.129/test.txt
=> “test.txt”
Connecting to 192.168.188.129:21... connected.
Logging in as sahil ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD not needed.
==> SIZE test.txt ... 20
==> PASV ... done. ==> RETR test.txt ... done.
Length: 20 (unauthoritative)
100%[==========================================================================================>] 20 --.-K/s in 0s
2017-11-20 18:53:58 (58.5 KB/s) - “test.txt” saved [20]
[[email protected] ~]# ls -l test.txt
-rw-r--r--. 1 root root 20 Nov 20 18:53 test.txt

This is awesome but if you are downloading a big file then anyone could view your password when listing processes in the ps -ef command.
A safer way is to use the –ask-password option instead of –password so that wget will prompt you for the password.

[[email protected] ~]# wget --user=sahil --ask-password ftp://192.168.188.129/test.txt
Password for user “sahil”:
--2017-11-20 18:56:59-- ftp://192.168.188.129/test.txt
=> “test.txt.1”
Connecting to 192.168.188.129:21... connected.
Logging in as sahil ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD not needed.
==> SIZE test.txt ... 20
==> PASV ... done. ==> RETR test.txt ... done.
Length: 20 (unauthoritative)
100%[==========================================================================================>] 20 --.-K/s in 0s
2017-11-20 18:56:59 (75.5 KB/s) - “test.txt.1” saved [20]

Example 10: Limit download speed
While using wget we could also instruct it to limit the download speed if we are on a slow connection by using the –limit-rate=100k option.

[[email protected] ~]# wget --limit-rate=100k -o wget_logs.txt http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
[[email protected] ~]#

In this example, we’ve also used the -o option to redirect the output of the wget command to a file named wget_logs.txt.

Related concept:   How to implement RAID10?

The contents of the file are shown below:

[[email protected] ~]# cat wget_logs.txt
--2017-11-20 18:59:59-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: “wget-1.5.3.tar.gz.1”
0K .......... .......... .......... .......... .......... 11% 48.1K 8s
50K .......... .......... .......... .......... .......... 22% 84.4K 5s
100K .......... .......... .......... .......... .......... 34% 25.4K 7s
150K .......... .......... .......... .......... .......... 45% 40.2K 6s
200K .......... .......... .......... .......... .......... 57% 81.0K 4s
250K .......... .......... .......... .......... .......... 68% 92.7K 3s
300K .......... .......... .......... .......... .......... 80% 85.1K 2s
350K .......... .......... .......... .......... .......... 91% 156K 1s
400K .......... .......... .......... ...... 100% 76.4K=7.4s
2017-11-20 19:00:12 (59.1 KB/s) - “wget-1.5.3.tar.gz.1” saved [446966/446966]

Example 11: Increase retry attempts while downloading files.
By default, the wget command retires a download 20 times to download a file successfully. We can increase this count by using the –tries option.

[[email protected] ~]# wget --tries=101 http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso --2017-11-20 19:05:18-- http://centos.mirror.net.in/centos/6.9/isos/x86_64/CentOS-6.9-x86_64-bin-DVD1.iso
Resolving centos.mirror.net.in... 103.240.105.131

In the above example, I increased the retry limit from the default 20 to 101.

Example 12:
In the final example for this article, we’ll show how you can use the wget command to download a webpage and save it to a file.

[[email protected] ~]# wget http://www.linuxnix.com/7-mpstat-command-examples-in-linux/
--2017-11-20 19:28:35-- http://www.linuxnix.com/7-mpstat-command-examples-in-linux/
Resolving www.linuxnix.com... 104.28.4.110, 104.28.5.110
Connecting to www.linuxnix.com|104.28.4.110|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “index.html”
[ <=> ] 100,658 151K/s in 0.6s
2017-11-20 19:28:38 (151 KB/s) - “index.html” saved [100658]

The above command saved the web page http://www.linuxnix.com/7-mpstat-command-examples-in-linux/ to a file named index.html.
Don’t be confused by the file name.
If you open the file you’ll find that it has the content of the actual web page and is by no means a conventional index.html file.

Conclusion:

In this article, we explored different methods to retrieve content using the wget command.
We hope that you’ve found this information useful and we encourage you to try out these features on your own as well.