The basics of GNU Wget

GNU Wget is an open source software for downloading files with compatibility with the many popular internet protocols through command line use.

Wget is a popular tool and highly versatile, however it can be confusing or overwhelming but here are some commands to get started.

If you’re using Windows you can download Wget here or for Linux based systems simply do

sudo apt install wget

or

yum install wget -y

Downloading a website

wget -r www.website.com

-r stands for recursive and goes 5 levels deep as default, to set levels deep do as

wget -r -20 www.everydaylinuxuser.com

This is (up to) 20 levels deep.

Download a file

wget saveas downloadurl

wget /save/here/file.zip www.website.com/filelinktodownload.zip

Download from folder

wget -r --no-parent /save/here -A jpeg,jpg,gif,png https://website/images

--no-parent means Wget wont enter directories above the listed one, however as we have the -r option Wget will go into directories below the listed one.

-A Means accept. Follow this with file extensions that you want to download. In the example above Wget will only download images (.jpeg, .jpg, .gif, .png).

Another option is -R meaning Reject. It is the opposite of accept and can be use to download all file except say .bmp, .html and .tmp which seems to be unwanted in cases.

Download list from text file

If you have a text file that has a list of links like:

link1
link2
link3
link4

You can download them all by using -i which means list

wget -i downloadlist.txt /save/here

Download in background

For Linux systems to get Wget running in the background

wget -bqc /save/here/file.zip https://website.com/file.zip

-b is background, -q turns of output and -c is for resuming broken downloads.

Download with authentication

To download from FTP with a username and password use the following, replacing with your credentials.

--user=USERNAMEHERE --password=PASSWORDHERE

Set user agent

A user agent for requests can be set with -U

-U Mozilla/5.0

Skip SSL checking

 

--no-check-certificate

Only download from https links

--https-only

 

A full documentation on Wget can be viewed here.