Don't have a Linux machine? Sign up to BitLaunch and use one of our VMs to perform a cURL download.
Though there are a few command-line tools a Linux VPS can utilize to download files, Client URL (cURL) is one of the most used. Its inclusion on most Unix-like operating systems and wide protocol support has made it popular with server admins. This is partly because of its lack of interactivity - when you download a file with cURL it doesn't ask for confirmation and can transfer multiple files at once. This makes it ideal for automation if you trust the source.
Today we'll be walking you through the basics of using curl to download a file. Our sample file will be the readme for blcli, BitLaunch's command-line interface, which is hosted on GitHub.
Fetching a File with cURL
The basic structure of cURL is curl http://yoururl.filename
.
curl
naturally invokes our command line tool, while the URL points to the location of the remote file you want to download with cURL. In the case of our readme, the complete command would like this:
curl https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md
You'll notice that the contents of the file are displayed in your command line:
blcli is a command-line interface for BitLaunch.io
```
Usage:
blcli [command]
Available Commands:
account Retrieve account information
create-options View images, sizes, and options available for a host when creating a new server.
help Help about any command
server Manage your virtual machines
sshkey Manage SSH Keys
transaction Manage transactions
version blcli version
Download a file with cURL
So what if we want to use cURL to save a file to our server? For that, we must use the -O
option:
curl -O https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md
You'll notice that cURL will display a download progress table rather than the file contents this time:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2559 100 2559 0 0 17174 0 --:--:-- --:--:-- --:--:-- 17174
To read verify that the file has downloaded, we can then type nano README.md
.
If you'd like the file to have a different file name (perhaps readme is too bossy for you), specify it after-O
:
curl -o dontreadme.md
https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md
You should then be able to read the file with sudo nano dontreadme.md
.
Use cURL to download multiple files
That's all well and good, but downloading lots of files in this way would quickly become a hassle. You can download more than one file in a single command by using the following syntax:
curl -O -O https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md https://bitlaunch.io/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2559 100 2559 0 0 5067 0 --:--:-- --:--:-- --:--:-- 5067
100 23 100 23 0 0 101 0 --:--:-- --:--:-- --:--:-- 101
You'll see that this time the output lists two rows for downloads - both of our files.
If both files are in the same sub-directory on the same site, it's even easier:
curl -O -O https://raw.githubusercontent.com/BitLaunchIO/blcli/master/{README.md,LICENSE.txt}
Or, if you want to download a series of files in the same directory that are numbered:
curl examplewebsite.com/filename[1-20].jpeg
If you have a long list of different files you want to download, you can place them in a text file and run cURL with xargs:
xargs -n 1 curl -O fileurls.txt
You'll get the normal download output with each file transfer listed in its own row.
Get cURL to follow redirects
As it's common for site admins to move the location of a file, then 301 redirect to the new one, it can be good practice to include the -L
option in your cURL command.
For example, if we try to access BitLaunch's robots.txt with this command:
curl -O www.bitlaunch.io/robots.txt
You get no output. That's because www.bitlaunch
traffic is redirected to https://bitlaunch
. If we use curl -L -O www.bitlaunch.io/robots.txt
, cURL will follow any redirects:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 23 100 23 0 0 82 0 --:--:-- --:--:-- --:--:-- 82
You can see that there are two rows in our download table that didn't download any data. These are the URLs that were redirected. We can review this redirect chain with:
curl -v www.bitlaunch.io/robots.txt
* Trying 104.22.72.97...
* TCP_NODELAY set
* Connected to www.bitlaunch.io (104.22.72.97) port 80 (#0)
> GET /robots.txt HTTP/1.1
> Host: www.bitlaunch.io
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Sun, 15 Nov 2020 12:50:43 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Sun, 15 Nov 2020 13:50:43 GMT
< Location: https://www.bitlaunch.io/robots.txt
< cf-request-id: 066d8e4110000015dc31949000000001
< Server: cloudflare
< CF-RAY: 5f29197b4be815dc-ARN
<
* Connection #0 to host www.bitlaunch.io left intact