How to use cURL to download a file

Don't have a Linux machine? Sign up to BitLaunch and use one of our VMs to perform a cURL download.

Though there are a few command-line tools a Linux VPS can utilize to download files, Client URL (cURL) is one of the most used. Its inclusion on most Unix-like operating systems and wide protocol support has made it popular with server admins. This is partly because of its lack of interactivity - when you download a file with cURL it doesn't ask for confirmation and can transfer multiple files at once. This makes it ideal for automation if you trust the source.

Today we'll be walking you through the basics of using curl to download a file. Our sample file will be the readme for blcli, BitLaunch's command-line interface, which is hosted on GitHub.

Fetching a File with cURL

The basic structure of cURL is curl http://yoururl.filename.

curl naturally invokes our command line tool, while the URL points to the location of the remote file you want to download with cURL. In the case of our readme, the complete command would like this:

curl https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md

You'll notice that the contents of the file are displayed in your command line:

blcli is a command-line interface for BitLaunch.io

```
Usage:
  blcli [command]

Available Commands:
  account        Retrieve account information
  create-options View images, sizes, and options available for a host when creating a new server.
  help           Help about any command
  server         Manage your virtual machines
  sshkey         Manage SSH Keys
  transaction    Manage transactions
  version        blcli version

Download a file with cURL

So what if we want to use cURL to save a file to our server? For that, we must use the -O option:

curl -O https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md

You'll notice that cURL will display a download progress table rather than the file contents this time:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2559  100  2559    0     0  17174      0 --:--:-- --:--:-- --:--:-- 17174

To read verify that the file has downloaded, we can then type nano README.md.

If you'd like the file to have a different file name (perhaps readme is too bossy for you), specify it after-O :

curl -o dontreadme.md https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md

You should then be able to read the file with sudo nano dontreadme.md.

Use cURL to download multiple files

That's all well and good, but downloading lots of files in this way would quickly become a hassle. You can download more than one file in a single command by using the following syntax:

curl -O -O https://raw.githubusercontent.com/BitLaunchIO/blcli/master/README.md https://bitlaunch.io/robots.txt

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2559  100  2559    0     0   5067      0 --:--:-- --:--:-- --:--:--  5067
100    23  100    23    0     0    101      0 --:--:-- --:--:-- --:--:--   101

You'll see that this time the output lists two rows for downloads - both of our files.

If both files are in the same sub-directory on the same site, it's even easier:

curl -O -O https://raw.githubusercontent.com/BitLaunchIO/blcli/master/{README.md,LICENSE.txt}

Or, if you want to download a series of files in the same directory that are numbered:

curl examplewebsite.com/filename[1-20].jpeg

If you have a long list of different files you want to download, you can place them in a text file and run cURL with xargs:

xargs -n 1 curl -O fileurls.txt

You'll get the normal download output with each file transfer listed in its own row.

Get cURL to follow redirects

As it's common for site admins to move the location of a file, then 301 redirect to the new one, it can be good practice to include the -L option in your cURL command.

For example, if we try to access BitLaunch's robots.txt with this command:

curl -O www.bitlaunch.io/robots.txt

You get no output. That's because www.bitlaunch traffic is redirected to https://bitlaunch. If we use curl -L -O www.bitlaunch.io/robots.txt, cURL will follow any redirects:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    23  100    23    0     0     82      0 --:--:-- --:--:-- --:--:--    82

You can see that there are two rows in our download table that didn't download any data. These are the URLs that were redirected. We can review this redirect chain with:

curl -v www.bitlaunch.io/robots.txt

*   Trying 104.22.72.97...
* TCP_NODELAY set
* Connected to www.bitlaunch.io (104.22.72.97) port 80 (#0)
> GET /robots.txt HTTP/1.1
> Host: www.bitlaunch.io
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Sun, 15 Nov 2020 12:50:43 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Sun, 15 Nov 2020 13:50:43 GMT
< Location: https://www.bitlaunch.io/robots.txt
< cf-request-id: 066d8e4110000015dc31949000000001
< Server: cloudflare
< CF-RAY: 5f29197b4be815dc-ARN
< 
* Connection #0 to host www.bitlaunch.io left intact