How to Download Files With cURL?

Saving a file with the original name

To save a remote file to your local system with the same filename as in the URL, use the -O option:

In this case, the downloaded file will keep its original name.

Saving a file with a specified name

To download a response to a file with a specific name, use the -o (or --output) argument, followed by the local file name to which you’d like to save the contents.

In this example, the file will be saved under the "example.txt" name, and you'll avoid overwriting the existing file with the same name.

How to append responses to a file?

If you want to append the response to an existing file rather than overwrite it, you can use shell redirection in Unix-like systems or the equivalent in other operating systems. Here's how to append to a file in a Unix-like system:

Note that when using shell redirection to append or save output (> for overwrite; >> for append), you're capturing the standard output (stdout) of the cURL command. This means that any headers or verbose information sent to standard error (stderr) won't be captured in the downloaded file unless you explicitly redirect stderr to stdout using 2>&1. Here’s how this would look in a complete cURL request, including both the redirection of output to a file and capturing verbose information:

• curl is the command to use cURL.

• -v enables verbose mode, which sends detailed request and response information to stderr.

• https://example.com is the URL you are making the request to.

• -o output.txt directs stdout (typically the response body) to a file named output.txt.

• 2>&1 redirects stderr (where the verbose information goes) to stdout so that both the response data and the verbose details are captured in output.txt.

Handling redirects in cURL

Redirects are a standard mechanism on the web, instructing clients to fetch a resource from a different URL than the one initially requested. When saving a remote file using cURL, handling redirects ensures that your application can follow the server’s response just as a web browser would.

How to automatically follow redirects?

By default, cURL doesn’t follow redirect responses (like 301 Moved Permanently or 302 Found). To enable automatic following of redirects, use the -L or --location flag:

With -L, if the server responds with a redirect, cURL will make a new request to the URL specified in the response's Location header.

Dealing with cookies across redirects

You might need to manage cookies across different domains or paths when handling redirects, especially with authentication flows. Use the -b option to send cookies and -c to store cookies provided in the response:

In this cURL command, -b cookies.txt tells cURL to use the existing cookies from the cookies.txt file for the request, while -c cookies.txt updates cookies.txt with any new cookies received in the response.

Bypassing authentication

To bypass HTTP or FTP authentication when downloading files, use the -u or --user flag:

This method is straightforward but not the most secure, as the credentials are sent in plaintext. It's strongly recommended to use HTTPS when employing basic authentication to prevent eavesdropping.

Downloading files via proxy

For a step-by-step guide, check our help documentation on setting up and implementing our proxies to download files, but it can be as simple as generating a proxy list on our dashboard and using it like this:

Here, the -U option introduces proxy authentication, which should be followed with your proxy user’s credentials. The -x option specifies the proxy server: in this case, it’s a location from the US, but you can customize this and other parameters on our dashboard.

It's also possible to simply whitelist your IP and use the -x option, followed by the address of the proxy server, port, and target website:

Saving headers and body

To save both the response headers and body to the same file, you can use the -i (or --include) option, which includes the HTTP response headers in the output:

Alternatively, if you want to save headers and body to separate files, you can use the -D option for headers:

Downloading multiple files

To download multiple files from different URLs, you can simply specify URLs and filenames in one cURL command line:

To download multiple files for URLs that share a common pattern, use curly braces {}:

In this case, all files share the same "https://example.com/images/" path.

Handling binary files

When downloading binary files, such as an image or an archive, ensure that you write the output in binary mode to prevent data corruption. cURL handles this automatically when using -o or -O, but it's an important consideration when redirecting output through the shell:

Saving file in verbose mode

If you're using verbose mode (-v) to debug a request and want to save the output (including verbose information) to a file, you can redirect both stdout and stderr to the file:

What is the verbose mode in cURL?

Verbose mode in cURL is an invaluable feature providing a detailed view of the request-response cycle. This mode outputs additional details to stderr about the HTTP conversation between the client (cURL) and the server. It includes information such as request headers, response headers, and the negotiation process of the protocols. Understanding and effectively using verbose mode can significantly aid in debugging, optimizing, and understanding web interactions.

How to enable verbose mode?

To enable verbose mode in cURL, you use the -v or --verbose flag. Here's a basic example:

The -v flag instructs cURL to print detailed information about the request and response, including the full set of HTTP headers.

How can the verbose mode help to debug a request?

Verbose mode provides insights into several aspects of the HTTP request and response process:

• Connection details. Shows the IP address and port number of the server to which cURL connects.

• TLS/SSL handshake. If connecting over HTTPS, verbose mode displays the TLS handshake process, including the chosen protocol version and cipher suite.

• Request headers. Displays all headers sent to the server, which can help ensure that your request is formatted correctly.

• Response headers. Lists all headers returned by the server, useful for debugging caching issues, cookies, and more.

• Redirects. Shows the process of following redirects if the -L flag is used, including the response codes and the Location headers that guide the redirects.

How to combine verbose mode with output redirection?

When using verbose mode, the details are printed to stderr, while the actual content of the response is sent to stdout. This separation allows you to redirect them separately, for instance, saving the response to a file while still viewing the verbose output on the screen:

To capture both the verbose data and the response in files, you can redirect stderr to a separate file:

Mind the security

While the verbose mode is highly useful for debugging, be cautious with the output, especially in shared environments or when saving to logs, as it can contain sensitive information like authentication tokens or cookies.

Rate limiting

You can use the --limit-rate [speed] flag in cURL to restrict the maximum download speed, which is particularly helpful on slow internet connections. You can specify the rate limit value with a letter suffix: K for kilobytes, M for megabytes, or G for gigabytes.

In this example, the maximum download speed is 300 kilobytes.

Final thoughts

If you're working with web services and APIs, using cURL for file downloads can enhance your efficiency when saving responses, handling redirects, or managing authentication. Using various options to download multiple files, save files via proxy, automatically follow redirects, and other tasks, cURL becomes an essential tool for optimizing your workflow.