Cookbooks
C
Chapter 11. HTML and the Web
Downloading a web page

C:
Downloading a web page

How to:

To download a web page in C, one popular approach is using the libcurl library, an efficient and portable client-side URL transfer library. Ensure you have libcurl installed and linked in your project. Here’s an example demonstrating how to use libcurl to download the content of a web page:

#include <stdio.h>
#include <curl/curl.h>

size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
    size_t written = fwrite(ptr, size, nmemb, stream);
    return written;
}

int main(void) {
    CURL *curl;
    FILE *fp;
    CURLcode res;
    char *url = "http://example.com";
    char outfilename[FILENAME_MAX] = "./downloaded_page.html";

    curl = curl_easy_init(); // Initialize a libcurl easy session
    if (curl) {
        fp = fopen(outfilename,"wb");
        curl_easy_setopt(curl, CURLOPT_URL, url);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data); // Callback for writing received data
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp); // Set the file pointer to write the data to

        res = curl_easy_perform(curl); // Perform the file download
        if(res != CURLE_OK) {
            fprintf(stderr, "curl_easy_perform() failed: %s\n",
                    curl_easy_strerror(res));
        }

        /* always cleanup */
        curl_easy_cleanup(curl); // Clean up the easy session
        fclose(fp); // Close the file stream
    }
    return 0;
}

Sample output (no visible output in the console): This code downloads the content at the specified URL and saves it to a file named downloaded_page.html. Check your program’s directory for this file to see the downloaded content.

Deep Dive:

Historically, downloading web content in C was more cumbersome, requiring manual socket programming and HTTP protocol handling. Libcurl abstracts these complexities, offering a robust and high-level API for data transfer over the web.

While libcurl simplifies HTTP requests in C, modern programming languages like Python with their requests library or JavaScript (Node.js) with various HTTP client libraries may offer more intuitive syntax and built-in support for JSON and other data formats commonly used in web communication. However, C and libcurl provide a high-performance and stable solution for systems where efficiency, fine-grained control, or integration into existing C codebases are critical. It’s also worth noting that C, combined with libcurl, can be used for more than just downloading web pages—it’s capable of FTP, SMTP, and much more, making it a versatile tool in a programmer’s toolkit.

Parsing HTML