Cookbooks
C
Chapter 11. HTML and the Web
Parsing HTML

C:
Parsing HTML

How to:

Parsing HTML can seem daunting due to HTML’s complexity and its frequent deviations from clean, well-formed structures. However, using a library such as libxml2, specifically its HTML parsing module, simplifies the process. This example demonstrates how to use libxml2 to parse HTML and extract information.

First, ensure libxml2 is installed in your environment. In many Linux distributions, you can install it via the package manager. For example, on Ubuntu:

sudo apt-get install libxml2 libxml2-dev

Now, let’s write a simple C program that uses libxml2 to parse an HTML string and print the text inside a specific element:

#include <stdio.h>
#include <libxml/HTMLparser.h>

void parseHTML(const char *html) {
    htmlDocPtr doc = htmlReadDoc((const xmlChar *)html, NULL, NULL, HTML_PARSE_RECOVER | HTML_PARSE_NOERROR | HTML_PARSE_NOWARNING);
    
    // Assuming we're looking for content inside <p> tags
    xmlNode *root_element = xmlDocGetRootElement(doc);
    for (xmlNode *current_node = root_element; current_node; current_node = current_node->next) {
        if (current_node->type == XML_ELEMENT_NODE && strcmp((const char *)current_node->name, "p") == 0) {
            printf("Found paragraph: %s\n", xmlNodeGetContent(current_node));
        }
    }
    
    xmlFreeDoc(doc);
    xmlCleanupParser();
}

int main() {
    const char *html = "<html><body><p>Hello, world!</p></body></html>";
    parseHTML(html);
    return 0;
}

Sample Output:

Found paragraph: Hello, world!

This example focuses on extracting text within paragraph tags, but libxml2 offers robust support for navigating and querying various parts of an HTML document.

Deep Dive

Parsing HTML in C dates back to the early days of web development. Initially, developers had to rely on custom, often rudimentary parsing solutions, due to the lack of standardized libraries and the chaotic state of HTML on the web. The introduction of libraries like libxml2 marked a significant progression, offering more standardized, efficient, and resilient approaches to parsing HTML.

Despite C’s unmatched speed and control, it’s worth noting that C may not always be the best tool for parsing HTML, especially for tasks requiring quick development cycles or dealing with exceptionally malformed HTML. Languages with high-level HTML parsing libraries, such as Python with Beautiful Soup, provide more abstracted, user-friendly interfaces at the cost of some performance.

Nevertheless, for performance-critical applications, or when operating in resource-constrained environments, parsing HTML in C remains a viable and often preferred method. The key is leveraging robust libraries such as libxml2 to handle the intricacies of HTML, allowing developers to focus on extracting the data they need without getting bogged down in the details of parsing mechanics.

Downloading a web pageSending an HTTP request