Cookbooks
Go
Chapter 11. HTML and the Web
Downloading a web page

Go:
Downloading a web page

How to:

In Go, the standard library provides powerful tools for web requests, notably the net/http package. To download a web page, we primarily use http.Get method. Here is a basic example:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    url := "http://example.com"
    response, err := http.Get(url)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer response.Body.Close()

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Error reading body:", err)
        return
    }

    fmt.Println(string(body))
}

Sample output could be the HTML content of http://example.com, which is a basic example web page:

<!doctype html>
<html>
<head>
    <title>Example Domain</title>
...
</html>

This simple program makes an HTTP GET request to the specified URL, then reads and prints the body of the response.

Note: In contemporary Go programming, ioutil.ReadAll is considered deprecated since Go 1.16 in favor of io.ReadAll.

Deep Dive

The Go language has a design philosophy that emphasizes simplicity, efficiency, and reliable error handling. When it comes to network programming, and specifically downloading web pages, Go’s standard library, notably net/http, is efficiently designed to handle HTTP request and response operations.

The approach to network requests in Go dates back to the language’s origins, borrowing concepts from predecessors but significantly improving on efficiency and simplicity. For downloading content, Go’s concurrency model using goroutines makes it an exceptionally powerful tool for making asynchronous HTTP requests, handling thousands of requests in parallel with ease.

Historically, programmers relied heavily on third-party libraries in other languages for simple HTTP requests, but Go’s standard library effectively eliminates this need for most common use cases. While there are alternatives and more comprehensive packages available for complex scenarios, such as Colly for web scraping, the native net/http package is often sufficient for downloading web pages, making Go an attractive choice for developers looking for a built-in, no-frills solution.

In comparison to other languages, Go provides a notably straightforward and performant way to perform network operations, underlining the language’s philosophy of doing more with less. Even as better alternatives might be available for specialized tasks, Go’s built-in features strike a balance between ease of use and performance, making it a compelling option for downloading web content.

Parsing HTML