Cookbooks
Clojure
Chapter 11. HTML and the Web
Downloading a web page

Clojure:
Downloading a web page

How to:

In Clojure, you can use clj-http to quickly download a web page. Here’s a barebones example:

(require '[clj-http.client :as client])

(defn download-page [url]
  (client/get url))

;; Use it like this:
(defn -main []
  (println (download-page "http://example.com")))

If you try that out, you’ll get a map full of details. The juicy bits are under :body and :status.

Deep Dive

Historically, web downloading was a ‘wget’ or ‘curl’ at the command line. Now, languages like Clojure abstract this with libraries. clj-http is one such library that wraps Java’s Apache HttpComponents for Clojure’s functional style.

Alternatives? Sure. You could conjure up java.net.HttpURLConnection directly or pick another library like http-kit – but clj-http is comfy and packs most things you’ll need out of the box.

As for nuts and bolts, clj-http turns your request into a Java HTTP entity, makes the call, and hands the response back. Behind the scenes, it’s handling redirects, parsing headers, and managing the response body so you can focus on your data, not the plumbing.

See Also

Parsing HTML