Need all high-resolution images from a fashion collection on Vogue’s often-paywalled slideshow viewer.

deps.edn

{:paths ["script"]
 :deps {etaoin/etaoin {:mvn/version "1.0.40"}}}

script/dl.clj

(ns dl
  (:require [etaoin.api :as e]))

(defn grab
  [url]
  (e/with-safari d
    (e/go d url)
    (e/wait-exists
     d {:css "ul li picture.responsive-image > source"})
    (loop
     [links []]
      (let [img (e/get-element-attr
                 d {:css "ul li picture.responsive-image img"}
                 "src")]
        (if (= "disabled" (e/get-element-attr
                           d {:css "div[data-testid='RunwayGalleryControlNext'] > svg"}
                           "arrowdisabled"))
          (conj links img)
          (do
            (when (e/visible?
                   d {:css ".persistent-bottom button[aria-label='Collapse']"})
              (e/click
               d {:css ".persistent-bottom button[aria-label='Collapse']"})
              (e/wait d 1))
            (e/click
             d {:css "div[data-testid='RunwayGalleryControlNext']"})
            (e/wait d 1)
            (recur (conj links img))))))))

Then

(def links
  (dl/grab "https://www.vogue.com/fashion-shows/spring-1993-ready-to-wear/john-galliano/slideshow/collection"))

(doseq [link links]
  (spit "ss-93-rtw.txt" (format "%s\n" link) :append true))

while read p; do curl "${p}" -so "ss-93-rtw/${p##*/}"; done < ss-93-rtw.txt

Sorry Vogue..

deps.edn

{:paths ["script"]
 :deps {etaoin/etaoin {:mvn/version "1.0.39"}}
 :aliases {:grab {:exec-fn dl/grab}}}

script/dl.clj

(ns dl
  (:require [etaoin.api :as e]))

(def links
  ["https://www28.zippyshare.com/v/xxxxxx/file.html"
   "https://www11.zippyshare.com/v/xxxxxx/file.html"
   ...])

(defn grab
  [& _]
  (e/with-chrome-headless
    d
    (doseq [link links]
      (e/go d link)
      (e/wait d 1)
      (e/click d {:css "#dlbutton"})
      (e/wait d 30))))

On the command line

clj -X:grab

Shook that works

Black-beard in the springtime crushed roses in his teeth for hard luck on enemies through the year. His memoir indicated this idea came from vagrants on the way to Mendocino from coastal B.C. in the 1970s, whose profession—wild leather—had brought them around. The pair were too open about distinctly confused Indic beliefs and a warlike approach to their craft. This clashed with locals, and did not lead to success. The cows were safe but in time some things drug into the stocky and greyed man’s brain. Black-beard’s disposition lent him little during purchases of cattle supplies in town, every time arriving darkened at feed shops, let alone the big annual market offing, his reputation starting unpopular and continuing unshaken from its pit 25 years later. In summer 1988 he burned his entire farm down claiming betrayal by a vague mistress, stole an acquaintance’s boat and launched off the Eureka coastline as a pirate. He felt unusual and troubled in town and this did not pass on the water. Like dry business piracy is difficult to undertake crewless.

Modern cargo shipping has natural crime resistance outside the dangerous areas of the world with the impermeable shells of its massive freighters, and seabound sightings of crown luxury appropriate for looting are uncommon on the northern Pacific coast even today. Small-time yachtsmen generally do not carry precious assets on short day trips out of local yacht clubs, except maybe a few rare bottles of spirits or sparkling wine.

Black-beard’s rule was arbitrary. Intuition only gets people so far and should often fail. He dredged old destinations from stories and days past to chart his new life. Sometimes picking up much younger people, unsure if they were mates or passengers, receiving their suspicions toward his way of being with cracks in his disciplined performance, he started to fade. They looked at him chewing something at the stern.

Less and less was heard about the local legend, maybe more an anecdote, until one day young people at the shore encountered a small beached craft broken opposite large rocks, carrying a peaceful, sunken seafaring man smiling at the low sky. His hand closed around wiry saltgrass like a bouquet and he passed from life satisfied. A group of local Californians gathered for Black-beard’s wake. Some even drove from areas of Oregon to attend. No one arrived from as far north as Portland, but Black-beard the saved sea criminal had apparently held a small sway in agricultural pocket suburbs up the coast in the later 20th century.

Flaunt magazine

I'm a hobbyist programmer, love to learn and improve. I've gravitated toward small tools that deliver unique utility, then bury that utility in flavor and inconsequential content.

Most software, pretty or not, lacks a voice. This is an absolute fact. Across many domains, big or small, consequential software (on the desktop, websites, mobile apps, tools, engines, firmware, networks, services) cleave functionality from personality.

Personality includes visual decoration, but goes further. A domain you tend to see personality in is game design. That makes sense, because video games are: interactive processes where meaningless actions are wrapped in arbitrary content, imbuing actions with ideas to entertain, provoke thought, and provoke emotion.

Video games are ultimately distractionware—nothing against Terry Cavanagh!—they're fun, but I said it above, their basic mechanical actions are meaningless. I’m not that interested in making games.

In the history of digital design, I see the span of time from the dotcom boom to the pandemic being the rock bottom of craft at the modern forefront of utility objects. Design makes me think of a table and chairs, or an article of clothing. Like with mass manufacturing, handicraft is lost in the pursuit of practical software design.

People do love personal websites. That has been a notable location of personality for decades. I am not making a case that solutions are getting carpal tunnel from handwriting html, or limiting your design skills to figma and webflow. Branded experiences may have personality, but are made hollow by their aspects all needing to serve commerce. Toys and visualizations are a middle ground, a case can be made for meaning in their utility. I do think that beyond these instances, I've started to have my interests piqued in recent years. In the weird browser I currently use, copying the URL makes blue cubes float around the corner of the screen. Why?! Decor. Now, become personal.

My belief that inspires me to do code, is that personality and utility should be separate and always consuming each other. Boundaries on usefulness don't inhibit creativity.

Personality in software should constrain the user, and press on the user. It should mean decoration, and obfuscation, and where text is present, it should create a conversation with the user. Software should fade and die. Most importantly (as I see it, because it's what I see the least of), the subjects of software personalities shouldn't have to couple with the content of their utility. My favorite part of personality is the oblique.

Yess…avatar…indigenous blue man group

— The Way of Water

My idea of fun is compulsively downloading stuff…

I pared back an impromptu script recently that archives the entirety of a substack blog as a standardized data format (ePub via JSON). What if I could expand my domain and download from different publications as well?

Thanks for reading!

Ripping a new source

I tried pulling articles from spikeartmagazine.com the same way that substack-to-json does, by just changing what elements it looks for. Well, surprise, after an hour or two it worked fine! It automatically batch downloaded everything from a specific contributor (probably also works with subjects and categories). But writing python gets old fast…

I learned more about webdrivers on the way. Chiefly, that etaoin is a perfectly good webdriver library in clojure. With a little work, the python substack-to-json can be adapted—then later, generalized.

Why did I want to do this? Lol.. dean-kissick dean-kissick-0 I just wanted this guy’s articles. He made a good playlist, gets referenced in the vibes scene, and now I trust him. I demonstrate my trust by repossessing your content, in some grey area between archiving and piracy.

Compare and contrast

substack-to-json, the technique is: access an index page and crawl through to find every relevant URL, then iterate those URLs to grab fully rendered page content. How general is this method on other written content?

Caveats certainly vary across platforms, like to go through an entire substack archive you need to simulate scrolling to the bottom; on Spike everything loads at once.

Styling static pages once articles are rehosted is another story. Substack is fine, just put classes on a few tags and include their universal CSS file, and posts look as good or better than on their origin. It was kind of magic. Spike styles were a horrifying thicket of jQuery and Drupal, and their resources are CORS-blocked. Other than just being better engineered, Substack may enable cross-origin requests due to being a platform for serial content hosted on custom domains, rather than a single outlet? I don’t know what situations warrant one config or the other.

Webdrivers

So, webdrivers seem absurd and testament to how inverted the web is. You are creating a temporary engine that simulates an entire browser, invisibly or even visibly, and allows ‘RPC’-style calls to interact with DOM minutia in the most granular way. Much more control than I anticipated is possible, down to filling and submitting individual form elements, individual keypresses and clicks, etc, that is scratching the surface. Web is bizarre. Robots imitate human access patterns, humans are conformed to be data robots, and the entire ecosystem has become content farms.

But, in my vague first tests playing with etaoin, I am having a difficult time understanding how to adapt to a basic functional style. It’s giving imperative. It seems like interaction with a webdriver leans in to mimicking an actual web surfing experience, the ‘driver’ you reference cloaks a giant mutable browser runtime that is one giant side-effecting entity.

Other things

I paid attention to and leveraged head metadata tags for the first time. Instead of throwing myself into the trenches of body content, much of the relevant information, like post dates, authors, bylines, all exist sanitized and isolated in the heads of pages! Who knew lol. (Every single web dev down to the entry level)

I had been using XPath as that’s what the substack-to-json author was using. Overkill! Looks like webdriver software generally allows the user to query the DOM in whatever way is most convenient, like.. CSS selector chains. Easier, more readable, basically just as flexible! I would rather avoid yet another DSL and reuse a structure-referencing syntax I already know.

Trying a clj webdriver

etaoin, referenced above, is a pleasant surprise after I got a little mystified in python. I know enough python to adapt and hack on existing scripts to fit my use cases, but joy sparks are not exactly what float through my amygdala.

I will adapt each individual action I’d taken to clojure. It seems like the original script I’ve used,

Starts a headless (invisible) browser instance:

(require '[etaoin.api :as e])

(e/with-chrome-headless
 {:path-browser "..."} ;; As, woops, I didn't have chrome installed
 driver ;; A binding, not a reference! Gotcha
 ;; Now, all subsequent forms are executed declaratively
 )

with-chrome-headless (or with-chrome) completely self-contains a webdriver instance, executes anything in the body with a driver binding, then unalives itself no matter what! That is clean, and pleasant.

Visits a page which, at some point, will contain a link to every desired article:

(e/go "url ...")
;; You then perform subsequent actions.

Easy!

(e/go returns nothing relevant)¹

Loads the whole page:

I forsook this when parsing a simpler archive, this snippet is just a directly adapted proof of concept to demonstrate waiting.

(let [heights (atom [1 0])]
  (while (not (zero? (apply - @heights)))
    (e/scroll-bottom d)
    (e/wait 0.5)
    (e/wait-invisible d {:class :post-preview-silhouette})
    (reset! heights [(e/js-execute d "return document.body.scrollHeight")
                     (first @heights)])))

Substack archives have infinite scroll. When you hit the bottom, it temporarily shows their version of a spinner with a particular classes. More posts have loaded, or the end has been reached.

This is the exact behavior recreated from the original script. How can it be improved? It’s messy, and depends on the temporary element loading by a given time, then compares the current and previous scroll heights.

Saves the desired information, like the main author, and list of all posts:

In theory… The entire archive of articles will now be loaded on the page.

(e/get-element-text-el
 driver
 (e/query driver {:css "#content header h3.author"})) ;; e.g
;; => "Some person"

(mapv #(get-element-inner-html-el driver %)
	  (e/query-all driver {:css "#content .article-block"}))
;; => ["<h4>My big article</h4>\n<a href='https://...'><img></a>"
;;     "<h4>Another post..</h4>\n<a href='https://...'><img></a>"]

This snippet and the above would either be in a driver block, or relying on (def driver (e/chrome ...)).

I was having a bit of a hurdle getting text/HTML results from different parts of a node all at once. But, the answer is straightforward!

Simplified mvs:

(defn get-articles
  [d]
  (let [els (e/query-all d {:css ".article"})
        f (fn [el]
            {:title (e/get-element-text-el d
                      (e/child d el {:css ".article-title"}))
             :url (e/get-element-attr-el d
                    (e/child d el {:css ".article-title a"}))})]
    (mapv f els)))

(defn scrape
  [archive-url]
  (e/with-chrome-headless driver
    (e/go archive-url)
    (get-articles driver)))
;; => [{:title "My article" :url "https://..."}
;;     {:title "Another article" :url "https://.."}]

That is actually soooo sexy, simple, and reasonable. Once I got out of the py/selenium style of mutating collectors, wow! This could be further streamlined or greatly expanded, it’s just an example for clarity.

And scrapes the content of a post.

Hm, it started to get odd here. The script individuates all top-level elements within the content block, filters out substack subscribe nudges, and pastes the article back together. That’s easy to query all the elements, and retaining each element separately makes later parsing easier. But why can’t I find an outer-html function?

Not a hint why, but one solution, @borkdude mentions in 2018 when trying to accomplish an outer-html grab:

(e/get-element-attr-el driver element "outerHTML")

Yeah works fine… But I agree it’s kind of weird it doesn’t exist. What about defining my own to learn more?

Here’s the source for e/get-element-inner-html-el:

(defmethod get-element-inner-html-el
  :default
  [driver el]
  {:pre [(some? el)]}
  (:value (execute {:driver driver
                    :method :get
                    :path [:session (:session driver)
                           :element el
                           :property :innerHTML]})))

Go deeper, etaoin.api/execute → etaoin.impl.client/call → it’s just using clj.http/client to make a browser request for the given property of a given element!

It should be trivial to copy the syntax² and just change the property! And, it is:

(defn get-element-outer-html-el
  [driver el]
  (:value (e/execute {:driver driver
                      :method :get
                      :path [:session (:session driver)
                             :element el
                             :property :outerHTML]})))

Then, it can be plugged into a query for an individual article page’s content:

(defn text-html
  []
  (e/with-chrome-headless
    driver
    (e/go driver "https://...")
    (mapv #(get-element-outer-html-el driver %)
          (e/query-all driver {:css ".field-name-body .field-item > *"}))))
;; => ["<p>Thanks for reading this essay.</p>"
;;     "<p>Next paragraph</p>"
;;     "<img src=\"photo.jpg\">"]

Wow, that’s really all I need. It’s surprising actually. I have achieved the goal of having a robot visit a website, read an article really quickly, and report back to me what it learned. Thanks robot!

The practice project that implements what I described is here, kees-/mag-stripe.

1. Browser-mutating commands (some? all?) return a minimal session ID map.↩
2. Why is the original code a mm? Because, for another webdriver type (phantom), the syntax is slightly different to get a property. I’m not using phantom, it’s fine, I just write a function.↩