I first met XPath in 2007, but we didn't become friends until just recently. For the most part I had avoided it; when forced to use it, I made do with trial and error. XPath just didn't really make sense to me.
But then I came across a peculiar parsing problem—too complex for CSS selectors, too simple to warrant hand-rolled code—and decided to give XPath another shot. I discovered, much to my surprise and glee, that it does make sense, and once it makes sense, it's actually quite useful.
This is my story.
The problem
Say you're working on a website full of song lyrics, and in order to maintain a consistent reading experience, you want to capitalize the first word of every line. If the lyrics are stored in plain text this is pretty straightforward:
lyrics.gsub!(/^./) { |character| character.upcase }But it gets more interesting if the lyrics are stored as an HTML fragment. A DOM doesn't have any built-in concept of "lines." You can't just break it up with a simple regular expression.
So the first thing we'll need to do is define, for ourselves, what "the beginning of a line" means in a DOM. Here's a simple version:
- The first text node inside a
<p>tag - The first text node following a
<br>tag
So, in the simplest case:
<p>This is the beginning of a line.<br>This is too.</p>But we also want to handle nested inline elements:
<p><em>This</em> is the beginning of a line. <strong>This is not.</strong></p>I'll take the low road
My first instinct was to just write a Ruby method to scan over relevant parts of the DOM and recursively seek out text nodes that fit our criteria. I used some very light CSS selectors, but nothing too fancy:
def each_new_line(document)
document.css('p').each { |p| yield first_text_node(p) }
document.css('br').each { |br| yield first_text_node(br.next) }
end
def first_text_node(node)
if node.nil? then nil
elsif node.text? then node
elsif node.children.any? then
first_text_node(node.children.first)
end
end
def first_text_node(node)
if node.nil? then nil
elsif node.text? then node
elsif node.children.any? then
first_text_node(node.children.first)
end
end This is a perfectly reasonable solution, but it's a whopping 11 lines of code. Further, it feels like we're using the wrong tool for the job: why are we using Ruby iterators and conditionals to get at DOM nodes? Can we do better?
Enter XPath
XPath is confusing for a couple of reasons. The first is that there are surprisingly few good references on the Internet (don't even think about looking at W3Schools!). The best doc I've found is the RFC itself.
The second is that XPath looks deceptively like CSS. The word "path" is right there in the name, and so I had always assumed, mistakenly, that the / in an XPath expression plays the same role as the > in a CSS selector:
document.xpath('//p/em/a') == document.css('p > em > a')As it turns out, the XPath expression involves a lot of shorthand, which we'll want to explode in order to really understand what's going on. Here's the same expression written out in longhand:
/descendant-or-self::node()/child::p/child::em/child::a/This XPath expression and the CSS selector above are equivalent, but not for the reason I had always assumed. An XPath expression consists of one or more “location steps” separated by forward slashes. The / at the beginning means the context of the first step is the root node of the document. Each location step knows which nodes have already been matched, and uses that context to answer three questions:
Where do I want to move from the current context?
This is called the Axis, and it's optional. The default axis is child, meaning "select all of the children of the currently selected nodes." In the above example, descendant-or-self is the axis for the first location step, meaning "all of the currently selected nodes and all of their descendants." Most of the axes defined by the XPath spec are likewise intuitively named.
What sort of nodes do I want to select?
Am I selecting <p> tags, text nodes, or is it a free-for-all? This is specified by the node test, which is the only required part of the location step. In our above example, node() is the most permissive node test: it selects everything. text() would only select text nodes; element() would only select elements; and explicitly specified node names like p and emabove, of course, would only select elements with those names.
Are there additional filters I want to add?
Maybe I only want to select the first child of every node in the current context, or I only want to select <a> tags that have an href attribute. For this sort of assertion, we can use predicates, which filter the matched nodes based on additional tree traversals. So I can filter the matched nodes based on a property of those nodes' children, parents, or siblings.
Our example doesn't have any predicates, so let's add one to only match <a> tags that have an href attribute:
/descendant-or-self::node()/child::p/child::em/child::a[attribute::href]As you can see, the predicate looks just like a location step in brackets, although the "node test" portion of a predicate has much more functionality than the node test of a true location step.
A better way to think about XPath
So instead of thinking of XPath as beefed-up CSS selectors, a more helpful analogy is to chained jQuery traversals. For instance, we could approximate our expression above in jQuery like so:
$(document).find('*').
children('p').
children('em').
children('a').filter('[href]')In our analogy, the jQuery traversal method we use is the equivalent of the axis: .children() corresponds to the XPath child axis, and .find() to the descendant axis.
The selector we pass to the jQuery methods is transparently equivalent to the XPath node test, although jQuery doesn't allow us to select text nodes at all.
The jQuery .filter() method stands in for the XPath predicate, although in this case again jQuery is much less powerful than XPath.
Note that the .children('em') call above will match all <em> children of all the matched <p> tags; this is exactly how XPath axes work as well.
OK, back to the beginning-of-line thing
Now that we've got a solid grasp of how XPath actually works, let's put it to work to solve the beginning-of-line problem. To start, we'll simplify the problem even further and just look for the first text node in each paragraph:
/descendant-or-self::node()/child::p/child::text()[position()=1]Translated into English, we're saying:
- Find all the nodes in the document
- Find all the children of those nodes that are paragraphs
- Find all the children of those paragraphs that are text nodes
- And filter those text nodes down to only the first match
Note that the position() function is evaluated within each matching node from the context: so the filter is saying "first text child of each paragraph," not "first text child of a paragraph in the entire document."
This is a good start, but it's not going to match text nodes that are deeply nested inside paragraphs. To do that, let's change our child axis to descendant:
/descendant-or-self::node()/child::p/descendant::text()[position()=1]Now we're really cooking! The beauty of putting a position filter on a descendantaxis is that it gives us exactly what we want, namely "the first text node descendant of each paragraph."
Now we just need to account for line breaks. To start, let's just build a separate expression that matches the first text node following a line break. I'll divide this one into multiple lines (perfectly legal in XPath) since it's a bit long:
/descendant-or-self::node()/child::br/following-sibling::node()[position=1]/descendant-or-self::text()[position()=1]Again, breaking it down:
- Find all the nodes
- Find children of those nodes that are
brelements - Find all the siblings that follow the
brelement - Filter those down to only the immediate next sibling of each
br - Find all the text nodes that are either the first sibling we already matched,
or one of its descendants - Filter those text nodes down to only the first one we matched
Voila! Now we've got a way to select both newlines inside p tags and newlines after br tags. Time to combine this all into a single expression:
(/descendant-or-self::node()/child::p|/descendant-or-self::node()/child::br/following-sibling::node()[position=1])/descendant-or-self::text()[position()=1]To put the finishing touches on it, let's tighten up the expression a little bit with some shorthand, and collapse a couple of the steps:
(//p|//br/following-sibling::node()[position=1])/descendant-or-self::text()[position=1]That's a pretty concise expression of a fairly sophisticated concept. And if we wanted to handle more ways of generating "lines" on the screen, we could simply add more element names to the list of possible matches.
So what does all this get us?
Hopefully at this point XPath is making some sense, but why go through all that when we had a perfectly good, easily understandable Ruby implementation?
For the most part, Ruby is best used for high-level code like business logic, integrating various application components, and writing rich domain models. The best Ruby code is about intent, not implementation. That's why it's so unsatisfying to use Ruby to do something as low-level and application-agnostic as traversing a DOM tree looking for nodes with a certain property.
One good reason to use XPath might be speed: Nokogiri's XPath traversal is implemented by libxml, and native code is fast! However, for this particular task, XPath is actually considerably slower than the pure-Ruby implementation. Interestingly, that's not true if you take out the <br> part and only look for text at the beginning of paragraphs. My guess is that the following-sibling axis is the culprit, since it has to select all the following siblings of the br tags, and then filter them down to only the first sibling.
So XPath may be slower or may be faster depending on exactly what you're doing, but it’s only hard to read if you’re new to it. It's a purpose-built tool, allowing you to traverse the DOM in any way you'd like with expressions that are compact and idiomatic.