Getting started with microformats2

2014-03-05 14:12

Classic microformats have been serving the web community’s need to extend HTML’s expressive power since 2004. Through an evolutionary, open, rigorous community process and human-first design principles, structured use of the class and rel attributes have paved the cowpaths of publishing data about people, places, events, reviews, products and more.

Microformats2 is the next big effort by the community to improve how microformats are authored, parsed and defined. Version two has multiple working open source implementations which independents are using in production and is easier to publish and consume than ever.

In this series of guides I’ll show you how to be the next site publishing and consuming microformats.

Part 1: This post
Part 2: How to Consume Microformats 2 Data

You can see how a microformats parser sees your markup by pasting any of the code samples below into this php-mf2 sandbox. Go ahead and experiment with adding more properties and see what happens!

Incremental Steps

In order to demonstrate some of the differences between microformats 2 and Classic Microformats/other competing technologies, I’ll use the process of content-out markup — going from plain text to HTML and finally adding a sprinkling of microformats. Let’s start with my favourite example: mentioning a person.

As plain text:

Barnaby Walters

With HTML:

<a href="http://waterpigs.co.uk">Barnaby Walters</a>

With classic microformats:

<span class="vcard"><a class="fn n url" href="http://waterpigs.co.uk">Barnaby Walters</a></span>

That’s 37 extra characters and a whole extra nested element just to say “This link is to a person”, not to mention the strangely named root classname (vcard? I thought this was an hcard?) and multiple cryptic fn n classnames. Competing technologies are typically even longer and messier.

With microformats 2 this all becomes much simpler:

<a class="h-card" href="http://waterpigs.co.uk">Barnaby Walters</a>

Weighing in at only 15 characters, this is quicker to type, easier on the eyes and easier to remember.

There are two fundamental changes in microformats 2 which make this helium-esque lightness possible: Implied Properties and prefixed classnames.

Implied Properties

When you give class=h-card to an element, you’re saying “This element represents a person”. In many cases the element will be simple; just a name, perhaps with a link or photo. Why should you add extra elements and classnames just to tell a dumb computer which bit is the name, URL or photo URL when that information is already expressed by the markup?

Implied properties save you from this tedium. When you specify an element as an h-card without explicitly defining which parts are the name, url or photo url, the parser will figure out what you meant. And it’s not just for h-cards either — thanks to the new generic parsing in microformats 2, this shorthand works for any microformat.

Prefixed Classnames

Classic microformats used plain classnames which looked like any other (e.g. vcard, n or note). There were a few problems with this — classnames would clash, cause false positives or be thrown away by developers who weren’t microformats-aware (“these classnames aren’t doing anything!”).

This also meant parsers were tricky to write, as each one had to maintain a long list of classnames used by each microformats, resulting in many parsers quickly going out of date.

Prefixing classnames solves both of these problems: semantic microformats2 classnames are set apart from styling hooks, and parsers can figure out which classnames to look for, cutting down on maintenance. There are 5 prefixes:

h-* root classnames specify that an element is a microformat, e.g. <span class="h-card">
p-* specifies an element as a plain-text property, e.g. <span class="p-name">My Name</span>
u-* parses an element as a URL, e.g. <a class="u-url" href="/"></a>
dt-* parses an element as a date/time, e.g. <time class="dt-published" datetime="2013-05-02 12:00:00" />
e-* parses an element’s whole inner HTML, e.g. <div class="e-content">

I’ll demonstrate the use of all of these prefixes with some real-world examples.

Firstly, another h-card, more fleshed out than the earlier example. This might be the sort that you put on your homepage:

<div class="h-card">
    <p><img class="u-photo" href="/me.png" alt="" /></p>
    
    <p class="p-name"><a href="u-url" href="http://waterpigs.co.uk">Barnaby Walters</a></p>
</div>

p-name, u-url and u-photo are fairly standard properties you’ll see over and over again. Another improvement in microformats 2 is increased consistency between different microformat specifications — again, making them easier to authors to remember and consumers to understand. A nice side effect is that a single element can be more than one type of microformat at once — for example a h-entry and h-review.

To demonstrate dt-* and e-*, here’s a note (like a tweet or short blog post), marked up using h-entry:

<article class="h-entry">
    <div class="e-content p-summary p-name">
        <p>
            Just writing a guide to using <a href="http://microformats.org/wiki/microformats-2">microformats-2</a>
        </p>
    </div>
    
    <time class="dt-published" datetime="2013-05-01 12:00:00">20 minutes ago</time>
</article>

Here, I want the HTML inside the content to be passed to the parser, so I mark it up as e-*. Notice I’m also specifying that content as the summary and name — one element can be parsed as multiple properties.

I’m using the time element to mark up the time this note was published. Because I’ve prefixed the classname with dt- and it’s on a time element, parsers know to look in the datetime attribute if it exists.

Combining Microformats

The third area in which microformats2 improves on the previous version is in combining microformats and making each microformat specification more reusable — for example, both a person or an event might have an address, so it makes sense to reuse the same markup for both.

There are many reasons to combine microformats — say you want to specify the author of a blog post or review. You would do so by making the p-author of the post an h-card:

<article class="h-entry">
    <p class="p-author h-card">Barnaby Walters</p>
    <p class="p-content">Blah blah blah</p>
    …

Or a comment on an article, via h-cites as p-comments:

    …
    <article class="p-comment h-cite">
        <p class="p-author h-card">Jón Jónsson</p>
        <p class="p-summary">Woah that’s insightful.</p>
        <p>
            <a class="u-url" href="http://jonsson.com/replies/1">
                <time class="dt-published" datetime="2014-03-01T14:00:25+00:00>2014-03-01 14:00</time>
            </a>
        </p>
    </article>
</article>

The reason comments are h-cites instead of h-entries is that h-entry implies syndication — it’s something you’ve posted, or have re-posted, whereas a comment should be a reference to a post on another site.

Or the address of a person or event, using h-adr:

<div class="h-event">
    <h1 class="p-name">Microformats Meetup</h1>
    <p>Join us at <b class="p-adr h-adr">
        <span class="p-street-address">Some Bar</span>,
        <span class="p-locality">Someplace</span></b>
    </p>
</div>

Incremental Steps

As plain text:

Barnaby Walters

With HTML:

<a href="http://waterpigs.co.uk">Barnaby Walters</a>

With classic microformats:

<span class="vcard"><a class="fn n url" href="http://waterpigs.co.uk">Barnaby Walters</a>

With microformats 2 this all becomes much simpler:

<a class="h-card" href="http://waterpigs.co.uk">Barnaby Walters</a>

Weighing in at only 15 characters, this is quicker to type, easier on the eyes and easier to remember.

There are two fundamental changes in microformats 2 which make this helium-esque lightness possible: Implied Properties and prefixed classnames.

Implied Properties

Prefixed Classnames

This also meant parsers were tricky to write, as each one had to maintain a long list of classnames used by each microformats, resulting in many parsers quickly going out of date.

h-* root classnames specify that an element is a microformat, e.g. <span class="h-card">
p-* specifies an element as a plain-text property, e.g. <span class="p-name">My Name</span>
u-* parses an element as a URL, e.g. <a class="u-url" href="/"></a>
dt-* parses an element as a date/time, e.g. <time class="dt-published" datetime="2013-05-02 12:00:00" />
e-* parses an element’s whole inner HTML, e.g. <div class="e-content">

I’ll demonstrate the use of all of these prefixes with some real-world examples.

Firstly, another h-card, more fleshed out than the earlier example. This might be the sort that you put on your homepage:

<div class="h-card">
  <p><img class="u-photo" href="/me.png" alt="" /></p>
  <p class="p-name">
    <a href="u-url" href="http://waterpigs.co.uk">Barnaby Walters</a>
  </p>
</div>

p-name, u-url and u-photo are fairly standard properties you’ll see over and over again. Another improvement in microformats 2 is increasing consistency between different microformat specifications — again, making them easier to authors to remember and consumers to understand. A nice side effect is that a single element can be more than one type of microformat at once — for example a h-entry and h-review.

To demonstrate dt-* and e-*, here’s a note (like a tweet or short blog post), marked up using h-entry:

<article class="h-entry>
  <div class="e-content p-summary p-name">
    <p>Just writing a guide to using <a href="http://microformats.org/wiki/microformats-2">
microformats-2</a></p>
  </div>
  <time class="dt-published" datetime="2013-05-01 12:00:00">20 minutes ago</time> 
</article>

Combining Microformats

There are many reasons to combine microformats — say you want to specify the author of a blog post or review. You would do so by making the p-author of the post an h-card:

<article class="h-entry">
  <p class="p-author h-card">Barnaby Walters</p>
  <p class="p-content">Blah blah blah</p>
  …

Or a comment on an article, via h-cites as p-comments:


  …
  <article class="p-comment h-cite">
    <p class="p-author h-card">Jón Jónsson</p>
    <p class="p-summary">Woah that’s insightful.</p>
    <p><a class="u-url" href="http://jonsson.com/replies/1">
      <time class="dt-published" datetime="2014-03-01T14:00:25+00:00>
        2014-03-01 14:00
      </time>
    </a></p>
  </article>
</article>

The reason comments are h-cites instead of h-entrys is that h-entry implies syndication — it’s something you’ve posted, or have re-posted, whereas a comment is a reference to

Or the address of a person or event, using h-adr:

<div class="h-event">
  <h1 class="p-name">Microformats Meetup</h1>
  <p>Join us at <b class="p-adr h-adr">
    <span class="p-street-address">Some Bar</span>,
    <span class="p-locality">Someplace</span></b>
  </p>
</div>

Choose a Parser

To turn a web page containing data marked up with microformats 2 (or classic microformats, if supported) into a canonical MF2 JSON data structure, you’ll need a parser.

At the time of writing, there are actively supported microformats 2 parsers available for the following programming languages:

Parsers for various other languages exist, but might not be actively supported or support recent changes to the parsing specification.

There are also various websites which you can use to experiment with microformats markup without having to download a library and write any code:

My own live-updating php-mf2 sandbox
The various parser comparison tools hosted on microformats.io
Aaron Parecki’s pin13.net microformats parser for parsing either URLs or HTML fragments

If there’s not currently a parser available for your language of choice, you have a few options:

Call the command-line tools provided by one of the existing libraries from your code, and consume the JSON they provide
Make use of one of the online mf2 parsers capable of parsing sites, and consume the JSON it returns (only recommended for very low volume usage!)
Write your own microformats 2 parser! There are plenty of people happy to help, and a language-agnostic test suite you can plug your implementation into for testing.

Considerations During Fetching and Parsing

Most real-world microformats data is fetched from a URL, which could potentially redirect to a different URL one or more times. The final URL in the redirect chain is called the “effective URL”. HTML often contains relative URLs, which need to be resolved against a base URL in order to be useful out of context.

If your parser has a function for “parsing microformats from a URL”, it should deal with all of this for you. If you’re making the request yourself (e.g. to use custom caching or network settings) and then passing the response HTML and base URL to the parser, make sure to use the effective URL, not the starting URL! The parser will handle relative URL resolution, but it needs to know the correct base URL.

When parsing microformats, an HTTP request which returns a non-200 value doesn’t necessarily mean that there’s nothing to parse! For example, a 410 Gone response might contain a h-entry with a message explaining the deletion of whatever was there before.

Storing Raw HTML vs Parsed Canonical JSON vs Derived Data

When consuming microformats 2 data, you’ll most often be fetching raw HTML from a URL, parsing it to canonical JSON, then finally processing it into a simpler, cleaned and sanitised format ready for use in your website or application. That’s three different representations of the same data — you’ll most likely end up storing the derived data somewhere for quick access, but what about the other two?

Experience shows that, over time:

the way a particular application cleans up mf2 data will be tweaked and improved as you add new features and handle unexpected edge-cases
mf2 parsers gradually get improved, fixing bugs and occasionally adding entirely new features.

Therefore, if it makes sense for your use case, I recommend archiving a copy of the original HTML as well as your derived data, leaving out the intermediate canonical JSON. That way, you can easily create scripts or background jobs to update all the derived data based on the original HTML, taking advantage of both parser improvements and improvements to your own code at the same time, without having to re-fetch potentially hundreds of potentially broken links.

As mentioned in the previous section, if you archive original HTML for re-parsing, you’ll need to additionally store the effective URL for correct relative URL resolution.

For some languages, there are already libraries (such as XRay for PHP) which will perform common cleaning and sanitisation for you. If the assumptions with which these libraries are built suit your applications, you may be able to avoid a lot of the hard work of handling raw microformats 2 data structures!

If not, read on…

Navigating Microformat Structures

A parsed page may contain a number of microformat data structures (mf structs), in various different places.

Take a look at the parsed canonical microformats JSON for the article you’re reading right now, for example.

items is a list of top-level mf structs, each of which may contain nested mf structs either under their properties or children keys.

Each individual mf struct is guaranteed to have at least two keys, type and properties. type is the primary way of identifying what sort of thing that struct represents (e.g. a person, a post, an event). Structs can have more than one type if they represent multiple things at once without wanting to nest them — for example, a post detailing an event might be both a h-entry and a h-event at the same time. Structs can also have additional top-level keys such as id and lang.

Generally speaking, type information is most useful when dealing with top-level mf structs, and mf structs nested under a children key. Nested mf structs found in properties will also have type information, but their usage is usually implied by the property name they’re found under.

For many common use cases (e.g. a homepage feed and profile) there are several different ways people might nest mf structs to achieve the same goals, so it’s important that your code is capable of searching the entire tree, rather than just looking at the top-level mf structs. Never assume that the microformat struct you’re looking for will be in the top-level of the items list! You need to search the whole tree.

I recommend writing some functions which can traverse a mf tree and return all structs which match a filtering callback. This can then be used as a basis for writing more specific convenience functions for common tasks such as finding all microformats on a page of a particular type, or where a certain property matches a certain value.

See my microformats2 PHP functions for some working examples.

Possible Property Values

Each key in a mf struct’s properties dict maps to a list of values for that property. Every property may map to multiple values, and those values may be a mixture of any of the following:

A plain string value, containing no HTML, and leaving HTML entities unescaped (e.g. <)

{
  "items": [{
    "type": ["h-card"],
    "properties": {
      "name": ["Barnaby Walters"]
    }
  }]
}

(In future examples I will leave out the encapsulating {"items": [{"type": [•••], •••}]} for brevity, focusing on the properties key of a single mf struct.)

An embedded HTML struct, containing two keys: html, which maps to an HTML representation of the property, and value, mapping to a plain text version.

"properties": {
  "content": [{
    "html": "<p>The content of a post, as <strong>raw HTML</strong> (or not).</p>",
    "value": "The content of a post, as raw HTML (or not)."
  }]
}

An img/alt struct, containing the URL of a parsed image under value, and its alt text under alt.

"properties": {
  "photo": [{
    "value": "https://example.com/profile-photo.jpg",
    "alt": "Example Person"
  }]
}

A nested microformat data structure, with an additional value key containing a plaintext representation of the data contained within.

"properties": {
  "author": [{
    "type": ["h-card"],
    "properties": {
      "name": ["Barnaby Walters"]
    },
    "value": "Barnaby Walters
  }]
}

All properties may have more than one value. In cases where you expect a single property value (e.g. name), simply take the first one you find, and in cases where you expect multiple values, use all values you consider valid. There are also some cases where it may make sense to use multiple values, but to prioritise one based on some heuristic — for example, an h-card may have multiple url values, in which case the first one is usually the “canonical” URL, and further URLs refer to external profiles.

Let’s look at the implications of each of the potential property value structures in turn.

Firstly, Never assume that a property value will be a plaintext string. Microformats publishers can nest microformats, embedded content and img/alt structures in a variety of different ways, and your consuming code should be as flexible as possible.

To partially make up for this complexity, you can always rely on the value key of nested structs to provide you with an equivalent plaintext value, regardless of what type of struct you’ve found.

When you start consuming microformats 2, write a function like this, and get into the habit of using it every time you want a single, plaintext value from a property:

def get_first_plaintext(mf_struct, property_name):
  try:
    first_val = mf_struct['properties'][property_name][0]
    if isinstance(first_val, str):
      return first_val
    else:
      return first_val['value']
  except (IndexError, KeyError):
    return None

Secondly, Never assume that a particular property will contain an embedded HTML struct — this usually applies to content, but is relevant anywhere your application expects embedded HTML. If you want to reliably get a value encoded as raw HTML, then you need to:

Check whether the first property value is an embedded HTML struct (i.e. has an html key). If so, take the value of the html key
Otherwise, get the first plaintext property value using the approach above, and HTML-escape it
If neither is found, the property has no value.

In Python 3.5+, that could look something like this:

from html import escape

def get_first_html(mf_struct, property_name):
  try:
    first_val = mf_struct['properties'][property_name][0]
    if isinstance(first_val, dict) and 'html' in first_val:
      return first_val['html']
    else:
      plaintext_val = get_first_plaintext(mf_struct, property_name)

      if plaintext_val is not None:
        plaintext_val = escape(plaintext_val)

      return plaintext_val
  except (IndexError, KeyError):
    return None

In some cases, it may make sense for your application to be aware of whether a value was parsed as embedded HTML or a plain text string, and to store/treat them differently. In all other cases, always use a function like this when you’re expecting embedded HTML data.

Thirdly, when expecting an image URL, check for an img/alt structure, falling back to the plain text value (and either assuming an empty alt text or inferring an appropriate one, depending on your specific use case). Something like this could be a good starting point:

def get_img_alt(mf_struct, property_name):
  try:
    first_val = mf_struct['properties'][property_name][0]
    if isinstance(first_val, dict) and 'alt' in first_val:
      return first_val
    else:
      plaintext_val = get_first_plaintext(mf_struct, property_name)

      if plaintext_val is not None:
        return {'value': plaintext_val, 'alt': ''}

      return None
  except (IndexError, KeyError):
    return None

Finally, in cases where you expect a nested microformat, you might end up getting something else. This is the hardest case to deal with, and the one which depends the most on the specific data and use-case you’re dealing with. For example, if you’re expecting a nested h-card under an author property, but get something else, you could use any of the following approaches:

If you got a plain string which doesn’t look like a URL, treat it as the name property of an implied h-card structure with no other properties (and if you need a URL, you could potentially take the hostname of the effective URL, if it works in context as a useful fallback value)
If you got an img alt struct, you could treat the value as the photo property, the alt as the name property, and potentially even take the hostname of the photo URL to be the implied fallback url property (although that’s pushing it a bit, and in most cases it’s probably better to just leave out the url)
If you got an embedded HTML struct, take its plaintext value and use one of the first two approaches
If you got a plain string, check to see if it looks like a URL. If so, fetch that URL and look for a representative h-card to use as the author value
If you get an embedded mf struct with a url property but no photo, you could fetch the url, look for a representative h-card (more on that in the next section) and see if it has a photo property
Treat the author property as invalid and run the h-entry (or entire page if relevant) through the authorship algorithm

The first three are general principles which can be applied to many scenarios where you expect an embedded mf struct but find something else. The last three, however, are examples of a common trend in consuming microformats 2 data: for many common use-cases, there are well-thought-through algorithms you can use to interpret data in a standardised way.

Know Your Algorithms and Vocabularies

The authorship algorithm mentioned above is one of several more-or-less formally established algorithms used to solve common problems in indieweb usages of microformats 2. Some others which are worth knowing about include:

“Who wrote this post?”: authorship algorithm
“There’s more than one h-card on this page, which one should I use?”: representative h-card
“I want to get a paginated feed of posts from this page”: How to consume h-feed
“How do I find and display the main post on this page?”: How to consume h-entry
“I received a response to one of my posts via webmention, how do I display it?”: How to display comments

Library implementations of these algorithms exist for some languages, although they often deviate slightly from the exact text. See if you can find one which meets your needs, and if not, write your own and share it with the community!

In addition to the formal consumption algorithms, it’s worth looking through the definitions of the microformats vocabularies you’re using (as well as testing with real-world data) and adding support for properties or publishing techniques you might not have thought of the first time around. Some examples to get you started:

If an h-card has no valid photo, see if there’s a valid logo you can use instead
When presenting a h-entry with a featured photo, check both the photo property and the featured property, as one or the other might be used in different scenarios
When dealing with address or location data (e.g. on an h-card, h-entry or h-event), be aware that either might be present in various different forms. Co-ordinates might be separate latitude and longitude properties, a combined plaintext geo property, or an embedded h-geo. Addresses might be separate top-level properties or an embedded h-adr. There are many variations which are totally valid to publish, and your consuming code should be as liberal as possible in what it accepts.
If a h-entry contains images which are marked up with u-photo within the e-content, they’ll be present both in the content html key and also under the photo property. If your app shows the embedded content HTML rather than using the plaintext version, and also supports photo properties (which may also be present outside the content), you may have to sniff the presence of photos within the content, and either remove them from it or ignore the corresponding photo properties to avoid showing photos twice.

Sanitise, Validate, and Truncate

In the vast majority of cases, consuming microformats 2 data involves handling, storing and potentially re-publishing untrusted and potentially dangerous input data. Preventing XSS and other attacks is out of the scope of the microformats parsing algorithm, so the data your parser gives you is just as dangerous as the original source. You need to take your own measures for sanitising and truncating it so you can store and display it safely.

Covering every possible injection and XSS attack is out of the scope of this article, so I highly recommend referring to the OWASP resources on XSS Prevention, Unicode Attacks and Injection Attacks for more information.

Other than that, the following ideas are a good start:

Use plaintext values where possible, only using embedded HTML when absolutely necessary
Pass everything (HTML or not) through a well-respected HTML sanitizer such as PHP’s HTML Purifier. Configure it to make sure that embedded HTML can’t interfere with your own markup or CSS. It probably shouldn’t contain any javascript ever, either.
In any case where you’re expecting a value with a specific format, validate it as appropriate.
More specifically, everywhere that you expect a URL, check that what you got was actually a URL. If you’re using the URL as an image, consider fetching it an checking its content type
Consider either proxying resource such as images, or storing local copies of them (reducing size and resolution as necessary), to avoid mixed content issues, potential attacks, and missing images if the links break in the future.
Decide on relevant maximum length values for each separate piece of external content, and truncate them as necessary. Ideally, use a language-aware truncation algorithm to avoid breaking words apart. When the content of a post is truncated, consider adding a “Read More” link for convenience.

Test with Real-World Data

The web is a diverse place, and microformats are a flexible, permissive method of marking up structured data. There are often several different yet perfectly valid ways to achieve the same goal, and as a good consumer of mf2 data, your application should strive to accept as many of them as possible!

The best way to test this is with real world data. If your application is built with a particular source of data in mind, then start off with testing it against that. If you want to be able to handle a wider variety of sources, the best way is to determine what vocabularies and publishing use-cases your application consumes, and look at the Examples sections of the relevant indieweb.org wiki pages for real-world sites to test your code against.

Don’t forget to test your code against examples you’ve published on your own personal site!

Next Steps

Hopefully this article helped you avoid a lot of common gotchas, and gave you a good head-start towards successfully consuming real-world microformats 2 data.

If you have questions or issues, or want to share something cool you’ve built, come and join us in the indieweb chat room.

2022-02-19 11:48

Fundor 333 2022-02-24 19:55

A complete and easy guide to MicroFormat and how to use them. The article isn't new but it is easier to understand than the official site explanation about what is MicroFormat and how to use theme

https://fundor333.com/

2023-02-12 20:56

Barnaby Walters waterpigs.co.uk

Incremental Steps

Implied Properties

Prefixed Classnames

Combining Microformats

Further Reading