Character encoding mismatches in ONIX

The following post was featured in the international ONIX implementation group — a mailing list that you really should follow for ONIX announcements and discussion. Want to stay on top of ONIX implementation issues in a global discussion led by EDItEUR? Join the group here and continue reading.

Nâzım Hikmet

Marie-Adélaïde Barthélemy-Hadot

‘Excellently written’ – Wisława Szymborska

It’s hard to make any sense of text like this. But it’s a sure sign of a mismatched character encoding. In fact, the above should look like this…

Nâzım Hikmet

Marie-Adélaïde Barthélemy-Hadot

‘Excellently written’ — Wisława Szymborska

The three names, of Turkish, French and Polish writers, require some special characters. And those characters can be “encoded” digitally in a variety of ways. Ultimately, for data to look correct, the encoding method needs to match all the way along the data supply chain. But you often see character encoding issues like this, in online bookstores and elsewhere, and a publisher asked about it earlier this week because author names on their own website did not match the names in their internal system.

In a publisher’s system, the publisher’s staff probably know how to make these names look correct in their own system. If the system exports ONIX, then the character encoding method used by that system is listed in the ONIX itself — it’s on the first line of the ONIX message:

<?xml version="1.0" encoding="UTF-8"?>

It needn’t always be UTF-8 (Unicode) — it can be Windows-1252 or a range of other encodings.

And then an ONIX recipient needs to respect that encoding when the ONIX data is imported into another system, for example, a retailer’s system or a website server. The retailer's system or webserver might use that same encoding internally, or they might use a different encoding — and if different, the encoding of the data needs to be explicitly changed into whatever the retail system uses, as it is imported.

The top example here illustrates what happens if the ONIX encoding is not respected as the data is imported. This is data that was encoded using UTF-8 Unicode in the ONIX, but was imported into a retailer system that uses Windows-1252 encoding as if it were already encoded using Windows-1252.

Conversion from one encoding into another is mostly straightforward for developers.

It’s a fact that some encodings are more comprehensive than others. Sometimes this means that you can have a character in the ONIX that cannot be ‘converted’ into whatever encoding is used by the recipient system because the encoding used in that recipient system doesn’t have an encoding for that particular character at all. Cyrillic or Arabic characters cannot be encoded using Windows-1252, for example. Unicode is the most universal, and forward-looking developers will mostly be using Unicode internally.

There is an appendix in the ONIX Implementation and Best Practice Guide that covers the question of character encodings in more detail.

And you might ask why the ONIX tags themselves — like <Header> or <PersonName> — don’t get affected by this? That’s a consequence of the fact that the ONIX tags themselves use only ASCII characters. Of course, the data within the tags can use any available characters. ASCII is a tiny subset of all the characters available, and for historical reasons almost every encoding method encodes them in the same way — so whichever character encoding you’re using, the tags themselves will remain recognizable. Whichever encoding you are using, <PersonName> will look like <PersonName>, even if Marie-Adélaïde Barthélemy-Hadot ends up looking like Marie-Adélaïde Barthélemy-Hadot.

Graham Bell is Executive Director of EDItEUR, responsible for the overall development of EDItEUR’s standards and the management services it provides on behalf of other standards agencies (including the International ISNI agency and the International DOI Foundation).

He joined EDItEUR as its Chief Data Architect in 2010, focused on the continuing development and application of ONIX for Books, and on other EDItEUR standards for both the book and serials sectors.