This blog post is based on a couple of things. The first was correspondence about a problem a retailer contacted us about: Their new in-house ONIX 3.0 metadata system had made an ONIX specification-based assumption and they didn’t understand why a composite, Block 6’s “Market Date”, wasn’t typically populated.
The second thing comes from committee work where publishers and retailers advocate for using fully implemented, good-quality ONIX while arguing against or being surprised by the functional expectations of ONIX 3.0.
I want to be clear, this isn’t a one is right and the other is wrong scenario. It costs money to do metadata well and not everything can be covered. Choices need to be made. But here’s my point: Those choices shouldn’t be led by our experience using ONIX 2.1. Developers tend to repeat what they know (ONIX 2.1) when ONIX 3.0 offers simpler and better solutions in a different place.
We (those of us who make decisions about metadata) need to instruct our developers on where they should be mapping and why. We need to do that because “good ONIX” depends on implementing the system, the metadata system that EDItEUR has spent more than 20 years developing and that the ONIX standard represents.
ONIX is not complex, but what ONIX supports — books and a publishing supply chain where a huge number of vastly diverse products are sold in relatively small numbers — is complex. Therefore, the metadata system needs to support complexity.
Going back to the first thing that led me to write this blog post, the retailer’s design assumptions were right, sensible, and true to the ONIX system. What my committee colleagues discuss is data as it’s used. That cannot be wrong because data-that’s-being-accepted-in-the-supply-chain can never be “wrong” however not-right it might be.
The difference between those two things is what defines “good ONIX”, and “Market Date” is a nice example. It’s nice because what you supported in ONIX 2.1 can be used in ONIX 3.0, almost without change as data — it’s the ONIX structure that’s different. ONIX 3.0 allows the data to be simpler with less repetition and usable with greater accuracy. And there’s an educational bonus to it: It forces you to consider what is a market relative to your book selling needs.
Unfortunately, the data-that’s-being-accepted-in-the-supply-chain is modelled on ONIX 2.1 practices that, frankly, were not following EDItEUR’s system very well even for that version. When such practices are implemented in ONIX 3.0 they might not be wrong, but they definitely cannot be described as good.
EDItEUR’s ONIX Specifications and Best Practices
I’m going to start by highlighting and providing my annotations on what’s said in the Specifications and Best Practices as published by EDItEUR to show why a company would expect to find a “Market Date” composite in ONIX 3.0 metadata.
An ONIX record SHOULD contain at least one “Product Supply” composite.
This means that it's not mandatory that an ONIX record contains a “Product Supply” but, a book record without supply information is functionally useless to support book sales. EDItEUR designs ONIX to support a number of use cases and there are atypical ones that don’t involve supplying a product to a retailer, making the “Product Supply” statement unnecessary.
When EDItEUR says:
“SHOULD contain”, they mean always, except for an atypical metadata exchange, and they mean it in the context of the structure they’re describing. There are many “expected composites” that aren't mandatory in all ONIX records.
“MUST contain”, they mean mandatory in all cases, or within the context they’re describing it has to be there.
Having clarified our terms let’s continue down EDItEUR’s path using them:
A “Product Supply” composite SHOULD contain: the <Market> composite.
A “Market” composite MUST contain a “Territory” composite. (Ergo: A “Market” is a defined territory.)
EDItEUR provides a statement about when you can plausibly not provide a “Market” composite: ”<Market> may be omitted when the product is available in only a single market since the market would be identical to the <SalesRights>.” This use case implies a simplicity in “supply” that I don’t think applies to many physical books.
A “Product Supply” composite SHOULD contain: the <MarketPublishingDetail> composite.
A “Market Publishing Detail” composite MUST contain a “Market Publishing Status” — you need to define the publisher’s relationship to the book in the described territory.
A “Market Publishing Detail” composite SHOULD contain a “Market Date” composite. Market-specific dates are needed but there are publishing statuses like "Cancelled" or "Postpone indefinitely" where a date is meaningless and should not be provided.
You don't have to supply a "Market Publishing Detail" when the product is available only in a single market, but, again, I would say that print ISBNs are seldom sold in a single market.
A “Product Supply” composite MUST contain: the <SupplyDetail> composite with pricing details for the product in the market.
While a single exclusive supplier is typical of North American book metadata, the ONIX system can support multiple suppliers with different "supplier roles" where each supplier is able to find pricing specific to different parts of the market — which is of course defined by the “Product Supply”.
What’s important to note is that a “Product Supply” statement is designed to facilitate reporting on multiple suppliers. Even if you’re supporting a single supplier you should understand that the system can support more and respect its design needs.
I won’t carry on through the “Supply Detail” — a few points will come up later — but let’s loop back to “Product Supply”. It should be there and Block 6 is a special block in ONIX 3.0 because the block “Product Supply” can repeat (all other blocks do not repeat).
The reason Block 6 can repeat is that one “Product Supply” composite represents all the supply information for the market being described. The “Market” is a geographical description of that: it’s where all the “Product Supply” information is functionally the same. You can’t separate a “Product Supply” statement from its “Market Statement”, even if the record omits a “Market” statement, it’s omitted because as a “single market” other data in the ONIX record replaces it. The territory could be World or it could be a province but it usually is defined by a country or groups of countries where the publisher has a business relationship with trading partners whose “where” defines the market and the contracts it implies.
The simple case would be an exclusive supplier in one territory and a different exclusive supplier in another territory. Those would be mutually exclusive distinct markets that should be provided in different “Product Supply” composites and defined status and dates can be conveniently and easily defined. Even without “exclusive” supply roles it’s possible that the details and dates will be different between territories. For example, ONIX offers support for information down to the level of Publisher Representative and contacts so you might need to define a territory based on that instead of the supplier.
A “Product Supply” can easily hold multiple “Supply Details” each defined by their unique role. It’s normal in North American metadata to only list the typically “exclusive” supplier but EDItEUR has designed the “Product Supply” to allow for the possibility of providing data from the wholesalers they supply. It’s what you need to and choose to trade as metadata that defines what a market is. Each market is a different “Product Supply” statement.
Stop and think about that for a second: ONIX 3.0 offers a simpler way for you to focus on the business-based information you need to supply by territory.
I hear the screams: We can’t change! It costs us! I know that most companies supply market-driven files because BiblioShare receives files made to support the Canadian market. Here I’ll just note that you don’t need to distribute every “Product Supply” to every trading partner. You send them the one(s) they need for the ISBN-based records for the market(s) they serve. You could simplify your metadata exchange with not much development work to use “Product Supply” fully. Simplicity will provide a return on investment long term.
All of this is normal in good ONIX.
A “Product Supply” composite should contain a “Market” composite that must contain a “Territory” statement. It should define the publisher’s relationship to the product in that market so that data must contain a defining status and should contain market-specific dates. Block 6 must contain a supplier with a defined role relative to the market and an availability status so a retailer in the market area knows who to buy from and if it can be supplied. That “Supply Detail” must contain some form of price (price can include an unpriced item code) and should contain other things that we won’t cover here today.
What is good ONIX?
An ONIX file has to be valid as an XML document. That means that its tag structure has to be readable and sound, codes need to match their code lists, and for any part of an ONIX record that you use all of the embedded tags MUST include data points. Missing or mismatched bits violate the XML defining document, a.k.a. the schema, which is where the business rules of the ONIX “system” are defined. To say that an ONIX file is valid means it’s valid as an XML document — if your file is compared to the schema’s rules using XML software, it “passes” the rules set by EDItEUR. This represents the minimum, the least you can do and both EDItEUR and BookNet Canada recommend that everyone knows how to validate their ONIX files (or know that your software does it).
In addition to being valid as XML, “good ONIX” means that if you choose to communicate about something in ONIX you SHOULD include the supporting “should contain” data points. The exact ONIX record needed to support any given ISBN-based product will vary by what you need to describe.
Before this is taken too dogmatically it would be true to say some “should contain” statements within ONIX’s structure are more important than others, but I have to emphasize that supporting “Markets” and a “Market Date” is basic in supporting “good ONIX”. The reason I say that is because a “Product Supply” statement is required to communicate basic information needed to sell books. Any “should contain” statement in ONIX’s business rules associated with basic information is similarly basic enough to be fundamental to “good ONIX”.
The retailer’s assumption that “Market Date” is an “expected composite” in ONIX 3.0 metadata is sound and based on ONIX best practices. Before I move to why this is better and more like how we handle data in North America, let’s consider why the retailer contacted BookNet Canada: The expected data was missing.
Reality check on data-that’s-being-accepted-in-the-supply-chain: ONIX 3.0 numbers delivered to BiblioShare
When looking at the ONIX 3.0 records delivered to BiblioShare, I found that:
Out of over 630,000 ONIX 3.0 records, all carry at least one Block 6 “Product Supply”
93% carried a single Block 6 and the balance carried two to five Block 6 entries
<45% of the Block 6 entries carried a “Market” composite:
Virtually all of the records with multiple Block 6 entries carried a “Market” composite
>47% of the records with a single Block 6 carried a “Market” composite
<11% of the records carried a “Market Publishing Detail” composite
<10% of the records carried a “Market Date”:
Virtually all dates supplied are “Market Publication Date” and I did not compare that value to the “Global Publication Date” found in Block 4
16 records carry a “Sales Embargo Date” as a “Market Date” entry
All of Block 6 carry one or more “Supply Details” and just over 50% of the records carry a “Supply Date”
>37% — more than 165,000 dates were a Sales Embargo Date as a Supplier Date entry
I’ve made a spreadsheet available here where I share in a lot more detail the counts of ONIX 3.0 records BiblioShare has received. If you feel I’m leading you to a conclusion (I hope I am!) look for additional support for (or against) it in the weeds I’ve provided.
A quick note about the data: the counts were made against file-based metadata, so if an ISBN was supported by two (or more) accounts, each account represents one record. That means the counts are not “unique by ISBN” but are “unique by ONIX record”. The amount of ISBN duplication is under 3% and many if not most ONIX records represent a unique representation and the data practices of its individual sender.
I cannot think of a better example of a piece of “market metadata” than a Sales Embargo Date (On-Sale Date). Its purpose is to ensure that within some territories all retailers have an equal opportunity for sales by sharing the same start date. How do we reconcile that with these numbers?
16 records carry a “Sales Embargo Date” as a “Market Date” entry
>37% — more than 165,000 dates were a “Sales Embargo Date” as a “Supplier Date” entry
The reason companies are using a “Supplier Date” entry is because this date was contained within the “Supply Detail” in ONIX 2.1. Companies are not recognizing — or taking advantage of — the simplicity offered by the changes EDItEUR made in ONIX 3.0. The IT department repeats what it knows and, because no one asked them to do it differently, that becomes the data that arrives but not the ONIX that is good.
Isn’t “Supply Date” an option? It is an option. As noted above, EDItEUR has identified the simple case of a single market as a reason to not supply market-related details and if the “global” details are identical to the market details then this is just a way to simplify the metadata. Digital products would often meet that criteria and our typical practice of naming the publisher as the supplier would support a great example where both the “Sales Rights” and “Supply” statements are “publisher based.” If you’re naming other companies as the supplier ask yourself:
Does the publisher or the company(ies) named as “Supplier(s)” set the sales embargo date (On-sale date)?
In the North American market, the normal practice is that the publisher determines when retail sales start. The case for using the “Market Publishing Detail” in “good ONIX” is now pretty obvious.
Some data senders create ONIX files that have been manipulated to make the records a single market representation. I can find ISBNs in BiblioShare whose “Sales Rights” are specified as “Not for Sale” outside Canada but a quick Google search confirms the ISBN is available outside of Canada. I think companies doing this are wasting resources. It’s actually simpler to use “Product Supply” properly as your market divisions than to set up whole feeds that misrepresent your data. How do these files work for you when a retailer opens a store in another market? I’d think adding an existing “Product Supply” statement to a feed is easier than creating a third manipulated ONIX joint market feed for that retailer.
I even would suggest that if you don’t support the metadata feeds worldwide (other companies support the ISBN in Europe for example) it would still be more accurate to provide your home market details in a “Product Supply”. It reduces the potential for conflict should a retailer get both feeds.
The bottom line: “Supply Date” remains an option and is needed for supplier-specific dates like “Last date for returns.” As for “Sales Embargo Date”, it’s hard to call it “good ONIX” when used as a part of “Supplier Detail” assigned to a different company than the publisher but retailers would be wise to be prepared for its use there.
Is Canada a Market?
Canadian retailers have had complaints about the lack of actionable business dates in ONIX feeds for a long time. They don’t get the product when the dates say they should. Their buyers' plans are not managed as efficiently as they need. I cannot help but think that the US supply chain, with its high density, supporting multiple distribution locations represents a fundamentally different “market” than the lower density, long-distance one that exists in Canada.
Canadian publishers who support different dates for the Canadian market from the US complain that companies selling into Canada ignore the Canadian date and use the US date in Canada. This works to US companies' disadvantage given that the Canadian date for a Canadian-originated product is likely sooner than the one used in the US, but it still costs the publisher sales by mismatching on the publicity push. I cannot help but think that the ONIX 2.1 norm of sloppily using “Sales Rights” as proxy statements for distribution rights and the tendency by both US and Canadian ONIX creators to stuff their metadata together as a single market for convenience contributes. The fact that half of the ONIX 3.0 data in BiblioShare doesn’t even support a “Market” statement and less than 10% supports dates shows the same sloppiness carrying on.
The strange thing is that the sloppy data practices typical of ONIX 2.1 in North American metadata make for pretty good ONIX if, as EDItEUR intends, it’s mapped to “Product Supply” with “Market” details.
“Market Territory” statements are “simple”, positive statements of the territory and possibly a good match to the proxy for distributor statements often found in “Sales Rights”.
As noted above, foreign data senders make a substantial effort to “make” Canadian metadata out of their dataset as special file outputs. It would be better if that effort was put into supporting “Product Supply” statements as a solution for working in multiple markets. BookNet is in a good position to see the time spent by Canadian sales reps and branch publishers fixing foreign data for our market and that effort seems needed no matter which foreign market holds the source publisher. It should be easier to integrate that into updating a market specific Product Supply held by the creator than being overlayed after the fact.
I can only say that a Canadian retailer designed their system to get actionable business metadata from the appropriate ONIX data point; and that the data-that’s-being-accepted-in-the-supply-chain wasn’t provided there. If the data-that’s-being-accepted-in-the-supply-chain meets the criteria of being actionable business metadata isn’t easily determined outside of a business, but the care taken by the retailer to look to the “Market Date” points to their unmet need.
You can’t separate a “Market Territory” from a working “Product Supply” statement any more than you can understand “good ONIX” outside of accurate business communication.
Featuring River in an Ocean: Essays on Translation edited by Nuzhat Abbas.