Vocabulary Drift Is an ML Bug: A Model Is Only as Right as the Word It Predicts
In 2019 a study in Science examined a widely used commercial algorithm of the kind health systems rely on to identify patients with complex health needs, and found a serious bias in it. The authors’ explanation was precise: the algorithm predicted health care costs rather than illness. Cost was a convenient label. It stood in for need, and the gap between the two was where the harm sat. The model did what it was trained to do, and the trouble was the word it had been trained on.
When I first encountered an example of vocabulary drift, I realised how easily such a small difference in a definition could result in a model predicting the wrong thing, because all the documents and data the model reads use abbreviations, and the developer of the model (or its user) simply thinks their definition is the only one that exists. To be accurate, the model must be about the right thing. Many organisations define “right” through an abbreviation, and multiple groups within the organisation define that abbreviation differently. Thus, the model correctly predicts the wrong event. This article discusses how that occurs, why the model will not tell you it is wrong, and what a development team can do before any training code exists.
One word, several meanings
Most organisations have at least one acronym similar to this. A common acronym or abbreviated name like AOD can carry many meanings within the same organisation. For example, AOD may mean Attempt of Delivery, After Order Discounts, Average Outage Duration and Always on Display. While all of these are legitimate and understandable to those who use them regularly, when they are brought into a meeting room together, they talk past each other, because they do not agree on what the abbreviation represents. For operational teams, AOD is a number (an integer), whereas for technical and SRE teams it is in seconds or minutes. That is when it became apparent. Every planning document therefore uses whatever interpretation its author intended. While no one is wrong individually within their own domain, everyone is wrong collectively. Vocabulary drift is typically silent because every usage is appropriate on its own page. It only shows when two pages are read side by side, and in most organisations there is generally no role or responsibility assigned to read both pages together.
A model on the wrong label still scores well
That is where the potential danger lies. A model trained on an inappropriate target still learns something. The model still generates a good score, and there is nothing in that score that indicates the target was wrong. Accuracy based on the wrong target appears identical to accuracy based upon the correct target until someone takes action on it. Google’s Rules of Machine Learning suggest selecting “a simple, observable and attributable metric” for a first objective. That is good counsel, and it quietly assumes that everyone agrees on what the metric means. A label created on the basis of the wrong field meets those criteria, but it remains a label based on the wrong field.
Several outcomes, not two
The most typical culprit here is a binary label that defines success or failure, because it collapses distinct outcomes. A delivery is an example. A delivery can be completed. It can also be dispatched and never reach the customer. Alternatively, it can be cancelled, often before anyone has been dispatched at all. Late is another example. By this particular vocabulary, a delivery that arrives after the deadline is considered a success, although that success is measured by a different metric from the other outcomes. These distinct outcomes have different owners, and they therefore need different interventions. A single label that combines them gives every owner a number that partly reflects somebody else’s issue.
When the column is meant, write the column
The correction required mainly involves discipline in how documentation is written. Business prose uses everyday words, such as “a delivery that failed at the door”. The score, the label, the features used to create it and any service associated with them use one common prefix, so that no one confuses them with anything else. Additionally, when a document refers to a raw source column in a database, the wording provided by the database itself must be kept, because documentation should never rename a column that already exists. That final rule seems overly formalistic. In reality, it is the one that identifies whether a label was defined using the wrong field, because it requires an author to state explicitly which column they are referring to. Most importantly, I feel strongly that you should use full names in your databases, which gives the best chance of eliminating ambiguity from the outset. Whether it is a person or an agent reading this data, both will get the meaning that was intended.
A glossary settles words, not intent
Eventually, a glossary will run into a document that uses a term in a way that nobody can decide. At this point there is a natural desire to rename it and move on. Instead, it is better to flag it for review by the owner of the document in question, and let them address the intent themselves. A glossary can resolve what a word means, but not what someone else was attempting to measure. More often than not, the person originally responsible for a term is no longer actively maintaining it. As such, it is preferable to confirm what was being referred to, rather than making assumptions.
Make the glossary binding
An optional terminology document does little more than provide suggestions. Your glossary should be mandatory for every planning document, as well as for every prompt given to an assistant. In conversational products, a vaguely defined term becomes a wrong answer given to an actual person. The costs of failing to do this are well documented. A 2021 study of machine learning practitioners identified data cascades, “compounding events causing negative, downstream effects from data issues”, and found them widespread, with 92% prevalence. A poorly defined term is clearly one place for such a cascade to begin.
Who should own it
Your glossary should have one named owner, but the larger decision is who formally accepts what a model attempts to measure. That role belongs to the owner of each business outcome, who accepts the definition of any label built on it, while the people developing the model own neither. Simply stated, no training target is defined in prose alone. Rather, it is defined against your glossary and the exact source column, and formally accepted before any training code is written. The early indicators of trouble are two separate documents that use the same term with different meanings while nobody is tasked with resolving it, or a target that has reached the coding stage but is still defined only by a planning document’s description.
Feature engineering begins with vocabulary, and the majority of label bugs ultimately turn out to be dictionary bugs. Just as it takes little effort to agree on a language before engaging in a dialogue, agreeing on vocabulary means that vague definitions are eliminated or resolved early on, which yields quality results from the outset.
Which word in your planning documents means one thing to finance and another to operations?