Goodhart's law in marketing: why your best number can eat your brand

Goodhart's law warns that a statistical relationship can deteriorate when used for control. In marketing, optimizing clicks, leads or AI citations can improve the indicator without improving buyer outcomes. Keep metrics as evidence: assign a human decision owner, protect independent evaluation, specify unacceptable trade-offs and stop when the work deteriorates.

Every marketer knows the report where everything is green while the business gets no better. That is neither bad luck nor bad measurement. It is what happens to any number you turn into a target. Profit has one dimension; people do not. How do you nourish something infinite by striving for a higher number?

What did Goodhart say, and what did Strathern add?

Goodhart’s law concerns a relationship that can change under control. The opening also raises a different question: whether a number can adequately express what a business should care about. That is a question of values. Neither question establishes that every target necessarily damages the work.

The historical reference is Charles Goodhart’s 1975 conference contribution, Problems of Monetary Management: The U.K. Experience. The Reserve Bank of Australia’s bibliography identifies the July 1975 conference and a revised collection published in 1976. The wording attributed to Goodhart appears in footnote 1 of Manheim and Garrabrant’s paper:

any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

I could not retrieve the 1975 original. This quotation was checked in that later paper, not against Goodhart’s original text. Repetition of an attribution should not be mistaken for direct verification.

The sentence most people recognize is in Marilyn Strathern’s 1997 article ‘Improving ratings’: audit in the British University system, page 308:

When a measure becomes a target, it ceases to be a good measure.

These are Strathern’s words on that page. She immediately refers to Keith Hoskin’s discussion of Goodhart. Quoting the sentence as Goodhart’s own wording collapses that distinction. Her argument also reaches beyond people gaming a score. Audit demands particular descriptions of an institution; practices can then be reorganized to make those descriptions possible. What is easy to demonstrate starts to shape what is done.

Our extension to marketing is that a reporting system can gradually acquire authority over the definition of useful work. An explanation that helps an unsuitable buyer withdraw may be excellent marketing and a poor contributor to a conversion target. If only measurable conversion gains justify a page, the organization can lose the capacity to recognize that explanation as valuable. The dashboard has begun to write the job description.

Donald Campbell’s Assessing the impact of planned social change appeared in 1979. His argument is that an indicator becomes more exposed to corruption as its use in social decision-making increases. In the primary text available here as a later reprint, page 34, the relevant phrase is:

the more subject it will be to corruption pressures

Campbell continues by warning about distortion of the social processes being monitored. That distinguishes damage to the indicator from damage to the activity. He also describes his supporting evidence there as predominantly anecdotal. The opened 2011 reprint identifies its original edition as 1976; I could not inspect the 1979 journal version. This is an influential analysis of institutional pressure, not an experimentally established universal law of marketing.

Which useful activity would your reporting system struggle to defend?

Four different ways a marketing number can fail

Manheim and Garrabrant’s taxonomy separates selection on noise, selection beyond a relationship’s valid range, interventions that change the relationship, and other actors redirecting the measure toward their own goals. They write: “These varied forms often occur together”. The marketing diagnoses below are ours. The underlying studies document the phenomena; they do not classify each case as a uniquely identified Goodhart mechanism.

Regressional: the winning observation includes favorable error. Picture the actual experimental setting studied by Robertson and colleagues: alternative headlines compete above the same news article. Their randomized experiments found that negative wording increased click-through. This is evidence about clicking. It does not establish increased understanding, brand trust or better purchasing decisions.

A separate problem arises when selecting the highest observed click-through rate. Our statistical explanation is that an observed rate contains both an underlying propensity and sampling error. Selecting its maximum rewards favorable error as well as genuine performance. The observed winner can therefore exaggerate the advantage that will survive independent repetition. This remains possible with accurate tracking and entirely honest people. It should not be confused with the study’s demonstrated effect of negative wording, which need not disappear when measured again.

Three claims have to stay separate: the clicks were recorded correctly; the estimate will repeat; clicking is a valid indicator of the desired outcome. Better instrumentation addresses the first. Independent evaluation helps with the second. Neither establishes the third. A perfectly repeatable way to attract clicks can still be a poor way to help a reader.

Extremal: the relationship changes outside the setting that supported it. Blake, Nosko and Tadelis’s paid-search field experiments found different effects across buyer groups. New and infrequent buyers were more influenced by advertising than frequent buyers. A click therefore did not carry a uniform implication for additional purchases. Our application is the scaling decision: evidence from a responsive population does not license indefinite expansion into populations where the relationship differs. Optimization can move the business out of the setting that made its indicator useful.

Lead volume creates a closely related practical question. In Sabnis and colleagues’ research, marketing supplies leads while sales leaves some unpursued. Follow-up is associated with perceived prequalification quality, competing demands and salesperson characteristics. The research does not demonstrate that every increase in volume reduces quality. It does show why the same lead count cannot be assumed to mean the same sales workload or willingness to follow up.

Our extrapolation is that expansion requires a fresh check of composition and capacity. A changed acquisition source can invalidate an earlier relationship between submitted forms and useful conversations. Rejection by sales alone cannot diagnose that change: weak routing, limited attention or mistaken rejection remain alternatives. Qualification that also audits rejected records keeps those alternatives visible rather than treating sales acceptance as an infallible label.

Are you applying evidence from the buyers you studied to buyers you have never studied?

Causal: the intervention changes how the number is produced. Last-click attribution awards credit to the final counted click before a purchase. It cannot, by that rule alone, tell whether the purchase depended on that click. In the field experiments above, turning off brand-keyword advertising caused almost all the forgone click traffic and attributed sales to move to organic search. Buyers could still reach the destination through another route.

The classification needs care. Existing purchase intent causing both clicks and purchases is confounding; that fact alone is not causal Goodhart under the paper’s strict taxonomy. Our causal application concerns an intervention that changes the measured route: intercepting an existing buyer journey can raise credited performance without creating corresponding additional demand. The act used to improve the indicator changes its relationship with the objective.

This is a bounded conclusion from a particular research setting. It does not establish that brand advertising never helps. Nor does replacing last-click with another attribution model identify the missing counterfactual. Credit allocation asks who receives recognition for an observed purchase. Incrementality asks what would have happened without the intervention. A credible experimental comparison can address the latter. Choosing what an early GTM experiment measures should begin with that distinction, before the reporting model is chosen.

Adversarial: another actor learns how to satisfy your evaluation. Cresci and colleagues’ research on fake followers used a reference dataset of genuine and artificially acquired follower accounts. The documented phenomenon is a rising follower count without a corresponding audience of genuine people. A supplier evaluated on the visible total and a business seeking relevant attention can therefore succeed and fail on the same delivery.

Our interpretation depends on who is acting. When an executor exploits the client’s follower target to serve a different objective, the problem involves misalignment between actors. When the business owner deliberately inflates their own counter, it is direct manipulation of the measure. The visible output does not uniquely identify the mechanism. This is why the taxonomy is more useful as a question about incentives and interventions than as a label attached to a metric.

Dishonesty is sufficient for some failures, but unnecessary for others. Sampling noise requires no deception. A conscientious team can also meet its instructions while neglecting work those instructions omit. A better metric can repair a specific defect; it still has limits. Combining several into a weighted score makes the weights consequential rather than eliminating trade-offs. Even a genuinely improved outcome does not prove that every sacrifice made to obtain it was acceptable.

Why an AI citation panel is a particularly exposed proxy

A citation panel samples selected questions under recorded conditions. Its score describes that sample. Whether suitable buyers encounter accurate, useful information is a further question, central to visibility in AI answers.

Aggarwal and colleagues’ GEO research supplies a concrete optimization setting: source text is modified and its visibility in generated answers is evaluated, including through the addition of citations, quotations and statistics. The resulting impression measures concern representation in answers, not observed purchasing decisions. Separately, Liu, Zhang and Liang’s 2023 audit found citations that did not support their associated claims. Those findings describe historical systems and samples. They are not a current error rate for every AI search product.

Our extension is that a marketing panel can expose all four mechanisms at once. A lucky run is selected as proof of improvement. Results on familiar questions are generalized to unfamiliar buying situations. Copy is altered to win inclusion without improving the evidence a buyer needs. An executor can then choose flattering questions, denominators or examples to satisfy the person commissioning the work. These are proposed failure pathways. The cited research does not establish their joint prevalence in commercial monitoring.

The practical boundary is between an event and the claims built on it. A citation is not automatically source support. Source support is not automatically an accurate recommendation. A recommendation is not an observed purchase. Counting the events separately preserves those distinctions; it does not itself prove that one caused the next. Our proposal is to investigate the connections rather than compress them into a visibility score whose commercial meaning remains untested. Getting discovered through AI search should retain that separation throughout the work.

If the panel disappeared tomorrow, would this page still deserve to exist?

How to give metrics a useful but bounded role

The following is our operating proposal, assembled from the preceding distinctions. It is not a validated package that the cited studies show will eliminate Goodhart effects. Its purpose is to preserve evidence capable of contradicting the optimization.

Read also: turning marketing goals into agent permissions and stop rules.

Assign a person with authority beside each signal. The operating record should name the actual person who reviews underlying cases, the judgment they must make and the evidence that can override the number. A sales owner examines suitable and rejected enquiries against explicit fit criteria. An editor checks whether a headline’s promise survives reading the page. Record the colleague’s name, not merely a department, and give that person the power to stop rollout. Human judgment is fallible too: retain reasons, disagreements and examples rather than replacing an unexplained score with an unexplained opinion.

Reserve a holdout you never optimize for. Keep unseen buyer questions or independently judged cases out of development. Decide sampling, evaluation rules and the review occasion before opening them. The people changing the work should not see the individual holdout items beforehand. Dwork and colleagues explain why adaptive reuse can overfit even a holdout. A separate file is insufficient if feedback repeatedly determines the next candidate. Once findings become material for improvement, treat that material as development data and reserve fresh cases for the next independent assessment. This simple precaution is our recommendation, not the authors’ technical method for controlled reuse.

A held-back question set and an experimental control group have different jobs. The former can check performance away from development items. The latter withholds an intervention to help estimate its effect. Neither guarantees that the sampled buyers represent a different market. Independence, representativeness and causal identification must be assessed separately; naming something a holdout grants none of them automatically.

Pair every improvement with what it may not cost. State exclusions before seeing the winning result. A headline may not promise an answer the article does not deliver. A qualification improvement may not be manufactured by hiding difficult but suitable buyers from the records. Citation growth may not depend on unsupported claims. Include time: whose reading, checking or follow-up grows when the number improves? Give someone other than the person rewarded for the score authority to assess a boundary violation. A supposed limit that can always be outweighed by enough improvement is a preference in the score, not a limit.

Let contradictory evidence change the decision. Preserve the original text, denominator, counting rule and scope. Examine disappointing cases as well as winning ones. Keep observation, explanation and action separate in the experiment record. When the score rises while reviewed work deteriorates, pause further rollout, inspect the discrepancy and restore the previous version where appropriate. One incident does not establish a trend. A demonstrably false promise does not need to become a trend before you remove it.

Who in your company can reject a better number, and what evidence gives them that authority?

I make two commitments of my own here. I ask whether every page would still have a reason to exist without the metric. I stop optimizing when the number improves while the work gets worse. This leaves room to retire a page that serves nobody or consumes disproportionate attention. Its justification must nevertheless survive a conversation about usefulness, rather than rest entirely on its score.

That conversation starts with positioning and messaging: who benefits, what can be substantiated and when another option fits better. Honest marketing can recognize an informed rejection as evidence of useful work. Helping a buyer make a better decision remains valuable even when the decision goes against the seller.

How this connects to my larger vision

I want to build businesses whose value reaches further than their accounts can show. Profit has one dimension; a person’s inner life does not. Money is useful in my pocket, where it enables work. I do not want it occupying my head so completely that a rising number becomes my definition of a good day. That would make the thing I build smaller than the human possibilities it could serve.

I see this danger in every kind of striving where the description starts replacing the purpose. That is my broader reading of Goodhart, not a conclusion established by the research. My response is to work with full involvement and as little attachment to a particular outcome as I can manage. I develop value within myself first, then let it take shape in the work. I want the benefit to the world, the team and people beyond the business to become part of the product itself, clear enough to carry a purchasing decision.

Marketing can help people and their agents recognize that value. But making something harmful more effective would deepen the harm. I want to add knowledge to our shared information rather than pollute it to win another mention. My ambition is that becoming a better business and becoming the business a buyer chooses move closer together. Numbers can help me see whether that is happening. They cannot decide what deserves to happen.

Questions people ask

Does Goodhart's law mean targets never work?

No. A useful indicator can support genuine improvement even while overstating it. The question is whether the relationship survives the way you select, reward or intervene. Distinguish inaccurate recording, selection bias, a changed population and a changed causal process. Calling every disappointing result Goodhart's law hides the diagnosis needed to make the next decision.

Would revenue or profit be a safe replacement for clicks?

They may bring measurement closer to the commercial objective, but neither is a complete specification of worthwhile work. Attribution, delayed effects and consequences for customers and staff still require judgment. Replacing a distant proxy can help. It does not establish that every way of increasing the replacement is acceptable, or that the commercial objective exhausts what matters.

How can a holdout remain independent?

Keep its cases out of development and specify the evaluation before seeing results. Repeatedly choosing changes from its feedback makes the holdout part of optimization, even if its file remains separate. Once you use its failures to improve the work, reserve fresh material for the next independent assessment. Independence also does not guarantee that the sample represents a new market.

Is a higher AI citation rate a real improvement?

It can be a real improvement in citation frequency under the recorded conditions. That is a narrower claim than better recommendations or additional sales. Preserve the denominator and raw answers, inspect whether citations support their associated claims, and keep unfamiliar buyer questions outside development. Assess commercial usefulness separately rather than treating the citation score as its substitute.

Related questions

Sources

  1. Reserve Bank of Australia: Conference Volumes, bibliography of the July 1975 conference
  2. Marilyn Strathern (1997): 'Improving ratings': audit in the British University system, European Review, pp. 308 and 311
  3. Donald T. Campbell: Assessing the Impact of Planned Social Change, 2011 reprint of the 1976 text, pp. 34 and 35
  4. David Manheim and Scott Garrabrant (2018): Categorizing Variants of Goodhart's Law, version 1
  5. Claire E. Robertson and colleagues (2023): Negativity drives online news consumption, Nature Human Behaviour
  6. Gaurav Sabnis and colleagues: The Sales Lead Black Hole, 2010 author manuscript and 2013 publication abstract
  7. Thomas Blake, Chris Nosko and Steven Tadelis (2014): Consumer Heterogeneity and Paid Search Effectiveness, author manuscript
  8. Stefano Cresci and colleagues (2015): Fame for sale: efficient detection of fake Twitter followers
  9. Pranjal Aggarwal and colleagues (2024): GEO: Generative Engine Optimization
  10. Nelson F. Liu, Tianyi Zhang and Percy Liang (2023): Evaluating Verifiability in Generative Search Engines
  11. Cynthia Dwork and colleagues (2015): Generalization in Adaptive Data Analysis and Holdout Reuse

About the author: Piet Baudoin

I am Piet. I help startups with their brand, positioning and go-to-market, and I set up AI automations that turn customer notes into draft content and follow-up tasks. I discuss the choices with you so the work helps your business and its buyers. Want to apply this to your company? Try the growth scan or book a conversation.