“Should I Trust…?” · The Series · Episode #03
No claim about “everyone” can be proven true. It can only be proven false. Here’s why.
NONSENSE !
What? Of course you can validate things. I do it all the time. I check the numbers, I confirm the assumption, I tick the box.
Telling me I cannot prove these things sounds like wordplay.
Fair. But the word "validate" is doing two different jobs, and only one of them is possible. People mix them up all the time.
Picture a turkey on a farm.
Every morning, the farmer brings food. Every single day confirms what the turkey believes: people are kind, food always arrives, life is safe.
A thousand days of evidence, all pointing the same way. The turkey has never felt more certain.
Until one single bad day.
Two days before Thanksgiving, the farmer does what he’s supposed to do two days before Thanksgiving.
A thousand validations that weren’t. The 1,001st broke what was believed to be reality.
Job one is checking something finished. Did the invoice add up. Did the form submit. Do these two files match. You can do this all the way to the end, because there is a fixed list of things to check and you can check every one. This is real. Go ahead and call it validating.
Job two is proving a general statement about the world. "Our buyers are subcontractors." "Customers want this feature." "This onboarding flow works." These are claims about everyone, including people you have never met and people who are not customers yet. You can never check all of them, because the list never ends.
Here is the mistake. You feel how solid job one is, and you assume job two works the same way. It does not. The moment your claim says "all," or "always," or quietly means "in general," you have walked out of the things you can finish.
This is the part people miss. It is not that you did not gather enough evidence. More evidence does not close the gap.
Everything you have seen is already behind you. Your claim is also about what happens next. So your evidence sits on one side and your claim reaches past it, every time. You are guessing from the cases you have seen to all the cases you have not. A big pile of seen cases is still not all cases.
Sourced David Hume gave this a name: the problem of induction. To go from "it has always happened" to "it will always happen," you have to assume the future behaves like the past. And your only reason to believe that is that the future has behaved like the past so far. You are using the idea to prove itself. It goes in a circle. A useful habit, but not a proof.
Source: David Hume, An Enquiry Concerning Human Understanding (1748).So stop trying to confirm things. You were never going to get there. Do these two instead.
Validating is not a faster version of this. It is pointed the wrong way. It counts the feedings and calls them proof.
Don’t ask “what agrees with me” — the turkey had plenty of that. Ask: “what would kill this, and has it shown up yet?”
That’s the whole idea in one move: stop trying to confirm, start trying to break. If this clicked, I write more like it.
Join my Substack — freeNew pieces in the “Should I Trust…?” series. Free.
Deep dive
That is the claim. Here is the long way to it: how the mistake starts, what a serious test actually looks like, and the single sheep that cracks the whole thing open. Every step has a panel you can open for the rigorous version and the named sources.
The starting sentence was a task of this shape: "Validate your assumptions on three key traits of your work-in-progress customer profile by doing X and Y."
The word "validate" is the failure point. The scientific method does not validate assumptions. It tries to kill them.
The trouble is that "validate" decides the outcome before the test runs. If the goal is to validate, you will build X and Y to succeed, read unclear data as support, and treat a weak test surviving as proof. The conclusion is smuggled into the task.
Sourced The asymmetry between confirming and refuting is the engine here. No finite number of confirming observations can prove a universal claim, but a single genuine counter-example refutes it. A claim that survives hard testing is not called "proven." It is called corroborated, meaning it has not yet been refuted, and the status is always provisional.
Source: Karl Popper, The Logic of Scientific Discovery (1959), as set out in the Stanford Encyclopedia of Philosophy.My reasoning The corrected form of the task: "Attempt to break your three highest-risk assumptions. For each, state in advance the observation that would prove it wrong, then run X and Y to provoke that observation." The rewrite, and the idea of a pre-declared kill condition per trait, is my framing.
Most people hear "validate X" as one clean chore with an end. You look things up, line them up against other sources, see they match, tick the box. The thing is now "validated," meaning true and safe to build on. That picture fails in three quiet ways.
The honest translation: "I can raise or lower my confidence in X by checking it against reality and trying to break it." Less tidy. No tick-box. No finish line.
Sourced The first failure has a name and a documented history. The tendency to seek evidence that fits a hypothesis in hand, rather than evidence that would break it, is confirmation bias. A sharper version is the positive test strategy: people test cases where their hypothesis predicts success, which is not the same as trying to disconfirm it.
Sources: Peter Wason (1960), Quarterly Journal of Experimental Psychology; Raymond Nickerson (1998), Review of General Psychology. Positive test strategy associated with Klayman.Trust in a claim does not come from finding support. It comes from this: you went looking, in good faith, for the evidence that would kill the claim, and it survived.
Pin down the words. The claim is what is on trial. What it survived is your attempt to find its killer. It did not survive because you found things that agree with it. It survived because you searched for its weak point and the weak point did not show up.
Said flat: "I tried to prove X wrong. I could not. So I trust X more, for now." The "for now" stays. A killer could still turn up tomorrow.
The strength of the claim equals the seriousness of the attempt to break it. A lazy search proves nothing. A hard search that comes up empty is what earns trust.
If "earned trust" depends on the seriousness of the attempt, then "seriousness" has to be defined, or anyone can run a few searches and declare victory.
Seriousness, defined. An attempt is serious to the degree that it would have caught the claim being false, if the claim were in fact false. Not effort. Not time spent. Not how many tabs you opened. Only this: had the claim been wrong, would this method have shown it.
Count is the wrong unit. My reasoning Ten searches on the same tool are not ten attempts. They are one attempt run ten times: same source, same blind spots. Correlated tests do not stack. What stacks is independence: attempts that can fail for different reasons, touching different parts of reality. One customer interview, one churn number, one competitor's absence are three real attempts. Ten searches with synonyms are one.
Other attempts, ranked by how much teeth they have. My reasoning The ranking and selection are mine; individual items carry their own source notes.
When are you "done." My reasoning You are never "done" in the sense of proven. You are "done enough, for now," and that is set by two things together: the surviving attempts were severe and independent enough that you would most likely have caught the claim if it were false, and the cost of still being wrong is acceptable for what you are about to bet on it. Higher stakes demand more survival before you act.
Sourced The definition of seriousness above is the severity idea from the philosophy of statistics. In plain terms: data count as evidence for a claim only if the test would probably have exposed the claim as flawed, if it were flawed.
Source: Deborah Mayo, Statistical Inference as Severe Testing (2018). Lineage note: Mayo extends and repairs Popper rather than restating him, and is usually filed as neo-falsificationist.The independence and "done" points above are standard reasoning about correlated evidence and decision-making under stakes; they are my framing, not borrowed from a single cited authority.
A person driving in Ireland sees one black sheep in a field and says, "All sheep in Ireland are black." The next day he finds a white one.
The claim "all sheep in Ireland are black" is dead. One white sheep refutes it. Sourced This is Popper's standard example inverted: his was "all swans are white," refuted by a single black swan.
The important part: the white sheep did not reveal new complexity. It collected a bill that was already owed. The mistake was on day one, not day two. He saw one sheep and asserted "all." That leap was never earned. His test had no severity, because after one glance he stopped looking. The white sheep just exposed a claim that had no support to begin with.
What he is actually entitled to say, after two sheep:
He still cannot say "all are white," or "half are black," or anything universal. He has two data points.
The sheep and the scientists
My reasoning A classic, widely retold joke, included as illustration; the origin is not mine. Three scientists on a train in Scotland see one black sheep.
The mathematician made the only statement one observation can support, and the only one no second sheep can ever falsify. He priced in the next sheep. The other two were refuted the moment a second animal walked into frame.
A fair objection: what if Irish sheep really are 99 percent black and 1 percent white? Then the refutation came from hitting a rare case, so surely the conclusion was "largely accurate."
The objection hides two different claims under one sentence. Pull them apart and it dissolves.
So the white sheep did not make your picture of the world wrong. The world really is about 99 percent black. What it made wrong was the form of claim you chose to carry that picture. The gap was between your all-or-nothing claim and a world that has a tail. It was not a gap between reality and your eyes. Your observation was fine. Your conclusion was brittle.
The rarity cuts against you, not for you. My reasoning The calculation below is my own, using standard probability. If 1 percent of sheep are white, a single glance lands on a black sheep 99 percent of the time, so "all black" will survive day after day, looking corroborated, purely by luck. To even have a better-than-even chance of catching one white sheep by single glances, you would need to look at roughly 300 sheep. (The arithmetic: 0.99 to the power n drops below 0.05 at about n = 299.) You did not get unlucky on day two. You ran a test with almost no power, and it happened to fire early. A rarer exception demands a more severe test, not a more forgiving rule.
My reasoning This marks a real boundary between two regimes; the mapping is my framing, the named tools are sourced. Pure falsification of the Popper kind is clean for universal laws, the physics-shaped claims where one counter-example settles it. Sheep populations and customer bases are statistical, and that is the regime built for by Mayo's error statistics: you do not falsify a distribution on a single draw. You estimate a proportion and you put bounds on the error.
Sources for the named tools: Popper (1959) and Mayo (2018), as above.The same discipline applies away from sheep, to any claim about a population of customers.
The white sheep was never the problem. The word "all" was.
The other half
We opened with a hard claim: you cannot prove a general statement about the world true. That is true, and it is easy to over-read. It does not mean "validate" is a junk word. It is a precise tool with a real job. The job just has an edge to it.
You can validate anything whose cases are finite and you can reach the end of. You cannot validate anything whose cases never end. That single line sorts almost every real example.
The rescue is simple: make "validate" take a closed object. Do not validate "the strategy," "product-market fit," or "the assumption." Those never end, so the word is writing a cheque it cannot cash. Validate the finishable thing next to them instead.
An A/B test is the clean example. The test does not validate "users prefer B." That is an open claim about everyone, including the users you do not have yet. What the test does validate, exactly and honestly, is closed:
All three are the rigorous core of the experiment, and all three are validatable, because each one has an end. Keep the object closed and "validate" earns its place again. Let the object drift open, to "the flow works for everyone," and you are back to counting feedings like the turkey.
Falsify what is open. Validate what is closed. The word was never broken — it was standing in front of the wrong noun.
Sourced This positive list is the logical dual of the asymmetry that runs the whole record. In the logic of quantifiers, a universal statement ("all X are Y") is falsifiable: one case can refute it, no number of cases can confirm it. An existential statement ("at least one X is Y") is the mirror: one case confirms it, no amount of further looking can refute it. "Validate" is the right verb for the second kind, and for closed, finite sets. "Falsify" is the right verb for the first.
Source: Karl Popper, The Logic of Scientific Discovery (1959), on the asymmetry between universal and existential statements.My reasoning The five-way taxonomy of validatable cases, and the A/B-test rescue (validate the test, the threshold, and the null, never the open claim), are my framing, built on that asymmetry.
The trap, named out loud
The electric-sheep line was not false. It was non-falsifiable: there is no observation that could prove it wrong. That is its own category, and most weak claims live in it. Naming the category is half the work. Once you can say "that is non-falsifiable," you stop arguing about it and start fixing it. Here are the usual shapes.
No defined terms, no threshold. Nothing says what would count as failure.
"Our UX is intuitive." "This will improve engagement."
A forecast with no deadline and no metric, so it never comes due. It can always be pushed to next quarter.
"This will be huge." "Eventually customers will want this."
True by definition. The predicate just restates the subject, so no fact can break it.
"Our power users are the ones who use the product most."
Comes with a built-in escape hatch. Any failure gets blamed on the conditions, never on the claim.
"It works, when it is done right." "It helps, in the right context."
A "somewhere out there" claim. One example confirms it, but nothing can refute it, because you can never check everyone. Confirmable, not falsifiable.
"There is a market for this." "Some customers want X."
Faster, better, more scalable, with no referent and no number. Faster than what, measured how.
"The new flow is faster." "This stack is more scalable."
A claim about something undefined or non-existent. No instances exist to test, so there is nothing for it to be true or false about.
"Electric sheep often dream of androids."
Editorial statements are the bucket people fight hardest to defend, so handle them on purpose. "This design is elegant." "Notion beats Linear." "This is the right strategy." These are value judgments. They state a preference or a taste, not a condition of the world, so no observation can refute them. They are non-falsifiable, and that is fine, as long as you call them what they are.
Two honest moves, and only two:
The danger is never holding an opinion. It is smuggling an opinion in as if it were evidence.
The repair
A falsifiable hypothesis points straight at the door marked "this is how you would know I am wrong." You build it in four moves. Miss any one and you slide back into non-falsifiable.
Two examples. Clarity first, cleverness never.
"Self-serve onboarding will improve activation."
"Among new B2B trial sign-ups in Q3, the share reaching first value within 24 hours will rise from 30% to at least 45%. Kill condition: if it is still below 38% after 1,000 sign-ups, the claim is dead."
"The new route is faster."
"My commute on Route B will average under 25 minutes across 10 weekday morning trips. Kill condition: if the 10-trip average is 25 minutes or more, the claim is dead."
If you cannot name the observation that would kill it, you do not have a hypothesis yet. You have a wish.
The discipline behind all of this
Everything above is one idea applied. It is worth seeing the idea on its own: where it came from, how it was corrected, and how working people actually use it.
Sourced Karl Popper started from a simple question: what separates science from everything that only sounds like science. His answer was risk. A claim earns the name scientific only if it forbids something, only if there is some observation that could prove it wrong. "All swans are white" is scientific, because one black swan kills it. "Things happen for a reason" forbids nothing, so nothing can test it.
Sourced From there the method follows. You never prove a theory true. You put up a bold guess, you work out what it forbids, and you attack it with the hardest test you can build. What keeps surviving those attacks is kept, for now. Knowledge grows by killing wrong ideas, not by stacking up agreement. Popper called this conjecture and refutation.
Sources: Karl Popper, The Logic of Scientific Discovery (1959); Conjectures and Refutations (1963).Pure falsification has a flaw: in real life a single failed test rarely kills one claim cleanly. The people who came after spent decades fixing that.
In science. Sourced The engine is conjecture and refutation. State a bold claim that forbids something, work out a risky prediction, build the most severe test you can, and try to break it. Popper's favourite example was Einstein: general relativity predicted that starlight would bend by a specific amount near the sun, a prediction that could have come back wrong and sunk the theory. It did not. That is what a real test looks like, one the idea could have failed. Source: Karl Popper, Conjectures and Refutations (1963), on Einstein as the model of a falsifiable theory.
In product and customer discovery. My reasoning Your roadmap sits on assumptions about customers. Treat them as things to confirm and you will confirm them, then ship the turkey. Treat them Popper's way instead. Write each assumption as a claim that forbids something: "if this is true we should see X, and if we see Y it is dead." Name the riskiest one, the assumption that would do the most damage if it is wrong. Then design the cheapest test that could actually produce that killing Y. A customer interview built so it can only produce agreement is not a test. One that could produce a clear "no" is. Good discovery is not gathering support for the idea. It is trying to disqualify the idea before it costs you a build.
Decoding reality and growing knowledge. Sourced This is why falsification is generative, not negative. You never reach final truth. You get closer to it by removing what is false. Every refuted claim deletes an error and sharpens the map. What you know grows by subtraction, by the steady elimination of mistakes, not by piling up confirmations that, as the turkey learned, can all point the wrong way at once. Source: Karl Popper, Conjectures and Refutations (1963), on the growth of knowledge through error elimination.
The story above is one small worked example of this discipline. "Do not validate, try to break it" is just falsification, brought down to the scale of a single product decision.
The standing principle
Never assert a universal over a messy population. Assert a magnitude with its uncertainty.
Do not set out to validate. Set out to break the claim, and state the most you can claim that survives the next data point. If it survives a serious attempt, raise your confidence, for now. If it does not, you learned something real and cheap.
And keep the two verbs straight: validate what is closed and countable, falsify what is open and endless. The error is never the word, it is the noun you put behind it.
The split between what rests on external sources and what is my own reasoning, gathered in one place.
Therefore, it’s a hypothesis. Go falsify it.
The plain layer is the storyline. Read it top to bottom and it stands on its own. The rigorous layer sits inside each step under a closed panel, and opens on click. It carries the precise mechanics and the named sources.
Two content tags appear throughout:
One disclosure for the record. The storyline, the structure, the examples, and the prose in this record are all my generated synthesis of the conversation. External sources are limited to the items marked Sourced, each named on the spot and listed again in the Source Audit. Everything not tied to a source is my own reasoning, marked My reasoning.
Plain English is used throughout. A technical term from the discipline appears in mono only where precision requires it, and is defined when it does.
Keep going