← Back to Topic Saiaus

Does "Probably Correct" Mean "What We Should Do"?

AI can use past information to predict likely outcomes and likely choices. But a future with a high probability is not the same as a future humans ought to choose. This article considers the risk that prediction, while helping judgment, can get ahead of human will, and explores the distance needed between probability and choice.

General

We use predictions in many parts of daily life. If rain is likely, we carry an umbrella; if congestion is likely, we change our departure time. Because AI can process large amounts of information, it can sometimes make such predictions in greater detail. But a high probability that an outcome will occur is different from saying that outcome should be chosen. "Likely to happen" is a description of the future, while "should do so" includes purposes and values. If these are treated as the same, prediction quietly turns into command.

Suppose AI predicts from your past choices that "you are likely to choose this product." The more accurate the prediction, the more convenient the suggestion becomes. But humans are not beings who only repeat past choices. New purposes can arise: wanting to save money, try something different, or consider environmental impact. A probability calculated from the past may describe tendencies before that change very well, but it does not determine present intention.

One reason stronger predictions can close down thought is that numbers look objective. When shown "an 80 percent chance," the remaining 20 percent can feel unimportant. But the meaning of a probability changes depending on what was observed and which population or period was used as the basis. In individual judgment, a low-probability possibility may also carry great value. Knowing what is likely is useful, but that alone does not determine what we want.

What thought needs here is not rejection of prediction but limitation of its role. Prediction provides material for considering "what is likely under these conditions." By contrast, "which outcome should we pursue," "which risks should we accept," and "what do we value" are different questions. By separating prediction from will, humans can use AI analysis while checking their own purposes. Whether to follow a prediction is itself a judgment outside the prediction.

For example, in a study plan, suppose AI predicts from past grades that "your score is unlikely to improve in this field." If the only goal is to raise exam scores in the short term, allocating time elsewhere can be rational. But for someone who wants to study that field deeply in the future, difficulty improving may itself be a reason to work on it. The same prediction leads to different actions when the purpose changes. Prediction can be used not as an answer that narrows choice, but as information that makes the conditions of choice visible.

For AI to respect human will, it is important not to automatically determine present wishes from past tendencies. There is an unverified leap from "this is how it has been" to "this must be how you want it to be from now on." Humans learn from experience, change their values, and make choices different from earlier ones. Preserving the possibility of that change does not deny predictive accuracy. It means not confusing the scope handled by prediction with the scope decided by humans.

Humans also need an attitude of not entrusting too much of their judgment to prediction. When AI shows a high probability, instead of ending with "then that must be the right answer," we can ask again, "What assumptions does this prediction make?" and "Do I have the same purpose as those assumptions?" With this small distance, numbers become material for thought rather than authority that stops thought. The more accurate prediction becomes, the more important that distance is.

When Sai*aus handles prediction, what matters is not only getting the future right. It is distinguishing what conditions a prediction is based on, how far it can be used, and where human choice begins. If AI can show "what is likely to happen" while humans can consider "what to choose nevertheless," prediction and will do not compete. AI capability can be used not to replace human decisions, but to make the conditions of those decisions clearer.

"Probably correct" is important information. But it is not the same sentence as "this is what should be done." Probability gives a view of the world ahead, while will chooses where to go within that world. By maintaining the distance between them, humans need not oppose AI predictions; they can understand those predictions and still retain their own judgment. Thinking is not only accepting the most likely future, but continuing to ask which future we want to choose.

Not answers. Understanding.

← Back to Topic