We can't live in a state of perpetual doubt, so we make up the best story possible and we live as if the story were true.
Doubting what you see is a very odd experience. And doubting what you remember is a little less odd than doubting what you see. But it's also a pretty odd experience, because some memories come with a very compelling sense of truth about them, and that happens to be the case even for memories that are not true.