I'm sharing two answers I gave in 2019 explaining why we should reject an idea if we know one criticism of it. In short, a criticism is an explanation of why an idea fails at its purpose. It never makes sense to act on or accept an idea you think won't work.
I will also add that we don't reject a theory just from 1 failed observation. We must also have a better theory in place. One that explains what the previous theory successfully explained, and accounts for the mismatch in observation.
If it's a universal theory (X), and you (tentatively) accept one failed observation, and accept the arguments about why it's a counter-example, then you must reject the theory, immediately. It is false. You may temporarily accept a replacement, e.g. "X is false but I will keep using it as an approximation in low-risk, low-consequences daily situations for now until I figure out a better replacement. A replacement could be a new theory in the usual sense, but could also e.g. be a new combination of X + additional info which more clearly specifies the boundaries of when X is a good approximation and when it's not."
For a non-universal theory Y which applies to a domain D, then the same reasoning applies for one failed relevant observation – a counter-example within D.
As I understood it before, we don't reject it until we have a better explanation. Like for the theory or relativity, we have "failed observations" at the quantum level right? But we don't reject it because we don't have another better theory yet. What am I missing?
If you know something is false, you should never accept it because it's false.
The theory of relativity is accepted as true by (approximately) no one. Call it R. What people accept is e.g. "R is a good approximation of the truth (in context C)." This meta theory is not known to be false. I call it a meta theory because it contains R within it, plus additional commentary governing the use of R.
This meta theory, which has no known refutation, is better than R, which we consider false.
KP and DD did not make this clear. I have.
If you believe a theory is false, you must find a variant which you don't know to be false. You should never act on known errors. Errors are purely and always bad and known errors are always avoidable and best to avoid. Coming up with a great variant can be hard, but a quick one like "Use theory T for purposes X and Y but not otherwise until we know more." is generally easy to create and defend against criticism (unless the theory actually shouldn't be used at all, in any manner).
This is fundamentally the same issue as fixing small errors in a theory.
If someone points out a criticism C of theory T and you think it's small/minor/unimportant (but not wrong), then the proper thing to do is create a variant of T which is not refuted by C. If the variant barely changes anything and solves the problem, then you were correct that C was minor (and you can see that in retrospect). Sometimes it turns out to be harder to create a viable variant of T than you expected (it's hard to accurately predict how important every criticism is before you've come up with a solution. that can be done only approximately, not reliably).
It's easy to make a variant if you allow arbitrary exceptions. "Use T except in the following cases..." That is in fact better than "Always use T" for a T with known exceptions. It's better to state and accept the exceptions than accept the original theory with no exceptions. (It's a different matter if you are doubtful of the exceptions and want to double check the experiments or something. That's fine. I'm just talking from the premises that you accept the criticism/exception.) You can make exceptions for all kinds of issues, not just experiments. If someone criticizes a writing method for being bad for a purpose, let's say when you want to write something serious, then you can create the variant theory consisting of the writing method plus the exception that it shouldn't be used for serious writing. You can take whatever the criticism is about and add an exception that the theory is for people in situations where they don't care about that issue.
Relativity is in the situation or context that we know it's not universally true but it works great for many purposes so we think there's substantial knowledge in it. No one currently has a refutation of that view of relativity, that meta theory which contains relativity plus that commentary.