What prompted scientists to introduce a theory in contradiction with “common sense”?
They must have had a reason, and a good one:

Imagine that you have a cat, and you noticed that when your cat doesn’t eat for 5 hours, she goes to the kitchen supposedly asking for food. Day in and day out, the cat never deviates from this pattern. Five hours without eating and the cat rushes to the
kitchen. Summer, winter, rainy or sunny, whether the house is full of guests or no one in the house, the cat does exactly the same thing. So you devised a “theory”. This theory says that: “When healthy, the cat rushes to the kitchen after five hours of fasting.”

Now imagine that on a weekend you left the cat alone with plenty of cat food in the kitchen, so the cat wouldn't starve. You came back on Sunday night only to find the cat food untouched and the cat perfectly healthy!

Then you could say that the theory doesn’t work during this weekend while you were away. So you devised a “modern theory” which says: “When away, the cat doesn’t eat at all while staying healthy.” You thought: What a strange theory, it’s even conflicting with common sense. Who cares? As long as the theory explains the real behavior of the cat. When your friends didn’t believe you, you simply repeated the same “experiment” under their supervision. They believed, but they weren’t comfortable.

This story is almost the exact story of quantum mechanics. We had the classical theory from Galileo to Newton, all the way to Einstein. This theory explained our world pretty well, except for one thing: our observations!

Consider for example the “classical” atom, i.e. the solar system model of the atom as introduced by Rutherford in 1911. The basic flaw with this “classical” atom is that as the orbiting electron circles the nucleus, it should emit electromagnetic waves of an intensity increasing rapidly to infinity in a tiny
fraction of a second, as it spirals inwards and plunges into the nucleus. However, nothing like this is observed. Thus our observation contradicts our “classical” physics theory.

This is why Quantum theory, which certainly was not wished upon by scientists, was forced upon them despite their great reluctance. They found themselves driven into this strange, and in many ways, philosophically unsatisfying view of the world.