Reality in Fantasy

Every now and then someone will argue that armor should affect spell damage, that plate would deflect or absorb some amount of damage. They say it only makes sense, it's realistic. Is it realistic? Does realism play any role in this clearly fantasy world?

Reality does play a part. This is not a purely chaotic world with no natural laws, subject entirely to the whims of magic-users. I believe that the basis of the world is much like the real world. There are all the natural forces and laws that we are familiar with. There is gravity and magnetism, objects have inertia, the conservation laws hold for the physical world. These rules are not entirely broken by magic.

What is magic anyway? When a mage throws a fireball, is it a normal ball of fire, aside from the source, or does it actually have magical properties? If it is simply fire, then it would be subject to natural laws as we know them. It would be absorbed by the armor, spread out, the heat would be dissipated. Frost damage could be treated in a similar manner. Nature seems to be a grouped category of electricity and chemical attacks from either poisons or the toxins released by diseases. Electricity would at least be spread out and would hopefully not need to take a path through anything important like the heart. The resistance of the armor would cause heating and therefore burning though.

But what about holy, arcane, or shadow damage? Those might be something beyond normal physics, so I have no idea how they might react with armor. Perhaps they are actually electromagnetic radiation (it's not as fancy as it sounds, these are all just flavors of light, though most that we cannot see).

I completely failed at getting to my point here. :(

Magical damage may actually follow normal physical laws, so armor would influence the effects of magical attacks. Reality can and should play a role in fantasy. We only need to discover the rules.