I think the best approach would be to throw an exception by every computation can cannot be computed, with an additional information of what type the problem is (like divide by zero, square root of negative, etc). Then on top of that can build new types with the desired behavior. You can't catch exceptions locally in any suggested extension of Haskell, so this is impossible. I think it's much cleaner to keep Infinity and NaN, and then raise an exception if these value should ever occur where you cannot handle them. IEEE754 propagates the special values in a nice way so it work pretty well.
That said, don't think Haskell's Float&Double are ideal for serious numerical work; you might need finer control of things like rounding modes etc. -- Lennart