While I share your distrust of the whole Safe Haskell movement as being a lot of effort for an unproven benefit from a definition of "safe" that is not demonstrated to be of practical usefulness or practical concern, I think you're wrong here.

unsafePerformIO is unsafe.

Data.ByteString.map is only unsafe if it allows unsafePerformIO to be abused.

If it can verify that nothing actually unsafe takes place — which it does, by dint of the promise inherent in it being exposed as pure — Data.ByteString.map is *not* unsafe.  The mechanical application of "oh, it uses unsafePerformIO, we don't care whether it proves it has used it safely:  it must by definition be unsafe" just complicates things even more.  If indeed it's not simply a strawman.