I think how you want to do this depends on the motivation of why you want to rollback. Note that rolling back based on information *inside* the transaction means that the transaction must be consistent to successfully rollback. If you want to trigger rolling back from *outside* the transaction you can do something like this:
atomicallyWithCancel :: TVar Bool -> STM x -> IO (Maybe x)
atomicallyWithCancel cancel act = atomically ((readTVar cancel >>= check . not >> return Nothing) `orElse` (Just <$> act))
Note that this still has to have a consistent view of memory to successfully "cancel", but if it reaches the end of the transaction it will notice an inconsistent read of the "cancel" tvar and restart the transaction. Then it will quickly commit the cancel transaction that only reads a single TVar and returns Nothing.
Ryan