I know you're asking for a function that will either return an Int or a (Int, Int). However, in my limited experience, I haven't seen Ints used as error codes in Haskell. Typically, I see something like the following:
checkNum3 :: Int -> Int -> Either String (Int,Int)
checkNum3 a b = if check a b
then Right (a, b)
else Left "Error: something happened"
Then you can use a function like (either error id) to either extract the (Int, Int) value or to raise the error with the message.
The (either error id) function will look at an Either value. If it's a Left value, it will call error on that value -- so (error "Error: something happened"). If it's a Right value, it will call id on that value -- so id (a, b) which evaluates to (a, b).
I usually use this setup if I lazily parse a huge file of text data, where each line has some constraint that makes the file "valid" or "invalid". I end up with a long list of Either String (Int, Int)s, and I then map (either error id) over the list.