The first option is
class Monad m => MonadRef r m | m -> r where
newRef :: a -> m (r a)
...
This has the benefit of using quite portable extensions.
The second option is
class Monad m => MonadRef m where
type Ref m :: * -> *
newRef :: a -> m (Ref m a)
This takes us into GHC specific territory, by using type families, but avoids polluting every type that uses one with an extra 'ref' param. I use this variant in my as-yet-unreleased 'revisions' package.
Both of these have the benefit that they can work with transformers, but they carry the limitation that you can't have multiple reference types for the same monad. e.g. you can't use the same combinators for both IORefs and, say, MVars or TVars within the same monad. This is arguably not so much a problem as they have very different operational semantics!
In other code I've done horrible things for lifting TVars out into monad transformer stacks, while well aware of the rather ad hoc meaning of chains of operations on TVars when you are working over something IO based. Peeking at some haddocks from my analytics project:
This had the benefit of staying in 98 style, but solved a different problem than what we're discussing here, because the reference type was fixed.
In yet other code i've had the reference type select the monad it works with. The primitive package that is used by vector is at least morally of this flavor, though, for array types rather than references, but another example is the 'var' package that was released the other day in response to this thread. It doesn't use any functional dependencies between the monad and the reference type in order to support multiple reference types per monad, including unboxed variants.
-Edward