Did you also remove ~/.ghc?  Libraries are actually installed and registered there, not under ~/.cabal.  Although this shouldn't be a problem if any Cabal-installed libraries were for a different version of ghc, it's safest to remove both.