The ghc-config Bash script in multi-ghc uses symlinks to determine which is the "current" version to use. The instructions on the GitHub page explain it in detail, but here's how my setup works.
* I create /opt/ghc for storing GHC, clone the multi-ghc repository (or copy the files) here.
* I create /opt/ghc/6.10.4/src, download
http://www.haskell.org/ghc/dist/6.10.4/ghc-6.10.4-i386-unknown-linux-n.tar.bz2 , and untar it here.
* cd ..; ln -s ../Makefile; make install
ghc-config creates a $HOME/.ghc-config directory for tracking the current version and symlinks. Now, the key for your situation (I think) is your $PATH. If you append $HOME/.ghc-config/ghc/bin to your $PATH after /usr/bin, then you can use ghc-6.10.4 to call this specific version.
Unfortunately, there is still a problem. ghc-config also manages $HOME/.cabal as a symlink for the current version. So, that would most likely conflict with your setup. There may be a way to fix that, though I haven't looked into it, or I could separate the cabal configuration from the ghc configuration.