pandoc, git-annex, and xmonad are available in [haskell-core]; no need to compile anything yourself or pull down extra dependencies.
As far as providing only build of binary command-line programs (and not libs) and building needed libs for each tool in a sandbox build; that's been discussed over on the mac side w/ brew.
I for one *really* appreciate having binary installs of haskell libs available. Some take *forever* to compile (e.g. llvm-general), and I really like having one consistent set of libs to worry about and not having to figure out library version number juggling myself. I use a lot of haskell tools, and I also compile haskell programs / do development in haskell. If a lib's already been compiled into a package, it's making that available for other computers to install, rather than make other boxes recompile that themselves.
For things not in [haskell-core], I've been using cabal2arch to make PKGBUILDs, put those on AUR, and install those myself. I don't have anything cabal-install'd at the user level.