The problem is that the package db contains only what ghc needs to be able to use the library; not the additional information needed to safely remove it. (There are other package systems with this problem, notably Apple's. Apple decided that instead of solving it, they would only support applications that are complete bundles in and of themselves.) And arguably it doesn't actually belong in ghc-pkg insofar as it's not intended to do anything but store the information needed for ghc to link against packages.
This is something of a nasty problem, considering that storing uninstall information separately is not particularly robust. Perhaps ghc-pkg should, if it doesn't already, support extension fields that e.g. cabal can use to store uninstall information. (But even that potentially has problems, given that people are known to copy package registration information between package databases. If there is uninstall information in there, what happens if someone uninstalls via one or the other copy?)