> Is this the only sane way to do this?

Would Mutable Vector per column do?

http://hackage.haskell.org/package/vector-0.12.0.1/docs/Data-Vector-Mutable.html