It's a difference in the Z3 theorem prover rather than the library. SInteger maps to the theory of integers while SWord32 maps to theory of bitvectors. The two are implemented completely differently.
Of course, this just punts the question to why Z3 can be faster for Integers than for BitVectors. BitVectors are just fixed-length patterns of bits--32 bits in this case. Integers, on the other hand, cannot be reasoned about as bits because Z3 has to prove things about *all* integers in the mathematical sense. I *think* the difference comes up because the theory of bitvectors is optimized for exhaustively searching over all possible bit patterns given a wide range of constraints while integers, by their very nature, have to be dealt with symbolically.
In fact, working with bitvectors is pretty much the same as doing normal SAT solving. I don't know about Z3 in particular, but this is actually how some solvers implement bitvectors: they just compile the bitvector constraints to normal SAT constraints (called "bit blasting").
For simpler queries, symbolically working with integers can prove faster, but for complex ones it might just not return an answer at all. (In SBV, it would just return `Unknown'.) Since the domain of bitvectors of a given size is finite we can, in principle, check everything which means that any set of constraints is solvable although it might take a really long time.
Also, the solver uses different strategies for the same types depending on the actual constraints you have. I've noticed slow-downs in a very similar vein just using SInteger. I had a problem that would quickly return `Unknown' with unbounded integers; when I added some bounds (ie x .> 0 &&& x .< 10000000000) the process started eating up a lot more CPU and not returning in a reasonable time-frame.
I should add that while I've worked *with* a few SMT solvers, I still only have a vague idea of how they are implemented. If you really want to understand what's going on, you'll have to ask somebody knowledgeable or just read their publications :).