virt-viewer is grabbing the keyboard but not the pointer? o.O This seems awkward at best. (There is also a distinct shortage of useful documentation to see if there's some way to make it do a more complete grab / "capture mode".)

I don't think there's a good way to work around this aside from using a different modifier for the mouse events on one side or the other. You would have to check which window the mouse event was applied to, and synthesize and forward a new event in the case of virt-viewer; but many programs ignore those events, as the server forcibly tags them as synthetic for security reasons (consider the local-GUI version of various web injection attacks).