I have never really understood xmodmap, and I suspect it's broken in a
few ways. You can probably fix your problem by moving numlock to a
different keysym, like this:
Sort of broken. The problem is that modern X servers don't actually use xmodmap; they use Xkb, and that has certain modifiers "baked in" to its keymaps (notably caps and num lock) and you can only reliably alter that by rewriting the Xkb mappings.