I agree with others who have stated that the interpretation as a hint is bad for performance. My intuition has always been that we pre-allocate an array of length N, fill that in with a recursive function while accumulating the length, and rejecting anything incorrect.
I think the motivation behind it being a hint is still mostly a mystery to me, until I can find some discussion. I can see how a hint might be useful from the perspective of the desugarer, but for people who are calling fromListN in source code, this seems much less intuitive, if not potentially useful.
It might be illuminating if we could find some discussion about why this decision was made, since there might be something more I'm not considering. As it stands, my intuition (and that of most library authors I know, and most libraries I have seen) does not align with that as a hint. E.g. if primitive were to switch to treating N as a hint, it would require a majour version bump, and also it would be a place where a size hint would be most unwelcome because of perf.