But the delay caused by those buffers does not depend on the internal workings of your crypto algorithm.
The delay caused by potential cache misses could very well leak information. If your algorithm terminates early on a string comparison, and therefore doesn't end up evicting more pages out of the cache performing whatever operations would have come after had the strings matched, there could be real world measurable differences, even if you waited for the clock to hit some threshold before continuing on with execution.
Regardless of how much delay there is in other sources, if it is not directly correlated, it can be averaged out.