Re: [Haskell-cafe] inotify-alike for mac os x?

On Dec 4, 2009, at 5:30 PM, Bryan O'Sullivan wrote:

On Fri, Dec 4, 2009 at 8:39 AM, Svein Ove Aas wrote:

That said.. you say you have to handle the events "fast". What happens if you don't?

If you don't handle events quickly, they're typically thrown away by the kernel without you ever getting to read them. That is, for instance, what happens on Linux with inotify. Throwing away events means that your app's internal mirror of the filesystem state becomes wrong, which is Very Bad for most applications that care. (i.e. Ross's assertion than nothing bad will happen is generally not true.)

Ah hah yeah, I meant in the context of it won't block the kernel or cause your computer to melt. It varies between applications whether dropping events is bad or not so I wasn't commenting there.

However, with inotify you also can't afford to perform a single read system call per event, because that will cause your "watch the filesystem" event to soak up most of the system's CPU time. So what you have to do is select to listen for "there's an event ready to be read", then sleep a little while, then read in the hope that many (but not too many!) events will have been queued that you can all read at once.

And at that point, you'll be getting a stale notification about a file or directory that may no longer even exist, or may have changed type. Consider: I create a file f, write data into it, rename it to g, then create a directory named f. You wake up 10 milliseconds later, and the first event you hear about is that a file named f was created.

This is all by way of saying that working with filesystem change notification interfaces is extremely subtle and tricky, enormously more so than you'd think on casual inspection. It's very easy to write a program that uses these interfaces in ways that will make it either generate garbage or consume huge amounts of CPU, and in fact the common case is to write a program that does both.

Amen. I've written an application that does this kind of work using inotify and it was a nightmare. I think this is why fseventsd was invented for OS X, and I'm not sure if there's any linux equivalent. However, if someone were to write a library that uses kqueue / inotify / win32-call-I-forget-the-name-of-from-earlier-post in a way that is both efficient and correct, that would be totally awesome. -Ross