On 27 October 2011 16:31, Bas van Dijk wrote:

Should a Haskell' proposal be filed?

http://hackage.haskell.org/trac/haskell-prime/wiki/Process

Ian mentioned this issue in the GHC ticket at http://hackage.haskell.org/trac/ghc/ticket/4363. He says: """ I think the next step should be a Haskell' proposal to remove the locking from the report (including a clarification that a no-locking implementation is possible on Windows). Some people might argue that we should have an implementation first, though. """ So I guess we are seeing this patch as part of an "implementation first" approach. Max

On Thu, Oct 27, 2011 at 11:09 AM, Bas van Dijk wrote:

Right and to be clear, I'm totally fine with it.

It does create the situation that once this change gets released in a new base package, the haskell2010 package will loose Haskell 2010 compatibility because it depends on base for its IO functions. We could then of course add custom IO functions to haskell2010 that are compatible with Haskell 2010. However I think it's best to just quickly change the Haskell 2010 report and have a short period where the haskell2010 package is incompatible with Haskell 2010.

Bas

Wouldn't the Haskell2010 report remain unchanged by this? I imagine this would be incorporated into Haskell2012, or whatever the next revision is going to be. Antoine

On Thu, Oct 27, 2011 at 04:55:41PM +0100, Max Bolingbroke wrote:

On 27 October 2011 16:31, Bas van Dijk wrote:

...

Should a Haskell' proposal be filed?

http://hackage.haskell.org/trac/haskell-prime/wiki/Process

Ian mentioned this issue in the GHC ticket at http://hackage.haskell.org/trac/ghc/ticket/4363. He says:

""" I think the next step should be a Haskell' proposal to remove the locking from the report (including a clarification that a no-locking implementation is possible on Windows).

Some people might argue that we should have an implementation first, though. """

So I guess we are seeing this patch as part of an "implementation first" approach.

Actually, I realised when revisiting the ticket that this is really only a change in the libraries, and we've taken the approach that library changes are handled by the library proposal process, and the standard will follow along in the next release. Thanks Ian

On Thu, Oct 27, 2011 at 07:10:27PM +0200, Bas van Dijk wrote:

On 27 October 2011 19:03, Ian Lynagh wrote:

...

Actually, I realised when revisiting the ticket that this is really only a change in the libraries, and we've taken the approach that library changes are handled by the library proposal process, and the standard will follow along in the next release.

Great, that's settled then.

It's probably best to just put an upper version constraint on the dependency on base of the haskell2010 package so that it won't depend on the new base package.

Well, haskell2010 comes with ghc, so it needs to use the version of base that GHC comes with. If we stopped shipping it and set dependencies such that it won't build with the new version of base, then no-one with the latest GHC would be able to use it. Thanks Ian

On Thu, Oct 27, 2011 at 3:10 PM, Bryan O'Sullivan wrote:

On Wed, Oct 26, 2011 at 3:59 PM, Ian Lynagh wrote:

...

I propose that we remove all the automatic locking from the libraries, and let the user do any locking that they wish to do themselves.

+50

+1. In general I don't like when low-level libraries try to do too many smart things behind my back. That normally ends up in me having to write my own version of the library without those smarts. -- Johan

On Tue, 2011-11-08 at 11:54 +0000, Duncan Coutts wrote:

On Wed, 2011-10-26 at 23:59 +0100, Ian Lynagh wrote:

...

Hi all,

Haskell currently requires that multiple-reader single-writer locking is used on files. I understand the motivation for this was to try to protect people from running into problems due to lazy IO, but in practice I think it just causes more problems than it solves.

...

I propose that we remove all the automatic locking from the libraries, and let the user do any locking that they wish to do themselves.

I have to say, I'm a little reticent about this.

Certainly it's useful to be able deliberately to have multiple writers, or to be able deliberately to read a file while also writing to it. But doing so accidentally seems like a bug waiting to happen doesn't it?

BTW, I realise this proposal has received a good deal of support, however I suspect this is from people who are thinking about the current inability to opt-out from locking in the various cases where it would be useful for them to do so. Perhaps the people who have spoken up in favour of this proposal could comment on the following: We all seem to agree that its important to be able to open a file without any locking. There is a distinction between being able to opt-out from locking and removing all locking by default. * do you think it important to remove all locking? * would you be satisfied (or put up with) having to explicitly opt-out of locking when opening a file? * do you see any value at all in locking by default to catch bugs where files are read and written concurrently by accident? Duncan

On Tue, 2011-11-08 at 12:04 +0000, Duncan Coutts wrote:

On Tue, 2011-11-08 at 11:54 +0000, Duncan Coutts wrote:

...

On Wed, 2011-10-26 at 23:59 +0100, Ian Lynagh wrote:

...

Hi all,

Haskell currently requires that multiple-reader single-writer locking is used on files. I understand the motivation for this was to try to protect people from running into problems due to lazy IO, but in practice I think it just causes more problems than it solves.

...

...

I propose that we remove all the automatic locking from the libraries, and let the user do any locking that they wish to do themselves.

...

I have to say, I'm a little reticent about this.

Certainly it's useful to be able deliberately to have multiple writers, or to be able deliberately to read a file while also writing to it. But doing so accidentally seems like a bug waiting to happen doesn't it?

BTW, I realise this proposal has received a good deal of support, however I suspect this is from people who are thinking about the current inability to opt-out from locking in the various cases where it would be useful for them to do so.

I should also point out that in the referenced ticket http://hackage.haskell.org/trac/ghc/ticket/4363 nobody was asking to remove all locking. The request was to change the behaviour on windows to match that of unix which is felt to be more useful. That is, to change from OS-wide locking to intra-process locking. Both are compatible with the H98/2010 spec, but the latter is preferable as a default. Additionally, there is clearly value in being able to opt out of all locking. I have a suggestion for how that should be done. Not so long ago we cleaned up the messy System.Process API by adding: createProcess :: CreateProcess -> IO (...stuff...) The CreateProcess record has a whole load of parameters and we provide functions for creating simple default values of these parameters, e.g. proc :: FilePath -> [String] -> CreateProcess Crucially, if you want to customise anything, e.g. specify cwd or env vars then you can do so easily using record update syntax. Also, we can add new parameters to CreateProcess without breaking old programs. We should take the same approach with openFile. Currently we have: openFile :: FilePath -> IOMode -> IO Handle I suggest changing the IOMode for a record of options which would include the IO mode (read/write/read-write/etc), and other things like file locking, default create permissions, create exclusive etc. Again, we'd use a default value and override extra options as necessary. Duncan

On 9 November 2011 03:02, Ian Lynagh wrote:

On Tue, Nov 08, 2011 at 01:07:23PM +0000, Duncan Coutts wrote:

...

We should take the same approach with openFile. Currently we have:

openFile :: FilePath -> IOMode -> IO Handle

I suggest changing the IOMode for a record of options which would include the IO mode (read/write/read-write/etc), and other things like file locking, default create permissions, create exclusive etc. Again, we'd use a default value and override extra options as necessary.

I've just reread the thread, and I think the best way forwards would be to make the original proposed change now, and for the above change to be proposed separately.

By making the proposed change now, we eliminate the unexpected locking behaviour, and the API effectively doesn't change (I say "effectively", because the semantics are obviously different, but I doubt anyone is relying on getting an exception due to the Haskell locking).

But you're still proposing removing all protections when the initial complaint was simply to change the behaviour on Windows to match the behaviour on Unix. That complaint is about Haskell processes locking files such that other processes cannot read them (which we all agree is annoying). A slightly different (and also legitimate) complaint is the current inability to opt-out of locking. So I think the best way forward is to scap the windows OS-level locking implementation and to use the RTS-based one on all platforms. Then separately we can talk about redesigning openFile to give people control over the locking (mostly the ability to opt-out). Of course you'll not hear complaints from people relying on getting a locking exception, that's the whole point about it being there to catch accidental cases. The only complaints you'll hear will be from the people who need to opt-out of locking (there are few such cases but they're important where they do occur). Are you suggesting we turn off all locking by default, and then turn it back on again by default once we change openFile? That kind of switching back and forth doesn't seem like a good idea. If you're saying now, lets turn it off and keep it off, then there's almost no point in changing openFile. Nobody is going to use openFile locking for the prevention of accidents. It has to be a default for that. And that was presumably the original rationale for H98. So far, I've not hear a single justification for turning off locking by default. Well, Ian says it's "unexpected", but has not addressed the purpose of the locking to prevent "unexpected"ly concurrently modifying a file that you're reading, i.e. said if he thinks this is worthless or not. The only justifications I've seen have been for changing the Windows locking approach to match unix, and the need for people to opt-out of locking. Note also that in future we could make the "unexpected" aspect better if the locking error message points people in the right direction to turn off locking (via the future openFile change), if they decide that's what they decide is the right solution. Duncan

On Wed, Nov 9, 2011 at 1:50 PM, Duncan Coutts wrote:

On 9 November 2011 12:13, Duncan Coutts wrote:

...

On 9 November 2011 03:02, Ian Lynagh wrote:

...

...

I've just reread the thread, and I think the best way forwards would be to make the original proposed change now, and for the above change to be proposed separately.

By making the proposed change now, we eliminate the unexpected locking behaviour, and the API effectively doesn't change (I say "effectively", because the semantics are obviously different, but I doubt anyone is relying on getting an exception due to the Haskell locking).

...

So far, I've not hear a single justification for turning off locking by default. Well, Ian says it's "unexpected", but has not addressed the purpose of the locking to prevent "unexpected"ly concurrently modifying a file that you're reading, i.e. said if he thinks this is worthless or not. The only justifications I've seen have been for changing the Windows locking approach to match unix, and the need for people to opt-out of locking.

Henning raises a good point:

writeFile path . process =<< readFile path

This is an excellent motivating example for why the Haskell committee included locking in the first place.

This example will currently give a runtime error (which no doubt could be improved). If we remove the default locking then this example changes from a situation where a new user gets an unexpected locking error into one where the new user gets unexpected file corruption. The latter is *much* worse.

Concurrently modifying a file that you are reading can happen in any language, so why is it that Haskell has locking by default and e.g. Python does not (except perhaps historical accident that Python copied C which has no protection for newbies). Well, it's simply more likely to happen in Haskell, epsecially for new users who do not understand the implications of lazy IO. Since we are not proposing to remove lazy IO then we should also not remove the default locking.

Experts don't make these kinds of mistakes, so only bump into the cases where they are intentionally modifying a file they're reading and so are annoied by the default locking. That's fine, the experts understand what is going on and can opt out.

Note also that readFile, writeFile etc are not a low level API where we should avoid adding extra "cleverness". The Handle API is high level, feature rich, convenient and used by newbies.

Note also that the file locking in the Haskell report has nothing to do with some OS's doing locking by default. The H98/2010 report says:

 Implementations should enforce as far as possible, at least locally to the Haskell process, multiple-reader single-writer locking on files.

http://www.haskell.org/onlinereport/haskell2010/haskellch41.html#x49-3240004...

All in all, the original reasons for including the default file locking still hold. All we need is a way for experts to opt-out.

Further, I claim that the proposal should not be adopted as is, because there is no consensus on the specific point of the default locking. There is no consensus because that specific point has not been addressed by more than two people. I suggest we extend the discussion period and address the specific issue.

I support Duncan's view. Have a way to opt-out for experts. -- Nicolas Pouillard http://nicolaspouillard.fr

On 9 November 2011 15:09, Jon Fairbairn wrote:

Duncan Coutts writes:

...

Henning raises a good point:

writeFile path . process =<< readFile path

This is an excellent motivating example for why the Haskell committee included locking in the first place.

Indeed. It’s a long time ago, but I dimly remember discussions on these lines. Certainly there was an issue with lazy IO (attempting to make it morally referentially transparent), but one of the arguments was that, ideally, things like readFile should behave as if they were atomic, perhaps by means of shadow pages. So in the ideal semantics, the above programme would do what it appears to do, but there were complaints about efficiency and difficulty of implementation, so we settled on the locking approach (with a nod towards the difficulty of implementation in Windows).

Aye, if the OS would let us take copy-on-write snapshots of files then that would be an improvement on locking while giving the same serialisability semantics. This is much like what databases did with things like MVCC multi-version concurrency control to increase concurrency by removing row or table-based locking and making copies as needed. Some new filesystems do support copy-on-write but the OS doen't yet provide much of an interface for apps to take advantage of it. Something that is almost as good is atomically writing files. Unfortunately this only works on Unix where you can atomically rename over existing files that are open by your own or other processes.

* * *

For the cases where it is a problem, we need new modes, don’t we?

Yes, or maybe locking modes orthogonal to the read/write mode. Duncan

On Tue, 2011-11-08 at 13:27 +0000, Ian Lynagh wrote:

On Tue, Nov 08, 2011 at 11:54:02AM +0000, Duncan Coutts wrote:

...

And if you switch the default and users have to do their own locking, how can they do it? I'm not sure there is any sensible way to do it. Any bit of IO code can open any file. If they're already tightly coupled and can share an MVar it's fine, but that's not the sort of accidental situation that the current implicit locking protects against.

We have the same problem now with any library which you FFI import that does any file IO.

Sure, but that doesn't mean we should remove all protections. Duncan