July 2015 - Haskell-Cafe

Discussion: The CLOEXEC problem
by Niklas Hambüchen 30 Aug '15

30 Aug '15

Hello Cafe, I would like to point out a problem common to all programming languages, and that Haskell hasn't addressed yet while other languages have. It is about what happens to file descriptors when the `exec()` syscall is used (whenever you `readProcess`, `createProcess`, `system`, use any form of `popen()`, Shake's `cmd` etc.). (A Markdown-rendered version containing most of this email can be found at https://github.com/ndmitchell/shake/issues/253.) Take the following function f :: IO () f = do inSomeTemporaryDirectory $ do BS.writeFile "mybinary" binaryContents setPermissions "mybinary" (setOwnerExecutable True emptyPermissions) _ <- readProcess "./mybinary" [] "" return () If this is happening in parallel, e.g. using, forkIO f >> forkIO f >> forkIO f >> threadDelay 5000000` then on Linux the `readProcess` might often fail wit the error message mybinary: Text file busy This error means "Cannot execute the program 'mybinary' because it is open for writing by some process". How can this happen, given that we're writing all `mybinary` files in completely separate temporary directories, and given that `BS.writeFile` guarantees to close the file handle / file descriptor (`Fd`) before it returns? The answer is that by default, child processes on Unix (`fork()+exec()`) inherit all open file descriptors of the parent process. An ordering that leads to the problematic case could be: * Thread 1 writes its file completely (opens and closes an Fd 1) * Thread 2 starts writing its file (Fd 2 open for writing) * Thread 1 executes "myBinary" (which calls `fork()` and `exec()`). Fd 2 is inherited by the child process * Thread 2 finishes writing (closes its Fd 2) * Thread 2 executes "myBinary", which fails with `Text file busy` because an Fd is still open to it in the child of Process 1 The scope of this program is quite general unfortunately: It will happen for any program that uses parallel threads, and that runs two or more external processes at some time. It cannot be fixed by the part that starts the external process (e.g. you can't write a reliable `readProcess` function that doesn't have this problem, since the problem is rooted in the Fds, and there is no version of `exec()` that doesn't inherit parent Fds). This problem is a general problem in C on Unix, and was discovered quite late. Naive solutions to this use `fcntl`, e.g. `fcntl(fd, F_SETFD, FD_CLOEXEC)`: http://stackoverflow.com/questions/6125068/what-does-the-fd-cloexec-fcntl-f… which is the equivalent of Haskell's `setFdOption` to set the `CLOEXEC` flag to all Fds before `exec()`ing. Fds with this flag are not inherited by `exec()`ed child processes. However, these solutions are racy in multi-threaded programs (such as typical Haskell programs), where an `exec()` made by some thread can fall just in between the `int fd = open(...); exec(...)` of some other thread. For this reason, the `O_CLOEXEC` flag was added in Linux 2.6.23, see e.g. `man 2 open` http://man7.org/linux/man-pages/man2/open.2.html to the `open()` syscall to atomically open a file and set the Fd to CLOEXEC in a single step. This flag is not the default in Haskell - but maybe it should be. Other languages set it by default, for example Python. See PEP-433: https://www.python.org/dev/peps/pep-0433/ and the newer PEP-446: https://www.python.org/dev/peps/pep-0446/ for a very good description of the situation. Python >= 3.2 closes open Fds *after* the `exec()` when performed with its `subprocess` module. Python 3.4 uses O_CLOEXEC by default on all Fds opened by Python. It is also noted that "The programming languages Go, Perl and Ruby make newly created file descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0 (1987) and Ruby 2.0 (2013)": https://www.python.org/dev/peps/pep-0446/#related-work A work-around for Haskell is to use `O_CLOEXEC` explicitly, as in this example module `System/Posix/IO/ExecSafe.hsc`: https://gist.github.com/nh2/4932ecf5ca919659ae51 Then we can implement a safe version of `BS.writeFile`: https://gist.github.com/nh2/4932ecf5ca919659ae51 Using this form of `writeFileExecSafe` helps in cases when your program is very small, when you can change all the code and you don't use any libraries that open files. However, this is a very rare case, and not a real solution. All multi-threaded Haskell programs that write and execute files will inherently trigger the `Text file busy` problem. We need to discuss what to do about this. Let us run this discussion on haskell-cafe and move to the libraries@ mailing list once we've got some ideas and opinions. My personal stance is that we should follow Python's example, and all functions in our standard libraries should open files with the O_CLOEXEC flag set. Niklas

6 12

[ANN] frpnow-0.12
by Atze van der Ploeg 25 Aug '15

25 Aug '15

Dear Cafe, We have released the (nearly) first version of FRPNow, the functional reactive programming library based on the ICFP 2015 paper "Principled Practical FRP: Forget the Past, Change the Future, FRPNow!" ( https://www.reddit.com/r/haskell/comments/3ai7hl/principled_practical_frp_f… ) The main package: http://hackage.haskell.org/package/frpnow Examples: https://github.com/atzeus/FRPNow/tree/master/Examples Gloss interoperability: http://hackage.haskell.org/package/frpnow-gloss GTK interoperability: http://hackage.haskell.org/package/frpnow-gtk (hackage doesn't like the newer GTK docs, so you can read the docs at http://www.cse.chalmers.se/~atze/frpnow-gtk/ ) Cheers, Atze

3 9

[ANN] base91
by Alvaro J. Genial 13 Aug '15

13 Aug '15

Howdy, I'd like to announce a first version of base91 <http://hackage.haskell.org/package/base91>, a Haskell package to enable the encoding and decoding of data in Base91 format, as documented in its specification <http://base91.sourceforge.net>. The main benefits of said scheme are a good reduction in overhead (from 31% up to 58%) when compared to Base64 due to the clever use of two printable ASCII characters (consuming 16 bits) to encode 13 arbitrary bits, as well as neither need for nor ambiguity over padding or alphabets. It includes support for... - Plain Strings (module Codec.Binary.Base91.String) - Data.ByteString (module Codec.Binary.Base91.ByteString) - Data.Text (module Codec.Binary.Base91.Text) - The latter two combined (module Codec.Binary.Base91.Efficient) - Data.ByteString.Lazy (module Codec.Binary.Base91.ByteString.Lazy) - Data.Text.Lazy (module Codec.Binary.Base91.Text.Lazy) - The latter two combined (module Codec.Binary.Base91.Efficient.Lazy) ...which are all just concretely typed versions of the generic implementation in Codec.Binary.Base91. I had fun exploring some of the latest GHC extensions, as evidenced by Codec.Binary.Base91.Control, which relies on a certain amount of type trickery to reconcile the incongruence between monomorphic containers and the standard type classes. Anyway, suggestions are very welcome. In particular, I'm interested in: 1. Whether the generic decode and encode signatures can be simplified or improved elsehow. 2. Whether there already exist classes similar in purpose to Applicative' and Foldable'. 3. Whether it makes sense to offer decode and encode variants that use lazy (left) folds--currently folding is always strict, even with lazy types. (Testing comes nearly free courtesy of QuickCheck and the identity property of decode . encode, plus a couple of concrete examples. Please open an issue or otherwise let me know if you find a bug.) Thank you, Alvaro <http://alva.ro>

4 5

Where is Haskell Weekly News syndicated?
by Jeremy 06 Aug '15

06 Aug '15

Haskell Weekly News used to be syndicated on http://contemplatecode.blogspot.com/ and http://sequence.complete.org/hwn. sequence only has HWN up to March 2010, while contemplatecode goes up to the beginning of February. Is there any way that I can subscribe to HWN (preferably as RSS/Atom), and not other stuff that doesn't interest me? -- View this message in context: http://haskell.1045720.n5.nabble.com/Where-is-Haskell-Weekly-News-syndicate… Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.

5 18

Re: [Haskell-cafe] ambiguous module name Network.URI
by Kees Bleijenberg 05 Aug '15

05 Aug '15

Thanks Christopher, -----Oorspronkelijk bericht----- Check out the Hackage page for network-uri: https://hackage.haskell.org/package/network-uri The parseURI function is in the version of network you are using already so there is no need to import modules from network-uri. ---------------------------------------- If I remove the import of Network-URI and leave the import Network.Http and use parseURI, ghc complains: not in scope parseURI. Do I have to change someting in the packages? Kees

3 2

Why is the no tuple syntax for sum types
by Silvio Frischknecht 05 Aug '15

05 Aug '15

Protuct types have this nice tuple syntax (a,Char,Int),(a,'c',1). Why is there no syntax like this for Sum types (a|Char|Int),(||1). This would be very usefull for some more explicit error handling. I.E. You have a library that has errors. Error1, .. ErrorN. Currently there are only two ways of making a function that can throw Error3 and Error8. Use 'Dynamic' typed Exceptions or make a general Error type that is the sum of all Errors. In both cases it is not clear from the type of the function what Errors can occur. With sumTuples you could just return (Result|Error3|Error8). Also, I guess I'm just one of those people who occasionally confuse Left and Right :) Silvio Frischknecht

7 9

STM Finalizers
by Michael Schröder 04 Aug '15

04 Aug '15

Hi everyone, I want to finally announce some work I did as part of my master's thesis [1]. It's an extension to GHC's STM implementation. The main idea is to introduce a new primitive function atomicallyWithIO :: STM a -> (a -> IO b) -> IO b Like the existing atomically operation, atomicallyWithIO performs an STM computation. Additionally, it takes a *finalizer*, which is an arbitrary I/O action that can depend on the result of the STM computation, and combines it with the transaction in such a way that: 1. The finalizer is only performed if the STM transaction is guaranteed to commit. 2. The STM transaction only commits if the finalizer finishes without raising an exception. A motivation of why this is useful: Say we want to save the results of some STM transaction, i.e. persist them to disk. Currently, this is not possible in a transactionally safe way. The naive approach would be to first atomically perform some STM computation and then independently serialise its result: do x <- atomically m serialize x There are two issues with this. First, another thread could perform a second transaction and serialise it before we have finished serialising ours, which could result in an inconsistent state between the memory of our program and what is stored on disk. Secondly, the function serialize might not terminate at all; it could throw an exception or its thread could die. Again we would end up with an inconsistent state and possibly data loss. We might try to simply move serialisation into the atomic block, but to do this we have to circumvent the types, which should already be a huge red flag: atomically $ do x <- m unsafeIOToSTM (serialize x) The problem here is of course that an STM transaction can retry many times, which will also result in serialize being executed many times, which is probably not something we want. Furthermore, if the thread receives an asynchronous exception, the transaction will abort in an orderly fashion, while serialize, with its irrevocable side effects, cannot be undone. But with finalizers, the solution is easy: atomicallyWithIO m serialize I've written a small example application that uses finalizers and other stuff from my thesis to build a lightweight STM database framework: [2] There are more possible uses cases besides serialisation (such as interactive transactions) and some interesting issues surrounding finalizers (like nesting of transactions) which are discussed in greater detail in my thesis [1], which also includes a formal operational semantics of the whole thing. I have implemented finalizers in my fork of GHC [3], if any of you want to play around with them yourself. The atomicallyWithIO function is exported from the GHC.Conc.Sync module. Cheers, Michael [1] https://github.com/mcschroeder/thesis [2] https://github.com/mcschroeder/social-example [3] https://github.com/mcschroeder/ghc

7 14

Different forms for RandomT?
by Zemyla 03 Aug '15

03 Aug '15

Normally, a monad transformer to provide a random number generator would be of the form StateT g, where g is a RandomGen. But I've seen some libraries (like QuickCheck) define their RandomT as: newtype RandomT g m a = RandomT { runRandomT :: g -> m a } with their monadic bind operation defined as (RandomT m) >>= f = RandomT $ \g -> let (ga, gb) = split g in m ga >>= (\a -> runRandomT (f a) gb) and return and fail as in ReaderT. Can someone describe the advantages and disadvantages of doing RandomT this way? I mean, if your generator has a subpar split operation (and most do), this will obviously exacerbate any problems with it. Does it give any comparable advantages?

3 3

Annotated ASTs, and the tension between Bound, Free, and Cofree...
by Merijn Verstraaten 02 Aug '15

02 Aug '15

Hi, So I'm playing around with using bound to define a simple AST using the bound library to deal with substitution/abstraction. Being a lazy haskeller I figured that ASTs map nicely onto Free and that I could get all the necessary Applicative, Monad, etc. instances for free by using something like: newtype Expr a = Expr { unExpr :: Free (ExprF Expr) a } deriving (Functor,Applicative,Monad,Foldable,Traversable) data ExprF f a = App (f a) (f a) | Lam Type (Scope () f a) | TmTrue | TmFalse | If (f a) (f a) (f a) deriving (Eq,Ord,Show,Read,Functor,Foldable,Traversable) Now, for an AST to be useful, it has to be annotated with things like source location, type info, etc. So I started looking into how to accomplish this in a way where it's easy to add/modify annotations on any node of the AST. My initial idea was to use FreeT with ((,) a) as a base monad for annotations, but I quickly realised this fails, because "((,) a)" is only a Monad if 'a' is a Monoid and there's no sensible Monoid for type information/source locations. Then I looked around more and realised that this actually sounds a lot like Cofree. But here's where I run into problems. Bound expects the type of variables to be in the functor position of my Expr type, whereas Cofree expects the type of *annotations* to be in the functor position. So I can't figure out an elegant way to both use bound to deal with substitution for me *and* use Cofree to deal with annotations, since there's no way to make the types line up sanely. So my question boils down too: Has anyone done all the hard work for me already? i.e. how can I have an annotated AST with substitution *and* an easy way to modify the annotations, without writing a ton of boiler plate to deal with substitution and/or annotations? Cheers, Merijn

4 4

ANNOUNCE: Haskell Platform 7.10.2
by Mark Lentczner 01 Aug '15

01 Aug '15

Haskellers, we are pleased to announce the release of Haskell Platform 7.10.2 *get it here... <https://www.haskell.org/platform/>* Highlights include: - GHC 7.10.2 - packages & tools bumped to very latest - major package updates to: - attoparsec - case-insensitive - cgi - GLUT, GLURaw, OpenGL, & OpenGLRaw - mtl - network - primitive & vector - QuickCheck - random - syb - text - new website - new installer script for linux - 10 bugs closed: Issues · haskell/haskell-platform <https://github.com/haskell/haskell-platform/issues?utf8=%E2%9C%93&q=is%3Acl…> This is our first release concurrent with GHC's release. You'll notice that the version number has changed to reflect that from now on, our aim is to keep HP releases sync'd to GHC releases. *Note:* Haskell Platform download pages are often cached - you may need to hit reload to see the release. *About the last minute update to text package:* *tl;dr: Vast majority of code will never notice the difference. Go ahead and get the HP now and start enjoying 7.10.2 You can always update to the point release when it comes out.* Despite all the testing, GHC central noticed just a day before release that there was a regression with text literals in 7.10.2. It was possible to work around the issue via a change to the text package, and text-1.2.1.3 was released just today. Alas, there wasn't enough time to re-build and test the platform installers between then and now, so this release has text-1.2.1.1. There will be a point release (7.10.2.1) of the Platform this weekend or next with the updated text package. The issue does not result in incorrect code, only slow compilation times for very text literal laden code (as in thousands of literals), and slightly higher one-time construction cost at run-time. In full stack builds and many other test builds, these slow downs were inconsequential. *Windows Notes:* The Haskell Platform on Windows now provides the MSys2 tools. These tools are needed when installing packages that use conf-tools (generally rare). These tools are not automatically placed onto the PATH in order avoid troubles due to MSys2 tools which have the same name as a standard Windows tool (e.g., echo, find, dir). - Mark "release monad" Lentczner Special thanks to Erik Rantappa, Wasif Hasan Baig, & Ben Gamari for the new web site. Randy Polen for Windows build wrangling.

4 6