Growing Haskell Platform

In my Haskell Implementors 2012 talk Haskell Platform: Field Report and Future Goals, I called for an expansion of the content of the platform to bring it up to par with what other programming language platforms provide. Please see my blog post today: http://mtnviewmark.wordpress.com/2012/12/06/growing/) Where I outline what is missing, and call for a ramp up in our efforts to grow the platform. - Mark

Crypto is rather well addressed by Vincent's packages
(crypto{cipher,hash}). Please don't use Crypto as it's not really for
production use. In some cases we could do with high performance
bindings to C instead of pure Haskell, and I'm willing to put in a
little effort to see that happen.
The staging page needs fleshed out with not just the immediate
packages but also their build deps.
Thomas
On Thu, Dec 6, 2012 at 11:24 PM, Mark Lentczner
In my Haskell Implementors 2012 talk Haskell Platform: Field Report and Future Goals, I called for an expansion of the content of the platform to bring it up to par with what other programming language platforms provide.
Please see my blog post today: http://mtnviewmark.wordpress.com/2012/12/06/growing/)
Where I outline what is missing, and call for a ramp up in our efforts to grow the platform.
- Mark
_______________________________________________ Haskell-platform mailing list Haskell-platform@projects.haskell.org http://projects.haskell.org/cgi-bin/mailman/listinfo/haskell-platform

On Fri, Dec 7, 2012 at 9:24 AM, Mark Lentczner
In my Haskell Implementors 2012 talk Haskell Platform: Field Report and Future Goals, I called for an expansion of the content of the platform to bring it up to par with what other programming language platforms provide.
Please see my blog post today: http://mtnviewmark.wordpress.com/2012/12/06/growing/)
Where I outline what is missing, and call for a ramp up in our efforts to grow the platform.
I agree with Thomas's call for using Vincent's crypto packages. I also think blaze-html is the best option for HTML generation. Let me bring up one other package: yaml (written by me). I think it's a pretty good fit for the standard YAML packaging library, since it simply reuses the existing infrastructure from the aeson package (i.e. the ToJSON and FromJSON typeclasses, and the Value datatype). But I'm actually a bit concerned about proposing it, based on some history. There was a major improvement put forward in version 0.11.2 of the text package, which meant that compile times were drastically reduced due to a better set of rewrite rules for pack. This affected a number of my packages directly, particularly persistent, which had a large number of compile-time Text values for representing table and column names. Moving from text 0.11.1 to 0.11.2 meant in some cases that programs transformed from uncompilable to compilable. In other cases, it was the difference between 10 minute compiles and 10 second compiles. I thought the natural response to this would be to put a lower bound on text. After all, following the PVP, there should be no problem with updating a minor version number. However, when I did so, many users of the Haskell Platform were no longer able to compile due to conflicting dependencies. The result: I had to put in some dirty conditional compilation tricks in persistent to avoid inlining `pack` when using older versions of text. Now this problem may be completely resolved in newer versions of cabal-install, but I think having a large number of packages in the global package database pegged at specific versions is a very strong recipe for reintroducing version hell. Coming back to yaml: it depends on conduit. Suppose after the HP is released, I release a new version of conduit. And suppose some other package (say, xml-conduit) depends on this newer version. What happens when the user tries to install a package that depends on the newer xml-conduit and yaml at the same time? Ideally I'd want them to get yaml recompiled against a newer version of conduit, but conflicting user and global databases can be a very sore point. I'm all in favor of increasing the scope of the HP, but having a lot of version numbers set in stone could be problematic. Perhaps a solution would be to have the HP packages go into the user package database instead of the global database, I'm really not sure. Michael

Now this problem may be completely resolved in newer versions of cabal-install, but I think having a large number of packages in the global package database pegged at specific versions is a very strong recipe for reintroducing version hell. Coming back to yaml: it depends on conduit. Suppose after the HP is released, I release a new version of conduit. And suppose some other package (say, xml-conduit) depends on this newer version. What happens when the user tries to install a package that depends on the newer xml-conduit and yaml at the same time? Ideally I'd want them to get yaml recompiled against a newer version of conduit, but conflicting user and global databases can be a very sore point.
But this is exactly the point of the "Haskell Platform": given some set of specific package versions, install them at the very beginning, and have all other packages be built with those packages. If you want to update one of those packages, you are almost definitionally going to have to rebuild--and this won't go away until Haskell libraries with no API changes get binary compatibility. Edward

On Fri, Dec 7, 2012 at 10:20 AM, Edward Z. Yang
Now this problem may be completely resolved in newer versions of cabal-install, but I think having a large number of packages in the global package database pegged at specific versions is a very strong recipe for reintroducing version hell. Coming back to yaml: it depends on conduit. Suppose after the HP is released, I release a new version of conduit. And suppose some other package (say, xml-conduit) depends on this newer version. What happens when the user tries to install a package that depends on the newer xml-conduit and yaml at the same time? Ideally I'd want them to get yaml recompiled against a newer version of conduit, but conflicting user and global databases can be a very sore point.
But this is exactly the point of the "Haskell Platform": given some set of specific package versions, install them at the very beginning, and have all other packages be built with those packages. If you want to update one of those packages, you are almost definitionally going to have to rebuild--and this won't go away until Haskell libraries with no API changes get binary compatibility.
Edward
I agree that you almost certainly will have to rebuild, my point is that the current setup strongly discourages this rebuilding. Let me give a more extreme example: suppose that the bug in text hadn't simply been slower compile times, but instead it segfaulted every time you appended two empty strings. What would be our response to users? Should we tell them to stick with the current HP until the next one is released, and hope they don't trigger the bug? Should they try and convince their whole system to get rebuilt? I think as the HP gets larger, we need to come up with an answer to the question of upgrading included packages. Michael

Excerpts from Michael Snoyman's message of Fri Dec 07 00:29:14 -0800 2012:
Let me give a more extreme example: suppose that the bug in text hadn't simply been slower compile times, but instead it segfaulted every time you appended two empty strings. What would be our response to users? Should we tell them to stick with the current HP until the next one is released, and hope they don't trigger the bug? Should they try and convince their whole system to get rebuilt?
I think the pragmatic solution here is to roll a platform point-release, just fixing the particular major bug. Edward

On Fri, Dec 7, 2012 at 10:50 AM, Edward Z. Yang
Excerpts from Michael Snoyman's message of Fri Dec 07 00:29:14 -0800 2012:
Let me give a more extreme example: suppose that the bug in text hadn't simply been slower compile times, but instead it segfaulted every time you appended two empty strings. What would be our response to users? Should we tell them to stick with the current HP until the next one is released, and hope they don't trigger the bug? Should they try and convince their whole system to get rebuilt?
I think the pragmatic solution here is to roll a platform point-release, just fixing the particular major bug.
Edward
That seems suboptimal from a few different standpoints: * HP maintainers will have to do a lot of work every time a bug is detected. * Users will need to redownload and reinstall the HP every time a bug is detected. In this process, they will almost certainly need to recompile *all* of their packages, instead of just the packages depending on the broken package. * Package maintainers cannot always get version bumps they need. The text example I gave is a perfect case: I would not expect the HP team to turn around and release a brand new HP to improve compile times of my users. But persistent users would have liked to be able to opt-in to getting that update, even if it meant recompiling everything. Perhaps we need to think about this as OSes do. Ubuntu will release version 12.04 (LTS), and then provide security patches as necessary. When enough security patches come in, they'll ultimately create a point release and then go through the effort of spinning a new CD. We're in a trickier situation due to the ABI issues, but I don't think we can solve this by simply forgoing interim updates and requiring a new point release for every fix we need. Michael

On Fri, Dec 7, 2012 at 12:50 AM, Edward Z. Yang
Excerpts from Michael Snoyman's message of Fri Dec 07 00:29:14 -0800 2012:
Let me give a more extreme example: suppose that the bug in text hadn't simply been slower compile times, but instead it segfaulted every time you appended two empty strings. What would be our response to users? Should we tell them to stick with the current HP until the next one is released, and hope they don't trigger the bug? Should they try and convince their whole system to get rebuilt?
I think the pragmatic solution here is to roll a platform point-release, just fixing the particular major bug.
This is the correct solution in this case. This is e.g. what Python would do. Note that we're quite conservative in what versions of libraries we include in the platform (i.e. we try to not use something that was released yesterday) so we hope we won't have too many bugs that are major enough to require a new platform release. The reasoning here is that if people have gotten by for e.g. the last 6 months without running into the bug they probably can live with it until the next platform release. -- Johan

* Michael Snoyman
Let me bring up one other package: yaml (written by me). I think it's a pretty good fit for the standard YAML packaging library, since it simply reuses the existing infrastructure from the aeson package (i.e. the ToJSON and FromJSON typeclasses, and the Value datatype). But I'm actually a bit concerned about proposing it, based on some history.
When I was looking at existing YAML libraries, I rejected yours when I saw JSON types. (I did see the "do not let that confuse you, it's intentional" warning, and no, it didn't help.) Besides giving the feeling that the API is a hack, this will inevitably confuse readers of my code who are not familiar with the library and, just based on the names, will assume that the code is working with JSON, not YAML. IIUC, "all of the existing tools for JSON processing" are mainly just existing FromJSON instances. I am not sure how common this situation is, but for that you could define a newtype whose FromYAML instance would internally use the FromJSON instance of the underlying type. Or perhaps the JSON-agnostic subset of aeson could be extracted into a separate library, which both aeson and yaml would depend on. As for toYAML/toJSON, I guess most of the time they are different anyway — otherwise it's defeating the purpose of YAML to be more human-readable than JSON. Roman

On Fri, 2012-12-07 at 10:54 +0200, Roman Cheplyaka wrote:
When I was looking at existing YAML libraries, I rejected yours when I saw JSON types. (I did see the "do not let that confuse you, it's intentional" warning, and no, it didn't help.)
Besides giving the feeling that the API is a hack
+1. I had the exact same thoughts when passing by this library. Nicolas

* Mark Lentczner
In my Haskell Implementors 2012 talk Haskell Platform: Field Report and Future Goals, I called for an expansion of the content of the platform to bring it up to par with what other programming language platforms provide.
Please see my blog post today: http://mtnviewmark.wordpress.com/2012/12/06/growing/)
Where I outline what is missing, and call for a ramp up in our efforts to grow the platform.
Couple of thoughts: 1. file utils I would suggest filemanip [1]. It has good coverage and sane API. Last time I looked, there were no good alternatives to it on hackage. [1]: http://hackage.haskell.org/package/filemanip 2. test framework I have mixed feelings here. On the one hand, it would be good to have one in the platform. On the other hand, we're seeing some competition in this area (the main competitors being test-framework, HTF and hspec), and including one of them into the platform would give it advantage on the market and might discourage development of its alternatives. Roman

PHP's Zend Framework actually offers two releases[1]: a minimal release, and a full release. Perhaps we should consider a similar setup. "haskell-platform-minimal" would contain the same packages that are currently packaged with the platform, while "haskell-platform-full" would also contain several additional packages that significantly improve the Out of the Box experience. At least in the beginning, this setup with two platforms may make it easier to mature the new packages that we would like to include: "haskell-platform-minimal" would still offer the solid basis that the platform offers today, while "haskell-platform-full" is a work-in-progress towards a more feature-rich OOTB experience (which, of course, will eventually mature as well, at which point packages from "full" could be ported to "minimal").
Something completely different: I would very much like to see an email library be included by default as well. One that "just works".
Jurriën
[1] http://framework.zend.com/downloads/latest
On 7 Dec 2012, at 09:17, Roman Cheplyaka
* Mark Lentczner
[2012-12-06 23:24:41-0800] In my Haskell Implementors 2012 talk Haskell Platform: Field Report and Future Goals, I called for an expansion of the content of the platform to bring it up to par with what other programming language platforms provide.
Please see my blog post today: http://mtnviewmark.wordpress.com/2012/12/06/growing/)
Where I outline what is missing, and call for a ramp up in our efforts to grow the platform.
Couple of thoughts:
1. file utils
I would suggest filemanip [1]. It has good coverage and sane API. Last time I looked, there were no good alternatives to it on hackage.
[1]: http://hackage.haskell.org/package/filemanip
2. test framework
I have mixed feelings here. On the one hand, it would be good to have one in the platform. On the other hand, we're seeing some competition in this area (the main competitors being test-framework, HTF and hspec), and including one of them into the platform would give it advantage on the market and might discourage development of its alternatives.
Roman
_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On Thu, Dec 06, 2012 at 11:24:41PM -0800, Mark Lentczner wrote:
In my Haskell Implementors 2012 talk Haskell Platform: Field Report and Future Goals, I called for an expansion of the content of the platform to bring it up to par with what other programming language platforms provide.
Please see my blog post today: http://mtnviewmark.wordpress.com/2012/12/06/growing/)
Where I outline what is missing, and call for a ramp up in our efforts to grow the platform.
Hi Mark, I agree in general with the goals of growing the haskell platform; I more than once expressed the same interests. However I really don't see how it can happens at the moment. The bar is too high in general. While i can understand the reason (and agree) for most of the requirements on the list of acceptance, in general this is working against the platform. I would prefer the platform to offer many more good packages that maybe doesn't meet all the high quality requirements, but provide a necessary feature for users, and see them improve over time through the platform. At the moment, I think we expect perfect packages that will enter a freezing state after being accepted. In my mind, having multiple levels of platform could be a good idea. the current platform would be platform-base, and you could have something like platform-unstable, platform-experimental, that would mark the intent of being in the platform: - users can start rely on the package with the clear expectation of change in next versions. - users/devs can proactively improve packages that are missing stuff I think there's clearly a middle ground between the graveyard-like property of the platform and pure cabal/hackage usage. I believe this is being worked on, from the other end with Michael's stackage, but more effort to meet in the middle would be a good idea. -- Vincent
participants (9)
-
Edward Z. Yang
-
J. Stutterheim
-
Johan Tibell
-
Mark Lentczner
-
Michael Snoyman
-
Nicolas Trangez
-
Roman Cheplyaka
-
Thomas DuBuisson
-
Vincent Hanquez