On 08/07/2010 21:34, John Meacham wrote:

On Tue, Jul 06, 2010 at 09:48:20PM +0300, Roman Cheplyaka wrote:

...

Greetings! Now, can different compiler/interpreter authors make statements about support of this standard in their compilers please?

Jhc supports all the language features of haskell 2010 except pattern guards. (ironic as I am a huge fan of said feature.)

The library situation is more complicated, there is tension between haskell 98, haskell 2010, and supporting code in the wild, which often assumes some specific version of ghc's base, which doesn't quite conform to either. My current thinking is to give up supporting base directly as a clone of ghc's and rather support just haskell 98 and haskell 2010 modes "natively", then compatability libraries (called things like ghc-base-4 and ghc-base-3 perhaps) would be independent projects and not tied to jhc releases.

The story we tentatively plan to provide in GHC 6.14.1 is a haskell2010 package that provides exactly the API specified by the report (by definition, since the source was used to generate the report :-). The modules of haskell2010 overlap with base, so it will be impossible to depend on both haskell2010 and base, without using non-portable extensions like PackageImports. I hadn't realised before, but this situation is better for portability, because it discourages people from using base package modules in pure Haskell 2010 code. The downside is exactly the reverse: if you wanted to use modules from base, then you don't get to use the pure Haskell 2010 modules too (although the base versions are virtually identical at the moment). So, if things stay like this, then you can write pure Haskell 2010 with GHC in two ways: - using Cabal: say "extensions: Haskell2010" and "build-depends: haskell2010". Maybe Cabal should have a new "language" directive, because "extensions" is a bit strange. - using GHC standalone: say "-package haskell2010 -hide-package base". -XHaskell2010 will be the default for standalone GHC, although for Cabal we'll probably require you to say what the language is, in the interests of being explicit and for backwards compatibility with existing packages.

jhc 0.7.4 which supports garbage collection and a speedier runtime and better support for external build systems will be out soon. My goal is one more point release before 0.8.0 which will have full haskell 2010 and 98 support.

I haven't looked at the new jhc yet, but I have a question about the GC support: is it conservative or accurate? If accurate, how are you finding the pointers - a shadow stack? Cheers, Simon

On 12/07/2010 12:43, John Meacham wrote:

On Mon, Jul 12, 2010 at 10:07:28AM +0100, Simon Marlow wrote:

...

The story we tentatively plan to provide in GHC 6.14.1 is a haskell2010 package that provides exactly the API specified by the report (by definition, since the source was used to generate the report :-). The modules of haskell2010 overlap with base, so it will be impossible to depend on both haskell2010 and base, without using non-portable extensions like PackageImports.

The way Jhc handles module overlap with packages is that it allows re-exporting modules from different libraries transparently, and reports conflicts lazily, as with normal haskell identifiers. For example, the 'haskell98' package might have a field 'Reexport-Modules:' with an entry 'Compat.Haskell98.Prelude as Prelude', meaning that if you pass -phaskell98 on the command line, you get a 'Prelude' that is transparently remapped to 'Compat.Haskell98.Prelude' under the scenes. This means that you can happily have both -phaskell98 and -phaskell2010 and get access to all of both their modules because they have identical 'Prelude' modules so will both re-export the same underlying module (implemented in some common base package) and there will be no conflict reported. Just like it is okay if you import the same haskell function from two different modules as long as they refer to the same original function.

Things are trickier if they do both -phaskell2010 and -pghc-base-3 since they conflict on some module names, there is nothing wrong with linking against both of them, but if you do an 'import Data.List' then a conflict will be reported as that particular module is different between the two. But if you only use the 'Prelude' or other modules that coincide, there will be no trouble.

Right. I like the idea of packages being able to declare re-exported modules, indeed I considered doing this in GHC (when we needed base3) but decided not to mainly because we would still need PackageImports, and once you have PackageImports you can implement re-exports using that, albeit with more boilerplate. The advantage of direct re-exports, as you say, is that it avoids a conflict when the module is imported. However, since some of the modules would conflict anyway, there didn't seem to be much advantage in avoiding the conflict in some cases but not others.

I think the ability for a package to define an 'interface' built up from other re-exported modules will be very useful going forward, I am not sure how hard something like that would be to implement in ghc, but it may be worth it in the future.

Isn't that what PackageImports lets you do?

...

I hadn't realised before, but this situation is better for portability, because it discourages people from using base package modules in pure Haskell 2010 code. The downside is exactly the reverse: if you wanted to use modules from base, then you don't get to use the pure Haskell 2010 modules too (although the base versions are virtually identical at the moment).

I would worry that it would discourage people from using 'haskell2010' at all to some degree. Are there issues in ghc with one library being built on haskell2010 and another being built on base and a third wanting to use both of them? If not, then I don't think it will be too bad.

No, there's no problem with mixing packages in that way.

The situation will be no worse than it is now, as it is, pretty much every library out there depends on 'base' making it theoretically incompatible with jhc. (in practice, most things compile just fine if I simply ignore the overly pedantic cabal dependencies and just add the obvious dependencies based on imports, aka, the franchise heuristic).

And hopefully things will improve over time, as fewer packages will need to depend on base. We could also start pulling out APIs that are currently in base into separate packages, without actually pulling out the code - e.g. concurrency, and that would be a good way to migrate people away from depending on base, but without breaking everything.

In the past I just had the boehm GC and the cross your fingers and hope static analysis can catch everything options. But as of 0.7.4 I have a real gc that I hope to make the new default in the next major release. (enabled with -fjgc on the command line)

It is an accurate GC that is implemented in portable C. I am using something based on the paper 'Accurate garbage collection in an Uncooperative Environment'[1] though the technique was independently discovered. I always pass the GC parameter as the first argument to functions, which is mapped to the same register, so I effectively have a dedicated register without having to resort to a dedicated declared register in gcc. Plus I can omit the parameter in leaf functions that don't allocate and free up the register for normal use. I compile with -mregparm=2 on i386 so the first two arguments to a function get mapped to registers.

I found that an independent shadow stack actually is faster than using the linked version described in the paper, (though, still passing around a pointer to the top of the stack as described), my theory being that taking the address of a stack allocated object will inhibit certain gcc optimizations.

The underlying allocator is based on Bonwick's slab allocator[2] which works quite well for a haskell runtime, I have a slab for each type, so a slab of 'cons' cells, a slab of size 3 tuples, and so forth.

[1] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.19.5570 [2] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.143.4374

Interesting. This is quite a different set of decisions to the way GHC does things, so I look forward to comparing some benchmarks. My intuition is that if allocation is anything more complicated than "reg += size; if (reg > lim) { ... }" then you have a lot of work to do to make up for the overhead, but let's see! Cheers, Simon

On 12/07/2010 18:29, Felipe Lessa wrote:

On Mon, Jul 12, 2010 at 9:50 AM, Simon Marlow wrote:

...

And hopefully things will improve over time, as fewer packages will need to depend on base. We could also start pulling out APIs that are currently in base into separate packages, without actually pulling out the code - e.g. concurrency, and that would be a good way to migrate people away from depending on base, but without breaking everything.

Is the plan to stop using base and start using 'haskell2010', even if you use extensions that aren't Haskell 2010?

You can make an independent choice about whether to use the Haskell 2010 language, and/or the Haskell 2010 libraries. If you need to use modules from base that are not part of Haskell 2010, then you will not be able to use the haskell2010 package, but you will still be able to use {-# LANGUAGE Haskell2010 #-}. Cheers, Simon

On 12/07/2010 22:12, John Meacham wrote:

On Mon, Jul 12, 2010 at 01:50:01PM +0100, Simon Marlow wrote:

...

Right. I like the idea of packages being able to declare re-exported modules, indeed I considered doing this in GHC (when we needed base3) but decided not to mainly because we would still need PackageImports, and once you have PackageImports you can implement re-exports using that, albeit with more boilerplate. The advantage of direct re-exports, as you say, is that it avoids a conflict when the module is imported. However, since some of the modules would conflict anyway, there didn't seem to be much advantage in avoiding the conflict in some cases but not others.

Well, a main useful case is that I can do -phaskell98 and -phaskell2010 at the same time. So I can make the default jhc behavior be the union of the two languages easily.

That works in GHC too: the modules of those two packages don't overlap. That is partly because we never moved Prelude from base to haskell98. But suppose the Haskell 2010 Prelude had some differences from the Haskell 98 Prelude - then a direct re-export wouldn't work, and you'd be stuck with conflicts again. So my main point is that while direct re-exports avoid conflicts sometimes, they're not a general solution, and eventually you run into the same problems again. There are differences between several Haskell 2010 modules and the equivalent base modules, so you won't be able to use re-exports for those.

There is also some philosophy behind it. I don't like coupling things that don't need to be coupled, such as the interface a library exports and the place that interface is implemented.

I agree with the philosophy, but I don't understand how package re-exports are part of its implementation - the Haskell module system already lets you re-export entities and therefore decouple the implementation from the API.

...

...

I think the ability for a package to define an 'interface' built up from other re-exported modules will be very useful going forward, I am not sure how hard something like that would be to implement in ghc, but it may be worth it in the future.

Isn't that what PackageImports lets you do?

Hmm.. maybe. Jhc doesn't have PackageImports, the idea being that you re-export modules rather than importing them from a specific package and exporting their interface. So, it sort of shifts the effort from the interface makers to the implementors, as in, the Haskell 98 Prelude will actually have to be in a module called Compat.Haskell98.Prelude and just re-exported by the 'haskell98' and 'haskell2010' modules.

I don't think the difference is as fundamental as you make out. In Haskell you can already write module A ( module B ) where import B which is the essence of re-exporting a module, and preserves the interface/implementation separation. Now suppose A and B are the same module name, from different packages. To implement this re-export you can either (1) Use a package re-export (in JHC) (2) Somehow avoid A and B being the same, e.g. by having an intermediate like Compat.Haskell98.Prelude (3) Use PackageImports (in GHC) the advantage of (1) is that you can have both packages in scope without a conflict, whereas in (2/3) the two modules can export different interfaces, while still separating the implementation from the interface. I think of PackageImports as just an optimisation that lets you avoid the intermediate module in (2). So again, my point is that in the general case the APIs diverge, so you don't get to do (1), or you only get to do it for some modules, which could be confusing for users.

In one sense 'Compat.Haskell98' sort of acts as a package import, but it isn't actually tied to a package, it just refers to the Compat.Haskell98.Prelude in scope. I also have a large aversion to allowing package names to infect source files carte blanche. It blurs the line between the language itself and the environment the language runs in. It may be neccesarry at some point, but I'll put it off until a case comes up that makes it abundantly clear that it is needed.

I agree with your aversion to putting package names in the source code, which is why we don't advertise PackageImports as an extension that we expect end users to use, and we don't let you use version numbers. Cheers, Simon

On 14/07/2010 03:36, John Meacham wrote:

On Tue, Jul 13, 2010 at 10:24:00AM +0100, Simon Marlow wrote:

...

...

Well, a main useful case is that I can do -phaskell98 and -phaskell2010 at the same time. So I can make the default jhc behavior be the union of the two languages easily.

That works in GHC too: the modules of those two packages don't overlap. That is partly because we never moved Prelude from base to haskell98.

But don't you still have to have things directly declare they depend on 'base' then in order to get 'Prelude'? The extra dependency on the implementation specific 'base' because you want both haskell98 and haskell2010 is what I am trying to avoid.

In GHC 6.14.1 you'll be able to depend on haskell2010 instead of base if you wish, and we'll recommend doing so where it makes sense (indeed, if you use haskell2010 then you *cannot* also depend on base). Also, haskell98 and haskell2010 can be used together if you really want to. Cheers, Simon

On Wed, Jul 14, 2010 at 10:35:50AM +0100, Simon Marlow wrote:

...

Yeah, I didn't realize how important the allocator was until I started benchmarking, spending time cutting the cost of marking garbage in half didn't help nearly as much as shaving a few cycles off the allocator. The fast pass of the allocator is actually very fast, each object type has its own allocator and free list so allocation is pretty much just pulling an object off of the free list, it is already of the appropriate size and its constant fields are pre-initialized as they arn't cleared during garbage collection. (there is a heuristic that claims full pages back to the general pool sometimes).

The slow path has to grab a full page and initialize it, but that isn't really much slower as I can prefetch the cache lines needed so the cost is on the order of another cache line fill. (thinking about computational complexity in terms of O(cache line fills) rather than O(operations) is much more appropriate on todays architectures.).

Right, you can see how important locality is by looking at these graphs that Don produced recently:

http://haskell.org/haskellwiki/Ghc-gc-tune

generational GC these days is important more for locality than for the benefits of avoiding repeated tracing.

Yeah, jhc's GC is currently not generational, it is certainly something I want to do, something I have considered as a quick hack was splitting up allocations based on whether they were updatable or not. allocating things that can statically be determined to be in HNF into an "older" generation and allocating updatable thunks in a younger one. The theory being updatable thunks are more likely to be overwritten by a indirection and become garbage soon, I was thinking this could give me some of the benefits of a generational collector without having to rewrite my GC around being able to promote objects to an older generation. I am not sure if it will work, but it is an idea I have been toying with. Implementing better heap profiling support for jhc generated code is a prerequisite in any case.

Speaking of prefetching, we get a lot of benefit in GHC from the automatic prefetching done by modern CPUs; I'm not sure how this would be affected by having multiple allocation regions. Manual prefetching is almost impossible to get right in my experience, see also Nethercote/Mycroft where they did some prefetching experiments with GHC:

http://portal.acm.org/citation.cfm?id=773044

Ah, this paper looks very interesting, I was wondering if you had experimented with prefetching just ahead of the allocation pointer. Looks like it helped :) John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/