On Tue, Jun 23, 2009 at 05:29:23PM -0700, Corey O'Connor wrote:

Outside of the compiler itself: A JHC equivalent to ghc-pkg would be nice. That, I'd imagine, would require some Cabal library integration.

The equivalent of ghc-pkg is delightfully simple. to list the packages you have installed, do a ls ~/lib/jhc to install new ones, simply do a cp package.hl ~/lib/jhc (the ~/lib/jhc is configurable, that is just where I keep my jhc packages) Any hl files in jhc's search path are considered available, jhc has no issues with package conflicts as it uses a hash of the API to uniquely identify packages, the human readable name is just a convinience. This and the portability of jhc libraries means the equivilant of something like cabal-install would simply be to list the dependencies and 'wget' them if they don't exist. This would probably be a pretty straightforward project if someone wanted to work on it, you would just need to add simple hooks to jhc to dump a list of package dependencies as hashes, and get the hash-name for a given hl file. The downloader could be a separate program then, called jhc-pkg.

As for the compiler:

0. The performance of the jhc compiler is sometimes downright slow. Given your description of how JHC works it makes sense why the compile times can be long. Looking into optimizing this would be nice.

Indeed. This is a major issue that I have not payed enough attention too. Back when lhc was a fork of jhc they did some very good profiling work and found some bottlenecks and I backported their fixes into jhc. I would love some help here. The makefile should generate a profiled build of jhc, just do 'make jhcp'. I have no doubt there are a lot of low hanging fruit that can be found and fixed.

1. Additional documentation of the compiler source. This isn't much of a problem given the code is quite clear.

Thanks! I tried to make it pretty clear, there are some huge ugly bits, but hopefully they are sectioned off. distressingly, Main.hs is one of those ugly files which can turn people off if they don't start looking in other files. Cleaning up Main would be nice.

2. Improved garbage collector? This might just be a documentation issue. The documentation states the region-based garbage collector leaks memory. And the configure script hints at being able to use a standard Boehm garbage collector library. Does the region based garbage collector still have issues? When should one be used over the other?

yeah, to compile with the boehm collector, simply add the -fboehm command to the command line when compiling. alternatively you can add a gc=boehm line to your targets.ini file to choose the collector on a target by target basis. This probably is the major issue with jhc right now. static analysis is fine for short running programs, but no good if your program runs a while. boehm works, but is pretty slow. This is tied in to #3 below actually. For some potential back ends (JVM,cmm) garbage collection becomes a lot easier to implement. So perhaps my time is better spent working on alternate back ends, on the other hand, the gcc back end is more 'universal' and works pretty well in practice. Now that jhc emits loops directly rather than relying on tail calls, GC really is the only thing that using gcc hurts.

3. Direct code generator instead of relying on the C compiler. In addition to opening up some generated code optimization possibilities this may tie in with #0 above. Perhaps some compiler performance optimizations are only possible with a native code generator?

It probably won't help much with #0, We would still have to perform the grin translations to get the code into a form we can send out as assembly and actually calling 'gcc' doesn't take much time relative to the rest of jhc. I definitely want to have multiple 'blessed' back ends in jhc. The main ones I have been interested in are a c-- back end and at least one 'VM' style back end. such as the JVM, .NET, or Dalvik. Dalvik actually looks like it might be the easiest, but targeting JVM might be more useful.

I would like to get involved in JHC more and see me mostly able to contribute to #0 and #1. Specifically I plan on attacking #0 and documenting everything I read in the source along the way.

That would be excellent! If you find yourself writing a lot and want to add a chapter to the manual, simply add comments like so {-@ChapterName markdown formatted text -} in the appropriate file then add ChapterName to utils/stitch.prl in the list depending on where you want it to go. multiple sections with the same chaptername are merged, you can add a section number after the chapter name to force an explicit order on the sections during merging. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On Tue, Jun 23, 2009 at 09:44:50PM -0400, Rick R wrote:

I don't know if you follow haskell cafe, but there was a lot of buzz over a new iPhone target. for GHC. It apparently took some hacks of the ghc source to get it into that state.

Yeah, that is part of the reason I pushed out the new release, since it makes compiling for targets like the iPhone a lot simpler. I was using jhc to write haskell programs for the iPhone back with my Gen1 phone before the AppStore existed. (jailbroken of course)

There is a lot of interest in being able to easily target other platforms, if it supported JVM that would be huge. Compiling to C/Native for the arm platform would be pretty great as well.

Compiling to the ARM works just fine via gcc right now. I have tested MIPS, ARM, and x86, but I don't think there is any reason jhc won't work with any back end that gcc supports. I would like to do a JVM target, but there are a couple things holding me back. The first is that a clear mapping from Grin to JVM isn't clear. jvm doesn't support some vital things such as structures (without the class overhead) and unsigned arithmetic (this may not be true anymore, does anyone know?) The other thing is library support, We would need to rewrite the Prelude functions to refer to JVM primitives rather than 'fread' 'fopen' and whatnot. I don't think the task is difficult, but it's time consuming and I don't know enough about the JVM api to get started. A syntax for 'foreign' declarations importing java primitives would also be needed. I have a syntax for dotnet primitives, perhaps it can be re-used pretty straightforwardly for java. In any case, I think a JVM or .NET back end would be quite doable, but it is not something I can do without help/support from someone fluent in the given VM and its APIs.

I'm sorry to hear about the region based GC. It would be neat if it worked. As an idea, it seems like the holy grail of GC. But everyone I've talked to says that it seems easy at 1st, but bucketing objects into regions seems to frequently place too much into the global pool.

Well, it is not lost work. as any effort I put into static analysis (which I lump all compile-time garbage reduction techniques) puts less of a strain on the run-time garbage collector and improves cache efficiency. No doubt some hybrid will evolve. Region inference can tell me what can be allocated on the stack for instance, and that is a big win even if I don't have general regions.

Timber sports a nice, incremental garbage collector that can be made to guarantee sub millisecond response times. Perhaps one could take some inspiration from it as a replacement for the current GC.

I'll look into it, an issue is that I am looking for a GC I can use with the gcc backend. Other backends (such as cmm or jvm) come with their own garbage collectors, and although a native code generator would be neat, I wouldn't want to rely on one. In fact, rather than write a native code generator, I would likely want to write a standalone c-- compiler that I can bundle with jhc. In the end, grin is close enough to c-- that I don't think I lose anything in the translation. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On Wed, Jun 24, 2009 at 11:17:29AM -0700, Taral wrote:

On Wed, Jun 24, 2009 at 7:15 AM, John Meacham wrote:

...

I'll look into it, an issue is that I am looking for a GC I can use with the gcc backend. Other backends (such as cmm or jvm) come with their own garbage collectors, and although a native code generator would be neat, I wouldn't want to rely on one. In fact, rather than write a native code generator, I would likely want to write a standalone c-- compiler that I can bundle with jhc. In the end, grin is close enough to c-- that I don't think I lose anything in the translation.

Doesn't LLVM have a GC?

Yes, I believe so. I have looked into LLVM a couple times and decided not to use it. (at least for now). The type system doesn't quite mesh with jhc's and it's primitives don't match up enough for it to be an issue. I think as far as back ends go, a C-- one will be a much better path to a processor-neutral assembly language than LLVM. I may change my mind at some point, and certainly if someone who knows LLVM more wants to look into it then I'd be interested. But for now it is not on my radar. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On Wed, Jun 24, 2009 at 02:56:24PM -0400, Rick R wrote:

...

I'll look into it, an issue is that I am looking for a GC I can use with the gcc backend. Other backends (such as cmm or jvm) come with their own garbage collectors, and although a native code generator would be neat, I wouldn't want to rely on one. In fact, rather than write a native code generator, I would likely want to write a standalone c-- compiler that I can bundle with jhc. In the end, grin is close enough to c-- that I don't think I lose anything in the translation.

Timber's only target at the moment is C. So you can use it with gcc.

Excellent, I'll look at how they solve the problem.

Let me see if I understand your intention for the long term correctly: For native code jhc would emit C/C++ and use the native garbage collector. For everything else it would emit c-- which would then get compiled to the appropriate VM, utilizing that VM's GC?

No, C-- is independent of a VM implementation. I am thinking 3 independent back ends, C - uses gcc, fully complete and good except for the lack of a good GC. C-- - basically this is the native code generator, c-- is a portable assembly code at a much lower level than C, I'd then use or write a C-- -> native assembler. C-- supports garbage collection, it doesn't come with one, but it allows root-finding on the stack which is the major problem that needs to be solved with the C back end, so it is much simpler. VM - some sort of VM back end, .NET or JVM. Simply because it would be a good thing to have and useful to people, I would use the VM's GC for this back end. The C-- back end is probably the easiest, as I designed GRIN with it in mind. However, the lack of C-- compilers means it is not as big of a win as we might hope. Now, a VM back end could be a killer feature for a lot of people, but it is something I would need help with, someone who knows the VM to spearhead the effort. And of course, I really need to bite the bullet and write a GC for the C back end, as that would fix the major issue with jhc right now. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On Wed, Jun 24, 2009 at 10:10:53AM -0400, Isaac Dupree wrote:

Evangelism (the honest sort), so that you have some users.

I am not so good at Evangelism, it makes me feel a little dirty :)

In your release announcement I would like to hear - that there are finally enough bugs fixed that JHC is usable. - that it's as easy to use as GHC for normal coding [can you make a Haskell-Platform-with-JHC to install?] getting a lot closer, the fact that containers,applicative,filepath and others compile without a hitch implies to me that it is getting quite usable as far as outstanding bugs go.

- Which extensions are implemented.

It supports quite a few extensions, but they don't always exactly line up with GHC. some are CPP, UnboxedTuples, EmptyDataDecls, TypeSynonymInstances, Arbitrary Rank N Types, RULES pragmas some more interesting ones in various stages of completion are: fully impredicative type system, [forall a . a] is a valid type for instance. first class existentials (exists a . a) is a valid type as well. unboxed values. working with unboxed values is almost as straightforward and elegant as working with boxed values. "strict" boxed values. values of kind !. they are boxed, but guarenteed evaluated. They are fully polymorphic unlike unboxed values. user defined kinds. top level IO actions Now, an issue is that even when jhc implements the same extension as ghc, it may not be exactly the same. as in, jhc's type system is different than ghcs. they both as rank-n types, but since there is no formal description of the ghc type system as it isn't in a language standard, I am sure there are programs ghc accepts and jhc doesn't and vice versa. In practice, these cases are probably rare but until haskell' is formalized they will be an issue.

If you get on the haskell-iphone list, you'll have more concentrated interest of people because the existing/past GHC overhead [and poorness at cross-compiling] matters more there. But make sure all of Haskell knows what's going on!

P.S. for my purposes, the Haskell Platform (including cabal-install) would be most important: its release made it 10x easier for me to get a working release of GHC! Then the only barrier might be if I needed some extension that Jhc didn't provide.

Unfortunately, the way cabal is designed makes it very difficult to use for compilers that don't share a codebase with ghc. The main issue is that when cabal files refer to something like 'base', they don't mean something compatible with base, they mean the precise 'base' that is on hackage. Furthurmore, cabal files refer to many ghc specifics, like 'split-base'. This is clearly an issue for something like jhc, which has its own base (that will likely be split up between several libraries, and perhaps not even called 'base'). We can have every package on cabal add explicit support for jhc, but those dependency lists are already getting complicated just to support a couple different versions of ghc. imagine 15 years down the road, with a half dozen new ghc versions, 3 uhc versions, 5 lhc ones and a dozen jhc releases that all need to be taken into account in every cabal file. Not scalable at all. Also, I couldn't bring myself to ask everyone to add explicit jhc support, it is just putting a band-aid on the problem and I don't like to do band-aids. All in all, cabal has a closed-world assumption which isn't true. It assumes all haskell code is on hackage and we can enumerate all compilers and language extensions in cabal itself. This makes haskell development more insular, rather than standardizing new versions of the language, people are dissuaded from using alternate versions by the design of the tools. Now, there are a couple paths... - wait for Haskell', let cabal packages depend on 'haskell'' rather than a specific implementation of it. Hope Haskell' is expansive enough for this to work out. - Lie to cabal, hack cabal's jhc support to pretend jhc has packages it doesn't and hope for the best. - Merge base with ghc base. No good at all. jhc has wildly different primitive concepts than ghc, files would be giant #if statements and I'd somehow have to coordinate releases and code repositories with ghc libraries. Also, it does not solve the issue, it just kludges around it. The inherent non-scalability is still there. - Use a better packaging system like 'franchise' that can understand cabal files and handle their dependencies cleanly. not great, but probably doable. implement a franchise-install. Now, all isn't that dire. jhc actually can do something much nicer than cabal-install due to the way its libraries work. jhc libraries are identified by a unique hash generated from their external interface, this means you don't actually have to worry about all the versioning issues that cabal handles or proscribe a specific version number format, if the hashes match, the libraries are compatible, if they don't, the libraries arn't. Multiple libraries implementing the same thing can coexist peacefully. A jhc-pkg install need only find the hash that matches a given name, download said file, and recursively look up each of its dependencies and download them. No compilation is necessary as the only thing needed to do to install an hl file is drop it in a directory. Think of something like arch-linuxes 'pacman' installer, very simple and it works. The only issue here is bootstrapping the hl files in the first place, the good news is that since hl files are portable, We are no longer constrained by cabal's least common denominator build system. You can use whatever the best tools for the job are and as long as you can produce an hl file, you can distribute it for all architectures. So, a combination of franchise and jhc-pkg could work together quite well. So, the short answer is, cabal is tricky. But jhc can do something like 'cabal install' but much better and that is mainly what people want out of it. Making transparent use of cabal packages on hackage is an open problem but 'franchise' gives us the start of a solution.

also I wonder, what is JHC's stance on monomorphism? (e.g. MonomorphismRestriction, MonoPatBinds, etc.)

No particular stance. It lets you turn on or off the monomorphism restriction with -fno-monomorphism-restriction. It doesn't seem to make too much different in the compiled code as the jhc optimizer is good at specialization, so I guess that is a data point in favor of dropping the monomorphism restriction. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On Thu, Jun 25, 2009 at 01:00:51AM -0400, Isaac Dupree wrote:

...

...

In your release announcement I would like to hear - that there are finally enough bugs fixed that JHC is usable. - that it's as easy to use as GHC for normal coding [can you make a Haskell-Platform-with-JHC to install?] getting a lot closer, the fact that containers,applicative,filepath and others compile without a hitch implies to me that it is getting quite usable as far as outstanding bugs go.

and it will naturally be incremental improvement like that. I mean, this is objective progress after all, it's worth listing in release notes (and online docs or wherever)

Yeah, I usually post a bit more. This most recent release was motivated by the recent iPhone interest so that is what I concentrated on in the release note. Honestly, I think I am paranoid that someone will try jhc and find it not to their liking and thus disapointed so I tend to undersell it. Which is pretty counter-productive since said person is likely to file a bug report and thus spur development to fix jhc's flaws (or may even decide to become a developer themselves). I know it's silly, but it's a hard habit to break.

...

...

- Which extensions are implemented.

It supports quite a few extensions, but they don't always exactly line up with GHC.

not responding to this in particular, but, in general; I think a lot of the Haskell community (including me I guess) will be coming from somewhat of a GHC perspective. And most of the available libraries will too (e.g. extensions required by Hackage packages are sometimes excessive). Do treasure your differences, don't just throw them away wantonly, but don't underestimate the value of working together! You've made a wonderful tool that is a compiler, but just think all the horrors of Unicode and threading and Windows and who knows what bugs GHC and its base libraries have been dealing with in the past years! (I mean I've tried things like maintaining unicode code, but I'm only one person and I have a lot of interests! It's usually just better in the end to increase code sharing, therefore bug-reports and patches and all those good things.). so... I think where possible it'd be good to try and share a code-base for libraries, and put effort into improving that. (okay I admit it's frustrating too, since GHC momentum will make it harder to start getting involved hacking. But things are IMHO less bad than the ghc-6.4 days when I entered haskell) Anyway also for the extensions that do line up JHC-GHC, at least you can try and give them the same names/understanding, which it looks like you're doing.

Yeah, I try to be GHC compatible when it makes sense, but I also don't have problems with breaking compatibility when there is a good reason. Sometimes I am able to improve on what ghc does, other times things just work out differently due to the natural evolution of code. But in general, if I can be compatible without compromising design, I will be. Though, you bring up another interesting issue when it comes to developing the libraries. JHC and GHC actually have a very different philosophy here in that in jhc, I rely on the underlying operating systems services whenever possible. For instance, I don't try to roll my own unicode or buffered file support. I rely on 'iconv' and 'FILE' and friends when they are available. In general, the thinking is, A whole lot of people have worked very hard already at optimizing FILE, the OS may even do things like use zero-copy mmaped buffers to back it. So, I use existing resources whenever possible. Incidentally, this makes porting to new platforms quite easy, I have a very shallow and high level library layer to switch out, rather than finding a way to implement a whole buffered IO system. For instance, the buffered IO system relys on pointers to alloced memory and being able to read and write into it. something that clearly won't work with a VM style back end that doesn't allow raw memory access. I need only replace putChar with the VM provided equivalent. It also creates small and fast executables. which is always good :)

...

some are CPP, UnboxedTuples, EmptyDataDecls, TypeSynonymInstances, Arbitrary Rank N Types, RULES pragmas

some more interesting ones in various stages of completion are:

fully impredicative type system, [forall a . a] is a valid type for instance.

first class existentials (exists a . a) is a valid type as well.

interesting. GHC has had such a hard time finding a *good* scheme for impredicative type inference, and there have been lots of papers written (so I hear) on the subject. Is JHC's current scheme described somewhere? (including limitations. Which mostly means "which type signatures you need to specify", IIRC).

It is pretty much exactly the algorithm described in this paper http://research.microsoft.com/en-us/um/people/simonpj/papers/boxy/ I should note that sometimes the front end type system bites off more than it can chew, in that it will correctly type something that eventually the back end realizes it can't handle. I need to formalize exactly what is allowed. In any case, pretty much all code in the wild seems to use a common subset of what GHC and JHC provide when it comes to advanced type system stuff. I should note that this problem bites both ways, for instance the core has full support for GADTs, in fact they are vital for my type class implementation. however the front end has no way to express them since I have no parse rules for them. That could be a good project for someone actually. But MPTCs are clearly the most pressing issue when it comes to the type system. Incidentally the 'boxy types' inference algorithm is also what allows unboxed polymorphism. unboxed types can be 'pushed down' in the same way higher rank types can, in the end, unknown types are defaulted to Bits32_ (think Int# from ghc) if they wern't otherwise constrained.

...

unboxed values. working with unboxed values is almost as straightforward and elegant as working with boxed values.

a possible pitfall of making them more usable, is it may become less obvious where they're evaluated (since the point of unboxed values, was to make sure not to rely on the optimizer, I think?)

My main motivation here was things like writing operating systems in haskell, where you need to mix strict and lazy code in interesting ways. implementing wait-free algorithms can depend critically on the exact order of memory writes for instance. Also, it helps the "haskell is a better C than C" idea if writing unboxed code isn't akward. I don't intend for working with unboxed values to become a mainstream haskell thing, mainly to make things you would normally have to drop into C for less painful. There is also another motivation, it tests my cores ability to handle mixed-mode code well, which opens up interesting possibilities like a ML from end for jhc that lets you seamlessly mix strict and lazy code. If unboxed strict haskell meshes seamlessly with lazy haskell then an ML front end is just a parsing issue rather than a major redesign.

...

"strict" boxed values. values of kind !. they are boxed, but guarenteed evaluated. They are fully polymorphic unlike unboxed values.

yay! I want this in ghc :-P

Jhc has a very rich internal type system, it is described in this section of the manual http://repetae.net/computer/jhc/manual.html#jhc-core-type-system Unlike some other attempts at mixing strict and lazy evaluation, it doesn't tread on new type theoretic principles, it is just a clever instantiation of a PTS (pure type system) so existing known proven properties of PTSs hold. A nice boon to know I am on steady ground theoretically. I think this and the way I transform monadic code into loops without giving up it functional character at any point are probably the main academic achievments of jhc. There is a lot of other neat stuff, but a lot is just cherry-picked from the current state of the art and extended in obvious ways.

...

Now, an issue is that even when jhc implements the same extension as ghc, it may not be exactly the same. as in, jhc's type system is different than ghcs. they both as rank-n types, but since there is no formal description of the ghc type system as it isn't in a language standard, I am sure there are programs ghc accepts and jhc doesn't and vice versa. In practice, these cases are probably rare but until haskell' is formalized they will be an issue.

yeah, that'll happen slowly now and then. Documenting individual extensions is a good thing, even when not part of finishing haskell'. I suspect haskell' won't exist until a non-GHC compiler looks likely to ever support it though!

...

We can have every package on cabal add explicit support for jhc, but those dependency lists are already getting complicated just to support a couple different versions of ghc. imagine 15 years down the road, with a half dozen new ghc versions, 3 uhc versions, 5 lhc ones and a dozen jhc releases that all need to be taken into account in every cabal file. Not scalable at all. Also, I couldn't bring myself to ask everyone to add explicit jhc support, it is just putting a band-aid on the problem and I don't like to do band-aids.

GHC/cabal folks seem to be moving towards a more portable model. e.g., separate "ghc-base" from "base" (which for GHC would depend on "ghc-base") so that Jhc could export an equivalent base (and maybe there the code divergence would be small enough that "base" itself could just use ifdefs, since most code would be in "jhc-base" or portable packages).

A problem is that my 'base' still wouldn't be the same. For instance, Data.Typeable is part of base, but jhc's Data.Typeable will never look like GHC's Data.Typeable due to some major design changes. Types can be examied by case statements in jhc core, meaning typerep manipulation is subject to the same case-optimizations as normal data constructors. This has nice properties, but it also means that handwritten Typeable instances arn't possible. deriving Typeable is the only way to get them. Now, this is just one example, but it means that no matter what even if jhc merged bases with ghc's, 'base-4' would still mean different things to the two of them, jhc will have to lie.

More importantly IMHO, they *want* to be interoperable and if you try to cooperate with them, they'll be more than happy to have a non-GHC compiler with which to fix their bad assumptions!

I am not sure this is universally true. Oh, it is generally true, but private correspondence has taught me there are some.. controlling.. personalities out there that want everything haskell to be subject to a certain world-view. But yes. you are right, in that a non-ghc compiler will hopefully shake things up some. And perhaps my impressions above were just based on having a bad day on a few peoples part (mine included). Certainly, cabals problems arn't as obvious when you only have one main compiler, but it gets harder and harder to get in the door as it were. perhaps jhc-pkg will at least illustrate a different way to do things, even if it doesn't directly compete with cabal..

...

All in all, cabal has a closed-world assumption which isn't true. It assumes all haskell code is on hackage

to some extent yes, the "hackage" centralisation seems like a bad thing, but it's also a great boon! (Any opinions on perl's CPAN and the like?)

Yeah, hackage has been useful. But I think that has to do with the haskell community rather than design decisions made by hackage. CPAN can be nice (and also frustrating at times) but there is a _major_ difference between CPAN and haskell. Perl is an implementation-defined single sourced language. Interoperability isn't something the CPAN design had to deal with. I think a lot of hackages issues arise from attempting to take the CPAN model and applying it to haskell without thinking about what makes haskell different. Things that work for single-source languages don't work for standards defined ones, and it is not surprising that attempting to squeeze haskell into that mold has made alternate implementions of haskell more difficult. I think following this path, either haskell will cease to exist, and we will just have 'glasgow haskell' defined as whatever the newest ghc implements or the system will break down and stagnate.

...

and we can enumerate all compilers and language extensions in cabal itself. This makes haskell development more insular, rather than standardizing new versions of the language, people are dissuaded from using alternate versions by the design of the tools.

The overhead isn't that great, and it makes you document each language extension at least a bit, so that other compilers might at least have a *chance* of implementing the same thing...

I was refering more to the fact that cabal hardcodes compilers and extensions into its source, rather than have some extensible format. It would be super nice if all I needed to do to get jhc to work with cabal was drop a file somewhere describing how to call jhc, or have jhc conform to some standard command line convention rather than become a cabal developer and modify its source...

...

So, the short answer is, cabal is tricky. But jhc can do something like 'cabal install' but much better and that is mainly what people want out of it.

that's true (but also think how many convenient features cabal is getting). I suspect the path through cabal to be the reasonable long term answer, as you'll then get smart people (e.g. libraries@haskell.org !) asking tough questions about what their tools need to do!

I am not so sure. There seems to be a fundamental difference in what they think a build system should do. One recent thing that illustrates it very well is the push to put 'maximum version numbers' on dependencies. As in, cabal is not just prone to bitrot, it is actively encouraged to make sure your packages become obsolete. Now, what I want out of a build system is for it to compile my code the best way it can. If I get called away for two years on a contract to do C# work (it's happened before) and want to start using haskell again because I hear of some new compiler that sounds interesting, what is vitally important is that I can just go into my old directories and tell it to build and have it just work if at all possible. I can take my sunsite CDs printed 12 years ago and ./configure those tarballs and they will in all likelyhood just work, on an OS that didn't even exist when they were made with a compiler and new C standard they didn't even know about. I consider this the _fundamental_ thing a build system should do. Try to build my code if at all possible, for all time, if at all possible. Now cabals dependency model is not only akward (having to specify dependencies without knowing what you are compiling on), but actively inhibits the one thing build systems should do, adapt to unknown environments. It is so concerned about not attempting to compile on something for which there is a slight chance the compile will fail, that it pretty much blocks out any ability for it to adapt to unknown systems. I mean, if a compile was going to fail, then it is going to fail anyway, so really the only thing the strictness does is cut out platforms where it would have worked. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On Fri, Jun 26, 2009 at 01:01:37PM -0400, Isaac Dupree wrote:

Interesting, interesting, interesting! Your points make more sense to me now -- i'll comment on a couple

John Meacham wrote:

...

Though, you bring up another interesting issue when it comes to developing the libraries. JHC and GHC actually have a very different philosophy here in that in jhc, I rely on the underlying operating systems services whenever possible. For instance, I don't try to roll my own unicode or buffered file support. I rely on 'iconv' and 'FILE' and friends when they are available. In general, the thinking is, A whole lot of people have worked very hard already at optimizing FILE, the OS may even do things like use zero-copy mmaped buffers to back it. So, I use existing resources whenever possible.

ah, re-use outside the haskell world! A worthy idea, encourages the existence of good C libraries etc. I guess the two ways I can think of weaknesses there is, - often C libraries don't think of features that would be obvious in Haskell, like duplicating/backtracking with state, which make a Haskell version have to be unnecessarily imperative. (But the C libraries should then be improved where possible! Sometimes they have an awful lot of momentum though, like iconv) - the Haskell compiler can't optimize as much, when lots of parts of the code are in C :-)

Yes indeed. of course, if the OS C library implements something like IO-Lite[1] under the scenes, then trying to reimplement plain old buffers assuming it is as good as what the OS does is clearly counterproductive. I guess my philosophy is more about reducing the barrier to code reuse between languages and to not second guess what the OS provides without good reason. Also, it's the "lazy" thing to do. :) [1] http://www.usenix.org/events/osdi99/full_papers/pai/pai.pdf There was a computer science professor at caltech who taught all the standard stuff. regular expressions, finite automata, computability theory, algorithms, etc. But each week after he gave the homework out he would say something as an afterthought like... "oh... and this week... do the homework in algol-68" or some other random language. A different one each week. The idea being that for a good computer scientist, learning a language shouldn't even register as a challenge, and more deeply, that the concepts being taught truely were intrinsic to computing itself and not any particular manifestation of it. I forget where I was going with this, other than to say that I think it is a good lesson. When thinking of cache coherency issues my mind natuarlly thinks in C, whereas I may go through a few iterations of a datalog program in my head for performing an optimization analysis before actually writing it in haskell. So reusing code from outside haskell (when it makes sense), and lowering the barrier of entry to doing so is something that is very natural to me.

still, it's an idea. Like when someone found out that replacing GHC's complicated (-threaded) thread system with basically just 1-thread = 1-OS-thread, on Linux it was just as efficient as the original [Windows and maybe OSX were slower though].

This isn't too surprising, I was in the OS kernel group at sun microsystems when a coworker took a look at the pathologically generalized many-to-many standard threaded library and decided to do a little experiment and wrote a simple wrapper that mapped exactly one pthread to one OS thread. turned out, it was pretty much superior in all practical cases in addition to being much smaller and with a scheduling model that didn't give you headaches. It ended up being shipped with Solaris 8 I believe. This is interesting though, I always figured I would implement concurrency with jhc via a simple wrapper around pthreads with little to no other magic. It is good to know that ghc has already treaded that ground and found it stable, though it doesn't surprise me. The many-to-many thread model was flawed from day-one IMHO (no matter what the incarnation).

I wonder if someday someone'll take on a project of minimizing GHC RTS code (GHC devs like the idea in the abstract but are in no great hurry to make it happen)... garbage-collector seemed to be the biggest feature that has to stay in the RTS (unless I wonder if JHC could detect automatically whether a program would benefit from having a garbage collector?)

That is the idea actually. I don't plan on turning off my static analysises ever, even if a garbage collector is enabled, it may well be that some programs end up not needing it. None of my GC ideas are that heavyweight, however the one I am playing with now requires libJudy at run-time, which probably won't do in general but is a useful start.

...

Now cabals dependency model is not only akward (having to specify dependencies without knowing what you are compiling on), but actively inhibits the one thing build systems should do, adapt to unknown environments. It is so concerned about not attempting to compile on something for which there is a slight chance the compile will fail, that it pretty much blocks out any ability for it to adapt to unknown systems. I mean, if a compile was going to fail, then it is going to fail anyway, so really the only thing the strictness does is cut out platforms where it would have worked.

Well, actually the strictness also allows/helps Cabal to choose the package-versions where the compile *will* succeed. But.. API breakages are usually truly in the interface, rather than silently changing the semantics. JHC could probably automatically search all the APIs for compatibility (though I'm not sure what kind of indexing you'd need to make that efficient for thousands+ of known package-versions).

My idea for jhc-pkg was to do something similar to yum. have whatever builds the jhc-pkg site create a sqlite database that can be downloaded and contains an index. certainly a mapping from module names to packages won't be that strenuous, even for many thousands of entries. yum handles on the order of 10,000 entries just fine for instance. But the idea isn't to completely get rid of dependencies, but rather make them smarter and adaptive and context independent. for instance, jhc's build depends are basically "a bunch of packages always + syb if it exists" This is quite simple and will probably work for quite a few versions of ghc and other compilers. furthermore, it can be interpreted unambiguously. cabal flags like 'split-base' or 'foo > 4.0' don't have meaning on their own. You can't know what 'split-base' means without knowing the history of ghc's libraries. whereas 'syb if it exists' is a truely intrinsic property that can be determined locally and for any compiler. Now, it may not be the _right_ rule for every compiler, but not every compiler is going to support backwards compatability with some odd version of ghc for all time either to compile the specific old (perhaps buggy?) versions of packages needed to satisfy some strict build-dependencies. You can add new intrinsic rules as needed and they have a nicely synergistic effect. two rules like 'syb if it exists, containers if it exists' catch the cases where none, either, or both exist. whereas 'syb if ghc = 6.10' only fixes a single case for a single iteration of a single compiler. So, you end up with sub-linear growth of build rules in one case, but linear or worse for the current cabal way and guarenteed bitrot to boot.

If you get to being able to compile even a moderate fraction of Hackage, then you have a lot of versions and APIs you can play with to see how your approach might numerically compare to Cabal's :-) (e.g. success rate and asymptotic speed)

Yeah. this goes into writing a new packaging system fully though. Which is something I think haskell needs, but I am not sure I have time for. I am hoping 'franchise' evolves into something. Perhaps it can be used as a base for cabal2. Cabal is sort of happered by its roots as a library instead of a program, the desire to keep API compatibility means they can't really refactor or clean up the code, even though very few programs use most of the API in more than trivial ways. A full reboot without the API baggage and a reasonable dependency model is needed. Perhaps jhc-pkg will inspire something new.. perhaps not. I do believe the current cabal will have to evolve into something like I describe, I just don't see how it can scale otherwise. Not to mention I went through this whole process before with 'xmkmf' finally being dropped in favor of 'autoconf' for X11 development, it had a lot of interesting parallels. But if such a system evolves we will end up with legacy systems built on top of each other with an accidentally cobbled together package description language (accidental turing complete languages end up with some odd properties) and an overly complicated codebase. Whereas if we designed it the right way from the start we would have had something much cleaner and well thought out. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On Mon, Jun 29, 2009 at 05:16:02PM -0400, Isaac Dupree wrote:

on the subject of deducing dependencies. What about code that can function with or without the functionality of a given library? Is there any trouble with using CPP for that (say, #ifdef PARSE_YAML. Or more annoyingly, #ifdef SOME_FEATURE_THAT_ONLY_SOME_COMPILER_VERSIONS_CAN_SUPPORT)

It depends actually, this is something that the cabal scheme doesn't work for, since you need to specify everything your package _might_ depend on, even though the dependencies may be irrelevant for a given system or not even compile. Ideally it would do something like franchise, build-depends are things you depend on for sure, but the initial configuration can decide that it needs some more dependencies. franchise-install (or whatever) is simply recursively invoked to install any dependencies it didn't get in the first pass. Incidentally, this is how pretty much every other CxAN style system works. I think the #ifdef SOME_FEATURE_etc... problem is more an issue with haskell' being so long in coming so such things are needed more often than they should be. Though, this touches on an interesting point that may become an issue for jhc. Since jhc libraries are machine independent, this means that your preprocessor doesn't have knowledge of the target architecture. so things like #if WORD_SIZE == 64 won't work. the proper way to do it is 'if sizeOf (undefined :: Ptr ()) == 64 then ... else ...' or "if isBigEndian then ... else ..." rather than relying on the preprocessor. I imagine it will be something that comes up when porting code to jhc, but I don't think it will be a big deal, just something to remember.

but well. I guess this is getting a little off topic.

The practicalities of writing haskell code that is portable between compilers is quite relevant. I certainly welcome discussion on the topic. :) John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

That reminds me. I had previously thought a hybrid approach for garbage collection would be beneficial. One could use regional GC for all objects that can be safely assigned to regions, for all others, use an incremental GC. Perhaps some additional assumptions could be made about the "global" region objects that would allow for further optimizations. On Tue, Jun 23, 2009 at 10:13 PM, Taral wrote:

On Tue, Jun 23, 2009 at 5:29 PM, Corey O'Connor wrote:

...

2. Improved garbage collector? This might just be a documentation issue. The documentation states the region-based garbage collector leaks memory. And the configure script hints at being able to use a standard Boehm garbage collector library. Does the region based garbage collector still have issues? When should one be used over the other?

+1

This is the main reason I don't use jhc code in production.

-- Taral "Please let me know if there's any further trouble I can give you." -- Unknown _______________________________________________ jhc mailing list jhc@haskell.org http://www.haskell.org/mailman/listinfo/jhc

-- "The greatest obstacle to discovering the shape of the earth, the continents, and the oceans was not ignorance but the illusion of knowledge." - Daniel J. Boorstin