Thank you all very much for your tips! @Ben
You might want to check out DDC [1]. It’s still in a pre-alpha state, but what it does have is a working external core language. You could write your own front end that produces System-F, then have DDC compile that. The individual compiler passes are also fairly well separated, so it’s easy to change.
DDC looks like quite interesting, but I don't think it is what I am looking for. For one thing the first parsing step is something I would prefer to "steal" wholesale, and I would also like to avoid messing with the type system where I'm not abusing it. On the other hand the claim that its purpose is to "investigate program transformation" sounds promising and I love how the code is littered with comments. So I will definitely look at it more closely, even if it's only to learn some tricks. @Sergey
If you still want to work with hackablel compiler look at jhc https://github.com/jimcrayne/jhc and it's fork ajhc https://github.com/ajhc/ajhc
JHC immediately got my +1 because they have a rough diagram of the core transformations (http://repetae.net/computer/jhc/big-picture.pdf). It's far from perfect, but as a visual learner I love such stuff and would like other complex programs to offer something similar. It also looks like there might be some shared code between JHC and UHC? Speaking of the latter, that one looks well documented, too. So both might be a good place to start. @Tom
GHC compiler plugins may also be what you're looking for.
Funnily enough GHC plugins are one of the reasons I'm shying away from GHC at the moment. Not that it's a bad idea, especially for optimizations. But focusing so much on one core structure (no pun intended) leads to a lot of dependencies on that structure and makes it hard to change it. At the same time I don't think such a linear pipeline approach is necessarily the best one. Say we want to implement a prolog-like sublanguage, makros, or even externalized typesystems at some point, as other languages are doing. (with makros serving as a core tool to keep all these kitchen sinks optional) The easiest approach would be to create separate branching paths in the compiler. Which in turn might be implemented by a DAG of independent transformer modules with separate pre- and postconditions, flexible intermediate types and semi-automatic dependency resolution. In other words tight integration with a powerful build system. That doesn't sound like something I will be able to offer, but I think it would be a good idea to at least experiment with approaches that might lead in that direction instead of relying on one core highway. Which lead me to the conclusion... @Sergey
You should go for a long run with GHC, it has most advanced extension system and you'll be able to integrate your extensions with existing codebase if they come out successful.
...that while that is certainly "the one true way to go" later, it's not the place I want to start at. Sorry, GHC. You will still be my build tool of choice, though. So for now my takeaway is that I will start by looking at UHC and JHC as possible places to start. DDC might be able to contribute parts or ideas. And if everything is too complicated I might just rip out parts from different projects and throw together my own minimalistic frankencompiler. So again, thanks for your contributions. It will probably take a long time for any significant results, but I'll keep you posted. Cheers, MarLinn