On Markdown in Haddock and why it's not going to happen

Mateusz Kowalczyk

29 Aug 2013 29 Aug '13

9:30 p.m.

Greetings café, Perhaps some saddening news for Markdown fans out there. As you might remember, there was a fair amount of push for having Markdown as an alternate syntax for Haddock. Unfortunately, this is probably not going to happen for reasons listed on the post I just published at [1]. This thread is meant to be for discussion about the post as many people, myself included, felt that Markdown would be a nice thing to have. I feel like I covered the topic fairly well on the post but feel free to give suggestions or ask questions. I would also like to remind you that if there's something that you'd like to see in Haddock or something that you feel is broken, a good way express this is to make a ticket on the Haddock Trac[2]. I will close the relevant Markdown ticket on the Trac[3] in about 3 days, unless someone can come up with a reasonable solution that meets the initial intent of this part of the project: a widely known markup format that could be used as an alternate syntax for Haddock so that it's possible to write the documentation without learning the vanilla syntax itself. [1]: http://fuuzetsu.co.uk/blog/posts/2013-08-30-why-Markdown-in-Haddock-can't-happen.html [2]: http://trac.haskell.org/haddock [3]: http://trac.haskell.org/haddock/ticket/244 -- Mateusz K.

Show replies by date

Omari Norman

31 Aug 31 Aug

10:37 a.m.

New subject: On Markdown in Haddock and why it's not going to happen

On Thu, Aug 29, 2013 at 9:30 PM, Mateusz Kowalczyk wrote:

...

Greetings café,

Perhaps some saddening news for Markdown fans out there. As you might remember, there was a fair amount of push for having Markdown as an alternate syntax for Haddock.

This is a little off-topic, but the Haddock website apparently is years out of date. http://www.haskell.org/haddock/ says the latest version is 2.8.0 released in September 2010, but apparently I have 2.13.2 on my machine.

Carter Schonwald

11:20 a.m.

New subject: On Markdown in Haddock and why it's not going to happen

Is there an up todate copy of the haddock manual online anywhere? On Saturday, August 31, 2013, Omari Norman wrote:

...

On Thu, Aug 29, 2013 at 9:30 PM, Mateusz Kowalczyk < fuuzetsu@fuuzetsu.co.uk > wrote:

...
Greetings café,

Perhaps some saddening news for Markdown fans out there. As you might remember, there was a fair amount of push for having Markdown as an alternate syntax for Haddock.

This is a little off-topic, but the Haddock website apparently is years out of date.

http://www.haskell.org/haddock/

says the latest version is 2.8.0 released in September 2010, but apparently I have 2.13.2 on my machine.

Mateusz Kowalczyk

12:30 p.m.

New subject: On Markdown in Haddock and why it's not going to happen

On 31/08/13 16:20, Carter Schonwald wrote:

...

Is there an up todate copy of the haddock manual online anywhere?

...

On Saturday, August 31, 2013, Omari Norman wrote:

...
On Thu, Aug 29, 2013 at 9:30 PM, Mateusz Kowalczyk wrote:

...
Greetings café,

Perhaps some saddening news for Markdown fans out there. As you might remember, there was a fair amount of push for having Markdown as an alternate syntax for Haddock.

This is a little off-topic, but the Haddock website apparently is years out of date.

http://www.haskell.org/haddock/

says the latest version is 2.8.0 released in September 2010, but apparently I have 2.13.2 on my machine.

The latest stable is indeed 2.13.2. Hopefully this will be updated fairly soon, AFAIK there was some miscommunication on who got the write

No. You can build your own documentation. In Haddock directory, go into ‘doc’ and read the README on how to build it. That is also outdated however: for example, it doesn't provide information about the image syntax (<<image>>). New documentation will be written soon as the stuff I have been doing added and changed quite a bit. This will most likely come out before any actual release is made because I'd like to include a small guideline on migrating your docs to the new version. Nothing big should break and in huge majority of cases you shouldn't need to do anything but there are a few quirks that are gone that might change how your things look under some cases (mostly in cases of ill-formed docs). permissions or something but I might just be misremembering. -- Mateusz K.

Niklas Hambüchen

2:14 p.m.

New subject: On Markdown in Haddock and why it's not going to happen

Hello, I disagree. While none of your detail points are wrong, they mainly focus on the fact that there is no 1-to-1 mapping between the existing haddock markup and Markdown. I don't think there needs to be. If Markdown can do something new, that something can be added; if something doesn't make sense in Haddock (like horizontal rules), we ignore them. I don't think original and markdown syntax should be mixed in the same file. That would make everything impossible to parse and difficult to write. As for which markdown implementation to use: I think it really doesn't matter that much in practice. Github and pandoc can both render documentation to my pleasing; I have yet to find a difference between them that would be a practical problem for my documentation efforts. Regarding module and type links: They are links, so probably just representing them as Markdown links would be cleanest. [SomeThing]() with an empty reference could make haddock automatically figure out what is wanted or default to a type, and you could be explicit with [SomeThing](module), [SomeThing](type) and [SomeThing](class). For headings, why is CPP a problem? CPP ignores haddock comments, haddock should ignore CPP. There is no reason to put CPP macros into comments. Regarding emphasis, **foo** would simply not be a heading in an export list. Using markdown haddock, we will have markdown headings for that. Markdown being claimed to be "for editing documents for the Web" doesn't make our efforts impossible. Pandoc can easily create latex output from it, and Github can use it as a documentation language for programming perfectly fine. So can we. Did I address all your points? Niklas On Fri 30 Aug 2013 10:30:51 JST, Mateusz Kowalczyk wrote:

...

Greetings café,

Perhaps some saddening news for Markdown fans out there. As you might remember, there was a fair amount of push for having Markdown as an alternate syntax for Haddock.

Unfortunately, this is probably not going to happen for reasons listed on the post I just published at [1].

This thread is meant to be for discussion about the post as many people, myself included, felt that Markdown would be a nice thing to have.

I feel like I covered the topic fairly well on the post but feel free to give suggestions or ask questions.

I would also like to remind you that if there's something that you'd like to see in Haddock or something that you feel is broken, a good way express this is to make a ticket on the Haddock Trac[2].

I will close the relevant Markdown ticket on the Trac[3] in about 3 days, unless someone can come up with a reasonable solution that meets the initial intent of this part of the project: a widely known markup format that could be used as an alternate syntax for Haddock so that it's possible to write the documentation without learning the vanilla syntax itself.

[1]: http://fuuzetsu.co.uk/blog/posts/2013-08-30-why-Markdown-in-Haddock-can't-happen.html

[2]: http://trac.haskell.org/haddock

[3]: http://trac.haskell.org/haddock/ticket/244

Mateusz Kowalczyk

3:27 p.m.

New subject: On Markdown in Haddock and why it's not going to happen

...

Hello,

I disagree. That's fine, that's why the thread is here.

While none of your detail points are wrong, they mainly focus on the fact that there is no 1-to-1 mapping between the existing haddock markup and Markdown. I don't think there needs to be. It doesn't have to be 1-to-1 but the features have to be expressible in both: it's useless if we have different features with one syntax but not

...

If Markdown can do something new, that something can be added; if something doesn't make sense in Haddock (like horizontal rules), we ignore them. This is precisely what I have been doing and why my post went over every

On 31/08/13 19:14, Niklas Hambüchen wrote: the other. point from the original Markdown documentation, talking about why each of these does or doesn't make sense. In the end, Markdown offers us nothing new feature wise and either forces us to make up new syntax for things it doesn't know about (definition lists) or makes us remove existing features all together. Once any not-HTML-specific (tags, rules) features that might be missing are implemented, we're left with something that looks very much like Markdown already. Alternate syntax would also look very much like Markdown but in a different, less known way. Not very nice. Basically, Markdown brings nothing new to the table that we can actually use and we're in fact making it less familiar by having to strip and add syntax.

...

I don't think original and markdown syntax should be mixed in the same file. That would make everything impossible to parse and difficult to write.

Agreed. See point above why this isn't practical anyway. We'd be piling features on top of Markdown, defeating the point of having familiar-to-others syntax.

...

As for which markdown implementation to use: I think it really doesn't matter that much in practice. Github and pandoc can both render documentation to my pleasing; I have yet to find a difference between them that would be a practical problem for my documentation efforts.

Unfortunately it _does_ matter in practice because each flavour solves Markdown quirks differently. On top of this, we have to accomodate for Haddock specific features. We're effectively creating our own flavour of Markdown + extra syntax. This sounds like… Haddock markup!

...

Regarding module and type links: They are links, so probably just representing them as Markdown links would be cleanest. [SomeThing]() with an empty reference could make haddock automatically figure out what is wanted or default to a type, and you could be explicit with [SomeThing](module), [SomeThing](type) and [SomeThing](class).

For headings, why is CPP a problem? CPP ignores haddock comments, haddock should ignore CPP. There is no reason to put CPP macros into comments.

I have to admit that I did not explore this very carefully but you we can't simply guarantee that no one every will run CPP on their files by hand. Better safe than sorry. I actually implemented headings for function docs using the ‘==== Heading 4’ syntax, avoiding this problem all together.

...

Regarding emphasis, **foo** would simply not be a heading in an export list. Using markdown haddock, we will have markdown headings for that.

But what if I want it to be a heading? Are we picking rules arbitrarily now?

...

Markdown being claimed to be "for editing documents for the Web" doesn't make our efforts impossible. Pandoc can easily create latex output from it, and Github can use it as a documentation language for programming perfectly fine. So can we.

That's because you can use LaTeX to render everything that HTML can (but not the other way around!). GitHub can use it because they don't have the burden of having to accommodate for an existing feature set of a different markup language. Note how GitHub Markdown doesn't have all the features of Haddock, therefore it is _not_ compatible with it. We can't just use their Markdown because it doesn't cover our use cases.

...

Did I address all your points?

You addressed some of them, but I hope I reasoned for them well above. What you haven't provided is how such Markdown could possibly look, noting the original goal of this: be an alternate syntax that's familiar to people already. If you can come up with such thing without losing any features and being distinct enough from Haddock markup to warrant it, I would love to hear it.

...

Niklas

-- Mateusz K.

Niklas Hambüchen

1 Sep 1 Sep

8:59 a.m.

New subject: On Markdown in Haddock and why it's not going to happen

On 01/09/13 04:27, Mateusz Kowalczyk wrote:

...

It doesn't have to be 1-to-1 but the features have to be expressible in both: it's useless if we have different features with one syntax but not the other.

I don't find that useless. Markdown does not have definition lists, but we use a normal list to achieve the same documentation goals. Already being able to simply write basic markdown would bring me big practical benefits, and I would prefer minor theoretical misalignments not to outweigh them.

...

...
If Markdown can do something new, that something can be added; if something doesn't make sense in Haddock (like horizontal rules), we ignore them. This is precisely what I have been doing and why my post went over every point from the original Markdown documentation, talking about why each of these does or doesn't make sense. In the end, Markdown offers us nothing new feature wise

If the features are the same, I would prefer to write them in markdown.

...

and either forces us to make up new syntax for things it doesn't know about (definition lists) or makes us remove existing features all together.

As I said, I think it doesn't matter. The basic functionality that people use is all there. I have not seen a haddock definition list in my life, the first one is encountered is the single one in attoparsec because I grepped for [@ after reading your post.

...

...
As for which markdown implementation to use: I think it really doesn't matter that much in practice. Github and pandoc can both render documentation to my pleasing; I have yet to find a difference between them that would be a practical problem for my documentation efforts. Unfortunately it _does_ matter in practice because each flavour solves Markdown quirks differently. On top of this, we have to accomodate for Haddock specific features. We're effectively creating our own flavour of Markdown + extra syntax. This sounds like… Haddock markup!

Once we have picked one of the flavours, I do not care about the others. These quirks have little practical relevance: I have put the markdown from my main github projects into the tool you mentioned, and into pandoc, and there were no bad surprises for me. Given that, even if I were to step into a pathologic case, after I write the documentation, a quick look at it immediately tells me whether everything is to my liking or not.

...

...
Regarding module and type links: They are links, so probably just representing them as Markdown links would be cleanest. [SomeThing]() with an empty reference could make haddock automatically figure out what is wanted or default to a type, and you could be explicit with [SomeThing](module), [SomeThing](type) and [SomeThing](class).

For headings, why is CPP a problem? CPP ignores haddock comments, haddock should ignore CPP. There is no reason to put CPP macros into comments. I have to admit that I did not explore this very carefully but you we can't simply guarantee that no one every will run CPP on their files by hand. Better safe than sorry.

This is unreasonable. You also can't guarantee that nobody will run the Handlebars templating language either, or any other tool. You cannot even tell what the user wants to achieve: Should -- | You can use `#if SHELL a #else b #endif` go into the documentation unexpanded or template processed? The only sane way is to make haddock agnostic of all those tools in the world, save the Haskell compiler which it addresses by being inside comments. If somebody wanted to run their custom tool by hand before running haddock by hand, sure they would write #if ... -- | Some haddock #else -- | Some other haddock # endif if they wished to conditionally generate different haddock.

...

...
Regarding emphasis, **foo** would simply not be a heading in an export list. Using markdown haddock, we will have markdown headings for that. But what if I want it to be a heading? Are we picking rules arbitrarily now?

I do not understand what you mean. If you want it to be a heading, you write: # My heading I don't find that worse than * My heading

...

...
Markdown being claimed to be "for editing documents for the Web" doesn't make our efforts impossible. Pandoc can easily create latex output from it, and Github can use it as a documentation language for programming perfectly fine. So can we. That's because you can use LaTeX to render everything that HTML can (but not the other way around!).

Yes, so what is the problem? You write markdown and it can be compiled to your target languages. Why is it a problem that you cannot go the other way around? My point here was saying that markdown can be compiled to the three targets you mentioned.

...

GitHub can use it because they don't have the burden of having to accommodate for an existing feature set of a different markup language. Note how GitHub Markdown doesn't have all the features of Haddock, therefore it is _not_ compatible with it. We can't just use their Markdown because it doesn't cover our use cases.

We don't have that burden either, save the easily replaceable definition list. Coming back to those, their markdown equivalent could be a list with links, or simply a ```definition fenced code block.

...

...
Did I address all your points? You addressed some of them, but I hope I reasoned for them well above. What you haven't provided is how such Markdown could possibly look, noting the original goal of this: be an alternate syntax that's familiar to people already.

It could look exactly like Github or pandoc markdown (fine with either, they are equal enough for me), with the difference that * they are enclosed in Haskell comments and start with `-- |` * they resolve type/module links via standard markdown links (this does not change the parsing)

Mateusz Kowalczyk

5:23 p.m.

New subject: On Markdown in Haddock and why it's not going to happen

On 01/09/13 13:59, Niklas Hambüchen wrote:

...

On 01/09/13 04:27, Mateusz Kowalczyk wrote:

...
It doesn't have to be 1-to-1 but the features have to be expressible in both: it's useless if we have different features with one syntax but not the other.

I don't find that useless. Markdown does not have definition lists, but we use a normal list to achieve the same documentation goals.

Already being able to simply write basic markdown would bring me big practical benefits, and I would prefer minor theoretical misalignments not to outweigh them.

I haven't considered it this way before. I'll try to implement it today as an experiment and hopefully show-off the result so we can see whether it's worth using. -- Mateusz K.

John MacFarlane

2 Sep 2 Sep

2:43 p.m.

New subject: On Markdown in Haddock and why it's not going to happen

When the proposal was first being discussed, I suggested that instead of adding markdown support to haddock, one might enhance the existing haddock markup, making it more expressive, so that it could encode the same range of structural features as markdown. If I'm not mistaken, currently haddock doesn't allow list items with multiple paragraphs or other block elements, or nested lists, or images, or blockquotes. This makes it impossible to create a pandoc writer for haddock markup (we already have a haddock reader, soon to be released). With a pandoc writer for haddock, one could write longer documentation sections (e.g. tutorials) in markdown and convert them to haddock for inclusion in source code. This should help people who don't like haddock markup or don't want to learn it. With the reader, haddock comments can already be converted into other formats. I haven't been following your work, but do I understand correctly that you've been making haddock markup more expressive and rational? Has this gotten to the point where a pandoc writer would be feasible? John +++ Mateusz Kowalczyk [Aug 30 13 02:30 ]:

...

Greetings café,

Perhaps some saddening news for Markdown fans out there. As you might remember, there was a fair amount of push for having Markdown as an alternate syntax for Haddock.

Unfortunately, this is probably not going to happen for reasons listed on the post I just published at [1].

This thread is meant to be for discussion about the post as many people, myself included, felt that Markdown would be a nice thing to have.

I feel like I covered the topic fairly well on the post but feel free to give suggestions or ask questions.

I would also like to remind you that if there's something that you'd like to see in Haddock or something that you feel is broken, a good way express this is to make a ticket on the Haddock Trac[2].

I will close the relevant Markdown ticket on the Trac[3] in about 3 days, unless someone can come up with a reasonable solution that meets the initial intent of this part of the project: a widely known markup format that could be used as an alternate syntax for Haddock so that it's possible to write the documentation without learning the vanilla syntax itself.

[1]: http://fuuzetsu.co.uk/blog/posts/2013-08-30-why-Markdown-in-Haddock-can't-happen.html

[2]: http://trac.haskell.org/haddock

[3]: http://trac.haskell.org/haddock/ticket/244 -- Mateusz K.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Mateusz Kowalczyk

4:34 p.m.

New subject: On Markdown in Haddock and why it's not going to happen

...

When the proposal was first being discussed, I suggested that instead of adding markdown support to haddock, one might enhance the existing haddock markup, making it more expressive, so that it could encode the same range of structural features as markdown. If I'm not mistaken, currently haddock doesn't allow list items with multiple paragraphs or other block elements, or nested lists, or images, or blockquotes. Paragraph level elements are indeed not allowed in lists. The list contents are currently (that is, on my working branch) allowed to be: monospace (delimited by @), anchors, identifiers, module names,

...

This makes it impossible to create a pandoc writer for haddock markup (we already have a haddock reader, soon to be released). Yes, indeed you might remember that my original proposal was to take care of this: improving Haddock to the point where a writer would be

On 02/09/13 19:43, John MacFarlane wrote: pictures, URLs, bold, emphasis, HTML escape codes, regular strings. Note that if a monospaced string is the only element of the list content, it gets turned into a code block. I'm not sure what you mean by block elements. Do you mean bird tracks and code blocks? There's a ticket about nested lists at http://trac.haskell.org/haddock/ticket/27 . While I say that nested paragraphs are not planned, I'm still playing with the thought. The issue is having to balance adding new features with not breaking a significant amount of existing documentation: we do not want to force everyone to go through all their libraries and change their markup. While I say on the ticket that it's hard to implement, I think it could be done with indentation. I would have to experiment. If you would like to see paragraph nesting, please make a ticket describing the behaviour! If list nesting is all you would be after then the ticket already exists. If you're looking for allowing more elements in lists, please make a ticket. possible and providing the writer. I even mentioned somewhere that my proposal and the proposal I'm currently working on would eventually end up with the same result, just in a different order of events.

...

With a pandoc writer for haddock, one could write longer documentation sections (e.g. tutorials) in markdown and convert them to haddock for inclusion in source code. This should help people who don't like haddock markup or don't want to learn it. With the reader, haddock comments can already be converted into other formats.

I haven't been following your work, but do I understand correctly that you've been making haddock markup more expressive and rational? Has this gotten to the point where a pandoc writer would be feasible?

I will enumerate some changes. Probably the most important part is the new parser which makes new markup additions and changes far easier and testable than working with two separate grammars fed into Alex and Happy. I also added a fair amount of tests and set up LaTeX backend testing (I still need to populate it with tests though). I hope to do the same with Hoogle. * Long missing bold was added, with double underscore as delimiter. "__foo bar baz__". You don't have to escape single underscores as you would have to in the past: "__fo_o__" is fine. Even "__foo bar\__baz__" is fine. * Nested markup is now possible and allowed where sensible. "__/foo/__" * Escaping of delimiters mid-markup is now possible. "/fo\/o/" will produce emphasised "fo\o". The old behaviour would only emphasise "fo\". URLs, anchors and images follow the old rules (terminate on first delimiter, even if escaped. I'll probably allow escaping in URL and image syntax though. * The parser should no longer fail on any strings. This means that a line like "This is / a line." will parse as a simple string. In the past this would fail because there's no closing slash and you'd be forced to escape it. If you happen to see the old ‘parsing failed’ message with new release, please file as a major bug! * Image ‘title’ text is now possible. "<<imagePath titleText>>" * You no longer need a line break between lists of the same type. "* foo * bar" is fine. "* foo 2. bar" is not fine if two lists are what you're after. * Headers are now allowed in function documentation: "= @Monospaced Header 1@ == __Bold Header 2__ === /Emphasised Header 3/ […] ====== Header 6" Header levers 3-6 all get rendered at the same size in LaTeX. Also put it into the Hoogle back-end but of course, it's out of our hands to decide how that will be rendered. I e-mailed Neil Mitchel and he said that this should be no problem however. Note that you do NOT need a line break between a header and a paragraph-level markup. That is things like "= Header

...

birdtracks can only be on their own paragraphs" are completely fine. I'm still thinking about this behaviour but I don't see any real problems with it.

Currently implementing per-module enabled extension listing, enabled for rendering with a Haddock pragma but I need to solve a testing problem before I push it up. On the side I started to implement a Markdown implementation that Niklas seems very eager to have, even if that means a lot less features available when using it. Personally, I don't think it will be nice to use at all seeing that some Markdown AND Hadock features will be missing and some of the syntax has to be semantically altered so it's more of a proof of concept. Hopefully a Pandoc writer would solve this problem for us: considering how far along we are in the GSoC timeline, I'd much rather keep adding functionality and fix bugs than to add implementation and maintenance burden for a rather incomplete Markdown implementation. Please, do tell us what you need doing for the writer. Best way to do this is to file a ticket on Haddock Trac http://trac.haskell.org/haddock/ . If the ticket exists already, let me know (here or on Trac) and I'll see whether I could prioritise it a bit more.

...

John

+++ Mateusz Kowalczyk [Aug 30 13 02:30 ]:

...
Greetings café,

Perhaps some saddening news for Markdown fans out there. As you might remember, there was a fair amount of push for having Markdown as an alternate syntax for Haddock.

Unfortunately, this is probably not going to happen for reasons listed on the post I just published at [1].

This thread is meant to be for discussion about the post as many people, myself included, felt that Markdown would be a nice thing to have.

I feel like I covered the topic fairly well on the post but feel free to give suggestions or ask questions.

I would also like to remind you that if there's something that you'd like to see in Haddock or something that you feel is broken, a good way express this is to make a ticket on the Haddock Trac[2].

I will close the relevant Markdown ticket on the Trac[3] in about 3 days, unless someone can come up with a reasonable solution that meets the initial intent of this part of the project: a widely known markup format that could be used as an alternate syntax for Haddock so that it's possible to write the documentation without learning the vanilla syntax itself.

[1]: http://fuuzetsu.co.uk/blog/posts/2013-08-30-why-Markdown-in-Haddock-can't-happen.html

[2]: http://trac.haskell.org/haddock

[3]: http://trac.haskell.org/haddock/ticket/244 -- Mateusz K.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Mateusz K.

Niklas Hambüchen

3 Sep 3 Sep

11:05 a.m.

New subject: On Markdown in Haddock and why it's not going to happen

On 30/08/13 10:30, Mateusz Kowalczyk wrote:

...

I would also like to remind you that if there's something that you'd like to see in Haddock or something that you feel is broken, a good way express this is to make a ticket on the Haddock Trac[2].

I made one: http://trac.haskell.org/haddock/ticket/257 This is very useful for us because haddock is broken by some TH issue. This way, we can have most of our docs anyway.

4279

Age (days ago)

4283

Last active (days ago)

List overview

Download

10 comments

5 participants

participants (5)

Carter Schonwald
John MacFarlane
Mateusz Kowalczyk
Niklas Hambüchen
Omari Norman