Markdown extension for Haddock as a GSoC project

23 Apr 2013

      -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

In light of fairly recent discussion, on this mailing list, I decided
to investigate the topic of Markdown support for Haddock. The archives
of the recent discussion can be seen at [1]. This post aims to
summarise the current state of discussion. I do aim to file a proposal
for a GSoC project on this issue but it'd be foolish to file a
proposal for a project aiming to benefit the community without
consulting the community itself.

Here are some main points and ideas gathered:
* reSt seems to have a small following - quite a bit smaller than the
Markdown community. In fact, there seems to be a significant amount of
people pushing for Markdown which contrasts what we can read in a
topic from 2008 at [2]. I guess it just shows how much Markdown has
gained in popularity in recent years.
* There are issues with using Markdown even before we attempt to use
it for Haskell documentation:
  * There exists no formal specification or semantics. It would seem
that a significant number of Markdown parsers are creating by reverse
engineering an already existing parser. This is bad because we end up
propagating the bugs and workarounds around ambiguity that the
original parser has.
  * As a follow-up to the previous point, the (vanilla) Markdown is
ambiguous and there is nothing to resolve it. As Richard A. O'Keefe
pointed out, there exist situations where it's not possible to infer
the semantics of Markdown from its official implementation and the
result is parser/writer-specific [6].
* John MacFarlane has already written a Markdown parser in Haskell. It
can be read at [3]. This means that the new extension would not need
to rely on Pandoc. He says ``I have an experimental thing here that
could be used as a basis (it's 7x faster than pandoc and uses 1/5 the
memory, BSD licensed)''. This is great! The post can be seen in full
at [4].
* An alternative idea was to simply write a writer module for Pandoc
for Haddock.
  * A reader module is already present.
  * According to John MacFarlane, Haddock is not expressive enough to
take advantage of this. Furthermore, Pandoc doesn't have some
constructions that Haddock does [4].
  * Comes back to the problem on relying on such a large package as
Pandoc.
* Yet another proposal was rather than introducing Markdown to
concentrate on fixing current issues and adding features to Haddock as
it stands [8]. This is one of the options listed at the short blog
post at [14] by Johan Tibell.
  * David Waern, one of Haddock maintainers admits that Haddock lacks
active development and it has been that way for a longer time. Having
said that, he seems to believe that Markdown integration is a project
that can realistically be completed over summer. Such project would be
able to use the existing HTML backend in Haddock. [9].
* Math expressions were requested and MathJAX was suggested as a
solution at [5]. math.stackexchange.com uses MathJAX and it works
quite well. Personally, I believe that Haskell documentation would
benefit from this simply due to the academic nature of the language.
* Support for Literate Haskell would be a welcome addition as
suggested by Andrew Butterfield at [7].
* There are issues with CPP and LHS in regards to using Markdown in
documentation. They are pointed out at [10] by John MacFarlane and
others that he's replying to.
* As pointed out 5 years ago at [11], we'd have to do some
preprocessing on current, fairly critical sections of comments used by
Haddock. I believe these are fairly useful and it would be a shame to
see them go.

I hope I haven't missed anything of high importance in a list above.
When researching the topic, issues with Markdown quickly become
apparent. Today, there are tens of different Markdown flavours: each
company has different needs and each company interprets the vague
original documentation in a way that's convenient to them. In these
situations, topics, e-mails and blog posts like this one happen. There
is a call to action from October 2012 at [12] but there seems to be
absolutely no progress on any of the miraculous things mentioned in
the post. As a result of that post, a W3C community was formed at
[13]. It's clear that the community is inactive and no progress
towards a solution was made.

Having considered all information gathered here, I believe this would
make a good GSoC project. There has been interest in this for Haskell
since 2008 judging by the size of the last thread, it's clear that the
community interest is high. In fact, it surprises me that this hasn't
been done in previous years.

The main issue that I see with using Markdown is the fact that we
can't simply add a {-# HADDOCK Markdown #-} and let some parser rip
through it. There are many issues that need to be resolved that I
outlined above. This will no doubt result in a Markdown flavour suited
to documenting Haskell rather than the wonderful, standardised
Markdown that [13] is talking about. Is this different in any way from
what GitHub or StackOverflow is doing? Introducing yet another
flavour? In essence, no. For all Haskellers however, there would be a
difference: this is a chance to use the pleasant-to-eye Markdown that
many seem to like and use it for documentation. We have the chance to
formalise it and make it work properly. It's not a perfect solution
for every single person writing documentation out there but it's a
good solution for Haskellers. Obviously, the ideal would be to have
the wonderful, standardised, formalised Markdown that [13] seems to
try and conjure and then extend that. Alas, we have no such luxury.

As a closing paragraph, I would like to ask some questions.
- - Is creating yet another Markdown flavour acceptable? I see no better
solution: we'll have to take Markdown and fix it up to our needs. I'm
sure most people are aware of the fairly famous Internet web-comic
relating to standards [15]. We're however not trying to push a new
standard - writing a proper standard, implementing it and then
extending it for Haskell documentation is far too much for a single
summer.
- - Is there a mentor that would be willing to oversee the project?
There's a fairly fresh ticket at [16] but there seems to be no
indication of mentor-ship.

I'm very interested in the project - it seems to hold high value for
the community while not being completely out of scope for a student
with a couple of months of Haskell experience, especially considering
the large ``Getting up to scratch'' period.

Thank you for your time.

Links
- -----
[1] - http://comments.gmane.org/gmane.comp.lang.haskell.cafe/104398
[2] -
http://www.haskell.org/pipermail/haskell-cafe/2008-February/039846.html
[3] - https://github.com/jgm/Markdown
[4] - http://permalink.gmane.org/gmane.comp.lang.haskell.cafe/104421
[5] - http://permalink.gmane.org/gmane.comp.lang.haskell.cafe/104443
[6] - http://permalink.gmane.org/gmane.comp.lang.haskell.cafe/104422
[7] - http://permalink.gmane.org/gmane.comp.lang.haskell.cafe/104428
[8] - http://permalink.gmane.org/gmane.comp.lang.haskell.cafe/104523
[9] - http://permalink.gmane.org/gmane.comp.lang.haskell.cafe/104545
[10] - http://permalink.gmane.org/gmane.comp.lang.haskell.cafe/104444
[11] -
http://www.haskell.org/pipermail/haskell-cafe/2008-February/039939.html
[12] -
http://www.codinghorror.com/blog/2012/10/the-future-of-markdown.html
[13] - http://www.w3.org/community/markdown/
[14] - http://blog.johantibell.com/2013/04/haskellorg-gsoc-ideas.html
[15] - http://xkcd.com/927/
[16] - http://trac.haskell.org/haddock/ticket/244
- -- 
Mateusz K.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iQIcBAEBAgAGBQJRdt+WAAoJEM1mucMq2pqXspkP/j+SCyfho65/ji5OvNCEBFTb
YLGC1YY/dk/zbb67gDDxmFtxw/UBGnUifliGUKhCy2Zzm6mwF7BI1U4NWhbR2nkQ
VWkysqHMn1JM8uQNtkdXEe/KUi2KxcM+VtOkuCUQeF0mlHB7Hthlz58lGZHD+cMK
EvhN0BGrL1/xnKwSexTSNexXIDCmvCW19jTxuR9uwPqVNVPMjWxO0wRVplnzcqyf
S8jeG0+Q28S0Su/CxNUdUQ+pE2+FWksNY4a/d/8Tce9sFoTTXUEGAp6Nbgyi+uKd
yS+AtBqvcsw7KYw03kQAB1qCD05dtc7ptp9ohjgwYY6jzgRMBrIRdDUvmqT6nPh1
MHgG9rkg9yj+gKhSlNiTZ7MCl/o/CkCtPG0rNtd6//QXDZlH3Xx6CpYKm/GnyAWh
a2vX5ZX0urVIapFPPEpClMkleLbHfi2UWGVbC43jP0lvsZTMlr4uvwHzJ8N4e9AI
2MdK4yZes6b/DSH47FPdoAeYksUvwI0xpI5rpkLuoOh+O094gQVhm4yb+ceqF6CE
VddqnHPPF4gn7NTeYh6fnsAvzCm5PYgyhEUySbkmlGHjAODtvs6uC9MW4zwYPeCp
BvGTSsGx7DbtjHwDYQIzMI/oyjFXMVsQ/+bE+pzIo/tQJevXkILaZKYhBvUyAbE5
ypdJkLdQb8EcWGT5Cx0v
=sTE0
-----END PGP SIGNATURE-----

Mateusz Kowalczyk

Alistair Bayley

Mateusz Kowalczyk

Bryan O'Sullivan

Ben

Bryan O'Sullivan

Ben

Mateusz Kowalczyk

Petr Pudlák

Andrew Butterfield

Mateusz Kowalczyk

Carter Schonwald

Mateusz Kowalczyk

tags

participants (7)