Re: aeson and dlist in HP 2013.4.0.0

28 Nov 2013

      On 28 November 2013 13:42, Sven Panne  wrote:
...
Just two add my 2c: Given all these new packages which would need to
be pulled into the HP just for aeson, let's not include aeson for
2013.4.0.0 and release 2013.4.0.0 soon without the need for lengthy
discussions.
As the proposer for inclusion of aeson in the HP I'm beginning to agree.

There's another reason I would like to postpone the aeson inclusion: I
just started working on improving the encoding performance of aeson.
This requires some significant changes to the API. Therefore I think
it would be better to see how well this new API works out. If it works
out, release it as aeson-7 (or aeson-8) and include that release in
the HP after next. This way we have time to discuss the new
dependencies and the HP remains stable.

The following is a brief explanation of the new aeson API (you can
stop reading here if you're not interested in it):

The idea is to use the same trick that is used in the upcoming binary
package[1].

First of all toJSON will return a JsonBuilder instead of a Value:

class ToJSON a where
  toJSON :: a -> JsonBuilder

A JsonBuilder is basically a difference list:

newtype JsonBuilder = JsonBuilder (IStream -> IStream)

instance Monoid JsonBuilder where ...

The "list", here represented as an IStream, is a sequence of
instructions to the encoder:

data IStream =
    INull                              IStream

  | ITrue                              IStream
  | IFalse                             IStream

  | IDoubleQuote                       IStream

  | IChar   {-# UNPACK #-} !Char       IStream
  | IString                !String     IStream
  | IText                  !Text       IStream

  | IInt    {-# UNPACK #-} !Int        IStream
  | IInt8   {-# UNPACK #-} !Int8       IStream
  | IInt16  {-# UNPACK #-} !Int16      IStream
  | IInt32  {-# UNPACK #-} !Int32      IStream
  | IInt64  {-# UNPACK #-} !Int64      IStream

  | IWord   {-# UNPACK #-} !Word       IStream
  | IWord8  {-# UNPACK #-} !Word8      IStream
  | IWord16 {-# UNPACK #-} !Word16     IStream
  | IWord32 {-# UNPACK #-} !Word32     IStream
  | IWord64 {-# UNPACK #-} !Word64     IStream

  | IFloat  {-# UNPACK #-} !Float      IStream
  | IDouble {-# UNPACK #-} !Double     IStream

  | IInteger               !Integer    IStream

  | IScientific            !Scientific IStream

  | IComma                             IStream

  | IBeginArray                        IStream
  | IEndArray                          IStream

  | IBeginObject                       IStream
  | IEndObject                         IStream

  | IColon                             IStream

  | IValue                 !Value      IStream

  -- Fused:
  | IBeginObject_IDoubleQuote          IStream
  | IComma_IDoubleQuote                IStream
  -- TODO; more

  | IEnd

Converting a JsonBuilder to a Builder (note that I'm using the new
bytestring Builder here) is simply a matter of executing the right
Builder for each instruction:

toBuilder :: JsonBuilder -> Builder
toBuilder (JsonBuilder g) = go (g IEnd)
 where
  go :: IStream -> Builder
  go is = case is of
    INull             is' -> nullB               <> go is'

    ITrue             is' -> trueB               <> go is'
    IFalse            is' -> falseB              <> go is'

    IDoubleQuote      is' -> char8          '"'  <> go is'

    IChar        c    is' -> char           c    <> go is'
    IString      cs   is' -> string         cs   <> go is'
    IText        t    is' -> text           t    <> go is'

    IInt         i    is' -> intDec         i    <> go is'
    IInt8        i8   is' -> int8Dec        i8   <> go is'
    IInt16       i16  is' -> int16Dec       i16  <> go is'
    IInt32       i32  is' -> int32Dec       i32  <> go is'
    IInt64       i64  is' -> int64Dec       i64  <> go is'

    IWord        w    is' -> wordDec        w    <> go is'
    IWord8       w8   is' -> word8Dec       w8   <> go is'
    IWord16      w16  is' -> word16Dec      w16  <> go is'
    IWord32      w32  is' -> word32Dec      w32  <> go is'
    IWord64      w64  is' -> word64Dec      w64  <> go is'

    IFloat       f    is' -> floatDec       f    <> go is'
    IDouble      d    is' -> doubleDec      d    <> go is'

    IInteger     i    is' -> integerDec     i    <> go is'
    IScientific  s    is' -> fromScientific s    <> go is'

    IComma            is' -> char8          ','  <> go is'

    IBeginArray       is' -> char8          '['  <> go is'
    IEndArray         is' -> char8          ']'  <> go is'

    IBeginObject      is' -> char8          '{'  <> go is'
    IEndObject        is' -> char8          '}'  <> go is'

    IColon            is' -> char8          ':'  <> go is'

    IValue       v    is' -> fromValue      v    <> go is'

    -- Fused:
    IBeginObject_IDoubleQuote is'-> fixed2('{','"')<> go is'
    IComma_IDoubleQuote       is'-> fixed2(',','"')<> go is'
    -- TODO: more

    IEnd -> mempty

nullB :: Builder
nullB = fixed4 ('n',('u',('l','l')))
{-# INLINE nullB #-}

trueB :: Builder
trueB = fixed4 ('t',('r',('u','e')))
{-# INLINE trueB #-}

falseB :: Builder
falseB = fixed5 ('f',('a',('l',('s','e'))))
{-# INLINE falseB #-}

fixed2 :: (Char, Char) -> Builder
fixed2 = P.primFixed (P.char8 >*< P.char8)
{-# INLINE fixed2 #-}

fixed4 :: (Char, (Char, (Char, Char))) -> Builder
fixed4 = P.primFixed (P.char8 >*< P.char8 >*< P.char8 >*< P.char8)
{-# INLINE fixed4 #-}

fixed5 :: (Char, (Char, (Char, (Char, Char)))) -> Builder
fixed5 = P.primFixed (P.char8 >*< P.char8 >*< P.char8 >*< P.char8 >*< P.char8)
{-# INLINE fixed5 #-}

This representation allows a lot of optimizations. For example we can
define rewrite rules that "fuse" the Builders of common sequences
like:

{-# RULES
  "IBeginObject_IDoubleQuote" forall is.
   IBeginObject (IDoubleQuote is) =
   IBeginObject_IDoubleQuote is #-}

{-# RULES
  "IComma_IDoubleQuote" forall is.
   IComma (IDoubleQuote is) =
   IComma_IDoubleQuote  is #-}

The encoder can handle these common sequences more efficiently.

Of course the JsonBuilder is abstract to the user. There will be a
safe API to construct well-formed JsonBuilders. (While writing this I
realize that users will be able to use the Monoid instance for
JsonBuilders which is undesirable. I will solve this by wrapping the
JsonBuilder returned from toJSON in another newtype which doesn't have
a Monoid instance)

What do we loose? In the current API of aeson, toJSON will directly
return a Value. This Value can then be inspected or extended. In order
to do the same in the new API the JsonBuilder first has to be parsed
to a Value which is less efficient. However, if the new API proves to
be significantly more efficient for encoding I think this extra
parsing cost is warranted since it's far less common than encoding.

A first version of this API will soon be ready and I will push that to
my github. Hopefully I can come up with some convincing benchmarks!

Bas

Re: aeson and dlist in HP 2013.4.0.0

Bas van Dijk