Hi, Does some one have made performance tests on the different XML libraries for Haskell? I have a 20MB xml file that I want to read. I remember from my earlier experiments (years ago) that all libraries were too slow and were consuming too much memory. I hoped that this situation had changed but maybe not. I looked at HaXML, libxml, HXML and HXT. HaXML eats a lot of memory and is still very slow. libxml is unfinished binding to the C library. Currently it only allows to create documents. HXML seems to be very promising. It works fast and it doesn't eat memory. Unfortunately it is that it seems to be rather old. It uses its own Arrow and Tree libraries instead of the standard libraries. I have not jumped into HXT yet because it seems to be very large library. Could someone recomend which one is the state of the art? Best Regards, Krasimir
I remember that HaXML has also a lazy XML parser. maybe if you just need to use some specific information stored in your XML file you can earn some time/memory with it.From my experience, HXT seems faster. Cheers, hugo 2008/10/23 Krasimir Angelov <kr.angelov@gmail.com>
Hi,
Does some one have made performance tests on the different XML libraries for Haskell? I have a 20MB xml file that I want to read. I remember from my earlier experiments (years ago) that all libraries were too slow and were consuming too much memory. I hoped that this situation had changed but maybe not. I looked at HaXML, libxml, HXML and HXT. HaXML eats a lot of memory and is still very slow. libxml is unfinished binding to the C library. Currently it only allows to create documents. HXML seems to be very promising. It works fast and it doesn't eat memory. Unfortunately it is that it seems to be rather old. It uses its own Arrow and Tree libraries instead of the standard libraries. I have not jumped into HXT yet because it seems to be very large library. Could someone recomend which one is the state of the art?
Best Regards, Krasimir
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
-- www.di.uminho.pt/~hpacheco
I've used HaXml's SAX parser to parse huge XML files. It is surprising that HXT doesn't seem to have a SAX parser as I understand that it is the successor to HaXml. DOM style parsing won't work with huge files. http://hackage.haskell.org/packages/archive/HaXml/1.19/doc/html/Text- XML-HaXml-SAX.html On Oct 23, 2008, at 11:38 PM, Hugo Pacheco wrote:
I remember that HaXML has also a lazy XML parser. maybe if you just need to use some specific information stored in your XML file you can earn some time/memory with it. From my experience, HXT seems faster.
_____________ David F. Place d@vidplace.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David F. Place wrote:
I've used HaXml's SAX parser to parse huge XML files. It is surprising that HXT doesn't seem to have a SAX parser as I understand that it is the successor to HaXml. DOM style parsing won't work with huge files.
It can with a lazy processor, can't it? - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJAR+e7M8hyUobTrERAqBkAJ4hplduxSf0527c9XZRO63UXiS1KwCePlKa vJkCMBTDZeVmbwfzFaV4oIo= =TBNZ -----END PGP SIGNATURE-----
Hello Krasimir, There is also the xml package from Galois: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/xml Hope that helps. __ Donnie 2008/10/23 Krasimir Angelov <kr.angelov@gmail.com>
Hi,
Does some one have made performance tests on the different XML libraries for Haskell? I have a 20MB xml file that I want to read. I remember from my earlier experiments (years ago) that all libraries were too slow and were consuming too much memory. I hoped that this situation had changed but maybe not. I looked at HaXML, libxml, HXML and HXT. HaXML eats a lot of memory and is still very slow. libxml is unfinished binding to the C library. Currently it only allows to create documents. HXML seems to be very promising. It works fast and it doesn't eat memory. Unfortunately it is that it seems to be rather old. It uses its own Arrow and Tree libraries instead of the standard libraries. I have not jumped into HXT yet because it seems to be very large library. Could someone recomend which one is the state of the art?
Best Regards, Krasimir
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
2008/10/24 Donnie Jones <donnie@darthik.com>:
Hello Krasimir,
There is also the xml package from Galois: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/xml
That looks nice. Are there any examples of its usage? --vk
"Krasimir Angelov" <kr.angelov@gmail.com> writes:
Does some one have made performance tests on the different XML libraries for Haskell? I have a 20MB xml file that I want to read. I remember from my earlier experiments (years ago) that all libraries were too slow and were consuming too much memory.
For my XML needs, I ended up just using the TagSoup library to extract the parts I wanted. At least this allows lazy processing of large files (a typical file is in the 1 to 4 GB range). Performance is acceptable, and Neil is rumored to have a plan to improve it further. -k -- If I haven't seen further, it is by standing in the footprints of giants
Thanks to everyone who answered. HXML still seems to be the best for me. It is fast and it has good Arrow interface. It is also a small and simple library. I tried also HXT with this example: http://www.haskell.org/haskellwiki/HXT#Getting_started:_Hello_world_examples but it just died with out of memory. HXML just works. I have not tried the Light XML library because it doesn't provide Arrow interface. I updated HXML to use more stuff from the standard libraries. Now I can use the syntax sugar for arrows with it. Is the library maintained? Should I send a patch to someone? Best Regards, Krasimir On Thu, Oct 23, 2008 at 10:26 PM, Krasimir Angelov <kr.angelov@gmail.com>wrote:
Hi,
Does some one have made performance tests on the different XML libraries for Haskell? I have a 20MB xml file that I want to read. I remember from my earlier experiments (years ago) that all libraries were too slow and were consuming too much memory. I hoped that this situation had changed but maybe not. I looked at HaXML, libxml, HXML and HXT. HaXML eats a lot of memory and is still very slow. libxml is unfinished binding to the C library. Currently it only allows to create documents. HXML seems to be very promising. It works fast and it doesn't eat memory. Unfortunately it is that it seems to be rather old. It uses its own Arrow and Tree libraries instead of the standard libraries. I have not jumped into HXT yet because it seems to be very large library. Could someone recomend which one is the state of the art?
Best Regards, Krasimir
Hello Krasimir,
Thanks to everyone who answered. HXML still seems to be the best for me. It is fast and it has good Arrow interface. It is also a small and simple library. I tried also HXT with this example:
http://www.haskell.org/haskellwiki/HXT#Getting_started:_Hello_world_example s
but it just died with out of memory. HXML just works. I have not tried the Light XML library because it doesn't provide Arrow interface.
What you can try with HXT is to use the readDocument arrow with the option to use the tagsoup parser for parsing. Then HXT really does lazy input. And there is a memory optimization for sharing the strings representing element and attribute names. But there is no DTD processing and no validating functionality with tagsoup. Most efficient input encoding is ISO Latin1 (or ASCII). Cheers Uwe -- Uwe Schmidt Web: http://www.fh-wedel.de/~si/
I tried to use HXT's readDocument with its tagsoup option for my application. I couldn't find a way to construct the operation that didn't run out of memory. I'll attach some code using HaXml's saxParse so you can see what I want. Is that easy to do in HXT? I simply want the text of <PMID> and <AbstractText> elements. Please feel free to bounce this to Haskell Cafe if you feel it is more appropriate. On Oct 24, 2008, at 12:44 PM, Uwe Schmidt wrote:
What you can try with HXT is to use the readDocument arrow with the option to use the tagsoup parser for parsing. Then HXT really does lazy input.
_____________ David F. Place d@vidplace.com
there was a thread about xml parsing, one month ago. <http://thread.gmane.org/gmane.comp.lang.haskell.cafe/44708> well, i don't know much about xml, except what it looks like; but i know about that interesting parsing problem behind it. maybe Lev Walkin has fixed that in HXML. at least he wrote this patch... <http://article.gmane.org/gmane.comp.lang.haskell.cafe/45033> - marc ----- sometimes i think, i should write a paper about it. but then... naah, i'm like haskell: non-strict. Am Donnerstag, 23. Oktober 2008 schrieb Krasimir Angelov:
Hi,
Does some one have made performance tests on the different XML libraries for Haskell? I have a 20MB xml file that I want to read. I remember from my earlier experiments (years ago) that all libraries were too slow and were consuming too much memory. I hoped that this situation had changed but maybe not. I looked at HaXML, libxml, HXML and HXT. HaXML eats a lot of memory and is still very slow. libxml is unfinished binding to the C library. Currently it only allows to create documents. HXML seems to be very promising. It works fast and it doesn't eat memory. Unfortunately it is that it seems to be rather old. It uses its own Arrow and Tree libraries instead of the standard libraries. I have not jumped into HXT yet because it seems to be very large library. Could someone recomend which one is the state of the art?
Best Regards, Krasimir
Hi Marc, Thanks for the pointer. Fortunately I don't have this problem. Probably 2GB of memory are enough to parse 20MB file even with this space leak. HXML still works better than the other libraries and has a nice API so I use it. The patch is useful but it is not applied. I also did some other changes in HXML. It seems like the library is not maintained. Should I package it and upload to Hackage? Regards, Krasimir On Sat, Oct 25, 2008 at 2:14 AM, Marc A. Ziegert <coeus@gmx.de> wrote:
there was a thread about xml parsing, one month ago. <http://thread.gmane.org/gmane.comp.lang.haskell.cafe/44708> well, i don't know much about xml, except what it looks like; but i know about that interesting parsing problem behind it. maybe Lev Walkin has fixed that in HXML. at least he wrote this patch... <http://article.gmane.org/gmane.comp.lang.haskell.cafe/45033>
- marc
----- sometimes i think, i should write a paper about it. but then... naah, i'm like haskell: non-strict.
Am Donnerstag, 23. Oktober 2008 schrieb Krasimir Angelov:
Hi,
Does some one have made performance tests on the different XML libraries for Haskell? I have a 20MB xml file that I want to read. I remember from my earlier experiments (years ago) that all libraries were too slow and were consuming too much memory. I hoped that this situation had changed but maybe not. I looked at HaXML, libxml, HXML and HXT. HaXML eats a lot of memory and is still very slow. libxml is unfinished binding to the C library. Currently it only allows to create documents. HXML seems to be very promising. It works fast and it doesn't eat memory. Unfortunately it is that it seems to be rather old. It uses its own Arrow and Tree libraries instead of the standard libraries. I have not jumped into HXT yet because it seems to be very large library. Could someone recomend which one is the state of the art?
Best Regards, Krasimir
participants (9)
-
David F. Place -
Donnie Jones -
Hugo Pacheco -
Ketil Malde -
Krasimir Angelov -
Marc A. Ziegert -
Micah Cowan -
Uwe Schmidt -
Vesa Kaihlavirta