Incremental XML parsing with namespaces?
I'm trying to convert an XML document, incrementally, into a sequence of XML events. A simple example XML document: <doc xmlns="org:myproject:mainns" xmlns:x="org:myproject:otherns"> <title>Doc title</title> <x:ref>abc1234</x:ref> <html xmlns="http://www.w3.org/1999/xhtml"><body>Hello world!</body></html> </doc> The document can be very large, and arrives in chunks over a socket, so I need to be able to "feed" the text data into a parser and receive a list of XML events per chunk. Chunks can be separated in time by intervals of several minutes to an hour, so pausing processing for the arrival of the entire document is not an option. The type signatures would be something like: type Namespace = String type LocalName = String data Attribute = Attribute Namespace LocalName String data XMLEvent = EventElementBegin Namespace LocalName [Attribute] | EventElementEnd Namespace LocalName | EventContent String | EventError String parse :: Parser -> String -> (Parser, [XMLEvent]) I've looked at HaXml, HXT, and hexpat, and unless I'm missing something, none of them can achieve this: + HaXml and hexpat seem to disregard namespaces entirely -- that is, the root element is parsed to "doc" instead of ("org:myproject:mainns", "doc"), and the second child is "x:ref" instead of ("org:myproject:otherns", "ref"). Obviously, this makes parsing mixed-namespace documents effectively impossible. I found an email from 2004[1] that mentions a "filter" for namespace support in HaXml, but no further information and no working code. + HXT looks promising, because I see explicit mention in the documentation of recording and propagating namespaces. However, I can't figure out if there's an incremental mode. A page on the wiki[2] suggests that SAX is supported in the "html tag soup" parser, but I want incremental parsing of *valid* documents. If incremental parsing is supported by the standard "arrow" interface, I don't see any obvious way to pull events out into a list -- I'm a Haskell newbie, and still haven't quite figured out monads yet, let alone Arrows. Are there any libraries that support namespace-aware incremental parsing? [1] http://www.haskell.org/pipermail/haskell-cafe/2004-June/006252.html [2] http://www.haskell.org/haskellwiki/HXT/Conversion_of_Haskell_data_from/to_XM...
On 8 Jun 2009, at 19:39, John Millikin wrote:
+ HaXml and hexpat seem to disregard namespaces entirely -- that is, the root element is parsed to "doc" instead of ("org:myproject:mainns", "doc"), and the second child is "x:ref" instead of ("org:myproject:otherns", "ref").
Yes, HaXml makes no special effort to deal with namespaces. However, that does not mean that dealing with namespaces is "impossible" - it just requires a small amount of post-processing, that is all. For instance, it would not be difficult to start from the SAX-like parser http://hackage.haskell.org/packages/archive/HaXml/1.19.7/doc/html/Text-XML-H... taking e.g. a constructor value SaxElementOpen Name [Attribute] and converting it to your corresponding constructor value EventElementBegin Namespace LocalName [Attribute] Just filter the [Attribute] of the first type for the attribute name "xmlns", and pull that attribute value out to become your new Namespace value. Obviously there is a bit more to it than that, since namespace *defining* attributes, like your example xmlns:x="...", have an lexical scope. You will need some kind of state to track the scope, possibly in the parser itself, or again possibly in a post-processing step over the list of output XMLEvents. Regards, Malcolm
On Mon, Jun 8, 2009 at 1:44 PM, Malcolm Wallace<malcolm.wallace@cs.york.ac.uk> wrote:
Yes, HaXml makes no special effort to deal with namespaces. However, that does not mean that dealing with namespaces is "impossible" - it just requires a small amount of post-processing, that is all.
For instance, it would not be difficult to start from the SAX-like parser
http://hackage.haskell.org/packages/archive/HaXml/1.19.7/doc/html/Text-XML-H...
taking e.g. a constructor value SaxElementOpen Name [Attribute]
and converting it to your corresponding constructor value EventElementBegin Namespace LocalName [Attribute]
Just filter the [Attribute] of the first type for the attribute name "xmlns", and pull that attribute value out to become your new Namespace value.
Obviously there is a bit more to it than that, since namespace *defining* attributes, like your example xmlns:x="...", have an lexical scope. You will need some kind of state to track the scope, possibly in the parser itself, or again possibly in a post-processing step over the list of output XMLEvents.
The interface you linked to doesn't seem to have a way to "resume" parsing. That is, I can't feed it chunks of text and have it generate a (ParserState, [Event]) tuple for each chunk. Perhaps this is possible in Haskell without explicit state management? I've tried to write a test application to listen on a socket and print events as the arrive, but with no luck. Manually re-parsing the events isn't attractive, because it would require writing at least part of the parser manually. I had hoped to re-use an existing XML parser, rather than writing a new one.
John Millikin schrieb:
On Mon, Jun 8, 2009 at 1:44 PM, Malcolm
The interface you linked to doesn't seem to have a way to "resume" parsing. That is, I can't feed it chunks of text and have it generate a (ParserState, [Event]) tuple for each chunk. Perhaps this is possible in Haskell without explicit state management? I've tried to write a test application to listen on a socket and print events as the arrive, but with no luck.
Manually re-parsing the events isn't attractive, because it would require writing at least part of the parser manually. I had hoped to re-use an existing XML parser, rather than writing a new one.
I think you could use the parser as it is and do the name parsing later. Due to lazy evaluation both parsers would run in an interleaved way. But you may also be interested in: http://hackage.haskell.org/packages/archive/wraxml/0.4.2/doc/html/Text-XML-W... using http://hackage.haskell.org/packages/archive/xml-basic/0.1/doc/html/Text-XML-...
On Mon, Jun 8, 2009 at 3:39 PM, Henning Thielemann<lemming@henning-thielemann.de> wrote:
I think you could use the parser as it is and do the name parsing later. Due to lazy evaluation both parsers would run in an interleaved way.
I've been trying to figure out how to get this to work with lazy evaluation, but haven't made much headway. Tips? The only way I can think of to get incremental parsing working is to maintain explicit state, but I also can't figure out how to achieve this with the parsers I've tested (HaXml, HXT, hexpat). Here's a working example of what I'm trying to do, in Python. It reads XML from stdin, prints events as they are parsed, and will terminate when the document ends: ########################## from xml.sax import handler, saxutils, expatreader class ContentHandler (handler.ContentHandler): def __init__ (self): self.events = [] self.level = 0 def startElementNS (self, ns_name, lname, attrs): self.events.append (("BEGIN", ns_name, lname, dict (attrs))) self.level += 1 def endElementNS (self, ns_name, lname): self.events.append (("END", ns_name, lname)) self.level -= 1 def characters (self, content): self.events.append (("TEXT", content)) def main (): parser = expatreader.ExpatParser () content = ContentHandler () parser.setFeature (handler.feature_namespaces, True) parser.setContentHandler (content) got_events = False while content.level > 0 or (not got_events): text = raw_input ("Enter XML:\n") parser.feed (text) print content.events content.events = [] got_events = True if __name__ == "__main__": main() ############################### $ python incremental.py Enter XML: <test xmlns="urn:test"><test2><test3> [('BEGIN', (u'urn:test', u'test'), u'test', {}), ('BEGIN', (u'urn:test', u'test2'), u'test2', {}), ('BEGIN', (u'urn:test', u'test3'), u'test3', {})] Enter XML: </test3></test2><test2 a="b"/>text content goes here [('END', (u'urn:test', u'test3'), None), ('END', (u'urn:test', u'test2'), None), ('BEGIN', (u'urn:test', u'test2'), u'test2', {(None, u'a'): u'b'}), ('END', (u'urn:test', u'test2'), None), ('TEXT', u'text content goes here')] Enter XML: </test> [('END', (u'urn:test', u'test'), None)] ############################# As demonstrated, the parser retains state (namespaces, nesting) between text inputs. Are there any XML parsers for Haskell that support this incremental behavior?
John Millikin wrote:
On Mon, Jun 8, 2009 at 3:39 PM, Henning Thielemann<lemming@henning-thielemann.de> wrote:
I think you could use the parser as it is and do the name parsing later. Due to lazy evaluation both parsers would run in an interleaved way.
I've been trying to figure out how to get this to work with lazy evaluation, but haven't made much headway. Tips? The only way I can think of to get incremental parsing working is to maintain explicit state, but I also can't figure out how to achieve this with the parsers I've tested (HaXml, HXT, hexpat).
Can you please look at http://code.haskell.org/~thielema/tagchup/example/Escape.hs http://code.haskell.org/~thielema/tagchup/example/Strip.hs You just have to replace Text.XML.Basic.Name.LowerCase by Text.XML.Basic.Name.Qualified for use of qualified names. -- Mit freundlichen Gruessen Henning Thielemann Viele Gruesse Henning Martin-Luther-Universitaet Halle-Wittenberg, Institut fuer Informatik Tel. +49 - 345 - 55 24773 Fax +49 - 345 - 55 27333
On Mon, Jun 8, 2009 at 20:39, John Millikin<jmillikin@gmail.com> wrote:
I'm trying to convert an XML document, incrementally, into a sequence of XML events. A simple example XML document:
<doc xmlns="org:myproject:mainns" xmlns:x="org:myproject:otherns"> <title>Doc title</title> <x:ref>abc1234</x:ref> <html xmlns="http://www.w3.org/1999/xhtml"><body>Hello world!</body></html> </doc>
The document can be very large, and arrives in chunks over a socket, so I need to be able to "feed" the text data into a parser and receive a list of XML events per chunk. Chunks can be separated in time by intervals of several minutes to an hour, so pausing processing for the arrival of the entire document is not an option. The type signatures would be something like:
type Namespace = String type LocalName = String
data Attribute = Attribute Namespace LocalName String
data XMLEvent = EventElementBegin Namespace LocalName [Attribute] | EventElementEnd Namespace LocalName | EventContent String | EventError String
parse :: Parser -> String -> (Parser, [XMLEvent])
I've looked at HaXml, HXT, and hexpat, and unless I'm missing something, none of them can achieve this:
+ HaXml and hexpat seem to disregard namespaces entirely -- that is, the root element is parsed to "doc" instead of ("org:myproject:mainns", "doc"), and the second child is "x:ref" instead of ("org:myproject:otherns", "ref"). Obviously, this makes parsing mixed-namespace documents effectively impossible. I found an email from 2004[1] that mentions a "filter" for namespace support in HaXml, but no further information and no working code.
I would recommend hexpat to do the job. Contrary to what you are saying, hexpat does offer namespace handling: http://hackage.haskell.org/packages/archive/hexpat/0.8/doc/html/Text-XML-Exp... Perhaps you need more than that? Personally I found hexpat to be fast, space efficient and easy to use. Here is the representation I got for your example. Please note the namespaces in right places. * > (toNamespaced ( toQualified t')) Element {eName = NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "doc"}, eAttrs = [(NName {nnNamespace = Just "http://www.w3.org/2000/xmlns/", nnLocalPart = "x"},"org:myproject:otherns"),(NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "xmlns"},"org:myproject:mainns")], eChildren = [Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "title"}, eAttrs = [], eChildren = [Text "Doc title"]},Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "org:myproject:otherns", nnLocalPart = "ref"}, eAttrs = [], eChildren = [Text "abc1234"]},Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "html"}, eAttrs = [(NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "xmlns"},"http://www.w3.org/1999/xhtml")], eChildren = [Element {eName = NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "body"}, eAttrs = [], eChildren = [Text "Hello world!"]}]},Text "\n"]} Best regards Krzysztof Skrzętnicki
And just to provide an example of working program: --- module Main where import Text.XML.Expat.Qualified import Text.XML.Expat.Namespaced import Text.XML.Expat.Tree import qualified Data.ByteString.Lazy as BSL main = do f <- BSL.readFile "doc1.xml" let (tree,error) = parseTree Nothing f case error of Nothing -> putStrLn "Here you are: " >> (print . toNamespaced . toQualified $ (tree :: Node String String)) Just err -> putStrLn "Error!" >> print err --- $ ./hexpat-test Here you are: Element {eName = NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "doc"}, eAttrs = [(NName {nnNamespace = Just "http://www.w3.org/2000/xmlns/", nnLocalPart = "x"},"org:myproject:otherns"),(NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "xmlns"},"org:myproject:mainns")], eChildren = [Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "title"}, eAttrs = [], eChildren = [Text "Doc title"]},Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "org:myproject:otherns", nnLocalPart = "ref"}, eAttrs = [], eChildren = [Text "abc1234"]},Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "html"}, eAttrs = [(NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "xmlns"},"http://www.w3.org/1999/xhtml")], eChildren = [Element {eName = NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "body"}, eAttrs = [], eChildren = [Text "Hello world!"]}]},Text "\n"]} # we mess with doc1.xml and exchange </doc> for </do> $ ./hexpat-test Error! XMLParseError "mismatched tag" (XMLParseLocation {xmlLineNumber = 5, xmlColumnNumber = 2, xmlByteIndex = 205, xmlByteCount = 0}) Best regards Krzysztof Skrzętnicki 2009/6/9 Krzysztof Skrzętnicki <gtener@gmail.com>:
On Mon, Jun 8, 2009 at 20:39, John Millikin<jmillikin@gmail.com> wrote:
I'm trying to convert an XML document, incrementally, into a sequence of XML events. A simple example XML document:
<doc xmlns="org:myproject:mainns" xmlns:x="org:myproject:otherns"> <title>Doc title</title> <x:ref>abc1234</x:ref> <html xmlns="http://www.w3.org/1999/xhtml"><body>Hello world!</body></html> </doc>
The document can be very large, and arrives in chunks over a socket, so I need to be able to "feed" the text data into a parser and receive a list of XML events per chunk. Chunks can be separated in time by intervals of several minutes to an hour, so pausing processing for the arrival of the entire document is not an option. The type signatures would be something like:
type Namespace = String type LocalName = String
data Attribute = Attribute Namespace LocalName String
data XMLEvent = EventElementBegin Namespace LocalName [Attribute] | EventElementEnd Namespace LocalName | EventContent String | EventError String
parse :: Parser -> String -> (Parser, [XMLEvent])
I've looked at HaXml, HXT, and hexpat, and unless I'm missing something, none of them can achieve this:
+ HaXml and hexpat seem to disregard namespaces entirely -- that is, the root element is parsed to "doc" instead of ("org:myproject:mainns", "doc"), and the second child is "x:ref" instead of ("org:myproject:otherns", "ref"). Obviously, this makes parsing mixed-namespace documents effectively impossible. I found an email from 2004[1] that mentions a "filter" for namespace support in HaXml, but no further information and no working code.
I would recommend hexpat to do the job. Contrary to what you are saying, hexpat does offer namespace handling: http://hackage.haskell.org/packages/archive/hexpat/0.8/doc/html/Text-XML-Exp... Perhaps you need more than that?
Personally I found hexpat to be fast, space efficient and easy to use.
Here is the representation I got for your example. Please note the namespaces in right places. * > (toNamespaced ( toQualified t')) Element {eName = NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "doc"}, eAttrs = [(NName {nnNamespace = Just "http://www.w3.org/2000/xmlns/", nnLocalPart = "x"},"org:myproject:otherns"),(NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "xmlns"},"org:myproject:mainns")], eChildren = [Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "org:myproject:mainns", nnLocalPart = "title"}, eAttrs = [], eChildren = [Text "Doc title"]},Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "org:myproject:otherns", nnLocalPart = "ref"}, eAttrs = [], eChildren = [Text "abc1234"]},Text "\n",Text " ",Element {eName = NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "html"}, eAttrs = [(NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "xmlns"},"http://www.w3.org/1999/xhtml")], eChildren = [Element {eName = NName {nnNamespace = Just "http://www.w3.org/1999/xhtml", nnLocalPart = "body"}, eAttrs = [], eChildren = [Text "Hello world!"]}]},Text "\n"]}
Best regards
Krzysztof Skrzętnicki
Hi, you may also want to look at: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/xml It knows about namespaces and, also, it's parser is lazy. -Iavor On Mon, Jun 8, 2009 at 11:39 AM, John Millikin<jmillikin@gmail.com> wrote:
I'm trying to convert an XML document, incrementally, into a sequence of XML events. A simple example XML document:
<doc xmlns="org:myproject:mainns" xmlns:x="org:myproject:otherns"> <title>Doc title</title> <x:ref>abc1234</x:ref> <html xmlns="http://www.w3.org/1999/xhtml"><body>Hello world!</body></html> </doc>
The document can be very large, and arrives in chunks over a socket, so I need to be able to "feed" the text data into a parser and receive a list of XML events per chunk. Chunks can be separated in time by intervals of several minutes to an hour, so pausing processing for the arrival of the entire document is not an option. The type signatures would be something like:
type Namespace = String type LocalName = String
data Attribute = Attribute Namespace LocalName String
data XMLEvent = EventElementBegin Namespace LocalName [Attribute] | EventElementEnd Namespace LocalName | EventContent String | EventError String
parse :: Parser -> String -> (Parser, [XMLEvent])
I've looked at HaXml, HXT, and hexpat, and unless I'm missing something, none of them can achieve this:
+ HaXml and hexpat seem to disregard namespaces entirely -- that is, the root element is parsed to "doc" instead of ("org:myproject:mainns", "doc"), and the second child is "x:ref" instead of ("org:myproject:otherns", "ref"). Obviously, this makes parsing mixed-namespace documents effectively impossible. I found an email from 2004[1] that mentions a "filter" for namespace support in HaXml, but no further information and no working code.
+ HXT looks promising, because I see explicit mention in the documentation of recording and propagating namespaces. However, I can't figure out if there's an incremental mode. A page on the wiki[2] suggests that SAX is supported in the "html tag soup" parser, but I want incremental parsing of *valid* documents. If incremental parsing is supported by the standard "arrow" interface, I don't see any obvious way to pull events out into a list -- I'm a Haskell newbie, and still haven't quite figured out monads yet, let alone Arrows.
Are there any libraries that support namespace-aware incremental parsing?
[1] http://www.haskell.org/pipermail/haskell-cafe/2004-June/006252.html [2] http://www.haskell.org/haskellwiki/HXT/Conversion_of_Haskell_data_from/to_XM... _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
participants (5)
-
Henning Thielemann -
Iavor Diatchki -
John Millikin -
Krzysztof Skrzętnicki -
Malcolm Wallace