On Tue, 26 Jul 2011 10:17:22 +0200 (CEST) Henning Thielemann wrote:

On Mon, 25 Jul 2011, Manfred Lotz wrote:

...

Hi there, If I take example imap.hs

import System.IO import Network.HaskellNet.IMAP import Text.Mime import qualified Data.ByteString.Char8 as BS import Control.Monad

-- the next lines were changed to fit to my local imap server imapServer = "imap.mail.org" user = "" pass = ""

main = do

s> > con <- connectIMAP imapServer

...

login con user pass mboxes <- list con mapM print mboxes

This should be mapM_ and 'ghc -Wall' spots this problem since 6.12.

The compiler (7.04) doesn't tell me anything about it.

...

select con "INBOX" msgs <- search con [ALLs] mapM_ (\x -> print x) (take 4 msgs) forM_ (take 4msgs) (\x -> fetch con x >>= print)

I'm not quite sure I understand what you mean. Stack overflow comes from this: forM_ msgs (\x -> fetch con x >>= print) If I change it to: mapM_ (\x -> fetch con x >>= print) msgs there is the same stack overflow. -- Thanks, Manfred

On Tue, 26 Jul 2011 14:17:08 -0700 (PDT) Donn Cave wrote:

Quoth Manfred Lotz , ...

...

I'm not quite sure I understand what you mean. Stack overflow comes from this: forM_ msgs (\x -> fetch con x >>= print)

If I change it to: mapM_ (\x -> fetch con x >>= print) msgs

there is the same stack overflow.

I didn't understand that myself, but neither do I know what might be wrong. One thing to consider is that email messages can be very large. Looking at messages received in the last 10 days I see I have one that exceeds your reported stack size, and that isn't counting the extra space required for text representation of non printing characters etc. There may be messages that you simply can't "print".

The HaskellNet IMAP "fetch" is actually FETCH ... BODY[], i.e., the whole contents of the message. Normal practice for giant data files is to send them as part of a MIME multipart/mixed message, and something like the above can proceed with a reasonable chance of success if it avoids these attachments by fetching BODY[1] (or BODY[1.1], etc. depending on actual structure.) I just fetched the 10Mb message I mentioned above to check the structure, and it happened in the blink of an eye - BODY[1] is smaller than the header.

I don't see any support for fetch by part, you might have to hack that up yourself. You may ideally also want to fetch BODYSTRUCTURE, but practically I might go out on a limb and predict that you won't run into messages where the first part is a multipart/mixed with a large attachment - so if the object is just a survivable first part, you could live without BODYSTRUCTURE analysis and optimistically ask for BODY[1].

Moving on to practical use of email via IMAP, you'd also want to be able to fetch and decode the attachments. At this point, it's interesting to return to the question of space requirements.

Donn

The problem seems to lie in the HaskellNet package. If for example I only fetch a specific message m <- fetch con 2092 having a size of some 1.2m then I get the same stack overflow. If at runtime I specify +RTS -K40M -RTS it works but takes over 40 seconds to fetch the message. -- Manfred

On Wed, 27 Jul 2011 09:47:52 -0700 (PDT) Donn Cave wrote:

Quoth Manfred Lotz , ...

...

The problem seems to lie in the HaskellNet package. If for example I only fetch a specific message m <- fetch con 2092 having a size of some 1.2m then I get the same stack overflow.

If at runtime I specify +RTS -K40M -RTS it works but takes over 40 seconds to fetch the message.

That's not so good, but I wouldn't be surprised if it's a natural parsing problem, I mean it's just a lot of data to run through a Haskell parser.

Yep, I agree. Perhaps the library should provide a fetchRaw function to get the whole message without much parsing. Perhaps I could tell the package author what the problem is, and he is happy to provide a solution. In the end the only thing I need is to get the full message because I want to feed bogofilter to learn that a message is ham or spam. For the time being I decided to write my own program to fetch the data because it is a good exercise for a Haskell beginner as I am. -- Manfred

Hello, I guess I should stick my hand up as the supposed maintainer of HaskellNet. Unfortunately I can't say that I know the code that well. Two years ago I rescued it from bitrot cabalized it and when I couldn't get any response from the original author put myself down as the maintainer. It is a package which is starting to show its age. Michael Snoyman and I had a conversation in February agreeing that I should try to revamp it by applying some techniques such as those used in blaze html. Unfortunately, I haven't had the time. I agree with the post above that for mime mail HaskellNet shouldn't be retreiving all of the messages with their message bodies. I might see if I can get a chance to work on it a little this weekend but if someone is using the library and has some time to make some changes that person woould be very welcome (and I'd be more than happy if someone wishes to take over as the maintenance). -Rob On Wed, Jul 27, 2011 at 5:47 PM, Donn Cave wrote:

Quoth Manfred Lotz , ...

...

The problem seems to lie in the HaskellNet package. If for example I only fetch a specific message m <- fetch con 2092 having a size of some 1.2m then I get the same stack overflow.

If at runtime I specify +RTS -K40M -RTS it works but takes over 40 seconds to fetch the message.

That's not so good, but I wouldn't be surprised if it's a natural parsing problem, I mean it's just a lot of data to run through a Haskell parser.

IMAP does give you the means to mitigate the problem - the big data transfer in a FETCH response is preceded by a byte count - but to really take advantage of that, how far do you go? I don't have much experience with general purpose parsers, do they often support an efficient counted string read? Is it OK to return String, or do we need a more efficient type (e.g., ByteString?) Is it OK to return any kind of string value - given that a message part could be arbitrarily long (and needs to be decoded), do you go to a lot of trouble to support large message parts but not extremely large ones?

For me, the answer is for the parser to bail out, reporting the counted input as a count but leaving it to the application to actually effect the data transfer and return to finish the parse. That's only moderately complicated, but it's part of a general philosophy about application driven I/O vs. protocol parsing that seems to be mine alone.

I have no idea how much could be done to tighten up HaskellNet.IMAP. Someone who understands it well enough might be able to get a miraculous improvement with a strictness annotation or something. Maybe you could track that down with profiling.

Donn

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

...

...

This should be mapM_ and 'ghc -Wall' spots this problem since 6.12.

...

The compiler (7.04) doesn't tell me anything about it.

Henning> It seems that it is no longer part of -Wall. Indeed, that's not part of -Wall. http://www.haskell.org/ghc/docs/7.0.4/html/users_guide/options-sanity.html Am I the only one who assumed so far that Wall turned on all existing warnings ?

From the doc :

-Wall: Turns on all warning options that indicate potentially suspicious code. The warnings that are not enabled by -Wall are -fwarn-tabs, -fwarn-incomplete-record-updates, -fwarn-monomorphism-restriction, -fwarn-unused-do-bind, and -fwarn-implicit-prelude. -w: Turns off all warnings, including the standard ones and those that -Wall doesn't enable. If there were no backward compatibility issues, I'd prefer to just see -w and -Wall swaped. -w would mean "We let the GHC team decide what subset of warnings they really want us to observe", and -Wall would mean "We really want them all". -- Paul

Hem hem ... I should never try to write anything sensible before putting my thick glasses. -w does not turn ON all warnings, but turns them OFF, so my previous comment regarding swapping its definition with -Wall is just nonsense. Sorry for the noise. Still, do you think there could be room for a -Wsuspicious that would be defined as current -Wall, and for a more intuitive meaning for -Wall : turns on really all warnings ? Paul> Indeed, that's not part of -Wall. Paul> http://www.haskell.org/ghc/docs/7.0.4/html/users_guide/options-sanity.html Paul> Am I the only one who assumed so far that Wall turned on all Paul> existing warnings ? Paul> From the doc : Paul> -Wall: Turns on all warning options that indicate potentially Paul> suspicious code. The warnings that are not enabled by -Wall Paul> are -fwarn-tabs, -fwarn-incomplete-record-updates, -fwarn-monomorphism-restriction, -fwarn-unused-do-bind, Paul> and -fwarn-implicit-prelude. Paul> -w: Turns off all warnings, including the standard ones and those Paul> that -Wall doesn't enable. Paul> If there were no backward compatibility issues, I'd prefer to just Paul> see -w and -Wall swaped. -w would mean "We let the GHC team decide Paul> what subset of warnings they really want us to observe", and -Wall Paul> would mean "We really want them all". -- Paul