- Beginners - Haskell.org

Writing huge result of `show` to file results in out of memory
by Mahdi Dibaiee 30 Oct '16

30 Oct '16

Hi, So I have a data instance which contains a few big matrices, and I want to save my instance to a file so I can `read` it back later to avoid the long computation every time (I'm training a recurrent neural network). First attempt, the simplest method: writeFile "rnn" (show dataInstance) It starts to take all of the memory and then bails out with `out-of-memory` error. So I wrote a function to write the string chunk by chunk, without buffering, here is the code: https://github.com/mdibaiee/sibe/blob/728df02fbdd6f134af107c098f5477094c61e… Copy/pasted from the link: saveRecurrent :: FilePath -> String -> Int -> IO () saveRecurrent path str chunkSize = do handle <- openFile path AppendMode hSetBuffering handle NoBuffering loop handle str hClose handle where loop _ [] = return () loop handle s = do hPutStr handle $ take chunkSize s hFlush handle loop handle $ drop chunkSize s But it doesn't work either, I still get `out-of-memory` errors. From what I understand, this should work, but it isn't. I asked on IRC and someone said "Show is not lazy enough", if that's the case, I would appreciate an explanation of that. Thanks, Mahdi

4 4

Re: [Haskell-beginners] Beginners Digest, Vol 100, Issue 19
by Ashish Negi 30 Oct '16

30 Oct '16

Can you write your own custom function for converting the data structure to string ?

3 2

Re: [Haskell-beginners] Beginners Digest, Vol 100, Issue 20
by Mahdi Dibaiee 29 Oct '16

29 Oct '16

That's not generally true (but I will give my guess about your assumption after this). See the example in the article I linked: ``` import Data.Char(toUpper) main = do inpStr <- readFile "input.txt" writeFile "output.txt" (map toUpper inpStr) ``` Here, `inputStr` has to be evaluated in order to run `map toUpper` on it, but it's done character by character. Character is read, `toUpper` is run on it, it's written to the file. (Though it might not be a character unless we use `NoBuffering`, it's going to be lines or blocks) Now, I guess your assumption _might_ be true in my case, because `show` might not be able to generate the string character by character (which is very likely), in this case, show will give us the whole string in one piece, leading to an out-of-memory error as the big string is getting loaded to memory in order to be used by `writeFile`. This is just a guess, I would appreciate it if someone could actually prove it. What do you think Imants? -----Original Message----- From: beginners-request <beginners-request(a)haskell.org> To: beginners <beginners(a)haskell.org> Sent: Sat, Oct 29, 2016 3:44 pm Subject: Beginners Digest, Vol 100, Issue 20 Send Beginners mailing list submissions to beginners(a)haskell.orgTo subscribe or unsubscribe via the World Wide Web, visit http://mail.haskell.org/cgi-bin/mailman/listinfo/beginnersor, via email, send a message with subject or body 'help' to beginners-request(a)haskell.orgYou can reach the person managing the list at beginners-owner(a)haskell.orgWhen replying, please edit your Subject line so it is more specificthan "Re: Contents of Beginners digest..."Today's Topics: 1. Writing huge result of `show` to file results in out of memory (Mahdi Dibaiee) 2. Re: Writing huge result of `show` to file results in out of memory (Imants Cekusins) Attached Message From Mahdi Dibaiee <mdibaiee(a)aol.com> To beginners(a)haskell.org Subject [Haskell-beginners] Writing huge result of `show` to file resultsin out of memory Date Fri, 28 Oct 2016 12:57:22 -0400 I know that there are other ways of doing it, I just want to understand where lies the problem here. I just read the realworldhaskell book by O'Reilly, in one section, Lazy I/O [0], it's explained that there should be no problem writing big strings, without consuming the whole string to files, however big they are. I would appreciate it if someone could point out the problem there. Thanks [0]: http://book.realworldhaskell.org/read/io.html#io.lazy Attached Message From Imants Cekusins <imantc(a)gmail.com> To The Haskell-Beginners Mailing List - Discussion of primarily beginner-level topics related to Haskell <beginners(a)haskell.org> Subject Re: [Haskell-beginners] Writing huge result of `show` to fileresults in out of memory Date Fri, 28 Oct 2016 19:13:23 +0200 > there should be no problem writing big strings, without consuming the whole string lazy means evaluated when needed. However when a string is evaluated, it is evaluated fully, I guess. _______________________________________________Beginners mailing listBeginners@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/beginners

2 1

Writing huge result of `show` to file results in out of memory
by Mahdi Dibaiee 28 Oct '16

28 Oct '16

I know that there are other ways of doing it, I just want to understand where lies the problem here. I just read the realworldhaskell book by O'Reilly, in one section, Lazy I/O [0], it's explained that there should be no problem writing big strings, without consuming the whole string to files, however big they are. I would appreciate it if someone could point out the problem there. Thanks [0]: http://book.realworldhaskell.org/read/io.html#io.lazy

2 1

The meaning of categories constructed from HASK
by Dimitri DeFigueiredo 25 Oct '16

25 Oct '16

In category theory, there are many ways one can make new categories out of an old one. In particular, given a category C one can construct: 1. The arrows category of C: arrows in C become objects and commutative squares in C become arrows 2. The slice category of C given an object A: arrows into a distinguished object A become objects in the slice commutative triangles become arrows There are also functors going from C to these new categories (and back). Are these constructed categories useful when C = `Hask` (the category of haskell types and functions)? What do they represent in programming terms? In other words, is there intuition for what the arrows category of Hask is? What about the slice category of Hask over a specific type? Do the functors between these match some programming abstractions? Any pointers are much appreciated. Thanks, Dimitri

3 2

Iterate in terms of scanl
by Atrudyjane 23 Oct '16

23 Oct '16

Hello All, Sorry about my previous post, email was hosed. I was working on a homework problem where the task is to write the iterate function in terms of scanl. Came up with this: myIterate f x = scanl (const.f) x (repeat x) I went looking around for other solutions to check my work and found the following solution on the Haskell Wiki: iterate f x = scanl f x (repeat x) myIterate seems to work checked against the Prelude iterate but I don't know if it's a good solution or not, because the iterate solution on the Haskell Wiki throws a type error. Types of Prelude iterate and scanl are different, so I can see why, or am I missing something? Here is some REPL output: Prelude iterate: λ> take 10 (iterate (+1) 1) [1,2,3,4,5,6,7,8,9,10] myIterate: λ> take 10 (myIterate (+1) 1) [1,2,3,4,5,6,7,8,9,10] Haskell Wiki solution: λ> take 10 (iterate' (+1) 1) error: • Occurs check: cannot construct the infinite type: a ~ a -> a Expected type: a -> a -> a Actual type: (a -> a) -> a -> a • In the first argument of ‘iterate'’, namely ‘(+ 1)’ In the second argument of ‘take’, namely ‘(iterate' (+ 1) 1)’ In the expression: take 10 (iterate' (+ 1) 1) • Relevant bindings include it :: [a] (bound at <interactive>:63:1) It will work if you pass it a function that takes two parameters though: λ> take 10 (iterate' (+) 1) [1,2,3,4,5,6,7,8,9,10] Any thoughts would be much appreciated! Andrea

1 0

Iterate in terms of scanlIterate in
by Andrea ML 23 Oct '16

23 Oct '16

1 0

Re: [Haskell-beginners] Beginners Digest, Vol 100, Issue 14
by Lai Boon Hui 22 Oct '16

22 Oct '16

Hi David, Thanks for chipping in. I found the solution to be to include "directory" in cabal file "build-depends". I was confused because my stack ghci config didn't need :m + System.directory to work. But somehow cabal repl does not work without some kind of import. On Sat, Oct 22, 2016 at 8:00 PM, <beginners-request(a)haskell.org> wrote: > Send Beginners mailing list submissions to > beginners(a)haskell.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners > or, via email, send a message with subject or body 'help' to > beginners-request(a)haskell.org > > You can reach the person managing the list at > beginners-owner(a)haskell.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Beginners digest..." > > > Today's Topics: > > 1. cabal repl issue (Lai Boon Hui) > 2. Re: cabal repl issue (David McBride) > 3. Re: Monadic functions definitions for free monadic DSL > (Sumit Raja) > 4. Re: Monadic functions definitions for free monadic DSL > (Kim-Ee Yeoh) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 21 Oct 2016 22:35:19 +0800 > From: Lai Boon Hui <laiboonh(a)gmail.com> > To: beginners(a)haskell.org > Subject: [Haskell-beginners] cabal repl issue > Message-ID: > <CAJdQggnSvw8z7V+SBY=y2TEfhpcK2znLHpkp9i+Edq5QAgXM7w@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi all, > > when i execute cabal repl i get this: > > <interactive>:1:8: error: > > Not in scope: ‘System.Directory.getCurrentDirectory’ > > No module named ‘System.Directory’ is imported. > because my ghci.conf has > :def pwd (\_-> System.Directory.getCurrentDirectory >>= print >> return > "") > > Does anyone know how i can include System.Directory somehow even though its > not used in my cabal project? > > -- > Best Regards, > Boon Hui >

1 0

Re: [Haskell-beginners] Monadic functions definitions for free monadic DSL
by Sumit Raja 21 Oct '16

21 Oct '16

> I would really like to help you, but without your imports, packages, etc, > it is really hard to interpret your program. Like where does decodeUtf8 > come from, or receive, or TCPSocket? If they are functions you wrote, I > don't need their code, the types would be sufficient. > Imports are: import Protolude import Control.Monad.Free import System.Socket import System.Socket.Family.Inet import System.Socket.Type.Stream import System.Socket.Protocol.TCP import Control.Exception ( bracket, catch ) import Data.ByteString as BS (uncons) decodeUtf8 :: ByteString -> Text encodeUtf8 :: Text -> ByteString I'm using the socket library for the actual networking (https://hackage.haskell.org/package/socket-0.6.0.1) type TCPSocket = Socket Inet Stream TCP receive :: Socket f t p -> Int -> MessageFlags -> IO ByteString Source send :: Socket f t p -> ByteString -> MessageFlags -> IO Int accept :: (Family f, Storable (SocketAddress f)) => Socket f t p -> IO (Socket f t p, SocketAddress f) If it helps the full source is at https://bitbucket.org/sumitraja/network-free/src/a4fcbc74c9e178e81d8b10b60d…. Looking forward to your assistance. Thanks Sumit

2 1

Re: [Haskell-beginners] Monadic functions definitions for free monadic DSL
by Sumit Raja 21 Oct '16

21 Oct '16

> I feel like if you can figure out what you actually want Accept to do, it > will become clearer. Here's my attempt. Accept takes a chan, takes a > procedure to loop on, a procedure to accept on, and then returns the server > chan to continue the loop. I don't know if this is entirely right, but it > type checks and hopefully it will give you some ideas. As you've said Accept needs refinement. I tried writing a interpretUDP which doesn't have an accept loop but does have a bind + listen. I suspect that accept needs to move into the interpretTCP somehow and the DSL needs to be Bind or similar. Bind makes more sense as well if I wanted to write a pipes or a chan based interpreter. Thanks for the rewrite and the pointers. -Sumit

1 0