Sounds bad. But it'll need someone with bytestring expertise to debug. Maybe there's a GHC problem underlying; or maybe it's shortcoming of bytestring.
Simon
| -----Original Message-----
| From: Glasgow-haskell-users [mailto:glasgow-haskell-users-
| bounces@haskell.org] On Behalf Of Dominic Steinitz
| Sent: 25 August 2016 10:11
| To: GHC users
| Subject: GHC Performance / Replacement for R?
|
| I am trying to use Haskell as a replacement for R but running into two
| problems which I describe below. Are there any plans to address the
| performance issues I have encountered?
|
| 1. I seem to have to jump through a lot of hoops just to be able to
| select the data I am interested in.
|
| {-# LANGUAGE ScopedTypeVariables #-}
|
| {-# OPTIONS_GHC -Wall #-}
|
| import Data.Csv hiding ( decodeByName )
| import qualified Data.Vector as V
|
| import Data.ByteString ( ByteString )
| import qualified Data.ByteString.Char8 as B
|
| import qualified Pipes.Prelude as P
| import qualified Pipes.ByteString as Bytes import Pipes import
| qualified Pipes.Csv as Csv import System.IO
|
| import qualified Control.Foldl as L
|
| main :: IO ()
| main = withFile "examples/787338586_T_ONTIME.csv" ReadMode $ \h -> do
| let csvs :: Producer (V.Vector ByteString) IO ()
| csvs = Csv.decode HasHeader (Bytes.fromHandle h) >-> P.concat
| uvectors :: Producer (V.Vector ByteString) IO ()
| uvectors = csvs >-> P.map (V.foldr V.cons V.empty)
| vec_vec <- L.impurely P.foldM L.vector uvectors
| print $ (vec_vec :: V.Vector (V.Vector ByteString)) V.! 17
| print $ V.length vec_vec
| let rockspring = V.filter (\x -> x V.! 8 == B.pack "RKS") vec_vec
| print $ V.length rockspring
|
| Here's the equivalent R:
|
| df <- read.csv("787338586_T_ONTIME.csv")
| rockspring <- df[df$ORIGIN == "RKS",]
|
| 2. Now I think I could improve the above to make an environment that
| is more similar to the one my colleagues are used to in R but more
| problematical is the memory usage.
|
| * 112.5M file
| * Just loading the source into ghci takes 142.7M
| * > foo <- readFile "examples/787338586_T_ONTIME.csv" > length foo
| takes me up to 4.75G. But we probably don't want to do this!
| * Let's try again.
| * > :set -XScopedTypeVariables
| * > h <- openFile "examples/787338586_T_ONTIME.csv" ReadMode
| * > let csvs :: Producer (V.Vector ByteString) IO () = Csv.decode
| HasHeader (Bytes.fromHandle h) >-> P.concat
| * > let uvectors :: Producer (V.Vector ByteString) IO () = csvs >->
| P.map (V.map id) >-> P.map (V.foldr V.cons V.empty)
| * > vec_vec :: V.Vector (V.Vector ByteString) <- L.impurely P.foldM
| L.vector uvectors
| * Now I am up at 3.17G. In R I am under 221.3M.
| * > V.length rockspring takes a long time to return 155 and now I am
| at 3.5G!!! In R > rockspring <- df[df$ORIGIN == "RKS",] seems
| instantaneous and now uses only 379.5M.
| * > length(rockspring) 37 > length(df$ORIGIN) 471949 i.e. there are
| 37 columns and 471,949 rows.
|
| Running this as an executable gives
|
| ~/Dropbox/Private/labels $ ./examples/BugReport +RTS -s ["2014-01-
| 01","EV","20366","N904EV","2512","10747","1074702","30747",
| "BRO","Brownsville, TX","Texas","11298","1129803","30194",
| "DFW","Dallas/Fort Worth, TX","Texas","0720","0718",
| "-2.00","8.00","0726","0837","7.00","0855","0844","-11.00","0.00",
| "","0.00","482.00","","","","","",""]
| 471949
| 155
| 14,179,764,240 bytes allocated in the heap
| 3,378,342,072 bytes copied during GC
| 786,333,512 bytes maximum residency (13 sample(s))
| 36,933,976 bytes maximum slop
| 1434 MB total memory in use (0 MB lost due to
| fragmentation)
|
| Tot time (elapsed) Avg pause
| Max pause
| Gen 0 26989 colls, 0 par 1.423s 1.483s 0.0001s
| 0.0039s
| Gen 1 13 colls, 0 par 1.005s 1.499s 0.1153s
| 0.6730s
|
| INIT time 0.000s ( 0.003s elapsed)
| MUT time 3.195s ( 3.193s elapsed)
| GC time 2.428s ( 2.982s elapsed)
| EXIT time 0.016s ( 0.138s elapsed)
| Total time 5.642s ( 6.315s elapsed)
|
| %GC time 43.0% (47.2% elapsed)
|
| Alloc rate 4,437,740,019 bytes per MUT second
|
| Productivity 57.0% of total user, 50.9% of total elapsed
|
| _______________________________________________
| Glasgow-haskell-users mailing list
| Glasgow-haskell-users@haskell.org
| https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fmail.h
| askell.org%2fcgi-bin%2fmailman%2flistinfo%2fglasgow-haskell-
| users&data=01%7c01%7csimonpj%40microsoft.com%7c5017a5fe26cb4df9c41d08d
| 3ccc7b5bd%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=2Ku1Fr5QttHRoj5
| NSOJREZrt2Fsqhi63iJOpxmku68E%3d
