filesystem verification utility
Hi, I had a requirement to generate various kinds of I/O patterns on a filesystem and subsequently verify this. The initial version of the program below implements a random I/O pattern with multiple threads. Even when the number of I/O is as small as 200 and 10 concurrent theads, the amount of memory used is huge. When I run the program it consumes close to 1to2GB memory. Moreover the rate at which it generates the I/O is very low which is not good for testing a filesystem. I have used System.POSIX.IO but I tried System.IO and did not see much difference either. I would appreciate help in identifying ways to improve this. {-# OPTIONS -fglasgow-exts #-} import System.Random import Data.List import Monad import System.Posix.IO import System.Posix.Types import Data.Time.Clock import System.Posix.Files import Data.Maybe import GHC.IO.Device (SeekMode(..)) import Control.Exception import Data.Typeable import System import Control.Concurrent --myrandomRlist :: (Num t, Random t) => t -> IO [t] --myrandomRlist x = liftM (randomRs (0,x)) newStdGen myrandomRlist :: (Num t, Random t) => t ->t -> StdGen -> [t] myrandomRlist min max seed = randomRs (min,max) seed data IoLoc = IoLoc {offset :: FileOffset, size::ByteCount, num::Int } deriving (Show, Typeable) instance Exception IoLoc data Corrupt = Corrupt String IoLoc deriving (Show, Typeable) instance Exception Corrupt data FileHdr = FileHdr {fileName::FilePath, seed::StdGen, minIoSize::Int, maxIoSize::Int, ioCount::Int} deriving (Show) data FileIO = FileIO {fd::Fd, params::FileHdr, fileData::[IoLoc]} genPattern :: FilePath -> IoLoc -> String genPattern f l = take (read $ show $size l) (cycle $ "(" ++ f ++ ")" ++ "[" ++ (show (offset l)) ++ ":" ++ (show (size l)) ++ (show (num l)) ++ "]") doHdrRead :: Fd -> IO [Char] doHdrRead fd = do (str, count) <- fdRead fd 100 return (takeWhile (\x-> not (x == '\n')) str) doGetHdr :: Fd -> IO FileHdr doGetHdr fd = do file <- doHdrRead fd seed <- doHdrRead fd minIoSize <- doHdrRead fd maxIoSize <- doHdrRead fd ioCount <- doHdrRead fd return (FileHdr file (read seed) (read minIoSize) (read maxIoSize) (read ioCount)) doHdrFromFile :: FilePath -> IO FileIO doHdrFromFile name = do fd <- openFd name ReadWrite (Just ownerModes) (OpenFileFlags {append=False, exclusive=False, noctty=True, nonBlock=False, trunc=False}) hdr <- doGetHdr fd return (FileIO fd hdr []) doHdrWrite :: Fd -> [Char] -> IO ByteCount doHdrWrite fd str = do fdWrite fd (take 100 (str ++ "\n" ++ (cycle ['\0']))) doWriteHdr :: Fd -> FileHdr -> IO ByteCount doWriteHdr fd hdr = do doHdrWrite fd (show (fileName hdr)) doHdrWrite fd (show (seed hdr)) doHdrWrite fd (show (minIoSize hdr)) doHdrWrite fd (show (maxIoSize hdr)) doHdrWrite fd (show (ioCount hdr)) doWriteFile :: FilePath -> IO FileHdr doWriteFile name = do fd <- openFd name ReadWrite (Just ownerModes) (OpenFileFlags {append=False, exclusive=False, noctty=True, nonBlock=False, trunc=True}) seed <- newStdGen hdr <- return (FileHdr name seed 4096 (2*8096) 200) doWriteHdr fd hdr return hdr overLap (IoLoc off1' sz1' num1) (IoLoc off2' sz2' num2) = ((off1 > off2) && (off1 < off2 +sz2)) ||((off1+sz1 >off2) &&(off1+sz1 < off2 +sz2)) where off1 = read (show off1') sz1 = read (show sz1') off2 = read (show off2') sz2 = read (show sz2') genIoList1 :: [IoLoc] -> [Int] -> [Int] -> Int -> Int -> [IoLoc] genIoList1 list offset size _ 0 = list genIoList1 list offset size 0 _ = list genIoList1 list (offset:os) (size:ss) count bound = if isNothing $ find (overLap x) list then genIoList1 ([x] ++ list) os ss (count - 1) (bound -1) else genIoList1 list os ss count (bound -1) where x = IoLoc (read (show offset)) (read(show size)) count -- offset = (read (show offset1)) -- size = (read(show size1)) genIoList :: FileHdr -> [IoLoc] genIoList (FileHdr name seed min max count) = genIoList1 [] (myrandomRlist 4000 1099511627776 seed) (myrandomRlist min max seed) count (count*2) doActualIo :: Fd -> IoLoc -> String -> IO () doActualIo fd (IoLoc off sz num) str = do off <- fdSeek fd AbsoluteSeek off fdWrite fd str return () doVerifyIo :: Fd -> IoLoc -> String -> IO () doVerifyIo fd (IoLoc off sz num) str = do off <- fdSeek fd AbsoluteSeek off (filedata, count) <- fdRead fd sz if str == filedata then return () else throwIO (Corrupt ("Data corruption in #" ++ (take 200 str) ++ "#" ++ (take 200 filedata))(IoLoc off sz num)) mainWrite file = do hdr <- doWriteFile file hdrIO <- doHdrFromFile file iolst <- return $ genIoList hdr app <- return $ zip iolst (map (genPattern (fileName hdr)) iolst) putStrLn (fileName hdr) mapM (\(x,y)-> doActualIo (fd hdrIO) x y) app mapM (\(x,y)-> doVerifyIo (fd hdrIO) x y) app return hdr mainVerify file = do hdrIO <- doHdrFromFile file hdr <- return (params hdrIO) iolst <- return $ genIoList hdr file <- return $ filter (\x->not ( x=='"')) (fileName hdr) app <- return $ zip iolst (map (genPattern file) iolst) putStrLn $ filter (\x->not ( x=='"')) (fileName hdr) mapM (\(x,y)-> doVerifyIo (fd hdrIO) x y) app return hdr main = do x <- getArgs if (length x) == 3 then do main1 else do putStrLn "USAGE:\nfile-io write/verify <full-path file name> <number of threads> \ \\n\n\ \Simple IO load generator with write verification. This utility is designed\n\ \to generate multi-threaded IO load which will write a pattern to the file. \n\ \When this is invoked with the same parameters with the verify option \n\ \the data written will be verified.\n\ \ " main1 = do [op, name, numProcs] <- getArgs m <- newEmptyMVar n <- return $read numProcs case op of "write" -> do mapM forkIO [(fillfile (name ++ (show i)) ) m|i<-[1..n]] "verify" -> do mapM forkIO [(verifyfile (name ++ (show i)) ) m|i<-[1..n]] x <- mapM takeMVar $ take (read numProcs) $ repeat m putStrLn $ show $ and x fillfile filename m = do mainWrite filename putMVar m True verifyfile filename m = do mainVerify filename putMVar m True
I don't know much, but you're using show, read, and String. These may be part of your problem. On Mon, Jan 10, 2011 at 2:32 PM, Anand Mitra <mitra@kqinfotech.com> wrote:
Hi,
I had a requirement to generate various kinds of I/O patterns on a filesystem and subsequently verify this. The initial version of the program below implements a random I/O pattern with multiple threads. Even when the number of I/O is as small as 200 and 10 concurrent theads, the amount of memory used is huge. When I run the program it consumes close to 1to2GB memory. Moreover the rate at which it generates the I/O is very low which is not good for testing a filesystem. I have used System.POSIX.IO but I tried System.IO and did not see much difference either. I would appreciate help in identifying ways to improve this.
{-# OPTIONS -fglasgow-exts #-} import System.Random import Data.List import Monad import System.Posix.IO import System.Posix.Types import Data.Time.Clock import System.Posix.Files import Data.Maybe import GHC.IO.Device (SeekMode(..)) import Control.Exception import Data.Typeable import System import Control.Concurrent
--myrandomRlist :: (Num t, Random t) => t -> IO [t] --myrandomRlist x = liftM (randomRs (0,x)) newStdGen
myrandomRlist :: (Num t, Random t) => t ->t -> StdGen -> [t] myrandomRlist min max seed = randomRs (min,max) seed
data IoLoc = IoLoc {offset :: FileOffset, size::ByteCount, num::Int } deriving (Show, Typeable) instance Exception IoLoc
data Corrupt = Corrupt String IoLoc deriving (Show, Typeable) instance Exception Corrupt
data FileHdr = FileHdr {fileName::FilePath, seed::StdGen, minIoSize::Int, maxIoSize::Int, ioCount::Int} deriving (Show)
data FileIO = FileIO {fd::Fd, params::FileHdr, fileData::[IoLoc]}
genPattern :: FilePath -> IoLoc -> String genPattern f l = take (read $ show $size l) (cycle $ "(" ++ f ++ ")" ++ "[" ++ (show (offset l)) ++ ":" ++ (show (size l)) ++ (show (num l)) ++ "]")
doHdrRead :: Fd -> IO [Char] doHdrRead fd = do (str, count) <- fdRead fd 100 return (takeWhile (\x-> not (x == '\n')) str)
doGetHdr :: Fd -> IO FileHdr doGetHdr fd = do file <- doHdrRead fd seed <- doHdrRead fd minIoSize <- doHdrRead fd maxIoSize <- doHdrRead fd ioCount <- doHdrRead fd return (FileHdr file (read seed) (read minIoSize) (read maxIoSize) (read ioCount))
doHdrFromFile :: FilePath -> IO FileIO doHdrFromFile name = do fd <- openFd name ReadWrite (Just ownerModes) (OpenFileFlags {append=False, exclusive=False, noctty=True, nonBlock=False, trunc=False}) hdr <- doGetHdr fd return (FileIO fd hdr [])
doHdrWrite :: Fd -> [Char] -> IO ByteCount doHdrWrite fd str = do fdWrite fd (take 100 (str ++ "\n" ++ (cycle ['\0'])))
doWriteHdr :: Fd -> FileHdr -> IO ByteCount doWriteHdr fd hdr = do doHdrWrite fd (show (fileName hdr)) doHdrWrite fd (show (seed hdr)) doHdrWrite fd (show (minIoSize hdr)) doHdrWrite fd (show (maxIoSize hdr)) doHdrWrite fd (show (ioCount hdr))
doWriteFile :: FilePath -> IO FileHdr doWriteFile name = do fd <- openFd name ReadWrite (Just ownerModes) (OpenFileFlags {append=False, exclusive=False, noctty=True, nonBlock=False, trunc=True}) seed <- newStdGen hdr <- return (FileHdr name seed 4096 (2*8096) 200) doWriteHdr fd hdr return hdr
overLap (IoLoc off1' sz1' num1) (IoLoc off2' sz2' num2) = ((off1 > off2) && (off1 < off2 +sz2)) ||((off1+sz1 >off2) &&(off1+sz1 < off2 +sz2)) where off1 = read (show off1') sz1 = read (show sz1') off2 = read (show off2') sz2 = read (show sz2')
genIoList1 :: [IoLoc] -> [Int] -> [Int] -> Int -> Int -> [IoLoc]
genIoList1 list offset size _ 0 = list genIoList1 list offset size 0 _ = list
genIoList1 list (offset:os) (size:ss) count bound = if isNothing $ find (overLap x) list then genIoList1 ([x] ++ list) os ss (count - 1) (bound -1) else genIoList1 list os ss count (bound -1) where x = IoLoc (read (show offset)) (read(show size)) count -- offset = (read (show offset1)) -- size = (read(show size1))
genIoList :: FileHdr -> [IoLoc] genIoList (FileHdr name seed min max count) = genIoList1 [] (myrandomRlist 4000 1099511627776 seed) (myrandomRlist min max seed) count (count*2)
doActualIo :: Fd -> IoLoc -> String -> IO () doActualIo fd (IoLoc off sz num) str = do off <- fdSeek fd AbsoluteSeek off fdWrite fd str return ()
doVerifyIo :: Fd -> IoLoc -> String -> IO () doVerifyIo fd (IoLoc off sz num) str = do off <- fdSeek fd AbsoluteSeek off (filedata, count) <- fdRead fd sz if str == filedata then return () else throwIO (Corrupt ("Data corruption in #" ++ (take 200 str) ++ "#" ++ (take 200 filedata))(IoLoc off sz num))
mainWrite file = do hdr <- doWriteFile file hdrIO <- doHdrFromFile file iolst <- return $ genIoList hdr app <- return $ zip iolst (map (genPattern (fileName hdr)) iolst) putStrLn (fileName hdr) mapM (\(x,y)-> doActualIo (fd hdrIO) x y) app mapM (\(x,y)-> doVerifyIo (fd hdrIO) x y) app return hdr
mainVerify file = do hdrIO <- doHdrFromFile file hdr <- return (params hdrIO) iolst <- return $ genIoList hdr file <- return $ filter (\x->not ( x=='"')) (fileName hdr) app <- return $ zip iolst (map (genPattern file) iolst) putStrLn $ filter (\x->not ( x=='"')) (fileName hdr) mapM (\(x,y)-> doVerifyIo (fd hdrIO) x y) app return hdr
main = do x <- getArgs if (length x) == 3 then do main1 else do putStrLn "USAGE:\nfile-io write/verify <full-path file name> <number of threads> \ \\n\n\ \Simple IO load generator with write verification. This utility is designed\n\ \to generate multi-threaded IO load which will write a pattern to the file. \n\ \When this is invoked with the same parameters with the verify option \n\ \the data written will be verified.\n\ \ "
main1 = do [op, name, numProcs] <- getArgs m <- newEmptyMVar n <- return $read numProcs case op of "write" -> do mapM forkIO [(fillfile (name ++ (show i)) ) m|i<-[1..n]] "verify" -> do mapM forkIO [(verifyfile (name ++ (show i)) ) m|i<-[1..n]] x <- mapM takeMVar $ take (read numProcs) $ repeat m putStrLn $ show $ and x
fillfile filename m = do mainWrite filename putMVar m True
verifyfile filename m = do mainVerify filename putMVar m True
_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners
-- Alex R
On 10 January 2011 22:26, Stephen Tetley <stephen.tetley@gmail.com> wrote:
And append (++) inside the genPattern function...
Changing to use "functional strings" which have the ShowS type is the obvious way to avoid the cost of (++) append. There is a library on Hackage that provides many utility functions for a wrapped ShowS type and an IsString instance so you can still use string literals. http://hackage.haskell.org/package/dstring However you might be better using one of the alternative String packages such as Text or ByteString.
Based on feedback I inferred that the huge memory usage was mostly because of the String handling in the patern generation. To make it more efficient I have used Data.ByteString.Lazy.Char8 but now I get the following error when I execute stress: tmp/asdf-1: hPutBuf: illegal operation (handle is closed) stress: tmp/asdf-3: hPutBuf: illegal operation (handle is closed) stress: tmp/asdf-5: hPutBuf: illegal operation (handle is closed) stress: tmp/asdf-2: hPutBuf: illegal operation (handle is closed) stress: tmp/asdf-7: hPutBuf: illegal operation (handle is closed) stress: tmp/asdf-10: hPutBuf: illegal operation (handle is closed) I have explicitly remove all calls to close despite this I get this error. I tried both version of seek the fdSeek as well as hSeek. The documentation on hSeek had a confusing comment that "The offset i is given in terms of 8-bit bytes" At this point I am unable to understand how the handle is getting explicitly closed. {-# OPTIONS -fglasgow-exts #-} import Data.Int import qualified Data.ByteString.Lazy.Char8 as L import System.Random import Data.List import Monad import System.Posix.IO import System.Posix.Types import Data.Time.Clock import System.Posix.Files import Data.Maybe import GHC.IO.Device (SeekMode(..)) import Control.Exception import Data.Typeable import System import Control.Concurrent import System.IO --myrandomRlist :: (Num t, Random t) => t -> IO [t] --myrandomRlist x = liftM (randomRs (0,x)) newStdGen myrandomRlist :: (Num t, Random t) => t ->t -> StdGen -> [t] myrandomRlist min max seed = randomRs (min,max) seed data IoLoc = IoLoc {offset ::FileOffset, size::Int, num::Int } deriving (Show, Typeable) instance Exception IoLoc data Corrupt = Corrupt String IoLoc deriving (Show, Typeable) instance Exception Corrupt data FileHdr = FileHdr {fileName::FilePath, seed::StdGen, minIoSize::Int, maxIoSize::Int, ioCount::Int} deriving (Show) data FileIO = FileIO {fd::Handle, params::FileHdr, fileData::[IoLoc]} check hdl = do closed <- hIsClosed hdl if closed then do putStrLn $ "file handle was closed" ++ (show hdl) else do return () genPattern :: FilePath -> IoLoc -> L.ByteString genPattern f l = L.take (fromIntegral (size l)) (L.cycle $ L.pack ("(" ++ f ++ ")" ++ "[" ++ (show (offset l)) ++ ":" ++ (show (size l)) ++ (show (num l)) ++ "]")) doHdrRead :: Handle -> IO [Char] doHdrRead x = do fd <- handleToFd x (str, count) <- fdRead fd 100 return (takeWhile (\x-> not (x == '\n')) str) doGetHdr :: Handle -> IO FileHdr doGetHdr fd = do file <- doHdrRead fd seed <- doHdrRead fd minIoSize <- doHdrRead fd maxIoSize <- doHdrRead fd ioCount <- doHdrRead fd return (FileHdr file (read seed) (read minIoSize) (read maxIoSize) (read ioCount)) doHdrFromFile :: Handle -> IO FileIO doHdrFromFile fd = do -- fd <- openBinaryFile name ReadWriteMode -- (Just ownerModes) (OpenFileFlags {append=False, exclusive=False, noctty=True, nonBlock=False, trunc=False}) hSeek fd AbsoluteSeek 0 hdr <- doGetHdr fd return (FileIO fd hdr []) doHdrWrite :: Handle -> [Char] -> IO () doHdrWrite fd str = do hPutStr fd (take 100 (str ++ "\n" ++ (cycle ['\0']))) doWriteHdr :: Handle -> FileHdr -> IO () doWriteHdr fd hdr = do doHdrWrite fd (show (fileName hdr)) doHdrWrite fd (show (seed hdr)) doHdrWrite fd (show (minIoSize hdr)) doHdrWrite fd (show (maxIoSize hdr)) doHdrWrite fd (show (ioCount hdr)) doWriteFile :: FilePath -> IO FileIO doWriteFile name = do fd <- openBinaryFile name ReadWriteMode -- (Just ownerModes) (OpenFileFlags {append=False, exclusive=False, noctty=True, nonBlock=False, trunc=True}) seed <- newStdGen hdr <- return (FileHdr name seed 4096 (2*8096) 200) doWriteHdr fd hdr return (FileIO fd hdr []) overLap (IoLoc off1' sz1' num1) (IoLoc off2' sz2' num2) = ((off1 > off2) && (off1 < off2 +sz2)) ||((off1+sz1 >off2) &&(off1+sz1 < off2 +sz2)) where off1 = read (show off1') sz1 = read (show sz1') off2 = read (show off2') sz2 = read (show sz2') genIoList1 :: [IoLoc] -> [Int] -> [Int] -> Int -> Int -> [IoLoc] genIoList1 list offset size _ 0 = list genIoList1 list offset size 0 _ = list genIoList1 list (offset:os) (size:ss) count bound = if isNothing $ find (overLap x) list then genIoList1 ([x] ++ list) os ss (count - 1) (bound -1) else genIoList1 list os ss count (bound -1) where x = IoLoc (read (show offset)) (read(show size)) count -- offset = (read (show offset1)) -- size = (read(show size1)) genIoList :: FileHdr -> [IoLoc] genIoList (FileHdr name seed min max count) = genIoList1 [] (myrandomRlist 4000 1099511627776 seed) (myrandomRlist min max seed) count (count*2) doActualIo :: Handle -> IoLoc -> L.ByteString -> IO () doActualIo fd (IoLoc off sz num) str = do -- hSeek fd AbsoluteSeek off rfd <- (handleToFd fd) fdSeek rfd AbsoluteSeek off L.hPut fd str return () doVerifyIo :: Handle -> IoLoc -> L.ByteString -> IO () doVerifyIo fd (IoLoc off sz num) str = do -- hSeek fd AbsoluteSeek off rfd <- (handleToFd fd) fdSeek rfd AbsoluteSeek off filedata <- L.hGet fd sz if str == filedata then return () else throwIO (Corrupt ("Data corruption in #" ++ (take 200 (L.unpack str)) ++ "#" ++ (take 200 (L.unpack filedata)))(IoLoc off sz num)) mainWrite file = do (FileIO hd _ _) <- doWriteFile file hdrIO <- doHdrFromFile hd check hd (FileIO _ hdr _) <- return hdrIO check hd iolst <- return $ genIoList hdr app <- return $ zip iolst (map (genPattern (fileName hdr)) iolst) putStrLn (fileName hdr) mapM (\(x,y)-> doActualIo (fd hdrIO) x y) app mapM (\(x,y)-> doVerifyIo (fd hdrIO) x y) app -- hClose hd return hdr mainVerify file = do hd <- openBinaryFile file ReadWriteMode hdrIO <- doHdrFromFile hd hdr <- return (params hdrIO) iolst <- return $ genIoList hdr file <- return $ filter (\x->not ( x=='"')) (fileName hdr) app <- return $ zip iolst (map (genPattern file) iolst) putStrLn $ filter (\x->not ( x=='"')) (fileName hdr) mapM (\(x,y)-> doVerifyIo (fd hdrIO) x y) app -- hClose hd return hdr main = do x <- getArgs if (length x) == 3 then do main1 else do putStrLn "USAGE:\nfile-io write/verify <full-path file name> <number of threads> \ \\n\n\ \Simple IO load generator with write verification. This utility is designed\n\ \to generate multi-threaded IO load which will write a pattern to the file. \n\ \When this is invoked with the same parameters with the verify option \n\ \the data written will be verified.\n\ \ " main1 = do [op, name, numProcs] <- getArgs m <- newEmptyMVar n <- return $read numProcs case op of "write" -> do mapM forkIO [(fillfile (name ++ (show i)) ) m|i<-[1..n]] "verify" -> do mapM forkIO [(verifyfile (name ++ (show i)) ) m|i<-[1..n]] x <- mapM takeMVar $ take (read numProcs) $ repeat m putStrLn $ show $ and x fillfile filename m = do mainWrite filename putMVar m True verifyfile filename m = do mainVerify filename putMVar m True On Tue, Jan 11, 2011 at 1:02 AM, Anand Mitra <mitra@kqinfotech.com> wrote:
Hi,
I had a requirement to generate various kinds of I/O patterns on a filesystem and subsequently verify this. The initial version of the program below implements a random I/O pattern with multiple threads. Even when the number of I/O is as small as 200 and 10 concurrent theads, the amount of memory used is huge. When I run the program it consumes close to 1to2GB memory. Moreover the rate at which it generates the I/O is very low which is not good for testing a filesystem. I have used System.POSIX.IO but I tried System.IO and did not see much difference either. I would appreciate help in identifying ways to improve this.
Hi Anand Firstly apologies - my advice from yesterday was trivial advice, changing to a better representation of Strings and avoiding costly operations (++) is valuable and should improve the performance of the program, but it might be a small overall improvement and it doesn't get to the heart of things. Really you need to do two things - one is consider what you are doing and evaluate whether it is appropriate for a performance sensitive app, the other is to profile and find the bits that are too slow. I rarely use Control.Concurrent so I can't offer any real experience but I'm concerned that it is adding overhead for no benefit. Looking at the code and what the comments say it does - I don't think your situation benefits from concurrency. A thread in your program could do all is work in one go, its not that you need to be servicing many clients (cf. a web server that needs to service many clients without individual long waits so it makes sense to schedule them) or that you are waiting on other processes making resources available. So for your program, any changes to execution caused by scheduling / de-scheduling threads (probably) just add to the total time. If you have a multi-core machine you could potentially benefit from parallelism - splitting the work amongst available cores. But in GHC forkIO spawns "green threads" which run in the same OS thread so you won't be getting any automatic parallelism from the code even if you have multi-core. However don't take my word for this - I could easily be wrong. If you want performance you really do need to see what the profiler tells you. Best wishes Stephen
Hi Stephen, Thanks for you reply. I will give you my motivation behind this application so that you have better context. We have been porting ZFS to linux and have written a few programs to stress the filesystem and generate IO load similar to what would be expected in a production environment. The key objective is to find bugs in the ZFS code. I have been interested in haskell for quite some time and used this as an excuse to write something which might be useful. It is possible that what I am doing would be much more efficient in C than in haskell. However if my objective is fulfilled to within a fair degree, the effort will be fun as well as worth the time invested in it. On the high level I expect the program to do the following - ability to perform multi-threaded i/o both threadlevel and process level to increasing contention on locks and shared resources in the filesystem data structures to expose race conditions. - verify each write that has been written. i.e. the content of the file is a function of a random seed which will allow us to detect errors like misplaced writes lost writes etc. - the contents should be self identifying for debugging purposes. If mysterious data appears within a file the contents would make it obvious where it should actually belong. I.e. file name, offset, size and io sequence number. - ability to generate metadata load The existing code does only some of these but it could be expanded to cover all aspects if the existing performance is promising. Getting back to the problem at hand. I had some luck in identifying the cause of the "handle is closed". I suspected that it could be because I was mixing calls from System.IO and System.POSIX.IO. After making the them uniform atleast I don't get a "handle is closed" error but it is hung. Not being familiar with the debugger I resorted to more traditional means of putting putStrLn to find what was happening. And it appeared that the program was getting hung just before starting the random IO. At this point I was distracted with some other work while the apparently hung program was executing. When I came back to the xterm there was the debug putStrLn I had added. This told me that it wasn't hung but just taking a lot of time. From this evidence it was instantly clear what was happening. To simplify the verification process I was checking that there weren't any overlaping I/O. It seems that a large number of them were overlapping and hence an inordinate amount of time was being spent generating the list. I could bet that is where my memory was going as well. I have removed the constraints on the random list generator and now program is able to saturate the disk bandwidth without any trouble. Thanks for you help I'll welcome suggestions to improve the code. regards -- mitra On Tue, Jan 11, 2011 at 3:38 PM, Stephen Tetley <stephen.tetley@gmail.com> wrote:
Hi Anand
Firstly apologies - my advice from yesterday was trivial advice, changing to a better representation of Strings and avoiding costly operations (++) is valuable and should improve the performance of the program, but it might be a small overall improvement and it doesn't get to the heart of things.
Really you need to do two things - one is consider what you are doing and evaluate whether it is appropriate for a performance sensitive app, the other is to profile and find the bits that are too slow.
I rarely use Control.Concurrent so I can't offer any real experience but I'm concerned that it is adding overhead for no benefit. Looking at the code and what the comments say it does - I don't think your situation benefits from concurrency. A thread in your program could do all is work in one go, its not that you need to be servicing many clients (cf. a web server that needs to service many clients without individual long waits so it makes sense to schedule them) or that you are waiting on other processes making resources available. So for your program, any changes to execution caused by scheduling / de-scheduling threads (probably) just add to the total time.
If you have a multi-core machine you could potentially benefit from parallelism - splitting the work amongst available cores. But in GHC forkIO spawns "green threads" which run in the same OS thread so you won't be getting any automatic parallelism from the code even if you have multi-core.
However don't take my word for this - I could easily be wrong. If you want performance you really do need to see what the profiler tells you.
Best wishes
Stephen
_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners
participants (3)
-
Alex Rozenshteyn -
Anand Mitra -
Stephen Tetley