problem with System.Directory.Tree
Hello All, I want to build a program which will recursively scan a directory and build md5sum for all the files. The intent is to do something similar to unison but more specific to my requirements. I am having trouble in the initial part of building the md5sums. I did some digging around and found that "System.Directory.Tree" is a very close match for what I want to do. In fact after a little poking around I could do exactly what I wanted. ,---- | import Monad | import System.Directory.Tree | import System.Directory | import Data.Digest.Pure.MD5 | import qualified Data.ByteString.Lazy.Char8 as L | | calcMD5 = | readDirectoryWith (\x-> liftM md5 (L.readFile x)) `---- This work perfectly for small directories. readDirectoryWith is already defined in the library and exactly what we want ,---- | *Main> calcMD5 "/home/mitra/Desktop/" | | "/home/mitra" :/ Dir {name = "Desktop", contents = [File {name = | "060_LocalMirror_Workflow.t.10.2.62.9.log", file = | f687ad04bc64674134e55c9d2a06902a},File {name = "cmd_run", file = | 6f334f302b5c0d2028adeff81bf2a0d9},File {name = "cmd_run~", `---- However when ever I give it something more challenging it gets into trouble. ,---- | *Main> calcMD5 "/home/mitra/laptop/" | *** Exception: /home/mitra/laptop/ell/calc-2.02f/calc.info-27: | openFile: resource exhausted (Too many open files) | *Main> 29~ `---- If I understand what is happening it seems to be doing all the opens before consuming them via md5. This works fine for small directories but for any practical setup this could potentially be very large. I tried forcing the md5 evaluation in the hope that the file descriptor will be freed once the entire file is read. That did not help, either because I could not get it right or there is some more subtle I am missing. I also had a look at the code in module "System.Directory.Tree" and although it gave me some understanding of how it works I am no closer to a solution. regards -- Anand Mitra
On Monday 07 June 2010 14:06:22, Anand Mitra wrote:
Hello All,
I want to build a program which will recursively scan a directory and build md5sum for all the files. The intent is to do something similar to unison but more specific to my requirements. I am having trouble in the initial part of building the md5sums.
I did some digging around and found that "System.Directory.Tree" is a very close match for what I want to do. In fact after a little poking around I could do exactly what I wanted.
,----
| import Monad | import System.Directory.Tree | import System.Directory | import Data.Digest.Pure.MD5 | import qualified Data.ByteString.Lazy.Char8 as L | | calcMD5 = | readDirectoryWith (\x-> liftM md5 (L.readFile x))
Does calcMD5 = readDirectoryWith (\x -> do txt <- readFile x return $! md5 txt) help?
`----
This work perfectly for small directories. readDirectoryWith is already defined in the library and exactly what we want
,----
| *Main> calcMD5 "/home/mitra/Desktop/" | | "/home/mitra" :/ Dir {name = "Desktop", contents = [File {name = | "060_LocalMirror_Workflow.t.10.2.62.9.log", file = | f687ad04bc64674134e55c9d2a06902a},File {name = "cmd_run", file = | 6f334f302b5c0d2028adeff81bf2a0d9},File {name = "cmd_run~",
`----
However when ever I give it something more challenging it gets into trouble.
,----
| *Main> calcMD5 "/home/mitra/laptop/" | *** Exception: /home/mitra/laptop/ell/calc-2.02f/calc.info-27: | openFile: resource exhausted (Too many open files) | *Main> 29~
`----
If I understand what is happening it seems to be doing all the opens before consuming them via md5. This works fine for small directories but for any practical setup this could potentially be very large. I tried forcing the md5 evaluation in the hope that the file descriptor will be freed once the entire file is read. That did not help, either because I could not get it right or there is some more subtle I am missing.
I also had a look at the code in module "System.Directory.Tree" and although it gave me some understanding of how it works I am no closer to a solution.
regards
Hi Daniel, That works just as intended, Thanks. On Tue, Jun 8, 2010 at 1:31 AM, Daniel Fischer <daniel.is.fischer@web.de>wrote:
Does
calcMD5 = readDirectoryWith (\x -> do txt <- readFile x return $! md5 txt)
help?
`----
This work perfectly for small directories. readDirectoryWith is already defined in the library and exactly what we want
,----
| *Main> calcMD5 "/home/mitra/Desktop/" | | "/home/mitra" :/ Dir {name = "Desktop", contents = [File {name = | "060_LocalMirror_Workflow.t.10.2.62.9.log", file = | f687ad04bc64674134e55c9d2a06902a},File {name = "cmd_run", file = | 6f334f302b5c0d2028adeff81bf2a0d9},File {name = "cmd_run~",
`----
However when ever I give it something more challenging it gets into trouble.
,----
| *Main> calcMD5 "/home/mitra/laptop/" | *** Exception: /home/mitra/laptop/ell/calc-2.02f/calc.info-27: | openFile: resource exhausted (Too many open files) | *Main> 29~
`----
If I understand what is happening it seems to be doing all the opens before consuming them via md5. This works fine for small directories but for any practical setup this could potentially be very large. I tried forcing the md5 evaluation in the hope that the file descriptor will be freed once the entire file is read. That did not help, either because I could not get it right or there is some more subtle I am missing.
I also had a look at the code in module "System.Directory.Tree" and although it gave me some understanding of how it works I am no closer to a solution.
regards
_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners
I did something similar where I built up an md5sum of all the files in a directory for comparing whether two directories were identical (I was cleaning up some server storage). One difference is that I only read the first 4096 bytes of the file because if files are going to differ they will likely differ in those bytes (and definitely would in my case) and that is the default page read size is I recall, so even if you use hGet handle 512, the system still reads 4192 bytes into memory anyway, so why not use them. I think I had a similar problem to yours with open file handles until i used `withFile` from System.IO. This handy function took care of closing up file resources for me so I wouldn't have a ton of open file handles. My getFileHash function is as follows: getFileHash :: FilePath -> IO (Maybe String) getFileHash path = (do contents <- withFile path ReadMode (\h -> hGet h 4096) return . Just $! md5sum contents) `catch` (\e -> printFileError e >> return Nothing) printFileError is just a function for printing out pretty errors related to files. You can see that it reads some contents of the file through withFile and then md5sums them. I have the $! to force evaluation so it will compute as we go, otherwise it builds a huge tree of sums waiting to be computed before computing the result for display at the root. There are other $! operators in the tree operations to collapse at that level, and now the program runs in constant memory space. -- Drew Haven drew.haven@gmail.com On Mon, Jun 7, 2010 at 5:06 AM, Anand Mitra <anand.mitra@gmail.com> wrote:
Hello All,
I want to build a program which will recursively scan a directory and build md5sum for all the files. The intent is to do something similar to unison but more specific to my requirements. I am having trouble in the initial part of building the md5sums.
I did some digging around and found that "System.Directory.Tree" is a very close match for what I want to do. In fact after a little poking around I could do exactly what I wanted.
,---- | import Monad | import System.Directory.Tree | import System.Directory | import Data.Digest.Pure.MD5 | import qualified Data.ByteString.Lazy.Char8 as L | | calcMD5 = | readDirectoryWith (\x-> liftM md5 (L.readFile x)) `----
This work perfectly for small directories. readDirectoryWith is already defined in the library and exactly what we want
,---- | *Main> calcMD5 "/home/mitra/Desktop/" | | "/home/mitra" :/ Dir {name = "Desktop", contents = [File {name = | "060_LocalMirror_Workflow.t.10.2.62.9.log", file = | f687ad04bc64674134e55c9d2a06902a},File {name = "cmd_run", file = | 6f334f302b5c0d2028adeff81bf2a0d9},File {name = "cmd_run~", `----
However when ever I give it something more challenging it gets into trouble.
,---- | *Main> calcMD5 "/home/mitra/laptop/" | *** Exception: /home/mitra/laptop/ell/calc-2.02f/calc.info-27: | openFile: resource exhausted (Too many open files) | *Main> 29~ `----
If I understand what is happening it seems to be doing all the opens before consuming them via md5. This works fine for small directories but for any practical setup this could potentially be very large. I tried forcing the md5 evaluation in the hope that the file descriptor will be freed once the entire file is read. That did not help, either because I could not get it right or there is some more subtle I am missing.
I also had a look at the code in module "System.Directory.Tree" and although it gave me some understanding of how it works I am no closer to a solution.
regards -- Anand Mitra
_______________________________________________ Beginners mailing list Beginners@haskell.org http://www.haskell.org/mailman/listinfo/beginners
Hi Again, Thanks to the help from this group I have got past my first problem and now stuck on another. I have succeeded in building a DirTree using library "System.Directory.Tree" with properties of md5sum and modification time. I have also built the parts that can compare two such trees and find the files that have changed. I was trying to serializing the tree so that it can be saved for later identification of changed files. The module "Tree" incidentally derives Show but not Read and hence I cannot read the file serialized file. Searching a bit I decided to use the infrastructure in "Data.Binary" to do the job for me. As soon as I started this I realized that I would have to modify Tree.hs module. This was required because DirTree does not derive from Typeable and Data which is required for it to be serialized via "Data.Binary". After patching the Binary module to derive from Typeable and Data I get the following error. System/Directory/Tree.hs:95:47: No instance for (Data Exception) arising from the 'deriving' clause of a data type declaration at System/Directory/Tree.hs:95:47-50 Possible fix: add an instance declaration for (Data Exception) or use a standalone 'deriving instance' declaration instead, so you can specify the instance context yourself When deriving the instance for (Data (DirTree a)) Failed, modules loaded: BinaryDerive. Secondly I suspect that I could have derived it without having to modify the original module source. The compilation error does give a hint about "standalone deriving instance" but trying stuff at http://www.haskell.org/ghc/docs/6.12.2/html/users_guide/deriving.html did not help me much. In short what is the simplest way I can serialize "System.Directory.Tree" using Binary. Is there a better alternative to Binary for serialization ? Are there solutions to the problem I have outlined above ? or is my approach incorrect. Is it possible to add the deriving of datatype DirTree without modifying the module ? regards -- Anand Mitra On Mon, Jun 7, 2010 at 5:36 PM, Anand Mitra <anand.mitra@gmail.com> wrote:
Hello All,
I want to build a program which will recursively scan a directory and build md5sum for all the files. The intent is to do something similar to unison but more specific to my requirements. I am having trouble in the initial part of building the md5sums.
I did some digging around and found that "System.Directory.Tree" is a very close match for what I want to do. In fact after a little poking around I could do exactly what I wanted.
,---- | import Monad | import System.Directory.Tree | import System.Directory | import Data.Digest.Pure.MD5 | import qualified Data.ByteString.Lazy.Char8 as L | | calcMD5 = | readDirectoryWith (\x-> liftM md5 (L.readFile x)) `----
This work perfectly for small directories. readDirectoryWith is already defined in the library and exactly what we want
,---- | *Main> calcMD5 "/home/mitra/Desktop/" | | "/home/mitra" :/ Dir {name = "Desktop", contents = [File {name = | "060_LocalMirror_Workflow.t.10.2.62.9.log", file = | f687ad04bc64674134e55c9d2a06902a},File {name = "cmd_run", file = | 6f334f302b5c0d2028adeff81bf2a0d9},File {name = "cmd_run~", `----
However when ever I give it something more challenging it gets into trouble.
,---- | *Main> calcMD5 "/home/mitra/laptop/" | *** Exception: /home/mitra/laptop/ell/calc-2.02f/calc.info-27: | openFile: resource exhausted (Too many open files) | *Main> 29~ `----
If I understand what is happening it seems to be doing all the opens before consuming them via md5. This works fine for small directories but for any practical setup this could potentially be very large. I tried forcing the md5 evaluation in the hope that the file descriptor will be freed once the entire file is read. That did not help, either because I could not get it right or there is some more subtle I am missing.
I also had a look at the code in module "System.Directory.Tree" and although it gave me some understanding of how it works I am no closer to a solution.
regards -- Anand Mitra
Hello Anand System.Directory.Tree is not a good candidate for serializing directly - as you have found out, one of the constructors - Failed - carries an IOException which cannot be serialized (IOExceptions may contain file handles which are inherently runtime values). Data.Binary is usually the best option for serialization. In your case, you wont be able to make a 1-1 mapping between a runtime DirTree and its on disk representation as you'll have to work out what to do about 'Failed' - maybe you would want to only serialize the good constructors - Dir and File - instead. As you will have to coerce the data type a bit, I'd recommend using Data.Binary to write the serialization, but rather than make instances of Put and Get for DirTree instead give the serialize and deserialize functions characteristic names e.g. serializeGoodTree / deserializeGoodTree. Best wishes Stephen
Hi, Continuing on my project I have hit another hurdle. Summarizing this is what I have been tring to do. Using System.Directory.Tree to traverse a directory tree and build a DirTree of type Props where Props are data Props = Prop { md5sum :: MD5Digest, modTime :: ClockTime, filenam::FilePath } | Blank deriving (Show, Eq, Typeable) -- , Data I am trying to capture the md5sum and time. Next I wanted to serialize this structure to disk for comparison at a later invocation. I got stuck trying to serialize DirTree using Data.Binary and encode because DirTree has a field of type Exception which does not derive from Binary/Data which is required for serialization. I felt that using "deriving instance " could solve the problem but I don't understand that mechanism well enough to solve my problem. How ever an alternative was suggested and this is what I did. It is in-elegant but sort of works. I defined a alternate private DirTreeW which mirrors the DirTree except that the Exception is converted to a String. The data type is defined below data DirTreeW a = DirW { name :: FileName, contents :: [DirTreeW a] } | FileW { name :: FileName, file :: a } | FailedW { name :: FileName, err :: String } deriving (Show, Eq, Typeable, Data) I then wrote a function to convert a DirTree to this type. convert (Dir a b) = DirW a $ map convert b convert (File a b) = FileW a b convert (Failed a e) = FailedW a "some error" This version of DirTree I can serialize. However a DirTreeW of type Props fails to serialize because of the MD5Digest and ClockTime. I tried deriving Props from Data but that fails with the following error. *BinaryDerive Data.Binary Data.Digest.Pure.MD5> :r [2 of 2] Compiling Main ( dir-recurse.hs, interpreted ) Ok, modules loaded: BinaryDerive, Main. *Main Data.Binary Data.Digest.Pure.MD5> :r [2 of 2] Compiling Main ( dir-recurse.hs, interpreted ) dir-recurse.hs:16:56: No instances for (Data MD5Digest, Data ClockTime) arising from the 'deriving' clause of a data type declaration at dir-recurse.hs:16:56-59 Possible fix: add an instance declaration for (Data MD5Digest, Data ClockTime) or use a standalone 'deriving instance' declaration instead, so you can specify the instance context yourself When deriving the instance for (Data Props) Failed, modules loaded: BinaryDerive. It seems to suggest that md5 and clocktime are not derived from Data and hence I cannot derive Props from Data. Please suggest on a way forward. I would appreciate pointers on exactly what the purpose of "deriving instace" is. I found the relevant references in the ghc compiler reference but the are far too cryptic and I cannot understand it. I am attaching the full source in the attachment if you need to examine it. Hoping to see useful suggestions soon. regards -- Anand Mitra On Mon, Jun 14, 2010 at 1:26 PM, Stephen Tetley <stephen.tetley@gmail.com>wrote:
Hello Anand
System.Directory.Tree is not a good candidate for serializing directly - as you have found out, one of the constructors - Failed - carries an IOException which cannot be serialized (IOExceptions may contain file handles which are inherently runtime values).
Data.Binary is usually the best option for serialization. In your case, you wont be able to make a 1-1 mapping between a runtime DirTree and its on disk representation as you'll have to work out what to do about 'Failed' - maybe you would want to only serialize the good constructors - Dir and File - instead.
As you will have to coerce the data type a bit, I'd recommend using Data.Binary to write the serialization, but rather than make instances of Put and Get for DirTree instead give the serialize and deserialize functions characteristic names e.g. serializeGoodTree / deserializeGoodTree.
Best wishes
Stephen
Hi Anand MD5Digest is an abstract type (the constructor is not exported from its module) but it is an instance of Binary. ClockTime (from System.Time) is not an instance of Binary but it does export its constructor. Neither are instances of Data.Data. So I would hand-craft an instance of Binary for the Props datatype rather than try to first make them instances of Data. The code will be something like this, as I don't have MD5 installed it is unchecked: class Binary Props where put (Prop md5 tim name) = do { putWord8 0 -- number each constructor ; put md5 -- MD5Digest has a Binary instance ; putTOD tim ; put name } put Blank = do { putWord8 1 } -- number each constructor get = do { typ <- getWord8 -- get the constructor tag... ; case typ of 0 -> getProp 1 -> return Blank } getProp :: Get Props getProp = do { md5 <- get ; tim <- getTOD ; name <- get ; return (Prop md5 tim name) } -- ClockTime doesn't have a binary instance -- but a it has a single constructor -- -- > TOD Integer Integer - -- -- and Integer has a Binary instance, so I -- would make auxillaris for put and get: putTOD :: ClockTime -> Put () putTOD (TOD a b) = do { put a ; put b } getTOD :: Get ClockTime getTOD = do { a <- get; b <- get; return (TOD a b) }
All the advice and help I got on my difficulties till now have been very useful. My current problem is a little weird and can't figure out what is happening. I have been able to get the serialization working with DirTree based on the suggestions I have received till now. I have a function calcMD5 which given a FilePath will traverse the entire tree calculating the checksum of each file it encounters. The resultant structure is serializable by encode. But when I do a encodeFile to store the result to a file I get nothing. ,---- | *Main> calcMD5 "/tmp/tmp" | AncTree "/tmp" (DirW {name = "tmp", contents = [FileW {name = "passwd", file = Prop {md5sum = f54e7cef69973cecdce3c923da2f9222, modTime = Tue Jul 6 07:18:16 IST 2010, filenam = "/tmp/tmp/passwd"}}]}) | | *Main> liftM encode $ calcMD5 "/tmp/tmp" | Chunk "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\EOT/tmp\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\ETXtmp\NUL\NUL\NUL\NUL\NUL\NUL\NUL\SOH\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL\ACKpasswd\NUL\245N|\239i\151<\236\220\227\201#\218/\146\"\NULL2\139`\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\SI/tmp/tmp/passwd" Empty `---- clearly the encoding is working. But if I try to use encodeFile to write it to a file the file is not created. ,---- | *Main> liftM (encodeFile "/tmp/tmp-list") $ calcMD5 "/tmp/tmp" | | $ ls /tmp/tmp-list | ls: cannot access /tmp/tmp-list: No such file or directory `---- I tried a few other things like converting the Bytestring from encode into a string using show and then doing a writeFile on it. Unfortunately none of them worked. regards -- Anand Mitra On Mon, Jun 28, 2010 at 1:23 PM, Stephen Tetley <stephen.tetley@gmail.com>wrote:
Hi Anand
MD5Digest is an abstract type (the constructor is not exported from its module) but it is an instance of Binary. ClockTime (from System.Time) is not an instance of Binary but it does export its constructor. Neither are instances of Data.Data.
So I would hand-craft an instance of Binary for the Props datatype rather than try to first make them instances of Data.
The code will be something like this, as I don't have MD5 installed it is unchecked:
class Binary Props where put (Prop md5 tim name) = do { putWord8 0 -- number each constructor ; put md5 -- MD5Digest has a Binary instance ; putTOD tim ; put name }
put Blank = do { putWord8 1 } -- number each constructor
get = do { typ <- getWord8 -- get the constructor tag... ; case typ of 0 -> getProp 1 -> return Blank }
getProp :: Get Props getProp = do { md5 <- get ; tim <- getTOD ; name <- get ; return (Prop md5 tim name) }
-- ClockTime doesn't have a binary instance -- but a it has a single constructor -- -- > TOD Integer Integer - -- -- and Integer has a Binary instance, so I -- would make auxillaris for put and get:
putTOD :: ClockTime -> Put () putTOD (TOD a b) = do { put a ; put b }
getTOD :: Get ClockTime getTOD = do { a <- get; b <- get; return (TOD a b) }
On Tuesday 06 July 2010 04:01:58, Anand Mitra wrote:
All the advice and help I got on my difficulties till now have been very useful. My current problem is a little weird and can't figure out what is happening.
I have been able to get the serialization working with DirTree based on the suggestions I have received till now. I have a function calcMD5 which given a FilePath will traverse the entire tree calculating the checksum of each file it encounters. The resultant structure is serializable by encode. But when I do a encodeFile to store the result to a file I get nothing.
,----
| *Main> calcMD5 "/tmp/tmp" | AncTree "/tmp" (DirW {name = "tmp", contents = [FileW {name = | "passwd",
file = Prop {md5sum = f54e7cef69973cecdce3c923da2f9222, modTime = Tue Jul 6 07:18:16 IST 2010, filenam = "/tmp/tmp/passwd"}}]})
| *Main> liftM encode $ calcMD5 "/tmp/tmp" | Chunk
"\NUL\NUL\NUL\NUL\NUL\NUL\NUL\EOT/tmp\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\ET Xtmp\NUL\NUL\NUL\NUL\NUL\NUL\NUL\SOH\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL\ACKp asswd\NUL\245N|\239i\151<\236\220\227\201#\218/\146\"\NULL2\139`\NUL\NUL\ NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\SI/tmp/tmp/passwd" Empty `----
clearly the encoding is working. But if I try to use encodeFile to write it to a file the file is not created.
,----
| *Main> liftM (encodeFile "/tmp/tmp-list") $ calcMD5 "/tmp/tmp"
calcMD5 :: IO something I suppose? then liftM (encodeFile "/tmp/tmp-list") $ calcMD5 "/tmp/tmp" has type IO (IO ()) and executing that only evaluates the action (encodeFile "/tmp/tmp-list" calcMD5Result) , it doesn't execute it. What you want is calcMD5 "/tmp/tmp" >>= encodeFile "/tmp/tmp-list"
| | $ ls /tmp/tmp-list | ls: cannot access /tmp/tmp-list: No such file or directory
`----
I tried a few other things like converting the Bytestring from encode into a string using show and then doing a writeFile on it. Unfortunately none of them worked.
regards
participants (5)
-
Anand Mitra -
Anand Mitra -
Daniel Fischer -
Drew Haven -
Stephen Tetley