High-level technique for program options handling
Hello! I've used Haskell to create various command-line utitities for unix-like systems. In the process I developed a simple yet powerful and flexible technique for processing program options. What you can read below is my unfinished attempt at writing an article about it. The current form is probably far from good, but I decided I'll rather release it as it is than waste my effort. I will highly appreciate your opinions and criticism for both technical and literary side of this text. High-level technique for program options handling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Introduction ~~~~~~~~~~~~ Sven Panne's GetOpt module does a fine job at parsing command line options. It has a simple, easy to use interface, handles short and long options, abbreviated options and most needed option argument modes (no / optional / required argument). It can even produce a nice usage info from option descriptions. [Documentation and example code using this module can be found at http://www.haskell.org/ghc/docs/latest/html/libraries/base/System.Console.Ge...] However, I checked over half a dozen serious programs written in Haskell and saw that in most of them the code responsible for option handling is quite ugly, repetitious and error prone (of course there were exceptions, see below). Saying 'option handling' I don't mean the work of analyzing the command line, but rather how supplied options influence program's behaviour, how error situations are handled, etc. (The exception is Wolfgang Thaller's VOP program. Wolfgang used a nice, but a little lengthy technique there. There are probably other ,,exceptional'' programs out there). I think there are two major reasons of the current situation. The first reason is that option handling is rarely of primary concern to the programmer. Often there are not that much options to handle in the beginning and it seems that the simplest solution will do. Alas, if the program is evolving and its users ask for new functionality, new options appear, and the initial design starts to be an obstacle. The other reason is lack of good examples. The example delivered with GetOpt shows how to use the library, but it doesn't show that there are other, better ways to use it. I don't propose to introduce more advanced techniques to this example, because it would make it rather heavy. But it would be nice if there would be a pointer for interested users. About typical use of GetOpt ~~~~~~~~~~~~~~~~~~~~~~~~~~~ In a typical use of GetOpt module (like in the example) there is a definition of a sum datatype with data constructors corresponding to individual command-line options. Options with no arguments are represented by nullary constructors, and options that must or can have arguments - by unary constructors. For example, below is a slightly modified example from GetOpt's documentation: data Flag = Verbose -- this option has no arguments | Version -- no arguments | Input (Maybe String) -- optional argument | Output String -- mandatory argument | LibDir String -- mandatory argument GetOpt uses 'ArgDescr a' datatype to specify argument modes of options: data ArgDescr a = NoArg a | ReqArg (String -> a) String | OptArg (Maybe String -> a) String Because every constructor of Flag datatype has one of types Flag, (String -> Flag) or (Maybe String -> Flag), they can be given verbatim as first arguments to appropriate constructors of 'ArgDescr Flag' datatype. Used in this way, getOpt parses a list of strings to a list of Flag values. It also separates options from non option arguments, signals unrecoginized, ambiguous or improperly used options. But after all, it is not that big step. There is still much processing of this Flag list to do. One has to check if specific options are there in the list, some options have to be combined, etc, etc. Advertised technique ~~~~~~~~~~~~~~~~~~~~ ( Because I wanted it to be a fully functional Literate Haskell program, here come the required imports )
module Main (main) where
import System.Console.GetOpt import System import Control.Monad import IO import List import Char
Let's look at program options not from the command-line side, but rather from program(mer)'s side. How to encode them? What would be the best way to present them to the programmer. How should they influence program's behaviour? 'Verbose' could be an easily accessible Bool value, False by default. 'Version', if supplied, could just print the program's version and exit. 'Input' could just yield a String, either from stdin or from a specified file. 'Output' could just take a String and do something with it. We need some kind of a fixed-size polytypic dictionary of option values - a perfect application for Haskell's records.
data Options = Options { optVerbose :: Bool , optInput :: IO String , optOutput :: String -> IO () }
Note that I didn't place optVersion in the record. We will handle this option differently. There is one special combination of options, the start (or default) options:
startOptions :: Options startOptions = Options { optVerbose = False , optInput = getContents , optOutput = putStr }
But in the end we would like to get the record reflecting the options given to the program. We do this by threading the Options record through functions processing single options. Each such function can change this record. Why not just put such functions in ArgDescr and OptDescr datatypes? Here we benefit from first-class citizenship of functions. I won't use a pure function with type (Options -> Options), but rather an effectful function in the IO Monad, because I want to easily perform side effects during option parsing (here only in 'verbose' and 'help' options, but I could also check validity of input and output files during option processing). You may prefer to use a pure function, a State+Error monad, a State+IO monad, or something different...
options :: [ OptDescr (Options -> IO Options) ] options = [ Option "i" ["input"] (ReqArg (\arg opt -> return opt { optInput = readFile arg }) "FILE") "Input file"
, Option "o" ["output"] (ReqArg (\arg opt -> return opt { optOutput = writeFile arg }) "FILE") "Output file"
, Option "s" ["string"] (ReqArg (\arg opt -> return opt { optInput = return arg }) "FILE") "Input string"
, Option "V" ["version"] (NoArg (\_ -> do hPutStrLn stderr "Version 0.01" exitWith ExitSuccess)) "Print version"
, Option "h" ["help"] (NoArg (\_ -> do prg <- getProgName hPutStrLn stderr (usageInfo prg options) exitWith ExitSuccess)) "Show help" ]
Now we combine all this pieces:
main = do args <- getArgs
-- Parse options, getting a list of option actions let (actions, nonOptions, errors) = getOpt RequireOrder options args
-- Here we thread startOptions through all supplied option actions opts <- foldl (>>=) (return startOptions) actions
let Options { optVerbose = verbose , optInput = input , optOutput = output } = opts
when verbose (hPutStrLn stderr "Hello!")
input >>= output
Voila! As you can see most of work is done in option descriptions. I have attached a longer example. It is a simple text file filter that has options for uppercasing characters, reversing characters and lines, dropping initial characters, etc. It also shows how easily you can handle the event of mandatory parameter omission with IO exceptions. Best regards, Tomasz -- .signature: Too many levels of symbolic links
On Sunday 18 January 2004 15:42, Tomasz Zielonka wrote:
[much explanation of his option processing approach elided]
Interesting technique - lots of cool ideas there. I too find getOpts to be a great base but have taken a different approach when writing console-mode Unix programs. Part of my approach is implemented (download http://www.cs.utah.edu/flux/knit/cmi.html for GPL'ed program and look in cmi/src/utils/FluxUtils/Prog.hs) and part is still in my head waiting for an excuse to go cleanup the code. Some of these tricks can be merged with Tomasz's technique (e.g., replace a call to writeFile with a call to writeOutput) while others are orthogonal (e.g., Tomasz deals with arguments one at a time whereas some of my tricks look for duplicated arguments or omitted arguments). I would be very interested in comments on this code, the Unix idioms they implement, Unix idioms omitted, applicability to Windows, MacOS, improving error messages, etc. Here's a list of common Unix idioms and how I implement them: 1) Interpreting the filename "-" as stdin or stdout I use this function (and a similar function for reading input). [Trivial detail: I use pretty printing for all I/O in my programs.] -- | -- Write to file ("-" means stdout) writeOutput :: FilePath -> Doc -> IO () writeOutput "-" output = do printDoc PageMode stdout output writeOutput outfile output = do h <- openFile outfile WriteMode printDoc PageMode h output hClose h 2) Treating arguments of the form 'VARNAME=VALUE' like environment variables (cf. GNU make) and 3) Printing usage info for malformed command lines type StringEnv = [(String,String)] -- | -- Split command line arguments into flags, variable bindings and other. compilerOpts :: (Show a, Eq a) => String -> [OptDescr a] -> IO ([a], StringEnv, [String]) compilerOpts usage options = do argv <- getArgs return $ case getOpt Permute options argv of (o,n,[]) -> (o, env, args) where (env, args) = getEnv n (_,_,errs) -> error (concat errs ++ usageInfo usage options) getEnv :: [String] -> (StringEnv,[String]) getEnv args = (map split env,rest) where (env, rest) = partition ('=' `elem`) args split x = (pre, tail post) where (pre,post) = break (== '=') x An alternative function is the following: -- | -- Extract command line arguments that are inside '+FOO' '-FOO' parentheses -- then split command line arguments into flags, variable bindings and other. -- This is usually used in preference to compilerOpts when a program has to -- (mostly) behave like another program - that is, when the options have to -- be somewhat hidden. runtimeOpts :: (Show a, Eq a) => String -> String -> String -> [OptDescr a] -> IO ([a], StringEnv, [String]) [Incidentally, a cleanup pass might well replace calls to 'error' with calls to one of the following functions: -- | -- Print an error message and exit program with a failure code failWith :: Doc -> IO a failWith msg = do printDoc PageMode stderr msg exitFailure -- | -- Print an error message and exit program with a failure code abortWith :: Doc -> a abortWith msg = unsafePerformIO (failWith (text "" $$ text "Error:" <+> msg)) ] 4) An option can be specified 0 or 1 times: Filter options using this function -- | -- Extract value from a list of length at most one. uniqueWithDefault :: String -> a -> [a] -> a uniqueWithDefault what d [] = d uniqueWithDefault what d [a] = a uniqueWithDefault what d _ = error $ "At most one " ++ what ++ " may be specified" For example, the CMI program starts off like this: -- src/cmi/CMI.hs main = do (flags,env,args) <- compilerOpts usage options let budget = uniqueWithDefault "-b" 0 [ i | Budget b <- flags, (i,"") <- reads b ] let outfile = uniqueWithDefault "-o" "flat.c" [ f | Outfile f <- flags ] let request_files = [ f | Requests f <- flags ] ... 5) An option must be specified exactly once: Filter options using this function: -- | -- Extract value from a list of length one. uniqueNoDefault :: String -> [a] -> a uniqueNoDefault what [] = error $ "You must specify " ++ what uniqueNoDefault what [a] = a uniqueNoDefault what _ = error $ "At most one " ++ what ++ " may be specified" For example, let outfile = uniqueNoDefault "-o" [ f | Outfile f <- flags ] 6) Implementing --help, --version, --numeric-version, --verbose Not implemented yet but I plan to handle these by having the 'compilerOpts' function implement these flags for me. That is, I would define: data StandardOptions = Help | Version | ... and 'compilerOpts' would add these options into the list it passes to getOpts. (The functions to add the options in and separate out the results are a little tedious but not hard.) I'm pretty much agnostic about whether the strings for version, numeric-version, help, etc. should be provided as individual arguments or as a Haskell record. 7) --verbose output and varying levels of verbosity I generate all informational output using this function where the first argument is either True (generate output) or False (don't generate output). -- | -- Print message to stderr if condition holds blurt :: Bool -> Doc -> IO () blurt False msg = return () blurt True msg = printDoc PageMode stderr msg The first argument is usually based on the --verbose flag which is initialized by code like this: let verbosity = length (filter (==Verbose) flags) and a typical call looks like this: blurt (verbosity > 4) $ text "Stripped input:" <+> vmap pp cs [This could probably be improved on using one of a variety of ways of distributing command line flags around a program.] -- Alastair Reid www.haskell-consulting.com
On Mon, Jan 19, 2004 at 02:17:42PM +0000, Alastair Reid wrote:
On Sunday 18 January 2004 15:42, Tomasz Zielonka wrote:
[much explanation of his option processing approach elided]
Interesting technique - lots of cool ideas there.
Thanks :-)
I too find getOpts to be a great base but have taken a different approach when writing console-mode Unix programs. Part of my approach is implemented (download http://www.cs.utah.edu/flux/knit/cmi.html for GPL'ed program and look in cmi/src/utils/FluxUtils/Prog.hs) and part is still in my head waiting for an excuse to go cleanup the code.
Some of these tricks can be merged with Tomasz's technique (e.g., replace a call to writeFile with a call to writeOutput) while others are orthogonal (e.g., Tomasz deals with arguments one at a time whereas some of my tricks look for duplicated arguments or omitted arguments).
Both techniques - the traditional approach with sum Flag datatype and my approach with product Options datatype - can be seen as being dual to themselves, as you can implement one of top of the other. Moreover, there is an easy migration path from traditional option processing to my technique - you just build a list of Flag values as one of Options fields. This way you can migrate your options one at a time. data Flag = Verbose | Version -- | Input (Maybe String) -- this one moved to Options field | Output String deriving Show data Options = Options { optFlags :: [Flag] -> [Flag] , optInput :: IO String } startOptions :: Options startOptions = Options { optFlags = id , optInput = getContents } options :: [OptDescr (Options -> IO Options)] options = [ Option "h" ["help"] (NoArg (\opt -> exitHelp)) "Show usage info" , Option "i" ["input"] (ReqArg (\arg opt -> return opt { optInput = readInput arg }) "FILE") "Input file, - for stdin" , Option "o" ["output"] (ReqArg (appendFlag . Output) "FILE") "Output file, - for stdout" , Option "v" ["verbose"] (NoArg (appendFlag Verbose)) "Be verbose" , Option "V" ["version"] (NoArg (appendFlag Version)) "Print version" ] where appendFlag :: Flag -> (Options -> IO Options) appendFlag flag opts = return opts { optFlags = (flag :) . (optFlags opts) } main = do (opts, _) <- parseOptions let flags = optFlags opts [] ... I also thought about nesting one Options datatype in another datatype. This could be done with a function with type: (a -> b) -> (a -> b -> a) -> [OptDescr (b -> IO b)] -> [OptDescr (a -> IO a)] It would allow to divide options in groups.
I would be very interested in comments on this code, the Unix idioms they implement, Unix idioms omitted, applicability to Windows, MacOS, improving error messages, etc.
I have a question about error reporting. You use 'error' quite often. I think that this can cause errors to pop up at strange moments during program evaluation. It this a real problem? I prefer reporting errors early in the IO monad. I think there is some trade-off involved, but I can't name it now.
Here's a list of common Unix idioms and how I implement them:
1) Interpreting the filename "-" as stdin or stdout
I use this function (and a similar function for reading input).
[Trivial detail: I use pretty printing for all I/O in my programs.]
Interesting approach, you keep compositionality of Doc up to the last moment. It certainly helps to make program's output look prettier.
-- | -- Write to file ("-" means stdout) writeOutput :: FilePath -> Doc -> IO () writeOutput "-" output = do printDoc PageMode stdout output writeOutput outfile output = do h <- openFile outfile WriteMode printDoc PageMode h output hClose h
2) Treating arguments of the form 'VARNAME=VALUE' like environment variables (cf. GNU make) and 3) Printing usage info for malformed command lines
Shouldn't be too difficult to adapt to my technique. I've got to make some name for it :)
4) An option can be specified 0 or 1 times:
Filter options using this function
-- | -- Extract value from a list of length at most one. uniqueWithDefault :: String -> a -> [a] -> a uniqueWithDefault what d [] = d uniqueWithDefault what d [a] = a uniqueWithDefault what d _ = error $ "At most one " ++ what ++ " may be specified"
I used to ignore superfluous options, but I agree that reporting this would be nicer. In my approach you can still use your technique, if you build a list of values for an option. data Options = Options { optOutfile :: [String] , ... } startOpts :: Options startOpts = Options { optOutfile = [] , ... } options :: [ OptDescr (Options -> IO Options) ] options = [ Option "o" ["output"] (ReqArg (\arg opt -> return opt { optOutfile = arg : optOutfile out }) "FILE") "Output file" , ... You could also use some other Monoid, for example one that explicitly discourages more than one value: data T a = Zero | One a | Many instance Monoid (T a) where mempty = Zero mappend Zero x = x mappend (One a) Zero = One a mappend _ _ = Many BTW. What would be a good name for T?
5) An option must be specified exactly once:
Same as above. I would love to write more, but I have to go to work (where I mostly struggle with C++ :( but occasionally manage to smuggle Haskell :)) Best regards, Tomasz -- .signature: Too many levels of symbolic links
I have a question about error reporting. You use 'error' quite often. I think that this can cause errors to pop up at strange moments during program evaluation. It this a real problem? I prefer reporting errors early in the IO monad. I think there is some trade-off involved, but I can't name it now.
You're right, it can lead to late error messages. For example, if two output files are specified then the program might read its input, spend some time processing and only report an error after some considerable time has passed. (I haven't actually seen this happen but I'm sure it would.) One reason for using error in functions like 'uniqueNoDefault' (which checks that a list has precisely one element and either returns it or prints a useful error message) is that I use this function both from the IO monad and from pure code. I'm reluctant to duplicate the code just to avoid this. But maybe I should put it in a monad anyway (and go back and 'fix' all non-monadic uses)? The error messages produced are basically telling the user that they made a mistake so I must have just read some input from a file, the command line or the console. (Another issue with error reporting is that I should probably print the 'usage' message whenever flags are incorrectly omitted, duplicated, etc. This too suggests that tying the checks more tightly into command line parsing (as you do) would be a good idea.) Thanks for your comments. -- Alastair
I have a question about error reporting. You use 'error' quite often. I think that this can cause errors to pop up at strange moments during program evaluation. It this a real problem? I prefer reporting errors early in the IO monad. I think there is some trade-off involved, but I can't name it now.
You're right, it can lead to late error messages. For example, if two output files are specified then the program might read its input, spend some time processing and only report an error after some considerable time has passed. (I haven't actually seen this happen but I'm sure it would.)
One reason for using error in functions like 'uniqueNoDefault' (which checks that a list has precisely one element and either returns it or prints a useful error message) is that I use this function both from the IO monad and from pure code. I'm reluctant to duplicate the code just to avoid this.
But maybe I should put it in a monad anyway (and go back and 'fix' all non-monadic uses)? The error messages produced are basically telling the user that they made a mistake so I must have just read some input from a file, the command line or the console.
I'm not certain this applies, but it should be possible to force evaluation order with a technique similar to deepSeq. It might be cleaner than using IO. http://www.haskell.org/pipermail/haskell/2001-August/007712.html Cheers, JP. __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus
I'm not certain this applies, but it should be possible to force evaluation order with a technique similar to deepSeq. It might be cleaner than using IO.
I think in this case, I'd prefer to use the IO monad. 1) It keeps the sequencing very, very explicit and not likely to be confused with a strictness annotation. seq is more about performance than semantics and, although all current Haskell compilers happen to impose some sequencing on the evaluation order of seq's arguments, future compilers could break that property and still be semantically correct. 2) To use seq, I have to link the evaluation of the error check to the evaluation of something else. It's not quite clear what a good choice would be. Point #1 is my main reason. -- Alastair Reid www.haskell-consulting.com
(Reply-To: haskell-cafe) Alastair Reid <alastair@reid-consulting-uk.ltd.uk> writes:
I have a question about error reporting. You use 'error' quite often. I think that this can cause errors to pop up at strange moments during program evaluation.
You're right, it can lead to late error messages. For example, if two output files are specified then the program might read its input, spend some time processing and only report an error after some considerable time has passed. (I haven't actually seen this happen but I'm sure it would.)
I'm not sure this brings anything new on the table, but after this thread I went back and reworked some of my option handling code. Hopefully it will either be an inspirational example, or provide me with feedback on my grave errors and mistakes :-) here it is: ------------------------------------------------------------
-- Args is a record for the arguments, easily accessible by field -- in the program proper
data Output = G | C | X deriving (Read,Eq) data Args = Args { kval :: Int, output :: Output, writer :: String -> IO (), parser :: Fasta.FHParser}
-- usage prints using "error", this is perhaps Not Nice? (it -- makes it rather difficult to print usage and exit with success, -- should that happen to be important)...
usage :: [String] -> a usage errs = error (usageInfo (concat errs ++ "\nUsage: xegen -k <kval> [-u] -{G|C|X} [-o FILE] <filename>\n") options)
-- ...but since usage has polymorphic type, I can use it to -- initialize required fields in the default argument struct:
defaultArgs = Args { kval = usage ["You must specify a k value"] , output = usage ["Please specify -K, C, or X"] , writer = putStr, parser = bmfparser }
-- mkK is used to parse the argument string for the -k option -- which needs to be an integer. This is IMHO superior to getting an -- anonymous "Prelude.read: no parse" message
-- add a k value to p mkK :: Args -> String -> Args mkK p s = p {kval = if (and $ map isDigit s) then read s else usage ["The k value must be an integer"]}
-- and here's the table of options
options :: [OptDescr (Args -> Args)] options = [ Option ['k'] ["word-size"] (ReqArg (\s p -> mkK p s) "INT") "Word size" ,Option ['G'] ["graph-output"] (NoArg (\p -> p { output = G })) "Output the graph in GraphViz format" ,Option ['C'] ["consensus-output"] (NoArg (\p -> p { output = C })) "Output assembled consensus sequences" ,Option ['X'] ["exons-output"] (NoArg (\p -> p { output = X })) "Output the (concatenated) exons" ,Option ['u'] ["input-upper"] (NoArg (\p -> p { parser = ugparser })) "accept only upper case characters" ,Option ['o'] ["output"] (ReqArg (\s p -> p { writer = writeFile s }) "FILE") "output file name (default is stdout)" ]
-- comments are most welcome!
(Another issue with error reporting is that I should probably print the 'usage' message whenever flags are incorrectly omitted, duplicated, etc.
I think that's achieved in the above. And I think the late error message problem can be solved by making sure the Args struct is entirely evaluated before any "real code" is run. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
participants (4)
-
Alastair Reid -
JP Bernardy -
Ketil Malde -
Tomasz Zielonka