Hi everyone, I have been working on an initial benchmarking suite for generic programs, based on the programs that Stephanie, James, Johan and others have committed to the repository. Also most of the ideas here have been collected from the discussion that was carried out in the list, so thanks to all! We plan to be use this as the starting point for the suite that will allow us to compare the different approaches. The main ideas of this benchmarking suite are the following: * The benchmark suite consists of a set of tests in the form of generic programs, i.e. gshow/gread, gmap, geq, etc. * Each test consists of a set of modules written by different users, i.e. library writers, power users and end users. * Each test tries to solve a particular problem for the end-user, using the different generic programming approaches. * The end-user code for a test (the application of a specialized generic function to a data type) is the same across approaches. API differences (e.g. class based approaches vs. type representation argument) would be hidden behind a wrapper that the end-user code will use. * The rest of the code (generic function definition) for a test usually varies across approaches. * There will be a test per generic function proposed in the mailing list. However we will not include tests that give the same information that a previous test already does for all approaches. Such a test cannot be used to distinguish approaches. For example a generic zip test won't be included if it doesn't say anything more than a generic map. * The comparison results will use the wiki template. I would like to emphasize the decision of using the same end-user code in the different versions of a test. This ensures that, for example, all variants of equality are really doing the same thing for the end-user. This becomes more useful when you consider Stephanie's foldTree example versus the reduce function. Though they feel similar, these functions would be used in different ways by the end user since they have different ways to specialise their behaviour. This suggests that these two functions should not be compared against each other, instead they are two different test cases. I think that requiring the reuse of end-user code gives a nice criterion for telling which things are comparable, and it will be useful should we become interested in comparing performance as well. The tests that will become part of the suite are: * gshow/gread * greduce (generic reduce function) * foldTree * raise aka paradise aka update * generic map * generic equality * serialization (though similar to gshow/gread, the output is influenced by the data type structure, this might tell interesting things). * gmin (though I am not sure on how much more than gread this tells) The tests that will not become part of the suite are: * generic zip. Generic map is already a function that is defined on kinds different than * , and generic equality already has two arguments (difficult to define in some approaches). Tests I plan to look into: * Oleg's int2float You can get a "branch" of the generics repository by typing
The testsuite is in the "comparison" directory. To run the tests, type:
runghc tests.hs
The end-user code/drivers are in "comparison/TestGEq", "comparison/ TestGShow", etc. And the approach varying code is in "comparison/LIGD/", "comparison/ SYB1_2", etc. At the moment the SYB gshow code prints parenthesis around nullary constructors so you will get a failure since LIGD doesn't. This is something that the original code already did and I plan to fix it. I guess that the best license would be the BSD license that Manuel proposed, if the original authors agree, of course. We would like to push it to the official repository eventually, but meanwhile we would like to hear your comments. Cheers, Alexey