On Tuesday 14 May 2002 07:22 pm, Andrew J Bromage wrote:

On the other hand, it's an exciting time to do engineering in declarative languages, because we can invent the design patterns and discover what the good habits are as we go along.

This is very interesting to me, as I have a great deal of experience applying and documenting patterns in object-oriented environments. From the research I've done to date, functional programming provides enough of a paradigm shift that there are significant new patterns/idioms (you choose the lingo) that need to be documented.* I can think of several off of the top of my head: - Tying the knot - Phantom types - Appropriate use of strictness in data structures - The excellent techniques used in Edison for parameterization Etc. With that in mind, I'd be interested in starting such a documentation project. And by this I mean something more formal than the existing Wiki, perhaps in hyperlinked/book form. In this way, you could download a bunch of patterns with good example code, and get a head start on understanding proven approaches for structuring systems, etc. So, if you have any patterns, or better yet, source code that demonstrates an approach or technique, please send them my way. I will put something together on this in the next couple of weeks, and let everyone know where to find it. And, if there are no objections, I'll probably harvest some from the Wiki and various other papers that have them buried beneath formalisms. :) If anyone has any suggestions, I'd love to hear them. Regards, - j * To avoid any pattern debates: I personally believe that patterns are especially useful for getting novices up to speed on concepts that might not be readily apparent at first glance. Rather than treating patterns as anything "special" in their own right, for me they're simply a really convenient way of teaching people new and interesting concepts. -- The river is moving. The blackbird must be flying.

...

However, it appears that the only place (short of Ericsson) I can actually work on a complex functional system is in academia. Unfortunately, this is not an option, as I have no Ph.D., and going back to school is probably not realistic.

The latter depends on your background. With a first degree and solid experience in FP and software engineering, you might be just the right fit for some research associate positions (if you are willing to think about academic salaries for the serious work involved in getting a PhD..). Academia is investigating the implications of FP being put to practical use, and for that, we need input from people with strong practical background. A few weeks ago, we at UKC were looking for someone to investigate refactoring in functional languages (now taken). Heriot-Watt is looking for someone to pit Glasgow distributed Haskell against Erlang in a Motorola-sponsored project (see recent Haskell Communities Report for info and links).

There are other companies using Erlang, if not on as large a scale. The Erlang User Conference (proceedings online at erlang.org) is a good starting point to find out which. Some of the best stories, though, are from very small Erlang groups in companies mainly using something else...

If we can trust the recent download statistics, there have been 10.000 (!) downloads of the newest Erlang opensource release (that is, not counting the paying customers). And yes, Erlang has been used by other companies, see, e.g.: http://dmoz.org/Computers/Programming/Languages/Erlang/Products/

I would guess the biggest Haskell company is Galois Connections Inc (galois.com), although I know there are others. Funny there's no "Haskell in Industry" section on haskell.org -- it might be small, but it wouldn't be empty, if people using Haskell were willing to stand up and be counted.

As you say, there are lots of others, they just need to be willing to stand up and be counted (repeating a good suggestion sometimes helps;-). Meanwhile, we have made a start in the Haskell Communities Reports. http://www.haskell.org/communities/ The first edition had a longer section on Galois, this edition has a longer section on Xilinx. Both editions list Haskellers who have come forward to tell us what non-Haskell-related things they do with Haskell in the applications chapter. We only have a small sample, but the variety of applications is as encouraging as the confidence displayed in applying Haskell to complex problems. I've been trying to suggest to other fpl communities (Clean, Erlang, ML,..) that they produce similar activity reports. Joe Armstrong has just this week volunteered to organize such an effort for the Erlang community. Such reports help to document what, IMHO, has been a significant change in FP programmers attitude: we no longer wonder why "they in industry" don't take up FP, more and more of us simply start to use FP where it helps us. Btw, I wouldn't subscribe to Andrew's opinion that "there isn't a lot of functional (or even declarative) software engineering experience out there.". FP has been in the making for quite some time now, and all the time it didn't make it into mainstream, we have been "losing" FP people to industry. A few decades ago, universities had to send some of their lecturers on courses if they wanted them to teach FP; nowadays FP-aware students leave academia in the 100s (well, 1000s in some parts of Australia;-). Even if you assume that most of them forget what they learned in less than a year, and even if you completely ignore the LP figures, those numbers keep adding up. The ground is better prepared than ever. It remains up to you to decide whether you're confident enough to use FP, without needless hype, and just for the many things you know it can do well. Cheers, Claus

On Wed, 15 May 2002, Scott Finnie wrote:

As a naive but interested newbie, I'm very keen to understand those things that FP does well - and just as importantly, those things it doesn't. (I'm coming at this from use in an industrial context). Based on (_very_) limited experience so far, I get the feeling that:

. FP is well suited to transformation-type problems - i.e. those that can usefully be viewed as transforming some input to a required output. . FP - or at least Haskell - is not so well suited to reactive, event driven, parallel systems: i.e. those that can usefully be viewed as a CSP system.

In my personal experience it's not really specific things but rather in the more general area that: (1) FP generally makes it less painful to write higher level code (especially generic, reusable code), particularly because it of the ease of combining things using higher order functions, both from the standard prelude and designed for the particular domain the program is working in. Higher level code is generally less difficult (note not `easy') to write in the first place, understand and modify drastically. (2) FP is generally not more helpful than other approaches if you're trying to get close to optimal performance by ensuring that nothing redundant happens. (Redundant in the sense that without optimization "map f . map g" applied to a list will build a new intermediate list, including the `links' when applying the g only to break it apart again to apply the f; I know that the timing is actually more complicated due to laziness but it illustrates the point.) The key to using deciding when it's FP would be an appropriate choice lies in being able to honestly judge whether the program you are working on will benefit more from (1) than it will suffer from (2). I don't work in industry but rather am an academic researcher with a different field of research (i.e., I don't get any `publication credit' for doing things using a new FP-based technique rather than an existing one) and I use Haskell for: (1) Prototyping image-processing algorithms on toy data. When my initial ideas are in heavy flux and I'm trying to figure out if they will work at all, writing Haskell implementations on artificial data means I don't waste time coding in imperative-language detail things that won't work anyway. Performance issues & integration hassles mean I've always written C++ code before being able to try candidate algorithms on real (LARGE) data sets. (2) Scripting and small applications. For example, I've got a script that does some really nasty, tortous processing to build Makefiles that takes a couple of minutes to run under Hugs when I'm sure it would run in a couple of seconds if I rewrote it in C++. But I only run it a couple of times a day, and the ease of writing and modification far outweigh the slower running time.

Please don't flame if this is off-base, I'm trying to get a handle on things. This is based primarily on Haskell; I realise Erlang's primary domain is telecoms, which implies it does address the second category. Assuming that's so, are there extra concepts in Erlang that make it suitable for such problems?

There are certainly some different ideas in Erlang but I think the primary reason it's used is that it was developed in a research lab somewhere in Ericsson. ___cheers,_dave_________________________________________________________ www.cs.bris.ac.uk/~tweed/ | `It's no good going home to practise email:tweed@cs.bris.ac.uk | a Special Outdoor Song which Has To Be work tel:(0117) 954-5250 | Sung In The Snow' -- Winnie the Pooh

On Wed, 15 May 2002, Karl-Filip Faxen wrote:

On the performance (or not) of high level code: I'm working on a compiler with a strong emphasis on generating good code for

I wish you luck!

It is going to be interesting to see how much this will give. I suspect that part of the performance problems of (lazy) functional languages come from their encouraging the programmer to use linked data structures rather than arrays and similar biases, rather than just from overhead.

I tend to agree. I keep meaning for experimental purposes to define a list type called AList or something which is syntactically identical to lists (i.e., you can use the familiar (:) and [] operators/sugar), but gets preprocessed out as actually being implemented with an array with a pointer to the "current" element. Especially if we use unboxed types for such a thing, I imagine that on many applications this would give a boost in performance. I'm almost "mad" at the prelude for including so many functions which basically treat lists in ways they weren't meant, for instance, lookup, !!, etc. lookup is more suited to FiniteMap and !! is more suited to arrays. However, the syntactic sugar supporting lists is *so* strong that I find myself often using them because it's much easier to write functions using (:) and [] and especially [a,b,c] notation than it is to write functions using `addToFM` and emptyFM and the like. Perhaps if I (or someone else) gets around to writing a List->Array converter, I (or they) could apply some hueristics to see which lists (i.e., ones with lots of !!s or backwards traversals) should be implemented as Arrays and which (i.e., ones with lots of lookups) should be implemented as FiniteMaps*... Just some thoughts... - Hal * I believe such things aren't unheard of -- I don't know the name of it, but I know one of my friends up at MS Redmond was working on compilation where the compiler would determine optimal data structures to use by examining code...with typeclasses and all, I think this would be a very natural thing to see in Haskell (or a Haskell derivative).

Hi! The listlessness stuff is a precursor to deforrestation, so it replaces lists by (fewer lists and) scalars. I do not think there are any arrays involved. The most important property for being able to convert a list to an array is that there should never be two cons cells with the same tail (cdr). A somewhat stronger requirement is that the spine of the list can not be shared. Then there are the representation oriented conditions: For each (:), [] and case over lists in the program, it has to work over arrays or over linked lists or else the linked list representation and the array representation must be distinguishable at run-time. That's about it. But what I really meant is, if I may rephrase it, that imperative programs might often be both faster and harder to write because they embed more information about the abblication domain. That is, if you code in C and want an array, you must specify its size, so you have to think about your program and figure out that you only need 'x' items here, wheras in Haskell you'd use a list and never have to think about what the upper bound on the length of the list is. Cheers, /kff

Bjorn Lisper:

...sometimes the length of a list being returned from a function can be a simple function of the function arguments (or the sizes of the arguments), think of map for instance. In such cases, a static program analysis can sometimes find the length function. If we know thee functions for all list-producing functions in a closed program, then the lists could be represented by arrays rather than linked structures.

I know Christoph Herrmann worked on such a program analysis some years ago. Also, I think Manuel Hermenegildo has done this for some logic language.

Andrew Appel wrote something about "pointer-less" lists as well. What bothers me quite strongly is the algorithmic side of operations upon such objects. Typical iterations map- (or zip-) style: do something with the head, pass recursively to the tail, would demand "intelligent" arrays, with the indexing header detached from the bulk data itself. The "consumed" part could not be garbage collected. In a lazy language this might possibly produce a considerable amount of rubbish which otherwise would be destroyed quite fast. The concatenation of (parts of) such lists might also have very bad behaviour. Can you calm my anxiety? Jerzy Karczmarczuk

G'day all. On Wed, May 15, 2002 at 12:53:30PM +0100, Claus Reinke wrote:

Btw, I wouldn't subscribe to Andrew's opinion that "there isn't a lot of functional (or even declarative) software engineering experience out there.".

Just to clarify: I meant to emphasise the _declarative_ part rather than the functional part. There are plenty of LISP engineers, for example. However, I stand by the rest of it, as far as crude generalisations go. There's a lot of functional _programming_ happening, but not very much _engineering_. I hope everyone understands the difference. Cheers, Andrew Bromage