The Future of MissingH

older
Lazy data from pipe using MissingH...

John Goerzen

24 Nov 2006 24 Nov '06

11:32 a.m.

Hi, Josef Svenningsson posted a comment on my blog today that got me to thinking. He suggested that people may be "intimidated by the size of MissingH, confused by the undescriptive name, and don't quite know what's in there." And I think he's right. I've been passively thinking about what MissingH should be for awhile now, and wonder if you all would have some suggestions. First, let me back up and describe a bit of what MissingH is. It gets its name because I think that it has useful functions that are "missing" from the standard Haskell libraries. But I can see how it's not very descriptive. The major features are: * A general-purpose modular logging infrastructure * A virtual filesystem component (similar to VFS in Gnome, but written in Haskell) * A configuration file parser, compatible with Python's and Perl's, which can also modify and generate config files and do variable interpolation * A bunch of tools for setting up pipes to/from processes * Tools to format numbers as quantities, with default configs for the SI system (1000-based) and binary system (1024-based) * Tools to track the progress of long-running actions and display a status bar on the terminal * Pure-Haskell un-gzip code, *DBM infrastructure, etc * Assorted list, string, regexp, bit, pointer, etc. utilities * MIME and email parsers. I wrote most of the code myself, but did pull some large chunks from others where licenses were compatible. The API reference is at http://gopher.quux.org:70/devel/missingh/html/index.html So, some of the questions are: Should this all be one library? I initially wrote it that way to make resolving dependencies easier for end users. Also, it still has utility because modules make use of each other's features. The list and string functions are used almost everywhere, for instance. HVFS support is also fairly pervasive. Should the module naming scheme be changed? Modules are currently named things like MissingH.List, MissingH.IO.HVFS, MissingH.ProgressMeter, etc. I guess these could go to System.ListExt, System.IO.HVFS, System.Console.ProgressMeter, etc. Could, and should, any of this be integrated into the Haskell libraries project? (That set of libraries formerly known as fptools) A few bits of code have already found their way into them. But I bet there could be a lot more. Maybe even the majority of it. How could greater community participation be encouraged, while still encouraging quality control? I have received some very good contributions to MissingH from people, and that's been great. I've also received some that just aren't that great -- they don't have Haddock docs, the code is opaque, they don't come with unit tests, etc. But by and large, I've been maintaining it mostly myself. Whenever I write some code, I think about whether this is generic code that could be useful in other projects. If so, I consider whether it would be appropriate for MissingH (and it usually is). It would be wonderful if others would be interested in contributing code that solves some need as well. There is a public darcs repo already, and I have received some darcs patches from people (thanks!) I'm not trying to rid myself of MissingH, but I think that it could be a nice resource for small tools from the entire community, if we can maintain a structure and QA footprint to it. What else should be done to make this a valuable resource for Haskell programmers? And a showcase for what is possible with Haskell? The floor's open... Thanks, -- John

Show replies by date

Malcolm Wallace

24 Nov 24 Nov

12:14 p.m.

John Goerzen wrote:

...

The major features [of MissingH] are:

* A general-purpose modular logging infrastructure * A virtual filesystem component (similar to VFS in Gnome, but written in Haskell) * A configuration file parser, compatible with Python's and Perl's, which can also modify and generate config files and do variable interpolation * A bunch of tools for setting up pipes to/from processes * Tools to format numbers as quantities, with default configs for the SI system (1000-based) and binary system (1024-based) * Tools to track the progress of long-running actions and display a status bar on the terminal * Pure-Haskell un-gzip code, *DBM infrastructure, etc * MIME and email parsers.

Each of those bullet points sounds to me like a useful individual package in its own right. Certainly I had no idea that MissingH contained many of these things, and if I had wanted them, the single name MissingH would not have suggested that I look there. I think they could profitably be split off.

...

* Assorted list, string, regexp, bit, pointer, etc. utilities

By contrast, I guess these parts are what I always believed the "MissingH" name to refer to. They are in some sense missing from the original Haskell'98 libraries. But, given that the evolving 'base' package is now pretty-much the de facto standard, rather than the immutable 'haskell98' package, I see no reason for these functions to remain separate from it. There is a community process for adding new functions to the 'standard' libraries. Use it! Let these functions be "missing" no more! If they are truly useful, they will be adopted without too much fuss. (Note, when I say 'base', I might actually mean 'regex-*' or something else. I don't want to prejudge where these things live, just to encourage you to propose them to the whole Haskell community.)

...

I initially wrote it that way to make resolving dependencies easier for end users. Also, it still has utility because modules make use of each other's features. The list and string functions are used almost everywhere, for instance. HVFS support is also fairly pervasive.

With Cabal packages, you can express dependencies on other packages with ease. Provided it is not a rats nest of mutual recursion, this is the way to go.

...

Modules are currently named things like MissingH.List, MissingH.IO.HVFS, MissingH.ProgressMeter, etc.

I guess these could go to System.ListExt, System.IO.HVFS, System.Console.ProgressMeter, etc.

In general, yes. [ The bikeshed approacheth, but System.ListExt is not very descriptive - what is wrong with proposing additions to Data.List itself? ]

...

Could, and should, any of this be integrated into the Haskell libraries project? (That set of libraries formerly known as fptools)

Where your packages are stored, and how they are maintained, is a separate decision from the choices (a) to use the standard packaging mechanism, and (b) to divide the monolith up into smaller, more manageable chunks.

...

How could greater community participation be encouraged, while still encouraging quality control?

For quality, just maintain your own public darcs repositories. You get to decide which patches to apply and which to reject. For greater community participation, I think the simple strategy of splitting the libs up into smaller, more grok-able packages is the important first step. Publicity about what each package is good for is the next. Then, if you get significant numbers of users for these libraries, patches will arrive. :-) Regards, Malcolm

Neil Mitchell

12:26 p.m.

Hi John,

...

Should this all be one library?

No, several smaller libraries would be nice.

...

Should the module naming scheme be changed?

Yes, MissingH is not the place to put these things. You run the risk of more name clashes, but thats ok.

...

Could, and should, any of this be integrated into the Haskell libraries project?

Yes, some should, but only after careful review and being split up. Assuming the Haskell libraries project is still something that should be continued, once Hackage is up and running suddenly the distinction becomes minimal. A few of the functions should be picked out and moved into base, where they truely are missing.

...

How could greater community participation be encouraged, while still encouraging quality control?

Make sure more people know about it. I want to start using MissingH but never got around to it, by continually telling me how great it was I'm sure I would have started using it earlier. I have various bits of general libraries strewn around the place, which I might contribute to MissingH. Thanks Neil

Neil Mitchell

12:38 p.m.

...

...
How could greater community participation be encouraged, while still encouraging quality control?

One thing I forgot, serving the web pages and documentation over port 80 using HTTP would help. Not all people can access port 70 from all places, I can't from the university campus (only from computer science). Thanks Neil

Neil Mitchell

1:17 p.m.

...

...
...
How could greater community participation be encouraged, while still encouraging quality control?

It also took me quite a while to find the darcs repository, and as far as I can see there is no web page on what MissingH has in it, other than a textual readme and the GNU entry. If there was a single web page which said what MissingH was and where it could be found, that would help people. The darcs repo I found was: http://darcs.complete.org/missingh/ Thanks Neil

John Goerzen

3:22 p.m.

On Fri, 24 Nov 2006 18:17:06 +0000, Neil Mitchell wrote:

...

Hi

...
...
...
How could greater community participation be encouraged, while still encouraging quality control?

It also took me quite a while to find the darcs repository, and as far as I can see there is no web page on what MissingH has in it, other than a textual readme and the GNU entry. If there was a single web page which said what MissingH was and where it could be found, that would help people.

Your comments about the website are right on. I have been shoving stuff at my Gopher server for quite awhile now. Ironically, I tried to switch the whole mess to trac+darcs about a month ago, but trac crashes on the MissingH darcs repo. Go figure. I talked to Lele about it but he was really short on time, and I didn't have time to learn the trac code either.

...

The darcs repo I found was: http://darcs.complete.org/missingh/

That's the one. -- John

Simon Marlow

28 Nov 28 Nov

7:05 a.m.

Neil Mitchell wrote:

...

...
Could, and should, any of this be integrated into the Haskell libraries project?

Yes, some should, but only after careful review and being split up. Assuming the Haskell libraries project is still something that should be continued, once Hackage is up and running suddenly the distinction becomes minimal.

A few of the functions should be picked out and moved into base, where they truely are missing.

Yes please! This is what I hoped would happen to MissingH eventually. Cheers, Simon

Jason Dagit

24 Nov 24 Nov

3:37 p.m.

On 11/24/06, John Goerzen wrote:

...

What else should be done to make this a valuable resource for Haskell programmers? And a showcase for what is possible with Haskell?

I was going to try MissingH on win32 but when I did it refused to compile due to a dependency on, I think, Posix. It would be great if this could be fixed/relaxed. At the time I was looking forward to trying the logging facilities. Jason

John Goerzen

4:47 p.m.

On Fri, Nov 24, 2006 at 12:37:10PM -0800, Jason Dagit wrote:

...

On 11/24/06, John Goerzen wrote:

...
What else should be done to make this a valuable resource for Haskell programmers? And a showcase for what is possible with Haskell?

I was going to try MissingH on win32 but when I did it refused to compile due to a dependency on, I think, Posix. It would be great if this could be fixed/relaxed. At the time I was looking forward to trying the logging facilities.

That should be completely fixed for a long time now. But I will admit I haven't tried to build the latest version on GHC 6.6. In fact, I wrote an entire module (MissingH.IO.StatCompat) so that Windows users can also enjoy HVFS. -- John

Vyacheslav Akhmechet

5:42 p.m.

Until this email I was under the impression that the project is dead. For example, if I go to google and type 'MissingH' the first link is fsf's directory page. When I try to get to MissingH website from there the link appears to be down. I can't really figure out what MissingH includes and where to get it from searching the web, not easily anyway. I think fixing this will go a long way to a more widespread adoption of the library. Thanks, - Slava Akhmechet On 11/24/06, John Goerzen wrote:

...

On Fri, Nov 24, 2006 at 12:37:10PM -0800, Jason Dagit wrote:

...
On 11/24/06, John Goerzen wrote:

...
What else should be done to make this a valuable resource for Haskell programmers? And a showcase for what is possible with Haskell?

I was going to try MissingH on win32 but when I did it refused to compile due to a dependency on, I think, Posix. It would be great if this could be fixed/relaxed. At the time I was looking forward to trying the logging facilities.

That should be completely fixed for a long time now. But I will admit I haven't tried to build the latest version on GHC 6.6.

In fact, I wrote an entire module (MissingH.IO.StatCompat) so that Windows users can also enjoy HVFS.

-- John

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Bulat Ziganshin

26 Nov 26 Nov

2:54 p.m.

Hello John, Friday, November 24, 2006, 7:32:55 PM, you wrote:

...

Josef Svenningsson posted a comment on my blog today that got me to thinking. He suggested that people may be "intimidated by the size of MissingH, confused by the undescriptive name, and don't quite know what's in there." And I think he's right.

first, is it possible to integrate MissingH inside existing core libs, i.e. Haskell libs supported by Haskell community? i think that it will be impossible if MissingH will hold its GPL status. i think that such fundamental library as MissingH should be BSDified to allow use it both in commercial and non-commercial code if library will be not BSDified it can remain as a whole or be splitted to smaller conceptual parts (strings, file system, logging...) but these parts should remain separate from other haskell libs and these features can't be made, say, part of a Haskell standard if library will be BSDified, and somewhat "advertized". i hope that its parts will start moving to the more specific libs of core set, say HVFS system into the Files library, logging facilities into the Unix library, so on next. why your library isn't well recognized. i can suggest in each announce of new library version write the full list of its features or at least url to such advertizing page. second, are you included your library in HCAR and hswiki/Libraries_and_tools pages? third, while i personally prefer to read source code and fascinated with quality of code documenting in your lib, most peoples prefer to read Haddocks, which again should be made available on web next, while you accept patches to the lib, this's not declared in your announces. best way is just to open darcs repository - most peoples thinks that having darcs repository and accepting patches is the same thing :) i can also propose you the idea that Pupeno, packager of Streams library used - he included in the tgz files copy of darcs repository, again facilitating use of darcs and developing new patches for library and, about WindowsCompat.hs - stat() function is available on Windows and even used to implement getModificationTime :)

...

I initially wrote it that way to make resolving dependencies easier for end users.

now Cabal handles this

...

How could greater community participation be encouraged, while still encouraging quality control?

...

I have received some very good contributions to MissingH from people, and that's been great. I've also received some that just aren't that great -- they don't have Haddock docs, the code is opaque, they don't come with unit tests, etc.

...

But by and large, I've been maintaining it mostly myself.

i think that this is more general question for all haskell core libraries -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Dougal Stanton

3:32 p.m.

Quoth Bulat Ziganshin, nevermore,

...

while i personally prefer to read source code and fascinated with quality of code documenting in your lib, most peoples prefer to read Haddocks, which again should be made available on web

I just thought I should point out, cos there appears to be some confusion on the matter, that there is a Haddock API for MissingH, which I've found to be very informative. http://quux.org:70/devel/missingh/html/index.html As Neil Mitchell (I think) mentioned earlier it's not on a standard HTTP port which may be unsuitable for some people. Cheers, D.

Benjamin Franksen

5:46 p.m.

Bulat Ziganshin wrote:

...

Friday, November 24, 2006, 7:32:55 PM, you wrote:

...
Josef Svenningsson posted a comment on my blog today that got me to thinking. He suggested that people may be "intimidated by the size of MissingH, confused by the undescriptive name, and don't quite know what's in there." And I think he's right.

first, is it possible to integrate MissingH inside existing core libs, i.e. Haskell libs supported by Haskell community? i think that it will be impossible if MissingH will hold its GPL status. i think that such fundamental library as MissingH should be BSDified to allow use it both in commercial and non-commercial code

I hate to be nitpicking but GPL is not only compatible with but encourages commerce in general and commercial software in particular. It is incompatible with proprietary software. There's a difference. Cheers Ben

Max Vasin

27 Nov 27 Nov

2:04 a.m.

...

...
...
...
...
"Benjamin" == Benjamin Franksen writes:

Benjamin> Bulat Ziganshin wrote:

...

...
Friday, November 24, 2006, 7:32:55 PM, you wrote: Josef Svenningsson posted a comment on my blog today that got me to

...
thinking. He suggested that people may be "intimidated by the size of MissingH, confused by the undescriptive name, and don't quite know what's in there." And I think he's right.

first, is it possible to integrate MissingH inside existing core libs, i.e. Haskell libs supported by Haskell community? i think that it will be impossible if MissingH will hold its GPL status. i think that such fundamental library as MissingH should be BSDified to allow use it both in commercial and non-commercial code

Benjamin> I hate to be nitpicking but GPL is not only compatible Benjamin> with but encourages commerce in general and commercial Benjamin> software in particular. It is incompatible with Benjamin> proprietary software. There's a difference. A small addition: some GPLed libraries (libstdc++ AFAIK) allow linking with proprietary software by adding clause to lisence which relaxes GPL requirements. -- WBR, Max Vasin.

Bulat Ziganshin

4:08 a.m.

Hello Max, Monday, November 27, 2006, 10:04:15 AM, you wrote:

...

A small addition: some GPLed libraries (libstdc++ AFAIK) allow linking with proprietary software by adding clause to lisence which relaxes GPL requirements.

a GPL license text is the same for any GPLed software, otherwise it cannot be called "GPL" :) this should be a LGPL. also there are many double-licensed libs. you can find details about this in LGPL and GPL pages of wikipedia (where i've read this) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Max Vasin

8:56 a.m.

...

...
...
...
...
"Bulat" == Bulat Ziganshin writes:

Bulat> Hello Max, Hello, Bulat> Monday, November 27, 2006, 10:04:15 AM, you wrote:

...

A small addition: some GPLed libraries (libstdc++ AFAIK) allow

...
linking with proprietary software by adding clause to lisence which relaxes GPL requirements.

Bulat> a GPL license text is the same for any GPLed software, Bulat> otherwise it cannot be called "GPL" :) this should be a Bulat> LGPL. also there are many double-licensed libs. you can Bulat> find details about this in LGPL and GPL pages of wikipedia Bulat> (where i've read this) from /usr/share/doc/libstdc++6/copyright: The libstdc++-v3 library is licensed under the terms of the GNU General Public License, with this special exception: As a special exception, you may use this file as part of a free software library without restriction. Specifically, if other files instantiate templates or use macros or inline functions from this file, or you compile this file and link it with other files to produce an executable, this file does not by itself cause the resulting executable to be covered by the GNU General Public License. This exception does not however invalidate any other reasons why the executable file might be covered by the GNU General Public License. -- WBR, Max Vasin.

Alex Queiroz

6:35 a.m.

Hallo, On 11/27/06, Max Vasin wrote:

...

A small addition: some GPLed libraries (libstdc++ AFAIK) allow linking with proprietary software by adding clause to lisence which relaxes GPL requirements.

Nope, it's LGPL with an extra binary linking permission. Cheers, -- -alex http://www.ventonegro.org/

Bulat Ziganshin

4:03 a.m.

Hello Benjamin, Monday, November 27, 2006, 1:46:34 AM, you wrote:

...

I hate to be nitpicking but GPL is not only compatible with but encourages commerce in general and commercial software in particular. It is incompatible with proprietary software. There's a difference.

of course, but on practice most of commercial software are closed-source. i personally use this license when i want to show code to the world but don't want that but will be used in commercial software (without paying royalties). my own program contains several general-purpose modules that is BSDified and rest of program, which is a task-specific, is GPLed -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Benjamin Franksen

9 p.m.

Bulat Ziganshin wrote:

...

Monday, November 27, 2006, 1:46:34 AM, you wrote:

...
I hate to be nitpicking but GPL is not only compatible with but encourages commerce in general and commercial software in particular. It is incompatible with proprietary software. There's a difference.

of course, but on practice most of commercial software are closed-source. i personally use this license when i want to show code to the world but don't want that but will be used in commercial software (without paying royalties).

This is impossible. GPL expressly allows commercial use of your software. You cannot license under GPL and at the same time disallow making money out of it, this would be incompatible. If, however, you actually meant to say 'proprietary', then I would kindly ask you to say so. Imprecise usage of the term 'commercial' as a synonym (**) for 'proprietary' only serves to promote common misconceptions. It is neither in the spirit nor the words of the GPL to be opposed to commerce (=earning money by selling work or things of value to others), and there is ample proof(*) that it isn't opposed to commerce in practice, too. And no, I don't intend to pursue this (somewhat off-) topic any further ;-) Cheers Ben (*) anyone know how much Novell paid for buying SuSE? Not that it's gotten any better since... (**) some would say 'euphemism'

John Goerzen

30 Nov 30 Nov

8:56 a.m.

Bulat Ziganshin wrote: Hi Bulat, Many thanks for the *great* comments.

...

first, is it possible to integrate MissingH inside existing core libs, i.e. Haskell libs supported by Haskell community? i think that it will be impossible if MissingH will hold its GPL status. i think that such fundamental library as MissingH should be BSDified to allow use it both in commercial and non-commercial code

As others have pointed out, the GPL is not a commercial vs. non-commercial license. That said, I am scrupulous about copyrights and licensing. I know exactly which bits of MissingH I own the copyright to, and which bits are under which license. I have, for quite some time already, maintained an LGPL branch of MissingH. This branch contains all of the code in MissingH that is: a) compatible with the LGPL b) not depending on LGPL-incompatible components That means basically the code I wrote, plus any LGPL or BSD code others wrote. It would be easy enough to figure out which bits can suitably fall under BSD license; it would be nearly the same bits as can fall under LGPL. Again, since I own copyright to most of the code, I can put it under as many different licenses as I like, so long as I respect everyone else's copyrights properly. In any case, most of the stuff that would be suitable for base was written by me anyway.

...

if library will be BSDified, and somewhat "advertized". i hope that its parts will start moving to the more specific libs of core set, say HVFS system into the Files library, logging facilities into the Unix library, so on

Planning to do so.

...

quality of code documenting in your lib, most peoples prefer to read Haddocks, which again should be made available on web

Already are, and will continue to be.

...

next, while you accept patches to the lib, this's not declared in your announces. best way is just to open darcs repository - most peoples thinks that having darcs repository and accepting patches is the same thing :) i can also propose you the idea that Pupeno, packager of

Have it, but it's under-documented. That will change. See http://software.complete.org/offlineimap/ for an example of what I intend to do with MissingH as well.

...

Streams library used - he included in the tgz files copy of darcs repository, again facilitating use of darcs and developing new patches for library

That is an interesting idea, but the MissingH repo has nearly 1000 darcs patches by now. This would seriously bloat the tarball, plus it's easy enough to just download it off the 'net with darcs.

...

and, about WindowsCompat.hs - stat() function is available on Windows and even used to implement getModificationTime :)

Err, how? Is this new in ghc 6.6? Last I tried, -package unix wouldn't even work on Windows.

...

...
I initially wrote it that way to make resolving dependencies easier for end users.

now Cabal handles this

No, it just complains when dependencies aren't resolved. People still have to go out and download/install each piece manually. -- John

6746

Age (days ago)

6752

Last active (days ago)

List overview

Download

19 comments

11 participants

participants (11)

Alex Queiroz
Benjamin Franksen
Bulat Ziganshin
Dougal Stanton
Jason Dagit
John Goerzen
Malcolm Wallace
Max Vasin
Neil Mitchell
Simon Marlow
Vyacheslav Akhmechet