RE: new Library Infrastructure spec.

Simon Peyton-Jones

2 Jun 2004 2 Jun '04

3:49 a.m.

| > I like the simplicity but would also like the spec | > to make it easy for me to guarantee that that I | > don't end up running/installing malware. | > | > I think Haskell's typesystem and purity should | > make it relatively easy to make sure that: I don't think so, alas. The IO monad lets you do *anything*, and of course Setup.lhs runs in the IO monad, else it would not be able to move files or run a compiler. So I'm not optimistic. Perhaps a package whose Setup.lhs did nothing but import Distribution.Simple (which you perhaps trust) would be more trustworthy than a big pile of goop. But remember that you are installing a library that you will later (presumably) run, and that might be bad too. I'm not optimistic here. Simon

Show replies by date

S. Alexander Jacobson

2 Jun 2 Jun

2:22 p.m.

New subject: new Library Infrastructure spec.

On Wed, 2 Jun 2004, Simon Peyton-Jones wrote:

...

But remember that you are installing a library that you will later (presumably) run, and that might be bad too.

We have two different issues here: the safety of *running* a library and the safety of *installing* a library. Obviously, if I run a lib in the IO monad without a sandbox, I am totally at risk of whatever the code does (but see below) However, if I run a library outside the IO Monad (and I can be certain that the lib does not call unsafeperformIO) then I have a high level of confidence in the safety of running the code. And, if that lib also comes with quickcheck/hunit/proof code, then I can have a really high level of confidence in using the code almost immediately. Isaac Jones observed that unless you and the other compiler authors say that it is easy/possible to make it safe to *run* libraries, there is no point in trying to make it safe to *install* them. So, the questions on the table for you is whether there is an easy way of guaranteeing that non IO Monad code does not do unsafeperformIO. If ensuring safety of non IO Monad code IS possible, it may be worth thinking about making installation safe as well (depending among other things on the volume of anticipated IO Monad libraries w/r/t non-IO Monad libraries, but see #1 below). Regarding making installation safe, Isaac Jones asked:

...

What about packages that install binary tools or data files? I don't want to limit the system to just libraries for the sake of this security feature.

and Simon noted:

...

Setup.lhs runs in the IO monad else it would not be able to move files or run a compiler.

Moving (data) files can be done completely declaratively and safely. Bob the Builder and Sam Sysadmin just needs to know if existing files or paths are being changed in the process. Compiling is perhaps more difficult but it is not clear to me that you can't define compilers available to the installation system and know that these compilers do nothing except produce binaries. (Especially because 99% of compilation will be the Haskell compiler itself and involving other compilers is guaranteed to be a headache no matter what you do) The only other IO that an installer may want is the ability to solicit user input and the ability to download files from the net based on that input. I assume the former awaits a further proposal elaboration if it turns out to be necessary at all (as compared to command line flags). I assume the later can be defined in terms of the former also in some declarative fashion. For exampe, here is a simple safe strawman installer data structure: data Install = Install [CommandLineOption] [Download] [Compile] [FileMove] type CommandLineOption = String type CommandLineValue = String type CommandLinePair = (String,String) data Download = [CommandLinePair] -> URL data Compile = (String,[CommandLinePair]) -> (Compiler,Options,[String])) data FileMove = (String,FilePath) For more flexibility, I could imagine a more procedural one that has only the verbs GET, PUT, POST with respect to various resources including compilers and file systems. -Alex- #1 Java and Python provide control/visibility into what libraries/native-functions a foreign lib uses. I assume that Haskell could do this at compile/download time rather than runtime which would be substantially cooler. Downloadable code in combination with Haskell's emerging generic programming infrasturcture could be really really useful and powerful. I imagine a server to which you upload code that navigates arbitrary data structures (e.g. a database) and acts based on the data it finds in some safe manner. http://java.sun.com/j2se/1.4.2/docs/api/java/lang/SecurityManager.html http://www.procoders.net/pythonstuff/SandBox/) _________________________________________________________________ S. Alexander Jacobson mailto:me@alexjacobson.com tel:917-770-6565 http://alexjacobson.com

...

| > I like the simplicity but would also like the spec | > to make it easy for me to guarantee that that I | > don't end up running/installing malware. | > | > I think Haskell's typesystem and purity should | > make it relatively easy to make sure that:

I don't think so, alas. The IO monad lets you do *anything*, and of course Setup.lhs runs in the IO monad, else it would not be able to move files or run a compiler.

So I'm not optimistic. Perhaps a package whose Setup.lhs did nothing but import Distribution.Simple (which you perhaps trust) would be more trustworthy than a big pile of goop.

But remember that you are installing a library that you will later (presumably) run, and that might be bad too.

I'm not optimistic here.

Simon

Isaac Jones

9:14 p.m.

New subject: new Library Infrastructure spec.

"S. Alexander Jacobson" writes: (snip)

...

Regarding making installation safe, Isaac Jones asked:

...
What about packages that install binary tools or data files? I don't want to limit the system to just libraries for the sake of this security feature.

and Simon noted:

...
Setup.lhs runs in the IO monad else it would not be able to move files or run a compiler.

Moving (data) files can be done completely declaratively and safely. Bob the Builder and Sam Sysadmin just needs to know if existing files or paths are being changed in the process.

I assume that your scheme has some kind of executable on Bob and Sam's machines which knows how to install libraries and such, and this system will also have to know how to perform the configuration. Let's call that StaticSetup (since it performs the tasks of Setup.lhs, but on static information provided by Angela). StaticSetup is a program that Sam trusts. Here are some more issues you're scheme will have to deal with: 1) Marcus Makefile. He's important to us; he's a hard-core haskell developer and we want to support him. How does he go about calling through to the "make install" command? 2) If you allow executables to be installed, you'll have to watch out for attacks which replace "ls" or something. I guess you could have a warning for any package which tries to overwrite an existing file, though this may happen a lot for upgrading a current library. 3) There is some grey area between the Angela and Marcus use cases where she can perform extra preprocessing steps and configuration steps within the context of Setup.lhs (since she has all of Haskell at her disposal.) In your scheme, there is no space between Angela and Marcus. You're either in or out when it comes to utilizing Distribution.Simple. peace, isaac

S. Alexander Jacobson

3 Jun 3 Jun

5:48 p.m.

New subject: new Library Infrastructure spec.

On Wed, 2 Jun 2004, Isaac Jones wrote:

...

Here are some more issues you're scheme will have to deal with:

1) Marcus Makefile. He's important to us; he's a hard-core haskell developer and we want to support him. How does he go about calling through to the "make install" command?

I thought this proposal was an alternative to "make install." Are people doing things with makefiles that can't be done using StaticSetup? If they are, do these things involve multiple compilers/interpreters that may or may not be installed together on a given machine? If yes, then perhaps they are better served with a more generic install functionality that interacts with Haskell via the more restricted StaticSetup.

...

2) If you allow executables to be installed, you'll have to watch out for attacks which replace "ls" or something. I guess you could have a warning for any package which tries to overwrite an existing file, though this may happen a lot for upgrading a current library.

But typically it would happen all within the same directory so there it should be possible to offer the user the option to accept all updates to a particular directory. Simply telling the user in which directories updates may be happening is a big plus.

...

3) There is some grey area between the Angela and Marcus use cases where she can perform extra preprocessing steps and configuration steps within the context of Setup.lhs (since she has all of Haskell at her disposal.) In your scheme, there is no space between Angela and Marcus. You're either in or out when it comes to utilizing Distribution.Simple.

I'm not sure I follow here. Nothing stops you from chaining/recursive StaticSetup... -Alex- _________________________________________________________________ S. Alexander Jacobson mailto:me@alexjacobson.com tel:917-770-6565 http://alexjacobson.com

Isaac Jones

6:12 p.m.

New subject: new Library Infrastructure spec.

"S. Alexander Jacobson" writes:

...

On Wed, 2 Jun 2004, Isaac Jones wrote:

...
Here are some more issues you're scheme will have to deal with:

1) Marcus Makefile. He's important to us; he's a hard-core haskell developer and we want to support him. How does he go about calling through to the "make install" command?

I thought this proposal was an alternative to "make install." Are people doing things with makefiles that can't be done using StaticSetup?

Please see (Section 1.1): Marcus Makefile is like Angela, but more sophisticated. He has a complicated tool, and uses makefiles. Still, he wants to arrange that Roland, Donald, Bob, Sam, and Joe don't need to know about his internal complexity. (Chapter 4): where the interface for the Setup script is explained. (Section 5.4) The Haskell libraries that support the simple build infrastructure can, of course, also be re-used to make setup scripts that work quite differently. At one extreme is a setup script that immediately shells out into make, which does all the work. It's important to remember that the Setup script is an _interface_ that hides underlying complexity. Under the hood might be Distribution.Simple (section 5.3), or it might be a bunch of makefiles, or it might be hmake, or it might be a haskell program that you write yourself. We're trying to give the end users (and tools that may be layered on top of the packaging system) a consistent interface to packages whose needs and underlying implementations are very different. (Think of the different kinds of needs between HUnit and wxHaskell). Now Distribution.Simple is a particular implementation of this interface which is meant for very simple libraries and tools. What you propose to provide is a secure infrastructure for building and installing very simple libraries and tools, but not providing a common interface across diverse libraries and tools. I feel that this is a completely different goal. (I also think that if this is what's desired, the user is better off using a packaging system that has more guarantees, and not installing code from untrusted sources at all (see my work in "apt-secure" btw) but that's neither here nor there.)

...

...
2) If you allow executables to be installed, you'll have to watch out for attacks which replace "ls" or something. I guess you could have a warning for any package which tries to overwrite an existing file, though this may happen a lot for upgrading a current library.

But typically it would happen all within the same directory so there it should be possible to offer the user the option to accept all updates to a particular directory.

Simply telling the user in which directories updates may be happening is a big plus.

That's a good point.

...

...
3) There is some grey area between the Angela and Marcus use cases where she can perform extra preprocessing steps and configuration steps within the context of Setup.lhs (since she has all of Haskell at her disposal.) In your scheme, there is no space between Angela and Marcus. You're either in or out when it comes to utilizing Distribution.Simple.

I'm not sure I follow here. Nothing stops you from chaining/recursive StaticSetup...

I don't know what you mean by recursive StaticSetup, so I'll try again. Distribution.Simple provides some mechanisms for making Angela's life easier, especially with very simple libraries. Now say Angela has a library that uses Distribution.Simple, but she's altered it to utilize the Happy preprocessor, and (let's say) Distribution.Simple doesn't know anything about Happy. Now Angela can add a "system" call to her Setup file to preprocess what she needs to, and continue using the rest of the Distribution.Simple mechanism she was using before. In the StaticSetup scheme, Angela is out of luck. She can't change the behavior of StaticSetup to do the right thing, so she has to get rid of her build system altogether and start from scratch with makefiles or something. This is what we want to avoid. peace, isaac

S. Alexander Jacobson

4 Jun 4 Jun

1:36 p.m.

New subject: new Library Infrastructure spec.

Perhaps it makes sense to define the mission more precisely. The document begins: The Haskell Library Infrastructure Project is an effort to provide a framework for developers to more effectively contribute their software to the Haskell community. Does "software" include an IDE written in C, Python, or Java? Does it include GHC itself? Are we really trying to design a Haskell wrapper around autoconf, make, or Installshield? Why reinvent the wheel? Given that this document is part of "Library Infrastructure", perhaps it makes sense to restrict the mission to: The Haskell Library Infrastructure Project is an effort to provide a framework that makes it easy for Haskell developers to package *haskell libraries* for easy installation into any Haskell development environment. Following this more restricted mission, the HPS would be something that autoconf calls to interact with the local Haskell environment rather than vice-versa. Perhaps it makes sense to work through some actual real life examples. If I have a package that combines C, Perl, and Haskell? How should I distribute install it? HPS or Autoconf? -Alex- _________________________________________________________________ S. Alexander Jacobson mailto:me@alexjacobson.com tel:917-770-6565 http://alexjacobson.com On Thu, 3 Jun 2004, Isaac Jones wrote:

...

"S. Alexander Jacobson" writes:

...
On Wed, 2 Jun 2004, Isaac Jones wrote:

...
Here are some more issues you're scheme will have to deal with:

1) Marcus Makefile. He's important to us; he's a hard-core haskell developer and we want to support him. How does he go about calling through to the "make install" command?

I thought this proposal was an alternative to "make install." Are people doing things with makefiles that can't be done using StaticSetup?

It's important to remember that the Setup script is an _interface_ that hides underlying complexity. Under the hood might be Distribution.Simple (section 5.3), or it might be a bunch of makefiles, or it might be hmake, or it might be a haskell program that you write yourself.

We're trying to give the end users (and tools that may be layered on top of the packaging system) a consistent interface to packages whose needs and underlying implementations are very different. (Think of the different kinds of needs between HUnit and wxHaskell).

Now Distribution.Simple is a particular implementation of this interface which is meant for very simple libraries and tools.

What you propose to provide is a secure infrastructure for building and installing very simple libraries and tools, but not providing a common interface across diverse libraries and tools. I feel that this is a completely different goal. (I also think that if this is what's desired, the user is better off using a packaging system that has more guarantees, and not installing code from untrusted sources at all (see my work in "apt-secure" btw) but that's neither here nor there.)

...
...
2) If you allow executables to be installed, you'll have to watch out for attacks which replace "ls" or something. I guess you could have a warning for any package which tries to overwrite an existing file, though this may happen a lot for upgrading a current library.

But typically it would happen all within the same directory so there it should be possible to offer the user the option to accept all updates to a particular directory.

Simply telling the user in which directories updates may be happening is a big plus.

That's a good point.

...
...
3) There is some grey area between the Angela and Marcus use cases where she can perform extra preprocessing steps and configuration steps within the context of Setup.lhs (since she has all of Haskell at her disposal.) In your scheme, there is no space between Angela and Marcus. You're either in or out when it comes to utilizing Distribution.Simple.

I'm not sure I follow here. Nothing stops you from chaining/recursive StaticSetup...

I don't know what you mean by recursive StaticSetup, so I'll try again.

Distribution.Simple provides some mechanisms for making Angela's life easier, especially with very simple libraries. Now say Angela has a library that uses Distribution.Simple, but she's altered it to utilize the Happy preprocessor, and (let's say) Distribution.Simple doesn't know anything about Happy. Now Angela can add a "system" call to her Setup file to preprocess what she needs to, and continue using the rest of the Distribution.Simple mechanism she was using before.

In the StaticSetup scheme, Angela is out of luck. She can't change the behavior of StaticSetup to do the right thing, so she has to get rid of her build system altogether and start from scratch with makefiles or something. This is what we want to avoid.

peace,

isaac

Isaac Jones

2:08 p.m.

New subject: new Library Infrastructure spec.

"S. Alexander Jacobson" writes:

...

Perhaps it makes sense to define the mission more precisely. The document begins:

The Haskell Library Infrastructure Project is an effort to provide a framework for developers to more effectively contribute their software to the Haskell community.

Software includes Haskell libraries and other tools writen in Haskell.

...

Does "software" include an IDE written in C, Python, or Java?

No.

...

Does it include GHC itself? Are we really trying to design a Haskell wrapper around autoconf, make, or Installshield? Why reinvent the wheel?

What do you mean? I don't think we ever claimed to be trying to implement autoconf or installshield. There are certain kinds of packages for which we think using make won't be necessary. Likewise, autoconf is overkill for certain kinds of packages. Wrapping autoconf is not the same as reinventing autoconf.

...

Given that this document is part of "Library Infrastructure", perhaps it makes sense to restrict the mission to:

We're going to rename the "Library Infrastructure Project" to reflect that it is for other tools written in Haskell as well. The idea that it is for libraries only keeps creeping in and I keep pusing it back, but I'm not always as firm as I should be. We really need to rename it.

...

The Haskell Library Infrastructure Project is an effort to provide a framework that makes it easy for Haskell developers to package *haskell libraries* for easy installation into any Haskell development environment.

I don't think we should restrict it to Haskell libraries only. I still don't see any real gain in doing so.

...

Following this more restricted mission, the HPS would be something that autoconf calls to interact with the local Haskell environment rather than vice-versa.

It can work both ways. The Setup.lhs script can call autoconf, which can query the package database, for instance.

...

Perhaps it makes sense to work through some actual real life examples. If I have a package that combines C, Perl, and Haskell? How should I distribute install it? HPS or Autoconf?

HPS can wrap autoconf in this case. It doesn't have to, though. You could implement whatever you need directly in the Setup script. peace, isaac

S. Alexander Jacobson

5:09 p.m.

New subject: new Library Infrastructure spec.

On Fri, 4 Jun 2004, Isaac Jones wrote:

...

...
Does "software" include an IDE written in C, Python, or Java? No. Software includes Haskell libraries and other tools writen in Haskell.

...
The Haskell Library Infrastructure Project is an effort to provide a framework that makes it easy for Haskell developers to package *haskell libraries* for easy installation into any Haskell development environment.

I don't think we should restrict it to Haskell libraries only. I still don't see any real gain in doing so.

I don't see any real gain in restricting it only to libraries and tools written in Haskell. Given the flexibility you are trying to achieve (compilation of arbitrary source, random mucking around in the local environment, calling autoconf!), it certainly *looks* like we are really talking about another generic config tool, this time written and scripted in Haskell? Note: It might be perfectly reasonable to write such a tool to supercede autoconf, just as it was totally reasonable to write Darc to supercede CVS. (Though perhaps the community should start thinking about eating the dogfood). Aside: A Haskell tool that calls autoconf will eventually evolve into a Haskell tool that replaces autoconf. The flip side of teh above is that I do see a substantial gain in having a really simple tool focused exclusively on installing Haskell libraries together with their associated data and binaries into a Haskell development environment such that they may be easily used from new Haskell programs. This gain may include perhaps: * an easy way to install Haskell code from autoconf * a standard way to access a Haskell interpreter/compiler * a safe way to install/run foreign code * a much smaller overall project to implement * etc.. Perhaps it would be helpful if we walk throug some scenarios: Case 1: Suppose I write a package that makes it easy to call back and forth between Python and Haskell using some code written in C. The installer needs to compile some C libraries to be called by Python wrapper code and some C libraries to be called by Haskell wrapper code. This C code is generated by code written in Python and Haskell. Should the outer layer be autoconf, distutils, or HPS? Case 2: Suppose I write a relational database library that stays outside the IO Monad. Its just a a few Haskell files that need to be installed in the Data directory? What is the simplest thing that I can do to package this code? The full flexibility of setup.lhs seems like overkill in this case. Case 3: Suppose I write a http server that library that relies on a mime-types file.... My gut is that the data file in this case should be part of the documentation... Note: I find the runhugs requirement in the current document frustrating. I feel like there should be a command in the local installation like hugs -install <setupscript> and ghc -install <setupscript> etc. -Alex- _________________________________________________________________ S. Alexander Jacobson mailto:me@alexjacobson.com tel:917-770-6565 http://alexjacobson.com

Arie Peterson

6:10 p.m.

New subject: new Library Infrastructure spec.

At 18:12 2004-06-03 -0400, Isaac Jones wrote:

...

We're trying to give the end users (and tools that may be layered on top of the packaging system) a consistent interface to packages whose needs and underlying implementations are very different. (Think of the different kinds of needs between HUnit and wxHaskell).

Now Distribution.Simple is a particular implementation of this interface which is meant for very simple libraries and tools.

What you propose to provide is a secure infrastructure for building and installing very simple libraries and tools, but not providing a common interface across diverse libraries and tools. I feel that this is a completely different goal. (I also think that if this is what's desired, the user is better off using a packaging system that has more guarantees, and not installing code from untrusted sources at all (see my work in "apt-secure" btw) but that's neither here nor there.)

I do not understand why a strictly defined data structure would not be a common interface, and why a (for ignorant windows users like me, at least) obscure and opaque makefile would. Do you mean by 'interface' the way in which a user invokes the installation? I would like to be able to choose whether I type "make install" myself, let a linux package manager handle that for me, or perhaps use some customised homemade install manager to give me exactly the information and options I want. Alex' idea of an installation data structure suits that wish perfectly. As mentioned before, it would be particularly nice to know what files/directories are to be modified or added.

...

Distribution.Simple provides some mechanisms for making Angela's life easier, especially with very simple libraries. Now say Angela has a library that uses Distribution.Simple, but she's altered it to utilize the Happy preprocessor, and (let's say) Distribution.Simple doesn't know anything about Happy. Now Angela can add a "system" call to her Setup file to preprocess what she needs to, and continue using the rest of the Distribution.Simple mechanism she was using before.

In the StaticSetup scheme, Angela is out of luck. She can't change the behavior of StaticSetup to do the right thing, so she has to get rid of her build system altogether and start from scratch with makefiles or something. This is what we want to avoid.

If the preprocessor is to be executed by the user, then this is just a dependency and can be listed as such in the install structure. If the preprocessor is to be executed by Angela, then she does so happily (sorry ;-)) and the user need not know that a preprocessor was used. Am I missing the point? Let me suggest a different approach. We could define a Setup/Installation monad, together with some some primitives (display a message, ask for user input, move files, check dependencies, request installation of other packages, etc.). One special primitive might be 'IO', containing an arbitrary IO operation, to accommodate installation actions that cannot be defined in terms of the given primitives. A package description/installation might consist of a haskell data structure, holding some basic properties of the package and a Setup () structure. This package structure may be distributed: 1. as haskell source code alone, to allow users to install the package with their own customised installer or a default ghc installer; 2. in source form with an installer that the user may or may not use; 3. compiled by the author with her installer to produce a binary executable installer. Notice that an 'installer' in this context is a haskell program, typically run with hugs or ghci (unless dynamic linking will be widely supported), that merely has a Setup () as a parameter and interprets this structure on the fly, in a way that the author of the installer thinks best: install the package in the right directory, do or do not download necessary libraries automatically, do or do not ask the user to authorise the IO parts, register the program with the compiler of choice, etc.. The installer might get settings from a conf file or bother the user with every detail - that is up to the author of the installer. I feel somewhat awkward to suggest this wild idea, because the proposal seems well-thought out and quite mature. So please convince me that this approach is useless :-). Regards, Arie

Isaac Jones

8:18 p.m.

New subject: new Library Infrastructure spec.

Hi Arie, Arie Peterson writes:

...

At 18:12 2004-06-03 -0400, Isaac Jones wrote:

...
We're trying to give the end users (and tools that may be layered on top of the packaging system) a consistent interface to packages whose needs and underlying implementations are very different. (Think of the different kinds of needs between HUnit and wxHaskell).

Now Distribution.Simple is a particular implementation of this interface which is meant for very simple libraries and tools.

What you propose to provide is a secure infrastructure for building and installing very simple libraries and tools, but not providing a common interface across diverse libraries and tools. I feel that this is a completely different goal. (I also think that if this is what's desired, the user is better off using a packaging system that has more guarantees, and not installing code from untrusted sources at all (see my work in "apt-secure" btw) but that's neither here nor there.)

I do not understand why a strictly defined data structure would not be a common interface, and why a (for ignorant windows users like me, at least) obscure and opaque makefile would.

I'm afraid you must misunderstand something. A makefile is not the interface. The interface is the Setup script and the arguments to it (install, build, etc).

...

Do you mean by 'interface' the way in which a user invokes the installation? I would like to be able to choose whether I type "make install" myself,

I don't understand; the "choice" of how to install the system is not made by you (Joe, Bob, or Sam), the installer, but rather by the author (Angela or Marcus).

...

let a linux package manager handle that for me, or perhaps use some customised homemade install manager to give me exactly the information and options I want.

This doesn't replace RPM or Debian-type package management; it facilitates it (for Ronald, Donald, and Willie) . For instance, the "common debian build system" can be designed to wrap the Setup interface; making it very easy to make Debian packages for Haskell libraries and tools. In particular, all a packager has to do for a package that conforms to the interface, is write a "control" file that says "build this like a standard haskell package". I already implemented a prototype of this, and it works nicely. OTOH, for those of you who aren't lucky enough to have a nice package manager on your operating system, the Setup script will provide a consistent interface for installing packages, and in the future perhaps for wrapped tools for downloading and installing packages from a remote database, such as CPAN.

...

Alex' idea of an installation data structure suits that wish perfectly. As mentioned before, it would be particularly nice to know what files/directories are to be modified or added.

Package managers such as dpkg typically handle this kind of thing. If you use one of those, you'll still have this ability for packages installed via dpkg. However, providing such a facility is not possible for packages by Marcus. This kind of security is properly handled on the operating system level. If you are downloading random code from the web and installing it outside your package manager, haskell packages are the least of your problems.

...

...
Distribution.Simple provides some mechanisms for making Angela's life easier, especially with very simple libraries. Now say Angela has a library that uses Distribution.Simple, but she's altered it to utilize the Happy preprocessor, and (let's say) Distribution.Simple doesn't know anything about Happy. Now Angela can add a "system" call to her Setup file to preprocess what she needs to, and continue using the rest of the Distribution.Simple mechanism she was using before.

In the StaticSetup scheme, Angela is out of luck. She can't change the behavior of StaticSetup to do the right thing, so she has to get rid of her build system altogether and start from scratch with makefiles or something. This is what we want to avoid.

If the preprocessor is to be executed by the user,

Which user are you referring to? We gave them all names :)

...

then this is just a dependency and can be listed as such in the install structure. If the preprocessor is to be executed by Angela, then she does so happily (sorry ;-)) and the user need not know that a preprocessor was used.

I'm confused by this. Maybe if you told me which user you're referring to I could explain it.

...

Let me suggest a different approach.

We could define a Setup/Installation monad, together with some some primitives (display a message, ask for user input, move files, check dependencies, request installation of other packages, etc.). One special primitive might be 'IO', containing an arbitrary IO operation, to accommodate installation actions that cannot be defined in terms of the given primitives.

A package description/installation might consist of a haskell data structure, holding some basic properties of the package and a Setup () structure.

If the installer system, provided by Angela, can perform arbitrary IO actions, then you have lost the property of knowing which files it installs.

...

This package structure may be distributed: 1. as haskell source code alone, to allow users to install the package with their own customised installer or a default ghc installer;

How does the user customize an installer for darcs?

...

2. in source form with an installer that the user may or may not use; 3. compiled by the author with her installer to produce a binary executable installer.

Our system covers 2 and 3.

...

Notice that an 'installer' in this context is a haskell program, typically run with hugs or ghci (unless dynamic linking will be widely supported), that merely has a Setup () as a parameter and interprets

I'm confused by who is providing this installer, and how this Setup structure gets provided to an executable.

...

this structure on the fly, in a way that the author of the installer thinks best: install the package in the right directory, do or do not download necessary libraries automatically, do or do not ask the user to authorise the IO parts,

Why ask the user if they want to authorize the IO parts? They will have to read the code to find out what the IO actions do.

...

register the program with the compiler of choice, etc.. The installer might get settings from a conf file or bother the user with every detail - that is up to the author of the installer.

I feel somewhat awkward to suggest this wild idea, because the proposal seems well-thought out and quite mature. So please convince me that this approach is useless :-).

I'm afraid I do not completely understand your approach. Most especially, I'm confused about what it buys you that our approach does not. I do suspect that we're losing support for Marcus, which is not acceptable. Further, I think that once you completely understand our approach, you'll like it. Please provide suggestions for the proposal for bits which are unclear. peace, isaac

Arie Peterson

7 Jun 7 Jun

11:44 a.m.

New subject: new Library Infrastructure spec.

At 20:18 2004-06-04 -0400, Isaac Jones wrote:

...

Hi Arie,

Hi Isaac,

...

I'm afraid you must misunderstand something. A makefile is not the interface. The interface is the Setup script and the arguments to it (install, build, etc).

OK. As I understand it, Marcus creates a Setup.lhs that merely passes its arguments to a makefile. All the relevant information is contained in that makefile. Is that makefile supposed to be platform independent? (And with platform independent I mean really independent, so no emulation (Cygwin, for example) is needed.)

...

...
Do you mean by 'interface' the way in which a user invokes the installation? I would like to be able to choose whether I type "make install" myself,

I don't understand; the "choice" of how to install the system is not made by you (Joe, Bob, or Sam), the installer, but rather by the author (Angela or Marcus).

As I see it, the author (Angela or Marcus) decides what needs to be done to perform the installation, but the user (Joe, Bob or Sam) decides how the installation is performed. (See below for more about this distinction.)

...

...
Alex' idea of an installation data structure suits that wish perfectly. As mentioned before, it would be particularly nice to know what files/directories are to be modified or added.

Package managers such as dpkg typically handle this kind of thing. If you use one of those, you'll still have this ability for packages installed via dpkg.

Still, I think that that information should be directly available from the installation file, without actually performing the installation.

...

...
...
Distribution.Simple provides some mechanisms for making Angela's life easier, especially with very simple libraries. Now say Angela has a library that uses Distribution.Simple, but she's altered it to utilize the Happy preprocessor, and (let's say) Distribution.Simple doesn't know anything about Happy. Now Angela can add a "system" call to her Setup file to preprocess what she needs to, and continue using the rest of the Distribution.Simple mechanism she was using before.

In the StaticSetup scheme, Angela is out of luck. She can't change the behavior of StaticSetup to do the right thing, so she has to get rid of her build system altogether and start from scratch with makefiles or something. This is what we want to avoid.

If the preprocessor is to be executed by the user,

Which user are you referring to? We gave them all names :)

That would be Bob or Sam (Joe doesn't seem too involved in the entire package system).

...

If the installer system, provided by Angela, can perform arbitrary IO actions, then you have lost the property of knowing which files it installs.

That's right. However, I imagine that the IO action contains a String field with a message from Angela, explaining what the IO action does. If the setup contains no IO actions, then the user (Bob or Sam) can be sure that nothing unexpected can happen. If the setup does contain IO actions, then the user (Bob or Sam) may be made aware of that so he can decide if he wants to continue.

...

...
This package structure may be distributed: 1. as haskell source code alone, to allow users to install the package with their own customised installer or a default ghc installer;

How does the user customize an installer for darcs?

I am not familiar at all with darcs. What makes it a special case? In general, an installer can be written from scratch (in haskell), derived from another installer (by editing it's source code) or customised (by parametrising or configuring an existing installer).

...

...
2. in source form with an installer that the user may or may not use; 3. compiled by the author with her installer to produce a binary executable installer.

Our system covers 2 and 3.

Suppose that, for any reason, someone (Sam, say) does not want to run the supplied installer (supplied by Angela, that is). With the your system, Sam can find out what must be done to install the package - only by extracting the necessary information from Setup.lhs (case 2), which can be very hard if Angela did not comment her code very well; - not at all (case 3). Right?

...

...
Notice that an 'installer' in this context is a haskell program, typically run with hugs or ghci (unless dynamic linking will be widely supported), that merely has a Setup () as a parameter and interprets

I'm confused by who is providing this installer, and how this Setup structure gets provided to an executable.

The installer can be provided by Angela, if she thinks Bob or Sam will like her installer or does not have an installer at hand (which is unlikely, but hey). Often, the installer will be the one supplied by the compiler that Bob or Sam uses. Configuration options (of the installer) can be saved for installation of other packages. Some people will want to hack on an existing installer or write one from scratch. Maybe Sam wants to create an installer for Bob to automate installations as much as possible, or Bob wants to have an installer that does exactly what he tells it to do, and nothing more. The Setup structure can be interpreted/executed (by Bob or Sam) by loading both the installer and the module containing the Setup structure in Hugs or GHCi and issuing the right command ("Install.install angela'sSetup" or "Install.install (defaultOptions { path = "/usr/local/" }) angela'sSetup" or something like that). If dynamic linking is an option, that can be used as well: in that case the Setup is compiled and linked (by Bob or Sam) with the (probably already compiled) installer. Angela can also compile her Setup structure with her installer to produce an executable (installer and Setup in one file).

...

Why ask the user if they want to authorize the IO parts? They will have to read the code to find out what the IO actions do.

As said above, a message (written by Angela) can be included in the IO construct to inform the user (Bob or Sam) about its purpose. Perhaps there are some common tasks that need to be done using an IO construct? In that case we could catch their presence in a datatype to enable further automation (although it would be preferable to keep as much as possible out of the IO part): ioTask :: IOTaskType -> IO -> String -> Setup () ioTask taskType task comment = ... data IOTaskType = FiddleWithKernel | Other (I don't know if the IOTaskType is a useful addition; it is not essential to this approach.)

...

I'm afraid I do not completely understand your approach. Most especially, I'm confused about what it buys you that our approach does not. I do suspect that we're losing support for Marcus, which is not acceptable.

In my approach there is a clear distinction between what needs to be done (this is contained in the Setup monad) and how it is done (this is directed by the installer). It keeps information in a useful, non-opaque form as long as possible, allowing for more flexibility in the installation.

...

Further, I think that once you completely understand our approach, you'll like it. Please provide suggestions for the proposal for bits which are unclear.

That is probably true. One aspect that is unclear to me is how version management is facilitated. However, it may be wise to let that rest for a while.

...

peace,

isaac

Greetings, Arie

Alastair Reid

12:46 p.m.

New subject: new Library Infrastructure spec.

On Monday 07 June 2004 16:44, Arie Peterson wrote:

...

OK. As I understand it, Marcus creates a Setup.lhs that merely passes its arguments to a makefile. All the relevant information is contained in that makefile. Is that makefile supposed to be platform independent? (And with platform independent I mean really independent, so no emulation (Cygwin, for example) is needed.)

As I understand it, the idea is that the makefile is as portable or non-portable as Marcus felt like making it. Once the code has been packaged up and made available, someone (Marcus or a fan of Marcus's code) _might_ do any of the following: - Nothing more. In this case, the code integrates nicely with the rest of the packaging system on some platforms and is unavailable on others. Not ideal but better than nothing and it might be that the code isn't very portable anyway. - Make the makefile more portable. Anyone who has maintained a system of any complexity will know that porting (code, makefiles, configure scripts, etc.) can be an enormous amount of work. So, it would be nice if this gets done but if the code only ever runs/builds/installs on FreeBSD systems which have libfoo version 2.97a (not 2.97b, please) installed, then Marcus and his pals have still gained from using the packaging system because it can interoperate with other packages. - Replace the makefile with something more portable like an hsmake file, maybe using Malcolm Wallace's cpphs and other portable tools, maybe writing a simple Haskell (or Perl, python, etc.) script to replace some key part of the makefile. Hopefully, this makes the installation process more portable so, again, some gain is made. We're still not using the simple installation system, we're still not fully portable but the code is more widely usable and it can interoperate with other packages. - Realize that the simple installation system has all the features needed to replace the Makefile. (Or extend the simple installation system to achieve this goal.) This is obviously simplest since it is portable, easier to port, etc. but it is ok if some packages can't reach this point. My point is that you don't have to be using the simple installation system to get benefit and to interoperate with packages which do use the simple system. This is very much a strength of the proposal since any single system powerful enough to do everything we want of it is likely to be complex, hard to port, etc. Better to have a solution that does 95% of the problem well and which can interoperate with whatever is used for the other 5%. -- Alastair Reid

7663

Age (days ago)

7668

Last active (days ago)

List overview

Download

11 comments

5 participants

participants (5)

Alastair Reid
Arie Peterson
Isaac Jones
S. Alexander Jacobson
Simon Peyton-Jones

RE: new Library Infrastructure spec.

tags

participants (5)