Optional C dependencies / build-time feature toggles

Hello Cafe, I have this project here: https://sprinkles.tobiasdammers.nl/ (source on github: https://github.com/tdammers/sprinkles), and it has a bunch of runtime dependencies on C libraries. Many of them are only required for certain use cases, e.g., the libmysqlclient dependency is only relevant if you want to access MySQL database backends, libfcgi is only required for hosting as FastCGI (which isn't even the recommended first choice), etc. Still, in order to build and run Sprinkles, all the dependencies must be installed, which is of course not ideal. The pipe dream would be for those dependencies to be optional *at run time*, such that the application would test for their presence on startup, or load them dynamically when and if they're needed. However, I'm also fine with making them compile-time feature toggles, such that users can enable only the ones they need. Doing some sort of autotools-style dependency discovery would of course be cool as well (i.e., only enable mysql support if libmysqlclient is present). So the broad question is, which of these are realistic, and how would I go about it? And, regarding feature toggles, this bit in the Cabal FAQ baffles me a little:
Question: I like to let the user enable extended functionality using a Cabal flag. Is this the right way?
Answer: Certainly not. Since other packages can distinguish packages only according to their name and their version, it is not a good idea to allow different APIs for the same package version. Cumbersome as it is you have to move extra features to a separate package.
Moving the features to a separate package wouldn't really solve anything AFAICT, because I'd still have to bring them together somehow. -- Tobias Dammers - tdammers@gmail.com

On Sun, Dec 11, 2016 at 11:53 AM, Tobias Dammers
And, regarding feature toggles, this bit in the Cabal FAQ baffles me a little:
Question: I like to let the user enable extended functionality using a Cabal flag. Is this the right way?
Answer: Certainly not. Since other packages can distinguish packages only according to their name and their version, it is not a good idea to allow different APIs for the same package version. Cumbersome as it is you have to move extra features to a separate package.
Moving the features to a separate package wouldn't really solve anything AFAICT, because I'd still have to bring them together somehow.
This is the same as the variants problem with MacPorts: you can't have another port depend on a variant, because then dependency calculation (already nightmarish) gets a combinatorial explosion on top of all the existing problems. -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

On 12/11/2016 11:53 AM, Tobias Dammers wrote:
The pipe dream would be for those dependencies to be optional *at run time*, such that the application would test for their presence on startup, or load them dynamically when and if they're needed. However, I'm also fine with making them compile-time feature toggles, such that users can enable only the ones they need. Doing some sort of autotools-style dependency discovery would of course be cool as well (i.e., only enable mysql support if libmysqlclient is present).
This actually isn't ideal, because once everything is installed, you have no idea what is important. Can I uninstall libfoo-mysql? Well, bar-utils is making use of it, but it always does when libfoo-mysql is present, so there's no way to tell if the mysql support is important (i.e. whether or not the user wanted it when he installed bar-utils). The end result is that every "optional" dependency has to remain installed unconditionally forever, because removing them might break software that auto-detects their presence. (Google: "automagic dependencies")
So the broad question is, which of these are realistic, and how would I go about it?
And, regarding feature toggles, this bit in the Cabal FAQ baffles me a little:
Question: I like to let the user enable extended functionality using a Cabal flag. Is this the right way?
Answer: Certainly not. Since other packages can distinguish packages only according to their name and their version, it is not a good idea to allow different APIs for the same package version. Cumbersome as it is you have to move extra features to a separate package.
This is a cabal-install or stack deficiency, not one inherent to Haskell packages or package management in general. The real solution: use a feature flag, and then use a package manager that knows what to do with them. As an added bonus, a real package manager can handle the C library dependencies as well, and your application won't break randomly when those libraries move around or disappear.

Excerpts from Tobias Dammers's message of 2016-12-11 17:53:07 +0100:
The pipe dream would be for those dependencies to be optional *at run time*, such that the application would test for their presence on startup, or load them dynamically when and if they're needed.
It's not completely unthinkable to directly interface with dlopen() and similar functions to dynamically load symbols. posix even has a (not very widely used) module for precisely this: https://hackage.haskell.org/package/unix-2.7.2.1/docs/System-Posix-DynamicLi...
However, I'm also fine with making them compile-time feature toggles, such that users can enable only the ones they need.
This is certainly what flags in Cabal files can be used for. More below.
Doing some sort of autotools-style dependency discovery would of course be cool as well (i.e., only enable mysql support if libmysqlclient is present).
This is also possible using build-type: Configure, see: http://cabal.readthedocs.io/en/latest/developing-packages.html#system-depend...
Question: I like to let the user enable extended functionality using a Cabal flag. Is this the right way?
Answer: Certainly not. Since other packages can distinguish packages only according to their name and their version, it is not a good idea to allow different APIs for the same package version. Cumbersome as it is you have to move extra features to a separate package.
Moving the features to a separate package wouldn't really solve anything AFAICT, because I'd still have to bring them together somehow.
Yeah, this is something that gets a bit of push and pull, even amongst Cabal developers. It is definitely true that there's no way to depend on a library, AND say that it must have some feature flag toggled, which makes it impossible to accurately specify the dependency. But for an executable application, this seems like less of a problem because no one can depend on you. Seems a lot more reasonable in this situation. Edward

On 12/11/16 8:53 AM, Tobias Dammers wrote:
The pipe dream would be for those dependencies to be optional *at run time*, such that the application would test for their presence on startup, or load them dynamically when and if they're needed. However, I'm also fine with making them compile-time feature toggles, such that users can enable only the ones they need. Doing some sort of autotools-style dependency discovery would of course be cool as well (i.e., only enable mysql support if libmysqlclient is present).
Hello Tobias, I asked for this, but indeed it's not easy to see how to implement it. I will say that the C libs required by sprinkles are all widely used so installing them wasn't too hard; and, it's nice having all of the power available from the start. Still, you may add more C-based storage back ends in future, and even one C lib does create friction and slow down adoption. compile-time cabal flags (cabal install sprinkles -fmysql -fpostgres) add complexity for maintainer, tools and users. I suppose they might be justified here. You'd have to decide on manual vs automatic flags (toggleable by Cabal), and enabled or disabled by default. (If they are enabled and automatic, and the required C lib is missing, will Cabal quietly skip that feature or give an error ?). The build-type: Configure suggestion sounds intriguing. A custom Setup.hs for fancy install behaviour might also be interesting, at minimum you could detect the C lib status and give more useful user advice. These also add complexity. Or, you could publish multiple semi-redundant packages - sprinkles, sprinkles-postgres, sprinkles-mysql, sprinkles-sqlite, sprinkles-all, sprinkles-lib... simpler packages, but more of them.
participants (5)
-
Brandon Allbery
-
Edward Z. Yang
-
Michael Orlitzky
-
Simon Michael
-
Tobias Dammers