Design of an extensible system

Hi there, I'm currently programming my first non toy project in Haskell. I come from Ruby background, where everything is possible no matter how much entropy the universe gains after you finish your work, so I'm still not sure how to do certain things without this side-effect :) I'll try to be concise. My idea is that the project would consist of several different Haskell packages. The central one, call it Core, would define a certain data type (with info about certain URLs). Other packages would define each one an strategy to do something with that data type (processing the URL info). Each strategy would have its own dependencies, and as a user probably won't need all of the strategies, I would like to keep them in separated packages so she or he has no need to download the unneeded stuff. The problem is that Core should act as the central dashboard, whether through a CLI or even a web frontend. For example, in the CLI case, I would like to be able to install Core package along with package StrategyA and package StrategyB and then do something like the following: ``` bash> core --action=fetch --strategy=A bash> core --action=fetch --strategy=B ``` Two generic ideas come to my mind but I don't know how to translate them to Haskell: - Infer module name from the `--strategy` argument and call it dynamically. However, as far as I know, in Haskell modules are not first class citizens, so no way I can dynamically call them. I have read something to achieve this through Template Haskell or through outdaded [plugins](http://hackage.haskell.org/package/plugins) package, but I'm not sure whether the need is the same and anyway it seems quite unnatural to Haskell (I would not like to go against it, but to learn to do thing in its way). - Define from Core some kind of global list where strategy packages can register a map between CLI argument value and a defined instance of the core data type. But, again, how to do global things in Haskell (and I would prefer not to need external tools like RDBMS, filesystem or whatever)? And, anyway, surely I don't want to do this because concurrency and everything... Any idea? Thanks in advance! Marc Busqué http://waiting-for-dev.github.io/about/

plugins is outdated because the functionality is now in ghc-api and exposed
in more convenient ways by the hint package. (Some parts of it were always
in ghc-api; plugins provided the parts it originally didn't, notably safe
unloading of modules.) That said, dynamic loading is something of a bad fit
for Haskell; often, the easiest way to do this is to build a new program
including the module and run it in place of the original. See for example
how xmonad configuration works. The dyre package also provides a form of
this.
Another way to do this is to use programs instead of modules, and the Core
driver provides the UI and plumbs the programs together with sockets or
other IPC mechanisms. This is slower in general. (Either way requires some
form of serialization and marshaling of data, so you will have that
overhead regardless.)
On Sun, Jul 22, 2018 at 11:27 AM Marc Busqué
Hi there,
I'm currently programming my first non toy project in Haskell. I come from Ruby background, where everything is possible no matter how much entropy the universe gains after you finish your work, so I'm still not sure how to do certain things without this side-effect :)
I'll try to be concise. My idea is that the project would consist of several different Haskell packages. The central one, call it Core, would define a certain data type (with info about certain URLs). Other packages would define each one an strategy to do something with that data type (processing the URL info). Each strategy would have its own dependencies, and as a user probably won't need all of the strategies, I would like to keep them in separated packages so she or he has no need to download the unneeded stuff.
The problem is that Core should act as the central dashboard, whether through a CLI or even a web frontend. For example, in the CLI case, I would like to be able to install Core package along with package StrategyA and package StrategyB and then do something like the following:
``` bash> core --action=fetch --strategy=A bash> core --action=fetch --strategy=B ```
Two generic ideas come to my mind but I don't know how to translate them to Haskell:
- Infer module name from the `--strategy` argument and call it dynamically. However, as far as I know, in Haskell modules are not first class citizens, so no way I can dynamically call them. I have read something to achieve this through Template Haskell or through outdaded [plugins](http://hackage.haskell.org/package/plugins) package, but I'm not sure whether the need is the same and anyway it seems quite unnatural to Haskell (I would not like to go against it, but to learn to do thing in its way).
- Define from Core some kind of global list where strategy packages can register a map between CLI argument value and a defined instance of the core data type. But, again, how to do global things in Haskell (and I would prefer not to need external tools like RDBMS, filesystem or whatever)? And, anyway, surely I don't want to do this because concurrency and everything...
Any idea?
Thanks in advance!
Marc Busqué http://waiting-for-dev.github.io/about/ _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.
-- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

I think you're attempting to optimize the wrong thing. In a dynamic
language like Ruby, this approach makes sense because download times are a
significant fraction of the user's time, and otherwise the user experience
isn't affected much because the code is basically interpreted either way.
Haskell is primarily a compiled language. The total download size probably
won't be significantly larger from downloading all modules. The cost of
recompiling/dynamically loading plugins will be expensive (relative to the
rest of the user time), and payed with every execution. Plus it's
additional work for the developer to set up.
In all likelihood you should just build a binary with all available
strategies and distribute that.
On Sun, Jul 22, 2018, 08:27 Marc Busqué
Hi there,
I'm currently programming my first non toy project in Haskell. I come from Ruby background, where everything is possible no matter how much entropy the universe gains after you finish your work, so I'm still not sure how to do certain things without this side-effect :)
I'll try to be concise. My idea is that the project would consist of several different Haskell packages. The central one, call it Core, would define a certain data type (with info about certain URLs). Other packages would define each one an strategy to do something with that data type (processing the URL info). Each strategy would have its own dependencies, and as a user probably won't need all of the strategies, I would like to keep them in separated packages so she or he has no need to download the unneeded stuff.
The problem is that Core should act as the central dashboard, whether through a CLI or even a web frontend. For example, in the CLI case, I would like to be able to install Core package along with package StrategyA and package StrategyB and then do something like the following:
``` bash> core --action=fetch --strategy=A bash> core --action=fetch --strategy=B ```
Two generic ideas come to my mind but I don't know how to translate them to Haskell:
- Infer module name from the `--strategy` argument and call it dynamically. However, as far as I know, in Haskell modules are not first class citizens, so no way I can dynamically call them. I have read something to achieve this through Template Haskell or through outdaded [plugins](http://hackage.haskell.org/package/plugins) package, but I'm not sure whether the need is the same and anyway it seems quite unnatural to Haskell (I would not like to go against it, but to learn to do thing in its way).
- Define from Core some kind of global list where strategy packages can register a map between CLI argument value and a defined instance of the core data type. But, again, how to do global things in Haskell (and I would prefer not to need external tools like RDBMS, filesystem or whatever)? And, anyway, surely I don't want to do this because concurrency and everything...
Any idea?
Thanks in advance!
Marc Busqué http://waiting-for-dev.github.io/about/ _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

On Mon, 23 Jul 2018, John Lato wrote:
I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either way. Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's additional work for the developer to set up.
What you say makes a lot of sense in terms of optimization. But, what about extensibility? Take for example the Pandoc package, which would be similar in design of what I'm trying to do. Say I want to add a converter to some mark-up format not supported by Pandoc, and that Pandoc team doesn't agree to merge my work in its repo or simply I don't have the time to wait until it happens. In this sense, it would be nice that Pandoc supported dynamically loading of extra plugins on demand. Marc Busqué http://waiting-for-dev.github.io/about/

that Pandoc supported dynamically loading of extra plugins on demand.
It means .dll/.so. But also there are applications with pre-built "plugins" - you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins). It's useful too. 24.07.2018 09:07, Marc Busqué wrotes:
On Mon, 23 Jul 2018, John Lato wrote:
I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either way. Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's additional work for the developer to set up.
What you say makes a lot of sense in terms of optimization. But, what about extensibility? Take for example the Pandoc package, which would be similar in design of what I'm trying to do. Say I want to add a converter to some mark-up format not supported by Pandoc, and that Pandoc team doesn't agree to merge my work in its repo or simply I don't have the time to wait until it happens. In this sense, it would be nice that Pandoc supported dynamically loading of extra plugins on demand.
Marc Busqué http://waiting-for-dev.github.io/about/
_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

On 24 Jul 2018, at 15:16, Paul
wrote: that Pandoc supported dynamically loading of extra plugins on demand.
It means .dll/.so. But also there are applications with pre-built "plugins" - you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins). It's useful too.
Pandoc plug-ins are separate executables rather than shared libraries. Haskell data types are serialised by Pandoc then de-serialised, modified and re-serialised in each plugin. The Pandoc approach is more like a data pipeline with Pandoc itself at both ends. Jeremy
24.07.2018 09:07, Marc Busqué wrotes:
On Mon, 23 Jul 2018, John Lato wrote:
I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either way. Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's additional work for the developer to set up.
What you say makes a lot of sense in terms of optimization. But, what about extensibility? Take for example the Pandoc package, which would be similar in design of what I'm trying to do. Say I want to add a converter to some mark-up format not supported by Pandoc, and that Pandoc team doesn't agree to merge my work in its repo or simply I don't have the time to wait until it happens. In this sense, it would be nice that Pandoc supported dynamically loading of extra plugins on demand.
Marc Busqué http://waiting-for-dev.github.io/about/
_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.
_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Hmm, so this is easier even. Plugin must have standard interface, available through stdin/stdout/pipes/sockets. Stdin/stdout/stderr is more simple and it's easy to debug (as first version, at least), also can be "mapped" to sockets easy (in inetd-style, for example). "Driver" can looks like "cmd" module in Python (it's better than to use command line arguments, due to length limitation). Even more, it's easy convertable to other protocols/formats which allows to switch to microservice architecture without big effort ;) (I did it in D: communication can be automatically mapped to JSON/XML/Show-Read/etc). It sounds as interesting task. I like Powershell: it extends standard way of Unix scripting languages communicating with strings (like Tcl, shell) to objects instead. So binary protocol can be supported too (MessagePack, BSON, Xdr, etc). It's totally language neutral plugin architecture ;) "Pipelining" can be done in WSGI style - with middleware which can help the "plugging" of protocols :) Dispatching can be achieved with special directory layout/some config file (better). 24.07.2018 10:30, Jeremy O'Donoghue wrotes:
On 24 Jul 2018, at 15:16, Paul
mailto:aquagnu@gmail.com> wrote: that Pandoc supported dynamically loading of extra plugins on demand.
It means .dll/.so. But also there are applications with pre-built "plugins" - you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins). It's useful too.
Pandoc plug-ins are separate executables rather than shared libraries. Haskell data types are serialised by Pandoc then de-serialised, modified and re-serialised in each plugin.
The Pandoc approach is more like a data pipeline with Pandoc itself at both ends.
Jeremy
24.07.2018 09:07, Marc Busqué wrotes:
On Mon, 23 Jul 2018, John Lato wrote:
I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either way. Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's additional work for the developer to set up.
What you say makes a lot of sense in terms of optimization. But, what about extensibility? Take for example the Pandoc package, which would be similar in design of what I'm trying to do. Say I want to add a converter to some mark-up format not supported by Pandoc, and that Pandoc team doesn't agree to merge my work in its repo or simply I don't have the time to wait until it happens. In this sense, it would be nice that Pandoc supported dynamically loading of extra plugins on demand.
Marc Busqué http://waiting-for-dev.github.io/about/
_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.
_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

On Tue, 24 Jul 2018, Paul wrote:
you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins).
Sorry if it's an obvious question, but, how do you do it? Marc Busqué http://waiting-for-dev.github.io/about/

On Tue, 24 Jul 2018, Paul wrote:
you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins).
Sorry if it's an obvious question, but, how do you do it? Depends on architecture. Suppose, plugins will be compiled with the application and will be "registered" in some container (list, for example). So, the turn-off will remove them from the list / will mark
24.07.2018 13:29, Marc Busqué wrotes: them with some flag as "disabled". So, your pipeline will skip them. As I understand you have cases: - plugins are DLLs - plugins are statically linked with application but can be disabled individually - plugins are EXEs, so easy can be found/called...
Marc Busqué http://waiting-for-dev.github.io/about/

Am 24.07.2018 um 08:07 schrieb Marc Busqué:
But, what about extensibility? Take for example the Pandoc package, which would be similar in design of what I'm trying to do. Say I want to add a converter to some mark-up format not supported by Pandoc, and that Pandoc team doesn't agree to merge my work in its repo or simply I don't have the time to wait until it happens.
You fork Pandoc and add the code you need. Then you put your extension into a public source repository so that others can share your work. And you ask the Pandoc developers to make their code friendly for extension. I.e. you do the plugins at the source code level, not at the binary level.
participants (6)
-
Brandon Allbery
-
Jeremy O'Donoghue
-
Joachim Durchholz
-
John Lato
-
Marc Busqué
-
Paul