On 10/03/14 18:41, Vagif Verdi wrote:

This is an excellent illustration of how fragmentation in this specific area hurts community. Not only you are forced to use 3 different tools (ghc-mod, hdevtools, cabal) each of which basically provides the same service but in an incompatible and incomplete way. But you decided to create yet another one instead of contributing to the ones you already use (they are all open source)

I'm not sure what you mean to say; ghc-mod and cabal certainly do not do the same. Hsdev doesn't either, and claiming that we don't contribute to these projects is strange.

While I do not condone Mr. Verdi's verbal bashing of the efforts of the Haskell community to provide development tools, I do feel his pain. There is quite a bit of divided effort which seems to result in having multiple inferior solutions. My all-time favorite is ghc-mod because "it just works", but it can be very slow for large projects, especially involving template-haskell (and there are no easy fixes for that). HDevTools, in theory, is great: a persistent server, allowing incremental builds/checks etc., but it's oblivious to .cabal files, let alone sandboxes, which makes it completely impractical for any degree of serious Haskell development (on that note, how come pull request 30 [1] is still open?! /rant). Finally, GHCi is supposed to be enough for IDE support in editors. However, as I understand it, the problem is that its output is not easily parsed and, hence, consumed by editors; on top of that, using it with .cabal files and sandboxes requires calling 'cabal repl', so using another tool. So, maybe, GHCi could be augmented with an easily consumed output format (like JSON), and 'cabal' be given support for that mode as well? Since it's that time of the year again (I mean, the GSoC proposal time :)), maybe, someone would look into this idea? [1] https://github.com/bitc/hdevtools/pull/30 /Andrey On 03/10/2014 03:44 PM, Vagif Verdi wrote:

Sorry, i meant ghci not cabal.

As for hsdev, in order to provide its features it will have to load and keep track of entire project information which is already done by ghc-mod and hdevtools separately. So you ushould see how much effort is wasted on doing the same thing again and again in each of 4 different tools (ghci, ghc-mod, hdevtools and hsdev)

On Monday, March 10, 2014 12:32:25 PM UTC-7, Niklas Hambüchen wrote:

On 10/03/14 18:41, Vagif Verdi wrote: > This is an excellent illustration of how fragmentation in this specific > area hurts community. Not only you are forced to use 3 different tools > (ghc-mod, hdevtools, cabal) each of which basically provides the same > service but in an incompatible and incomplete way. But you decided to > create yet another one instead of contributing to the ones you already > use (they are all open source)

I'm not sure what you mean to say; ghc-mod and cabal certainly do not do the same. Hsdev doesn't either, and claiming that we don't contribute to these projects is strange. _______________________________________________ Haskell-Cafe mailing list Haskel...@haskell.org javascript: http://www.haskell.org/mailman/listinfo/haskell-cafe http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

If we go down that road, it's not cabal-repl that would need to be extended. AFAIK, it's just a wrapper around GHCi that calls that latter with package info extracted from the cabal file and, optionally, the sandbox. So, really, it's GHCi that would need to be extended. I think, something as simple as adding an option to output JSON (or some other structured format with wide support in editors) instead of plain text would go a long way to allow editors to consume it's output: they could just pipe commands via stdin and read the JSON output via stdout, parse it and display the info in the buffer. Depending on how GHCi looks inside and how amenable it is to such modifications, it might actually be a doable summer project. Am I missing any technical gotchas? /Andrey On 03/10/2014 05:11 PM, Michal Antkiewicz wrote:

Cabal repl is absolutely great! Are there any reasons why it couldn't provide all information that ghc-mod and hdevtools do? It could be run as a background process, right? It knows about sandboxes as well.

That would be a great SoC project - extend cabal repls to subsume ghc-mod and hdevtools.

Michal

On Mon, Mar 10, 2014 at 4:59 PM, Andrey Chudnov mailto:achudnov@gmail.com> wrote:

While I do not condone Mr. Verdi's verbal bashing of the efforts of the Haskell community to provide development tools, I do feel his pain. There is quite a bit of divided effort which seems to result in having multiple inferior solutions.

My all-time favorite is ghc-mod because "it just works", but it can be very slow for large projects, especially involving template-haskell (and there are no easy fixes for that).

HDevTools, in theory, is great: a persistent server, allowing incremental builds/checks etc., but it's oblivious to .cabal files, let alone sandboxes, which makes it completely impractical for any degree of serious Haskell development (on that note, how come pull request 30 [1] is still open?! /rant).

Finally, GHCi is supposed to be enough for IDE support in editors. However, as I understand it, the problem is that its output is not easily parsed and, hence, consumed by editors; on top of that, using it with .cabal files and sandboxes requires calling 'cabal repl', so using another tool.

So, maybe, GHCi could be augmented with an easily consumed output format (like JSON), and 'cabal' be given support for that mode as well? Since it's that time of the year again (I mean, the GSoC proposal time :)), maybe, someone would look into this idea?

[1] https://github.com/bitc/hdevtools/pull/30

/Andrey

On 03/10/2014 03:44 PM, Vagif Verdi wrote:

...

Sorry, i meant ghci not cabal.

As for hsdev, in order to provide its features it will have to load and keep track of entire project information which is already done by ghc-mod and hdevtools separately. So you ushould see how much effort is wasted on doing the same thing again and again in each of 4 different tools (ghci, ghc-mod, hdevtools and hsdev)

On Monday, March 10, 2014 12:32:25 PM UTC-7, Niklas Hambüchen wrote:

On 10/03/14 18:41, Vagif Verdi wrote: > This is an excellent illustration of how fragmentation in this specific > area hurts community. Not only you are forced to use 3 different tools > (ghc-mod, hdevtools, cabal) each of which basically provides the same > service but in an incompatible and incomplete way. But you decided to > create yet another one instead of contributing to the ones you already > use (they are all open source)

I'm not sure what you mean to say; ghc-mod and cabal certainly do not do the same. Hsdev doesn't either, and claiming that we don't contribute to these projects is strange. _______________________________________________ Haskell-Cafe mailing list Haskel...@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org mailto:Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org mailto:Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

I played a bit with GHC API and Cabal (and even implemented .cabal support for hdevtools) and now I see issues with this approach: 1. GHC depends on Cabal (rly?) and this is most likely some old version of Cabal, not the one you use currently (e.g. I use cabal-install and Cabal 1.18, but GHC 7.6.3 -- the latest stable -- depends on Cabal 1.16). This makes it hard to develop tools that both use GHC API and latest Cabal. Hopefully this dependency will be removed in future versions of GHC. 2. Sometimes you get stuck with types and can't get source code compile and GHC API-based tools are of no help in this situation, because they won't be able to provide type for some variable in source file because this source file contains a syntax/type error in some other part. I'm not sure if GHC API can be tweaked to bypass errors and do the best effort of parsing/desugaring of other parts of sources with errors. Maybe some external parser (haskell-src-exts) should be used for source inspection. On Tue, Mar 11, 2014 at 10:25 AM, Carlo Hamalainen < carlo@carlo-hamalainen.net> wrote:

On 10/03/14 22:22, Andrey Chudnov wrote:

If we go down that road, it's not cabal-repl that would need to be extended. AFAIK, it's just a wrapper around GHCi that calls that latter with package info extracted from the cabal file and, optionally, the sandbox. So, really, it's GHCi that would need to be extended. I think, something as simple as adding an option to output JSON (or some other structured format with wide support in editors) instead of plain text would go a long way to allow editors to consume it's output: they could just pipe commands via stdin and read the JSON output via stdout, parse it and display the info in the buffer. Depending on how GHCi looks inside and how amenable it is to such modifications, it might actually be a doable summer project. Am I missing any technical gotchas?

Regarding the data format, ghc-mod uses the GHC API to get the de-sugared and type-checked representation of a module, e.g.

https://github.com/kazu-yamamoto/ghc-mod/blob/master/Language/Haskell/GhcMod...

Or, my tool ghc-imported-from needs to get the "guts" from the GHC API to get the global reader environment:

https://github.com/carlohamalainen/ghc-imported-from/blob/master/Language/Ha...

So my feeling is that we'd need to pass Haskell values back and forth, not just JSON.

Personally I would love it if GHCi was extended in the way that you are suggesting, as it would make my tool much faster. Ditto for ghc-mod.

-- Carlo Hamalainenhttp://carlo-hamalainen.net

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

ghc package depends on exact version of Cabal pacakge. I understand why old package depends on some other old package. I just don't understand why GHC depends on Cabal. On Tue, Mar 11, 2014 at 11:26 AM, Mateusz Kowalczyk

wrote:

On 11/03/14 07:19, Maxim Kulkin wrote:

...

I played a bit with GHC API and Cabal (and even implemented .cabal support for hdevtools) and now I see issues with this approach:

1. GHC depends on Cabal (rly?) and this is most likely some old version of Cabal, not the one you use currently (e.g. I use cabal-install and Cabal 1.18, but GHC 7.6.3 -- the latest stable -- depends on Cabal 1.16). This makes it hard to develop tools that both use GHC API and latest Cabal. Hopefully this dependency will be removed in future versions of GHC.

Actually, 1.16 was the bleeding edge release when 7.6.3 came out so technically it was new. Old compiler depends on an old version, I don't see why you find this odd. 7.8 will ship with a newer version (whether we like this or not).

Having said that, I don't think GHC API depends on Cabal API, does it?

...

2. Sometimes you get stuck with types and can't get source code compile and GHC API-based tools are of no help in this situation, because they won't be able to provide type for some variable in source file because this source file contains a syntax/type error in some other part. I'm not sure if GHC API can be tweaked to bypass errors and do the best effort of parsing/desugaring of other parts of sources with errors. Maybe some external parser (haskell-src-exts) should be used for source inspection.

-- Mateusz K. _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Ben, https://ghc.haskell.org/trac/ghc/ticket/8244 is what you're referring to, I think. JP On Tue, Mar 11, 2014 at 4:34 PM, Ben Gamari wrote:

Maxim Kulkin writes:

...

ghc package depends on exact version of Cabal pacakge. I understand why old package depends on some other old package. I just don't understand why GHC depends on Cabal.

I believe GHC uses a few types from Cabal. Namely InstalledPackageInfo and a few related types,

$ git grep 'import Distribution' compiler compiler/ghci/Linker.lhs:import Distribution.Package hiding (depends, PackageId) compiler/main/Finder.lhs:import Distribution.Text compiler/main/Finder.lhs:import Distribution.Package hiding (PackageId) compiler/main/PackageConfig.hs:import Distribution.InstalledPackageInfo compiler/main/PackageConfig.hs:import Distribution.ModuleName compiler/main/PackageConfig.hs:import Distribution.Package hiding (PackageId) compiler/main/PackageConfig.hs:import Distribution.Text compiler/main/PackageConfig.hs:import Distribution.Version compiler/main/Packages.lhs:import Distribution.InstalledPackageInfo compiler/main/Packages.lhs:import Distribution.InstalledPackageInfo.Binary compiler/main/Packages.lhs:import Distribution.Package hiding (PackageId,depends)

At one point I believe there was a proposal to move these to a separate package but I don't have a reference on hand.

Cheers,

- Ben

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- JP Moresmau http://jpmoresmau.blogspot.com/

HaRe gets around the cabal version mismatch by making sure that the return types from the relevant ghc-mod calls are types that are not exported by cabal. e.g. http://hackage.haskell.org/package/ghc-mod-3.1.4/docs/Language-Haskell-GhcMo... [String] for the target names. This could perhaps be extended to either use a separate library of stable types to be used by both cabal and ghc, or to return e.g. json Alan On Tue, Mar 11, 2014 at 9:19 AM, Maxim Kulkin wrote:

I played a bit with GHC API and Cabal (and even implemented .cabal support for hdevtools) and now I see issues with this approach:

1. GHC depends on Cabal (rly?) and this is most likely some old version of Cabal, not the one you use currently (e.g. I use cabal-install and Cabal 1.18, but GHC 7.6.3 -- the latest stable -- depends on Cabal 1.16). This makes it hard to develop tools that both use GHC API and latest Cabal. Hopefully this dependency will be removed in future versions of GHC.

2. Sometimes you get stuck with types and can't get source code compile and GHC API-based tools are of no help in this situation, because they won't be able to provide type for some variable in source file because this source file contains a syntax/type error in some other part. I'm not sure if GHC API can be tweaked to bypass errors and do the best effort of parsing/desugaring of other parts of sources with errors. Maybe some external parser (haskell-src-exts) should be used for source inspection.

On Tue, Mar 11, 2014 at 10:25 AM, Carlo Hamalainen < carlo@carlo-hamalainen.net> wrote:

...

On 10/03/14 22:22, Andrey Chudnov wrote:

If we go down that road, it's not cabal-repl that would need to be extended. AFAIK, it's just a wrapper around GHCi that calls that latter with package info extracted from the cabal file and, optionally, the sandbox. So, really, it's GHCi that would need to be extended. I think, something as simple as adding an option to output JSON (or some other structured format with wide support in editors) instead of plain text would go a long way to allow editors to consume it's output: they could just pipe commands via stdin and read the JSON output via stdout, parse it and display the info in the buffer. Depending on how GHCi looks inside and how amenable it is to such modifications, it might actually be a doable summer project. Am I missing any technical gotchas?

Regarding the data format, ghc-mod uses the GHC API to get the de-sugared and type-checked representation of a module, e.g.

https://github.com/kazu-yamamoto/ghc-mod/blob/master/Language/Haskell/GhcMod...

Or, my tool ghc-imported-from needs to get the "guts" from the GHC API to get the global reader environment:

https://github.com/carlohamalainen/ghc-imported-from/blob/master/Language/Ha...

So my feeling is that we'd need to pass Haskell values back and forth, not just JSON.

Personally I would love it if GHCi was extended in the way that you are suggesting, as it would make my tool much faster. Ditto for ghc-mod.

-- Carlo Hamalainenhttp://carlo-hamalainen.net

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Unifying all the different tools underlying IDEs has been talked again before, because the pain is real. For example, see https://github.com/fpco/haskell-ide/wiki. I've written about the GHC API in that wiki, and some of the issues that it has for such tools. Currently EclipseFP uses buildwrapper, which is a backend I wrote because I was dissatisfied with scion. It integrates the GHC API and Cabal, so that it aims to give the same results that running cabal configure/build. It usually run as a shortlived executable but for performance reason it is possible to have a long lived session to avoid reloading the modules in GHC all the times. Yes, GHC depends on the Cabal API for some types, which causes untold woes when you use buildwrapper after upgrading Cabal, for example. There is a bug report to remove that dependency but I had no time yet to work on it, so far I've used dynamic-cabal in buildwrapper to allow a newer version of Cabal, at the price of more complexity. But a complete tool would need to integrate both, so it can't be ghci only, otherwise Cabal and GHC would become really stuck together. The GHC API uses some global settings, which means you sometimes have no choice (if stcouldatic flags change) than to respawn a new process. This could probably be improved since it's not really the way a well behaved Haskell program should act. Memory usage is also a concern. So I think there should be two lines of work here: one would be to improve the GHC API and/or the ghci interface, to make it more IDE friendly, the other would be to unify all these tools like scion/buildwrapper/ghc-mod in one tool that would hopefully provide all services an IDE needs, and that would allow easy plugin of additional functionality (so for example HaRe could easily add its refactoring capabilities to it). I would happily switch from buildwrapper to another community supported tool if all the functionality I currently have is kept in one form or another. JP On Tue, Mar 11, 2014 at 9:17 AM, Maxim Kulkin wrote:

The thing is not with types: I would like to use a version of cabal in my tool, but dependency on GHC API (which I also use) dictates me other (older) version of Cabal for an unknown and unobvious reason. I want user of my tool to pick arbitrary Cabal version, the one that he actually uses. I'm even ok to write different implementations to support differencies in Cabal API.

I once googled a mailing discussion on that GHC uses methods for parsing and printing some types like package ids or something and they introduced a dependency for that reason.

On Tue, Mar 11, 2014 at 12:08 PM, AlanKim Zimmerman wrote:

...

HaRe gets around the cabal version mismatch by making sure that the return types from the relevant ghc-mod calls are types that are not exported by cabal.

e.g. http://hackage.haskell.org/package/ghc-mod-3.1.4/docs/Language-Haskell-GhcMo... [String] for the target names.

This could perhaps be extended to either use a separate library of stable types to be used by both cabal and ghc, or to return e.g. json

Alan

On Tue, Mar 11, 2014 at 9:19 AM, Maxim Kulkin wrote:

...

I played a bit with GHC API and Cabal (and even implemented .cabal support for hdevtools) and now I see issues with this approach:

1. GHC depends on Cabal (rly?) and this is most likely some old version of Cabal, not the one you use currently (e.g. I use cabal-install and Cabal 1.18, but GHC 7.6.3 -- the latest stable -- depends on Cabal 1.16). This makes it hard to develop tools that both use GHC API and latest Cabal. Hopefully this dependency will be removed in future versions of GHC.

2. Sometimes you get stuck with types and can't get source code compile and GHC API-based tools are of no help in this situation, because they won't be able to provide type for some variable in source file because this source file contains a syntax/type error in some other part. I'm not sure if GHC API can be tweaked to bypass errors and do the best effort of parsing/desugaring of other parts of sources with errors. Maybe some external parser (haskell-src-exts) should be used for source inspection.

On Tue, Mar 11, 2014 at 10:25 AM, Carlo Hamalainen < carlo@carlo-hamalainen.net> wrote:

...

On 10/03/14 22:22, Andrey Chudnov wrote:

If we go down that road, it's not cabal-repl that would need to be extended. AFAIK, it's just a wrapper around GHCi that calls that latter with package info extracted from the cabal file and, optionally, the sandbox. So, really, it's GHCi that would need to be extended. I think, something as simple as adding an option to output JSON (or some other structured format with wide support in editors) instead of plain text would go a long way to allow editors to consume it's output: they could just pipe commands via stdin and read the JSON output via stdout, parse it and display the info in the buffer. Depending on how GHCi looks inside and how amenable it is to such modifications, it might actually be a doable summer project. Am I missing any technical gotchas?

Regarding the data format, ghc-mod uses the GHC API to get the de-sugared and type-checked representation of a module, e.g.

https://github.com/kazu-yamamoto/ghc-mod/blob/master/Language/Haskell/GhcMod...

Or, my tool ghc-imported-from needs to get the "guts" from the GHC API to get the global reader environment:

https://github.com/carlohamalainen/ghc-imported-from/blob/master/Language/Ha...

So my feeling is that we'd need to pass Haskell values back and forth, not just JSON.

Personally I would love it if GHCi was extended in the way that you are suggesting, as it would make my tool much faster. Ditto for ghc-mod.

-- Carlo Hamalainenhttp://carlo-hamalainen.net

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- JP Moresmau http://jpmoresmau.blogspot.com/

literally just for sharing the code for that parsing afaik. I think theres a ticket for doing this. On Tue, Mar 11, 2014 at 2:48 PM, Bardur Arantsson wrote:

On 2014-03-11 09:20, JP Moresmau wrote:

...

Unifying all the different tools underlying IDEs has been talked again before, because the pain is real. For example, see https://github.com/fpco/haskell-ide/wiki. I've written about the GHC API in that wiki, and some of the issues that it has for such tools. Currently EclipseFP uses buildwrapper, which is a backend I wrote because I was dissatisfied with scion. It integrates the GHC API and Cabal, so that it aims to give the same results that running cabal configure/build. It usually run as a shortlived executable but for performance reason it is possible to have a long lived session to avoid reloading the modules in GHC all the times. Yes, GHC depends on the Cabal API for some types, which causes untold woes when you use buildwrapper after upgrading Cabal, for example. There is a bug report to remove that dependency but I had no time yet to work on it, so far I've used dynamic-cabal in buildwrapper to allow a newer version of Cabal, at the price of more complexity. But a complete tool would need to integrate both, so it can't be ghci only, otherwise Cabal and GHC would become really stuck together. The GHC API uses some global settings, which means you sometimes have no choice (if stcouldatic flags change) than to respawn a new process. This could probably be improved since it's not really the way a well behaved Haskell program should act. Memory usage is also a concern.

So I think there should be two lines of work here: one would be to improve the GHC API and/or the ghci interface, to make it more IDE friendly, the other would be to unify all these tools like scion/buildwrapper/ghc-mod in one tool that would hopefully provide all services an IDE needs, and that would allow easy plugin of additional functionality (so for example HaRe could easily add its refactoring capabilities to it). I would happily switch from buildwrapper to another community supported tool if all the functionality I currently have is kept in one form or another.

Since you've obviously been doing a lot of investigation here, I thought I might ask:

What is GHC using Cabal for in *functional terms*, i.e. what actual functionality is it using? Is it just parsing package information files? What is it using that information for? Would it be possible to supply that information via other mechanisms than GHC snooping in the files? I'm thinking something along the lines of pkg-config here.

(I understand the build system is a different issue, let's just ignore that completely.)

Regards,

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe