I'm going to disable DPH until someone starts maintaining it

Everyone, I'm still working on the AMP, and of course, things aren't going as planned. Why is that? I'm attempting to update `vector` to the latest version from GitHub, as it has some fixes we need for AMP. Mostly, it has some instances we need. Unfortunately, this is basically impossible because dph is locked into a vector fork of ours that we have been maintaining for *months*, and doesn't work with the latest vector upstream (0.11) due to changes in the stream representation. This means I can either: 1) Fix everything in DPH to work with vector 0.11, which is probably going to take a lot of work. 2) Merely fix our fork of vector and let things continue working. This is much easier than #1. Now, you might say that #2 is clearly a preferable solution, and it's very easy - so just do that, Austin! But I don't want to do it. You could say this is the straw that has broken the camel's back. Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain. In fact I'm the only person who's committed to it in *months*, and that has only been to fix breakage. The hackage packages are out of date and sync with what's in the repository (I can't upload them, nor can anyone else besides Ben I believe). So - why are we still building it, exactly? We had a conversation about this months ago. The concern was that things would break and we don't want it to fall out of sync. We're at that point right now - things are breaking, it's out of sync, and it's a pain to keep fixing it, and the actual *benefits* we get from doing so are completely unclear to me. It basically just seems like extra work for nothing, honestly. Unless someone speaks[1] up *very* soon, I'm going to disable DPH during ./validate and the regular build. It will be possible to build it with a '--dph' flag (the dual of the current '--no-dph' flag), although it will be broken very soon with these incoming changes. Providing someone starts fixing it, I'm completely, 100% open to re-enabling it in ./validate by default. But I'm personally tired of fixing it. I'm CC'ing Manuel, Geoff & Ben for their inputs. [1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches. -- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Adding Ben, Roman, Gabi SImon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | Austin Seipp | Sent: 04 August 2014 12:48 | To: ghc-devs@haskell.org | Cc: Manuel Chakravarty; Geoffrey Mainland | Subject: I'm going to disable DPH until someone starts maintaining it | | Everyone, | | I'm still working on the AMP, and of course, things aren't going as | planned. Why is that? | | I'm attempting to update `vector` to the latest version from GitHub, as | it has some fixes we need for AMP. Mostly, it has some instances we | need. | | Unfortunately, this is basically impossible because dph is locked into | a vector fork of ours that we have been maintaining for *months*, and | doesn't work with the latest vector upstream (0.11) due to changes in | the stream representation. This means I can either: | | 1) Fix everything in DPH to work with vector 0.11, which is probably | going to take a lot of work. | 2) Merely fix our fork of vector and let things continue working. | This is much easier than #1. | | Now, you might say that #2 is clearly a preferable solution, and it's | very easy - so just do that, Austin! | | But I don't want to do it. You could say this is the straw that has | broken the camel's back. | | Why? Because I'm afraid I just don't have any more patience for DPH, | I'm tired of fixing it, and it takes up a lot of extra time to build, | and time to maintain. | | In fact I'm the only person who's committed to it in *months*, and that | has only been to fix breakage. The hackage packages are out of date and | sync with what's in the repository (I can't upload them, nor can anyone | else besides Ben I believe). | | So - why are we still building it, exactly? | | We had a conversation about this months ago. The concern was that | things would break and we don't want it to fall out of sync. We're at | that point right now - things are breaking, it's out of sync, and it's | a pain to keep fixing it, and the actual *benefits* we get from doing | so are completely unclear to me. It basically just seems like extra | work for nothing, honestly. | | Unless someone speaks[1] up *very* soon, I'm going to disable DPH | during ./validate and the regular build. It will be possible to build | it with a '--dph' flag (the dual of the current '--no-dph' flag), | although it will be broken very soon with these incoming changes. | | Providing someone starts fixing it, I'm completely, 100% open to re- | enabling it in ./validate by default. But I'm personally tired of | fixing it. | | I'm CC'ing Manuel, Geoff & Ben for their inputs. | | [1] And by 'speak up', I mean I'd like to see someone actively step | forward address my concerns above in a decisive manner. With patches. | | -- | Regards, | | Austin Seipp, Haskell Consultant | Well-Typed LLP, http://www.well-typed.com/ | _______________________________________________ | ghc-devs mailing list | ghc-devs@haskell.org | http://www.haskell.org/mailman/listinfo/ghc-devs

On 4 Aug 2014, at 21:47 , Austin Seipp
Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain.
I'm not going to argue against cutting it lose.
So - why are we still building it, exactly?
It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches.
I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself. The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build. The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off. Sorry for the trouble. Ben.

I have patches for DPH that let it work with vector 0.11 as of a few months ago. I would be happy to submit them via phabricator if that is agreeable (we have to coordinate with the import of vector 0.11 though...I can instead leave them in a wip branch for Austin to merge as he sees fit). I am also willing to commit some time to keep DPH at least working in its current state. Geoff On 8/4/14 8:18 AM, Ben Lippmeier wrote:
On 4 Aug 2014, at 21:47 , Austin Seipp
wrote: Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain. I'm not going to argue against cutting it lose.
So - why are we still building it, exactly? It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches. I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself.
The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build.
The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off.
Sorry for the trouble. Ben.

On Mon, Aug 4, 2014 at 8:49 AM, Geoffrey Mainland
I have patches for DPH that let it work with vector 0.11 as of a few months ago. I would be happy to submit them via phabricator if that is agreeable (we have to coordinate with the import of vector 0.11 though...I can instead leave them in a wip branch for Austin to merge as he sees fit). I am also willing to commit some time to keep DPH at least working in its current state.
That would be quite nice if you could submit patches to get it to work! Thanks so much. As we've moved to submodules, having our own forks is becoming less palatable; we'd like to start tracking upstream closely, and having people submit changes there first and foremost. This creates a bit of a lag time between changes, but I think this is acceptable (and most of our maintainers are quite responsive to GHC needs!) It's also great you're willing to help maintain DPH a bit - but based on what Ben said, it seems like a significant rewrite will happen eventually. Geoff, here's my proposal: 1) I'll disable DPH for right now, so it won't pop up during ./validate. This will probably happen today. 2) We can coordinate the update of vector to 0.11, making it track the official master. (Perhaps an email thread or even Skype would work) 3) We can fix DPH at the same time. 4) Afterwords, we can re-enable it for ./validate If you submit Phabricator patches, that would be fantastic - we can add the DPH repository to Phabricator with little issue. In the long run, I think we should sync up with Ben and perhaps Simon & Co to see what will happen long-term for the DPH libraries.
Geoff
On 8/4/14 8:18 AM, Ben Lippmeier wrote:
On 4 Aug 2014, at 21:47 , Austin Seipp
wrote: Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain. I'm not going to argue against cutting it lose.
So - why are we still building it, exactly? It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches. I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself.
The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build.
The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off.
Sorry for the trouble. Ben.
-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Sounds good to me.
Thanks, Geoff, for doing this!
Manuel
Austin Seipp
On Mon, Aug 4, 2014 at 8:49 AM, Geoffrey Mainland
wrote: I have patches for DPH that let it work with vector 0.11 as of a few months ago. I would be happy to submit them via phabricator if that is agreeable (we have to coordinate with the import of vector 0.11 though...I can instead leave them in a wip branch for Austin to merge as he sees fit). I am also willing to commit some time to keep DPH at least working in its current state.
That would be quite nice if you could submit patches to get it to work! Thanks so much.
As we've moved to submodules, having our own forks is becoming less palatable; we'd like to start tracking upstream closely, and having people submit changes there first and foremost. This creates a bit of a lag time between changes, but I think this is acceptable (and most of our maintainers are quite responsive to GHC needs!)
It's also great you're willing to help maintain DPH a bit - but based on what Ben said, it seems like a significant rewrite will happen eventually.
Geoff, here's my proposal:
1) I'll disable DPH for right now, so it won't pop up during ./validate. This will probably happen today. 2) We can coordinate the update of vector to 0.11, making it track the official master. (Perhaps an email thread or even Skype would work) 3) We can fix DPH at the same time. 4) Afterwords, we can re-enable it for ./validate
If you submit Phabricator patches, that would be fantastic - we can add the DPH repository to Phabricator with little issue.
In the long run, I think we should sync up with Ben and perhaps Simon & Co to see what will happen long-term for the DPH libraries.
Geoff
On 8/4/14 8:18 AM, Ben Lippmeier wrote:
On 4 Aug 2014, at 21:47 , Austin Seipp
wrote: Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain. I'm not going to argue against cutting it lose.
So - why are we still building it, exactly? It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches. I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself.
The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build.
The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off.
Sorry for the trouble. Ben.
-- Regards,
Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Hi Austin, I've pushed wip/dph-fix branches to the dph and ghc repos. dph is not in Phabricator, so I didn't submit anything for review. I think this is small enough that we can probably just merge it directly, but it would be nice to have DPH in Phabricator eventually. I only validated on Linux x64. Is there an easy way for me to validate on other platforms? Thanks, Geoff On 08/04/2014 10:07 AM, Austin Seipp wrote:
On Mon, Aug 4, 2014 at 8:49 AM, Geoffrey Mainland
wrote: I have patches for DPH that let it work with vector 0.11 as of a few months ago. I would be happy to submit them via phabricator if that is agreeable (we have to coordinate with the import of vector 0.11 though...I can instead leave them in a wip branch for Austin to merge as he sees fit). I am also willing to commit some time to keep DPH at least working in its current state. That would be quite nice if you could submit patches to get it to work! Thanks so much.
As we've moved to submodules, having our own forks is becoming less palatable; we'd like to start tracking upstream closely, and having people submit changes there first and foremost. This creates a bit of a lag time between changes, but I think this is acceptable (and most of our maintainers are quite responsive to GHC needs!)
It's also great you're willing to help maintain DPH a bit - but based on what Ben said, it seems like a significant rewrite will happen eventually.
Geoff, here's my proposal:
1) I'll disable DPH for right now, so it won't pop up during ./validate. This will probably happen today. 2) We can coordinate the update of vector to 0.11, making it track the official master. (Perhaps an email thread or even Skype would work) 3) We can fix DPH at the same time. 4) Afterwords, we can re-enable it for ./validate
If you submit Phabricator patches, that would be fantastic - we can add the DPH repository to Phabricator with little issue.
In the long run, I think we should sync up with Ben and perhaps Simon & Co to see what will happen long-term for the DPH libraries.
Geoff
On 8/4/14 8:18 AM, Ben Lippmeier wrote:
On 4 Aug 2014, at 21:47 , Austin Seipp
wrote: Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain. I'm not going to argue against cutting it lose.
So - why are we still building it, exactly? It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches. I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself.
The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build.
The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off.
Sorry for the trouble. Ben.

Hi Austin, I haven't heard back from you about this, and D302 has been waiting for review in phab for two weeks. This seems to be a minor change, so if I don't hear otherwise by the end of the week, I'm going to go ahead and merge. Thanks, Geoff On 08/29/2014 01:29 PM, Geoffrey Mainland wrote:
Hi Austin,
I've pushed wip/dph-fix branches to the dph and ghc repos. dph is not in Phabricator, so I didn't submit anything for review. I think this is small enough that we can probably just merge it directly, but it would be nice to have DPH in Phabricator eventually.
I only validated on Linux x64. Is there an easy way for me to validate on other platforms?
Thanks, Geoff
On 08/04/2014 10:07 AM, Austin Seipp wrote:
On Mon, Aug 4, 2014 at 8:49 AM, Geoffrey Mainland
wrote: I have patches for DPH that let it work with vector 0.11 as of a few months ago. I would be happy to submit them via phabricator if that is agreeable (we have to coordinate with the import of vector 0.11 though...I can instead leave them in a wip branch for Austin to merge as he sees fit). I am also willing to commit some time to keep DPH at least working in its current state. That would be quite nice if you could submit patches to get it to work! Thanks so much.
As we've moved to submodules, having our own forks is becoming less palatable; we'd like to start tracking upstream closely, and having people submit changes there first and foremost. This creates a bit of a lag time between changes, but I think this is acceptable (and most of our maintainers are quite responsive to GHC needs!)
It's also great you're willing to help maintain DPH a bit - but based on what Ben said, it seems like a significant rewrite will happen eventually.
Geoff, here's my proposal:
1) I'll disable DPH for right now, so it won't pop up during ./validate. This will probably happen today. 2) We can coordinate the update of vector to 0.11, making it track the official master. (Perhaps an email thread or even Skype would work) 3) We can fix DPH at the same time. 4) Afterwords, we can re-enable it for ./validate
If you submit Phabricator patches, that would be fantastic - we can add the DPH repository to Phabricator with little issue.
In the long run, I think we should sync up with Ben and perhaps Simon & Co to see what will happen long-term for the DPH libraries.
Geoff
On 8/4/14 8:18 AM, Ben Lippmeier wrote:
On 4 Aug 2014, at 21:47 , Austin Seipp
wrote: Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain. I'm not going to argue against cutting it lose.
So - why are we still building it, exactly? It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches. I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself.
The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build.
The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off.
Sorry for the trouble. Ben.

Hi Geoff,
Sorry about that - this one slipped by my radar amongst a lot of other
things. I'm spending today in the review queue, so I'll get to it.
On Wed, Oct 15, 2014 at 10:43 AM, Geoffrey Mainland
Hi Austin,
I haven't heard back from you about this, and D302 has been waiting for review in phab for two weeks. This seems to be a minor change, so if I don't hear otherwise by the end of the week, I'm going to go ahead and merge.
Thanks, Geoff
On 08/29/2014 01:29 PM, Geoffrey Mainland wrote:
Hi Austin,
I've pushed wip/dph-fix branches to the dph and ghc repos. dph is not in Phabricator, so I didn't submit anything for review. I think this is small enough that we can probably just merge it directly, but it would be nice to have DPH in Phabricator eventually.
I only validated on Linux x64. Is there an easy way for me to validate on other platforms?
Thanks, Geoff
On 08/04/2014 10:07 AM, Austin Seipp wrote:
On Mon, Aug 4, 2014 at 8:49 AM, Geoffrey Mainland
wrote: I have patches for DPH that let it work with vector 0.11 as of a few months ago. I would be happy to submit them via phabricator if that is agreeable (we have to coordinate with the import of vector 0.11 though...I can instead leave them in a wip branch for Austin to merge as he sees fit). I am also willing to commit some time to keep DPH at least working in its current state. That would be quite nice if you could submit patches to get it to work! Thanks so much.
As we've moved to submodules, having our own forks is becoming less palatable; we'd like to start tracking upstream closely, and having people submit changes there first and foremost. This creates a bit of a lag time between changes, but I think this is acceptable (and most of our maintainers are quite responsive to GHC needs!)
It's also great you're willing to help maintain DPH a bit - but based on what Ben said, it seems like a significant rewrite will happen eventually.
Geoff, here's my proposal:
1) I'll disable DPH for right now, so it won't pop up during ./validate. This will probably happen today. 2) We can coordinate the update of vector to 0.11, making it track the official master. (Perhaps an email thread or even Skype would work) 3) We can fix DPH at the same time. 4) Afterwords, we can re-enable it for ./validate
If you submit Phabricator patches, that would be fantastic - we can add the DPH repository to Phabricator with little issue.
In the long run, I think we should sync up with Ben and perhaps Simon & Co to see what will happen long-term for the DPH libraries.
Geoff
On 8/4/14 8:18 AM, Ben Lippmeier wrote:
On 4 Aug 2014, at 21:47 , Austin Seipp
wrote: Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain. I'm not going to argue against cutting it lose.
So - why are we still building it, exactly? It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches. I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself.
The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build.
The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off.
Sorry for the trouble. Ben.
-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

On Mon, Aug 4, 2014 at 7:18 AM, Ben Lippmeier
On 4 Aug 2014, at 21:47 , Austin Seipp
wrote: Why? Because I'm afraid I just don't have any more patience for DPH, I'm tired of fixing it, and it takes up a lot of extra time to build, and time to maintain.
I'm not going to argue against cutting it lose.
So - why are we still building it, exactly?
It can be a good stress test for the simplifier, especially the SpecConstr transform. The fact that it takes so long to build is part of the reason it's a good stress test.
That's definitely fair. There's also the problem that SpecConstr has seen regressions lately causing explosions in the amount of time needed to compile, which might be making this more problematic (I can't remember the ticket # off hand).
[1] And by 'speak up', I mean I'd like to see someone actively step forward address my concerns above in a decisive manner. With patches.
I thought that in the original conversation we agreed that if the DPH code became too much of a burden it was fine to switch it off and let it become unmaintained. I don't have time to maintain it anymore myself.
Oh, I misremembered. It seems we're on the same page then :)
The original DPH project has fractured into a few different research streams, none of which work directly with the implementation in GHC, or with the DPH libraries that are bundled with the GHC build.
The short of it is that the array fusion mechanism implemented in DPH (based on stream fusion) is inadequate for the task. A few people are working on replacement fusion systems that aim to solve this problem, but merging this work back into DPH will entail an almost complete rewrite of the backend libraries. If it the existing code has become a maintenance burden then it's fine to switch it off.
I see, thanks. Is there any current roadmap on what might be done?
Sorry for the trouble. Ben.
No problem. I suppose after dealing with the frustration of tracking a single bug for a few days, this is just an annoyance that tipped me. :) -- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/
participants (5)
-
Austin Seipp
-
Ben Lippmeier
-
Geoffrey Mainland
-
Manuel M T Chakravarty
-
Simon Peyton Jones