Actually, the Primitive.Vector constructor should be irrelevant. Nevermind. It is the mutable one you need and that one is public. On Thu, Nov 21, 2013 at 11:11 AM, Edward Kmett wrote:

I'd like to hash this out a bit further. As this issue affects vector primarily, it is outside of the scope of the core library committee per se, and is largely up to Roman as the maintainer of vector.

That said if I put on my personal and not my 'committee' hat:

I would use these operations, personally, were they available to me.

The question is how to best expose them from an API perspective.

Vector currently exposes enough (or at least almost enough) of the implementation details that such support could be implemented 'out-of-line' from the main library. (I think the Data.Vector.Primitive constructor is currently not exported, and you'd need that.)

I don't *think* Roman would object too strenuously to exporting it though and I've personally wanted it for other reasons. This would open the door to this sort of thing being something you might supply entirely outside of the library, giving the final placement of the code some flexibility.

You could handle CAS operations for Primitive and boxed Vectors pretty easily at least, leaving off Unboxed and my various hybrid vector approaches as they are intrinsically tied to working on multiple underlying source vectors.

I'm pretty open to ideas about where the instances should live though. One argument for putting it in vector is you could extend Prim with the CAS operations directly rather than subclass it. An argument against is it could then iterate on a more flexible time table.

-Edward

On Thu, Nov 21, 2013 at 10:30 AM, Ryan Newton wrote:

...

[Popping up to the big-picture question for a moment, from the previous thread on Vector CAS.]

Given that a lot of people are really advertising Haskell on the parallel/concurrent front, it seems like we *should* be committed as a community to making our basic libraries support parallel/concurrent use cases effectively. This will necessarily mean touching some core libraries (container, vector...).

The two things that have come up for me recently are (1) CAS/RMW on [unboxed] Vectors and a general notion of balanced-splitting for tree-based structures (Map,Set). But I'm sure there are others!

Is any one else invested in getting this kind of thing moving? It would be great to have a partner. I can commit a small amount of cycles, but I don't have lots of bandwidth for non-research infrastructure work presently. Edward, is any of this useful to you?

In many of these cases it seems like the main problem is not a technical one, but the social one of consensus building (and fighting excessive "stop energy" if it appears). Edward, as chair of the core libraries committee, maybe you could help with this?

Cheers, -Ryan

On Thu, Nov 21, 2013 at 10:25 AM, Ryan Newton wrote:

...

[@Aleksey] Very good point! I'd missed that vector does the SOA->AOS business too. (I thought that was just Repa & Accelerate.)

In any case, an unboxed vector of tuples can't *actually* support atomic CAS anyway, so not supporting it is inevitable. It does mean that exposeMutableByteArray can't be a method of the Unbox class. It would need to be another class capturing the subset of vector types that are "Atomic" (support concurrent/atomic memory ops).

hehe, those are some of the other operations that llvm would let us write pretty easily! Do we have those on the inline atomics ticket? if not could you add those and any other ideas there? https://ghc.haskell.org/trac/ghc/ticket/7883 @ryan, as a stopgap, wouldn't just wrapping some unsafe ffi c calls around the analogous C intrinsics get you those primops? Its really easy to go back and forth between the types Ptr a and Addr# (after all, data Ptr a = Ptr Addr# !) cheers -Carter On Thu, Nov 21, 2013 at 4:00 PM, Ryan Newton wrote:

Yep, it is public! So it is fine to do this to rip the MutableByteArray out of the relevant "MVector s a" types:

https://github.com/rrnewton/haskell-lockfree/blob/cc2690f96ed616f9d1221770c5...

And then we can build up a "vector-atomics" package that exposes all the relevant ops on [un]boxed vectors (and Storable too if we want to use the bits-atomic package).

Personally, right this moment I need fetch-and-OR and fetch-and-AND because I want a concurrent hierarchical bit-vector for storing subsets of the integers [0,N). Edward, I don't suppose you have that one up your sleeve already ;-)?

-Ryan

On Thu, Nov 21, 2013 at 11:29 AM, Edward Kmett wrote:

...

Actually, the Primitive.Vector constructor should be irrelevant. Nevermind. It is the mutable one you need and that one is public.

On Thu, Nov 21, 2013 at 11:11 AM, Edward Kmett wrote:

...

I'd like to hash this out a bit further. As this issue affects vector primarily, it is outside of the scope of the core library committee per se, and is largely up to Roman as the maintainer of vector.

That said if I put on my personal and not my 'committee' hat:

I would use these operations, personally, were they available to me.

The question is how to best expose them from an API perspective.

Vector currently exposes enough (or at least almost enough) of the implementation details that such support could be implemented 'out-of-line' from the main library. (I think the Data.Vector.Primitive constructor is currently not exported, and you'd need that.)

I don't *think* Roman would object too strenuously to exporting it though and I've personally wanted it for other reasons. This would open the door to this sort of thing being something you might supply entirely outside of the library, giving the final placement of the code some flexibility.

You could handle CAS operations for Primitive and boxed Vectors pretty easily at least, leaving off Unboxed and my various hybrid vector approaches as they are intrinsically tied to working on multiple underlying source vectors.

I'm pretty open to ideas about where the instances should live though. One argument for putting it in vector is you could extend Prim with the CAS operations directly rather than subclass it. An argument against is it could then iterate on a more flexible time table.

-Edward

On Thu, Nov 21, 2013 at 10:30 AM, Ryan Newton wrote:

...

[Popping up to the big-picture question for a moment, from the previous thread on Vector CAS.]

Given that a lot of people are really advertising Haskell on the parallel/concurrent front, it seems like we *should* be committed as a community to making our basic libraries support parallel/concurrent use cases effectively. This will necessarily mean touching some core libraries (container, vector...).

The two things that have come up for me recently are (1) CAS/RMW on [unboxed] Vectors and a general notion of balanced-splitting for tree-based structures (Map,Set). But I'm sure there are others!

Is any one else invested in getting this kind of thing moving? It would be great to have a partner. I can commit a small amount of cycles, but I don't have lots of bandwidth for non-research infrastructure work presently. Edward, is any of this useful to you?

In many of these cases it seems like the main problem is not a technical one, but the social one of consensus building (and fighting excessive "stop energy" if it appears). Edward, as chair of the core libraries committee, maybe you could help with this?

Cheers, -Ryan

On Thu, Nov 21, 2013 at 10:25 AM, Ryan Newton wrote:

...

[@Aleksey] Very good point! I'd missed that vector does the SOA->AOS business too. (I thought that was just Repa & Accelerate.)

In any case, an unboxed vector of tuples can't *actually* support atomic CAS anyway, so not supporting it is inevitable. It does mean that exposeMutableByteArray can't be a method of the Unbox class. It would need to be another class capturing the subset of vector types that are "Atomic" (support concurrent/atomic memory ops).

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

Since vector is needed to build ghc with dph, and has gone fallow, I suppose it really does need a maintainer. I've personally spent a lot of time mucking around in its internals building hybrid-vector and vector-instances and many of my recent projects center on the guts of vector. To that end, I've forked vector over to my github account. If nobody has any objections, we can use that as a sort of staging ground for this sort of change before they filter up to the main repository. I'm happy to juggle whatever integration work is required as an individual maintainer, or to work through the committee if folks really prefer, just to make sure everything continues to work well, but I think things like the recent change to SPEC in HEAD, future SIMD integration via generalized stream fusion using Geoffrey's work, and this CAS proposal will require somewhat active management. That said, if someone else feels strongly about wanting to take over maintainership personally (Geoff?), I'm happy to have the conversation. -Edward On Thu, Nov 21, 2013 at 6:37 PM, Edward Kmett wrote:

I have several bit-vector variants floating around in various side-projects, such as github.com/ekmett/succinct and github.com/ekmett/structures but I don't currently have HBVs per se. I have some code that rolls up aggregations by that same complete tree structure though to build range-min-max trees and the like.

On Thu, Nov 21, 2013 at 4:00 PM, Ryan Newton wrote:

...

Yep, it is public! So it is fine to do this to rip the MutableByteArray out of the relevant "MVector s a" types:

https://github.com/rrnewton/haskell-lockfree/blob/cc2690f96ed616f9d1221770c5...

And then we can build up a "vector-atomics" package that exposes all the relevant ops on [un]boxed vectors (and Storable too if we want to use the bits-atomic package).

Personally, right this moment I need fetch-and-OR and fetch-and-AND because I want a concurrent hierarchical bit-vector for storing subsets of the integers [0,N). Edward, I don't suppose you have that one up your sleeve already ;-)?

-Ryan

On Thu, Nov 21, 2013 at 11:29 AM, Edward Kmett wrote:

...

Actually, the Primitive.Vector constructor should be irrelevant. Nevermind. It is the mutable one you need and that one is public.

On Thu, Nov 21, 2013 at 11:11 AM, Edward Kmett wrote:

...

I'd like to hash this out a bit further. As this issue affects vector primarily, it is outside of the scope of the core library committee per se, and is largely up to Roman as the maintainer of vector.

That said if I put on my personal and not my 'committee' hat:

I would use these operations, personally, were they available to me.

The question is how to best expose them from an API perspective.

Vector currently exposes enough (or at least almost enough) of the implementation details that such support could be implemented 'out-of-line' from the main library. (I think the Data.Vector.Primitive constructor is currently not exported, and you'd need that.)

I don't *think* Roman would object too strenuously to exporting it though and I've personally wanted it for other reasons. This would open the door to this sort of thing being something you might supply entirely outside of the library, giving the final placement of the code some flexibility.

You could handle CAS operations for Primitive and boxed Vectors pretty easily at least, leaving off Unboxed and my various hybrid vector approaches as they are intrinsically tied to working on multiple underlying source vectors.

I'm pretty open to ideas about where the instances should live though. One argument for putting it in vector is you could extend Prim with the CAS operations directly rather than subclass it. An argument against is it could then iterate on a more flexible time table.

-Edward

On Thu, Nov 21, 2013 at 10:30 AM, Ryan Newton wrote:

...

[Popping up to the big-picture question for a moment, from the previous thread on Vector CAS.]

Given that a lot of people are really advertising Haskell on the parallel/concurrent front, it seems like we *should* be committed as a community to making our basic libraries support parallel/concurrent use cases effectively. This will necessarily mean touching some core libraries (container, vector...).

The two things that have come up for me recently are (1) CAS/RMW on [unboxed] Vectors and a general notion of balanced-splitting for tree-based structures (Map,Set). But I'm sure there are others!

Is any one else invested in getting this kind of thing moving? It would be great to have a partner. I can commit a small amount of cycles, but I don't have lots of bandwidth for non-research infrastructure work presently. Edward, is any of this useful to you?

In many of these cases it seems like the main problem is not a technical one, but the social one of consensus building (and fighting excessive "stop energy" if it appears). Edward, as chair of the core libraries committee, maybe you could help with this?

Cheers, -Ryan

On Thu, Nov 21, 2013 at 10:25 AM, Ryan Newton wrote:

...

[@Aleksey] Very good point! I'd missed that vector does the SOA->AOS business too. (I thought that was just Repa & Accelerate.)

In any case, an unboxed vector of tuples can't *actually* support atomic CAS anyway, so not supporting it is inevitable. It does mean that exposeMutableByteArray can't be a method of the Unbox class. It would need to be another class capturing the subset of vector types that are "Atomic" (support concurrent/atomic memory ops).

I'm happy to do that as well if you feel strongly, it just requires more administrivia to coordinate, and the network graph on github shows the activity across repositories anyways, so I tend to work by fork and pull request, as I'm much more comfortable collapsing history on a repository off to on side than I am on the official repo. *shrug* -Edward On Thu, Nov 21, 2013 at 7:04 PM, Bryan O'Sullivan wrote:

On Thu, Nov 21, 2013 at 4:00 PM, Edward Kmett wrote:

...

If nobody has any objections, we can use that as a sort of staging ground for this sort of change before they filter up to the main repository.

I don't have a problem with you working on vector, but why wouldn't you just stage stuff on a branch in the main repo? If you do it in your own repo, that makes it effectively invisible to other people.

On Thu, Nov 21, 2013 at 8:11 AM, Edward Kmett wrote:

I'd like to hash this out a bit further. As this issue affects vector primarily, it is outside of the scope of the core library committee per se, and is largely up to Roman as the maintainer of vector.

Remember that Roman has fairly explicitly resigned as the maintainer of vector for want of time, so someone else needs to take up the crusade here.

I would be happy to share maintainership! Geoff On Thu, Nov 21, 2013 at 11:53:41AM -0500, Edward Kmett wrote:

Fair enough. I had missed that fact! In that case, we could bring vector under the arm of the core libraries committee, or we could seek a more direct maintainer.A [Adding Geoffrey Mainland to the To: list] Geoffrey, would you be willing to pick up maintainership?A I mention it mostly because I know your recent SIMD work has been very vector-centric, and you are the person most likely to affect it's course in the near term. If not, I could pick it up personally, or we could put it under the core libraries committee. -EdwardA

On Thu, Nov 21, 2013 at 11:45 AM, Bryan O'Sullivan wrote:

On Thu, Nov 21, 2013 at 8:11 AM, Edward Kmett wrote:

I'd like to hash this out a bit further. As this issue affectsA vectorA primarily, it is outside of the scope of the core library committee per se, and is largely up to Roman as the maintainer ofA vector.

Remember that Roman has fairly explicitly resigned as the maintainer of vector for want of time, so someone else needs to take up the crusade here.

I'm willing to help out on vector, as I use it quite a lot! I've not done much contrib to it as yet, but I like to think I'm familiar with a good chunk of the code base and that I obsessively care about making sure array based computation in Haskell is a great experience. -carter On Thursday, November 21, 2013, Bryan O'Sullivan wrote:

On Thu, Nov 21, 2013 at 8:11 AM, Edward Kmett

...

wrote:

...

I'd like to hash this out a bit further. As this issue affects vector primarily, it is outside of the scope of the core library committee per se, and is largely up to Roman as the maintainer of vector.

Remember that Roman has fairly explicitly resigned as the maintainer of vector for want of time, so someone else needs to take up the crusade here.