parallel queries would be helpful too. say i need to load independent data from 3 tables. now that has 3 roundtrips of latency to the DB. if there was a simple way to forkIO PersistentBackend functions and dump the value in an mvar, it might help. would anyone else be interested in this? max On Feb 22, 2011, at 11:53 PM, Michael Snoyman wrote:

Max: Let's address your desire for sharding in a separate thread: I think it's a good idea, and it's a feature I would like to implement. I also think that your approach seems fairly feasible and not too disruptive. I want to make sure that introducing sharding code doesn't negatively affect users not taking advantage of it.

As for joins (and other relational features): we should definitely add some helpers to Persistent. The main questions are:

* What should their type signatures be? * Should we offer optimizations for the SQL backends to use proper JOINs?

For the first, I think we still need to clarify a bit. Anton's code is a good start, but it certainly won't be amenable to SQL-backend optimizations. And I think we *should* strive to produce some efficient SQL code, though I want to avoid pouring too much effort into such an endeavor. Perhaps if we can write some code to handle the most common cases (eg, a 1-to-many relation between two tables) we'll be adhering well to the 80/80 rule.

My ideal would be having some nice high-level functions that handle joins, optimize those to proper SQL joins where possible, and for backends where that is not possible simply do the join logic in Haskell.

I definitely won't have time to get started on this coding for the next few weeks, but if anyone is interested in trying to implement any of the features under discussion, let me know, and I'll be happy to offer any assistance/code review I can.

Michael

On Tue, Feb 22, 2011 at 4:39 PM, Greg Weber wrote:

...

The basic Persistent backend is designed without joins ("web-scale"), and with the idea that it should be SQL agnostic. App-level join functionality is definitely needed. If someone wanted to design a layer on top of it with other SQL features that just the SQL backends can use that is fine. However, this might start re-inventing the wheel. For those interested in SQL joins and otherwise advanced SQL queries, it would be great if they started trying out a better tool for the job, like the recently released DSH [1] or (bitrotting?) HaskellDB. Relational Algebra sql query libraries can be very productive to use for 95% of use cases, even if they aren't "web-scale". [1] http://hackage.haskell.org/package/DSH [2] http://hackage.haskell.org/package/TableAlgebra _______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

I would imagine that Max is saying he'd like to see some built-in functions that automate the forkIO/mvar/tvar code, the same way we are discussing automating the join code. Am I right? Michael On Wed, Feb 23, 2011 at 4:25 PM, Greg Weber wrote:

What is stopping you from using forkIO and a mvar/tvar now? If all the use cases could be thought through, parallel queries might be able to be implemented transparently. But what if you don't want parallel queries- then we end up with a leaky abstraction where you expected a previous query to have already executed. On Tue, Feb 22, 2011 at 8:42 AM, Max Cantor wrote:

...

parallel queries would be helpful too.

say i need to load independent data from 3 tables.  now that has 3 roundtrips of latency to the DB.  if there was a simple way to forkIO PersistentBackend functions and dump the value in an mvar, it might help.

would anyone else be interested in this?

max

On Feb 22, 2011, at 11:53 PM, Michael Snoyman wrote:

...

Max: Let's address your desire for sharding in a separate thread: I think it's a good idea, and it's a feature I would like to implement. I also think that your approach seems fairly feasible and not too disruptive. I want to make sure that introducing sharding code doesn't negatively affect users not taking advantage of it.

As for joins (and other relational features): we should definitely add some helpers to Persistent. The main questions are:

* What should their type signatures be? * Should we offer optimizations for the SQL backends to use proper JOINs?

For the first, I think we still need to clarify a bit. Anton's code is a good start, but it certainly won't be amenable to SQL-backend optimizations. And I think we *should* strive to produce some efficient SQL code, though I want to avoid pouring too much effort into such an endeavor. Perhaps if we can write some code to handle the most common cases (eg, a 1-to-many relation between two tables) we'll be adhering well to the 80/80 rule.

My ideal would be having some nice high-level functions that handle joins, optimize those to proper SQL joins where possible, and for backends where that is not possible simply do the join logic in Haskell.

I definitely won't have time to get started on this coding for the next few weeks, but if anyone is interested in trying to implement any of the features under discussion, let me know, and I'll be happy to offer any assistance/code review I can.

Michael

On Tue, Feb 22, 2011 at 4:39 PM, Greg Weber wrote:

...

The basic Persistent backend is designed without joins ("web-scale"), and with the idea that it should be SQL agnostic. App-level join functionality is definitely needed. If someone wanted to design a layer on top of it with other SQL features that just the SQL backends can use that is fine. However, this might start re-inventing the wheel. For those interested in SQL joins and otherwise advanced SQL queries, it would be great if they started trying out a better tool for the job, like the recently released DSH [1] or (bitrotting?) HaskellDB. Relational Algebra sql query libraries can be very productive to use for 95% of use cases, even if they aren't "web-scale". [1] http://hackage.haskell.org/package/DSH [2] http://hackage.haskell.org/package/TableAlgebra _______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Michael look at this http://www-db.informatik.uni-tuebingen.de/files/publications/ferryhaskell.pd... http://www-db.informatik.uni-tuebingen.de/files/publications/ferryhaskell.pd...Very tempting. 2011/2/22 Michael Snoyman

Max: Let's address your desire for sharding in a separate thread: I think it's a good idea, and it's a feature I would like to implement. I also think that your approach seems fairly feasible and not too disruptive. I want to make sure that introducing sharding code doesn't negatively affect users not taking advantage of it.

As for joins (and other relational features): we should definitely add some helpers to Persistent. The main questions are:

* What should their type signatures be? * Should we offer optimizations for the SQL backends to use proper JOINs?

For the first, I think we still need to clarify a bit. Anton's code is a good start, but it certainly won't be amenable to SQL-backend optimizations. And I think we *should* strive to produce some efficient SQL code, though I want to avoid pouring too much effort into such an endeavor. Perhaps if we can write some code to handle the most common cases (eg, a 1-to-many relation between two tables) we'll be adhering well to the 80/80 rule.

My ideal would be having some nice high-level functions that handle joins, optimize those to proper SQL joins where possible, and for backends where that is not possible simply do the join logic in Haskell.

I definitely won't have time to get started on this coding for the next few weeks, but if anyone is interested in trying to implement any of the features under discussion, let me know, and I'll be happy to offer any assistance/code review I can.

Michael

On Tue, Feb 22, 2011 at 4:39 PM, Greg Weber wrote:

...

The basic Persistent backend is designed without joins ("web-scale"), and with the idea that it should be SQL agnostic. App-level join functionality is definitely needed. If someone wanted to design a layer on top of it with other SQL features that just the SQL backends can use that is fine. However, this might start re-inventing the wheel. For those interested in SQL joins and otherwise advanced SQL queries, it would be great if they started trying out a better tool for the job, like the recently released DSH [1] or (bitrotting?) HaskellDB. Relational Algebra sql query libraries can be very productive to use for 95% of use cases, even if they aren't "web-scale". [1] http://hackage.haskell.org/package/DSH [2] http://hackage.haskell.org/package/TableAlgebra _______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

-- Best regards, Cheshkov Anton Phone: +7 909 005 18 82 Phone: +7 931 511 47 37 Skype: cheshkov_anton