Actually, Persistent is fully usable without any special syntax, DSLs, or Template Haskell. In fact, Persistent itself has no template-haskell dependencies, specifically so that it can be built on ghc-iphone. Additionally, the Persistent DSL syntax is completely separate from any other Yesod DSL syntaxes that exist, so it's not like you have to learn five new things to get the automatic code generation. On Mon, Feb 13, 2012 at 9:30 PM, Tom Murphy wrote:

    With respect, I don't think that Persistent is a natural choice as the go-to tool for Haskell users, simply because it requires knowledge of a lot of Yesod-EDSL syntax.     The set of users with persistent data needs seems a very different set than that of those who are familiar with Yesod, and I think the syntax is quite confusing without fuller understanding of Yesod.

    The syntax of acid-state (not familiar with this one), and swapper (https://github.com/roman-smrz/swapper/blob/master/test/) seem to have a much more linear learning curve for an intermediate Haskell user.

amindfv / Tom

On 2/13/12, Greg Weber wrote:

...

Hi Sergiu,

Thanks you for your interest in that proposal. I rushed it off a year ago. Since then we have made a lot of improvements to Persistent and the library forms a basic building block for most Yesod users and other Haskellers. Persistent offers a level of type-safety and convenience not available elsewhere (except perhaps for libraries like acid-state that are limited to in-memory storage). That being said, there are still a lot of improvements that could be made. With the effort of a GSoC volunteer we could probably get it to the point of being the go-to data storage library for Haskellers, at least those planning on using the subset of backends (likely SQL) with great support. This proposal is vague and we would need to work with you to narrow things down a bit.

I am biased, but I believe the Yesod project is one of the most compelling in the Haskell ecosystem. There are a lot of different ways a GSoC project could help make things even better besides improving the associated Persistent library, and we would really like to mentor at least one GSoC student. I would open more tickets for this in the system, but I am not sure how helpful it will be. It seems that we need to reach out to more students like yourself, but I am not sure how to do that unless I see messages like these first.

Greg Weber

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Persistent is a database abstraction layer with no dependencies on Yesod. Those that need a persistence layer have the same needs to interface with the database in a type-safe way, regardless of whether their program presents a web interface. Have you tried using Persistent? We have never heard a report from a user that the Persistent DSL schema syntax is confusing. These complaints always seem to be from someone that hasn't actually tried it but is adverse to quasi-quoting. The DSL is basically the exact same as Haskell record syntax. I am not sure who the mythical users are that can figure out Haskell but can't understand dead-simple DSL's. That being said, I would like to have a Template Haskell interface instead of just a QQ interface. The reason why we haven't bothered with doing that ourselves is because the record name-spacing issue makes the current QQ interface much more convenient. I am actively working to solve the namespace issue. This all brings up a great point though: as part of the GSoC we should create a Template Haskell interface (similar to acid-state as you mention). Given the structure of things that Michael has already pointed out, and that we are already using Template Haskell with the DSL, this should only be a few day's work. On Mon, Feb 13, 2012 at 11:40 AM, Tom Murphy wrote:

It seems that all tutorials and resources for Persistent use Template Haskell along with several Yesod specifics.

But, I could be wrong, or new tutorials could be written.

Tom

On 2/13/12, Michael Snoyman wrote:

...

Actually, Persistent is fully usable without any special syntax, DSLs, or Template Haskell. In fact, Persistent itself has no template-haskell dependencies, specifically so that it can be built on ghc-iphone. Additionally, the Persistent DSL syntax is completely separate from any other Yesod DSL syntaxes that exist, so it's not like you have to learn five new things to get the automatic code generation.

On Mon, Feb 13, 2012 at 9:30 PM, Tom Murphy wrote:

...

    With respect, I don't think that Persistent is a natural choice as the go-to tool for Haskell users, simply because it requires knowledge of a lot of Yesod-EDSL syntax.     The set of users with persistent data needs seems a very different set than that of those who are familiar with Yesod, and I think the syntax is quite confusing without fuller understanding of Yesod.

    The syntax of acid-state (not familiar with this one), and swapper (https://github.com/roman-smrz/swapper/blob/master/test/) seem to have a much more linear learning curve for an intermediate Haskell user.

amindfv / Tom

On 2/13/12, Greg Weber wrote:

...

Hi Sergiu,

Thanks you for your interest in that proposal. I rushed it off a year ago. Since then we have made a lot of improvements to Persistent and the library forms a basic building block for most Yesod users and other Haskellers. Persistent offers a level of type-safety and convenience not available elsewhere (except perhaps for libraries like acid-state that are limited to in-memory storage). That being said, there are still a lot of improvements that could be made. With the effort of a GSoC volunteer we could probably get it to the point of being the go-to data storage library for Haskellers, at least those planning on using the subset of backends (likely SQL) with great support. This proposal is vague and we would need to work with you to narrow things down a bit.

I am biased, but I believe the Yesod project is one of the most compelling in the Haskell ecosystem. There are a lot of different ways a GSoC project could help make things even better besides improving the associated Persistent library, and we would really like to mentor at least one GSoC student. I would open more tickets for this in the system, but I am not sure how helpful it will be. It seems that we need to reach out to more students like yourself, but I am not sure how to do that unless I see messages like these first.

Greg Weber

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Mon, Feb 13, 2012 at 11:52, Greg Weber wrote:

That being said, I would like to have a Template Haskell interface instead of just a QQ interface. The reason why we haven't bothered with doing that ourselves is because the record name-spacing issue makes the current QQ interface much more convenient. I am actively working to solve the namespace issue. This all brings up a great point though: as part of the GSoC we should create a Template Haskell interface (similar to acid-state as you mention). Given the structure of things that Michael has already pointed out, and that we are already using Template Haskell with the DSL, this should only be a few day's work.

I'm joining this thread slightly late, but thought I point to a similar TH approach: http://hackage.haskell.org/package/structured-mongoDB The implementation is very much MongoDB-specific, but we're working on targeting different backends (package is on github, so we welcome other hackers). Of course, some of the issues that have been brought up (e.g., no support for projection) still exist in structured-mongoDB, but our goals have been more relaxed: 1) provide a type-safe interface to MongoDB, 2) to avoid QQ DSL approach if possible, and 3) support a HaskellDB-like query interface.

Hi Greg,

Thanks for bringing this to our attention - this is a neat project! It actually looks *exactly* like persistent - I can't actually discern a meaningful difference, although likely your internals are slightly simpler if you only support MongoDB. If your goals are to support multiple backends then it seems your goals are *exactly* the same as Persistent. Lets collaborate on this off of the mail list.

Great, I think collaborating on this will be useful! (Posting this on cafe to reflect the github url.)

The url listed on hackage for the git repo points to Stanford, not github.

Here is the github mirror: https://github.com/deian/structued-mongoDB Thanks, Deian

On 2/13/12, Paul R wrote: [...]

Abstraction over high level data stores is one of the worst idea in software engineering.

The most proeminent example is probably PostgreSQL, which is an incredibly strong product with high SQL power. But as soon as you access it through the ActiveRecord or Persistent API, it gets turned into a very limited store, with the SQL power of SQLITE or MongoDB.

"Limited" /= "Worst", though [0]. The popularity of SQLite and "NoSQL" prove that sometimes a limited feature set is worth the gains in abstraction. Definitely not for every project, of course. Tom [0] Prelude> "limited" == "worst" False

On Mon, Feb 13, 2012 at 9:53 PM, Paul R wrote:

So Sergiu, my POV is that universal data stores is at best a glue targeting small projects, so that they can be hacked quickly. They offer a set of features that, by design, is the greatest common divisor of the backends, which unfortunately isn't that great. This is certainly nice for a do-a-blog-in-5-minutes with "MyFramework init" video tutorial, but probably not for industrial projects in the long run.

[...]

Regarding your other option, the value behind the LLVM backend seems huge for the whole Haskell community. It has the power to lighten GHC, improve runtime performance, bring binaries to more platforms and much more. In my opinion, that's quiet exciting :)

I am absolutely not in the position to question or deny your conclusion. Obviously, I am not going to drop my commitment to the LLVM backend; however, I tried to find a second project idea that would interest me, and the idea of working on a universal data storage interface seemed quite attractive. It still seems attractive to me; mainly (but not only) because Greg said it could actually run of a purely Haskell backend, which means that, ideally, one could write Haskell programs with very little effort dedicated to storage. Sergiu

Hello Michael,

From Persistent documentation :

Persistent follows the guiding principles of type safety and concise, declarative syntax. Some other nice features are: * Database-agnostic. There is first class support for PostgreSQL, SQLite and MongoDB, with experimental CouchDB and MySQL support in the works. * By being non-relational in nature, we simultaneously are able to support a wider number of storage layers and are not constrained by some of the performance bottlenecks incurred through joins. * A major source of frustration in dealing with SQL databases is changes to the schema. Persistent can automatically perform database migrations. You raise a good point mentioning that persistent has raw queries facilities. That certainly makes is suitable for complex queries, but as soon as you do so, you are not anymore store-agnostic, so this part of Persistent becomes irrelevant in any project using SQL features specific to the store (all of mines, what about yours Michael ?). That makes the first two points not very convincing. The third point does not make much sense to me. Here we spend a lot of time designing efficient SQL schemas and migrations so that the store stays fast and migrations are safe. That's not the kind of job that can be automatically derived by a program by just looking at two models descriptions. Back to the feature of persistent raised by Greg, which is serializing to and from the database with type-safety, I agree that this is desirable. On Mon, 13 Feb 2012 22:36:17 +0200, Michael Snoyman said: Michael> You make it sound like your options are "use the crippled abstraction Michael> layer" or "use the full-powered database layer." You're leaving out Michael> two very important points: Michael> 1. There's a reason the abstraction layer exists: it can be clumsy to Michael> go directly to the full-powered database for simple stuff. Michael> 2. You can bypass the abstraction layer whenever you want. Michael> I like to describe Persistent's goal as doing 95% of what you need, Michael> and getting out of your way for the other 5%. You can write raw SQL Michael> queries with Persistent. I use this for implementing full-text search. Michael> I haven't needed to write deep analytical tools recently, but if I Michael> did, I would use raw SQL for that too. Michael> Persistent's advantage over going directly to the database is concise, Michael> type-safe code. Are you really telling me that `runSql "SELECT * FROM Michael> foo where id=?" [someId]` plus a bunch of marshal code is better then Michael> `get someId`? Michael> Michael Michael> On Mon, Feb 13, 2012 at 9:53 PM, Paul R wrote:

...

I have quiet a lot of experience in the business of web services strongly backed by data stores, in a company that allowed me to apply a technologies such as RubyOnRails, DataMapper, PostgreSQL, Redis, Riak, HappStack and Snap. Greg, with no offense intended, I will share with the café a conclusion that we took year to draw, but that made out job much better since : Abstraction over high level data stores is one of the worst idea in software engineering.

The most proeminent example is probably PostgreSQL, which is an incredibly strong product with high SQL power. But as soon as you access it through the ActiveRecord or Persistent API, it gets turned into a very limited store, with the SQL power of SQLITE or MongoDB.

So Sergiu, my POV is that universal data stores is at best a glue targeting small projects, so that they can be hacked quickly. They offer a set of features that, by design, is the greatest common divisor of the backends, which unfortunately isn't that great. This is certainly nice for a do-a-blog-in-5-minutes with "MyFramework init" video tutorial, but probably not for industrial projects in the long run.

As a side note, the acid-state package that Greg kindly mentioned, takes a very different approach to data storage, resulting in a haskell-centric solution with an original features set.

Regarding your other option, the value behind the LLVM backend seems huge for the whole Haskell community. It has the power to lighten GHC, improve runtime performance, bring binaries to more platforms and much more. In my opinion, that's quiet exciting :)

Greg> Hi Sergiu, Greg> Thanks you for your interest in that proposal. I rushed it off a year Greg> ago. Since then we have made a lot of improvements to Persistent and Greg> the library forms a basic building block for most Yesod users and Greg> other Haskellers. Persistent offers a level of type-safety and Greg> convenience not available elsewhere (except perhaps for libraries like Greg> acid-state that are limited to in-memory storage). That being said, Greg> there are still a lot of improvements that could be made. With the Greg> effort of a GSoC volunteer we could probably get it to the point of Greg> being the go-to data storage library for Haskellers, at least those Greg> planning on using the subset of backends (likely SQL) with great Greg> support. This proposal is vague and we would need to work with you to Greg> narrow things down a bit.

...

Greg> I am biased, but I believe the Yesod project is one of the most Greg> compelling in the Haskell ecosystem. There are a lot of different ways Greg> a GSoC project could help make things even better besides improving Greg> the associated Persistent library, and we would really like to mentor Greg> at least one GSoC student. I would open more tickets for this in the Greg> system, but I am not sure how helpful it will be. It seems that we Greg> need to reach out to more students like yourself, but I am not sure Greg> how to do that unless I see messages like these first.

...

Greg> Greg Weber

...

--  Paul

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Paul

Paul, I appreciate your experience and you might find it interesting that I have come to similar conclusions from my own experience! As Tom commented though, just because something is limited does not mean it is bad for every project. There are many small scale projects that do just fine with about any database abstraction layer, and are probably better off for it. And as Michael said, many find it a good approach to use a database abstraction layer for some of their code and then go raw for other parts of it. like-wise, the acid-state project (which I actually think is very similar to Persistent, but it only stores in the process memory) is a nice approach until the point that you run out of RAM. So lets assume our project has very demanding needs from our persistence layer, and we don't want to ever have to spend time re-writing an abstraction-layer query into a raw form. Does that mean it is impossible to write anything more than a database driver? I still believe there are several important areas that a database layer can be awesome for. The first is proper serialization - converting between Haskell values and database value. We have re-factored Persistent to separate this out from the querying. So you can use Persistent's serilization layer without using its query layer. The new rawSql function lets you write raw queries and get back real Haskell values. The second aspect is declaring the schema and leveraging that to help make things more type-safe. This is even more important in a schema-less data store like MongoDB. Using raw strings means you are a typo away from silent error. Declaring your schema in a database layer creates the possibility of catching these error in your code. This is less important overall in SQL, but Persistent still has your back by checking the schema and telling you what you need to migrate. But I want to take this a step further in Haskell: I want to know at compile time that my query is valid, even if I want to write it in a raw form. Something like Persistent is needed to make sure the columns referenced are correct, but we also want to know that the entire query is correct. There are some interesting but immature efforts in this area, which is yet another way that Persistent could be improved. On Mon, Feb 13, 2012 at 11:53 AM, Paul R wrote:

I have quiet a lot of experience in the business of web services strongly backed by data stores, in a company that allowed me to apply a technologies such as RubyOnRails, DataMapper, PostgreSQL, Redis, Riak, HappStack and Snap. Greg, with no offense intended, I will share with the café a conclusion that we took year to draw, but that made out job much better since : Abstraction over high level data stores is one of the worst idea in software engineering.

The most proeminent example is probably PostgreSQL, which is an incredibly strong product with high SQL power. But as soon as you access it through the ActiveRecord or Persistent API, it gets turned into a very limited store, with the SQL power of SQLITE or MongoDB.

So Sergiu, my POV is that universal data stores is at best a glue targeting small projects, so that they can be hacked quickly. They offer a set of features that, by design, is the greatest common divisor of the backends, which unfortunately isn't that great. This is certainly nice for a do-a-blog-in-5-minutes with "MyFramework init" video tutorial, but probably not for industrial projects in the long run.

As a side note, the acid-state package that Greg kindly mentioned, takes a very different approach to data storage, resulting in a haskell-centric solution with an original features set.

Regarding your other option, the value behind the LLVM backend seems huge for the whole Haskell community. It has the power to lighten GHC, improve runtime performance, bring binaries to more platforms and much more. In my opinion, that's quiet exciting :)

Greg> Hi Sergiu, Greg> Thanks you for your interest in that proposal. I rushed it off a year Greg> ago. Since then we have made a lot of improvements to Persistent and Greg> the library forms a basic building block for most Yesod users and Greg> other Haskellers. Persistent offers a level of type-safety and Greg> convenience not available elsewhere (except perhaps for libraries like Greg> acid-state that are limited to in-memory storage). That being said, Greg> there are still a lot of improvements that could be made. With the Greg> effort of a GSoC volunteer we could probably get it to the point of Greg> being the go-to data storage library for Haskellers, at least those Greg> planning on using the subset of backends (likely SQL) with great Greg> support. This proposal is vague and we would need to work with you to Greg> narrow things down a bit.

Greg> I am biased, but I believe the Yesod project is one of the most Greg> compelling in the Haskell ecosystem. There are a lot of different ways Greg> a GSoC project could help make things even better besides improving Greg> the associated Persistent library, and we would really like to mentor Greg> at least one GSoC student. I would open more tickets for this in the Greg> system, but I am not sure how helpful it will be. It seems that we Greg> need to reach out to more students like yourself, but I am not sure Greg> how to do that unless I see messages like these first.

Greg> Greg Weber

--  Paul

On 2/13/12, Sergiu Ivanov wrote: [...]

a stable, flexible, and easy-to-work-with, already existing storage interface should allow Haskell programmers to focus less on IO and more on the purely functional logic.

+1 - Very exciting! Tom