On Thu, May 5, 2011 at 1:28 PM, Felipe Almeida Lessa wrote:

On Thu, May 5, 2011 at 2:20 AM, Jeremy Hughes wrote:

...

Is Database.Persistent lazy wrt reading fields? I need to iterate over entities containing both small and large fields. I do not need to use the large fields in this instance, and so would rather they were not read from the database.

IIRC, they are read strictly.  I guess you should put them on a different entity.

That's correct. In fact, Persistent avoids any form of lazy I/O to ensure that database connections are returned to the pool as soon as possible (amongst other reasons). Michael

That's interesting: I'd never considered the idea of inserting undefined into fields you want excluded... That could work. The reason I've avoided putting in this (often requested) feature is that I could think of no way to do so and keep type safety. That might be an approach. We may consider requiring the user to supply the value instead, thereby avoiding the library inserting undefined. Michael On Thu, May 5, 2011 at 4:32 PM, Greg Weber wrote:

An alternative to laziness would be selecting a subset of fields. There is no support for this directly in persistent, but it might be possible to add it and the future and have the value of an unselected field be something like undefined. At the moment you can select sub fields by dropping down to lower-level sql methods (ask Michael about these methods if you are interested). I think there is a technique for building your persistent data structure back up from the return of raw sql, which again you might be able to do by inserting dummy error fields. Greg Weber

On Thu, May 5, 2011 at 4:18 AM, Michael Snoyman wrote:

...

On Thu, May 5, 2011 at 1:28 PM, Felipe Almeida Lessa wrote:

...

On Thu, May 5, 2011 at 2:20 AM, Jeremy Hughes wrote:

...

Is Database.Persistent lazy wrt reading fields? I need to iterate over entities containing both small and large fields. I do not need to use the large fields in this instance, and so would rather they were not read from the database.

IIRC, they are read strictly.  I guess you should put them on a different entity.

That's correct. In fact, Persistent avoids any form of lazy I/O to ensure that database connections are returned to the pool as soon as possible (amongst other reasons).

Michael

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Not sure that I feel *quite* so strongly about it as Max, but I agree with his sentiments here. I'm not convinced that a proper implementation that could deal with undefineds would be particularly error-prone, but I imagine that performance would be better anyways with TH. On Thu, May 5, 2011 at 9:02 PM, Max Cantor wrote:

Personally, I vote against this. Putting in undefineds maintains type-safety. However, in my mind the primary purpose of static type-safety is the avoidance of runtime errors. This only encourages that. Perhaps a better approach is to have some a TH function to generate "subselects" which would be records that only contain the requested data. Its a bit more constraining, but I fear that the alternative is tantamount to curing the disease by killing the patient.

furthermore, I worry that allowing undefined's here will start a slippery, runtime-error-laden slope which we're better off avoiding..

max

On May 5, 2011, at 9:55 PM, Michael Snoyman wrote:

...

That's interesting: I'd never considered the idea of inserting undefined into fields you want excluded... That could work. The reason I've avoided putting in this (often requested) feature is that I could think of no way to do so and keep type safety. That might be an approach.

We may consider requiring the user to supply the value instead, thereby avoiding the library inserting undefined.

Michael

On Thu, May 5, 2011 at 4:32 PM, Greg Weber wrote:

...

An alternative to laziness would be selecting a subset of fields. There is no support for this directly in persistent, but it might be possible to add it and the future and have the value of an unselected field be something like undefined. At the moment you can select sub fields by dropping down to lower-level sql methods (ask Michael about these methods if you are interested). I think there is a technique for building your persistent data structure back up from the return of raw sql, which again you might be able to do by inserting dummy error fields. Greg Weber

On Thu, May 5, 2011 at 4:18 AM, Michael Snoyman wrote:

...

On Thu, May 5, 2011 at 1:28 PM, Felipe Almeida Lessa wrote:

...

On Thu, May 5, 2011 at 2:20 AM, Jeremy Hughes

wrote:

...

...

...

Is Database.Persistent lazy wrt reading fields? I need to iterate over entities containing both small and large fields. I do not need to use the large fields in this instance, and so would rather they were not read from the database.

IIRC, they are read strictly. I guess you should put them on a different entity.

That's correct. In fact, Persistent avoids any form of lazy I/O to ensure that database connections are returned to the pool as soon as possible (amongst other reasons).

Michael

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

I can think of 2 other approaches: 1) lazy IO, but don't hold a connection open- accessing a lazy field creates an entirely new request. This means a change in the type signature of the field accessor to be in IO. 2) A to be excluded field must be nullable, and excluding it will return a Nothing. Instead of a runtime failure, you get a runtime wtf? moment if you don't realize why your data is not being displayed. I didn't mention TH sub-selects idea because originally I didn't see how it was practical. Lets say that we force users to declare the possible sub-selects so that we avoid exponential TH. A first approach is for sub-selects be of the same data type as the full select. data Person = PersonAll String String | PersonJustName String For all your functions that operate on your data, you will now have to pattern match against all the declared permutations, and some of the pattern matches are going to result in runtime errors or dummy values. So it seems they should be different data types. But now we need to have the same functions operate over different types. So we need a bunch of typeclasses that define which fields are operated on. data Person = Person String String data PersonJustName = Person String class PersonName p where personName :: p -> String instance PersonName Person personName (Person name _) = name instance PersonName PersonJustName personName (Person name) = name But I really don't know if the different data types means an entire re-working of Persistent, with even more obscure types. I will let Michael shoot this down :) Greg Weber On Thu, May 5, 2011 at 7:50 PM, Felipe Almeida Lessa

wrote:

On Thu, May 5, 2011 at 11:35 PM, Ian Duncan wrote:

...

Not sure that I feel *quite* so strongly about it as Max, but I agree with his sentiments here. I'm not convinced that a proper implementation that could deal with undefineds would be particularly error-prone, but I imagine that performance would be better anyways with TH.

Although undefined isn't nice, it's kind of difficult to provide a TH solution. For example, if your entity has 20 fields, would you generate all 2^20 different combinations of this-field-is-not-needed datatypes? Would the data types be generated on the call site (this is impossible, however). So should the user specify on the persistent entity declaration that a field may not be needed? Even so, if you said that you may not need 6 fields, would you generate 2^6 different data types. Or should you tell not only which fields, but also which combination of fields? And what names would these data types get?

IOW, this is complicated. =)

The undefined route doesn't seem so bad. Actually, as Michael said, we could have the user pass a default value. Then you could use as default value 'error "fileX.hs, function foo, field zyw"' or something like that, giving you decisive information that you used an undefined field, and which field that was.

Cheers,

-- Felipe.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

How about generating a version of the records lifted to Maybe? Am 06.05.2011 04:03 schrieb "Max Cantor" :

Personally, I vote against this. Putting in undefineds maintains type-safety. However, in my mind the primary purpose of static type-safety is the avoidance of runtime errors. This only encourages that. Perhaps a better approach is to have some a TH function to generate "subselects" which would be records that only contain the requested data. Its a bit more constraining, but I fear that the alternative is tantamount to curing the disease by killing the patient.

furthermore, I worry that allowing undefined's here will start a slippery, runtime-error-laden slope which we're better off avoiding..

max

On May 5, 2011, at 9:55 PM, Michael Snoyman wrote:

...

That's interesting: I'd never considered the idea of inserting undefined into fields you want excluded... That could work. The reason I've avoided putting in this (often requested) feature is that I could think of no way to do so and keep type safety. That might be an approach.

We may consider requiring the user to supply the value instead, thereby avoiding the library inserting undefined.

Michael

On Thu, May 5, 2011 at 4:32 PM, Greg Weber wrote:

...

An alternative to laziness would be selecting a subset of fields. There is no support for this directly in persistent, but it might be possible to add it and the future and have the value of an unselected field be something like undefined. At the moment you can select sub fields by dropping down to lower-level sql methods (ask Michael about these methods if you are interested). I think there is a technique for building your persistent data structure back up from the return of raw sql, which again you might be able to do by inserting dummy error fields. Greg Weber

On Thu, May 5, 2011 at 4:18 AM, Michael Snoyman wrote:

...

On Thu, May 5, 2011 at 1:28 PM, Felipe Almeida Lessa wrote:

...

On Thu, May 5, 2011 at 2:20 AM, Jeremy Hughes

wrote:

...

...

...

Is Database.Persistent lazy wrt reading fields? I need to iterate over entities containing both small and large fields. I do not need to use the large fields in this instance, and so would rather they were not read from the database.

IIRC, they are read strictly. I guess you should put them on a different entity.

That's correct. In fact, Persistent avoids any form of lazy I/O to ensure that database connections are returned to the pool as soon as possible (amongst other reasons).

Michael

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

So a quick review: we have a few general approaches here:

1) Create a few different datatypes, or a single datatype with multiple constructors. 2) Somehow annotate the fields to indicate that data may or may not be

3) Fill in some dummy values for fields not loaded.

I think option (1) is by far the most type safe. It's also the most involved and unintuitive. Greg's point about multiple constructors being ugly is very true. And two datatypes will be hard to manage.

Option (2) will work, but it's significantly less type safe. There's nothing at the type level distinguishing "get" and "getPartial". And it will have a strongly negative impact on all code branches using "get".

Option (3) is not as type-safe as option (1). In it's scariest form, it's going to allow undefined to float around programs wreaking havoc. In its more benign form, it will simply allow the user to provide dummy values. The downside versus option (1) is that there's nothing at the type level distinguishing get from getPartial. The downside versus option (2) is that there is nothing at the value level distinguishing get from getPartial.

But the advantage of (3) is that it's much simpler to implement and will have no effect on regular code at all. I think it's also much easier to understand how to use it to someone.

Anyone see something I'm missing?

Michael

On Fri, May 6, 2011 at 7:49 AM, Aristid Breitkreuz wrote:

...

How about generating a version of the records lifted to Maybe?

Am 06.05.2011 04:03 schrieb "Max Cantor" :

...

Personally, I vote against this. Putting in undefineds maintains type-safety. However, in my mind the primary purpose of static type-safety is the avoidance of runtime errors. This only encourages that. Perhaps a better approach is to have some a TH function to generate "subselects" which would be records that only contain the requested data. Its a bit more constraining, but I fear that the alternative is tantamount to curing

...

...

disease by killing the patient.

furthermore, I worry that allowing undefined's here will start a slippery, runtime-error-laden slope which we're better off avoiding..

max

On May 5, 2011, at 9:55 PM, Michael Snoyman wrote:

...

That's interesting: I'd never considered the idea of inserting undefined into fields you want excluded... That could work. The reason I've avoided putting in this (often requested) feature is that I could think of no way to do so and keep type safety. That might be an approach.

We may consider requiring the user to supply the value instead, thereby avoiding the library inserting undefined.

Michael

On Thu, May 5, 2011 at 4:32 PM, Greg Weber wrote:

...

An alternative to laziness would be selecting a subset of fields. There is no support for this directly in persistent, but it might be possible to add it and the future and have the value of an unselected field be something like undefined. At the moment you can select sub fields by dropping down to lower-level sql methods (ask Michael about these methods if you are interested). I

What about data Foo = X { a :: Int } data FooLifted = XLifted { aLifted :: Maybe Int } As I indicated earlier. Sorry for not being more eloquent on my phone. Aristid Am 06.05.2011 11:48 schrieb "Michael Snoyman" : present. the think

...

...

...

...

there is a technique for building your persistent data structure back up from the return of raw sql, which again you might be able to do by inserting dummy error fields. Greg Weber

On Thu, May 5, 2011 at 4:18 AM, Michael Snoyman wrote:

...

On Thu, May 5, 2011 at 1:28 PM, Felipe Almeida Lessa wrote: > On Thu, May 5, 2011 at 2:20 AM, Jeremy Hughes > wrote: >> Is Database.Persistent lazy wrt reading fields? I need to iterate >> over >> entities containing both small and large fields. I do not need to

use

...

>> the large fields in this instance, and so would rather they were not >> read from the database. > > IIRC, they are read strictly. I guess you should put them on a > different entity.

That's correct. In fact, Persistent avoids any form of lazy I/O to ensure that database connections are returned to the pool as soon as possible (amongst other reasons).

Michael

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

On Fri, May 6, 2011 at 9:48 PM, Michael Snoyman wrote:

So a quick review: we have a few general approaches here:

Thanks for all the answers. I guess I will have to normalize my schema a bit more so large fields need not be fetched. Jeremy