On Fri, Feb 15, 2008 at 07:21:55PM +0100, Lemmih wrote:

Greetings,

I've found a few hotspots that'll be working on. I'd be very interested in discussing solutions.

Performance flaws: * IdMaps are used to generate new ids. * Ho files contain huge amounts of duplicate information. * Ho files aren't saved lazily. * C code is used for generating atoms.

Repeatedly mapping variables to 'const Nothing' is very expensive. It is currently the most expensive procedure in Jhc, taking ~20% CPU time when compiling the base library.

Hmm.. yeah, sometimes I used Maps, sometimes Sets, depending on what I already have and sometimes it helped to switch between them, sometimes not. it wasn't always obvious. Ideally all new id selection will be done in Name.Id as part of the general plan to turn Id into a newtype. It has the newIds routine, a couple variants of that to work on IdMap and IdSet would be good. if I am just doing a map (const Nothing) before passing to the id selection routine then that can probably just be dropped since the id selection stuff doesn't care about the actual values in the map. The id selection can be finicky, using Set.size to seed the iterations helped a bunch but I wanted to try a hash function from Id -> Id at some point as it should reduce the time spent linearly probing for an open Id.

The base library contains 65 megabytes of uncompressed data. Most of that is duplicate information that disappears when it is compressed. However, parsing that amount of data takes considerable time.

Which duplicate data in particular is concerning you? I am in the process of completely reorganizing the Ho file layout so it is probably best to hold off here. Some of the redundancy is there on purpose, but most probably isn't.

Ho files are completely serialized before they're written to disk.

Yeah, the issue with fully lazy writing is we need the length of the chunk before we can produce the bytestring, however, it would be straightforward to write a routine that counted the data as it writes it out to disk then seeks back to fill in the length field. It would mean passing in a filehandle rather than getting out a lazy bytestring... but that is a little acceptable uglyness.

Using C code for generating atoms results in no performance improvement. There are only a few cases where using C is beneficial and this is not one of them.

The C code here was mainly about reducing GC pressure. in any case, I'd like to leave it in, I am not sure I have explored all the uses of that cuckoo hash and it is relatively new code. And C is one of my native languages. :) John -- John Meacham - ⑆repetae.net⑆john⑈