On 5/22/11 8:40 AM, Aleksandar Dimitrov wrote:
If you have too much trouble trying to get SRILM to work, there's also the Berkeley LM which is easier to install. I'm not familiar with its inner workings, but it should offer pretty much the same sorts of operations.
Do you know how BerkeleyLM compares to, say MongoDB and PostgresQL for large data sets? Maybe this is also the wrong list to ask for this kind of question.
Well, BerlekelyLM is specifically for n-gram language modeling, it's not a general database. According to the paper I mentioned off-list, the entire Google Web1T corpus (approx 1 trillion word tokens, 4 billion n-gram types) can be fit into 10GB of memory, which is much smaller than SRILM can do. Databases aren't really my area so I couldn't give a good comparison. Though for this scale of data you're going to want to use something specialized for storing n-grams, rather than a general database. There's a lot of redundant structure in n-gram counts and you'll want to take advantage of that.
For regular projects, that integerization would be enough, but for your task you'll probably want to spend some time tweaking the codes. In particular, you'll probably have enough word types to overflow the space of Int32/Word32 or even Int64/Word64.
Again according to Pauls & Klein (2011), Google Web1T has 13.5M word types, which easily fits into 24-bits. That's for English, so morphologically rich languages will be different. I wouldn't expect too many problems for German, unless you have a lot of technical text with a prodigious number of unique compound nouns. Even then I'd be surprised if you went over 2^64 (that'd be reserved for languages like Japanese, Hungarian, Inuit,... if even they'd ever get that bad). -- Live well, ~wren