On Tue, Aug 17, 2010 at 2:23 PM, Yitzchak Gale <gale@sefer.org> wrote:
Michael Snoyman wrote:
> Regarding the data: you haven't actually quoted any
> statistics about the prevalence of CJK data
True, I haven't seen any - except for Google, which
I don't believe is accurate. I would like to see some
good unbiased data.
To my knowledge the data we have about prevalence of encoding on the web is accurate. We crawl all pages we can get our hands on, by starting at some set of seeds and then following all the links. You cannot be sure that you've reached all web sites as there might be cliques in the web graph but we try our best to get them all. You're unlikely to get a better estimate anywhere else. I doubt few organizations have the machinery required to crawl most of the web.