The author(s) are from Microsoft .. they claimed to reverse engineer internals. My understanding is that MapReduce internals **is** a Google closely held secret ... hence the open source version is "dumbed down" .. I have forgotten the name. Vasili On Wed, Feb 25, 2009 at 8:49 AM, Eugene Kirpichov wrote:

I've read this article too, and I must say that it is indeed a very interesting and exciting read, both in terms of understanding MapReduce and its capabilities somewhat better, and in terms of beholding the beauty of Haskell.

It is not exactly reverse engineering, but it is expressing the essense of MapReduce algorithms, prerequisites, axioms, dimensions of its design space etc. in Haskell.

2009/2/25 Thomas DuBuisson :

...

Vasili, What do you mean? Googles MapReduce is already a published / well understood concept so no reverse engineering is needed. If you are asking about pre-existing implementations, there is at least one [1] but only for reference, not speed. If you are asking about community interest, great and you might want to say something on the haskell proposals reddit [2].

[1] http://www.cs.vu.nl/~ralf/MapReduce/http://www.cs.vu.nl/%7Eralf/MapReduce/ [2] http://www.reddit.com/r/haskell_proposals

2009/2/24 Galchin, Vasili :

...

Hello,

Here is an interesting paper of Google's MapReduce reverse engineered into Haskell. I apologize if already posted ..... http://www.cs.vu.nl/~ralf/MapReduce/http://www.cs.vu.nl/%7Eralf/MapReduce/

Kind regards, Vasili

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Eugene Kirpichov Web IR developer, market.yandex.ru

I agree. A distributed database could be made as usable as a standard RDBMS by offering an interface by which you supply a map/reduce pair of functions and a list (range?) of keys. This could be easily implemented with a database such as Scalaris, in which the Chord algorithm is responsible for placing and finding the data among nodes. The user would interface with any node in the distributed database, supplying a map and reduce function. It would distribute the map function to nodes of its choosing (weighted by some metrics such as idle cpu), retrieve the intermediate sets and run reduce if supplied. 2009/2/25 Alberto G. Corona :

Galchin,

Maybe you are asking not only about remote execution, but also mobility of code. This is a problem that is previous to mapReduce, since mapReduce assumes that all the code (and the data) is in place in the respective nodes. In fact, the distribution of resources in order to efficiently use mapReduce is a design problem that the google people has done by hand. But  my intuition says that there are a general algorithm  for distribution of  code, data, bandwidth and resources in general that  moves around at execution time to achieve better and better performance for a given grid of nodes and for any task, for example, a mapReduce task. I would be very interesting to read something about this. I know that some efforts have been carried out the past , for example mobile haskell http://homepages.inf.ed.ac.uk/stg/workshops/TFP/book/DuBois/duboismhaskell/c... which is a first step for this goal but I this has been discontinued and the source code is not available.

2009/2/25 Galchin, Vasili

...

Hello,

     Here is an interesting paper of Google's MapReduce reverse engineered into Haskell. I apologize if already posted ..... http://www.cs.vu.nl/~ralf/MapReduce/

Kind regards, Vasili

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- We can't solve problems by using the same kind of thinking we used when we created them. - A. Einstein