since mapReduce assumes that all the code (and the data) is in place in the respective nodes.


As far as I can tell, the Hadoop, the open source implementation of map reduce, doesn't require your map reduce code to be in all nodes.  It copies the jar files of the your application to the nodes that would execute the code. 
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Job+Submission+and+Monitoring

Data is also generally available via a network protocol such as http.  

I am wondering if  distribution tools (such as Cabal) would be preferable to  hadoop model.  Instead of having the framework copy the code why not use the existing tools to install the code prior to running the map-reduce? 

daryoush

2009/2/25 Alberto G. Corona <agocorona@gmail.com>
Galchin,

Maybe you are asking not only about remote execution, but also mobility of code. This is a problem that is previous to mapReduce, since mapReduce assumes that all the code (and the data) is in place in the respective nodes. In fact, the distribution of resources in order to efficiently use mapReduce is a design problem that the google people has done by hand.

But  my intuition says that there are a general algorithm  for distribution of  code, data, bandwidth and resources in general that  moves around at execution time to achieve better and better performance for a given grid of nodes and for any task, for example, a mapReduce task. I would be very interesting to read something about this. 

I know that some efforts have been carried out the past , for example mobile haskell 

http://homepages.inf.ed.ac.uk/stg/workshops/TFP/book/DuBois/duboismhaskell/cameraready.pdf  

which is a first step for this goal but I this has been discontinued and the source code is not available.

2009/2/25 Galchin, Vasili <vigalchin@gmail.com>

>
> Hello,
>
>      Here is an interesting paper of Google's MapReduce reverse engineered into Haskell. I apologize if already posted ..... http://www.cs.vu.nl/~ralf/MapReduce/
>
> Kind regards, Vasili
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe@haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe