Re: [Haskell-cafe] [haskell.org Google Summer of Code 2013] Approved Projects
Hi Edward, Although the project I am interested in (as a user) has been accepted :-), I can't help feeling the selection process is a bit opaque. Is it documented somewhere and I just missed it? Apologies if I did. BTW I appreciate all the hard work that goes into the selection process. Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com
Hi Dominic, The proposal is admittedly rather unfortunately opaque. The parts I can shed light on: Students come up with proposals with the help of the community and then submit them to google-melange.com. A bunch of folks from the haskell community sign up as potential mentors, vote on and discuss the proposals. (We had ~25 candidate mentors and ~20 proposals this year). The student application template contains a number of desirable criteria for a successful summer of code application, which is shown on the google-melange website under our organization -- an old version is available http://hackage.haskell.org/trac/summer-of-code/wiki/StudApply2012contains Once we have the proposals in hand, and some initial ranking, we ask google for slots. Allocation is based on past performance, arcane community parameters that only they know, mentor ratio, etc. This should be our largest year in the program, despite the fact that in general organizations have been getting fewer slots as more organizations join, so we're apparently doing rather well. In general we do try to select projects that maximize the public good. Most of the time this can almost be done by just straight cut off based on the average score. There is some special casing for duplicate applications between different students and where students have submitted multiple applications we can have some flexibility in how to apply them. This year we also received an extra couple of special-purpose darcs slots from Google in exchange for continuing to act as an umbrella organization over darcs at the request of the administrator of the program at Google. In previous years I had requested an extra slot for them, this year the request came in the other direction. We do inevitably get more good proposals than we get slots. This year we could have easily used another 3-4 slots to good effect. The main part I can't shed light on: Google requests that the final vote tallies remain private. This is done so that students who put in proposals to a high volume orgs and don't get accepted, or who are new to the process and don't quite catch all the rules, don't wind up with any sort of publicly visible black mark. This unfortunately means, that we can't really show the unaccepted proposals with information about how to avoid getting your proposal rejected. I hope that helps. If you have any more questions or if my answer didn't suffice please feel free to follow up! -Edward Kmett On Tue, May 28, 2013 at 6:52 AM, Dominic Steinitz <dominic@steinitz.org>wrote:
Hi Edward,
Although the project I am interested in (as a user) has been accepted :-), I can't help feeling the selection process is a bit opaque. Is it documented somewhere and I just missed it? Apologies if I did.
BTW I appreciate all the hard work that goes into the selection process.
Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com
On 29/05/2013, at 1:11 AM, Edward Kmett wrote:
This unfortunately means, that we can't really show the unaccepted proposals with information about how to avoid getting your proposal rejected.
You can if you rewrite the key points of proposal to retain the overall message, but remove identifying information. I think it would be helpful to write up some of the general reasons for projects being rejected. I tried to do this for Haskell experience reports, on the Haskell Symposium experience report advice page. http://www.haskell.org/haskellwiki/HaskellSymposium/ExperienceReports I'd imagine you could write up some common proposal / rejection / advice tuples like: Proposal: I want to write a MMORPG in Haskell, because this would be a good demonstration for Haskell in a large real world project. We can use this as a platform to develop the networking library infrastructure. Rejection: This project is much too big, and the production of a MMORPG wouldn't benefit the community as a whole. Advice: If you know of specific problems in the networking library infrastructure, then focus on those, using specific examples of where people have tried to do something and failed. Ben.
The majority of the rejections are of precisely that form, or just were slightly out-competed by another similar proposal in their space. These items either are stated or should be stated in the student application guidelines, but a successful summer of code submission probably has most of the following attributes: * *A good timeline* -- It is easy to miss this one! At one extreme, we've had proposals that just never submitted any details beyond a paragraph saying something would be nice to have. Other red flags are projects that promise the moon, but come from someone nobody knows (there is nothing wrong with promising the moon if everyone knows you can deliver it!) or which look like they close out an easy issue that someone could bang out in a weekend. We've passed on otherwise decent proposals that just completely lacked any details about how they were going to get there, as there is really no good way for a mentor to judge progress. Speaking of whom, it is useful to have... * *Some idea of who the mentor could be* -- We came very close to not finding mentors for a couple of projects, despite community interest. As a corollary, if you are interested in having something happen in the community, stepping up to mentor it along is a good first step to making it a reality! We generally can find mentors for projects, but having already reached out to someone beforehand helps show that you have... * *A student who has already started to get involved in the community* -- Haskell generally has a long ramp up, its rather hard to learn it on the fly if you've never really used it, and still accomplish some other goal. Students with a long history of good commits to a project are generally taken before ones of whom we're less sure. Without knowledge of haskell you are unlikely to have... * *Demonstrable impact on the community as a whole* -- Proposals that work on a specific aspect of something we all use generally see good responses. We definitely tend to favor projects that benefit the community as a whole over ones that improve niches. Work on Cabal, GHC, Hackage is a much easier to sell, than say me getting someone to hack on my kan-extensions library, which doesn't have many users, but at least it is... * *Not a greenfield project* -- In general we're a bit gunshy about students designing libraries from scratch. Your MMORPG example fits that mold. It'd likely be a completely new project, which brings with it a lot of design decisions, and it comes with no pre-existing users impacted by the project, so it is easy to flounder and have nobody care. For instance, we have routinely received submissions to build a library for type level programming, but we already have several that have failed to gain much traction. What discriminates the new design from the old? Now, if the library already exists and has begun to pick up traction and has users? You might be able to make a case for a summer of code project to improve it. I just can't think of such a proposal that we've had in recent years. -Edward On Tue, May 28, 2013 at 8:24 PM, Ben Lippmeier <benl@ouroborus.net> wrote:
On 29/05/2013, at 1:11 AM, Edward Kmett wrote:
This unfortunately means, that we can't really show the unaccepted proposals with information about how to avoid getting your proposal rejected.
You can if you rewrite the key points of proposal to retain the overall message, but remove identifying information. I think it would be helpful to write up some of the general reasons for projects being rejected.
I tried to do this for Haskell experience reports, on the Haskell Symposium experience report advice page. http://www.haskell.org/haskellwiki/HaskellSymposium/ExperienceReports
I'd imagine you could write up some common proposal / rejection / advice tuples like:
Proposal: I want to write a MMORPG in Haskell, because this would be a good demonstration for Haskell in a large real world project. We can use this as a platform to develop the networking library infrastructure.
Rejection: This project is much too big, and the production of a MMORPG wouldn't benefit the community as a whole.
Advice: If you know of specific problems in the networking library infrastructure, then focus on those, using specific examples of where people have tried to do something and failed.
Ben.
Hi Edward, Thanks for this comprehensive answer (and also thanks to participants in the follow-up dissuasion). How is the "public good" determined? (sounds rather Benthamite). I would have been disappointed if "charts using diagrams" had not been selected yet I don't recall being canvassed. Sorry to sound picky. I think from what you say that in this particular year it was obvious which projects should be selected; in future it may not be. I think an acceptable reason would be "there was only one user who wanted it". Maybe we should use something like: https://www.uservoice.com. Sadly it seems this requires payment but there may be a free equivalent Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com On 28 May 2013, at 16:11, Edward Kmett <ekmett@gmail.com> wrote:
Hi Dominic,
The proposal is admittedly rather unfortunately opaque.
The parts I can shed light on:
Students come up with proposals with the help of the community and then submit them to google-melange.com.
A bunch of folks from the haskell community sign up as potential mentors, vote on and discuss the proposals. (We had ~25 candidate mentors and ~20 proposals this year).
The student application template contains a number of desirable criteria for a successful summer of code application, which is shown on the google-melange website under our organization -- an old version is available http://hackage.haskell.org/trac/summer-of-code/wiki/StudApply2012 contains
Once we have the proposals in hand, and some initial ranking, we ask google for slots. Allocation is based on past performance, arcane community parameters that only they know, mentor ratio, etc. This should be our largest year in the program, despite the fact that in general organizations have been getting fewer slots as more organizations join, so we're apparently doing rather well.
In general we do try to select projects that maximize the public good. Most of the time this can almost be done by just straight cut off based on the average score. There is some special casing for duplicate applications between different students and where students have submitted multiple applications we can have some flexibility in how to apply them.
This year we also received an extra couple of special-purpose darcs slots from Google in exchange for continuing to act as an umbrella organization over darcs at the request of the administrator of the program at Google. In previous years I had requested an extra slot for them, this year the request came in the other direction.
We do inevitably get more good proposals than we get slots. This year we could have easily used another 3-4 slots to good effect.
The main part I can't shed light on:
Google requests that the final vote tallies remain private. This is done so that students who put in proposals to a high volume orgs and don't get accepted, or who are new to the process and don't quite catch all the rules, don't wind up with any sort of publicly visible black mark. This unfortunately means, that we can't really show the unaccepted proposals with information about how to avoid getting your proposal rejected.
I hope that helps. If you have any more questions or if my answer didn't suffice please feel free to follow up!
-Edward Kmett
On Tue, May 28, 2013 at 6:52 AM, Dominic Steinitz <dominic@steinitz.org> wrote: Hi Edward,
Although the project I am interested in (as a user) has been accepted :-), I can't help feeling the selection process is a bit opaque. Is it documented somewhere and I just missed it? Apologies if I did.
BTW I appreciate all the hard work that goes into the selection process.
Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com
Public good is a nebulous concept, but it is something that each of the folks who sign up as mentors judges independently when they are rating the projects and talking about them. Most of the folks who are offering to mentor have been involved in the community for quite some time and have a pretty good overview of what is going on, and what are currently active pain points. With 25 mentors we get a pretty good cross section of the community. We aren't really able to canvas outside of the mentor group during the approval process by google's guidelines, since we shouldn't leak information about unaccepted projects. Something like that uservoice site might be used to gauge public opinion of general ideas before the proposals start coming in, but in the end students write the proposals we get, so the things we would have polled about are inevitably not quite what we're rating anyways. We rarely get something that is just cut and pasted from the ideas list. Consequently a generic rating that doesn't take into consideration the actual proposal isn't worth a whole lot, beyond giving students an idea of what might be a successful proposal. There is a lot of variability in the ratings for projects based simply on what we know about the student, how clear the proposal is, and how achievable his or her particular goals are. In practice, we've been able to make sure that a couple of slots go to separable tasks in projects like cabal, haddock, and ghc that benefit everyone and that exceptional one-off projects don't get shut out completely just by asking each mentor to rate all of the projects, even the ones they aren't interested in mentoring, and from the discussions between the mentors and between the mentors and students that ensue within melange. My main advice is that if you want to get involved in the process, the easiest way to peel back the curtain is to volunteer to mentor! We're generally quite open to adding new voices to the discussion. -Edward On Sun, Jun 2, 2013 at 10:14 AM, Dominic Steinitz <dominic@steinitz.org>wrote:
Hi Edward,
Thanks for this comprehensive answer (and also thanks to participants in the follow-up dissuasion).
How is the "public good" determined? (sounds rather Benthamite). I would have been disappointed if "charts using diagrams" had not been selected yet I don't recall being canvassed.
Sorry to sound picky. I think from what you say that in this particular year it was obvious which projects should be selected; in future it may not be. I think an acceptable reason would be "there was only one user who wanted it". Maybe we should use something like: https://www.uservoice.com. Sadly it seems this requires payment but there may be a free equivalent
Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com
On 28 May 2013, at 16:11, Edward Kmett <ekmett@gmail.com> wrote:
Hi Dominic,
The proposal is admittedly rather unfortunately opaque.
The parts I can shed light on:
Students come up with proposals with the help of the community and then submit them to google-melange.com.
A bunch of folks from the haskell community sign up as potential mentors, vote on and discuss the proposals. (We had ~25 candidate mentors and ~20 proposals this year).
The student application template contains a number of desirable criteria for a successful summer of code application, which is shown on the google-melange website under our organization -- an old version is available http://hackage.haskell.org/trac/summer-of-code/wiki/StudApply2012contains
Once we have the proposals in hand, and some initial ranking, we ask google for slots. Allocation is based on past performance, arcane community parameters that only they know, mentor ratio, etc. This should be our largest year in the program, despite the fact that in general organizations have been getting fewer slots as more organizations join, so we're apparently doing rather well.
In general we do try to select projects that maximize the public good. Most of the time this can almost be done by just straight cut off based on the average score. There is some special casing for duplicate applications between different students and where students have submitted multiple applications we can have some flexibility in how to apply them.
This year we also received an extra couple of special-purpose darcs slots from Google in exchange for continuing to act as an umbrella organization over darcs at the request of the administrator of the program at Google. In previous years I had requested an extra slot for them, this year the request came in the other direction.
We do inevitably get more good proposals than we get slots. This year we could have easily used another 3-4 slots to good effect.
The main part I can't shed light on:
Google requests that the final vote tallies remain private. This is done so that students who put in proposals to a high volume orgs and don't get accepted, or who are new to the process and don't quite catch all the rules, don't wind up with any sort of publicly visible black mark. This unfortunately means, that we can't really show the unaccepted proposals with information about how to avoid getting your proposal rejected.
I hope that helps. If you have any more questions or if my answer didn't suffice please feel free to follow up!
-Edward Kmett
On Tue, May 28, 2013 at 6:52 AM, Dominic Steinitz <dominic@steinitz.org>wrote:
Hi Edward,
Although the project I am interested in (as a user) has been accepted :-), I can't help feeling the selection process is a bit opaque. Is it documented somewhere and I just missed it? Apologies if I did.
BTW I appreciate all the hard work that goes into the selection process.
Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com
I will certainly volunteer (to mentor) next year if I feel I can add value. Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com On 2 Jun 2013, at 17:23, Edward Kmett <ekmett@gmail.com> wrote:
Public good is a nebulous concept, but it is something that each of the folks who sign up as mentors judges independently when they are rating the projects and talking about them.
Most of the folks who are offering to mentor have been involved in the community for quite some time and have a pretty good overview of what is going on, and what are currently active pain points.
With 25 mentors we get a pretty good cross section of the community. We aren't really able to canvas outside of the mentor group during the approval process by google's guidelines, since we shouldn't leak information about unaccepted projects.
Something like that uservoice site might be used to gauge public opinion of general ideas before the proposals start coming in, but in the end students write the proposals we get, so the things we would have polled about are inevitably not quite what we're rating anyways. We rarely get something that is just cut and pasted from the ideas list. Consequently a generic rating that doesn't take into consideration the actual proposal isn't worth a whole lot, beyond giving students an idea of what might be a successful proposal. There is a lot of variability in the ratings for projects based simply on what we know about the student, how clear the proposal is, and how achievable his or her particular goals are.
In practice, we've been able to make sure that a couple of slots go to separable tasks in projects like cabal, haddock, and ghc that benefit everyone and that exceptional one-off projects don't get shut out completely just by asking each mentor to rate all of the projects, even the ones they aren't interested in mentoring, and from the discussions between the mentors and between the mentors and students that ensue within melange.
My main advice is that if you want to get involved in the process, the easiest way to peel back the curtain is to volunteer to mentor! We're generally quite open to adding new voices to the discussion.
-Edward
On Sun, Jun 2, 2013 at 10:14 AM, Dominic Steinitz <dominic@steinitz.org> wrote: Hi Edward,
Thanks for this comprehensive answer (and also thanks to participants in the follow-up dissuasion).
How is the "public good" determined? (sounds rather Benthamite). I would have been disappointed if "charts using diagrams" had not been selected yet I don't recall being canvassed.
Sorry to sound picky. I think from what you say that in this particular year it was obvious which projects should be selected; in future it may not be. I think an acceptable reason would be "there was only one user who wanted it". Maybe we should use something like: https://www.uservoice.com. Sadly it seems this requires payment but there may be a free equivalent
Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com
On 28 May 2013, at 16:11, Edward Kmett <ekmett@gmail.com> wrote:
Hi Dominic,
The proposal is admittedly rather unfortunately opaque.
The parts I can shed light on:
Students come up with proposals with the help of the community and then submit them to google-melange.com.
A bunch of folks from the haskell community sign up as potential mentors, vote on and discuss the proposals. (We had ~25 candidate mentors and ~20 proposals this year).
The student application template contains a number of desirable criteria for a successful summer of code application, which is shown on the google-melange website under our organization -- an old version is available http://hackage.haskell.org/trac/summer-of-code/wiki/StudApply2012 contains
Once we have the proposals in hand, and some initial ranking, we ask google for slots. Allocation is based on past performance, arcane community parameters that only they know, mentor ratio, etc. This should be our largest year in the program, despite the fact that in general organizations have been getting fewer slots as more organizations join, so we're apparently doing rather well.
In general we do try to select projects that maximize the public good. Most of the time this can almost be done by just straight cut off based on the average score. There is some special casing for duplicate applications between different students and where students have submitted multiple applications we can have some flexibility in how to apply them.
This year we also received an extra couple of special-purpose darcs slots from Google in exchange for continuing to act as an umbrella organization over darcs at the request of the administrator of the program at Google. In previous years I had requested an extra slot for them, this year the request came in the other direction.
We do inevitably get more good proposals than we get slots. This year we could have easily used another 3-4 slots to good effect.
The main part I can't shed light on:
Google requests that the final vote tallies remain private. This is done so that students who put in proposals to a high volume orgs and don't get accepted, or who are new to the process and don't quite catch all the rules, don't wind up with any sort of publicly visible black mark. This unfortunately means, that we can't really show the unaccepted proposals with information about how to avoid getting your proposal rejected.
I hope that helps. If you have any more questions or if my answer didn't suffice please feel free to follow up!
-Edward Kmett
On Tue, May 28, 2013 at 6:52 AM, Dominic Steinitz <dominic@steinitz.org> wrote: Hi Edward,
Although the project I am interested in (as a user) has been accepted :-), I can't help feeling the selection process is a bit opaque. Is it documented somewhere and I just missed it? Apologies if I did.
BTW I appreciate all the hard work that goes into the selection process.
Dominic Steinitz dominic@steinitz.org http://idontgetoutmuch.wordpress.com
participants (3)
-
Ben Lippmeier -
Dominic Steinitz -
Edward Kmett