Registering GHC for Coverity SCAN
Hello all, On IRC today, Nicolas Trangez brought up the idea of registering GHC for Coverity's SCAN project. SCAN is essentially a free service run by Coverity, which runs their Static Analyzer on open source projects ('open source' being defined by OSI) and gives the results back to developers. Coverity is widely regarded as having the absolute best tool in the business for C codebases as it stands. The idea is to try and rattle bugs out of the runtime system if possible. I think this is a good idea, but it needs some public discussion. Namely, Coverity requires an official Point of Contact within the project (maintainer/author/governing body) to register the project in their name. We are left up to determine who this is and verify it. Bug reports which are found are not also made public, as they could be potentially serious security vulnerabilities.* So, if done, this is a rather piece-by-piece project which requires a bit of commitment, because it will require the registree to not only assess the bugs, but sensibly move them into the issue tracker or fix it themselves. It's also a question of whether it's worth it. Personally, I think it has the chance to uncover some real bugs. But the RTS is highly unusual C code (by some standards) and it will probably require tuning and patience to get tangible results. Bugs will also need to be assessed by someone. False positives happen etc. This is the double edged sword of static analysis.** Anyone who does it should be very aware of this. So, what do you all think? I'd particularly like input from Simon M here since he's the primary author of the RTS, of course. :) And anyone who does this will undoubtedly have to be in contact with him to some degree, for all the aforementioned reasons. If nobody else would step up for this, I would be interested in contacting them if it seems worth it and giving it a shot. * I think the chance of this in our case is very very low of course. The cases we really care about for a safety perspective are type-safety violations at the Haskell level, obviously. Nonetheless this requires an individual who is committed to sifting through reports and assessing them in a detailed way. ** Coverity at one point found out the hard way that more analysis is not necessarily better, and people don't respond well to the results sometimes. Or they're too complicated to explain. The following is a great read: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later... -- Regards, Austin - PGP: 4096R/0x91384671
Austin brings up an interesting point, which is that the last time we had a security vulnerability situation was when it was discovered that CGI scripts could pass +RTS options to Haskell programs and get programs to do things that they shouldn't. And, AFAICT, we haven't ever had any CVEs issued against GHC. So we don't really have any infrastructure in place for dealing with security problems (not even a "private security bug tracker"). While in principle having this infrastructure is probably a good idea, I don't think it should block a motivated person who wants to do some Coverity analysis. Edward Excerpts from Austin Seipp's message of Wed May 08 17:36:49 -0700 2013:
Hello all,
On IRC today, Nicolas Trangez brought up the idea of registering GHC for Coverity's SCAN project. SCAN is essentially a free service run by Coverity, which runs their Static Analyzer on open source projects ('open source' being defined by OSI) and gives the results back to developers. Coverity is widely regarded as having the absolute best tool in the business for C codebases as it stands. The idea is to try and rattle bugs out of the runtime system if possible.
I think this is a good idea, but it needs some public discussion. Namely, Coverity requires an official Point of Contact within the project (maintainer/author/governing body) to register the project in their name. We are left up to determine who this is and verify it. Bug reports which are found are not also made public, as they could be potentially serious security vulnerabilities.*
So, if done, this is a rather piece-by-piece project which requires a bit of commitment, because it will require the registree to not only assess the bugs, but sensibly move them into the issue tracker or fix it themselves.
It's also a question of whether it's worth it. Personally, I think it has the chance to uncover some real bugs. But the RTS is highly unusual C code (by some standards) and it will probably require tuning and patience to get tangible results. Bugs will also need to be assessed by someone. False positives happen etc. This is the double edged sword of static analysis.** Anyone who does it should be very aware of this.
So, what do you all think? I'd particularly like input from Simon M here since he's the primary author of the RTS, of course. :) And anyone who does this will undoubtedly have to be in contact with him to some degree, for all the aforementioned reasons.
If nobody else would step up for this, I would be interested in contacting them if it seems worth it and giving it a shot.
* I think the chance of this in our case is very very low of course. The cases we really care about for a safety perspective are type-safety violations at the Haskell level, obviously. Nonetheless this requires an individual who is committed to sifting through reports and assessing them in a detailed way.
** Coverity at one point found out the hard way that more analysis is not necessarily better, and people don't respond well to the results sometimes. Or they're too complicated to explain. The following is a great read: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later...
On 09/05/13 01:36, Austin Seipp wrote:
Hello all,
On IRC today, Nicolas Trangez brought up the idea of registering GHC for Coverity's SCAN project. SCAN is essentially a free service run by Coverity, which runs their Static Analyzer on open source projects ('open source' being defined by OSI) and gives the results back to developers. Coverity is widely regarded as having the absolute best tool in the business for C codebases as it stands. The idea is to try and rattle bugs out of the runtime system if possible.
I think this is a good idea, but it needs some public discussion. Namely, Coverity requires an official Point of Contact within the project (maintainer/author/governing body) to register the project in their name. We are left up to determine who this is and verify it. Bug reports which are found are not also made public, as they could be potentially serious security vulnerabilities.*
So, if done, this is a rather piece-by-piece project which requires a bit of commitment, because it will require the registree to not only assess the bugs, but sensibly move them into the issue tracker or fix it themselves.
It's also a question of whether it's worth it. Personally, I think it has the chance to uncover some real bugs. But the RTS is highly unusual C code (by some standards) and it will probably require tuning and patience to get tangible results. Bugs will also need to be assessed by someone. False positives happen etc. This is the double edged sword of static analysis.** Anyone who does it should be very aware of this.
So, what do you all think? I'd particularly like input from Simon M here since he's the primary author of the RTS, of course. :) And anyone who does this will undoubtedly have to be in contact with him to some degree, for all the aforementioned reasons.
By all means go ahead. But I'm dubious that they'd be able to find anything useful, at least without adding some GHC-specific knowledge about conventions in the RTS to their static analyser. We hardly use malloc/free at all, for example. Also I can't guarantee that I'll have much time to look into any potential issues they uncover, but I'll try my best. Cheers, Simon
If nobody else would step up for this, I would be interested in contacting them if it seems worth it and giving it a shot.
* I think the chance of this in our case is very very low of course. The cases we really care about for a safety perspective are type-safety violations at the Haskell level, obviously. Nonetheless this requires an individual who is committed to sifting through reports and assessing them in a detailed way.
** Coverity at one point found out the hard way that more analysis is not necessarily better, and people don't respond well to the results sometimes. Or they're too complicated to explain. The following is a great read: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later...
On Wed, 2013-05-08 at 19:36 -0500, Austin Seipp wrote:
Hello all,
On IRC today, Nicolas Trangez brought up the idea of registering GHC for Coverity's SCAN project. SCAN is essentially a free service run by Coverity, which runs their Static Analyzer on open source projects ('open source' being defined by OSI) and gives the results back to developers. Coverity is widely regarded as having the absolute best tool in the business for C codebases as it stands. The idea is to try and rattle bugs out of the runtime system if possible.
I think this is a good idea, but it needs some public discussion. Namely, Coverity requires an official Point of Contact within the project (maintainer/author/governing body) to register the project in their name. We are left up to determine who this is and verify it. Bug reports which are found are not also made public, as they could be potentially serious security vulnerabilities.*
So, if done, this is a rather piece-by-piece project which requires a bit of commitment, because it will require the registree to not only assess the bugs, but sensibly move them into the issue tracker or fix it themselves.
It's also a question of whether it's worth it. Personally, I think it has the chance to uncover some real bugs. But the RTS is highly unusual C code (by some standards) and it will probably require tuning and patience to get tangible results. Bugs will also need to be assessed by someone. False positives happen etc. This is the double edged sword of static analysis.** Anyone who does it should be very aware of this.
So, what do you all think? I'd particularly like input from Simon M here since he's the primary author of the RTS, of course. :) And anyone who does this will undoubtedly have to be in contact with him to some degree, for all the aforementioned reasons.
If nobody else would step up for this, I would be interested in contacting them if it seems worth it and giving it a shot.
Although I believe this is a good idea (who'd have thought ;-)), I won't step up as a potential 'lead' for this due to a lack of knowledge, and I can't be sure I'll be able to spend enough time on the project. If someone does give it a shot, I'd love to help out triaging etc (a project 'owner' can provide access to the result to others as well if I'm not mistaken). As Simon M. mentioned, the analysis might be confused since the RTS is in several places not 'your average C code'. This makes sense (although adding some annotations in comments can help), but maybe some false positives caused by this could also point at 'too magic' code, which might need some simplification love? Lua, Mono & Racket are some projects already part of the program which might have similar code patterns etc. Nicolas
participants (4)
-
Austin Seipp -
Edward Z. Yang -
Nicolas Trangez -
Simon Marlow