July 2014 - ghc-devs - Haskell.org

Re: Multi-instance packages status report
by Joachim Breitner 22 Jul '14

22 Jul '14

[Replying to the list, in case it was sent to me in private by accident] Hi Edward, Am Montag, den 21.07.2014, 23:25 +0100 schrieb Edward Z.Yang: > Excerpts from Joachim Breitner's message of 2014-07-21 21:06:49 +0100: > > maybe a stupid question, but how does the package key relate to the hash > > that "ghc-pkg" shows for package? > > Fine question---this is definitely something that is different from the > GSoC project. The short answer is, the current hash shown in ghc-pkg is > the ABI hash associated with the InstalledPackageId, which is computed > after GHC is done compiling your code; whereas the package key is a > hash of the dependency graph, which can be done before compilation. > > The longer answer is we now have three ID-like things, in order of > increasing specificity: > > Package IDs: containers-0.9 > These are the "user visible" things that we expect users to talk > about in Cabal file > Package Keys: md5("containers-0.9" + transitive deps) > These are the identifiers the compiler cares about: they are used > for type equality, and contain a bit more detail than we expect > a user to normally need---however, a user might need to refer to > this to disambiguate in some situations. > Installed Package IDs: ABI hash of compiled code > This uniquely identifies an installed package in the database, up > to ABI. > > So, if two packages have the same IPID, their package keys are > guaranteed to be the same, but not vice versa. (And likewise for package > IDs.) thanks for the explanations, it makes it clear to me. Do the package key contain the flags used to compile dependencies? In the example where it could matter the flag would change that package’s key, so maybe it is redundant.... And just to confirm my understandn: If we had a completely reproducible environment, the same key would (conceptually, not practically) imply the same IPID, right? Greetings, Joachim -- Joachim “nomeata” Breitner mail(a)joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata(a)joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata(a)debian.org

3 4

SV: Windows breakage -- again
by Niklas Larsson 22 Jul '14

22 Jul '14

I posted a working and tested patch last night. Please feel free to commit it, I haven't the rights to do it. Niklas ----- Ursprungligt meddelande ----- Från: "Simon Peyton Jones" <simonpj(a)microsoft.com> Skickat: ‎2014-‎07-‎18 15:55 Till: "Niklas Larsson" <metaniklas(a)gmail.com>; "Johan Tibell" <johan.tibell(a)gmail.com> Kopia: "ghc-devs(a)haskell.org" <ghc-devs(a)haskell.org> Ämne: RE: Windows breakage -- again Thank you all for pursuing this. I gather that you know what is going on, so no further info needed from me. Yell if it is otherwise. Meanwhile, is the fix imminent, or should we revert Johan’s patch? Simon From: Niklas Larsson [mailto:metaniklas@gmail.com] Sent: 16 July 2014 19:58 To: Johan Tibell; Simon Peyton Jones Cc: ghc-devs(a)haskell.org Subject: Re: Windows breakage -- again I get the same failure when I try to build HEAD. Turns out the error occurs on the 32-bit Windows build, and my successful build was a 64-bit build. My 64-bit build still succeeds. Also, gcc is 4.5.2 on 32-bit, not 4.6.3 as on 64-bit. Niklas 2014-07-16 14:48 GMT+02:00 Niklas Larsson <metaniklas(a)gmail.com>: I have built ghc on windows after that was added with no issue. I can take a look this evening and see how HEAD works for me. The standard gcc in the tarballs is 4.6.3, which is getting long in the tooth, there is an issue on trac to upgrade it. -- Niklas Från: Johan Tibell Skickat: ‎2014-‎07-‎16 09:57 Till: Simon Peyton Jones Kopia: ghc-devs(a)haskell.org Ämne: Re: Windows breakage -- again You can rollback the commit (git revert 4ee4ab01c1d97845aecb7707ad2f9a80933e7a49) and push that to the repo if you wish. I will try to re-add the primop again after I figure out what's wrong. On Wed, Jul 16, 2014 at 9:37 AM, Johan Tibell <johan.tibell(a)gmail.com> wrote: I added some primops about a month ago (4ee4ab01c1d97845aecb7707ad2f9a80933e7a49) that call __sync_fetch_and_add, a gcc/llvm builtin. I'm a bit surprised to see this error. The GCC manual [1] says: > " Not all operations are supported by all target processors. If a particular operation cannot be implemented on the target processor, a warning will be generated and a call an external function will be generated. The external function will carry the same name as the builtin, with an additional suffix `_n' where n is the size of the data type." I'm a bit surprised by this error for two reasons: * A call to that symbol should only be generated if the CPU doesn't support the atomic instructions. What CPU model does Windows report that you have? * gcc should define such a symbol. For me the following test program compiles: #include <stdint.h> uint8_t test(uint8_t* ptr, uint8_t val) { return __sync_fetch_and_add_1(ptr, val); } int main(void) { uint8_t n; return test(&n, 1); } Does that compile for you? Which version of GCC do we end up using on Windows? The reported symbol (___sync_fetch_and_add_1) has three leading underscores, that looks weird. Can you compile just libraries/ghc-prim/cbits/atomic.c and see if it's indeed GCC that generates a reference to that symbol? 1. http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html On Wed, Jul 16, 2014 at 12:29 AM, Simon Peyton Jones <simonpj(a)microsoft.com> wrote: Aargh! The Windows build has broken – again. I can’t build GHC on my laptop any more. [Hela det ursprungliga meddelandet tas inte med.]

4 8

RFC: unsafeShrinkMutableByteArray#
by Herbert Valerio Riedel 22 Jul '14

22 Jul '14

Hello Simon (et al.) While experimenting with refactoring/improving integer-gmp, I'd like to represent a GMP number just by a ByteArrays# (and thus save a redundant limb-count field). However, for that I'd need an efficient way to resize a MutableByteArray# for the result value in case its initial size over- (or under-)allocated. Right now I'd re-allocate via newByteArray# with the final size and copyMutableByteArray#, and now I was wondering if we couldn't simply have an unsafeShrinkMutableByteArray# :: MutableByteArray# s# -> Int# -> State# s -> State# s operation, which would allow for zero-copying. (the 'unsafe' denotes this wouldn't check if the new size is less-or-equal to the current size, and that one has to be careful when subsequently using sizeofMutableByteArray# which is currently a pure function) Is such an operation feasible, or is there something in the RTS/GC that would trip over when a ByteArray has suddenly a smaller byte-count than its originally newByteArray#'ed amount? PS: maybe unsafeShrinkMutableByteArray# could unsafe-freeze the ByteArray# while at it (thus be called something like unsafeShrinkAndFreezeMutableByteArray#), as once I know the final smaller size I would freeze it anyway right after shrinking. Cheers, hvr

2 3

GHC contribution guidelines and infrastructure talk on 6th September at HIW?
by Rob Stewart 22 Jul '14

22 Jul '14

Hi, On Saturday 6th September is the Haskell Implementers Workshop. There has been plenty of discussion over the last 12 months about making contributions to GHC less formidable. Is this story going to be told at HIW? A talk about revised contribution guidelines and helpful tool support might engage those sat on, or peering over, the fence. This might include: * Phabricator code review demonstration. * Continuous integration infrastructure. * Trac demonstration, e.g. how to contribute to design discussions. * Wiki navigation, and important new pages born in recent months. * GHC coding guidelines, e.g. using notes and haddock documentation. * Git policies, e.g. use of submodules. * What GHC needs.. Windows testers? * Old contribution guidelines that no longer apply. Is HIW on 6th September a good place to give a "GHC contributions and infrastructure" talk? -- Rob

5 7

Thread status constants
by Kyle Van Berendonck 22 Jul '14

22 Jul '14

Hi, I found these: https://github.com/ghc/ghc/blob/5f3c5384df59717ca8013c5df8d1f65692867825/in… They go only 0-14, so there's some long chains of branches and stuff in hot paths that could be cleaned up into single &-masked branches by changing these into a set of flags. And then I saw these: https://github.com/ghc/ghc/blob/master/libraries/base/GHC/Conc/Sync.lhs#L483 Where does 16 and 17 come from -- I couldn't find them in the header files anywhere? Kyle

2 2

Beta Performance dashboard
by Joachim Breitner 22 Jul '14

22 Jul '14

Hi, I guess it’s time to talk about this, especially as Richard just brought it up again... I felt that we were seriously lacking in our grip on performance issues. We don’t even know whether 6.8.3 was better or worse than 6.8.3 or 7.6.4 in terms of nofib, not to speak of the effect of each single commit. I want to change that, so I set up a benchmark monitoring dashboard. You can currently reach it at: http://ghcspeed-nomeata.rhcloud.com/ What does it do? ~~~~~~~~~~~~~~~~ It monitors the repository (master branch only) and builds each commit, complete with the test suite and nofib. The log is saved and analyzed, and some numbers are extracted: * The build time * The test suite summary numbers * Runtime (if >1s), allocations and binary sizes of the nofib benchmarks These are uploaded to the website above, which is powered by codespeed, a general performance dashboard, implemented in Python using Django. Under _Changes_, it provides a report for each commit (changes wrt. to the previous version, and wrt. to 10 revisions earlier, the so-called “trend”). A summary of these reports is visible on the front-page. The _Timeline_ is a graph for each individual performance number. If there are bumps, you can hopefully find them there! You can also compare to 7.8.3, which is available as a “baseline”. _Comparison_ will be more useful if we have more tagged revision, or if were benchmarking various options (e.g. -fllvm): Here you can do bar-chart comparisons. Why codespeed? ~~~~~~~~~~~~~~ For a long time I searched for a suitable software product, and one criterion is that it should be open source, rather simple to set up and mostly decoupled from other tools, i.e. something that I throw numbers at and which then displays them nicely. While I don’t think codespeed is the best performance dashboard out there (I find http://goperfd.appspot.com/perf a bit better; I wonder how well codespeed scales to even larger numbers of benchmarks and I wish it were more git-aware), it was the easiest to get started with. And thanks to the loose coupling of (1) running the tests to acquire a log, (2) parsing the log to get numbers and (3) putting them on a server, we can hopefully replace it when we come along something better. I was hoping for the Phabricator guys to have something in their tool suite, but doesn’t look like it. How does it work (currently)? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ My office PC is underused (I work on my laptop), so its currently dedicated to it. I have a simple shell script that monitors the repo for new versions. It builds the newest revision and works itself back to the commit where everything was turned into submodules: https://github.com/nomeata/codespeed/blob/ghc/tools/ghc/watch.sh It calls a script that does the actual building: https://github.com/nomeata/codespeed/blob/ghc/tools/ghc/run-speed.sh This produces a log file which should contain all the required numbers somewhere. A second script extracts these numbers (with help of nofib-analyze) and converts them into codespeed compatible JSON files: https://github.com/nomeata/codespeed/blob/ghc/tools/ghc/log2json.pl Finally, a simple invocation to curl uploads them to codespeed: https://github.com/nomeata/codespeed/blob/ghc/tools/ghc/upload.sh So if you want additional benchmarks to be tracked, make sure they are present in the logs and adjust log2json.pl. codespeed will automatically pick up new benchmarks in these logs. Reimplementations in Haskell are also welcome :-) The testsuite is run with VERBOSE=4, so the performance numbers are also shown for failing test cases. So once a test case goes over the limit, you can grep through previous logs try to find the real culprit. I uploaded the logs (so far) to https://github.com/nomeata/ghc-speed-logs (but this is not automated yet, ping me if you need an update on this). What next? ~~~~~~~~~~ Clearly, the current setup is only good enough to evaluate the system. Eventually, I might want to use my office PC again, and the free hosting on openshift is not very powerful. So if we want to keep this setup and make it “official”, we need find a permanent solution.¹ This involves: * A dedicated machine to run the benchmarks. This probably shouldn’t be a VM, if we want to keep the noise in the runtime down. * A machine to run the codespeed server. Can be a VM, or even run on any of the system that we have right now. Just needs a database (postgresql preferably) and a webserver supporting WSGI (i.e. any of them). * Maybe a better place to store the logs for public consumption. Also, there are way to improve the system: * As I said, I don’t think codespeed is the best. If we find something better, we can replace it. Since we have all the logs, we can easily fill the new system with the data, or even run both at the same time. * We might want to have more numbers. I am already putting lines-of-code and disk space usage numbers into the logs, but do not parse them yet. * In particular, we might want to put in each performance test case as a benchmark of its own, to easier find commits that degrade (or improve!) performance. I’m not sure how well the web page will handle that. * We might want to replace my rather simple watch.sh-script by something more serious. In particular, I imagine that our builder setup could manages this, with a dedicated builder doing the benchmark runs and the builder server scheduling a build for each commit. That’s it for now. Enjoy clicking around! Greetings, Joachim ¹ I guess that could be considered beta-reduction :-) -- Joachim Breitner e-Mail: mail(a)joachim-breitner.de Homepage: http://www.joachim-breitner.de Jabber-ID: nomeata(a)joachim-breitner.de

4 4

Multi-instance packages status report
by Edward Z. Yang 22 Jul '14

22 Jul '14

Hello all, As you may have noticed, I've been knocking around GHC and Cabal the past few weeks. One of the tasks that has been on my list is essentially reimplementing Philipp Schuster's 2012 GSoC, with a few small but important architectural differences. Here is a status report of what is going on. In my copy of GHC and Cabal [1,2], you can now install multiple copies of a package with differing dependencies to the package database, i.e. q-1.0 compiled against p-1.0, and against p-2.0. The packages in the database are distinguished via a *package key*, which is an md5 hash of the package id (e.g. q-1.0) and the sorted list of the package IDs of the transitive closure of dependencies (e.g. p-1.0 or p-2.0). The package key is used to generate linker symbols for packages, so it's possible to link together both copies of q in the same program, as long as you rename the modules appropriately (of course, the redefined types are considered unequal). When registering a package, ghc-pkg now checks and removes duplicate package keys, as opposed to package IDs. I've also implemented many of the necessary niceties for making it pleasant to deal with duplicated package IDs: for example, GHC's output logic has been adjusted to only qualify a package ID with the package key when there are multiples of the same package ID exposed in the database. I've also adjusted GHC's build system to use package keys rather than package IDs to refer to packages when building. Like the original GSoC project, we still need to have decisions on some of the major design choices. I think the most pressing one is "simplistic dependency resolution" when you use, e.g. ghc -package foo-1.0 or ./Setup configure --with-constraint="foo==1.0". Right now, I pick dependencies in an unspecified manner, check if they are consistent, and bail out if they are not. As it turns out, both GHC and ./Setup configure will already compute the transitive closure of dependencies, so I suspect we might be able to do something clever here. Less pressing but eventually necessary is clueing in cabal-install. Comments would be especially appreciated. Cheers, Edward [1] https://phabricator.haskell.org/D80 [2] https://github.com/haskell/cabal/pull/2002

2 1

Status updates
by Austin Seipp 21 Jul '14

21 Jul '14

Hello *, Here are some updates from this past week. - I sent an email earlier about 7.10.1 tickets - please vote on them! - Related to that - as I said last week, I finished remilestoning a lot of tickets, but more remain. Please excuse any mail spam, but do pay attention to your bugs if they change! - I cleaned up some tabs and whitespace in the compiler in low-activity areas. You may have seen the commits go by, but otherwise it shouldn't be a problem. I'll probably keep eliminating errors in low-contention areas of the codebase as I can. Ditto with 'arc lint'. - I've cleaned up a bunch of wiki pages relating to the git repositories, and submodule management, but more stuff remains. See below for my reply to Simon. - I've been lazy about the Applicative-Monad work, since some of it fell out of date. But now other work is pending on it, so it's time to finish it Real Soon Now. - I'm working out some of the kinks in the Phabricator automation for builds etc, but it's not quite done yet - sorry! Here are some things I plan to do this week, roughly in order of priority: - I'm going to keep cleaning up these Git pages to make them more sensible, since some of the information is still wrong. Please see the comments below this and let me know what's wrong! - I will finalize these Phabricator things, which I should be able to get done with soon. This is all mostly annoying HTTP endpoint and PHP/Python stuff and is a bit of a chore to test/setup completely. - I am going to spend time next on landing AMP and get it back into shape. Yes, really landing it; I need to send something upstream to Cabal still. - The patch queue on Trac needs to be drained and slipped my mind. - Given all that I'll also probably end up continuing to fix lint errors, and hopefully fix bugs based on what people want in 7.10. :) Below is a quick recap of what me and Simon discussed last week, mostly stuff from the above: > · Complete re-milestoning. Then email ghc-devs and ghc-users [send a draft to Simon & Simon first] > >o Giving links to suitable query(s) > >o Saying that any ticket that isn’t > >a) milestoned 7.8.4 or 7.10.1, and > >b) high or highest priority, > >isn’t going to get attended to by us. > >o Inviting them to bid (on the ticket) for high/highest 7.10.1 status, explaining why it is important to them. And that it’s more likely to be done if they help. I sent this mail earlier - please bid on 7.10.1 tickets! > · Trac/Phab integration Doing this in a complete manner is turning out to be a bit of work and I haven't gotten done yet with having comments appear on tickets just yet. My goal is to have it so that when you say 'arc diff', you can put the words: Trac: #9832, #9123 into the review, and it will auto appear on Phabricator as a custom field, hyperlink it, and then a comment will appear on Trac. I have to write some PHP and Python on both ends to get this all working, though. Also the builder script is currently really crappy, so I'm fixing that a bit, so it will correctly deal with concurrent builds and multiple builders. > · Document how to update a submodule; e.g. haddock > > o We worked through an example > > o We need a canonical place on the wiki to say which branch GHC should be tracking on the submodule; and how to check that it is > > o Need to talk to haddock and cabal folk about whether GHC should track master or a branch in their repo. They may want a more stable branch to work on. > > > · Wiki pages > > o Be sure to update https://ghc.haskell.org/trac/ghc/wiki/Building/GettingTheSources, to include simple cloning story > > o Consolidate “how to use Git” information into WorkingConventions/Git, perhaps headed “Git workflows”. Merging WorkingConventions/Git with the Git/Submodules page will make it intimidatingly long. > > § Think about how to split it up. > > § Tricks should be on a separate page. > > § Focus on tasks, not technology > > o wiki:Repositories page says where everything lives, but not how to use them. > > § Needs updating to have the correct data. > > § Also integrate https://ghc.haskell.org/trac/ghc/wiki/Repositories/Upstream This is now done with a first draft. I think I simplified the pages a lot. See here: https://ghc.haskell.org/trac/ghc/wiki/WorkingConventions/Git and: https://ghc.haskell.org/trac/ghc/wiki/WorkingConventions/Git/Submodules I plan on obsoleting the old Repositories pages since they're mostly redundant and out of date now. There may be more info stuffed into other places, but I imagine it will stay under this hierarchy. There are still some links and prose that need to be rewritten that I obviously missed however. > · Merge AMP stuff. Should be done now. (Without join in Monad for now, #9123.) https://phabricator.haskell.org/D13 Unfortunately this is still pending a few things, so I haven't gotten it done yet. :( > o Maybe make the bug tracker page easier to find. > > § Move https://ghc.haskell.org/trac/ghc/wiki/WorkingConventions from “Joining in” side-bar to “Documentation” and list it as “Working conventions” > > § Move newcomer material from wiki:WorkingConventions to wiki:Newcomers, and put *that* under “Joining in” > > Now the bug tracker stuff is in Working Conventions, easier to find. This has all been done. -- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

1 0

GHC 7.10.1 tickets
by Austin Seipp 21 Jul '14

21 Jul '14

Hello *, We need to be sorting out tickets for 7.10.1. Last week, I spent some time remilestoning things to get them into the rough correct places (roughly), but we all need some help to get an idea of what we should fix for the release. Here's a query to look at: https://ghc.haskell.org/trac/ghc/query?status=!closed&milestone=7.10.1&orde… I have already categorized these tickets roughly, but we'll probably still address some of the priorities on existing tickets of course. These are all of the 7.10.1 tickets currently, but there are some important things to note: 1) There are a lot of them! 2) Most of the development team is probably not going to pay attention to many of them unless they are high or highest priority in Trac. (That's just the result of having so many tickets for 7.10.1 at the moment.) So I'm asking if people out there with the time can help chip in and give some input on the ticket! - If there's a ticket that interests you, please comment on it! - Say why you think it should have its priority elevated for 7.10.1. - It will certainly be easier if you offer to help. Then we can move it upwards and we'll look at it. Finally, if you are *really* dedicated or a newcomer - please go through the tickets and see if any of them can be closed, reassigned, marked as duplicates, etc. There are certainly some dead tickets and probably easy tickets for newcomers, so just spend a few minutes looking if you're interested! (Otherwise they just end up getting punted off in batch modifications) Thanks! ~~~~~~~ BTW, for developers, you can look at your assigned tickets for 7.10.1 here, it's also always on the sidebar: https://ghc.haskell.org/trac/ghc/query?owner=%24USER&status=infoneeded&stat… -- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

1 0

Planning a Haskell Platfrom Release Candidate this weekend
by Mark Lentczner 21 Jul '14

21 Jul '14

Here's my remaining list before putting out a realse candidate. I'm aiming for this weekend: *-- General --* [ ] haddock linkage from HP packages to GHC core packages is broken [ ] decision on QuickCheck? *-- Source Tarball --* [ ] missing platform.sh scripts [ ] missing os-extras in hptool/src *-- Mac --* [ ] start.html needs a once over updating [ ] uninstall-hs should clean out /usr/local/bin symlinks (as it does with /usr/bin) *-- Windows --* [ ] GLUT issue in ghci *-- Website --* [ ] templatize info on pages [ ] add build mode to bptool Main - needs to take file of downloads & hashes Anyone have anything else? Otherwise... we're almost good to go!!!! - Mark

1 1