On Tue, Aug 2, 2011 at 21:09, Edward Z. Yang wrote:

Excerpts from Victor Nazarov's message of Tue Aug 02 19:12:55 -0400 2011:

...

But GHC always emit parse error on "javascript" keyword.

For now I'm using (abusing) ccall calling convention and simple imports works pretty well, but I would like to support exports and static/dynamic wrappers. GHC generates C-code to support them, and GHCJS should generate Javascript-code, but I have no idea how to use GHC API to generate custom (Javascript) stubs. Is it possible at all?

That is certainly not; you'll need to patch GHC's lexer to do that. But it's all a bit dodgy since this FFI doesn't make sense unless you are generating Javascript code (which GHC is not.) Maybe someone else can comment on that. (Perhaps we need fully extensible calling convention syntax? Hmmm!)

If making it a backend isn't acceptable then it's a bit of a lost cause; I'd expect it to be a backend, with backend hooks to support various -f* options (probably already exists) and the foreign syntax changed from expecting a keyword to expecting an identifier and optional parenthesized identifier list (the same syntactic category as import and export lists, "deriving" clauses, etc.), and that would be passed to the backend.

...

I'd like to create ghcjs and ghcjs-pkg tools that will use their own directories and package index and will not interfere with ghc. Is it possible with GHC API? How can I do it?

Check out the approach that cabal-dev uses, which is about what you are looking for here. (Though, handling base packages might be tricky.)

And I'd point out that this sounds similar to another recently reported shortcoming of ghc-pkg (mentioned in the context of cabal's dependency calculations but I miscall the details at the moment). I would expect JS packages to register a dependency on a version of base specific to the JS backend, and that dependency would prevent libraries using the standard base package from being considered. -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

On Wed, Aug 3, 2011 at 00:31, John Lask wrote:

What is really required is a "pluggable" back-end infrastructure - whereby various back-ends could be maintained (or not) at the discretion of their originators and separate to the official ghc back-ends.

I guess I'm confused; I thought the current back-end system *was* that kind of pluggable architecture. I recall a JVM backend being proposed based on it some time back. -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

On Wed, Aug 3, 2011 at 8:56 AM, John Lask wrote:

On 3/08/2011 2:10 PM, Brandon Allbery wrote:

...

On Wed, Aug 3, 2011 at 00:31, John Lask  wrote:

...

What is really required is a "pluggable" back-end infrastructure - whereby various back-ends could be maintained (or not) at the discretion of their originators and separate to the official ghc back-ends.

I guess I'm confused; I thought the current back-end system *was* that kind of pluggable architecture.  I recall a JVM backend being proposed based on it some time back.

my thoughts of pluggable infrastructure include consideration of ffi bindings and library integration as well as command line options i.e. as discussed in this thread with respect of GHC-JS, rather than just backend code generation - i.e. considerations of broader scope than those currently handled.

Yes, I can enumerate following cosideration when developing "non drop-in replacement for current GHC backend": * command line support. Developers should be able to emulate ghc and ghc-pkg to gain cabal support. Another way would be to standatize compiler command line interface in Cabal and to provide command line parsing library with it. * library support. New backend should really behave like stand-alone compiler, rather then GHC * ffi support, custom ffi calling conventions and custom format for call specification, Javascript or JVM will require radically different call specification then simple function-name as used with C-call. * cross-compilation. GHCJS doesn't work properly on 64bit systems. Javascript lacks real integers and we need to emulate them with Javascript bit-operations. But all javascript bit-operations work with 32 bits. To make Javascript code readable and fast we need to make Haskell Int type to be 32bit-integer, but it is imposible on 64bit system. * Built-in desugaring. Some desugaring provided by GHC is backend-specific. For instance string literals are desugared into application of GHC.unpackCString :: Addr# -> String to some address wich points to '\0' terminated C-string. In GHCJS we better provide our own desugaring, then add terminating '\0' to strings to emulate C-string with Javascript-strings. -- Victor Nazarov

On Wed, Aug 3, 2011 at 11:30 AM, Simon Peyton-Jones wrote:

Victor

GHC is supposed to be extensible, via its API, so your questions are good ones.  However, there are things that that the API doesn't support, or supports badly, so it is not cast in stone.  Please suggest improvements -- and better still implement them. GHC evolves largely in response to your suggestions and help.

In particular I don't think anyone has implemented a new back end via the API (rather than by building it into GHC itself) before.  So this is good.  It would be cool to have a compiler that behaved as if it had a JavaScript backend built in, but was actually built in a modular way on the API. Then people could use that as a model to build new back ends.

I imagine that your general plan is:  - use the GHC API to parse Hasekll, typecheck it, optimise it  - finishing with a [CoreBind] of optimised definitions  - then use your own code generator to convert that [CoreBind] into JavaScript

Yes, I'm doing it, but I'm moving slightly fearther - I convert [CoreBind] to [StgBind] through GHC API. - then convert StgBindings to Javascript.

| == Command line interface == | This works, but I'm not allowed to parse some custom flags used by | GHCJS code and not by GHC API. | ParseDynamicFlags throws UsageError if it encounters some unknown flags.

So perhaps that's the problem. parseDynamicFlags could perfectly well simply return any un-recognised flags. Indeed, I thought it did just that -- it certainly returns a list of un-consumed arguments.  If it doesn't perhaps that's a bug.

parseDynamicFlags returns un-consumed arguments if they are something like filenames, but it throws error if un-consumed argument starts with dash.

| == Foreign Function Interface == | | What I want is to provide FFI for Javascript, But GHC doesn't allow to | extend FFI declaration syntax. | I'd like to create some new FFI calling convention ("javascript") like this: | | foreign import javascript "alert" |   jsalert :: Ptr JSString -> IO ()

OK, so this is harder.  Presumably you want to use an *unmodified* Haskell parser to parse the Haskell programs. Adding *syntactic* extensions is therefore somewhat invasive:        - change the lexer        - change the parser        - change the HsSyn data structure        - change every function that traverses HsSyn

However in this particular case maybe things are not so bad.  I believe that perhaps *all* you want is to add a new calling convention. See ForeignImport in HsDecls, and CCallConv in ForeignCall.  Simply adding a new data constructor to CCallConv, and lexing the token for it, would not be too bad.  We could possibly add that part to the mainline compiler. The compiler would largely ignore such decls, and they'd just pop out at the other end for your back end to consume.

This would be cool. I will try to provide patch to add all calling conventions that backend implementors can use. But GHC should report errors about unsupported calling conventions sometime during compilation when should it?

There might be complications -- see DsForeign in particular -- but I expect they'd be minor.

| For now I'm using (abusing) ccall calling convention and simple | imports works pretty well, but I would like to support | exports and static/dynamic wrappers. GHC generates C-code to support | them, and GHCJS should generate Javascript-code, | but I have no idea how to use GHC API to generate custom (Javascript) | stubs. Is it possible at all?

Well, GHC generates the stub code in its code generator, doesn't it?  If you don't call the code generator, because you are using yours instead, then it'll be up to you to generate the stub code, no?

I can access ForeignStubs datatype, but I think it already have generated C-code in it, havn't it? What step during compilation I should generate stubs on? -- Victor Nazarov

On Wed, Aug 03, 2011 at 11:44:10AM +0100, Simon Marlow wrote:

On 03/08/2011 11:09, Victor Nazarov wrote:

...

On Wed, Aug 3, 2011 at 11:30 AM, Simon Peyton-Jones wrote:

...

...

So perhaps that's the problem. parseDynamicFlags could perfectly well simply return any un-recognised flags. Indeed, I thought it did just that -- it certainly returns a list of un-consumed arguments. If it doesn't perhaps that's a bug.

parseDynamicFlags returns un-consumed arguments if they are something like filenames, but it throws error if un-consumed argument starts with dash.

So then parseDynamicFlags should be split into two layers, the lower layer returning unused flags, and the upper layer generating errors.

It's not that simple. In ghcjs -O -someflag something -Wall is "something" an argument to someflag, or a file to be compiled? I think the best approach is to make a variant parseDynamicFlagsAnd which takes an extra argument of type [Flag (CmdLineP DynFlags)] which it prepends to dynamic_flags. Otherwise there's Edward's suggestion, but it would be a bit depressing to have to use -- even for the simplest invocations, e.g. ghcjs -- -O foo.hs although I guess you could mirror the most common GHC flags as ghcjs flags. Thanks Ian

On Wed, Aug 3, 2011 at 2:44 PM, Simon Marlow wrote:

On 03/08/2011 11:09, Victor Nazarov wrote:

...

On Wed, Aug 3, 2011 at 11:30 AM, Simon Peyton-Jones  wrote:

...

...

So perhaps that's the problem. parseDynamicFlags could perfectly well simply return any un-recognised flags. Indeed, I thought it did just that -- it certainly returns a list of un-consumed arguments.  If it doesn't perhaps that's a bug.

parseDynamicFlags returns un-consumed arguments if they are something like filenames, but it throws error if un-consumed argument starts with dash.

So then parseDynamicFlags should be split into two layers, the lower layer returning unused flags, and the upper layer generating errors. You could use the lower layer in your front end.

Please send us a patch...

...

...

OK, so this is harder.  Presumably you want to use an *unmodified* Haskell parser to parse the Haskell programs. Adding *syntactic* extensions is therefore somewhat invasive:        - change the lexer        - change the parser        - change the HsSyn data structure        - change every function that traverses HsSyn

However in this particular case maybe things are not so bad.  I believe that perhaps *all* you want is to add a new calling convention. See ForeignImport in HsDecls, and CCallConv in ForeignCall.  Simply adding a new data constructor to CCallConv, and lexing the token for it, would not be too bad.  We could possibly add that part to the mainline compiler. The compiler would largely ignore such decls, and they'd just pop out at the other end for your back end to consume.

This would be cool. I will try to provide patch to add all calling conventions that backend implementors can use. But GHC should report errors about unsupported calling conventions sometime during compilation when should it?

Right, you'll need some backend-specific desugaring of the FFI declarations.  Maybe we need desugarer plugins? :-) An easier approach would be to have a slot in the DynFlags for a callback, like we do for printing error messages, so the GHC API client passes in a callback to do whatever backend-specific desugaring is required.  The callback mechanism could be used for lots of things - I can imagine it growing into a record of backend-specific functions that the earlier stages of the compiler might need to call.

It's hard to predict exactly what's needed.  Again, I suggest you try doing this and send us a patch.

I think I should do it. From GHCJS perspective I need to abstract out literal desugaring and foreign exports/imports desugaring. Desugarer uses these functions from MkCore module now: mkIntExpr :: Integer -> CoreExpr mkIntExprInt :: Int -> CoreExpr mkWordExpr :: Integer -> CoreExpr mkWordExprWord :: Word -> CoreExpr mkIntegerExpr :: MonadThings m => Integer -> m CoreExpr mkFloatExpr :: Float -> CoreExpr mkDoubleExpr :: Double -> CoreExpr mkCharExpr :: Char -> CoreExpr mkStringExpr :: MonadThings m => String -> m CoreExpr mkStringExprFS :: MonadThings m => FastString -> m CoreExpr We should create some record like: data LiteralDesugaring m = LiteralDesugaring { desugarInt :: MonadThings m => Integer -> m CoreExpr , desugarWord :: MonadThings m => Integer -> m CoreExpr , desugarInteger :: MonadThings m => Integer -> m CoreExpr , desugarFloat :: MonadThings m => Float -> m CoreExpr , desugarDouble :: MonadThings m => Double -> m CoreExpr , desugarChar :: MonadThings m => Char -> m CoreExpr , desugarString :: MonadThings m => String -> m CoreExpr } and some constant like defaultLiteralDesugaring :: MonadThings m => LiteralDesugaring m defaultLiteralDesugaring = LiteralDesugaring { desugarInt = return . mkIntExpr, ... } and make desugaring take LitaralDesugaring as an argument, with defaultLiteralDesugaring being default. But I don't still understand what can I do with foreign imports/exports. DsForeign module seems to be too complicated. As I can see, I shouldn't make whole dsForeigns function replaceable, but I can't understand what part of it should be replaceble. -- Victor Nazarov

On Fri, Aug 5, 2011 at 12:02 AM, Simon Peyton-Jones wrote:

|  data LiteralDesugaring m = |    LiteralDesugaring |      { desugarInt :: MonadThings m => Integer -> m CoreExpr |      , desugarWord :: MonadThings m => Integer -> m CoreExpr ...

I am not sure why you want to control the desugaring of literals.  Why literals?  And why is literals enough?

I'm not sure if literals are enough, but literals essentially as dependencies on ghc-prim package: 5 is desugared to something like GHC.Num.fromInteger (GHC.Integer.SmallInteger (5# :: Int#)) and "Hello" is desugared to something like GHC.String.unpackCString ("Hello"# :: Addr#) Fore backends other than drop-in-replacement-for-current-native-code-ghc-backend we may want an alternative desugarings. For example, * we may want to use some very simple Integer implementation. We can use doubles instead of integers at first to get things rolling. * in javascript we better use other "unpacking" function for strings, since Javascript strings doesn't have terminating NUL and have length built-in. -- Victor Nazarov

Hi,

...

So perhaps that's the problem. parseDynamicFlags could perfectly well simply return any un-recognised flags. Indeed, I thought it did just that -- it certainly returns a list of un-consumed arguments. If it doesn't perhaps that's a bug.

parseDynamicFlags returns un-consumed arguments if they are something like filenames, but it throws error if un-consumed argument starts with dash.

Maybe you can wrap the GHCJS into a plugin now that plugin support is finally in GHC 7.2.* , and generate the javascript as a side-effect [1]? Then you can pass option specifically to GHCJS using the '-fplugin-opt' command line option. Cheers, Christiaan [1] It is my understanding that plugins can only do Core-to-Core passes