Simon,

That's good to hear. That addresses one of the complex use cases for dynCompileExpr that we have.

I'd like to present two more and see how we can address them. They both follow the same pattern so I will just describe the first one.

In order to let interpreted Haskell code display images and other complex media to the frontend, the interpreted code uses unsafePerformIO to create a TChan. This TChan holds images (roughly speaking). The interpreted Haskell has access to a function which writes to this TChan. 

The compiled Haskell runs the interpreted Haskell in a separate thread. That thread then writes to this TChan (which lives in the interpreted context). In order to access values in the TChan, IHaskell uses dynCompileExpr on a value of type `IO [Image]`. This value is an IO action that reads from the TChan that lives in the interpreted context. This value is then run in the compiled code, giving IHaskell access to the results generated by the interpreter.

I do not see how this can be easily replicated with a remote GHCi. The value is `IO [Image]`, so it is not serializable. `[Image]` is serializable; however, in order to get the Image values from the TChan, we would have to run interpreted code, and these images are generated during the runtime of interpreted code already, so we would need to be running two separate interpreted threads at once. This has to be completely invisible to the user and seems to me like it might be a logistical nightmare... And, if I understand correctly, GHC/GHCi are not thread safe, so I can't just use the new equivalent of dynCompileExpr from two threads. (Can I?)


Thanks,

-- Andrew



On Thu, Nov 19, 2015 at 1:44 AM, Simon Marlow <marlowsd@gmail.com> wrote:
Hi Andrew -

Since the interpreted code is running in a separate process that we spawn using createProcess, we can set stdin/stdout/stderr to be whatever we like, including new pipes.  GHC itself needs two pipes to communicate with the sub-process, but those use separate file descriptors from the std Handles.

So I think the answer is yes, we can support that more easily with remote GHCi.  I'll think about what API we can provide for it.

Cheers,
Simon


On 18/11/2015 16:26, Andrew Gibiansky wrote:
Simon,

I'd like to hear how we can support what IHaskell does with remote GHCi.

One core functionality that we use dynCompileExpr for (not quite
dynCompileExpr, but similar) is getting the standard output of code that
is being run. Any time code is run, we

1. Create a unix pipe.
2. Set stdout to point to that pipe using dupTo.
3. Use hscStmt with unsafeCoerce to get the other end of the pipe in the
compiled context.
4. Run the statement in the interpreted context in a separate thread;
meanwhile, read from the pipe to get the stdout of the code running in
the interpreted context.
5. When it is done running, move stdout back to point to the read stdout
and close the unix pipe file handle.
6. Send the stdout (both intermediate values and the final value) to the
frontend to display to the user.

The key here is that we can access directly the file handle created by
the interpreted code. If the interpreted code is remote, we clearly
cannot read from a pipe it creates. In your remote GHCi, how could we
solve this problem?

In general, how would stdin and stdout work? Would there be a clean way
to feed the remote process its stdin and receive its stdout and stderr?
That would effectively mean stdin/stdout/stderr are configurable which
would be a godsend for IHaskell.

-- Andrew

On Wed, Nov 18, 2015 at 1:45 AM, Simon Marlow <marlowsd@gmail.com
<mailto:marlowsd@gmail.com>> wrote:

    On 18/11/2015 01:41, Manuel M T Chakravarty wrote:

        Hi Simon,

        While this is an interesting proposal, Haskell for Mac strongly
        relies on running interpreted code in the same process. I’m using
        ’dynCompileExpr’ as well as ’hscStmtWithLocation’ and some other
        stuff.


    Let me say first of all that I'm not going to remove anything, so
    there's no need to worry.  But I'd like to explore exactly what you
    need, so that we can see whether there's a way to accommodate it
    with a separate-process implementation.

    hscStmtWithLocation is part of the core GHCi functionality, it is
    definitely supported.  It has a slightly different signature:

    hscStmtWithLocation :: HscEnv
                         -> String -- ^ The statement
                         -> String -- ^ The source
                         -> Int    -- ^ Starting line
                         -> IO ( Maybe ([Id]
                               , RemoteHValue {- IO [HValue] -}
                               , FixityEnv))

    RemoteHValue is a reference to a value in the interpreter's context.
    These have to be evaluated via an explicit API, rather than just
    unsafeCoercing HValue as we do now.  (this is not strictly speaking
    part of the GHC API, so a separate but interesting question is: why
    did you need to use this directly, and what should we add to the GHC
    API?)

    I believe that many uses of dynCompileExpr can be changed so that
    the code using the resulting value is moved into the interpreter's
    context, and then there's no problem.

        This is quite crucial for some of the interactive
        functionality. Imagine a game where the game engine is in Swift
        linked into the main application and the game logic is in
        *interpreted* Haskell code. The engine calls into the Haskell code
        multiple times per frame of the animation and for all
        keyboard/mouse/etc input (using StablePtr and ForeignPtr to
        construct
        the scene graph across the Swift and Haskell heap).


    So my question is, why wouldn't you run the whole game engine in the
    interpreter's context?  That's what would happen if you were to load
    the program into GHCi and run it.  Directly calling back and forth
    between the client of the GHC API and the program being interpreted
    is arguably a strange thing to do, and it's kind of accidental that
    we allow it.

        I actually also might have a use for the architecture that you are
        proposing. However, I really would like to keep the ability to, at
        least, optionally run interpreted code in the same process (without
        profiling etc). Do you think we could have both?


    We can certainly have both, it's straightforward to implement, but I
    don't get to throw away some of the hacks we have to support
    same-process execution, which would be a shame.  We just add more
    code rather than


        Cheers,
        Manuel

            Simon Marlow <marlowsd@gmail.com <mailto:marlowsd@gmail.com>>:

            Hi folks - I've been thinking about changing the way we run
            interpreted code so that it would be run in a separate
            process.  It turns out this has quite a few benefits, and
            would let us kill some of the really awkward hacks we have
            in GHC to work around problems that arise because we're
            running interpreted code and the compiler on the same runtime.

            I summarised the idea here:
            https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi

            I'd be interested to hear if anyone has any thoughts around
            this, particularly if doing this would make your life
            difficult in some way. Are people relying on dynCompileExpr
            for anything?

            Cheers,
            Simon
            _______________________________________________
            ghc-devs mailing list
             ghc-devs@haskell.org <mailto:ghc-devs@haskell.org>
            http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


    _______________________________________________
    ghc-devs mailing list
     ghc-devs@haskell.org <mailto:ghc-devs@haskell.org>
    http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs