Come to think of it, Joel Bartlett at DECWRL had a system called ezd
that looked a lot like this, except he did it the other way around:
his server (written in Scheme) provided high-level (for the time)
graphics primitives whose interface was a pair of pipes. I played with
it some back in the day, and I recall it as pretty spiffy (especially
using Bartlett's Scheme compiler for the front end).

And the internet hasn't forgotten it! I found a copy of his WRL Tech
Report on it:
http://www.hpl.hp.com/techreports/Compaq-DEC/WRL-91-6.html

      <mike