Re: [web-devel] [Yesod] Better web app deployment

On Fri, May 4, 2012 at 6:17 PM, Lyndon Maydwell wrote:

This is absolutely on par with what I've been thinking about over the last week. I take it you are drawing some inspiration from the Twelve Factor App Manifesto that you linked me to? It certainly seems that way :-)

I started building a management application to act as an interface for this kind of architecture, however, I'm stuck grappling with the Yesod form model as I haven't had much time to spend on it yet.

A couple of ideas:

Are you planning to have strong isolation between the applications using some kind of sand-boxing? Given an Nginx reverse proxy this should be made quite trivial by having the apps communicating over a port or socket. Just a question of how it would be implemented.

I wasn't planning on it, but you're right that it should be feasible. I suppose I've been taking the approach so far of letting the apps have much more freedom than say a Heroku deployment. The idea would be that only trusted apps would be running on the system.

I feel that a framework like this would benefit enormously from having strong partnership with a continuous integration framework for actually building the applications. This could run as an application inside the application deployment framework in order to be nicely circular! A benefit of this would be that you could guarantee that the build and deploy architecture is the same - freeing developers from the worry that their local machine is a different CPU architecture or operating system.

Like Greg said, the ideal would be building this on a separate virtual machine to avoid bogging down your app server with compilation. One possible idea would be to provide a nice, powerful EC2 instance as a community, and get a simple setup with Github commit hooks to be able to deploy from it automatically. I definitely want to explore this, but after the deployment system itself is in place.

Love the idea. You're always a step ahead.

On Fri, May 4, 2012 at 10:55 PM, Michael Snoyman wrote:

...

Hi all,

(Note: I originally was going to just send this to the Yesod mailing list, but frankly, most of the code I'm setting up would work for any web framework. If people using other frameworks want to use this deployment code, and are interested in working with me to make sure it's universally applicable, please be in touch.)

I'm planning on improving the deployment code starting next week. Instead of comparing to what we currently have (and which I think only I use), let me just give a high-level approach. Please give input on how you'd like to change things. Once I have the basic structure put together, I'll put it up on Github, and contributions will be greatly appreciated:

* We'll use Nginx as a reverse proxy. This will allow us to have virtual hosting, more efficient static file serving (possibly from a separate domain), and zero-downtime deployments. * The scaffolding will accept a number of environment variables for configuring it, such as PostgreSQL connection information, approot, static root, and port number. (Most of this is already implemented.) * Each site will have a deployment configuration file, for things like domain name, whether a PostgreSQL database is required, and any system packages are required to be installed (we'll assume a Debian-based system for now). The deployer will automatically create databases and provide that information via environment variables. * Apps will have a specific file format, which is essentially a deployment config file and a bunch of things (static folder, executable, etc) it needs. An app will then be a tarball of those folders. * The deployer will watch a folder for changes. Deployments will be performed by copying files into this folder. * When a new app is deployed, it will be unpacked, started up, and monitored. If it stops, it will be restarted. The Nginx config file will be updated to reverse proxy to the app's port (randomly assigned) and then reloaded. * When a new version of the app is deployed, we'll unpack and start it up to a new folder with a different port. Once the app responds to an HTTP HEAD request (doesn't matter *how* it responds), we'll update the Nginx config file, reload Nginx, send the TERM signal to the old app, and then delete the old app's folder. * When an app is deleted, remove it from the Nginx config file, reload Nginx, send the app the TERM signal, and delete the folder. * We'll automatically store log files for apps (recommendations on good approaches here are welcome). * The deployer itself will provide a web interface to monitor status of apps and view logs. * I have no specific plans for running shell scripts or the like. I believe most of that can be part of the app loading. With the zero-downtime deployment, it's acceptable for the load time to be significant. If anyone has better ideas here, let me know.

If we get a good deployment tool going, I think I would next want to set up an AMI (Amazon Machine Image) to make it dead simple to spin up an instance capable of hosting Yesod apps. If we can get it to configure DNS as well, and get a command line tool that automatically performs the EC2 instance creation, I think we'd be golden.

Michael