- web-devel - Haskell.org

Re: [web-devel] [Yesod] Better web app deployment
by Michael Snoyman 05 May '12

05 May '12

On Fri, May 4, 2012 at 6:17 PM, Lyndon Maydwell <maydwell(a)gmail.com> wrote: > This is absolutely on par with what I've been thinking about over the > last week. I take it you are drawing some inspiration from the Twelve > Factor App Manifesto that you linked me to? It certainly seems that > way :-) > > I started building a management application to act as an interface for > this kind of architecture, however, I'm stuck grappling with the Yesod > form model as I haven't had much time to spend on it yet. > > A couple of ideas: > > Are you planning to have strong isolation between the applications > using some kind of sand-boxing? Given an Nginx reverse proxy this > should be made quite trivial by having the apps communicating over a > port or socket. Just a question of how it would be implemented. I wasn't planning on it, but you're right that it should be feasible. I suppose I've been taking the approach so far of letting the apps have much more freedom than say a Heroku deployment. The idea would be that only trusted apps would be running on the system. > I feel that a framework like this would benefit enormously from having > strong partnership with a continuous integration framework for > actually building the applications. This could run as an application > inside the application deployment framework in order to be nicely > circular! A benefit of this would be that you could guarantee that the > build and deploy architecture is the same - freeing developers from > the worry that their local machine is a different CPU architecture or > operating system. Like Greg said, the ideal would be building this on a separate virtual machine to avoid bogging down your app server with compilation. One possible idea would be to provide a nice, powerful EC2 instance as a community, and get a simple setup with Github commit hooks to be able to deploy from it automatically. I definitely want to explore this, but after the deployment system itself is in place. > > Love the idea. You're always a step ahead. > > On Fri, May 4, 2012 at 10:55 PM, Michael Snoyman <michael(a)snoyman.com> wrote: >> Hi all, >> >> (Note: I originally was going to just send this to the Yesod mailing list, >> but frankly, most of the code I'm setting up would work for any web >> framework. If people using other frameworks want to use this deployment >> code, and are interested in working with me to make sure it's universally >> applicable, please be in touch.) >> >> I'm planning on improving the deployment code starting next week. Instead of >> comparing to what we currently have (and which I think only I use), let me >> just give a high-level approach. Please give input on how you'd like to >> change things. Once I have the basic structure put together, I'll put it up >> on Github, and contributions will be greatly appreciated: >> >> * We'll use Nginx as a reverse proxy. This will allow us to have virtual >> hosting, more efficient static file serving (possibly from a separate >> domain), and zero-downtime deployments. >> * The scaffolding will accept a number of environment variables for >> configuring it, such as PostgreSQL connection information, approot, static >> root, and port number. (Most of this is already implemented.) >> * Each site will have a deployment configuration file, for things like >> domain name, whether a PostgreSQL database is required, and any system >> packages are required to be installed (we'll assume a Debian-based system >> for now). The deployer will automatically create databases and provide that >> information via environment variables. >> * Apps will have a specific file format, which is essentially a deployment >> config file and a bunch of things (static folder, executable, etc) it needs. >> An app will then be a tarball of those folders. >> * The deployer will watch a folder for changes. Deployments will be >> performed by copying files into this folder. >> * When a new app is deployed, it will be unpacked, started up, and >> monitored. If it stops, it will be restarted. The Nginx config file will be >> updated to reverse proxy to the app's port (randomly assigned) and then >> reloaded. >> * When a new version of the app is deployed, we'll unpack and start it up to >> a new folder with a different port. Once the app responds to an HTTP HEAD >> request (doesn't matter *how* it responds), we'll update the Nginx config >> file, reload Nginx, send the TERM signal to the old app, and then delete the >> old app's folder. >> * When an app is deleted, remove it from the Nginx config file, reload >> Nginx, send the app the TERM signal, and delete the folder. >> * We'll automatically store log files for apps (recommendations on good >> approaches here are welcome). >> * The deployer itself will provide a web interface to monitor status of apps >> and view logs. >> * I have no specific plans for running shell scripts or the like. I believe >> most of that can be part of the app loading. With the zero-downtime >> deployment, it's acceptable for the load time to be significant. If anyone >> has better ideas here, let me know. >> >> If we get a good deployment tool going, I think I would next want to set up >> an AMI (Amazon Machine Image) to make it dead simple to spin up an instance >> capable of hosting Yesod apps. If we can get it to configure DNS as well, >> and get a command line tool that automatically performs the EC2 instance >> creation, I think we'd be golden. >> >> Michael

1 0

Better web app deployment
by Michael Snoyman 05 May '12

05 May '12

Hi all, (Note: I originally was going to just send this to the Yesod mailing list, but frankly, most of the code I'm setting up would work for any web framework. If people using other frameworks want to use this deployment code, and are interested in working with me to make sure it's universally applicable, please be in touch.) I'm planning on improving the deployment code starting next week. Instead of comparing to what we currently have (and which I think only I use), let me just give a high-level approach. Please give input on how you'd like to change things. Once I have the basic structure put together, I'll put it up on Github, and contributions will be greatly appreciated: * We'll use Nginx as a reverse proxy. This will allow us to have virtual hosting, more efficient static file serving (possibly from a separate domain), and zero-downtime deployments. * The scaffolding will accept a number of environment variables for configuring it, such as PostgreSQL connection information, approot, static root, and port number. (Most of this is already implemented.) * Each site will have a deployment configuration file, for things like domain name, whether a PostgreSQL database is required, and any system packages are required to be installed (we'll assume a Debian-based system for now). The deployer will automatically create databases and provide that information via environment variables. * Apps will have a specific file format, which is essentially a deployment config file and a bunch of things (static folder, executable, etc) it needs. An app will then be a tarball of those folders. * The deployer will watch a folder for changes. Deployments will be performed by copying files into this folder. * When a new app is deployed, it will be unpacked, started up, and monitored. If it stops, it will be restarted. The Nginx config file will be updated to reverse proxy to the app's port (randomly assigned) and then reloaded. * When a new version of the app is deployed, we'll unpack and start it up to a new folder with a different port. Once the app responds to an HTTP HEAD request (doesn't matter *how* it responds), we'll update the Nginx config file, reload Nginx, send the TERM signal to the old app, and then delete the old app's folder. * When an app is deleted, remove it from the Nginx config file, reload Nginx, send the app the TERM signal, and delete the folder. * We'll automatically store log files for apps (recommendations on good approaches here are welcome). * The deployer itself will provide a web interface to monitor status of apps and view logs. * I have no specific plans for running shell scripts or the like. I believe most of that can be part of the app loading. With the zero-downtime deployment, it's acceptable for the load time to be significant. If anyone has better ideas here, let me know. If we get a good deployment tool going, I think I would next want to set up an AMI (Amazon Machine Image) to make it dead simple to spin up an instance capable of hosting Yesod apps. If we can get it to configure DNS as well, and get a command line tool that automatically performs the EC2 instance creation, I think we'd be golden. Michael

3 3

Re: [web-devel] [Yesod] Re: Better web app deployment
by Vagif Verdi 04 May '12

04 May '12

Why not ? It's an intranet app, few users, it does what we need. Its simple to deploy (one app instead of several separate apps). Much simpler infrastructure. Now i understand those awesome startups who deploy to Heroku of AWS or what have you in hopes to build next Facebook. Sure, go wild, nosql, thousands workers complex infrastructure. But why would i overcomplicate my small intranet app, just because your deployment tool is not flexible enough ? It would be simpler for me just not to use your deployment tool :) On Friday, May 4, 2012 10:10:32 AM UTC-7, Lyndon wrote: > > I don't think that it's generally considered a good idea to have > long-running processes occurring in-app. I know that on Heroku there > are 'workers' that you can provision to take care of such tasks. These > could even run as their own apps that connect to the same resources > for queueing, etc, in order to reuse the deployment interfaces and > infrastructure. > > Deciding to hold off on a restart of an application due to what would > usually be asynchronous operations seems a bit fishy. > > On Sat, May 5, 2012 at 12:32 AM, Vagif Verdi wrote: > > I have an intranet app that has a couple of long running operations. Say > > batch print 50 pdf files to printer. Or submit 10,000 invoices to ADP > Tax > > server (end of month). > > > > Maybe killing the app server should be optional. > > > > > > > > On Friday, May 4, 2012 7:55:21 AM UTC-7, Michael Snoyman wrote: > >> > >> send the TERM signal to the old app, >

2 1

ANNOUNCE: hsx-0.10.1, hjscript-0.6.0, hsp-0.7.0
by Niklas Broberg 23 Apr '12

23 Apr '12

Hi all, I'm pleased to announce the release of the latest versions of the libraries in the Haskell Server Pages suite. Haskell Server Pages enables literal XML syntax to be used mixed with Haskell code (like e.g. PHP), in a sane way (unlike said PHP). In principle the functionality could be used together with any Haskell web framework, but is (currently) best integrated in and supported by Happstack. For this release I owe much to Jeremy Shaw, both for inspiration and for actual patches. There are some grand plans in the pipeline for some more extensive changes to the suite, to shape it up to be worthy of the current exciting state-of-the-art of Haskell web development. Stay tuned! Cheers, /Niklas Specific changes in this release: * In the XMLGen class, the following associated types have been renamed to avoid frequent name clashes, and to better match the intention of those types: XML => XMLType, Child => ChildType, Attribute => AttributeType. * Literal CDATA strings are now given an explicit type signature in the post-processed code. This to avoid ambiguities when OverloadedStrings is enabled. * class XMLGenerator m now has EmbedAsChild m () as another premise. * HSX.Transform now exports a method transformExp :: Exp -> Exp, intended to be used with quasi-quotation. * trhsx no longer performs fixity resolution before transforming, allowing the tool to be used on files using operators with (locally) unknown fixity. * hsp and hjscript updated to work with hsx-0.10.

1 0

Yesod study meeting
by Kazu Yamamoto 23 Apr '12

23 Apr '12

Hello, The Haskell community in Japan has Yesod study meeting. It has just started. We are chatting in Tiwitter with #yesod (in Japanese). --Kazu

1 1

http-enumerator: users?
by Michael Snoyman 22 Apr '12

22 Apr '12

Hi all, I'm wondering if there are still active users out there of http-enumerator. Four months ago I released http-conduit, and since then, my development efforts have gone almost exclusively there. At this point, I'm not longer actively using http-enumerator on any of my projects, and while I've backported any security fixes to http-enumerator, it is otherwise not be updated. I see three possibilities for the future of http-enumerator: 1. Deprecate it in favor of http-conduit 2. Someone else takes over as maintainer 3. Turn it into a frontend for http-conduit, exposing an enumerator interface. This will likely, though not necessarily, result in API breakage I'm curious how others feel on this. If I don't hear anything back on the topic, I'll assume that no one is interested in http-enumerator, and thus will opt for option (1). Michael

2 2

[ANN] blaze-html-0.5
by Jasper Van der Jeugt 21 Apr '12

21 Apr '12

Hello all, I've just released a new version of blaze-html on Hackage. It has some backward-incompatible changes, so feel free to contact me if you run into any trouble. Summary of changes: - Split into blaze-markup and blaze-html - Easy creation of custom HTML elements - Very simple HTML tree manipulation For more details, see: http://jaspervdj.be/posts/2012-04-21-blaze-html-0.5.html Cheers, Jasper

1 0

Happstack Irregular News #1
by Jeremy Shaw 21 Apr '12

21 Apr '12

Welcome to the Happstack Irregular News #1! - 2012-04-20 ======================================================== This is the first issue of the Happstack Irregular News which is published on an irregular basis. Today a bunch of new and updated packages where uploaded to hackage, so it seemed like a good time to make some news! Big thanks to Dag Odenhall, Roel van Dijk, and mekeor.melire for their contributions. Also big thanks to Niklas Broberg, Felipe Lessa, and Gershom Bazerman for being responsive maintainers of their respective packages! Details: NEW: happstack-clientsession ---------------------------- Dag built us a shiny new client-side session data library which stores session data in cookies. It builds on top of the clientsession library from Felipe Lessa and Michael Snoyman, adding the glue needed to make it easy to use with Happstack. The happstack-clientsession library is still new and only lightly tested. So, your feedback and patches are appreciated. The API might still be in flux (unless we managed to get it correct on the first try). hsx-jmacro & happstack-jmacro ------------------------------ Dag and I (mostly Dag) worked on the hsx-jmacro and happstack-jmacro packages. hsx-jmacro now includes a number of methods to embed XML generated via HSX inside javascript generated by JMacro. See the haddock documentation for some discussion and examples: http://www.happstack.com/docs/hsx-jmacro-7.0.1/doc/html/hsx-jmacro/HSX-JMac… Also check out these examples: http://patch-tag.com/r/mae/happstack/snapshot/current/content/pretty/hsx-jm… http://patch-tag.com/r/mae/happstack/snapshot/current/content/pretty/hsx-jm… In the first example you will see these lines: > do let html :: DOMNode > html = <p>Generate javascript from <span class="foo" id="h">HTML & XML</span>.</p> which generate some simple HTML. Then in these lines, that html is added as a child of the main div using the normal javascript appendChild DOM function: > js :: JStat > js = [jmacro| document.getElementById('main').appendChild(`(html)`); > |] Then we have our main handler function, which splices that Javascript into our final HTML document: > handler :: ServerPart XML > handler = defaultTemplate "js-example" () > <%> > <div id="main" /> > <% js %> > </%> So, we saw both HSX being embedded into JMacro and JMacro being embedded into HSX. All the escaping is taking care of automatically for us. And we get to use very familiar looking HTML and Javascript syntax. Dag also made some improvements to happstack-jmacro to prevent global namespace pollution. The 'ToMessage JStat' instance now wraps the javascript inside an anonymous function before it is called. This helps avoid namespace collisions when you import multiple different javascript files (that are generated from JStat). There is also a new 'jmResponse :: ServerMonad m => JStat -> m Response' function which can be used to convert javascript (JStat) into a Response. 'jmResponse' ensures that all identifiers are unique by prefixing them with a hash computed from the rqUri. happstack-server ---------------- Dag and Mekeor added a few new functions to happstack-server. - serveFileFrom checks that the file being served is rooted in a specific parent directory. This is useful for preventing ../../etc/passwd type attacks when a portion of the filename is provided by the user. - combineSafe is a helper function which is used by serveFileFrom and can be reused in your own code - serveDirectory' is exactly like serveDirectory except that it allows you to provide the mime-type mapping function - neverExpires is a helper function that sets an 'Expires' header for as far in the future as is supported by most browsers I also modified happstack-server so that it can be built on platforms with out Template Haskell (such as ARM). happstack-hsp ------------- - analyticsAsync - a helper function that generates the Google Async Google Analytics Tracking Javascript code happstack-lite -------------- Type type-signature for serveFile now expects the mime-type function to be (FilePath -> IO String) instead of (FilePath -> ServerPart String) misc -------------- - a number of packages where updated to accept more recent dependenncies on mtl, transformers, hamlet, etc - Dag provided a bunch of documentation fixes to ixset Other News: =========== HSX ------------ HSX and friends still do not build from Hackage on GHC 7.4. I submitted patches to Niklas, which he has pulled into darcs, but not released on Hackage. However, we then started a long discussion about refactoring HSX so that it no longer depends on HJScript/HJavaScript, plus a bunch of other changes to make HSX easier to install and use. So, hopefully we will have a smaller, better HSX out soon. If you need HSX support on GHC 7.4 right now, you will want to fetch the source from darcs. You can find the darcs repos on patch-tag: http://patch-tag.com/r/nibro Unfortunately, you do not actually want the latest version from darcs, because some of the refactoring work has already begun. In the latest version of HSX, some of the associated types where renamed to avoid name classes. For example XML is now XMLType, Attribute is now AttributeType, etc. You will want to unpull the patches that make those changes. Hopefully this will be straightened out properly on Hackage in the next week. happstack-authenticate ---------------------- I also upgraded happstack-authenticate to the latest version of the authenticate library. The latest version of authenticate dropped support for facebook, so I updated the facebook connect support to use the fb package. Unfortunately, the fb package had not been updated to the latest conduits -- so I submitted a patch for that and a new fb 0.8 was released today! happstack-authenticate is still an 'alpha' package and has not been released on hackage yet. It is available from darcs if you wish to contribute: darcs get http://src.seereason.com/happstack-authenticate/ The major blockers to releasing on hackage are: 1. updating it so that it can take advantage of https:// 2. finishing the cookie code so that expired sessions are deleted 3. reviewing the code and making sure we are not doing anything obviously wrong security wise acme-http --------- In case you missed it, we released acme-http on April 1: https://groups.google.com/group/happs/browse_thread/thread/9b078d23b59a7374… While the announcement and date are a bit jokey, the acme-http server is actually the beginnings of the new Happstack HTTP backend. It is a real server which allows us to investigate some of the low-level performance issues in very small and isolated environment. We will then apply what we learn to the real backend. acme-http delivered 221,693.0 req/s, making it the fastest Haskell web server on the planet. We expect to see a drop when we add things like support for timeouts, etc. But, we also hope to find some ways to increase the performance over the current baseline. web-routes for Snap ------------------- The web-routes package, which provides type-safe URLs, is intended to be usable with a wide range of Haskell web frameworks. Here is a demonstration of how web-routes can be used with Snap without modifying either library (though there is some boiler-plate that could be extracted): https://github.com/stepcut/snap-web-routes-demo Crash Course Updates -------------------- In response to user feedback the acid-state section of the documentation has received some rewritten. There is also a new section on the basics of using Template Haskell: http://www.happstack.com/docs/crashcourse/TemplateHaskell.html#template_has… happstack.com updates --------------------- A lot of work has been going into behind the scenes improvements to the happstack.com website. Those changes should be rolling out soon! That's all the news that's fit to print! (and that I remember!). No extra charge for the typos!

2 2

Happstack authentication and sessions
by Asafe Ribeiro 17 Apr '12

17 Apr '12

How is Session management usually done in Happstack. I found two packages, happastack-auth and happastack-authentication but both seem outdated and no longer manteined (could not even get them to build).

2 3

BlazeHtml: How to write quotation-marks in attributes?
by Morel Pisum 12 Apr '12

12 Apr '12

Hi. How can I write <body onload="javascript:document.getElemByID('q').focus();"> in BlazeHtml? The problem is that single quotes are always converted to HTML. I'd appreciate a quick response. Thanks in advance. -- Morel Pisum

4 5