Thanks Alberto! You can certainly set up permissions and security groups on EC2 nodes using CloudFormation, run your job, and then tear down the CloudFormation stack. You can use stratosphere to specify the AWS resources like an EC2 instance, a VPC for it, some security groups, and an instance role. Then, you can use something like Ansible or even just a bash script to set up and run your job on that machine. Once it's done, you can tear down the stack you just built. We actually do something similar to what you said for running pgbadger on our Postgres RDS logs. Our logs can reach a few GB per hour, so downloading them to a local machine can be pretty time-consuming, and processing can take long as well. In order to get RDS logs, you need to have permissions to do so on AWS. I made a stack in stratosphere that creates a large EC2 instance for processing, a strict security group that locks down the instance, and also sets up a temporary user that only has permission to download RDS logs. Using Ansible, I then install pgbadger and its dependencies, download the logs, run pgbadger, and then download the results. Once the job is done, everything that was just created is destroyed. In summary, what you're saying is indeed possible and is a great use-case for CloudFormation. You just need to using something on top of stratosphere, as this library is simply used to build the CloudFormation templates. I'm considering adding support for running the templates directly in this library, maybe using amazonka-cloudformation. So far that hasn't been necessary for us. Also note that CloudFormation has a way to estimate the cost of your created resources: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-payi... On Tue, Apr 19, 2016 at 12:21 PM, Alberto G. Corona wrote:

Very nice job.

I have tried AWS EC2 and I failed miserably since the permission system and his intricate configurations, his paranoid security, and their secret formulas for pricing are beyond an average human intelligence.

It is possible to have an example stratosphere application that could configure and run a Haskell program in one or many EC2 nodes , close the instances and send back the results?

Are there some elements still lacking to achieve such goal?

2016-04-19 20:56 GMT+02:00 David Reaver :

...

Github: https://github.com/frontrowed/stratosphere Hackage: https://hackage.haskell.org/package/stratosphere

CloudFormation is a system that provisions and updates Amazon Web Services (AWS) resources based on declarative templates. Common criticisms of CloudFormation include the use of JSON as the template language and limited error-checking, often only available in the form of run-time errors and stack rollbacks. By wrapping templates in Haskell, we are able to easily construct them and help ensure correctness.

The goals of stratosphere are to: - Build a Haskell EDSL to specify CloudFormation templates. Since it is embedded in Haskell, it is type-checked and generally much easier to work with than raw JSON. - Have a simple checking/linting system outside of the types that can find common errors in templates. - Be able to also read valid CloudFormation JSON templates so they can be type-checked. This also gives us free integration tests by using the huge amount of example templates available in the AWS docs.

Most of the commonly used CloudFormation resources are implemented, and adding new resources is very straightforward. (We created a web scraper that generates a JSON file from a given CloudFormation documentation page URL, and from that we generate a Haskell module.) So far, we have implemented resources we use at Front Row Education, and we will add more resources over time.

The library is very much in a usable state as-is. However, we want to make any sweeping changes while it is still young. If you have any suggestions at all, you want us to add your favorite resource, or if you find a bug, please file an issue on Github!

Also, we want to give a huge thanks to Brendan Hay, the author of amazonka, for his ideas and feedback on the project.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

-- Alberto.

David. Thanks a lot for the hints. I looked at some CloudFormation scripts and they look even more horrendous than what I have imagined. 2016-04-19 21:39 GMT+02:00 David Reaver :

Thanks Alberto!

You can certainly set up permissions and security groups on EC2 nodes using CloudFormation, run your job, and then tear down the CloudFormation stack. You can use stratosphere to specify the AWS resources like an EC2 instance, a VPC for it, some security groups, and an instance role. Then, you can use something like Ansible or even just a bash script to set up and run your job on that machine. Once it's done, you can tear down the stack you just built.

We actually do something similar to what you said for running pgbadger on our Postgres RDS logs. Our logs can reach a few GB per hour, so downloading them to a local machine can be pretty time-consuming, and processing can take long as well. In order to get RDS logs, you need to have permissions to do so on AWS. I made a stack in stratosphere that creates a large EC2 instance for processing, a strict security group that locks down the instance, and also sets up a temporary user that only has permission to download RDS logs. Using Ansible, I then install pgbadger and its dependencies, download the logs, run pgbadger, and then download the results. Once the job is done, everything that was just created is destroyed.

In summary, what you're saying is indeed possible and is a great use-case for CloudFormation. You just need to using something on top of stratosphere, as this library is simply used to build the CloudFormation templates. I'm considering adding support for running the templates directly in this library, maybe using amazonka-cloudformation. So far that hasn't been necessary for us.

Also note that CloudFormation has a way to estimate the cost of your created resources: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-payi...

On Tue, Apr 19, 2016 at 12:21 PM, Alberto G. Corona wrote:

...

Very nice job.

I have tried AWS EC2 and I failed miserably since the permission system and his intricate configurations, his paranoid security, and their secret formulas for pricing are beyond an average human intelligence.

It is possible to have an example stratosphere application that could configure and run a Haskell program in one or many EC2 nodes , close the instances and send back the results?

Are there some elements still lacking to achieve such goal?

2016-04-19 20:56 GMT+02:00 David Reaver :

...

Github: https://github.com/frontrowed/stratosphere Hackage: https://hackage.haskell.org/package/stratosphere

CloudFormation is a system that provisions and updates Amazon Web Services (AWS) resources based on declarative templates. Common criticisms of CloudFormation include the use of JSON as the template language and limited error-checking, often only available in the form of run-time errors and stack rollbacks. By wrapping templates in Haskell, we are able to easily construct them and help ensure correctness.

The goals of stratosphere are to: - Build a Haskell EDSL to specify CloudFormation templates. Since it is embedded in Haskell, it is type-checked and generally much easier to work with than raw JSON. - Have a simple checking/linting system outside of the types that can find common errors in templates. - Be able to also read valid CloudFormation JSON templates so they can be type-checked. This also gives us free integration tests by using the huge amount of example templates available in the AWS docs.

Most of the commonly used CloudFormation resources are implemented, and adding new resources is very straightforward. (We created a web scraper that generates a JSON file from a given CloudFormation documentation page URL, and from that we generate a Haskell module.) So far, we have implemented resources we use at Front Row Education, and we will add more resources over time.

The library is very much in a usable state as-is. However, we want to make any sweeping changes while it is still young. If you have any suggestions at all, you want us to add your favorite resource, or if you find a bug, please file an issue on Github!

Also, we want to give a huge thanks to Brendan Hay, the author of amazonka, for his ideas and feedback on the project.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

-- Alberto.

-- Alberto.