Web.Encodings + Hack - Not sure what I'm doing wrong.

The project that I'm working on requires user image galleries. The idea is to allow them to upload images via a form, do some rudimentary processing on the uploaded data and then send it directly to Amazon's S3 service. The problem that I'm running in to is that I can't seem to get the binary data from the post request correctly. When written out to disk with withBinaryFile (hopefully thereby bypassing the new encoding pitfalls in ghc 6.12.x), file(1) says it's 'data'. This seems to be happening if I push it to a new object on S3 as well. I'm secretly hoping I'm doing something obviously wrong here. The code in question is at http://gist.github.com/404636 . Hopefully it's obvious that it's way more frantic what-the-hell-is-going-on-here code than some crackpot's idea of production ready code :) Thanks, web-devel!

On Tue, May 18, 2010 at 8:16 AM, Clint Moore
The project that I'm working on requires user image galleries. The idea is to allow them to upload images via a form, do some rudimentary processing on the uploaded data and then send it directly to Amazon's S3 service. The problem that I'm running in to is that I can't seem to get the binary data from the post request correctly. When written out to disk with withBinaryFile (hopefully thereby bypassing the new encoding pitfalls in ghc 6.12.x), file(1) says it's 'data'. This seems to be happening if I push it to a new object on S3 as well.
I'm secretly hoping I'm doing something obviously wrong here.
The code in question is at http://gist.github.com/404636 . Hopefully it's obvious that it's way more frantic what-the-hell-is-going-on-here code than some crackpot's idea of production ready code :)
Thanks, web-devel!
Clint, Firstly, would it be possible to modify your gist so that it's runnable on it's own? It's hard to debug what I can't see first hand. Secondly, I'm not certain I understand what you're saying the problem is. You mean that the file contents are simply "data"? Michael

On Mon, May 17, 2010 at 10:48 PM, Michael Snoyman
Clint, Firstly, would it be possible to modify your gist so that it's runnable on it's own? It's hard to debug what I can't see first hand. Secondly, I'm not certain I understand what you're saying the problem is. You mean that the file contents are simply "data"? Michael
I was referring to the file command on most unix-es to determine the file type by the magic number. Here's a better explanation: a.png is the original image, and b.png is what I get back from the upload process. $ file a.png b.png a.png: PNG image, 42 x 36, 8-bit colormap, non-interlaced b.png: data $ ls -l a.png b.png -rw-r--r-- 1 cmoore cmoore 611 2010-05-17 23:01 a.png -rw-r--r-- 1 cmoore cmoore 1020 2010-05-17 23:01 b.png $ md5sum a.png b.png 0116a68ff260fcce936b9a60c28a34d7 a.png 46e0b264fb5ab73aa2dfe8f3d40241c7 b.png As far as making the code runnable, I can do that perhaps tomorrow. I'd love to just tar up the code, or push it to github or somesuch, but it's not mine. If I can't figure it out by then I'll write a one-off that demonstrates the problem and put it up. In the hope that it spurs someone's 'oh yea, I remember that, you have to x when you y!' memory, here's what I know about the problem: I know that 1. It's almost certainly not the form used to submit the data because 2. The example file upload script on the haskell wiki works. This is what led me to assume that it wasn't a problem with encoding in ghc 6.12, but now I'm not so sure due to the differences in the file size above. I feel compelled to point out that, as far as the data of the file is concerned, I'm not transforming it (at least my naive Haskell knowledge tels me I'm not). Whether I am writing it out to a file, or pushing it to s3, I'm still getting it from parseMultipart, and then from fileContent. That said, it doesn't make sense that the problem would be web.encodings. It's obviously being used, and more importantly, working in other places. It just seems like the places I could be making a mistake are rather few. Which, hehe, again points to an encoding issue. Thanks again for the help, and I will post it up tomorrow if I still haven't figured it out.

Hello Clint, Tuesday, May 18, 2010, 10:29:56 AM, you wrote:
$ ls -l a.png b.png -rw-r--r-- 1 cmoore cmoore 611 2010-05-17 23:01 a.png -rw-r--r-- 1 cmoore cmoore 1020 2010-05-17 23:01 b.png
"od ?.png|head -5" ? it seems like utf8 encoding :)) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Tue, May 18, 2010 at 12:10 AM, Bulat Ziganshin
Hello Clint,
Tuesday, May 18, 2010, 10:29:56 AM, you wrote:
$ ls -l a.png b.png -rw-r--r-- 1 cmoore cmoore 611 2010-05-17 23:01 a.png -rw-r--r-- 1 cmoore cmoore 1020 2010-05-17 23:01 b.png
"od ?.png|head -5" ?
it seems like utf8 encoding :))
$ od ?.png | head -5 0000000 050211 043516 005015 005032 000000 006400 044111 051104 0000020 000000 025000 000000 022000 001410 000000 144000 054027 0000040 000031 000400 050005 052114 177505 177777 122244 167244 0000060 021200 177376 176376 176374 143306 163706 163747 126655 0000100 165255 165352 171362 175362 175372 122645 176645 176775 I didn't know od existed. Thanks for that!

On Tue, May 18, 2010 at 9:29 AM, Clint Moore
On Mon, May 17, 2010 at 10:48 PM, Michael Snoyman
wrote: Clint, Firstly, would it be possible to modify your gist so that it's runnable on it's own? It's hard to debug what I can't see first hand. Secondly, I'm not certain I understand what you're saying the problem is. You mean that the file contents are simply "data"? Michael
I was referring to the file command on most unix-es to determine the file type by the magic number. Here's a better explanation:
a.png is the original image, and b.png is what I get back from the upload process.
$ file a.png b.png a.png: PNG image, 42 x 36, 8-bit colormap, non-interlaced b.png: data $ ls -l a.png b.png -rw-r--r-- 1 cmoore cmoore 611 2010-05-17 23:01 a.png -rw-r--r-- 1 cmoore cmoore 1020 2010-05-17 23:01 b.png $ md5sum a.png b.png 0116a68ff260fcce936b9a60c28a34d7 a.png 46e0b264fb5ab73aa2dfe8f3d40241c7 b.png
As far as making the code runnable, I can do that perhaps tomorrow. I'd love to just tar up the code, or push it to github or somesuch, but it's not mine. If I can't figure it out by then I'll write a one-off that demonstrates the problem and put it up.
In the hope that it spurs someone's 'oh yea, I remember that, you have to x when you y!' memory, here's what I know about the problem: I know that 1. It's almost certainly not the form used to submit the data because 2. The example file upload script on the haskell wiki works. This is what led me to assume that it wasn't a problem with encoding in ghc 6.12, but now I'm not so sure due to the differences in the file size above.
I feel compelled to point out that, as far as the data of the file is concerned, I'm not transforming it (at least my naive Haskell knowledge tels me I'm not). Whether I am writing it out to a file, or pushing it to s3, I'm still getting it from parseMultipart, and then from fileContent.
That said, it doesn't make sense that the problem would be web.encodings. It's obviously being used, and more importantly, working in other places. It just seems like the places I could be making a mistake are rather few. Which, hehe, again points to an encoding issue.
Thanks again for the help, and I will post it up tomorrow if I still haven't figured it out.
Sounds like Bulat already solved the problem... but here's a sample of using web-encodings. I recommend using parsePost instead of parseMultipart. Michael

On Tue, May 18, 2010 at 12:18 AM, Michael Snoyman
Sounds like Bulat already solved the problem... but here's a sample of using web-encodings. I recommend using parsePost instead of parseMultipart.
Well, at least I'm not completely crazy - same thing happens with that code too. Time to start digging around to find out how to fix the encoding issue. Thanks again for your help, Michael and Bulat!

On Tue, May 18, 2010 at 11:17 AM, Clint Moore
On Tue, May 18, 2010 at 12:18 AM, Michael Snoyman
wrote: Sounds like Bulat already solved the problem... but here's a sample of using web-encodings. I recommend using parsePost instead of parseMultipart.
Well, at least I'm not completely crazy - same thing happens with that code too. Time to start digging around to find out how to fix the encoding issue.
Thanks again for your help, Michael and Bulat!
A few questions (this might still be my fault): * What operating system are you working on? * Which version of GHC? * What hack handler is your original code using? I'm worried that hack-handler-simpleserver might not be working properly on Windows due to encoding issues. I just booted up my Windows virtual machine and am downloading Haskell (I usually only use it for testing sites in IE). Michael

On Tue, May 18, 2010 at 1:28 AM, Michael Snoyman
A few questions (this might still be my fault):
* What operating system are you working on? ubuntu 10.4
* Which version of GHC? 6.12.2 from the GHC site.
* What hack handler is your original code using? Epoll
I had to switch to that for the example code as well - for some reason, simpleserver wasn't replying to requests. I went as far as to telnet to the port and make a request for / and didn't get anything back. telnet localhost 3000 -> GET /<enter><enter> and a few more <enter>s for good measure :) Maybe I should try h-h-happstack and see if that changes anything.
I'm worried that hack-handler-simpleserver might not be working properly on Windows due to encoding issues. I just booted up my Windows virtual machine and am downloading Haskell (I usually only use it for testing sites in IE). Michael

On Tue, May 18, 2010 at 1:32 AM, Clint Moore
On Tue, May 18, 2010 at 1:28 AM, Michael Snoyman
wrote: A few questions (this might still be my fault):
* What operating system are you working on? ubuntu 10.4
* Which version of GHC? 6.12.2 from the GHC site.
* What hack handler is your original code using? Epoll
I had to switch to that for the example code as well - for some reason, simpleserver wasn't replying to requests. I went as far as to telnet to the port and make a request for / and didn't get anything back. telnet localhost 3000 -> GET /<enter><enter> and a few more <enter>s for good measure :)
Maybe I should try h-h-happstack and see if that changes anything.
I'm worried that hack-handler-simpleserver might not be working properly on Windows due to encoding issues. I just booted up my Windows virtual machine and am downloading Haskell (I usually only use it for testing sites in IE). Michael
That was it. It works using hack-handler-happstack! Yay, now I can make my deadline. Thanks once again for all of your help! -cm

On Tue, May 18, 2010 at 11:49 AM, Clint Moore
On Tue, May 18, 2010 at 1:32 AM, Clint Moore
wrote: On Tue, May 18, 2010 at 1:28 AM, Michael Snoyman
wrote: A few questions (this might still be my fault):
* What operating system are you working on? ubuntu 10.4
* Which version of GHC? 6.12.2 from the GHC site.
* What hack handler is your original code using? Epoll
I had to switch to that for the example code as well - for some reason, simpleserver wasn't replying to requests. I went as far as to telnet to the port and make a request for / and didn't get anything back. telnet localhost 3000 -> GET /<enter><enter> and a few more <enter>s for good measure :)
Maybe I should try h-h-happstack and see if that changes anything.
I'm worried that hack-handler-simpleserver might not be working properly on Windows due to encoding issues. I just booted up my Windows virtual machine and am downloading Haskell (I usually only use it for testing sites in IE). Michael
That was it. It works using hack-handler-happstack!
Yay, now I can make my deadline. Thanks once again for all of your help!
-cm
Sure. I'd write a bug report to the epoll author. Funny thing about simpleserver not responding, I wonder why... Michael
participants (3)
-
Bulat Ziganshin
-
Clint Moore
-
Michael Snoyman