The methodology does look reasonable, although I think you should wait for all the scotty threads to start before starting the benchmarks, as I see this interleaved output: Setting phasers to stun... (port 3002) (ctrl-c to quit) Setting phasers to stun... (port 3003) (ctrl-c to quit) Setting phasers to stun... (port 3001) (ctrl-c to quit) benchmarking bareScotty Setting phasers to stun... (port 3000) (ctrl-c to quit) Your numbers are wayyy slower than the ones I see on my dev machine: benchmarking bareScotty Setting phasers to stun... (port 3000) (ctrl-c to quit) time 10.94 ms (10.36 ms .. 11.52 ms) 0.979 R² (0.961 R² .. 0.989 R²) mean 12.53 ms (11.98 ms .. 13.28 ms) std dev 1.702 ms (1.187 ms .. 2.589 ms) variance introduced by outliers: 66% (severely inflated) benchmarking bareScottyBareLucid time 12.95 ms (12.28 ms .. 13.95 ms) 0.972 R² (0.951 R² .. 0.989 R²) mean 12.20 ms (11.75 ms .. 12.69 ms) std dev 1.236 ms (991.3 μs .. 1.601 ms) variance introduced by outliers: 50% (severely inflated) benchmarking transScottyBareLucid time 12.05 ms (11.70 ms .. 12.39 ms) 0.992 R² (0.982 R² .. 0.996 R²) mean 12.43 ms (12.06 ms .. 13.01 ms) std dev 1.320 ms (880.5 μs .. 2.071 ms) variance introduced by outliers: 54% (severely inflated) benchmarking transScottyTransLucid time 39.73 ms (32.16 ms .. 49.45 ms) 0.668 R² (0.303 R² .. 0.969 R²) mean 42.59 ms (36.69 ms .. 54.38 ms) std dev 16.52 ms (8.456 ms .. 25.96 ms) variance introduced by outliers: 92% (severely inflated) benchmarking bareScotty time 11.46 ms (10.89 ms .. 12.07 ms) 0.986 R² (0.975 R² .. 0.994 R²) mean 11.73 ms (11.45 ms .. 12.07 ms) std dev 800.6 μs (636.8 μs .. 975.3 μs) variance introduced by outliers: 34% (moderately inflated) but nonetheless I do also see the one using renderTextT to be substantially slower than the one without. I've sent you a PR [1] that isolates Lucid from Scotty and shows that renderTextT is twice as slow over IO than it is over Identity, and it's ~10% slower over Reader too: benchmarking renderText time 5.529 ms (5.328 ms .. 5.709 ms) 0.990 R² (0.983 R² .. 0.995 R²) mean 5.645 ms (5.472 ms .. 5.888 ms) std dev 593.0 μs (352.5 μs .. 908.2 μs) variance introduced by outliers: 63% (severely inflated) benchmarking renderTextT Id time 5.439 ms (5.243 ms .. 5.640 ms) 0.991 R² (0.985 R² .. 0.996 R²) mean 5.498 ms (5.367 ms .. 5.631 ms) std dev 408.8 μs (323.8 μs .. 552.9 μs) variance introduced by outliers: 45% (moderately inflated) benchmarking renderTextT Rd time 6.173 ms (5.983 ms .. 6.396 ms) 0.990 R² (0.983 R² .. 0.995 R²) mean 6.284 ms (6.127 ms .. 6.527 ms) std dev 581.6 μs (422.9 μs .. 773.0 μs) variance introduced by outliers: 55% (severely inflated) benchmarking renderTextT IO time 12.35 ms (11.84 ms .. 12.84 ms) 0.989 R² (0.982 R² .. 0.995 R²) mean 12.22 ms (11.85 ms .. 12.76 ms) std dev 1.159 ms (729.5 μs .. 1.683 ms) variance introduced by outliers: 50% (severely inflated) I tried replacing forM [1..10000] (\_ -> div_ "hello world!") with replicateM_ 10000 (div_ "hello world!") which discards the list of 10,000 () values that the forM thing generates, but this made very little difference. Hope this helps, David [1] https://github.com/vacationlabs/monad-transformer-benchmark/pull/2 On 29 January 2017 at 07:26, Saurabh Nanda wrote:

Hi,

I was noticing severe drop in performance when Lucid's HtmlT was being combined with Scotty's ActionT. I've tried putting together a minimal repro at https://github.com/vacationlabs/monad-transformer-benchmark Request someone with better knowledge of benchmarking to check if the benchmarking methodology is correct.

Is my reading of 200ms performance penalty correct?

-- Saurabh.

_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.