#13630: ByteString pinned memory can be leaky -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 8.0.1 System | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- My question on IRC: {{{ How does memory allocation for pinned blocks work? Let's say pinned blocks are 4KB in size, and I allocate first a 3 KB ByteString A and then an 8-byte ByteString B. Now I GC A, no longer need it. Then according to https://ghc.haskell.org/trac/ghc/wiki/Commentary/Rts/Storage/GC/Pinned "a single pinned object keeps alive the whole block in which it resides", my small ByteString B keeps the entire block alive. But what happens with the 3KB in the front of that block? Will it be re-used by the next ByteString allocation (say 1KB)? In other words, how smart is allocatePinned as an allocator? }}} The answer: {{{ slyfox: nh2: allocatePinned it quite dump. it only allocated from free tail space }}} {{{ nh2: slyfox: that sounds like a huge potential for memory leak then slyfox: yes, fragmentation is quite bad for bytestrings }}} So it seems that I can get into the unfortunate situation where a super short `ByteString` of a few bytes can waste an entire 4 KB block of memory; some migth call this a leak. One idea to solve it seems to be to change standard `ByteStrings` to not pinned, and to allocate pinned ones explicitly when needed. This seems to be an often-discussed topic and not trivial because many `ByteString` functions are implemented using libc FFI functions. However, it seems there will always be _some_ need for pinned memory, so we should better have an efficient way to manage it in any case. Efficient here means, for example, to re-use freed memory inside a block instead of only using free tail space. It seems that `jemalloc` has a feature to use given blocks of memory and provide a `malloc()` functionality inside them: https://stackoverflow.com/questions/30836359/jemalloc-mmap-and-shared- memory Perhaps this could be used to provide GHC with a simple method to use pinned memory more efficiently? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13630 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler