On Sun, 24 Aug 2008, Eric Kidd wrote:
Greetings, Haskell folks!
I'm working on a toy program for drum synthesis. This has lead me to explore the various sound-related libraries in Hackage. Along the way, I've learned several things:
1. There's a lot of Haskell sound libraries, but no agreement on how to represent buffers of audio data.
In my sound synthesis library http://darcs.haskell.org/synthesizer/ I use mainly two representations: 1. Chunky sequences based on StorableVector. This is for efficient storage and functions that need to process data in a non-causal way. They are not good for fusion. http://darcs.haskell.org/synthesizer/src/Synthesizer/Storable/Signal.hs 2. A Stream like sequences without wrappers. In fact, they are no data storages but generator functions equipped with a seed. I use them for causal signal processes. I think that most functions needed for drum synthesis are causal. By inlining them, many subsequent causal signal processes are fused to one. http://darcs.haskell.org/synthesizer/src/Synthesizer/State/Signal.hs
2. Some of the most useful sound libraries aren't listed in Hackage's "Sound" section, including HCodecs, SDL-mixer and hogg.
my library is also not at Hackage, it's too much experimental currently
3. The Haskell OpenAL bindings are great, but ALUT has been removed from MacOS 10.5. Of course, a pure-Haskell version of ALUT would be pretty easy to write, and it could be used as a fallback. 4. '[[a]]' is _not_ a good representation for multi-channel audio data. Manipulating even 10 seconds of audio is noticeably slow.
Using the signal representations I mentioned above, I can synthesize some songs in realtime. For stereo signals I use Signal (a,a). However, a StereoFrame data type with Storable instance might be even better.
5. HCodecs is probably the best library for reading WAVE files. It uses 'DiffUArray Int a' to represent audio data, and not '[[a]]'. It also supports MIDI and SoundFont, which is quite handy. 6. YampaSynth looks really cool, and it might be possible to turn it into a reusable MIDI synthesizer library for use with HCodecs. This would be extremely useful: it would provide a basis for all kinds of crazy MIDI-based programming in Haskell.
If you are after realtime synthesis you might also want to look into SuperCollider. I posted some code to control SuperCollider via MIDI to haskell-art. I can send it to you, if you like.
What would the ideal Haskell sound API look like? Personally, I would love to see:
a. Something like HCodecs's Data.Audio as the basic sound format. Data.Audio is an array-based API, and it supports converting between a wide range of common sample formats. I don't know how fast this is, but it already exists, and it's used by YampaSynth. b. OpenAL for sound playback, with a portable version of ALUT. This may require writing a pure-Haskell version of ALUT.
So far I used Sox' play command fed by a pipe with runInteractiveCommand.
Other nice-to-have features might include:
e. A standard MIDI format, based on either the HCodecs package or the midi package. (HCodecs is used by YampaSynth, and the midi package is used by alsa-midi.)
midi package is mainly used by Haskore
f. A modular version of YampaSynth which can convert MIDI data structures into Data.Audio values.
It looks like Haskell could be a really sweet audio programming environment with just a bit of integration work. What do folks think? Are there other libraries I should look at more closely? Other features that should be included in an ideal audio API?
What do you mean by 'API'? I think there should not be one library handling all kind of music tasks. For interoperability common data structures are nice. However, what signal representation concerns, I think we need more than one representation, because different applications have different needs.
Haskell audio libraries
I have advertised my libraries for years in Haskell-Wiki and HCAR, but it seems that people do not look there. :-(
Reading and writing sound files: HCodecs: (Audible a) => DiffUArray Int a hsndfile: MArray with Float and Double HSoundFile: [[Double]] ALUT: OpenAL.AL.Buffer WAVE: [[Int32]]
Playing sounds: OpenAL: Ptr UInt8, Ptr Int16, 1-N channels SDL-mixer: ForeignPtr ChunkStruct, ForeignPtr MusicStruct
Sox via pipe, using any data structure which can be written to a file
Sound processing libraries: dsp: Array a, [a] jack: Operates via mapping functions mainMono :: (CFloat -> IO CFloat) -> IO () mainStereo :: ((CFloat, CFloat) -> IO (CFloat, CFloat)) -> IO ()
synthesizer: StorableVector, State, Generic (older: list, fusable list)
MIDI-based: HCodecs: Reads and writes MIDI files midi: Reads and writes MIDI files alsa-midi: Uses midi library YampaSynth: Stand-alone program
Haskore uses MIDI and other back-ends like Haskell Synthesizer, CSound, SuperCollider
Special-purpose APIs only (FFIs, etc.): hCsound: CsoundPtr hsc3: UGen
HasSound
No public sound-buffer API: hbeat: The relevant source files are missing! hogg: Very low-level API for Ogg internals only libmpd: No sound buffer API sonic-visualizer: No sound buffer API truelevel: Stand-alone program (uses WAVE) wavconvert: Stand-alone program