Haskell audio libraries & audio formats

newer
Research language vs. professional...

Eric Kidd

24 Aug 2008 24 Aug '08

11:41 a.m.

Greetings, Haskell folks! I'm working on a toy program for drum synthesis. This has lead me to explore the various sound-related libraries in Hackage. Along the way, I've learned several things: 1. There's a lot of Haskell sound libraries, but no agreement on how to represent buffers of audio data. 2. Some of the most useful sound libraries aren't listed in Hackage's "Sound" section, including HCodecs, SDL-mixer and hogg. 3. The Haskell OpenAL bindings are great, but ALUT has been removed from MacOS 10.5. Of course, a pure-Haskell version of ALUT would be pretty easy to write, and it could be used as a fallback. 4. '[[a]]' is _not_ a good representation for multi-channel audio data. Manipulating even 10 seconds of audio is noticeably slow. 5. HCodecs is probably the best library for reading WAVE files. It uses 'DiffUArray Int a' to represent audio data, and not '[[a]]'. It also supports MIDI and SoundFont, which is quite handy. 6. YampaSynth looks really cool, and it might be possible to turn it into a reusable MIDI synthesizer library for use with HCodecs. This would be extremely useful: it would provide a basis for all kinds of crazy MIDI-based programming in Haskell. At the end of this e-mail, you can find a list of Haskell sound libraries and their supported audio formats. What would the ideal Haskell sound API look like? Personally, I would love to see: a. Something like HCodecs's Data.Audio as the basic sound format. Data.Audio is an array-based API, and it supports converting between a wide range of common sample formats. I don't know how fast this is, but it already exists, and it's used by YampaSynth. b. OpenAL for sound playback, with a portable version of ALUT. This may require writing a pure-Haskell version of ALUT. c. Utility functions to convert Data.Audio to OpenAL.AL.Buffer. d. An easy-to-use API for audio processing. It might be possible to start with something like Jack's 'mainMono' and 'mainStereo' functions, which map a function over an audio stream. Other nice-to-have features might include: e. A standard MIDI format, based on either the HCodecs package or the midi package. (HCodecs is used by YampaSynth, and the midi package is used by alsa-midi.) f. A modular version of YampaSynth which can convert MIDI data structures into Data.Audio values. It looks like Haskell could be a really sweet audio programming environment with just a bit of integration work. What do folks think? Are there other libraries I should look at more closely? Other features that should be included in an ideal audio API? Thank you for your advice and feedback! Cheers, Eric Haskell audio libraries Reading and writing sound files: HCodecs: (Audible a) => DiffUArray Int a hsndfile: MArray with Float and Double HSoundFile: [[Double]] ALUT: OpenAL.AL.Buffer WAVE: [[Int32]] Playing sounds: OpenAL: Ptr UInt8, Ptr Int16, 1-N channels SDL-mixer: ForeignPtr ChunkStruct, ForeignPtr MusicStruct Sound processing libraries: dsp: Array a, [a] jack: Operates via mapping functions mainMono :: (CFloat -> IO CFloat) -> IO () mainStereo :: ((CFloat, CFloat) -> IO (CFloat, CFloat)) -> IO () MIDI-based: HCodecs: Reads and writes MIDI files midi: Reads and writes MIDI files alsa-midi: Uses midi library YampaSynth: Stand-alone program Special-purpose APIs only (FFIs, etc.): hCsound: CsoundPtr hsc3: UGen No public sound-buffer API: hbeat: The relevant source files are missing! hogg: Very low-level API for Ogg internals only libmpd: No sound buffer API sonic-visualizer: No sound buffer API truelevel: Stand-alone program (uses WAVE) wavconvert: Stand-alone program

Show replies by date

John Van Enk

24 Aug 24 Aug

12:41 p.m.

Eric, I'm hoping to publish my library in the next week here: darcs get http://code.haskell.org/portaudio/ It implements the Haskell bindings to the PortAudio library. This is the library behind Audacious. I *have not* implemented the callback model yet, but I plan to do that. Perhaps this will be something which is useful to you? On Sun, Aug 24, 2008 at 11:41 AM, Eric Kidd wrote:

...

Greetings, Haskell folks!

I'm working on a toy program for drum synthesis. This has lead me to explore the various sound-related libraries in Hackage. Along the way, I've learned several things:

1. There's a lot of Haskell sound libraries, but no agreement on how to represent buffers of audio data. 2. Some of the most useful sound libraries aren't listed in Hackage's "Sound" section, including HCodecs, SDL-mixer and hogg. 3. The Haskell OpenAL bindings are great, but ALUT has been removed from MacOS 10.5. Of course, a pure-Haskell version of ALUT would be pretty easy to write, and it could be used as a fallback. 4. '[[a]]' is _not_ a good representation for multi-channel audio data. Manipulating even 10 seconds of audio is noticeably slow. 5. HCodecs is probably the best library for reading WAVE files. It uses 'DiffUArray Int a' to represent audio data, and not '[[a]]'. It also supports MIDI and SoundFont, which is quite handy. 6. YampaSynth looks really cool, and it might be possible to turn it into a reusable MIDI synthesizer library for use with HCodecs. This would be extremely useful: it would provide a basis for all kinds of crazy MIDI-based programming in Haskell.

At the end of this e-mail, you can find a list of Haskell sound libraries and their supported audio formats.

What would the ideal Haskell sound API look like? Personally, I would love to see:

a. Something like HCodecs's Data.Audio as the basic sound format. Data.Audio is an array-based API, and it supports converting between a wide range of common sample formats. I don't know how fast this is, but it already exists, and it's used by YampaSynth. b. OpenAL for sound playback, with a portable version of ALUT. This may require writing a pure-Haskell version of ALUT. c. Utility functions to convert Data.Audio to OpenAL.AL.Buffer. d. An easy-to-use API for audio processing. It might be possible to start with something like Jack's 'mainMono' and 'mainStereo' functions, which map a function over an audio stream.

Other nice-to-have features might include:

e. A standard MIDI format, based on either the HCodecs package or the midi package. (HCodecs is used by YampaSynth, and the midi package is used by alsa-midi.) f. A modular version of YampaSynth which can convert MIDI data structures into Data.Audio values.

It looks like Haskell could be a really sweet audio programming environment with just a bit of integration work. What do folks think? Are there other libraries I should look at more closely? Other features that should be included in an ideal audio API?

Thank you for your advice and feedback!

Cheers, Eric

Haskell audio libraries

Reading and writing sound files: HCodecs: (Audible a) => DiffUArray Int a hsndfile: MArray with Float and Double HSoundFile: [[Double]] ALUT: OpenAL.AL.Buffer WAVE: [[Int32]]

Playing sounds: OpenAL: Ptr UInt8, Ptr Int16, 1-N channels SDL-mixer: ForeignPtr ChunkStruct, ForeignPtr MusicStruct

Sound processing libraries: dsp: Array a, [a] jack: Operates via mapping functions mainMono :: (CFloat -> IO CFloat) -> IO () mainStereo :: ((CFloat, CFloat) -> IO (CFloat, CFloat)) -> IO ()

MIDI-based: HCodecs: Reads and writes MIDI files midi: Reads and writes MIDI files alsa-midi: Uses midi library YampaSynth: Stand-alone program

Special-purpose APIs only (FFIs, etc.): hCsound: CsoundPtr hsc3: UGen

No public sound-buffer API: hbeat: The relevant source files are missing! hogg: Very low-level API for Ogg internals only libmpd: No sound buffer API sonic-visualizer: No sound buffer API truelevel: Stand-alone program (uses WAVE) wavconvert: Stand-alone program _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- /jve

Eric Kidd

1:06 p.m.

On Sun, Aug 24, 2008 at 12:41 PM, John Van Enk wrote:

...

It implements the Haskell bindings to the PortAudio library. This is the library behind Audacious. I *have not* implemented the callback model yet, but I plan to do that.

Perhaps this will be something which is useful to you?

Looks like a great binding! We actually use portaudio in callback mode at work, with reasonable success (from C++). One question: I notice that your writeStream function represents audio using '[[a]]'. writeStream :: (Storable a) => PaStream a -- ^ The output stream -> [[a]] -- ^ The samples to be played -> Int -- ^ Number of frames -> IO (Either String ErrorCode) -- ^ The return status of the write In my experiments, I've been somewhat unsatisfied with the performance of '[[a]]' as an audio format. Would you be interested in also supporting an array-based format for audio data? I'm currently converting my program to use HCodec's Data.Audio representation, which looks pretty promising. I don't know how it performs yet, but I'll let you know. Thank you for the pointer to your library! It will make an excellent addition to the available Haskell sound libraries. Cheers, Eric

John Van Enk

4:13 p.m.

Eric, I was hoping to use a packed format like ByteString eventually. Right now, I want to get everything working nicely. As it stands, I end up marshaling a lot of information into/out of arrays which I'd much rather keep as a block of memory. I'm guessing that some sort of unboxed array would be close to what I want. If you have suggestions, I'm more than open to them. (This is my first attempt at writing a library.) /jve On Sun, Aug 24, 2008 at 1:06 PM, Eric Kidd wrote:

...

On Sun, Aug 24, 2008 at 12:41 PM, John Van Enk wrote:

...
It implements the Haskell bindings to the PortAudio library. This is the library behind Audacious. I *have not* implemented the callback model yet, but I plan to do that.

Perhaps this will be something which is useful to you?

Looks like a great binding! We actually use portaudio in callback mode at work, with reasonable success (from C++).

One question: I notice that your writeStream function represents audio using '[[a]]'.

writeStream :: (Storable a) => PaStream a -- ^ The output stream -> [[a]] -- ^ The samples to be played -> Int -- ^ Number of frames -> IO (Either String ErrorCode) -- ^ The return status of the write

In my experiments, I've been somewhat unsatisfied with the performance of '[[a]]' as an audio format. Would you be interested in also supporting an array-based format for audio data?

I'm currently converting my program to use HCodec's Data.Audio representation, which looks pretty promising. I don't know how it performs yet, but I'll let you know.

Thank you for the pointer to your library! It will make an excellent addition to the available Haskell sound libraries.

Cheers, Eric

-- /jve

Eric Kidd

4:52 p.m.

On Sun, Aug 24, 2008 at 4:13 PM, John Van Enk wrote:

...

I'm guessing that some sort of unboxed array would be close to what I want. If you have suggestions, I'm more than open to them. (This is my first attempt at writing a library.)

I'm still trying to answer this question myself. :-) I'll let you know how HCodec's AudioData format actually works in practice once I my code is converted. Honestly, an ideal format should allow iteration over one or more StorableArray chunks. This would make it much easier to interact with the outside world. Unfortunately, HCodec's DiffUArray format seems to require a conversion step to get a StorableArray, so I'm going to have to benchmark it. Good luck with your first Haskell library! Cheers, Eric

Henning Thielemann

25 Aug 25 Aug

3:01 a.m.

On Sun, 24 Aug 2008, John Van Enk wrote:

...

Eric,

I was hoping to use a packed format like ByteString eventually. Right now, I want to get everything working nicely. As it stands, I end up marshaling a lot of information into/out of arrays which I'd much rather keep as a block of memory.

I'm guessing that some sort of unboxed array would be close to what I want. If you have suggestions, I'm more than open to them. (This is my first attempt at writing a library.)

In recent versions of StorableVector (not at hackage yet, only in darcs repository) there is a chunky lazy variant. However there are also the uvector and vector packages, and no-one seems to know, on which one we should focus. Let me elaborate on my point, that there is not one data structre which fits all needs: The great thing about Haskell's laziness is that you can implement feedback easily. But this does not work with chunky lazy sequences (StorableVector.Lazy) and causes quadratic computation time with my State.Signal representation. It works only with element-wise lazy structures, aka lists. I have an idea for a more efficient data structure for that problem, but I'm uncertain if it is really more efficient in the end.

Don Stewart

1:23 p.m.

lemming:

...

On Sun, 24 Aug 2008, John Van Enk wrote:

...
Eric,

I was hoping to use a packed format like ByteString eventually. Right now, I want to get everything working nicely. As it stands, I end up marshaling a lot of information into/out of arrays which I'd much rather keep as a block of memory.

I'm guessing that some sort of unboxed array would be close to what I want. If you have suggestions, I'm more than open to them. (This is my first attempt at writing a library.)

In recent versions of StorableVector (not at hackage yet, only in darcs repository) there is a chunky lazy variant. However there are also the uvector and vector packages, and no-one seems to know, on which one we should focus.

There's no immediate plans to merge them, but uvector will likely gain a lazy variant soonish. As 'vector's announcement said, uvector is the stable, more conservative variant that works with 6.8.2, and has a richer API. -- Don

John Van Enk

2:16 p.m.

How well would the storablevector package (Data.StorableVector) work for storing audio data? One of the major issues I'm still working over is that I want to maintain something similar to a [[a]] format (since the underlying PortAudio library and hardware could support hundreds of interleaved channels) but I would like to be able to build in some typechecking to the functions to make sure the number of channels matches the nubmer expected in the functions. I've considered doing the following:

...

data SingleChannel data DualChannel data TriChannel type Frames = [[a]]

...

data AudioData a = AudioData Frames

...

adata = [[x] | x <- [...]] :: AudioData SingleChannel

I'd like to have a better way of: 1) packing the audio data in something other than a list 2) representing more information about the data parameters (such as sample rate and number of channels) without relying as heavily upon runtime checks/errors (which HCodecs Data.Audio format seems to rely on). The ideal format would be [(a,a,..a)], but this seems cumbersome (since each tuple would have to be defined individually), I'm pretty sure that a runtime check on both sample rate, and number of channels will be necessary, but I just want to ask a little more before continuing down that path. I also noticed that Data.Audio uses a DiffUArray. In this circumstance, would a StorableVector be better, or a DiffUArray? On Sun, Aug 24, 2008 at 1:06 PM, Eric Kidd wrote:

...

On Sun, Aug 24, 2008 at 12:41 PM, John Van Enk wrote:

...
It implements the Haskell bindings to the PortAudio library. This is the library behind Audacious. I *have not* implemented the callback model yet, but I plan to do that.

Perhaps this will be something which is useful to you?

Looks like a great binding! We actually use portaudio in callback mode at work, with reasonable success (from C++).

One question: I notice that your writeStream function represents audio using '[[a]]'.

writeStream :: (Storable a) => PaStream a -- ^ The output stream -> [[a]] -- ^ The samples to be played -> Int -- ^ Number of frames -> IO (Either String ErrorCode) -- ^ The return status of the write

In my experiments, I've been somewhat unsatisfied with the performance of '[[a]]' as an audio format. Would you be interested in also supporting an array-based format for audio data?

I'm currently converting my program to use HCodec's Data.Audio representation, which looks pretty promising. I don't know how it performs yet, but I'll let you know.

Thank you for the pointer to your library! It will make an excellent addition to the available Haskell sound libraries.

Cheers, Eric

-- /jve

Henning Thielemann

26 Aug 26 Aug

2:11 p.m.

On Mon, 25 Aug 2008, John Van Enk wrote:

...

How well would the storablevector package (Data.StorableVector) work for storing audio data? One of the major issues I'm still working over is that I want to maintain something similar to a [[a]] format (since the underlying PortAudio library and hardware could support hundreds of interleaved channels) but I would like to be able to build in some typechecking to the functions to make sure the number of channels matches the nubmer expected in the functions.

With data Stereo a = Stereo !a !a you could also use Stereo (Stereo a) for quadrophony and so on. Would this be convenient enough? StorableVector stores everything of fixed length for which a Storable instance is defined.

John Van Enk

2:30 p.m.

I think the problem I'll run into is the 128 channel case. I'm hoping for a general solution... I'm almost positive this will require runtime checks. Your solution is what I was thinking for functions requiring exactly N channels (I'm not sure if there are many functions like that). On Tue, Aug 26, 2008 at 2:11 PM, Henning Thielemann < lemming@henning-thielemann.de> wrote:

...

On Mon, 25 Aug 2008, John Van Enk wrote:

How well would the storablevector package (Data.StorableVector) work for

...
storing audio data? One of the major issues I'm still working over is that I want to maintain something similar to a [[a]] format (since the underlying PortAudio library and hardware could support hundreds of interleaved channels) but I would like to be able to build in some typechecking to the functions to make sure the number of channels matches the nubmer expected in the functions.

With data Stereo a = Stereo !a !a

you could also use Stereo (Stereo a) for quadrophony and so on. Would this be convenient enough?

StorableVector stores everything of fixed length for which a Storable instance is defined.

-- /jve

Henning Thielemann

6:17 p.m.

On Tue, 26 Aug 2008, John Van Enk wrote:

...

I think the problem I'll run into is the 128 channel case. I'm hoping for a general solution... I'm almost positive this will require runtime checks. Your solution is what I was thinking for functions requiring exactly N channels (I'm not sure if there are many functions like that).

If the number of channels is variable it might be better to use a list of StorableVectors instead. I think it is more common to process the channels separately instead of simultaneously.

Don Stewart

24 Aug 24 Aug

3:55 p.m.

haskell:

...

At the end of this e-mail, you can find a list of Haskell sound libraries and their supported audio formats.

Could you add that detail to the Haskell wiki Audio page? -- Don

Henning Thielemann

25 Aug 25 Aug

2:52 a.m.

On Sun, 24 Aug 2008, Eric Kidd wrote:

...

Greetings, Haskell folks!

I'm working on a toy program for drum synthesis. This has lead me to explore the various sound-related libraries in Hackage. Along the way, I've learned several things:

1. There's a lot of Haskell sound libraries, but no agreement on how to represent buffers of audio data.

In my sound synthesis library http://darcs.haskell.org/synthesizer/ I use mainly two representations: 1. Chunky sequences based on StorableVector. This is for efficient storage and functions that need to process data in a non-causal way. They are not good for fusion. http://darcs.haskell.org/synthesizer/src/Synthesizer/Storable/Signal.hs 2. A Stream like sequences without wrappers. In fact, they are no data storages but generator functions equipped with a seed. I use them for causal signal processes. I think that most functions needed for drum synthesis are causal. By inlining them, many subsequent causal signal processes are fused to one. http://darcs.haskell.org/synthesizer/src/Synthesizer/State/Signal.hs

...

2. Some of the most useful sound libraries aren't listed in Hackage's "Sound" section, including HCodecs, SDL-mixer and hogg.

my library is also not at Hackage, it's too much experimental currently

...

3. The Haskell OpenAL bindings are great, but ALUT has been removed from MacOS 10.5. Of course, a pure-Haskell version of ALUT would be pretty easy to write, and it could be used as a fallback. 4. '[[a]]' is _not_ a good representation for multi-channel audio data. Manipulating even 10 seconds of audio is noticeably slow.

Using the signal representations I mentioned above, I can synthesize some songs in realtime. For stereo signals I use Signal (a,a). However, a StereoFrame data type with Storable instance might be even better.

...

5. HCodecs is probably the best library for reading WAVE files. It uses 'DiffUArray Int a' to represent audio data, and not '[[a]]'. It also supports MIDI and SoundFont, which is quite handy. 6. YampaSynth looks really cool, and it might be possible to turn it into a reusable MIDI synthesizer library for use with HCodecs. This would be extremely useful: it would provide a basis for all kinds of crazy MIDI-based programming in Haskell.

If you are after realtime synthesis you might also want to look into SuperCollider. I posted some code to control SuperCollider via MIDI to haskell-art. I can send it to you, if you like.

...

What would the ideal Haskell sound API look like? Personally, I would love to see:

a. Something like HCodecs's Data.Audio as the basic sound format. Data.Audio is an array-based API, and it supports converting between a wide range of common sample formats. I don't know how fast this is, but it already exists, and it's used by YampaSynth. b. OpenAL for sound playback, with a portable version of ALUT. This may require writing a pure-Haskell version of ALUT.

So far I used Sox' play command fed by a pipe with runInteractiveCommand.

...

Other nice-to-have features might include:

e. A standard MIDI format, based on either the HCodecs package or the midi package. (HCodecs is used by YampaSynth, and the midi package is used by alsa-midi.)

midi package is mainly used by Haskore

...

f. A modular version of YampaSynth which can convert MIDI data structures into Data.Audio values.

It looks like Haskell could be a really sweet audio programming environment with just a bit of integration work. What do folks think? Are there other libraries I should look at more closely? Other features that should be included in an ideal audio API?

What do you mean by 'API'? I think there should not be one library handling all kind of music tasks. For interoperability common data structures are nice. However, what signal representation concerns, I think we need more than one representation, because different applications have different needs.

...

Haskell audio libraries

I have advertised my libraries for years in Haskell-Wiki and HCAR, but it seems that people do not look there. :-(

...

Reading and writing sound files: HCodecs: (Audible a) => DiffUArray Int a hsndfile: MArray with Float and Double HSoundFile: [[Double]] ALUT: OpenAL.AL.Buffer WAVE: [[Int32]]

Playing sounds: OpenAL: Ptr UInt8, Ptr Int16, 1-N channels SDL-mixer: ForeignPtr ChunkStruct, ForeignPtr MusicStruct

Sox via pipe, using any data structure which can be written to a file

...

Sound processing libraries: dsp: Array a, [a] jack: Operates via mapping functions mainMono :: (CFloat -> IO CFloat) -> IO () mainStereo :: ((CFloat, CFloat) -> IO (CFloat, CFloat)) -> IO ()

synthesizer: StorableVector, State, Generic (older: list, fusable list)

...

MIDI-based: HCodecs: Reads and writes MIDI files midi: Reads and writes MIDI files alsa-midi: Uses midi library YampaSynth: Stand-alone program

Haskore uses MIDI and other back-ends like Haskell Synthesizer, CSound, SuperCollider

...

Special-purpose APIs only (FFIs, etc.): hCsound: CsoundPtr hsc3: UGen

HasSound

...

No public sound-buffer API: hbeat: The relevant source files are missing! hogg: Very low-level API for Ogg internals only libmpd: No sound buffer API sonic-visualizer: No sound buffer API truelevel: Stand-alone program (uses WAVE) wavconvert: Stand-alone program

Chris Waterson

7:16 p.m.

On Aug 24, 2008, at 8:41 AM, Eric Kidd wrote:

...

Greetings, Haskell folks!

I'm working on a toy program for drum synthesis. This has lead me to explore the various sound-related libraries in Hackage. Along the way, I've learned several things:

Hey Erik, I've also created (minimal, for fun) bindings to libmad and OS/X CoreAudio. The original post is here: http://www.haskell.org/pipermail/haskell-cafe/2008-March/040796.html Here is a direct link to the source: http://maubi.net/~waterson/REPO/hmad http://maubi.net/~waterson/REPO/CoreAudio I represented the samples using a lazily generated list of numbers, and it worked pretty well. chris

Gwern Branwen

3 Sep 3 Sep

5:17 p.m.

On 2008.08.24 11:41:04 -0400, Eric Kidd scribbled 3.7K characters:

...

Greetings, Haskell folks!

I'm working on a toy program for drum synthesis. This has lead me to explore the various sound-related libraries in Hackage. Along the way, I've learned several things:

1. There's a lot of Haskell sound libraries, but no agreement on how to represent buffers of audio data. 2. Some of the most useful sound libraries aren't listed in Hackage's "Sound" section, including HCodecs, SDL-mixer and hogg.

Have you contacted the authors to ask they add the Sound categorization? I already sent a hogg patch, but couldn't find any darcs repositories for SDL-mixer or HCodecs. ...

...

No public sound-buffer API: hbeat: The relevant source files are missing!

The cabal file was missing the Other-modules field. I've sent the author a patch. -- gwern Surveillance Defcon S511 anarchy REP NSIRL grenades dort UT/RUS W

6107

Age (days ago)

6117

Last active (days ago)

List overview

Download

14 comments

6 participants

participants (6)

Chris Waterson
Don Stewart
Eric Kidd
Gwern Branwen
Henning Thielemann
John Van Enk