I end up with this type.

data Hdf5M a
  = H5Root (Hdf5M a)
  | H5Group ByteString [Hdf5M a]  -- A group can contain other groups and/or datasets
  | forall sh b. (NativeType b, Shape sh) => H5Dataset ByteString (Array F sh b)

[…]

And I would like to be able to describe a contain like this

hdf5 $ do
      group "name" $ do
            dataset "name1" array1
            dataset "name2" array2
            group "other-name" $ do
                     etc...

You don't need a monad instance for this. First of all, you don't even need do syntax to make something "pretty" similar to this.

	hdf5 $
	    group "name" $
	        [ dataset "name1" array1
	        , dataset "name2" array2
	        , group "other-name" $ [
	            …

But if you insist, you can just use an existing monad like WriterT. For example (simplified):

	data Composite a = Base a | Composite [Composite a]
	
	type CompositeWriter a = Writer [Composite a] ()
	
	base = tell . pure . Base
	
	composite = censor (pure . Composite)
	
	asComposite = Composite . snd . runWriter
	
	test = asComposite $ do
	    base 'a'
	    base 'b'
	    composite $ do
	      base 'c'
	      base 'd'

Hope that helps.