Making A Computer Sing

Making singing sounds via processed sampling or re-synthesis is good and effective, but I have - for years - wanted to do it from first principles.

Now I have managed. At least to some extent. The aim (at the moment and since I started about 18 months ago) is to make a convincing, sung vowel sound. This has proved very, very difficult indeed. I have made a few spooky whisper using white noise and formant filtering. The effect is used here:  

The problem always came when trying to move over to the intensity of a true sung note. Formant filtering of a sawtooth, for example, just sounds like a filtered sawtooth and not very nice either. I have spent a long time looking at the spectra of human speech started to notice that the frequency bands of the overtones, even from a single voice, as much wider than for an normal music instrument. Even when not accounting for vibrato, the human voice does not produce a sharp, well defined fundamental and overtones. Yes, the structure is there, but 'fatter' for each partial.

A few more experiments showed that convolving a sawtooth with white noise and then formant filtering did not do the job either. Trying to 'widen' the partials with frequency modulation also failed. Then I started to consider what a real human voice does. The voice system is not a rigid resonator like a clarinet or organ pipe. The vocal folds are 'floppy' and so there needs to be a band of frequencies around each partial. The power of human singing is all in the filtering done with the air column started in the lungs and ending with the nose and mouth. 

This though process lead me to use additive synthesis to produce a rich set of sounds around the base frequency and its partials. Then I upped the filtering a lot. Not just formant filtering, but band pass filtering around the formants. I.e. not just resonant peaks but cutting between the formant.

Listening to the results of this was really interesting. Some notes sounded quite like they were sung, others totally failed and sounded just like string instruments. Careful inspection of the spectra of each case showed that where partials lined up with format frequencies the result sounded like singing; where the formants lay between partials, it sounded like a string instrument. I realised that there is a huge difference between a bad singer (say, me) and a great singer. Maybe great singers are doing something subtle with the formants to get that pure sound.

This is my current point with the technique. Each vowel has 3 formants. I leave the bottom one untouched. However, the upper to I align the formants to the nearest harmonic of the note. Synthesis done this way produces a reliable, consistent sound somewhere between a human singing and a bowed string instrument. Here is an example:

Next I want to try using notch filters to knock out specific harmonics to see if I can get rid of some of that string sound.

Here is the main filter bank for the singing effect heard above:

def findNearestOvertone(fMatch,freq):

    q=float(fMatch)/float(freq)

    q=int(q)

    return freq*q

def doFormant(sig,f1,f2,f3,freq,intensity=4):

    f1b=f1

    f2b=findNearestOvertone(f2,freq)

    f3b=findNearestOvertone(f3,freq)

    print "Match: ",freq,f1,f2,f3,f1b,f2b,f3b

    for x in range(1,intensity):

        s1=sf.RBJBandPass(+sig,f1b,0.25)

        s2=sf.RBJBandPass(+sig,f2b,0.5)

        s3=sf.RBJBandPass(+sig,f3b,0.5)

        sig=sf.FixSize(

            sf.Mix(

                sf.Pcnt10(sig),

                sf.Pcnt50(sf.FixSize(s1)),

                sf.Pcnt20(sf.FixSize(s2)),

                sf.Pcnt30(sf.FixSize(s3))

            )

        )

        s1=sf.RBJPeaking(+sig,f1b,1.0,5)

        s2=sf.RBJPeaking(+sig,f2b,2.0,5)

        s3=sf.RBJPeaking( sig,f3b,2.0,5)

        sig=sf.FixSize(

            sf.Mix(

                sf.Pcnt50(sf.FixSize(s1)),

                sf.Pcnt20(sf.FixSize(s2)),

                sf.Pcnt30(sf.FixSize(s3))

            )

        )

    x=polish(sig,freq)

    x=sf.FixSize(x)

    sf.Check(x)

    return x

The big steps forward have been the frequency matching and the band pass filtering. I hope that 2 notch filters at the centres of s1,s2 and s2,3 will help; however, I expect it will be more complex than that; it always is with singing@

Mimicking Reale Note Evolution

Humans seem to like listening to real things, tubes, strings and such. 

These real things don't make one tone in a sound, their tone evolves. Here I discuss a way of using resampling (FM if you like) to achieve that effect.

Here is part of the patch I used to create Contrapunctus III

        if tp==0:

            env=sf.NumericShape((0,0),(96,1),(192,0.5),(length,0))

        elif tp==1:

            env=sf.NumericShape((0,0),(96,0.75),(length-128,1.0),(length,0))

        elif tp==2:

            env=sf.NumericShape((0,0),(96,0.75),(length-256,1.0),(length,0))

        elif tp==3:

            if length<1280:

                env=sf.NumericShape((0,0),(64,0.5),(256,1),(512,0.75),(length-256,0.5),(length,0))

            else:

                env=sf.NumericShape((0,0),(64,0.5),(256,1),(512,0.75),(length-512,.75),(length,0))

        else:

            env=sf.NumericShape((0,0),(64,0.25),(512,1),(length/2,0.75),(length,0))

        sig=sf.Multiply(sig,+env)

        sig=sf.FixSize(sig)

        mod=sf.NumericShape((0,0.995),(length,1.005))

        mod=sf.Mix(mod,sf.NumericVolume(env,0.01))

        sig=sf.FrequencyModulate(sig,mod)  

        sig=sf.FixSize(sig)

Here is the outcome:

So, let's go through the patch and see how it works:

        if tp==0:

            env=sf.NumericShape((0,0),(96,1),(192,0.5),(length,0))

        elif tp==1:

            env=sf.NumericShape((0,0),(96,0.75),(length-128,1.0),(length,0))

        elif tp==2:

            env=sf.NumericShape((0,0),(96,0.75),(length-256,1.0),(length,0))

        elif tp==3:

            if length<1280:

                env=sf.NumericShape((0,0),(64,0.5),(256,1),(512,0.75),(length-256,0.5),(length,0))

            else:

                env=sf.NumericShape((0,0),(64,0.5),(256,1),(512,0.75),(length-512,.75),(length,0))

        else:

            env=sf.NumericShape((0,0),(64,0.25),(512,1),(length/2,0.75),(length,0))

        sig=sf.Multiply(sig,+env)

First we have a dynamic envelope generator; it chooses the envelope based on the original length of the node (encoded in tp) and the current length (after some modification). The final line applies the envelope to the signal by simply multiplying one with the other. Note that NumericShape creates linear envelopes. Whilst Sonic Field can make exponential envelopes (via SimpleShape) and combination envelopes (by multiplying exponential and linear ones) I personally find a it easier to control linear envelopes and to add detail to them to mimic exponential behaviour.

So far so standard. This approach is pretty much text book synthesis plus some smarts over envelope shape. We need to make the shape 'full size' again so that we don't get unwanted volume effects. IE I tend to keep bringing signals back up to a deviation max of 1 so that when I do apply a volume to them, I get exactly what I want.

        sig=sf.FixSize(sig)

Does the the deviation correction. In the past I used Normalise a lot instead. This is actually more powerful because it removes any over all 'DC" component. By DC (an old fashion term to say the least) I mean an over all offset from zero. A pure tone (say a sine wave) should, when all the samples are added together, have a absolute offset of 0. Normalise makes this happen; in so doing it sometimes move the beginning or end sample of a signal away from zero. When the signals are mixed this puts clicks and pops into the sound. Now I try to use FixSize throughout and only use Normalise at the last step.

        mod=sf.NumericShape((0,0.995),(length,1.005))

        mod=sf.Mix(mod,sf.NumericVolume(env,0.01))

        sig=sf.FrequencyModulate(sig,mod)  

        sig=sf.FixSize(sig)

Finally we get tot the point of the post. I create an envelope which moves from 0.995 to 1.005. I then add to that the volume envelope which was used. However, I scale the volume envelop right down to 1% of its original size. What this gives me is a shape which moves over all up a bit but with more up when the volume is high. 

        sig=sf.FrequencyModulate(sig,mod)  

FrequencyModulate then 'bends' the note to the envelope. This mimics the pitch rising throughout the note (a little) and being higher when the volume is higher. That sort of bend - to my ears - is quite typical of a bowed instrument or a blown tube (flute or diapason).

Simple Harmonic Exciter

Harmonic Excitation is a technique to add harmonics which are missing or generally give more depth and energy to a top end - here is a patch I having quite a bit of success with:

def excite(sig_,mix,power):

    def exciteInner():

        sig=sig_

        m=sf.Magnitude(+sig)

        sigh=sf.BesselHighPass(+sig,500,2)

        mh=sf.Magnitude(+sigh)

        sigh=sf.Power(sigh,power)

        sigh=sf.BesselHighPass(sigh,1000,2)

        nh=sf.Magnitude(+sigh)

        sigh=sf.NumericVolume(sigh,mh/nh)

        sig=sf.Mix(sf.NumericVolume(sigh,mix),sf.NumericVolume(sig,1.0-mix))

        n=sf.Magnitude(+sig)

        return sf.Realise(sf.NumericVolume(sig,m/n))

    return sf_do(exciteInner)

The trick is to separate the top end using Bessel filters to avoid too much phase distortion. Then I use the 'power' filter which takes the signal and raises its instantaneous magnitude to a power. This adds distortion which add harmonics and strengthens the existing ones. The result is mixed back into the original after a second filter helps remove some of the original signal so we here a boost to the new harmonics.

I use Magnitude to make the effect not change the over all magnitude of the signal and to over come the volume reducing effect of power to the low magnitude filtered signal.

Simple - but it works nicely as part of the reverberation post processing for this track:

Bach Contrapunctus III