Spectrogram in Milkdrop

[SteveW]

Hi,

I know I'm risking being flamed for cross-posting, but my posting isn't without having spent quite some time searching throught the MilkDrop forum for answers to my question.

I orignially asked this question in the AVS forum, but since have decided that MilkDrop has more to offer my development efforts (use of 3D graphics acceleration), so I'm posting my question here.

I'm trying to create a real-time Spectrogram that responds to the live input of music, ultimately to accompany live instrumental performance. A static visual that gives you the basic idea has been attached to this posting.

Of course, I'm interested in going way beyond a plain real-time spectrogram, but want to enhance this with 3D effects to essentially create what I call a 'Spectrogram on Steroids'

I've read many impressive comments about the amazing capabilities of MilkDrop, but after having looked at MANY available presets, haven't seen anything like this.

And in looking through the tutorials, I haven't found a clear way of coupling the pitch (frequency, as determined by FFT I assume) with an associated graphical function.

Please excuse my inexperience here. If there is anyone who can direct me as to the appropriateness of MilkDrop for this project, and how to get started with this particular type of visualization, I'd most appreciate it.

Thanks,

SteveW

[SteveW]

Just for added clairification:

A horizontal line in the display would scroll from left to right, changing in relative height, based on the relative pitch within the scale of the instrument.

This differs fundamentally from all the Milkdrop visualizations I've seen, all of which seem to be radially designed - emanating from the center out, or randomly, not in a steady scroll from left to right. Is this type of movement possible with MilkDrop?

SteveW

(Sorry, was unable to attach the jpg in the last post. Looks similar to a heart rate monitor display though)

[redi jedi]

i'm not totaly sure what your looking for, but there is a preset somewhere that looks like a heart monitor-> i think its in the preset pack on my website - milkdrop.us/presets.aspx

as far as the left-right scrolling, just use the 'dx' variable, it centers on 0 so +-.001 will prolly work(depending in the speed you want it to scroll at) you can use the custom wave to draw the point(that gets dragged out into a line) that does what you want it too.. as far as getting good frequency data... i guess put the wave in spectrum mode and do some testing for the higest value(strongest frequency) but i'm still not sure what you want it to look like, you may be best off figureing out how to make the wave look the way you want while its static, then using the audio data to modulate its speeds or something....

ither way i would need a better discripton of what you want it to look like and how you want it to respond to the music.

[SteveW]

Thanks for your reply, Jedi.

Here's a link to a spectrogram that I created of a flute playing 'Amazing Grace' using a spectrogram program I found on the internet (called 'Spectrogram'):


Of course what I want to create will be an enhanced version of what you see here, without the scientific trappings, expandable to full screen, possibly with two insrtuments input - one into each of the stereo channels -shown in different colors, with 3D effects, etc., etc...

[redi jedi]

well it depends on what exactly is being graphed here, is that time/frequency? if so is it just the strongest frequency? cause thats not too hard.

basicly in side the wave you have value1 + value2 one for the right channel one for the left(if your gonna try pusing diffrent signals through the channels you'll want the stero sepration variable as high as posiable) normaly there just the amplitude of the channel, but you can put it in Spectrum mode and it will return the amplude of frequency. the Sample variable is how far through the current sample we are, so in non-Spectrum its just that, but in Spectrum mode it corresponds to the frequency, so if you do:
x=sample;
y=vaule1;

in the per-point of the custom wave, in non-Spectrum mode you get your normal time/amp graph, but in spectrum mode it gives you the time/frequency(spectrascope) for this sample(frame).....


from the looks of it you want this to be streched out over time(the 'time' quoted above is only a few hundreths of a second) and want to basicly draw a line that is/was the loudest frequency we can see,the bottom of the screen being the lowest frequency and the top of the screen being the highest, right?

assuming thats the case your gonna wanna just draw the dot for this frame with the custom wave, then use the 'dx' var to drag it across the screen.
in the wave you'll just need to basicly just put it in Spectrum mode, and make a test to see if the last value1(custom variables hold there value from one sample to the next, but not from one frame to the next.) is higher than this one, if it is higher, you want to save the value of sample(rember that holds how far we are through the frequencys) then on the last sample(when the variable sample is equal to 1) you want to draw your dot all the way to one side on x, and for y you would use your saved value of sample(witch should now hold the value of sample when vaule1 was at its higest)..... so something like:

per-point code:

bigger=above(value1,maxvalue);
bigsamp=bigsamp*(1-bigger)+bigger*sample; //basicly an if statment, if bigger use current sample, else keep value
maxvalue=max(value1,maxvalue);
x=.99; // just inside the edge of the screen
y=bigsamp;
a=equal(sample,1); // only draw the last one

you might need to play with the last line, not sure if sample ever actualy gets to one or not. also "usedots" needs to be on.

but it should just be that and the dx=-.001; in the per-vertex code of the preset to move it over.

[SteveW]

Great information - Thanks Jedi. Yes, this is what I'm trying to do, produce a realtime (live) visualization of the fundamental frequency line only, not the harmonics. This gives the audience a sense of the live music creation process, and can see a visual representation of what the musician is creating with their instrument. This may be somewhat interesting with a piano or guitar, but it really gets interesting with, say, a jazz saxaphonist that puts many improvisational slurs, glissandos and appregios into their performance.

Then add the accompaniment instrument (maybe MIDI piano), with a different color and 3D treatment into the same screen (isolated input to the other stereo channel, or maybe you handle realtime MIDI differently), and you have sort of 'musical canvase' and an entertaining visual accompaniment to the piece.

[Babylon-warp]

SEE pictures for my home-made spectrogram

this is an unusual post because I have actually programmed a very powerful sample accurate spectrogram that can do 50 to 2000 scan lines corresponding to each frequency and one dot for each sample, by using a sweeping filter with a special resonance -one such filter takes about 1% of my processor power so I can basically do a hundred lines at a time at audio rates.,
one of the most useful things I've found is that by multiplying one sinewave that comes out of the filter with a high resonance with a copy of itself at 90° you can do some maths on it that averages the peaks of all the frequencies rather than having phase accurate information which would give you peaks and troughs.

unfortunately the program I use only saves table files, so I would have it to find a way to get screenshots of all the segments of a sound and stick them together to publish a very impressive results, the most amazing thing was a Nightingale song that I got from free sound the website

In any case I'm good at audio and I could give some advice to anyone making spectrograms and trying to improve spectrogram some visualisation, as far as I know a sample accurate SpectroScope-oscilloscope takes about 1% processor power on a modern computer, that said, anyway, I will attach some pictures of the program I did

[Babylon-warp]

Winamp forums are messing up

[Babylon-warp]

phaseaccurate peaks and troughs across the frequencies

[Babylon-warp]

same stuff

[Babylon-warp]

a home-made fast Fourier spectrogram

[svshepherd]

Hi all,

I was looking into this myself - has anyone gotten a spectrogram working? I wasn't able to figure out, on a quick pass, how to activate a spectrogram as a custom waveform in milkdrop 1.04. I've wanted a nice log-f spectrogram for a long time. I'd also like to play around with using color to represent phase / stereo information or to highlight musical note value.

Does anyone have a script I could work from, or have a good pointer as to how to access the freq spectrum as a custom wave?

stephen

[Flexi]

crawled my milkdrop archives and found a folder including presets from this:
http://forums.winamp.com/showthread....hreadid=263823
thread.
There are some sort of cool analyzer presets - have a look.

[ryan]

http://forums.winamp.com/showthread....threadid=85171

Tada.

Edit: Actually, I don't think that's the preset I thought it was.

Oh well.

[yhscchpcbr]

messed around with spectro
i list goals at the end of presets for those interested
jpg is for a couple of the preset's textures

[yhscchpcbr]

1of2 attached

[yhscchpcbr]

2of2 attached

[yhscchpcbr]

here is an attempt at improving the evenness of the amplitude response from frequency to frequency. i kept playing with equations cuz i'm too dumb to predict rigorously what's going to happen. plus a mash with it (not a sentence)

[hexcollie]

so i was wandering aimlessly thru the internet and came across someone using that Spectrogram program to analyze an Aphex Twin song...this is what it produced


Impressive huh? not really too related to the subject, but it amazed me. check out the whole page at http://www.bastwood.com/aphex.php

[Saint Goody]

huh...

I'm gonna start putting that kinda thing into some synths...

Anyway, which of these presets seems to provide the most accurate spectrogram? I'm wanting to try out some Pixel shader effects with one.

[Saint Goody]

Okay, here's one that I edited to make the bass, mid, and treb values and their position on the spectrograph correspond with the red, green, and blue values of the wave.

It's not perfect, but it's a start.

[yhscchpcbr]

the thing i really wanted to do i found i couldn't because of the way milkdrop is built. that is, i wanted to be able to use the value1 / value2 vars outside the custom wave perpoint code using q or t vars, but course, they don't flow that direction. it would be nice if you could use, in perpixel or in shader code, those values to be able to do something based on specific frequencies or ranges.

the ultimate goal in my mind is to do some kind of voice separation or filtering to get events on the screen for each specific voice, ie, kick drum, vocalist 1, vocalist 2, symbol 3, guitar, random instrument #4, etc. of course, timbre is very complex and filtering probably has to be statistical if you want to do things real time (still with a big buffer) since all the voices are mixed and there is great information loss at an individual freq at a specific time, and that can only be regained by multiple plays of the same voice in different contexts (something we do with our ears and brain deceptively easily), and probably still with much interpolation since timbre varies depending on pitch being played on the instrument, the velocity at which it's played, etc.

milkdrop, of course, doesn't have enough for this as far as i can tell. you can't get at value1/2 outside perpoint code, and even there you can't draw anything more than a thick dot for each sample (and that's just too little on the screen to easily associate a visual event with an audio event), and you wouldn't have enough vars to do voice filtering anyway. maybe i'm wrong, and if so, that'd be nice.

i did at least like that i could see some individual voices in the presets i did mess with, and to boot, in stereo, which is really fun with stuff like venetian snares.

i'm checking out your preset, saint g. it looks good. i wish more people would try more stuff with this and maybe we can get a breakthrough.

like i said above with only being able to draw a dot per frequency, one is always left with too much information at too homogeneous of a level to really pick out anything but the most significant contrasts in the music visually. i wonder if pseudo-randomizing the position based on frequency and/or loudness combinations could create a more intuitive voice to visual association.

i was just thinking, maybe quantizing value1/2 vs sample, maybe by running them through some modulo arithmatic, and then using a warp shader like the disco ball , might more correlate a position / color / brightness / set of squares with a voice. you don't need to connect the value1/2 with the warp shader code because it's all just pixels on the canvas which can be sampled. the key would be to be strategic about where certain frequency combinations are placed. does it make sense? any thoughts anyone?

[hexcollie]

... >,< dextroamp. nasal spray...must resist compulsion to post nonsense presets...substituting nonsense nonsense

[Flexi]

@yhscchpcbr:

check out redi-jedis latest beta version, there are two very interesting concepts: first the sound(lofreq,hifreq) function which gives the summed volume in an interval. (a little bit tricky to find out: parameters are integers 0-1024)
the other thing is the w1-w8 variables which carry values from custom waves to the q1-q8 vars in the per-frame section.

Doll Doll Doll!

[yhscchpcbr]

i remember when reading thread from when md2.0 was first released and redi asking something about re-integrating his forked changes into md2.0, but it seems like geiss didn't care, or maybe it's just impractical for some reason. i'd attempt it myself if i had the md2.0 source. is it available? i had searched around that same time, but i couldn't find it. i hadn't really started learning the structure of milkdrop until 2.0, but those features sound very useful for what this thread might be trying to accomplish. but i don't know if i can live without shaders.

[Saint Goody]

Can winamp handle that yhscchpcbr?

I would have thought that by now someone would have recompiled milk1.04 for something other than winamp (something with lots of input possibilities?).

hmmm... standalone milkdrop1 with it's own media player... maybe with the possibility of playing acid loops... SHIT there's an idea...

I don't think that Geiss made the MD2.0 source public. It kind of looks like he just lost interest.

Take a lookat his webpage...

Not quite off topic, but getting there: Arkaos Grand VJ is supposedly being released soon. I'll wait for that for live shows.

goddamn... I'm babbling... need moar shroomery...

[yhscchpcbr]

alter goody

[fed0r]

I think that would be quite useful...
(Don't blame me for demolishing your eyes, tis looks ugly without pixelizing.)

[Saint Goody]

EYES???? My eyes became pixelated on... and off... and gibberish...

Anyway, I fixed the blurring along y, now I just need to figure out x.

It looks like y stretching needs to be at 1.0005 to stop the shrinking, but that leaves horizontal blur...

Alter yhscchpcbr

[fed0r]

WHERE IS MY FILE???????/

[fed0r]

enhanced prev. file:

[yhscchpcbr]

well, i tried with my might to figure out what's wrong with the x stretch, but couldn't, but i did find that the real problem with the y stretch wasn't at sy, but somewhere else, so now sy can remain at 1.000. I tried many values for sx, but it always stretches after half the screen. anyway, if anyone who knows milkdrop better could look, it would be nice to stop it blurring as it goes left.

jin tian kuai le sha ren jia!

[Drasla]

To add to the list of spectrogram presets...

this picks up quiet notes a little better than the others posted here, and doesn't having the blurring issues, but is otherwise pretty similar.

[Talwick]

After using various spectrogram programs and plugins, I have to say that none of these come close, no offence intended to you creators out there.

My reasoning for acquiring a spectrogram visualizer is to look for hidden images in music (such as by Aphex Twin, who has done this sort of thing before).

I went to the trouble of learning how to translate images into audio (using FL Studio's BeepMap), and have them seemingly print out an image using sound waves.

I tested each one of these milk visualizers on a recording that spells out "Hello" in big letters, and saw nothing from these presets but noise or a blank image. Even after many adjustments, I got some visual feedback, but nothing close to what a true Spectrogram would show.


Here's the highest input vs output quality I was able to achieve in my experiments. And it would have been clearer if I remembered to turn off Stereo Widening (the reason for those little lines through the image)

[Nitorami]

Waste of time. Milkdrop was never meant as a signal analyser, and what you try to do does not work for a few simple reasons:

1. Milkdrop does not analyse 100% of the audio information. It only provides an FFT chunk over a few milliseconds each frame. Roughly 70% of the audio is lost.
2. The phase information is not available. Another 50% information loss.
3. Probably due to the short FFT window used, the basses are missing.
4. At 30 fps, the time resolution is hardly sufficient in the first place.
5. This stuff certainly requires careful adjustment of the FFT parameters, the window type, length, overlap etc. None of this can be adjusted in milkdrop except by modifying the source code... which you can do, as it is open source. In this case, you should consider using the wavelet transform which would probably be the better choice than FFT for this sort of thing anyway.