Favor to ask

[Yathosho]

hi guys!

as i'm currently working on the new pimpbot 4, i'd like to ask you for your help. i've attached a small tool that will go through all your installed presets to detect which APEs you're using.

it would be extremely helpful if you could post your results in this thread. this tool will most likely take forever, depending on how many preset you have installed. so, in the worst case this will take a bit longer than forever.

once completed, all results will be displayed in its status window. you can right-click this window and copy the results to your clipboard - and paste them in your answer to this post.

thanks for your help!

[QOAL]

3 hours later...

AddBorder=199 (270)
BufferBlend=19 (21)
ChannelShift=721 (850)
ColorMap=2903 (4542)
ColorReduction=95 (103)
ConvolutionFilter=2260 (3817)
FramerateLimiter=33 (34)
GlobalVarMan=86 (90)
Multiplier=423 (503)
MultiFilter=115 (120)
NegativeStrobe=0 (0)
Normalise=7 (8)
Picture2=64 (80)
RGBFilter=0 (0)
ScreenReverse=0 (0)
Texer=175 (263)
Texer2=868 (3817)
TransAuto=65 (65)
Triangle=87 (700)
VfxAVIPlayer=0 (0)
VideoDelay=136 (170)
Completed

[Yathosho]

here are my statistics so far. the reason why i'm really interested in this shows in the last two columns: different authors use different apes are used. ultimately, i'm going to use the results to speed up the ape-detection in the upcoming pimpbot version. the more likely an ape is, the sooner it gets detected, the quicker the remaining detection runs.

unfortunately, you can also see how unreliable the detection is. sometimes it's enough to write the name of the ape into a comment and the detection will work (some authors solved this better than others.) texer is especially tricky, have a look at the results in the last two columns!

[Grandchild]

AddBorder=291 (407)
BufferBlend=49 (53)
ChannelShift=923 (1100)
ColorMap=4175 (6913)
ColorReduction=241 (263)
ConvolutionFilter=3331 (6301)
FramerateLimiter=116 (123)
GlobalVarMan=113 (119)
Multiplier=620 (744)
MultiFilter=177 (188)
NegativeStrobe=0 (0)
Normalise=111 (135)
Picture2=54 (66)
RGBFilter=0 (0)
ScreenReverse=0 (0)
Texer=144 (236)
Texer2=1252 (5088)
TransAuto=491 (498)
Triangle=332 (1309)
VfxAVIPlayer=2 (3)
VideoDelay=325 (388)

[The Earthquaker]

AddBorder=55(77)
BufferBlend=5(5)
ChannelShift=156(182)
ColorMap=657(1010)
ColorReduction=20(26)
ConvolutionFilter=525(1123)
FramerateLimiter=3(3)
GlobalVarMan=54(73)
Multiplier=103(110)
MultiFilter=41(44)
NegativeStrobe=0(0)
Normalise=11(14)
Picture2=35(51)
RGBFilter=0(0)
ScreenReverse=0(0)
Texer=44(87)
Texer2=319(1811)
TransAuto=39(39)
Triangle=51(379)
VfxAviPlayer=0(0)
VideoDelay=34(53)

[QOAL]

Yathosho, as this app is so slow I decided to code an app in C++ to do that same, but much quicker.

Example of current output:

code:
     - APE sniffer -



 Scanning...



 Finished!

 Scanned 6353 presets in 1 second(s)

 Found "Jheriko: Global\0" 92 times in 88 presets.

 Found "Misc: AVSTrans Automation\0" 67 times in 67 presets.

 Found "Render: Triangle\0" 702 times in 89 presets.

 Found "Acko.net: Texer II\0" 3819 times in 870 presets.

As I don't, from my scan, have all the APEs you're checking for, and for consistency, would it be possible for you let me in on the strings you're looking for?

Horrible hacked up source will come with the binary when it's ready.
(Doing all this probably isn't that necessary just for stats collection, but it is really slow ;P)

[Yathosho]

"Holden03: Convolution Filter"
"Color Map"
"Holden04: Video Delay"
"Jheriko: Global"
"Multiplier"
"Virtual Effect: Addborders"
"Misc: AVSTrans Automation"
"Render: Triangle"
"Channel Shift"
"Color Reduction"
"Misc: Buffer blend"
"VFX FRAMERATE LIMITER"
"Trans: Normalise"
"Picture II"
"Texer"
"Acko.net: Texer II"
"Jheriko : MULTIFILTER"
"VFX AVI PLAYER"
"Negative Strobe"
"Screen Reverse"
"RGB filter"

as you can see, many of them are easy to fool (especially ucd's). i was wondering about the slow speed before, compared to how much quicker search-in-files in notepad++ is. the old version of pimpbot used a script, that was equally slow but more unreliable. for the new version, animaether wrote me a new function. i consider him a nsis pro, but i still wonder about the terribe detection rate.

i guess the best solution would be a custom nsis plugin. if you can think you can turn your app into one, i'll gladly use it for pimpbot (i'd prefer a plugin without dependancies such as net framework etc.)

[Yathosho]

as i said before, texer(1) is especially unreliable to detect, as all texer2 will also be found when searching for texer. i tried (integer) subtraction, but as you can see in my table of results, in one case this produced a result of zero (could also mean a negative value!)

that was likely caused by the mentioning of the the word "Texer" or by a texer-bitmap including the name.

sucky, sucky!

[QOAL]

Thanks for the list!

I wouldn't of thought NSIS would be close to as fast as a native app anyway, I'm sure the code you use is very well written though.

Right now I have zero idea on how to make a nsis plugin, but I'll have a look. (if I never mention it again then things didn't go well - I am just winging this)

By adding \0 (null) to the end of my search strings I made the check far more accurate (probably could match something at the very end of a comment/code block though.)

Anyway, I've attached a zip with APE Sniffer and the code for it.
Place it in a folder and it will recursively scan for AVS files and then spit out APE_Sniffer.txt with the results.

[Warrior of the Light]

AddBorder=434 (612)
BufferBlend=32 (37)
ChannelShift=1241 (1466)
ColorMap=4925 (7583)
ColorReduction=253 (273)
ConvolutionFilter=3661 (6650)
FramerateLimiter=91 (95)
GlobalVarMan=169 (182)
Multiplier=710 (827)
MultiFilter=302 (310)
NegativeStrobe=0 (0)
Normalise=55 (68)
Picture2=127 (183)
RGBFilter=0 (0)
ScreenReverse=2 (7)
Texer=257 (457)
Texer2=1711 (7379)
TransAuto=162 (162)
Triangle=209 (1126)
VfxAVIPlayer=2 (5)
VideoDelay=373 (475)
Completed

[QOAL]

Okay, I have something working.

Basically call 'apesniffer::scan "DIR"' it'll return a string of 1 and 0, 1 means that APE is used. The string is in the same order as your checkers output (and the same as mine).
I thought that would be the easiest and most compact way of telling you what are used.
It should also be faster than my checker too, as it only needs to find an APE is needed once to stop checking for that APE.

Simple test.nsi included (just remembered it has relative path in there for the plugin location, you'll need to fix that, I would but I'm going out now, sorry) along with the source (which is probably quite fugly).
I'm annoyed at the file size, but I guess that is what I get for using the STD library (I think?).
Anyway, hope it helps you out.

[Yathosho]

the speed is amazing, will implement this asap.

think you can support subfolders as well? while it's no problem to solve this within nsis, it feels more right if the plugin did this (also to avoid double-scans.)

licensing might also be an issue. i haven't decided yet, but i think i will stick with a bsd-like license (basically that's "do what you want") for pimpbot 4. it's not important for me, but it might be for google code.

[QOAL]

Yay. I was a bit worried over what to return, but no complaints(?) so that's cool.
(It would be nice to get it's size down, 200k for that is a bit silly)

It does scan subfolders, doesn't it? It's the same code as the standalone sniffer I made, that scanned folders recursively.

Hmm licensing, well I assumed that any of the code snippets I used are in the public domain and I'll agree to whatever license is best for you.
Yes, google code doesn't like you mixing licenses in single repos.

[Yathosho]

Originally Posted by QOAL

It does scan subfolders, doesn't it? It's the same code as the standalone sniffer I made, that scanned folders recursively.

didn't seem to work yesterday, now it does. my fault.

Originally Posted by QOAL

Hmm licensing, well I assumed that any of the code snippets I used are in the public domain and I'll agree to whatever license is best for you.
Yes, google code doesn't like you mixing licenses in single repos.

forget it, actually i can use any nsis plugin without supplying its source. didn't think of that.

[Yathosho]

just for the record: it didn't scan all my presets before as the plugin-dir detection isn't flawless. the plugin itself works, i'm currently implementing it to pb4.

[QOAL]

Right, I couldn't stand the fact the dll was so big so I made it work without using STD and I went back to C. (Source should just be a drop in replacement for the example plugin)

DLL size has gone from 283KB to 8.5KB.

I don't think anything else has changed.

[Yathosho]

your previous test script gave me this:

code:
101111001100000110000Ûôv0.T

Completed

and it doesn't work in pimpbot anymore (gives me 000000000000000000000Ûôv0.T as result)

[QOAL]

Hmm okay, I think I know what would cause that and I've added a fix in the attached version.
Checks if the path given ends with a / or \, and appends one if not.
I couldn't find any other way to make it spit out all zeros.

I hope that fixes the issue. :S

[Yathosho]

working again!

[Yathosho]

i guess it's more difficult to build something like this for texer images, videos or gvm files?

[QOAL]

You want a program that goes through all the presets and checks for .bmp, .avi and .gvm files?
Or you'd like it to list the required file names? (I guess it would need to ignore the default texer images)

I would need to see if I can alter the binary search code, so it can find the extension and then find the first null byte before it and use that as the file name -- which is easier said than done for me, probably.

I'm not sure how to return a list like that to NSIS, which I assume is the end goal here.
Doing it as a comma separated string could easily be way too long.
Can a plugin just keep pushing strings to the stack?

[Yathosho]

Originally Posted by QOAL

You want a program that goes through all the presets and checks for .bmp, .avi and .gvm files?

right now i guess the most annoying part is creating installers for presets and you don't know which files they require. so scanning them and copying the bitmaps etc. automatically would really be a timesaver.

I would need to see if I can alter the binary search code, so it can find the extension and then find the first null byte before it and use that as the file name -- which is easier said than done for me, probably.

i guess it might be difficult, but maybe it helps matching that up with the files existing in the avs directory.

I'm not sure how to return a list like that to NSIS, which I assume is the end goal here.
Doing it as a comma separated string could easily be way too long.
Can a plugin just keep pushing strings to the stack?

i'm not sure which is the best way, but unless it's too slow i'd be okay with a solution per preset using a loop within nsis. i'd then have to decide whether i filter out duplicate files before copying them or if i don't care as those files are small anyway.

if you want to discuss this on chat let me know, i'm on about ever instant messaging service there is. just dm me your contact.

[QOAL]

Okay so...

I have made progress / might of hit a hole in one with this.

Usage is like so:
avsresources::listresources "$1\avs\QOAL\" "$1\avs\"
${Do}
Pop $R0
DetailPrint $R0
${LoopUntil} ${Errors}

First parameter: Dir to search for AVS in
Second parameter: Resource dir

Each string you pop contains the name of a filename, verified to exist, relative to the resource dir passed.

Apparently as a plugin, you can just keep on pushing strings, which you can then just pop off in a while loop. (Which does work with 1000's of strings, hooray)
Of course, I can also output the file names to a text file too. (Disabled right now)

Couple of things that I noticed while doing this:
One APE prepends(?) the bitmap file name with ð?, rather than NULL (Removed internally)
Lots of possibly old references to files are left in presets, leading to the plugin finding lots of names that have the start cut off. (All files are checked to see if they exist, and are otherwise ignored)

The attached zip contains avsresources.dll and a test.nsi (change the plugin path in it!)
The plugin is large this time as I used some STL stuff (Vector stuff), to hold the list of files, so it can be checked for dupes during the scan.

Hopefully this post makes (some) sense!

Edit:
Just realised that having it check the passed dir for the resource files is a bit silly, and probably only works correctly on a main AVS dir. I'll have to add a second parameter?
Also: The file extensions it looks for are: .gvm, .GVM, .bmp, .BMP, .avi and .AVI
Edit 2:
Updated the plugin with a second parameter for the resource dir and updated the post.

[QOAL]

Dum de dum...

So... why have two separate plugins when you can have just the one?!

Function names have changed, and a new one has been added for a laugh, please see attached.
I also made it so it checks for presets ending in .AVS along with the previous .avs.

(Contains complied DLL, source and an example nsi file)

[Yathosho]

doesnt work for me, all it does is creating an empty textfile. my next question wouldve been whether this scans recursively? if not, could you add another parameter for that?

edit: my post was regarding the first version, have to test the new one

edit2: avstools is now listing resources and already works recursively - nice!

[QOAL]

Yeah, I only noticed this morning that what I posted yesterday had the code to check parameters for trailing slashes (and append one if not) removed by accident - and it had spitting out to a file still enabled. Whoops!

But what a difference a day makes.

The apesniffer in AVS tools should, hopefully, return a string that is as long as the number of APEs (before it was 32 chars, which probably contained memory from other stuff at the end)

And yes, everything is recursive.

[Yathosho]

uploaded a first beta using your plugin

[Yathosho]

Originally Posted by QOAL

Also: The file extensions it looks for are: .gvm, .GVM, .bmp, .BMP, .avi and .AVI

maybe you could add JPG and SVP (yes, i know). i haven't really run into presets using JPEGs, but since the option is available in both APEs and PimpBot, it would be a good addition. i think i'm only support the JPG extension, not JPE or JPEG.

[QOAL]

Here we go:

[Yathosho]

i've tested avstools on a couple of scenarios and it's not always working. to reproduce my problem, download this file and put the resource files in your avs directory.

avs 1
detects only the bitmaps in the texer, texer 2, picture and picture 2 preset. i've also added two presets using videos and two more using the svp/uvs loader (i didn't know until know the svp plugin also handles uvs files, never heard of those).

avs 2
contains a preset using a picture, but although present in my avs directory, the bitmap never gets detected (the picture effect worked in my first example).

[QOAL]

Hmm, I'm really not quite sure what problems you're getting.
I've gone through both of the avs folders presets and checked the result compared to what is listed in the files (I haven't actually looked at them with AVS), and what's in the Res folder, and no matter where the folders are (different disks, deep or shallow) it still works correctly.

I don't want to doubt you, but are you sure you're sending the correct resource dir?

my brain is shot for the night now, I should probably give you my msn so it can be nailed down, which I will send now.

[Yathosho]

would it be possible to get a unicode build of this plugin as well? sooner or later, i will switch to the unicode version of nsis so i thought i better ask now that you still remember how the plugin works

[QOAL]

I had a attempt at this, but I just don't know what I'm doing when it comes to ansi to unicode conversion.
I'm really not that good at c/++, I just wing it.
I know my way around the plugin code still and I set up it up to use the unicode stuff and did some changes to TCHAR and std::basic_string<TCHAR> but...

"Number of AVS presets: 0���Ѩ 저鰢樀鑞谰럼ℂꀀ山럼젂鰢쑄럿礂띊���h
APEs used: 〰〰〰〰〰〰〰〰〰〰䐰ￄʷ䩹皷⟼棜
Resource files required:
〰〰〰〰〰〰〰〰〰〰䐰ￄʷ䩹皷⟼棜
Completed"




So yeah, I'll just attach the source for the ansi version for now. :S

[Yathosho]

there are currently 5 nsis plugins not available for unicode nsis, at least two of them can not be integrated using the CallAnsi plugin. i guess a unicode build of pimpbot could take a looong time to be realized. it's said as i have some unicode language files that i can't convert to ansi without losing information (e.g. cyrillics).

[Yathosho]

out of interest, what are you using to compile your plugin?

[QOAL]

I'm using Code::Blocks with GCC (I'm used to that set up due to previous open source work).
I didn't bother including any project files in the zip because I assumed very few people would use that set up.
(I'm also still not interested in spending hours flapping about in a naive attempt to produce a unicode version)

[Yathosho]

well, for a second i thought i might take a look at it and make it work for unicode. i doubt i would even get as far as compiling, as don't know any c++

[Yathosho]

okay, it looks like i found a workaround. pimpbot itself will still be built using the ansi version of nsis, but it will create unicode installers. that way i don't rely on avstools in unicode.

will post a working version in the next couple of days.