New more accurate benchmark technique

[Dan East]

I would like everyone to start using the following benchmark technique. It is far more accurate than the simple TIMEREFRESH we've been using. The downside is that this takes a couple minutes to run on the iPaq (and substantially longer on the slower devices). The resultant fps will reflect other optimizations, such as those involving particle effects, entities (the bad guys), etc.<br>Add the follwing line to the end of your autoexec.cfg file:<br><br>timedemo "demo1.dem"<br><br>Basically this just runs the first demo when you start Quake, while generating a cumulative fps. You can abort the timed demo by hitting Menu, so it behaves just like a normal demo in that respect. Only the first demo will play, and it will not loop through the demos infinately. You have to wait until the end of the demo to see the fps value. This should represent the average fps you would experience while actually playing Pocket Quake.<br>TIMEREFRESH would of course still be useful to get a quick idea of the success of an optimization, but again it will only show an increase in performance for some of the rendering categories.<br><br>Dan East

[CARPEDIEM]

I get 5.2 fps on my E-125 @ 200 with release 0.041 and 3.5 with release 0.03 (same device).<br>The latest release shows a 48% speed increase over the first one.<br>how many fps does an ipaq get?<br><br>Last modification: CARPEDIEM - 02/25/01 at 20:59:52

[Sparky009]

using the previous versions, I got 5.4 FPS (5.8 when oc'ed to 221MHz)... haven't tried the new one yet...

[killzat - Mark]

I'll bench both my Casio and iPAQ with MIPs v4/v41 and the latest ARM.. watch for results...

[killzat - Mark]

Ok first test done on default 206Mhz ARM with NO autoexec,cfg optimisations (as in download) - not overclocked.<br><br>6.7 FPS<br><br>Trying 236Mhz..

[killzat - Mark]

^^ That was for v004 ARM by the way, not the older one.<br><br>Well 236Mhz caused the iPAQ to crash and so instead I've set it to 221Mhz and used the optimised autoexec.cfg I'm working on - minimal quality loss/designed for Landscape mode:<br><br>8.1FPS (1.4 increase)<br><br>I also tested the same .EXE but with all the usual quality ON and speed = 221Mhz:<br><br>7.7FPS (1 increas over 6.7) - not bad.<br><br>Finally I tested @ 221Mhz with the basic autoexec:<br><br>6.9FPS<br><br>So in short just clocking to 221Mhz gets you a .2FPS increase, where as my .CFG that keeps full quality gets you a boost of .8FPS. Or even more if you go for the .CFG I've optimised for landscape mode - slight quality loss (textures, only slight).<br><br>Not bad eh =)? I'm going to do some more math and see if my new autoexec can get any faster without loosing quality.<br><br>MIPs will be done a little later as I've got to get into work now.

[Dr. Phat]

How do I open the config file?

[killzat - Mark]

Ok now for the MIPS Benchmarks using the LQ and High Quality Builds. Note my E125 is overclocked to 180Mhz and I was using my 'quality + speedup' autoexec.cfg file as well. So these results will be more than the basic .CFG + E125 Default Mhz etc:<br><br>Lower Image Quality 041 build = 4.8FPS<br>Standard 040 build = 4.5FPS<br><br>I have to say that I didn't see the famed 44% speed increase, that probably came from the old benchmarking technique. I don't thing the extra 0.3 FPS was worth the quality loss, even if it was only minor stuff.<br><br>The MIPS version isn't really playable yet because medium sized rooms kill the speed to about 1FPS, which is madness to work with. A bit more FPM and it should be iPAQ 001 release quality.

[killzat - Mark]

Sorry I forgot, this means the ARM full image quality version is roughly 75% faster than the MIPs. I have yet to true my .CFG designed for landscape, mainly because the Casio dies in Landscape mode and you can't see the menu properly anyway.

[killzat - Mark]

Oh and a word of advise, do not benchmark while in your cradle, re-charging or with program memory under 12-13mb.<br><br>The cradle and re-charging takes some CPU time and the memory is an issue for the large room on the benchmark. You may see a 'RAM' indictor, that means you need to free up more memory. If you don't then the FPS results will be lower for obvious reasons.<br><br>All of the above can effect your tests by between .3 and .8 FPS.

[Dan East]

Does anyone else see the RAM icon? Do you see it only with a specific version? (I'm assuming that only the MIPS people are seeing this). I don't think having extra free program memory would affect this - Quake allocates all the memory it needs up front. It's an all or nothing thing. The RAM icon indicates that the cache has been "thrashed", and will result in a slowdown. This may have resulted from an optimization.<br><br>Dan East

[killzat - mark]

I was wondering that myself, because side by side and RAM vs RAM, the iPAQ doesn't exibit the problem. Both PocketQuake MIPs builds do, doesn't seem to matter how much ram I free.<br><br>Diffecult to get an accurate MIPs benchmark because of it.

[Moose or Chuck]

What is this RAM icon you mention? When would I see this RAM icon? On the task bar or inside quake itself?

[norvain]

Using the latest mips build v0.041, default settings for everything<br><br>my E125@150mhz got 3.8fps

[Dr. Phat]

What doe the RAM icon look like? (I wouldn't know, I don't own quake)