KalNet Message Board

Ati, break out your lawyers cuz you're gonna need them.

I'm sure you all know about Ati and Nvidia "tweaking" their drivers (ie. cheating) to boost their scores in 3d Mark2003. Ati was inflating their scores by 2-3% while Nvidia was doing so by 25-30%. Rather significant gains in you're an Nvidiot. Ati apologized and removed the code. Nvidia pointed the finger at everyone but themselves. As it is, I'm not that OVERTLY concerned. Unlike 3d Mark2001 which was based off the Max Payne engine, 3dMK03 isn't based off a game.

But Splinter Cell is a game. In fact, it's based off the Unreal Engine. A rather popular engine, so I've heard. [H]ardOCP noticed large gaps in the ocean of their Splinter Cell demo. We're not talking optimizations here; we're talking huge gaps. To inflate benchies at the price of visual anomolies, that's not cool. This thread fleshes out the Splinter Cell ordeal.

http://hardforum.com/showthread.php?s=6 ... did=617197

And it doesn't stop there. This website shows bugs in the French Nvidia drivers:

http://216.239.37.104/translate_c?hl=en ... uage_tools

This is the one that really ticks me off. Some of you may watch the show the Screensavers on TechTV. What you may not know is that TechTV is in dire finicial straits. Guess who one of their huge sponsors is? Nvidia. Off an article from the Inquirer (ie. gossip mag with almost no credibility), they go to show that Ati cheated in 3dMK03. They go on and on about this and then bring up the Quake/Quack thing from a few years ago. For a split second they say that Nvidia cheated, but then get back to how Ati cheated. Then they show pics of the cheating. They show them with the understanding that they're Ati pics; they're pics from an Nvidia card. While informed people like you and me will know the difference, Joe Schmoe won't.
That, quite frankly, is slander and fraud. That's not cool in my books.

Ati themselves have responded to the TechTV thing here:

http://www.rage3d.com/board/showthread. ... genumber=1

They've been very nice about it. A lot nicer than I would have been.

I used to recommend Ati just for the performance (and I still do). Now, it seems I'll also be recommending based off good business ethics. Shape up, Nvidia.

This fellow [H]ard poster sums it up quite nicely:

"I believe that the semantics of all this cheating and optimizations ought to be clarified.

Let's first quote Tim Sweeney,

"...Therefore, any code optimization performed on a function that does not change the resulting value of the function for any argument, is uncontroversially considered a valid optimization. Therefore, techniques such as instruction selection, instruction scheduling, dead code elimination, and load/store reordering are all acceptable. These techniques change the performance profile of the function, without affecting its extensional meaning.

Optimization techniques which change your function into a function that extensionally differs from what you specified are generally not considered valid optimizations..."

http://www.beyond3d.com/#news6041

This is what ATI did:

"The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST."

So what did ATI do? They slightly reordered the shader instructions, and thus performed optimizations on the code, which for Tim Sweeney, does not constitute cheating. nVidia, however, did quite a bit of rewriting of shader code, inserting clip planes, and so on, so what they did constitutes cheating. Considering this, a 1.9% improvement in overall score is well within the limits of shader "optimizations", while a 24%, increase in the overall score is the result of clear "cheating"."

Well now. Things get interesting.

http://www.hardocp.com/article.html?art=NDgy

For those of you too lazy to click:


Introduction

The synthetic benchmarking landscape has been in complete turmoil for the past few weeks. Three players in the video card industry have either confessed to or have been accused of cheating in synthetic benchmarks. We are going to forego the entire "Optimization Vs. Cheat" discussion to focus on what is truly the matter here. At the heart of this fiasco is FutureMark. They may be a small company of under 30 people but they are heavy hitters when it comes to huge computer OEMs spending big money. FutureMark builds the 3DMark series of benchmark that started out as an enthusiast plaything that has morphed into what is considered by many to be the number one tool for evaluating 3D graphics hardware in the industry. One of FutureMark?s main sources of income is to charge companies for being members of their BETA Program. Some companies pay hundreds of thousands of dollars to be members of this "program". In exchange for that cash, they get to have input on the benchmarks that FutureMark develops.



Dell Steps In

Dell is one of the heavy hitters that uses FutureMark?s tools to make hardware evaluations and they are also a "BETA Member". Here is what Dell had to say recently about the latest benchmark from FutureMark, 3DMark03, quoted on the FutureMark site.

Dell uses many tools to evaluate system and graphics subsystem performance. We believe 3DMark03 is a solid synthetic graphics benchmark that covers a wide range of usage models and complements application-specific testing. Synthetic benchmarks like 3DMark03 help to differentiate graphics subsystem performance characteristics of both high end and lower end cards by utilizing sets of tests with varying degrees of graphics complexity. Entry level cards will be able to run at least one simple test to be used in comparisons for those interested in basic 3D functionality. Additionally, those interested in leading edge technology with be able to make graphics hardware comparisons with a range of tests using the new APIs, shaders, rendering techniques, etc. Dell believes 3DMark03 is a versatile tool that allows a fair comparison of today's wide range of 3D graphics solutions.

Now while I do not wish to focus on Dell, it does allow an illustration of the power of FutureMark and dollars at stake based on it. Do you think Dell would even comment on this issue if huge sums of money were not at stake? Do you think that a company that has based millions of dollars worth of purchases on FutureMark?s tools is going to say, "You know what, the tool we use for evaluating 3D performance is terribly flawed.? No, of course they are not.



BETA Really Means "MONEY"

If you review the ?Strategic BETA Members? list linked above you will notice two companies that are huge forces in the market place not listed. Quite simply, those companies not seen on the list have chosen not to pay hundreds of thousands of dollars to FutureMark. When one or more the market leaders in the industry are no longer paying FutureMark to participate in the 3DMark BETA Program, there is simply no possible way you can look at the tool as being a nonbiased utility for evaluation. If you pay, you get to see the benchmark beforehand and give input on how you think it should work. If you don?t pay FutureMark, you don?t get to the benchmark before it is released and you do not have the opportunity to direct how the testing works.



Our Thoughts

My thoughts on paid synthetic benchmarks are this; either everyone pays or no one plays. Of course it can also be easily argued that a benchmark company should not accept one dime from the companies it is evaluating. Ask yourself this. Are the companies that are paying FutureMark?s BETA Program fees getting anything in return, or are they paying so they can have their logo on that page? I think logic dictates that unless you are getting something for the huge sums of money you are spending, you do not spend it.

If you think that all of this hoopla in the last couple weeks is a technology discussion you are very much incorrect. It is all about the almighty dollar, although it has been shrouded in the context of something technically meaningful. This is all about money and whose pocket it is going to be in. FutureMark's past failed business models over the years have turned 3DMark03 into what it is today. In my opinion, 3DMark03 is a paid for tool. 3DMark03 scoring based on their game demos is not meaningful for depicting real world gaming performance since those specific tests do not represent real world game engines or technologies sold in retail. Anyone that asks you to rely on those scores simply does not see the big picture or is simply trying to cover their own interests, and that includes Dell.

Even based on FutureMark's own statements, none of the scores previous to their latest 3DMark03 Build 330 are to be relied on.


"Can 3DMark03 be used as a reliable benchmark for DirectX 9 generation graphics cards?

Yes, with the new 3DMark03 build 330, it can."

Logically, that statement would seem to suggest to me that all scores previous to using build 330 are not reliable. We of course think that with or without the latest build 3DMark03 is not trustworthy. Do we wait for FutureMark to declare in writing that their next build is really the one that will give you reliable results. I would have to suggest that anyone using 3DMark03 scoring tests is irresponsible in one form or another.

Don't let the fanboy rhetoric cloud your thoughts and judgment about this issue. Stay focused on the big picture. We need better benchmarks based on game and game engines that we will be using on our own desktops down the road. 3DMark03 should not be thought of to represent how your hardware is going to perform in actual games we will be playing later this year. Here is a HardOCP editorial from February that outlines our thoughts and points out some of the other dangers associated with a synthetic benchmark becoming too powerful. Here also is a PDF that was prepared by HardOCP for game developers that explains our wants and needs in benchmarks future benchmarks. Please feel free to share the document or use the information contained within as your own. The community working together is what is needed.

I ask that you approach this with an open mind and step back and see that this ugly situation that has been spread out over weeks stretches far beyond brand loyalty. We need better tools from the game developers community. With those tools we can ensure that we all have a better gaming experience, and that is after all, the focus of the matter. The old cliché, "There are lies, damn lies, and benchmarks" does not have to be true, but you can be assured if we sit around and let the dollars of huge companies decide our benchmarking fate, that will come to pass.

I humbly ask that anyone evaluating computer hardware to stop publishing the game demo scores taken from 3DMark03. The media is greatly responsible for giving FutureMark its power in the industry and it is my thought that if we work collectively, we can shift that power to a more responsible set of real world utilities that I suggest will become a reality this year.




----------------------------
I've always thought FutureMark03 was kinda pointless (3dMark01 was based off the Max Payne engine), BUT it is still indicative of DirectX 9 games. So until Doom 3/Half-Life 2/Deus Ex 2 come out, I think it should be used.

Futuremark retracting their statement reeks of Nvidia passing the green bills under the table, but that just might be paranoid me.

The video card soap opera continues

Ghost once said.."I dont go by synthetic benchmarks i go by the card that plays good fo me and the games i play."

If i expressed it wrong then please correct me Ghost.

Well he is right on the ball with that quote, to think that all this was goin on right under all our noses. Makes ya think about what other stuff they are tampering with.

I did? Hell, Im smarter than I thought!

Quake/Quack turns into 3dMark/3dMurk:

http://www.tech-report.com/etc/2003q2/3 ... dex.x?pg=1

I wonder what will happen when you change the names of your game exe's?


(On a side note, the Quake/Quack thing isn't as simple as I once thought. I actually read up on it and there were some interesting plot details that I wasn't aware of before. I'll explain if you're interested. If not, to type out a whole paragraph.........blah).

I'm intriuged. I read up on the Quak/Quake thing when it happened, but Im always ready to hear a new explanation.

"Rewriting shaders behind an application's back in a way that changes the output under non-controlled circumstances is absolutely, positively wrong and indefensible." - John Carmack 5/27/03



Quake/Quack thing tomorrow if I remember.

Quake/Quack Thinger

Facts:

-Nvidia releases GF2 GTS, GF2MX, GF2 Pro, and GF2 Ultra
-Ati releases original Radeon
-Radeon thought to have better image quality; GF2 faster due to "magic" driver release of Nvidia

-Nvidia releases GF3, GF3 Ti200, and GF3 Ti500
-Ati releases Radeon 8500
-initial benchies put 8500 between GF3 and Ti 500
-many sites think that the 8500 is rushed (no Smoothvision, etc.)

-[H]ardOCP notices visual degredation on 8500; produce script that replaces "Quake" with "Quack"; benchies drop for 8500, but texture quality restored:
http://www.hardocp.com/article.html?art=MTEx

-Firingsquad reproduces results:
http://firingsquad.gamers.com/hardware/ ... efault.asp

-some German site shows the visual differences, but the Tech Report's are clearer and in English so just skip to the next point

-Tech Report shows visual difference:
http://www.tech-report.com/etc/2001q4/r ... dex.x?pg=1

-Firingsquad does interview with Ati; they say that, to their knowledge, there is nothing wrong and it's nothing but an optimization:
http://firingsquad.gamers.com/hardware/ ... efault.asp

-Ati releases new drivers that restore visual quality and increase speed; [H]ardOCP reviews:
http://www.hardocp.com/article.html?art=MjU1LDM=

-the issue is solved and further driver upgrades from Ati place the 8500 competively against the Ti500, enable Smoothvision, and completely abolish any texture degradation.


Where Things Get Hazy:

-Nvidia's "magic" driver releases. Since their picture quality was inferior, many believe that they too had certain optimizations that degraded quality. The recent 3dMark03 fiasco seems to support this theory. Nvidia was just better at hiding it. Many argue that if no one noticed, it wasn't a problem.

-Ati claims to have studied Nvidia's optimizations and applied it to the original Radeon (NOT the 8500 yet). This too sped up the Radeon.

-Ati claims that code was carried over to the 8500. Due to architectural differences, it had the adverse effect of degrading texture quality noticeably. That's why they claim that they didn't know about it and rushed to fix it with the next driver release (which they did). They admitted their fault and fixed the problem. Or they knew they got caught and "made things right."


If we take the super paranoid, cynical view we arrive at this:

Ati rushed their card out, did a texture hack to make initial reviews look good, got caught, fixed the drivers, and produced an excellent card in the end.

This is polarized by Nvidia. They rush out their 5900, do various hacks to a benchmark they claim doesn't matter (3dMark03.......but if it doesn't matter, why waste resources tweaking it?), get caught, point the finger at everyone but themselves, use their money to silence those bringing forth the allegations, and leave consumers completely confused.

I used to like it when all I had to do was quickly read the graphs, see who was on top, and then buy that card. Now I need to read graphs, look at image quality (IQ), figure where the drivers will be in 6 months for the next coolest games, and find the cheapest one. It was soooooo much easier just to bash 3dfx and pimp Nvidia back in the day

It was a hell of alot easier when Graphics card makers could be trusted to produce drivers that you didnt need to worry about. The state of the graphics chip industry disgusts me nowadays.

"It's very easy for people to target synthetic benchmarks and optimize their drivers and hardware for them. That does not add anything to real world performance. You need to optimize your drivers to make game engines and API features go faster, not just specific game titles."--David Kirk, nVidia Corp. Nov 29, 2001

"We at NVIDIA don’t make it a practice to optimize our pipeline for specific benchmarks."--David Kirk, nVidia Corp. Nov 28, 2001

Yet, by summer 2003, the Detonators have a 73-element table of "recognized" applications + 50 "recognized" shaders.



All quotes and data taken from [H]ardforum.

In a rather interesting twist, Firingsquad has their MSI 5900 review up. They don't make it big deal about it, but they use their own demos for benchmarking. What's really thought provoking are the results. Benchmarks (ie. Quake 3) that the Nvidia cards usually win are won by the Radeons. Optimizing their drivers for certain demos? Or do Firingsquad's demos just more optimal for Ati hardware?

See for yourself:

http://firingsquad.gamers.com/hardware/ ... efault.asp


p.s. quite looking at Dusk and read the review...............pervert

For those too lazy:



Note that both scores should drop due to the more taxxing nature of the test. However, it's interesting to see the Nvidia card leading substantially on standard benchies and then lose on custom ones (ie. actual gameplay........you know, the ones that matter).

Seems like Nvidia has been doing some subtle optimization for Quake 3. I wonder what would happen if they performed the Quake/Quak test with a FX5900?

Guru3d has taken it to the next level. Every game benchmark exe is renamed (to do this for UT2k3, one needs to copy the open bitmap and rename it to the same thing as your exe), they do their own custom demos (unreleased to the public for an unset amount of time), and they run them a minimum of 3 times. If they all aren't within 2% of each, they're all scrapped and they start over. Rather impressive. But this is what we need now.