#MIPS & other benchmarks
38 messages in this thread
and MFLOPS can be used for a "relative" performance measurement, to get a
true "system performance measurement" requires additional measures.
Real World:southern Illinois, USA Internet:ca0008 at siucvmb.siu.edu
There's more to it than 'helper chips'.
How can you believe that MIPS can come even close when comparing different
CPU architectures?
-larry
Larry,
If you look at MIPS ratings as simplistic "bean counters" rather than
"wholistic measures", then it is possible to relate one MIP rating to
another, in a simple quantitative analysis. This is very similar to
"milage ratings" in automobiles.
So what if car A gets 45MPG and car B gets 52MPG, you would say if you
looked at these "numbers" the same way that you are looking at "MIPS".
What size engine does each car have, how big of a gas tank do they have,
what is the fuel-air ratio mix, in what conditions were the cars tested,
et cetera, et cetera, et cetera.
Certainly, MIPS ratings involve many factors, as do MPG ratings. But are
you then going to say that MPG ratings are worthless? Don't MPG ratings
cover a wide variety of vehicles, some of which you would never think to
ask for such a rating (eg; 'heh doc, what's the MPG rating of your '87
tractor there your using in your bean field?').
MIPS ratings may not be any more accurate than MPG ratings, or vice versa,
but they both ARE quantitative measures, albeit perhaps not the most
desirable measure, and can be used in simple comparisons. Don't you look
at the MPG ratings when you look at cars?
If MIPS and MFLOPS ratings aren't enough to satisfy the request for a more
thorough systems performance analysis, then we must ask for other measures
to be included AND explained. If these other measures could be designed
well enough to give a "true, total system performance" regardless of
architecture, then lets demand that they be included in all
advertisements, et cetera.
Unfortunately, I don't think that there IS one, universal test, hence,
MIPS and MFLOPS DO show and mean something, and on some uniprocessor
systems they probably DO show 'true system performance', or close to it.
Instead of arguing whether "MPG" ratings are "still relavent", we should
be asking vendors to come up with more definitive test results with more
explicit definitions of those tests, and through in MIPS and MFLOPS just
<continued>
Actually, The analogy is more like using spoonfuls (unknown measure of
liquid quantity) per reconing (unknown measuring of distance) for car
comparisons. Personally I would like the analogy of calorie counting that
occurs on various products (2 calories per serving) where the serving size
is undefined and totally unrelated to anything meaningful. People like to
have raw numbers that they can just compare, but it doesn't make those
numbers very meaningful. MIPS is one of the bext examples of such and many
of the manufacturers of CPUs like to play those games. Have you ever
noticed that the RS-6000 has a higher MIPS rating than the 3081?
Personally, I think that the 3081 blows the RS-6000 out of the water as
soon as you put 50 users on the box. The key here is all of the support
that the processor brings with it.
The closest thing to a universal test has been the SPEC benchmarks. A lot
of effort has been put into making them large meaningful tests of typical
application work. However, they do take a bit of resources to get running
and certainly won't allow you to compare to a PC with 64K of ram (assuming
that you can find one). The ONLY time that MIPS and MFLOPS are meaningful
are for the SAME instruction set with the SAME hardware. Comparing a 8086
to a 80386 would give a good realtive comparison of improvement, but
certainly doesn't tell you diddly squat compared to a 68K.
Ah – your hard data rating comparison of the RS/6000 and a 3081 IS a good
point. MIPS IS a very limited measurement, and if one ONLY used MIPS to
make a purchase decision, then said persons shouldn't bother asking why
there isn't a CD player included, and why the radio is monaural instead of
the expected quadraphonic! 🙂
Perhaps the current SPEC benchmarks are too big to run on a lowly
8-bitter, but the testing parameters, if they are well enough defined,
could be rewritten to produce similar results on ANY machine. Obviously,
some kinds of measurements may not be possible across all machines, in
which case, said measurements are measures of a CLASS of machines. There
should STILL BE, however, sufficient quantitative measures to get a
similar kind of RELATIVE system performance. After all, they have
applications, integer instructions, software/firmware emulated
trigonometric functions, et cetera, et cetera, et cetera. Why CAN'T I
measure the system performance of an 8-bitter system?
It is not trivial to define the necessary mathematical model that can be
applied universally, and it is probably impossible. But it SHOULD be
possible to define a sufficiently acceptable subset that COULD be applied
to any machine. Perhaps SPEC benchmarks can achieve this goal. Should "we"
send the vendors a petition to include SPEC benchmarks in all of their
system performance ratings?
I don't see that a MIPS ratings comparison between an 80xxx and a 68xxx is
invalid. That's like saying that the MPG ratings for a Ford is
incomparable to an MPG rating of a Suzuki (motorcycle, that is), or even
an 18-wheeler. It's the OTHER factors (mininum/maximum instruction cycle
time, number of concurrent users, et cetera) that really make the
difference. If I have a 21 MIP system and you have an 18 MIP system, then
it IS possible to say that "given all other things being (considered)
equal", that "my" machine will be faster than "your" machine, at LEAST in
<continued>
Jim,
Apparently you have NO conception of how worthless the MIPS rating
really is. The BIG problem here is what the word "instruction" means.
Your analogy to MPG is TOTALLY fallacious. At least with MPG the meanings
of 'miles' and 'gallon' are well defined.
Vic:
Yes, in order to equate MIPS with MPG, you'd need to let the
different car manufacturers make their own definitions of what a mile is.
(and not tell anyone. <grin>)
Clint <VRLI>
SOME things, not necessarily ALL things.
Boy, did I open a can of worms or what? Anyone catch any fish yet??
Real World:southern Illinois,USA Internet:ca0008 at siucvmb.siu.edu
Nope… because your 21 MIPS machine was specified at 21 MIPS based on the
number of single byte adds if could do in a second, and mine was specified
at 18 MIPS based on using all instructions, most of which take a lot longer
than a single byte ADD. In actual fact, my machine is twice as fast as
yours, because it can do all the common instructions in less clock cycles,
and it has some fast instructions that do a lot more in reasonably fast
time, that yours can only do by writing a routine that uses many more
instructions.
Here's some food for thought.
A friend of mine used to do a lot of work with the Commodore PET. I dropped
into his officce/shop one day, and he was showing me a new BASIC that he
was writing. It was a business oriented BASIC, and had some neat features.
I asked him how fast it was, and he pointed to the keyboard and told me to
try something. Well, I just happened to know how long a 'FOR I=1 TO 1000;
NEXT' took to execute on a standard PET, with its built-in Microsoft BASIC.
I also knew how long it took on a TRS-80, Atari, and Apple. So, I typed it
in. The 'READY' prompt returned virtually immediately. I was VERY impresed!
He could not keep a straight face. _EVERYONE_ who ever wanted to find out
how fast a BASIC was, would type that line, so all he did was to detect a
FOR loop with nothing in it, set the variable to its terminating value, and
carry on to the next instruction. Was it a measure of speed? Of course not.
Same with MIPS.
-larry
We sure have wrung the towel pretty well on MIPS. I had heard of a story
of application programmers in some governmental agency, some years ago,
who would specifically include "dummy loops" to add delays to a programs'
execution just so they could then reduce or remove those loops later and
make themselves look good to "management" (think what THAT would do to
system performance measurements!), but I think you Basic wiz beats them
hands down 🙂
Real World:southern Illinois,USA Internet:ca0008 at siucvmb.siu.edu
Perhaps what we can do is use something like: MIPS(1.2), MIPS(1.8), or
MIPS(2.4) to represent that average instruction count per clock cycle.
This would lead to more relevance between "MIP ratings" then.
Or, we could call them "mumps" on one system, "measles" on another system,
and "chickenpox" on all others 🙂
Real World:southern Illinois,USA Internet:ca0008 at siucvmb.siu.edu
A MIPS(1.2) rating would be fine, if the instructions rated this way were
all included in the tests performed, and if the instructions were similar
enough in all manufacturer's products, and the 'ifs' go on….
Calling them by different names when performed for each chip would reflect
accurately on the value of the results, if not the performance of the
products involved.
-larry
for comparison (is your penny worth more than my penny?).
Real World:southern Illinois,USA Internet:ca0008 at siucvmb.siu.edu
Jim,
Your analogy to MPG ratings is way off base. Citing MIPS as a performance
figure, assuming that by MIPS you mean the number of _native_ instructions
per second, is not at all like citing MPG rates for vehicles, simply
because a mile is a mile, and a gallon is a gallon. In the case of a CPU
chip, and instruction is only an instruction for that specific CPU chip,
and it is handled in different ways in different architectures.
Citing MIPS as a raw figure, is the logical equivalent to saying "My car
gets 60 distance units per fuel amount unit", wihtout specifying what the
units are.
RISC chips don't have complex instructions, and in general, execute one
instruction per clock cycle. To compare that with a CISC chip, with both
simple and complex insrtructions, some of which _may_ execute in a single
clock cycle, some of which may take many tens of cycles, is an excercise in
futility, giving results that are not even close to being meaningful.
Similarly, the instruction sets of two different architectures of CISC
chips are likely to be different, in both capabilty and in the number of
cycles taken per instruction. Similar, meaningless comparisons are obtained
by quoting MIPS. This does not take into account that MIPS is sometimes
quoted only on the fastest instructions, and sometimes on the whole set, an
sometimes on what the manufacturer deems to be a 'representative subset'.
MFLOPS is a whole lot closer to being meaningful, if for no other reason
than that it is limited in the scope of operations performed, and that
those operations are more complex, requiring many more clock cycles in both
CISC and RISC chips. But of course, it is only a measurement of a limited
part of a CPU chip's repertoire.
-larry
Larry,
Sorry, but my analogy between MIPS and MPG is not, way off base. MPG
ratings are based upon a known, fixed volume of fuel. BUT, what about tire
termperature, road surface conditions, wind factors, weight of vehicle,
weight of driver, size of engine, amount of acceleration along the test
track, how much fuel is being fed into the engine and how much is being
wasted (because of undo acceleration), engine timing, and on, and on, and
on.
THE POINT is that MIPS IS based upon at least ONE known factor (the cycle
speed of the CPU), and MPGS IS based upon at least ONE known factor (the
amount of fuel delivered to the engine).
It is true that most CISC processors are in the 1 1/2 to 2 cycles for
most of their instructions, but even if an all integer number test was
chosen, YOU CAN'T GET ANY MORE INSTRUCTIONS PROCESSED THAN CAN BE DONE
WITH INTEGERS ALONE.
I AGREE WITH YOU, that MIPS by itSELF is not an adequate measuring
"stick" for comparing 'system performance', but it STILL is ONE
QUANTITATIVE MEASURE, just like MPG.
By the way, haven't you ever purchased a car that a one of those EPA
stickers on it that said "this car, through testing, has shown that IT
will attain ….. MPG (on average, and depending upon how you drive, and
.., and ..)". I would venture to say that there are very few cars that
live up to their "performance rating", YET, don't you STILL use MPG as a
QUALIFIER? I have yet to purchase any vehicle soley, or even primarily,
because of this one "measurement tool" called MPG, but I DO keep it in
mind, because it gives me a RELATIVE value that I can use to compare other
vehicles.
You say that "a mile is a mile and a gallon is a gallon", but is that
true on every test? What is the allowable percentage of error?
The WHOLE POINT that I am offering is that MIPS can be used as a RELATIVE
guide, NOT an ABSOLUTE MEASURE of computer performance, and still be
meaningful.
I also AGREE WITH YOU that it would be good if there were OTHER
<continued>
Jim,
between processors of different architecture MIPS is a meaningless
comparison. You apparently simply canNOT understand that 'intruction' is
SO variable that the MIPS number means nothing. I'll grant you that there
is ONE known in the MIPS rating… seconds; which are pretty constant
(unless you subscribe to the theory that time is slowing down). By
contrast MPG has BOTH elements defined 'miles' AND 'gallons'. It's really
a Humpty Dumpty problem "Instuction means _exactly_ what I want it to
mean, nothing more, nothing less."
The point that you are missing is that the only thing that is the same for
both machines is the time that you are measuring. Perhaps if you were to
measure the EXACT SAME instruction on both machines you get a relative
comparison, but even the NOP instruction is different on the 68000 and 8086
(length of instruction relative to other operations). The best description
that I have heard of with using MIPS as a comparison figure is that of
measuring the speed of a car that has just run off a cliff. You might be
able to measure how fast the car is going (falling), but it ceratinly isn't
anything that you would use for comparisons.
quantitative measures proferred along with any MIPS ratings, so as to be
better able to compare these additional RELATIVE measures against other
RELATIVE measures (don't forget that some of the system performance one is
trying to "count" is also used "in the counting").
Since the basic measure of Millions of Instructions Per Second clearly
identifies the units involved, this is NOT like relating MPG to saying "my
car gets 60 unspecified distance units per some unspecified fuel units".
The units are clearly identified.
AGAIN, MIPS is NOT the best measure of system performance. I DO NOT ARGUE
THAT. In fact, if anything, it tends to HIDE real system performance (and
hearing this, the CPU choir offered up a solemn, AMEN).
Let's not quibble anymore about whether MIPS is a worthy number or not.
Let us be more concerned with obtaining, asking for, or generating the
additional quantitative measures that can LEAD to a more realistic
statement of "system performance". BTW – I downloaded AIBB and looked at
its output, and found the numbers, and comparisons, interesting, but of no
practical value to me. This is where a little mathematical statement of
what is being measured would be beneficial to have, since I have no way of
knowing just "what" is being measured.
In some ways, I am kind of surprised that there aren't more mathematical
theses in the area of computer system performance measurement. I would
think that this would be right up their "alley"?
OK – let's agree that you do not like using MIPS at all, but I do, with
the realization of its shortcomings, and move on to more fun things, like,
calculating Eigenvalues, or "have you gotten to level 100 in Lemmings,
yet?".
Amities 🙂
Real World:southern Illinois,USA Internet:ca0008 at siucvmb.siu.edu
Well, I do agree that in the MIPS measurement, there is one known, and
quantifiable factor; that being the unit if time, the SECOND. Instructions,
as I have pointed out, are not quantifiable without considerable further
qualification. That qualification must include architecture, which
insctructions are being used (a subset or the entire set), how many clock
cycles are taken for each one, and how much 'work' is done by each one. The
clock cycle speed and the number of instructions done per cycle also needs
further qualification. It is quite possible for a CPU to perform more than
one instruction per clock cycle, as evidenced by the latest RISC chips from
IBM. Note, however, that the IBM chip does not have a fixed, immutable MIPS
rate, since it depends entirely on the exact sequence and mix of
instructions. IBM of course, chooses its tests carefully. The customer has
no such option.
In short, MIPS is no more a valid comparison than raw clock speed. One
might as well measure the cabinet and quote a performance figure per cubic
inch.
The ONLY value of MIPS comes in rating chips whose architectures are VERY
similar, which comes in handy for rating faster chips from the same
manufacturer, though there are sometimes enough dissimilarities to throw
those comparisons off.
I'm probably a little too vehement in my arguments on this, but it's one of
my 'hot buttons'. I have gone through this argument many times, with Intel
bigots who just could not see that raw clock speed does not reflect
performance in widely differing chips, and with folks who thought that seek
times were the only criteria for disk throughput comparisons.
So… on to eigenvalues (don't even know what they are), and Lemmings
(never played it).
🙂
-larry
Larry,
Nope! you're not being vehement enough. I've been trying to eliminate
the worthless MIPS ratings every chance I get.
Somthing else here… that MPG is on the same type of surface, under the
same type of stress (getting from here to there at a particular speed with
the same driver (you)).
A 68040 under AmigaDOS has none of the horriffic restrictions on it that
an 80486 has when slogging through Windows 3.1, for instance. So, although
a 68040 is actually a considerably more powerful processor (factor of two
or so) than a 486 to start with, when running an Amiga, a 68040
*effectively* outpowers a 486 under windows by *huge* margins far
exceeding what you would expect from "MIPS" ratings.
At least cars, and MPG, compete for those ratings on an even field –
they're all gonna get driven nearly the same way by a particular buyer, so
MPG relationships of one car to one another have a chance of holding the
same pattern. Not so 486's/040's/and RISC CPU's over different operating
systems. The comparisons are so difficult as to be irrelevant.
In my opinion, of course. :^)
–Ben
…via AutoPilot
Ben, I'm confused. I own a 486/33, a 386/20, an A2000 w/040 a Mac SE30,
(and tho' I hate to admit it, an Atari ST). Each attracts different kinds
of software apps, but from a SUBJECTIVE point of view, the 486 eats the
040/28 alive! Perhaps its simply the Amiga's slow screen updates that leads
me to that conclusion. Whatever the reason, the PC is always the box that
comes to hand UNLESS a specific application demands otherwise. 3D
Renderings appear to be faster. Word Processors appear to be snappier.
Spreadsheets recalc quicker etc.. etc…
It could simply be the applications. But I find it difficult to believe
that PC apps are consistently 'better' and more 'efficiently' written than
their Amiga and Macintosh counterparts.
As gor the MIPS debate, as a humble end-user I USE IT only to measure
relative speeds between different processor configurations on my AMIGA. NOT
for cross processor comparisons. In the absence of easily available and
believable cross platform tools, subjective opinions win the day.
For me, the 486 wins the laurels.
That's very interesting. I've worked on a 486/33 DX system with an ATI
Ultra card, and I thought Windows was dog slow compared to the Amiga.
Were you talking about Windows applications?
Steve
Come on Steve. If you are working with a 486/33 DX with an ATI Ultra card,
the video is so much faster its not funny. I have written scrolling
routines for both the Amiga and the Windows 3.X environments. I have both
machines sitting right here and if I scroll the same amount of text in the
same size windows with the Amiga having only 4 bit planes and the PC having
8, the Amiga looks like a horrible dog. By the way, the text is colored so
the speed improvements incorporated into the console device under 3.0 would
be rendered useless had I chose to use it. The central code is very
similar so code efficiency can be ruled out. One other tid bit, my
scrolling routines on the PC with work with any video card with a driver
for Windows. Don't try that on the Amiga!
Tim
And when you run a 32 color animation on the Amiga, and a color animation
under Windows, what does it look like? 🙂 Blits are still terrible under
Windows, and I've seen many souped-up Windows machines (50 Mhz, graphics
accell'd.) having trouble keeping up with AVI or Quicktime animations. I
suspect one's eyes are more forgiving when it comes to scrolling text.
I just opened a few windows on the PC's desktop in windows and moved
things around. Did the same thing on the Amiga. The Amiga won. The
redraw on the PC was pretty good, but I could see it redrawing each
section when moving windows over other windows. Those operations on the
Amiga were quicker. The PC lost this round.
This brings up a good point though….It takes a really high powered card
to bring the PC close to the Amiga. I'm sure that in a while with things
like local bus the PC will be faster….but what platform isn't faster if
you keep throwing money and hardware at it? The Amiga stacks up pretty
darn well, I'd say.
Steve
You have brought up a good point Steve. "…but what platform isn't faster
if you keep throwing money and hardware at it?". I think the answer to
this question is the Amiga. The Windows environment on the PC supports
whatever high powered video card you throw at it providing it has a driver.
Since most of the new high powered video cards were designed with Windows
in mind, they all have Windows drivers. This translates to a transparent
increase in video performance with all my Windows based programs. I could
dump more money into an Amiga high powered video card, but it certainly
isn't worth it for the 3 applications that support the card. Most of my
software would not speed up due to my increased hardware. Matter of fact,
the Workbench would be just as slow (and if emulated by the high powered
video board slower) as it was before I bought the card. So to answer your
question Steve, the Amiga the platform with the highest rate of diminishing
returns when one throws money and hardware at it.
Tim
Gee. :^)
I'm sitting here in front of a 2500/030, with some of the oldest Amiga gfx
chips there are. I'm using AmigaDOS 2.04 on it, and I'm running with a
WorkBench display field that is 2400×1800. The screen, though, is only
640×400.
And if I pull my mouse to the side, the display scrolls (in any direction
including 45 degrees) faster than I can move the mouse, and smoooothly,
too. THat's not only scrolling text, it's scrolling gfx too. Tell me, can
your windows card do that? I'm really curious. If it can, can others, so
that if you write something to work that way, will it work at a reasonable
speed with, say, my 35$ VGA card from Taiwan?
Interestingly enough, we have the Gateway 486/33 with the ATI Ultra at the
company office; we use it to work with our PC Morphing product. And guess
what… that machine is NOT any kind of whiz-bang fast with gfx OR text.
The 040-based Amiga (still a 2000, it's using the RCS card) outruns it
most handily; and has the advantage of being able to run multiple gfx
environments at ones, something PC users like to forget as fast as they
can.
PC's have their moments – this isn't one of them, though.
–Ben
…via AutoPilot
First of all let me say that a virtual Workbench 2K X 2K when you can only
see 640X400 is somewhat ridiculous and is a very poor way to get "Better"
resolutions. With that said, yes there is a PC video card that does the
same thing for Windows. It is the Number Nine GXI TC. However, it doesn't
give you a 640 X 400 16 color view. It gives you a 1024 X 768 16.7 Million
color view on a virtual surface of 2,048 X 2,048. I suppose the Amiga
could have a card like this as well, but I wouldn't buy it for the hand
full of applications that would support it. This card works with all
Windows 3.X applications just fine. As for the ATI Ultra, well I just
uploaded a little line drawing benchmark to library 9 called linemark. I
plan on testing the Ultra tomorrow with it. I'll let you know how it turns
out.
Tim
I have a 486/33 (GateWay) running windows 3.1; it is no way as fast as my
2500/030, OR my 4000. So I guess we've got different things running, eh?
<grin>
–Ben
…via AutoPilot
PC apps are probably better written, not necessarily from a functionality
standpoint, but in terms of efficiency. PC software makers had to be
clever over the years to make up for the machine's limits, e.g., 640K main
memory. Many PC apps use tight code and clever overlays, and there are
plenty of languages and routines that programmers can avail themselves of.
For real elegance, look at GeoWorks for the PC. Great software (though
totally overshadowed by Windows). Runs OK even on an 8088 machine. And
where did the authors cut their eye teeth? On a Commodore 64, of course,
where efficiency and compact code was even more vital. The wealth of
Amiga/Mac features and programmability might tend to lead to sloppy use of
resources. Then again, top Amiga or Mac software can hold their own
against all competitors.
I'd change that to "efficient use of limited resources." On the PC or Mac,
lots of applications work in limited memory, swap temp files and data to
disk, and can process information in blocks larger than available memory,
because they were forced to do so. Can you name any Amiga programs that do
this? Offhand, I can't. But I can think of lots of programs that would
benefit from it – image processing, 3D rendering, etc. Every developer
always says "just buy more memory."
Edit
-larry
I can name one, excellence! It uses overlays, and for large documents,
it swaps parts of the doc to disk…
Bob Comer
Agreed. But I think I can name a program that fits the description:
Excellence! 3.0. It comes with VM, a built-in virtual memory manager to
let you work on docs bigger than system memory. Works only Excellence!,
not other programs. It comes with tool types to let you customize the
feature (e.g, how much spooling and when to start). However, the
developer warns that using VM could cause nasty things to happen to your
system. E.g., if you don't quit E! normally — say, you're forced to
reboot while a VM file is open — your hard drive could be corrupted. I've
got a 12-meg system, so I just keep the VMM turned off.
Depends, is your penny British, American, Canadian, Jamacan, …?
Gary Dunlap : Wyrd Software
: It just works like magic!