Benchmarking Java 1.4 versus 1.5

Posted in Computers & Technology by Elliott Back on July 7th, 2005.

Someone at work today was wondering whether Java 5 was significantly faster than Java 2. I couldn’t really say from experience, so I googled around and found a java port of linpack that should serve as a great benchmark. My system specs are:

OS Name: Microsoft Windows XP Professional
System Model: Inspiron 8200
Processor: x86 Family 15 Model 2 Stepping 4 GenuineIntel ~1694 Mhz
Total Physical Memory: 512.00 MB

After installing the latest versions of java 1.4.2_08 and java 1.5.0_01, I compiled and ran the linpack port a few times on each vm, using the corresponding javac. The timings I got for Java 5 were:

Mflops/s: 149.056 Time: 4.49 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 146.766 Time: 4.56 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 149.356 Time: 4.48 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 149.389 Time: 4.48 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 139.976 Time: 4.78 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 149.724 Time: 4.47 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 147.057 Time: 4.55 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 149.724 Time: 4.47 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 144.202 Time: 4.64 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 149.69 Time: 4.47 secs Norm Res: 10.1 Precision: 2.220446049250313E-16

Then, for Java 2, here’s 10 results:

Mflops/s: 151.419 Time: 4.42 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 152.108 Time: 4.4 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 149.023 Time: 4.49 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 151.728 Time: 4.41 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 151.728 Time: 4.41 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 151.419 Time: 4.42 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 151.419 Time: 4.42 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 140.27 Time: 4.77 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 150.364 Time: 4.45 secs Norm Res: 10.1 Precision: 2.220446049250313E-16
Mflops/s: 151.763 Time: 4.41 secs Norm Res: 10.1 Precision: 2.220446049250313E-16

I can then use software like Minitab to produce a histogram of the results (green=java2, red=java5):

Java 2 versus Java 5 benchmark

A boxplot clearly shows the speed difference between them:

Java 2 versus Java 5 box plot

So, it seems Java 2 is consistently faster than Java 5, but is it statistically significant? I leave that up to you to decide.

This entry was posted on Thursday, July 7th, 2005 at 8:10 pm and is tagged with precision 2, java port, genuineintel, linpack, physical memory, os name, professional system, javac, system model, java 2, latest versions, system specs, norm, benchmarking, microsoft windows, vm, benchmark, windows xp, microsoft. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback.

Viewing 4 Comments

Christopher Oezbek 3 years ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

Hi Elliot,
could you add a little bit more information about which kind of tests you ran (how big was the program etc)? It is especially troubling to compare such small runs (4 seconds?), because then a bigger start-up overhead might ruin your whole comparison.

Also there are some problems with the presentation of the data:

* Why are the red and green boxes culmulative in the upper chart? In generall a histogram should not be used to present individual data points. The box plot is much better.
* A good labeling would explain that the black graph belongs to Java 5 and that the red one to Java 2.
* Data is not a good way to represent a dimension ("MFlops high is better" makes a better label).
* Maybe you can add individual data-points to the boxplot and explain the individual lines and tails better.

Thanks,
Christopher

reply edit reblog flag

Elliott Back 3 years ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

Chris, I'm not sure why you're asking these generally good questions so out of context. I ran the java port of linpack I found--see link above--and recorded the times. The times are measured *after* startup, by the linpack program itself, which should be obvious if you realized what I was measuring by reading the above.

The presentation of the data is minimal--you can figure it out. It was just a quick test, my friend...

reply edit reblog flag

http://elliottback.com

Christopher Oezbek 3 years ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

Hi Elliot,
good. I had mistaken the output data for plain 'java ...' runs that you started a couple of times from the console.

The main motivation for these question was the fact, that I am working in an empirical software engineering department and those little omissions and imprecisions can make all the difference there is for an empirical result.

Cheers and thanks for the results, :-)
Christopher

reply edit reblog flag

Abhi 11 months ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

hey elliot,
thts a nice comparison.....n practical too.
i ve got a query....i was preparing for sun certified java programmer (SCJP)
frm 2 months...n i was looking for 1.5 , but my company wud re-emburse the const for 1.4 only...so ..to d surrprise ive 2 give scjp 1.4 now....can u provide me any link tht clearly specificies d difference bet both in already existing
apis or shud i just go n give test ..considering both work d same way (programatically)..! thanks in advance.

reply edit reblog flag