PolePositionthe open source database benchmark
PolePosition is a benchmark test suite to compare database engines and object-relational mapping technology. As of today it is by no means complete. Database vendors and open source database project participants are invited to improve the test implementations and to contribute further disciplines ("Circuits"). The PolePosition framework source code will help to implement own tests quickly and to time and output the results as number series and graphic visualisations.
The authors of the current test implementations do not see themselves as experts for SQL, JDO or Hibernate or for any of the database engines used. If you think that you can provide a faster implementation for a circuit, please supply it to the PolePosition project instead of criticising the authors for ignorance or the test for lacking objectivity.
This is a community project. It is for you! Please think positive, help us and be nice to us for starting the initiative. If you have a strong ego, good! Please write a better Circuit, Team, Car or Driver than we did instead of laughing about us.
Objectivity must be the most important goal of the benchmark tests. The results show very clearly that there can not be one single "best product". Depending on your application, some circuit results may be very important for you and some may be completely irrelevant. If you consider basing your choice of database on this benchmark, bring along lots of time to look at what every test does. You will not be able to judge the results without looking at the source code that produces them.
Sample results from one of our runs of the PolePosition benchmark with all the included database engines are available in HTML format here. You can also view or download the results as one big PDF file.
You are invited and encouraged to download the test suite and to run the tests yourself with your favourite database. To use PolePosition with different JDBC databases you only have to enter JDBC connection parameters into a settings property file. Test cases are also provided for JDO so you can quickly get the tests running against your favourite JDO implementation.
As already expressed above, there is no best product. The importance of individual results will depend on the respective application. However we think that we can draw some general conclusions from the results that may be valid for everyone. Because we have spent days and weeks looking at code and graphs, we think that we should point out some of our findings, that you may not see immediately, if you get lost in the multitude of graphics that are provided. Here are some thoughts:
All sources of the PolePosition test suite including a couple of open source database engines to run tests are available for download here.
We recommend you to use the Eclipse IDE to work with the test source code. The distribution contains a .project file for Eclipse to get you started quickly.
The full test suite takes at least four hours to run on a fast machine. You may like to take a look at the configuration possibilities first, so you can run fewer tests with your favourite engines.
The main class to start the PolePosition suite is org.polepos.RunSeason. At the beginning of the file a static String is set with the path to the circuits property file. If you change this variable to the uncommented debug version supplied there also, you can check out all the database engine settings on your machine first, running with few (insignificant) objects, before you wait for hours to find the last test to fail.
Here are the most important files, that you will find on your machine after you download and unzip the distribution:
Hopefully the source code has turned out readable enough to understand it. What you may want to look for:
PolePosition sourcecode is supplied under the GPL.