Comparison with other my databases

To compare PERST features with other my databases view the following document: compare.html.

PolePosition benchmark

PolePosition is a benchmark test suite to compare database engines and object-relational mapping technology. As of today it is by no means complete. Database vendors and open source database project participants are invited to improve the test implementations and to contribute further disciplines ("Circuits"). The PolePosition framework source code will help to implement own tests quickly and to time and output the results as number series and graphic visualisations.

The authors of the current test implementations do not see themselves as experts for SQL, JDO or Hibernate or for any of the database engines used. If you think that you can provide a faster implementation for a circuit, please supply it to the PolePosition project instead of criticising the authors for ignorance or the test for lacking objectivity.

This is a community project. It is for you! Please think positive, help us and be nice to us for starting the initiative. If you have a strong ego, good! Please write a better Circuit, Team, Car or Driver than we did instead of laughing about us.

Objectivity must be the most important goal of the benchmark tests. The results show very clearly that there can not be one single "best product". Depending on your application, some circuit results may be very important for you and some may be completely irrelevant. If you consider basing your choice of database on this benchmark, bring along lots of time to look at what every test does. You will not be able to judge the results without looking at the source code that produces them.

Original PolePosition test suite doesn't include Perst port. So I have created Perst team for PolePosition myself. Unfortunately I have to copy some files from PolePos framework and make my local copies of them (because Perst requires all persistent capable classes to be derived from Persistent base class). It is possible to avoid this requirement, using AspectJ or JAssist reflection packages but it will significantly complicate integration of Perst in this test suite. The sources of Perst team can be taken here. I have performed competition only between Perst and db4o because it seems to be the most close competitor for Perst and because all other databases requires some kind of installation and launching server. Please look at the results. I have to notice that advantage of db4o in write and update operations can be explained by lack of transaction support in db4o (at least in configuration used for testing - it doesn't perform synchronous write to the disk during transaction commit and so power or system failure can cause loosing of committed changes and even corruption of the whole database file. In Perst it is also possible to avoid synchronous writes, but I think that it will be not honest to switch on this option in the test measuring performance of transaction commits.

OO7 benchmark

OO7 is simple and popular benchmark for object oriented databases. Port of this benchmark at Java is included in Ozone (pure Java OODBMS). I have ported this benchmark for Perst and also made some changes and bugs fixed in original Ozone OO7 implementation. My changes are the following:

  1. Add missing file in the preprocessor list in build.xml
  2. Fix bug in "query match" implementation
  3. Increase number of searched object in "query match"
  4. Remove all trace messages to stdout which affect performance

Patched version of OO7 benchmark for Ozone is available here: oo7.zip
Results of the benchmark (in milliseconds) are as follows:

Productcreate smallquery traversalquery match
Perst1 0411 3520 431
Ozone112 73213 6800 591

Index performance

One of the most popular commercial pure Java OODBMS at this moment is ObjectStore PSE Pro. ObjectStore does not accept the OO7 benchmark at all and disallow the publication of any results based on this benchmark. That is why I have ported for ObjectStore my TestIndex benchmark which measures performance of such basic operations as storing/fetching object and locating object using index. This test contains three steps:

  1. Create specified number of records with random long integer and string key and include them in long integer and string indices. After inserting of all records, commit is performed.
  2. Search for each record using long integer and string key.
  3. Search and remove each record from the indices and deallocate it from the storage.

The version of TestIndex project for ObjectStore is available here: testindex.zip.

Another object oriented database for Java/C# is db4o.C# and Java version of TestIndex port for db4o is avilable here: db4o-testindex.zip.

One of the most popular databases for embedded applications is BerkeleyDB by Sleepycat Software, including a Java version of BerkeleyDB. It doesn't provide transparent integration with Java - it is not possible to transparently store/fetch instances of Java classes. Instead of it database is represented as set of <key,value> pairs and application classes can be stored using a serialization mechanism. The port of the TestIndex.java example for BerkeleyDB is here: je-testindex.zip.

And yet another competitor is JISP (Java Indexed Serialization Package). Jisp provides implementation of B-Tree and is able to store objects in the database using serialization mechanism. It seems to be all it can do. Jisp is written in pure Java and is very small. The version of TestIndex for JISP is here: jisp-testindex.zip.

We should also consider JDO compliant databases, such as ObjectDB, but its evaluation version does not allow storage of more than 5000 objects in a file and also does not support collections, so a meaningful comparison is not possible. A popular personal JDO-compatible database is FastObjects from Versant. It fully supports JDO and provides many of its own extensions. Unlike ObjectDB which enhances byte-code at runtime using a special class loader, FastObjects requires you to explicitly run a byte-code preprocessor after compilation. Unlike all other competitors, FastObjects is not written in pure Java - it requires loading of native code. The port of TestIndex for FastObjects is here: poet-testindex.zip.

Time of each step execution in milliseconds is measured. Number of records in each case is the same - 100000. Page pool size for Perst in is 32Mb, which is enough to hold all records in memory. All test were executed at the same computer: AMD Athlon 64 (3200+), 1.5Gb RAM, WinXP. I am using Sun Java JDK 1.4.2_04.

ProductLanguageCreateSearchRemove
PerstJava3 7751 6833 275
PerstCSharp4 4462 4033 975
ObjectStore PSE ProJava8 2729 4133 104
FastObjects J2Java13 39910 85638 435
FastObjects.NetCSharp43 0122 7147 461
db4o-4.0Java18 4576 27938 886
db4o-4.0CSharp31 72541 09988 517
Berkeley DB JEJava(*)15 51310 75512 758
JISPJava350 674343 063527 248
(*) - JE has to be started with -Xmx256M option to avoid memory overflow. Also it is necessary to notice that database close time is 64 seconds.

Large Database Support

Perst is able to deal with really large number of persistent objects. One of my tests (TestMaxOid) stores in database one billion objects (1000000000=10^9). At quite ordinal computer (PIV-1.8G, 512Mb RAM, IDE disk) it takes 12 hours. The total database size was 66Gb. Performing index search for each object in this database (i.e. 10^9 index searches) takes about 8 hours (32 thousands searches per second). Iteration through all these objects in key ascending order - 6 hours. Search and deletion of all 10^9 objects - 10 hours. I want to notice that 10^9 is not limit for maximal number of objects in Perst, the limit is 2**31, but I do not have so much free space on disk to perform test with larger number of objects.

I didn't perform this test for other DBMSes, because it requires a lot of time and free space on disk. But if you think that some other pure Java or C# DBMS can offer better results, I will be glad to perform competition.

Conclusion

Perst is faster than most of competitors. If you are interesting in comparing Perst with some other DBMS and has test for it - please send it to me and I will port it to Perst.


Back to my home page | Send me a mail