Benchmarks for Semantic Graph Databases usually offer the tools and guidelines for vendors to set up and run the benchmarks themselves and publish the results on public websites. However, it’s very common that vendors only publish partial non-audited results publicly, for example, data imports on W3C.org, ignoring other aspects of the benchmark. Also, only some benchmarks support reasoning — one of the main selling points of RDF Graph databases — or can simulate enterprise environments where concurrent users execute realistic workloads of use case motivated queries over a certain period of time, which is particularly important to determine the scalability of the system under test and expose data concurrency bottlenecks. The Benchmarks that do include all the above do not seem to be widely used by RDF Graph Database vendors.
Most semantic graph database benchmark websites haven't been maintained or released audited results for years. Apart from publication of partial results from some vendors, there is not much going on.
We see this as a severe issue for the semantic graph database market to overcome since graph database selection based on hard facts is extremely hard to do by prospective buyers who do not yet have any experience with using RDF Graph Databases for mission-critical use cases.
The following are the main graph database benchmarks available today.
For more info refer to Enterprise Knowledge Graph Foundation (EKGF) at ekgf.org
At agnos.ai we think that Graph database benchmarks are very important for the success of the market, the same way tpc.org has been for the relational database market for decades. Graph Databases are implemented by a growing number of storage systems and are used within enterprises and the Web. For RDF Graph Databases, as SPARQL is taken up by the community, there is a growing need for benchmarks to compare the performance of storage systems that expose SPARQL endpoints via the SPARQL protocol.
We have been working on enhancing the capabilities of some of the most popular benchmarks towards more comprehensive and mature sets of tests. We currently offer the following extended versions:
Please check our web site for more details on the benchmarks above and other benchmarking services we provide that can help your organization.
Co-author: Jacobus Geluk — agnos.ai CEO.
