|
|
Current conferences and/or journals do not encourage submission of
mostly (or purely) experimental results. It is often difficult or
impossible to reproduce the experimental results being published,
either because the source code of research prototypes is not made
available or because the experimental framework is under
documented. Most performance studies have limited depth because of
space limitation. Their validity is limited in time because
assumptions made in the experimental framework become obsolete.
EXPDB 2006 is meant as a forum for presenting quantitative
evaluation of various data management techniques and systems.
Keynote speaker: Jim Gray (Microsoft)
Triumphs, Sins, and Challenges of Database Benchmarking
It is useful to review the history of database benchmarks -- how they
have evolved and what they tell us about balanced systems. It also
makes sense to document our failure to create benchmarks that measure
total cost of ownership (operations costs and usability.)
Looking at benchmark reports suggests that few people have performance
problems. A single 4-processor 800-disk server delivers ~300k tpmC or
about 400 million business transactions per day. That is far beyond
the needs of almost any company -- so there is no performance problem.
But... that's not what I hear from people who build and manage
applications. On the surface, they have much less demanding
workloads, but they cannot achieve comparable performance on their
workloads. This guru-gap seems to be widening as we offer
more-and-more automatic tools, scripting interfaces and higher levels
of abstraction.
In part the gurus are filling in for the tools (providing "hints" to
the subsystems), in part the gurus are avoiding design pitfalls, and
in part the gurus are fixing their systems do to the right thing. In
the long term, we have to automate all three things: The software
has to work without hints, it has to "flag" design pitfalls and
recommend correct designs, and of course it has to be fixed to
perform well.
Accepted papers
|
