Data stream oriented applications are
typically dealing with huge volumes of data. Storing data
and performing off-line processing on this huge data set
can be costly, time consuming and impractical. This work
describes our research results while designing and
implementing an efficient data management system for
on-line and off-line processing of streaming data. We
present major existing data stream processing engines,
their internal architecture and how they compare to our
platform, Global Sensor Network (GSN) middleware. In order
to achieve high efficiency while processing large volumes
of streaming data using window-based continuous queries,
we present a set of optimization algorithms and techniques
to intelligently group and process different types of
continuous queries. Moreover we present an efficient query
scheduling component which not only increases the
performance at least by an order of magnitude but also,
decreases the response time and memory requirements.
Finally, we present techniques and algorithms to enable
scalable delivery of streaming data for high data rate
streams (e.g., Financial Ticks).