Skip to main content

Notes on Real Time Data Processing



Real time is real hot right now. Days became hours, hours became minutes, minutes became seconds, seconds became nano seconds. Time is money. Go ahead. Show me the money. This is the mantra from all of hardware and software vendors. Do the analytics in real time. Your decisions, results and revenues will automatically jump up. When you perform a database transaction, events are triggered by exchange of messages using some protocol. The systems are recording changes as the event are unfolding. Analysis of this data should happen immediately after the event.  So, colloquially, real time processing is analysis of data as the event is unfolding. 

But, What is real time?

In all of cosmos, there is one, just one phenomenon we know happening in real time.  EPR thought experiment was invented by Einstein, Podolsky, Rosen to show quantum theory was an incomplete explanation of reality.  Simply put, it shows the information can travel the length of the universe — instantly. This violates Einstein's special theory of relativity which states that nothing can travel faster than light. Experiments have shown EPR's REAL TIME effect, seemingly faster than light, does happen. Such is the strange quantum world. See http://en.wikipedia.org/wiki/EPR_paradox, if you’re curious about more information. 

So, in physics, you  hardly hear about real time travel or information processing.  Speed of information travel, except in quantum world, is restricted by speed of light.

Real Time in Software. 

Computer science and software, of course, won’t be bothered with such constraints.  Everything from streaming, fast data ingestion, messaging, business intelligence to queries on the stream claim real time computing.  There is, however,a restrictive and reasonable definition of real-time computing in Wikepedia:  hardware and software systems that are subject to a "real-time constraint", for example operational deadlines from event to system response. Real-time programs must guarantee response within strict time constraints, often referred to as "deadlines". The “real-time constraint” can be defined exactly in cases where violation has immediate effect, like fire extinguisher, furnace temperature monitor, etc.

In other cases, e.g. stock trade or option trading, real time means doing it faster than competition to maximize revenue & profit.  The faster you do it, the higher is the revenue/profit possibility. So, for that, what’s the limit? Nano second? Planck second? Here, every nano second counts. Every type of bandwidth counts. Every optimization counts.  All to save precious time.

 So, it's important to realize which use case your real time system is handling -- whether you're time-bound to ensure you avoid catastrophe or you're chasing time's tail to maximize profit. Once you determine your constraint, it's easier to design systems, select the right architecture & tools to achieve that.

For interesting evolution of the term real time, see Phillip Laplante's article on It Isn’t Your Father’s Realtime Anymore at: http://queue.acm.org/detail.cfm?id=1117409
Labels:

Comments

Post a Comment

Popular posts from this blog

Swami Vivekananda: The Monk That Nobody Sent to Chicago

  There’s a saying in Chicago: “We don’t want nobody that nobody sent.” This was the cold reception Swami Vivekananda faced when he arrived in the windy city in July 1893, determined to attend the World Parliament of Religions that September. He belonged to no organization, carried no letter of recommendation, his countrymen were nobody, and represented an alien religion to the Western world. As the days passed, his hope of attending the parliament dwindled. With money running out and the odds stacked against him, he left the Windy City and went to Boston, praying for a glimmer of opportunity.  Swamiji came to America to share India’s most profound gift: the wisdom of the Hindu sages, preserved through centuries of oral tradition and embodied by its monks. This was 1893, not 1993—India was under the British grip, its resources drained, and its spirit subdued. Swamiji’s mission was not just a cultural exchange; it was a bold step toward envisioning a future where India could re...
Post a Comment
Read more

Why Should Databases Go Natural?

From search to CRM, applications are adopting natural language and intuitive interactions. Should databases follow? This article provides a strategic perspective. Amid the many technological evolutions in software and hardware (CISC/RISC, Internet, Cloud, and AI), one technology has endured:  Relational Database Systems   (RDBMS), aka SQL databases. For over 50 years, RDBMS has survived and thrived, overcoming many challenges. It has evolved and adopted beneficial features from emerging technologies like object-relational databases and now competes robustly with   NoSQL databases .  Today, RDBMS dominates the market, with four of the top five databases and seven of the top ten being relational. RDBMS has smartly borrowed ideas, like JSON support, from NoSQL, while NoSQL has also borrowed from RDBMS. NoSQL no longer rejects SQL. From a user perspective, all modern databases have SQL-inspired query language and a set of APIs. All applications manage the respective data...
Post a Comment
Read more

Accelerating Insights With Couchbase Columnar

Insights come from observing and analyzing data from all sources. Couchbase Columnar helps quickly analyze data from a variety of data sources with zero-ETL. The purpose of computing is insight, not numbers. -  Richard Hamming Capella  Columnar  is an advanced real-time analytics database service from  Couchbase , targeted for real-time data processing, offering SQL++ for processing  JSON  (semi-structured) data and more. This service enables data to be managed locally and streamed continuously from both relational and NoSQL databases, or simply process data on S3. The columns or fields of the source are directly mapped to a field in the JSON document at the destination automatically. This is really a zero-ETL operation. A key feature of this system is its ability to continuously stream data, making it immediately available for querying, thus ensuring near real-time data processing.  At the heart of Columnar's architecture is a Massively Parallel Proce...
Post a Comment
Read more