The History of Real-Time Applications: Where’ve We Been, Where We’re Going
Technical and cultural obstacles that prevented more widespread use of real-time systems and applications are disappearing, thus making them more common.
Real-time systems have been around for decades but were only
used for special purposes. Now, real-time applications are critical for safety
and security, as well as for business success. Fortunately, technical and
cultural obstacles that prevented more widespread use are disappearing.
RTInsights recently sat down with Paul Butterworth, co-founder and CTO of
VANTIQ, to discuss these issues, what’s changed that makes it easier to develop
and deploy real-time applications, and more.
Here is a summary of our conversation.
RTInsights: For the last ten years, analysts and
companies have talked about the benefits of real-time business operations and
the need to move in that direction. What prevented this from happening?
hasn’t been completely prevented, but I would say it’s not developing as fast
as everybody expected ten years ago. The realities of both the physical
environment and everyone’s knowledge of building real-time systems have had a
play in slowing down the adoption of these kinds of systems. For example, many
people talk about real-time systems that do tracking. Imagine that you could, today,
pretty much track anything under any conditions. We and our customers have been
finding it is a lot harder to do than you think. Suppose you want to track,
let’s say, packages, and you’re interested in tracking them in real-time. That
means you must have some sort of a device on the package that can broadcast its
location and some way to pick up those broadcasts and get them back to the IT
systems trying to do the tracking.
you’re willing to spend enough money to track each package. In that case, you
can probably do that because you can use cellular technology to transmit the
data in pretty much any environment. But if you’re trying to do this at a low
cost, the capabilities to track it in an arbitrary location are difficult to
find. In many cases, they will work in some regions and not others. So, if
you’re tracking in major urban areas, you might not have m a problem. But if
you’re trying to track around the globe, those systems may not be available in
every location. That just makes it more difficult to accomplish and more costly
to do it. So, it may not be cost-effective.
like that often come up. Another example is using cameras to do tracking. I am
guilty of this. “Oh, well, we have cameras. We’ll just put them up, and
we’ll start tracking stuff.” Well, there might be environmental conditions
that will cause problems with that. You must think about what happens if you’re
tracking in very bright environments or very dark environments. I know your
average webcam doesn’t have all those capabilities, and many times you don’t
realize that until you get into the project. And a lot of times, those things
always slow you down. If you run into enough roadblocks and hurdles, people
will just give up on the project. That’s one area that slowed down the adoption
of real-time streaming systems.
other big thing that we see out there is that the technical people are used to
building traditional IT applications. They aren’t that familiar with real-time
systems. And real-time systems have a different style and a different
architecture to them.
happens is that people try to build real-time systems using traditional
techniques. That either becomes expensive because you must throw more and more
hardware at the problem. Or you just can’t get the systems to run fast enough.
Or you can’t get them to be reliable enough to do the real-time processing that
you expect in those environments.
thing we see is that you have to be able to think about real-time systems and
how they will help your business. Many times, if you haven’t got some
familiarity and some experience with building these kinds of real-time
streaming systems, it’s really hard to imagine how they’re going to help you.
And so, industry education is another element to consider. That’s another part
of what’s been slowing the adoption of real-time and streaming systems.
been some progress in real-time systems recently. Given the current state of the
real-time market, how did we get here?
causing things to improve? Well, it’s a little bit of everything. The
underlying technology’s improving, and networks are getting better. Computing
systems are getting faster and less expensive, so they can be more easily
applied to some of these problems in a cost-effective fashion. A big thing is that
edge computing is evolving; many of the real-time systems have an IoT flavor to
you’re building real-time systems that track physical stuff, there are sensors
out in the world. And those sensors must be connected to computing
infrastructure in some way. Typically, the most cost-effective way to do that is
to connect them to edge computing capabilities. There’s been a lot of talk
about it over the last 10 or 15 years, but it’s been hard to do unless you
already had the infrastructure deployed. If you were trying to monitor stuff in
every store, had thousands of stores, and didn’t already have computing
infrastructure in those stores, getting the compute capacity out there was a
there’s been a lot of progress in this area. Many suppliers are working hard to
make it easier to do edge computing. Mobile telephone network operators with
MEC [multi-access edge computing] technology, other edge suppliers, content
delivery networks, and others offer capabilities. The big cloud providers are
trying to do something to help support edge computing. All of that is making it
easier for customers to deploy devices out in the world and hook them up to
computing infrastructure. And more of that computing infrastructure is available
or is going to be available as a service. To go along with that, networking is
getting more ubiquitous and faster all the time. So, all that kind of
infrastructure is helping to solve the problem.
these different technologies’ availability is that we have more people becoming
educated on building real-time systems. More businesses are starting to
understand how real-time and streaming systems can help their businesses. They
see the use cases.
the tools that are available for doing real-time and streaming processing are
continuing to improve. When everything was a hard real-time system, it was very
difficult to build those systems in the early days. You had to work very close
to the hardware to get the performance that you were looking for.
hard real-time systems are easier to build because the technology is there. And soft real-time systems are easier to build
because there are much better tools out there, and you can work at much higher
levels. There’s the beginnings of low-code and no-code solutions that work in
these streaming environments. What we’re doing [at Vantiq] is an example of
we have better knowledge, better underlying technology, and better tools. That
is helping to accelerate the adoption of these systems today.
RTInsights: What’s the impact of all of this on
development and deployment environments?
Butterworth: Development environments are getting
better. That is important because building real-time systems, streaming
systems, differs from building traditional IT data processing applications. You
need tools that are specialized or that optimize the construction of real-time
streaming architectures to make teams the most productive.
Several things are going on. There’s software, such as
messaging systems and event brokers, that you need to run real-time
environments. There are productivity tools that make it possible for you to
build these applications by operating at a much higher abstraction level. Many
times, the tools will integrate the infrastructure underneath the application.
So, you never have to worry about that. All you have to think about is what
your real-time system needs to do and how you express that in a high-level development
environment. That makes a huge difference.
It’s one thing to say you can develop the systems more
quickly and efficiently, but you need a full suite of tools. Once you get applications
developed, you have to deploy them and deployment into distributed and edge environments.
That can be quite complicated. Having tooling to help you with that is always
You also need tools to help manage real-time distributed
systems. This is important because anyone familiar with distributed systems
knows that it’s very difficult to manage them unless you have some good
information and instrumentation on what each part of the system is doing. If
something goes wrong, which part of the system is responsible?
You need to have tooling across the life cycle of real-time
applications to take advantage of these systems and make them more
RTInsights: Isn’t there more than just technology
involved? Don’t you have to change mindsets for companies to embrace real-time
systems and applications?
Butterworth: Businesses need to understand how this
technology can help them. It might help by improving productivity or providing
better service to customers.
One of the things we look at is how to reduce the burden on
customers to provide them with information. Traditionally, pretty much all
businesses were looking for information for customers. They’d put up some form
and have folks type all that information in.
If real-time systems are available, it might be that the
sensors in the environment can provide context and help answer some questions.
An obvious example is location-tracking information. You already know where the
customer is. You don’t have to ask that question. If you know where a customer
is, you might be able to provide them with better information concerning how
your products or your service relates to the environment that they’re in.
Another example is using a smartphone or some other device,
so users don’t have to do a lot of data entry. Many times, you can just take a
picture of something. The system can extract data from that in real-time and
feed that back to you. That’s great, especially with industrial equipment where
you’re in a tough environment. You have many distractions, and typing in serial
numbers off a piece of equipment is incredibly error-prone. Taking a picture
will solve the problem instantly.
RTInsights: How does
VANTIQ help with real-time systems and real-time application development?
Butterworth: We think of this as our specialty. Our
goal is to help our customers conceptualize, build, deploy, and operate
real-time, event-driven systems. We want to make it as easy as possible for you
to build these things.
At the same time, we make you more productive. We’re
providing the mission-critical infrastructure that these applications run on. If
you have to build larger applications, you don’t have to give up on high
productivity. You can build very large applications with VANTIQ that ingest
data at very high rates, process that data at high rates, and can provide you
with a real-time response in such a high traffic environment. That’s a unique
proposition. Usually, high-performance systems and high-productivity tools
don’t fit together very well. But that’s been our goal from the start. If your
system runs thousands, tens of thousands, or hundreds of thousands of events a
second, you can build them in our production environment, deploy them, manage
them, and run them reliably. And that’s a unique proposition at this point in
RTInsights: What are some interesting use cases?
Butterworth: Some of the most popular ones in our
user community at the moment are security and safety applications. A lot of
this has been driven by the Coronavirus. People are trying to make their work
environments safer. When you start thinking about that, you realize there are
other kinds of security and safety things you can do using the same
technologies. You have cameras and sensors of various kinds that can track
where people are.
That’s very helpful in emergency situations when you need to
evacuate an area or evacuate a building. Knowing where people are and whether
there’s anyone left in the area becomes a valuable feature of those kinds of
real-time systems. A similar use case is congestion management, making sure
crowds are not growing in particular areas. If you can do these things in
real-time, you can keep bad situations from happening.
We see a number of energy-related real-time systems. One
kind is just monitoring an electrical grid. Customers are trying to see when
there’s a problem with the grid. Some of these applications are just about
tracking more precisely where the power outage is and then notifying their
customers in that area about what’s going on, the expected time to repair, and
how extensive the problems are. It’s not just about controlling the grid, but
providing better service to their customers.
We see some retail applications, particularly around
interacting with customers to provide them with a better service or more
relevant information based on where they are and what they are doing at the
moment. We also see more back-office applications of real-time things, for
instance, more real-time updating of prices in stores using more modern,
electronic pricing systems. Rather than updating these systems daily or once
every couple of days, they’re doing it in near real-time.
There also are more IoT-enabled, connected retail
environment applications. That includes smart shopping carts, smart shelf
labels, and things like that. With smart shopping carts for grocery stores, a
customer just drops things in the cart, and the system figures out what they
are and how much you pay for them. If there are things that you pay by weight,
they have a little scale, and they can just measure the things on the fly when
you fill up the basket. The price is right there, and you get charged for that
stuff, and you’re ready to go. Those are very cool.
And then, as a bigger picture, maybe more loosely defined is
smart buildings. What are all the pieces to pull together into creating a smart
space solution that might have elements of everything we’ve talked about but
combining them into one whole?
You see this happening with smart buildings. The buildings
are equipped with individual systems. What if you want to optimize the total
system across the whole building? That’s another area where VANTIQ comes into
play. We try to pull together all the different energy management systems, plus
the people tracking systems that can then interplay with the energy systems.
You put all of that together, and you can build a much more sustainable
Discover Past Posts