Neil is a solution architect for Hazelcast, the world’s leading open source in-memory data grid.
In more than 25 years of work in IT, Neil has designed, developed and debugged a number of software systems for companies large and small.
Java 8 gave us a new language abstraction, the stream, which is good but not good enough.
We have some nice abstractions that enable us to code succinctly, and the JVM can exploit multi-processor machines for internal parallelisation. But fundamentally, the implementation is not distributed, we can’t pass data from JVM to JVM cleanly for external parallelisation.
In this talk we’ll look at Jet, a new open-source offering from Hazelcast, that solves this very problem.
The core construct is a DAG – directed acyclic graph – that runs across multiple machines and allows you to plumb together a pipeline of data from place to place, so a thread in one JVM can emit Java objects that a thread in another JVM takes as an input stream. All this happens in-memory, so it’s super fast and you can scale across as many JVMs as you need to boost the parallelisation.
For the developer, familiar constructs such as maps and filters are available so it’s easy to get going, and there’s a library of connections being developed so you can plug in existing data sources and data sinks with minimal effort.