Europe Union

Home Case Studies Stream Processing Engine (SPE) That Makes All Sensor Data Usable at the Moment They’re Generated

Stream Processing Engine (SPE) That Makes All Sensor Data Usable at the Moment They’re Generated

Farms, factories, and industrial systems now generate continuous streams of information every second. But is this data really useful when it's needed?

Most organizations rely on systems designed for delayed analysis. Data is collected first and processed later, often in daily or hourly batches. When insights finally appear, the opportunity to react already passed.

And what about the fact that data comes from many different devices, each with its own software, format, and dashboard? This forced organizations to juggle multiple tools or invest heavily in custom integrations.

DAC.digital in a powerhouse collaboration with the University of West Bohemia and ICPS (Latvia) decided to solve this problem.

Originally posted 2020
Updated for 2026

Data was available but analysis lagged behind

We took upon the challenge while analyzing existing situation. Existing IT and analytics systems were not designed to handle continuous data streams. Most solutions relied on batch processing which means that data was collected first and analyzed later.

And since data came from various sources, integrating which required custom adapters, middleware, or manual aggregation, which slowed operations, increased costs, and created brittle architectures that were difficult to scale.

Even when companies attempted to use existing stream processing frameworks, they ran into another major obstacle: these tools were primarily developer-focused toolkits. Deploying and managing them required specialized expertise in distributed systems, message streaming, and open-source infrastructure. This made real-time processing expensive, slow to deploy, and hard to maintain, particularly for organizations without large technical teams.

Smart Agritech Solutions

So, in collaboration with the University of West Bohemia and ICPS (Latvia), DAC.digital set out to design a solution that could transform sensor data into actionable intelligence at the moment it is produced, without requiring organizations to assemble and maintain complex streaming architectures from scratch.

State of the art steam processing engine for industrial use

Stream Processing Engine (SPE) is a system capable of ingesting, processing, and analyzing data streams from multiple devices and software platforms in real time. By combining Apache Kafka with a cutting-edge Lambda Architecture, SPE delivers both immediate insights and accurate historical analytics within a single, unified system.

One of the most advanced applications of SPE is its support for Artificial Intelligence workflows. Data from a single topic, or a combination of topics, can be fed directly into AI training algorithms. These algorithms can be implemented as microservices, allowing continuous retraining and improvement as new data flows through the system. All of this operates within the streaming architecture.

Concept picture

Stream Processing Platform

There is also a need for a solution such as SPE that would be able to integrate multiple streams. Having all electronic devices with their own software makes it impossible to integrate with devices from other companies. It results in a major disadvantage of managing many different software apps to display and process data.

The SPE was developed by DAC.digital, addresses this gap in the state-of-the-art and provides businesses with an easy-to-use emerging technology tool, which could boost the businesses’ operations and productivity.

Get in touch to see how our solutions can address your needs.

Contact us

The Solution: How does it work?

The SPE provides real-time data analytics based on Lambda Architecture, i.e. a generic, scalable, and fault-tolerant data processing architecture. This architecture is based on an append-only and immutable data source. Thus the serving layer is decoupled from data (events) storage and processing. Figure 1 shows the SPE within the AFarCloud platform Semantic Middleware (High-Level Services layer). 

Figure 1: Stream Processing Engine within the AFarCloud platform Semantic Middleware

The aim is to process the data inbound from third-party artifacts (data sources, software systems, and devices such as sensors) within real-time constraints. An example of processing functionalities is as under.

  • Pre-processing actions, e.g. filtering and cleaning the inbound data in order to reject all irrelevant or corrupted data.
  • Data aggregation, i.e. combining multiple data sources in order to prepare combined datasets for further processing (grouping data into topics, for instance).
  • Data analytics, e.g. calculating statistics or specific functions (for example, Product Environmental Footprint).
  • Checking business rules for triggering specific action, i.e. creating an alert or calling a specific function.
  • Publish-subscribe mechanism, i.e. AFarCloud stakeholders can provide data on specific topics in order to observe and be consumed by other participants.

Data Processing Solution

Data Processing Solution

SPE utilizes the Apache Kafka platform to implement a Data Broker, as shown in the figure 1 above. This is the core element that manages the inbound data. Kafka provides tools for managing real-time data pipelines and creating provider, consumer, and streaming applications.

Figure 2 presents SPE within the AFarCloud architecture. It is a part of the Data Management layer. The data is directly provided from the AFarCloud Interfaces layer via the Data Access Manager. Therefore there is no need for any additional middleware component (converter or adaptor) for stakeholders that publish the data within the AFarCloud ecosystem. In order to forward data to the SPE as well, the SPE Data Provider must be implemented.

Figure 2: Stream Processing Engine within the AFarCloud architecture

Key features of the Stream Processing Engine that makes it a reliable solution for industrial use

  • Aggregates and processes data from multiple devices and software systems in a centralized environment, without extra tools or custom integrations.
  • New analytics, metrics, or AI models can be added seamlessly as microservices without disrupting existing workflows.
  • Handles millions of events per second and scales horizontally as data volumes grow, so performance keeps pace with expanding sensor networks.
  • Supports immediate, event-driven insights as well as historical analysis, thanks to the Lambda Architecture, allowing organizations to act fast without sacrificing accuracy.
  • Built on Apache Kafka, SPE ensures fault-tolerant data ingestion, ordered event streams, and persistent storage, so it works even under high load or system failures.

A comparison of the SPE with others publish/subscriber-based technologies

First, let’s discuss functionality

FunctionalitySPE based on KafkaRESTMQTI
Filtration and processing dataallowed on streameverything must be implemented from scratchno
Requests serviceflexible scalability, big throughputoverload by many requestssmall throughput

Then, let’s talk about architecture

ArchitectureSPE based on KafkaRESTMQTI
ComplexityA complete and standalone application. Use TCP binary protocol for communication.API exposing endpoints. Use HTTP protocol which is significantly slower than plain TCP binaryLightweight transmission protocol. Optimized for sensor networks and M2M
Persistence and reliabilityEnsures high reliability by adding the persistence layer and holding copies of streams.Everything must be implemented from scratchNo embedded persistence

Who can make use of this technology?

Farmer icon

Individual farmers who want their own tailor-made software with the possibility to further extend

Data analysis icon

Data analysts working in the agricultural field – for rapid development and tests of data processing algorithms in the same framework

Efficient Systems icon

SPE can be implemented by an integration/software enterprise as a core used for integrating a few separate agricultural apps into one software or system

Example Applications of the Stream Processing Engine (SPE) across agriculture, farming, and industrial IoT environments

Use Case.

Farm dangerous events detection

Scenario.

Sensors that monitor the farm ecosystem (plantation or animal breeding) produce data that can be used in the SPE to detect dangerous events.

Solution.

Stream Processor (see Figure 1) can be used to monitor the specific type of events originating from the farm ecosystem in order to check the defined business rules. In the case of fulfilling the condition, it triggers specific actions, for example, append an alert to the specific topic in the SPE stream. The alert can be handled by dedicated software, for example, a system that informs the end-user (farmer) about the dangerous situation or a system that manages the dedicated vehicle (e.g. drone) to start the mission.

Use Case.

Monitoring of cow breeding zone

Scenario.

Nowadays, modern farm is equipped with sensors that monitor the cow farming ecosystem. Data from these devices should be fused for further processing. This employs applications and algorithms that provide all necessary information about breeding. This approach enables detecting abnormalities, planning future breeding, calculating the costs of infrastructure and herd maintenance, etc.

Solution.

Farm Management System layer (e.g. Decision Support System) or third-party software can consume data from the SPE in order to use it in the breeding domain algorithms and applications that monitor and manage the cow breeding zone.

Read More of Our Case Studies