Gaze Estimation in the Wild. Creating a Ground-Breaking Eye-tracking System for In-context Market Research

After a successful collaboration on the Spark surveying platform, eye square came back to us with a new project for market research: developing gaze estimation solution to track user attention while browsing content on their smartphones.

What an innovative and uncharted territory! An eye-tracking software without extra hardware and in-lab environment.

Client

Name: eye square GmbH
Line of business: market research, human experience
Founding year: 1999
Size: 50-200 employees
Country: Germany (Berlin)

Challenge

We partnered again to create something that hasn’t been done before – a highly accurate eye-tracking solution that can work on any equipment with a built-in camera.

Solution

Our team managed to built a PoC of the solution that accurately detects the point of gaze without specialized equipment. We achieved:

80% – the target accuracy for the next phase
<1cm – the target margin of gaze point detection error

👋 Hey You, What Are You Working On?

Share a few details about your challenge or idea, no matter how early or complex. Our expert team will reach out to offer our expertise in terms of feasibility, budget, and scope. Let’s shape a solution for you together!

Let’s Discuss Your Project

What Developers Said About the Challenge

One of the biggest challenges was estimating the gaze point on the screen in a non-controlled environment with varying light and camera positions. Contemporary solutions rely on expensive hardware and extended sets of recorded people. Few solutions currently available on the market would do what ours does with just the built-in smartphone camera.

The challenge that we find to be most formidable is the complexity of the problem. We must perform several tasks simultaneously to find the touch point between the gaze and the screen surface. And these tasks are complicated research challenges on their own.

The biggest challenge is to achieve high accuracy of the point of gaze estimation, i.e. the point the smartphone user is looking at. Our client wants the margin of error not to exceed 1 cm for a wide range of phones and their users. It’s a difficult task for a variety of reasons. Firstly, every person is different, and their eyes have a unique shape, colour, etc. Secondly, each phone has a different camera, size and screen ratio. Thirdly, each use case has different lighting conditions, and the person holds the phone differently. All of these factors make the task and the ultimate goal so complex.

Eye-Catching Innovation in Computer Vision

We tackled a complex challenge – designing a precise gaze estimation solution that could work without requiring a controlled research environment.

Traditional eye-tracking systems rely on special hardware and strict conditions, making them less practical for real-world use. Our computer vision system, on the other hand, can track where a person is looking using just a smartphone camera.

This raised several obstacles, including calibrating the phone camera for accurate gaze estimation. Since phone models vary in screen size, camera placement, and undisclosed physical dimensions, finding a universal estimation method is difficult.

To ensure users can use their phones freely, the gaze estimation algorithm must be auto-calibrated across different smartphone models, accounting for varying angles, distances, and face detection capabilities. We managed to tackle this challenge.

Hardware-Free, Mobile Revolution for Eye Tracking

Before reaching out to us, the company experimented with several third-party solutions for laptop-related use cases, some of them required external hardware.

The main goal this time was to create a new solution that would be suitable for mobile phones.

Our team worked on the Proof of Concept to ensure the visions were aligned before starting the subsequent phases of such a complex R&D project.

All Eyes on Gaze Estimation

Our team comprised a Senior Computer Vision Engineer, Machine Learning Engineers, DevOps specialists, frontend developers and a project manager.

Additionally, eye square supported us with three technical Project Managers (one of whom has become a project coordinator), the CFO and a developer to provide extra help.

An essential part of their contribution was coordinating the crowd-sourcing platform to acquire testers and data sets. The technical project managers Garrit, Frederic and Olaf also provided their expertise and help whenever needed.

How We Exceeded State-of-the-art Tech, Technically Speaking

The crucial part of the process was to create a stable foundation that would allow the creation and development of new features that would bring it closer to the final project and what it should look like.

We applied Python, PyTorch and OpenCV, among other libraries, to create the base algorithm. We later based the development on testing data from early tests and larger data gathered via the ClickWorker crowd-sourcing platform.
JavaScript was used to develop WETSDK, Training Data Collection App (TDCA) and an example web application illustrating the production use case.
FastAPI was used to develop a communication interface between the end user – the web app and the algorithm running on the backend server
AWS allowed us to store the training and validation data in the cloud
Docker made it easier to encapsulate the algorithm in self-contained SW images that can run in the cloud

What Developers Said About Tech

One of the most important technologies we use would be computer vision systems (based on Pytorch framework), linear algebra, and our skills involving careful reading and applying the solutions we found in different papers.

The technologies I found especially helpful were Python, Numpy, OpenCV, PyTorch, MediaPipe, Scipy, GPU and CUDA.

The WET project wouldn’t be possible without applying machine learning algorithms. We use deep learning elements at every stage of image processing – from face detection in the three-dimensional surface to gaze point estimation and like. We find Python and PyTorch to be invaluable there.

Challenges with Gaze Estimation in the Wild

Due to its complexity and innovation, the project was divided into multiple stages and required extensive research, including the “trial and error” approach. The biggest challenges involved creating a solution that would work “in the wild”, without the need for any specific hardware and with minimum prerequisites from the user.

This raised several obstacles, including the complexity of calibrating the phone camera. Phone screens and cameras differ from model to model. Therefore, it’s hard to find a generic estimation method, especially since phone manufacturers don’t disclose the physical dimension of devices.

Since the user needs to have complete freedom to use their phone, there’s the challenge of making sure that the gaze estimation algorithm can be auto-calibrated in different models of smartphones. It is a difficult task given different angles, distances, and face detection capabilities. Our neural networks are trained on different faces and angles to get a result similar to regular use.

Due to the vast crowd-sourced data from the ClickWorker platform, the team needs to evaluate the data quality. An automatic framework was developed to filter out recordings that don’t satisfy basic quality metrics, like proper lighting, lack of blurring, etc.

Developers’ Insights – overcoming obstacles

One of the most prominent obstacles we are proud to have overcome is putting the product elements together. Both parts are strongly dependent on each other, and thanks to putting them together, we can quickly transfer the solution into the cloud.

We needed to overcome the barrier of using our data set to train the neural network. Especially in our case, when acquiring the training data was more complicated.

One of the most prominent barriers we overcame was collecting enough data for training the neural network models. Such data exists but isn’t available for commercial use and doesn’t always match the specific use case. That’s why we’ve done a lot of work to collect and analyse a large amount of data from hundreds of people to create our own training set.

First Steps Towards a Groundbreaking Eye-tracking Solution

The project began as a proof of concept with the goal of developing a groundbreaking eye-tracking solution. The initial milestones focused on establishing algorithm accuracy and creating foundational components for the solution.
- Milestone 1 (06.06-31.07.2022): Achieved a baseline level of algorithm accuracy.
- Milestone 2 (05.08-30.09.2022): Developed an SDK and a prototype application design.
- Milestone 3 (01.10-31.10.2022): Created a web-based eye-tracking service and improved the algorithm’s accuracy to meet initial criteria.
The next phase focused on continuous research and further refining the algorithm. Milestones included preparing an application (TDCA) for collecting large-scale data through the ClickWorker crowd-sourcing platform.
- Milestone 4: Improved the algorithm and prepared the application to collect more data.
- Milestone 5: Processed the data from ClickWorker, enhanced the TDCA features, and continued refining the algorithm’s accuracy.
The current focus is on conducting another round of testing, aiming to achieve a maximum of 1 cm average gaze estimation error across a diverse range of test subjects.

Outcomes and Further Steps

Even though the innovation threshold is set high for that project, the results are satisfying on both sides. The initial aim was to prepare the Proof of Concept. However, our partner keeps extending our work, as the results are good and the prospects promising.

We are gathering and processing more training data to improve the algorithm’s accuracy. We already exceeded the technological state-of-the-art and are continuing to work on achieving better accuracy and taking the next steps towards creating a working product.

Several elements of the processing pipeline developed by DAC.digital are currently considered for patent submission.

What eye square Said About the Project

It was the synergy between your coordination, efforts on our side, and all of your team effort. We are looking forward to the next project with you guys.

So, on this eye-tracking project in particular, we were very impressed with the competence of the team, the timelines, and how they are met. It’s very, good to work together, and you have a great project team on site and communication, as Michael mentioned, is very good.

So although the project is not yet over, we now have a prototype, and now the prototype has to become real live-action and has to be integrated in our technology. So these are the next steps on our journey. And we hope that DAC will continue to support us, as has before.

(…) we are confident in looking forward that we have a good partner on our side.

Contact us!

Send us an email: [email protected]