Fully Automated Manicure Robot Powered by Ultra-Precise Computer Vision
How using machine learning helped to build the first 5-step manicure machine in the world that requires no specialized labor.
In October 2021, DAC.digital partnered with an ambitious beauty tech startup from the United States who saw an opportunity to tackle the hospitality and personal beauty services industry, like spas, hotels, and hair salons. The aim was to make professional-level manicures, usually done by highly skilled manicure professionals, accessible in a wide range of hospitality businesses.
Company overview
- Name: 10beauty
- Branch: beauty, robotics, retail technology
- Origin: Burlington, Massachusetts, United States
- Website: 10beauty.co
How to Bring Automation to Skilled Beauty Services
With the aim and target audience specified in mind, our client developed a solid plan for building a completely automatic manicure robot that customers could install in waiting rooms, lobbies, etc.
For example, a customer could indulge in a quick manicure while waiting for spa treatments or as a pastime in a hotel, thus generating additional profit with significantly lower costs and no need for professional manicure artists. With the idea taking shape, it was time to find skilled experts to bring it to life.
👋 Hey You, What Are You Working On?
Share a few details about your challenge or idea, no matter how early or complex. Our expert team will reach out to offer our expertise in terms of feasibility, budget, and scope. Let’s shape a solution for you together!
We Were Tasked with Helping Beauty Hardware Achieve Precision and Accuracy with Custom CV
10beauty drafted a plan for a machine that can create a complete manicure in five easy steps.
- Old polish removal
- Cuticle treatment
- Nail shaping
- New polish application (including patterns)
- Drying
The process could be completely automated. The user would put their hand into the designated hole. Inside the robot, the cameras around the hand would track the hand’s position and guide the functional head that performed the manicure steps with meticulous precision.
The aim was to make it so accessible as to become an essential equipment for businesses. The project was highly complex and involved multiple areas of expertise, including software development, mobile application, hardware, machine learning, and computer vision algorithms, so 10beauty sought experts across various technologies. Our team’s work focused on developing the robot’s AI-powered computer vision capabilities.
One of the most considerable challenges was ensuring high accuracy when positioning the nail in 3D space. Our client requested an accuracy of 0.25mm. It meant meticulous calibration of all cameras and then precise detection of the nails in all views and angles. Additionally, we had to confirm the detection precision experimentally. It involved high-resolution laser scans of artificial hands, which we had to transform into the robot’s coordinate system according to the layout of the positioning markers.
DAC Team Proved to Be the Most Capable Off-Shore Computer Vision Experts
Finding available computer vision professionals in the U.S. job market is difficult because they usually engage with some businesses long-term. That’s why our client decided to look for offshore resources worldwide. DAC’s dedicated experts became a valuable asset to our clients’ teams. The company needed many specialists to realize every step of building a product at this complexity scale.
Landing the Best Team
One of the most complicated parts included designing a system to accurately detect the hand and positions of the nails. The search for top-tier computer vision experts for this part of the project ultimately brought our client to us. They deemed our engineers more than suitable for the task, thus starting the collaboration.
Establishing the Basics
During the discovery talks between both company leaders, we established the scope of work entrusted to our team. Ultimately, we deployed a team of skilled computer vision engineers whose initial job was to develop a 3D positioning system for hand detection based on stereo vision.
We Took Upon Additional Tasks that Helped the Startup Grow their Project Further
Leveraging our team’s expertise and thought leadership, we collaborated closely with the company’s tech lead, a seasoned computer vision engineer. His deep understanding of the project enhanced our strategic approach, allowing our specialists to apply their skills more effectively and efficiently.
Initially, our cooperation was to last three months and focus on creating only the basis for the nail 3D positioning system. However, upon seeing our engineers’ skills and their value, the client decided to extend our collaboration.
Our solid background and expertise in computer vision proved invaluable for our photometry and semantic segmentation work. We focused on stereovision and semantic segmentation as the project progressed. Our experts also prepared a continuous integration environment for the product, allowing the new features to be tested automatically to mitigate any accuracy regression.
Stereo Vision Helped us Fine Tune High-Precision Nail Detection
For such a complex robot to work correctly, it requires precise hand and nail detection. Our team displayed outstanding expertise and knowledge in computer vision and stood up to the challenge, delivering the desired results.
Since a complete manicure is a multistep process that works on nails that come in different shapes and sizes, one of the critical aspects of the device was high accuracy. The required threshold had to be no more than 0.25mm.
The device is equipped with five cameras (out of which four are used simultaneously, depending on the hand) that detect the position of the hand and fingernails inside and require high precision. Achieving that when working in small teams scattered around the world is challenging. We successfully overcame obstacles that included:
- Working in different timezones and coordinating work with a significant time difference.
- High technical precision requirements of the device – the accuracy of the fingernail point cloud had to be equal to or even higher than 250 microns (0.25mm).
- Since human hands come in all colors, shapes, and sizes, developing an algorithm that would be able to detect nails robustly, was a challenge and took a considerable amount of time to fine-tune.
We Recreated Testing Environments and Optimized Algorithms for Lightning-Fast Performance
An essential first step and a notable project milestone was recreating the exact testing environment since we couldn’t perform the tests on-site. There was also no prototype model for the device yet. One of our leading experts recreated the equipment using the same camera models the client’s team used, with additional items for positioning, calibration targets, lighting, and other components – just for testing and quality assurance.
Recreating this testing environment enabled us to compare the accuracy of algorithms against the ground truth 3D hand prints provided by the client. It made testing new features faster and ensured no regressions or errors while implementing new versions. Thus, we could determine the algorithm’s accuracy in detecting a hand or a finger and monitor the improvements.
After over a year of meticulous work, our engineers finished optimizing the algorithm responsible for stereovision, and it’s now working as intended, successfully wrapping up this part of the project.
Our work on the stereo vision produced a solid foundation for photometry – allowing the nail point cloud to be based on the efforts achieved in the 3D positioning. However, it wasn’t the end of the journey, as, during the project, our client also entrusted us with semantic segmentation – supporting the recognition and distinction of a nail from the rest of the finger. In simpler terms, it helps detect which pixels on the image captured by the cameras belong to the fingernail for more precise manicure application.
I’m proud of managing to speed up the algorithm in the later stage of the project. I was able to speed up the process severalfold so that the calculations only took several milliseconds per hand.
Through Semantic Segmentation, We Achieved an Even More Accurate Nail Detection
Every human hand and fingernail is different. We needed as precise detection as possible for each unique shape and size so the robot wouldn’t paint outside of the fingernail. That’s where the semantic segmentation comes into play.
Our client initially had their team responsible for this task. However, it ultimately required our expert’s assistance to perfect it. The main tasks revolved around maintaining and fine-tuning the existing semantic segmentation models for optimum accuracy.
The process is meticulous and takes time so that we can research, train, and test newer state-of-the-art segmentation architectures to best fit the project’s needs. Additionally, we invested time in migrating segmentation pipelines to more recent library versions to ensure up-to-date support and maintenance and, ultimately, better overall results.
This successful attempt resulted in an improved outcome of the overall nail segmentation process. We are working on fine-tuning the models to avoid minor errors that could snowball into considerable problems in the detection pipeline workflow.
The Technology Stack that Made the Stereovision Happen
The technology our computer vision experts used for developing the stereovision component was based on Python and included:
- OpenCV library for the core operations of the device 3D positioning.
- Open3D for point cloud data processing.
- Python notebooks (Jupyter Notebook) for quick experiments and visualizations.
Tools we Used for Perfecting the Semantic Segmentation
- Pytorch: a deep-learning framework for training the model.
- Additional wrapper libraries for Pytorch, including Open-mmlab’s mmsegmentation.
DAC.digital Team Composition
We have been dynamically adjusting the team composition to align with the needs of the project at different stages and comprised:
- Two stereovision engineers (initially four, including one supporting engineer aiding in the initial works).
- Two semantic segmentation experts, reduced to one after completing the most critical milestones.
- One project manager and coordinator.
A Successful Outcome, Resulting in Raising Another Round of Financing for the Beauty Startup
The development process is slowly nearing the finish line. On our end, we have completed the work on stereovision, achieving the desired result of 250-micron accuracy. With a few extra polishes on the semantic segmentation, our work on the project will be complete for this stage. It has been successfully integrated into the entire SW stack that controls the robot’s operation.
Now, our clients proudly present the robot during demos for investors, and a recent showcase has secured the company another financing round.
The client is already manufacturing the first working units of what will ultimately be called “The 10,” and they announced that they are actively preparing for launch with selected partners in the United States this year.
Got a Project in Mind? Let’s Discuss It.
Contact us!
Send us an email: [email protected]
