The software prototype allows users to create computer control gestures themselves and link them to any commands, from opening applications to managing interfaces.
Unlike existing recognition systems that offer a fixed set of movements, the new development makes it possible to create a personalized vocabulary of gestures. The user records several motion patterns, a computer vision-based system highlights key points of the hand from the video stream, and the KNN machine learning algorithm classifies the gesture and matches it with a given command. All data is stored locally on a device, which ensures privacy and independence from cloud services.
— The main difference between our development is that the user creates a dictionary of commands himself. This is not just recognition, but a full cycle: recording data, training a model, and executing a command,» explained Rafail Davar, a student at the SUAI Institute of Information Technology and Programming.
The technology can be used in accessibility systems for people with disabilities, in medicine and industry for contactless control, in educational and demonstration stands, presentations, as well as in AR/VR interfaces and «smart» spaces.
A working prototype has been created at the current stage. Local tests on a test set of gestures showed recognition accuracy of about 92,6%, while the balanced F1 metric reached 92,4%. In the near future, the project is planned to be presented at the JMLC Conference. In the future, the project may be submitted to business accelerators to assess its commercial potential.
