WORK PROJECTS (AISOLVE UK)
Aphasia App (Brain Exercise app)
This is a work in progress. I am the lead developer on this project. We are creating a brain exercise app for mentally challenged patients on a client's request.
The app is being developed in Flutter and is targeted for both iOS and Android. My responsibilities include
Developing the complete frontend app using Flutter for iOS and Android
Develop a serverless backend for storing players' data and game content
Record and persist analytics from the app to the cloud.
App includes several puzzles and games that require users to write directly on screen using a stylus, therefore hand writing recognition will be needed. Below are some of the interesting and challenging problems I am solving during the implementation.​
HAND WRITING RECOGNITION USING AI (MACHINE LEARNING)
There are quite a few AI based hand writing recognition cloud services that can be used for this. The problem with a cloud service is that it adds latency, especially when the end users are mentally challenged even a slight delay in responsiveness can add to their frustration. So it was evident that we could not use a ready made cloud service. We needed an AI based solution that could be run on-device and on both iOS and Android. I had not done any AI work before so didn't have any idea where to begin.
Tensorflow
From my preliminary research Tensorflow framework sounded the right tool to tackle this problem since it allowed you to run AI models on many platforms including mobile. Next was creating the model and training it with hand written data. We needed all English alphabets and digits to be recognizable. Luckily for me such a dataset already existed. I found emnist to be exactly what I wanted. Tensorflow also offered a good collection of models ready made. There was one image classification model for hand written digits that could be trained to recognize hand written alphabets too. I used it to train on the emnist dataset. There was some learning needed to write and understand Python scripts and Tensorflow APIs plus basic understanding of what machine learning and deep learning is and how neural networks work. It was an exciting challenge for me. It took me roughly a month to do that.
DEPLOYING TENSORFLOW MODEL TO FLUTTER MOBILE APP
My model was ready and it could recognize both alphabets and digits with a reasonable accuracy. Next challenge was to deploy it to mobile app. The model was in .tf format the default Tensorflow format. To run it on mobile in realtime I converted it to .tflite format, an optimized lite weight format for running inference on mobile platforms. But, there was another issue, although Tensorflow lite was available on both iOS and Android but I needed it for Flutter which meant I implement the model hosting bits on both platforms as plugins. This could be done but seemed like to be challenge to manage in future. Plus I wanted a deployment pipeline where I could collect more data when the app is in use and train on this data to improve model accuracy overtime and then deploy automatically on both platforms. This is where Firebase ML Kit came to the rescue. I could simply deploy the tflite model on Firebase and it would take care of deploying it to all the app clients automatically and I could update the model overtime without having to worry about updating the apps. On app end to integrate I used Firebase plugins for Flutter and everything came together smoothly to create the magic. In the example above you can see one of the puzzle games (Tic-Tac-Toe) where user can compete with computer by hand-writing 'X' in the custom widget that I created. The widget was created using the CustomPaint widget and implementing the CustomPainter interface that allows to directly render on the widget's canvas.