EEG object recognition: Studies for criminal investigation and neuro-applications in social care

This paper describes the research endeavor aiming to design a brainwave processor and a sustainable machine learning model capable together of supplying an online informed guess of what a subject is seeing at a certain moment in time, using solely voltage data provided by scalp mounted electrodes (EEG signals). Brain activity processing is not a new topic: extensive research has been conducted over the last 50 years. However, the proposed solution brings novelty by its way of approaching the whole strategy, the greatest achievement of this research consisting of devising a composite brainwave processing–machine learning method capable to some extent of real-time detection of outstanding objects a person is viewing. Using, among others, preprocessing methods like DC offset removal, notch filtering, bandpass filtering, detrending, resampling and classifiers like SVM, neural networks, AdaBoost, nearest neighbors, an online prediction accuracy of 100% was obtained for a set of six colors and an offline prediction accuracy of 83.3% for a set of five scenes, for a single subject. The results of the study can be applied to the fields of neuro management and neuromarketing and other domains, a couple of possible scenarios being presented.


Introduction
*In the past few years, brain-computer interfaces (BCI) have proved to be significant candidates for human-machine interaction, requiring little to no physical input and providing fantastic potential. Brainwaves are so fascinatingly complex in the sense that they carry an impressive amount of information that we are only just beginning to uncover. True brainwave pattern recognition can pave the way for huge technology advancements able to dramatically reshape the world we live in. Imagine never having to turn up the thermostat (because a brain sensor senses non-optimal temperature), or never having to take out the phone to lookup an address in your agenda (because your GPS already knows where you are heading). Some say the speed of thought is the fastest mean of transportation in the universe; the superpowers" this seemingly "alien technology" can bring to our world; such progress may be closer to reality, than to science fiction. The greatest achievement of this research endeavor would be devising a composite brainwave processing-machine learning method capable to some extent of real-time detection of outstanding objects a person is viewing. For this reason, we will use the term of an outstanding object to define an element in a given scene that has the greatest effect on the subject's brainwave patterns. It could also be considered the most important object of the scene, from the viewer's perspective, but, without further evidence supporting this hypothesis and it not being in the scope of this research, the paper will further refer to it as the outstanding object. Fig. 1 presents this concept, any of the four elements present in the picture could be considered valid outputs for the algorithm.
As stated before, given the difficulty of the problem, sub-problems similar to the final problem were devised, to prove the feasibility of the concept and build the required knowledge. The present research tackles progressively more complicated challenges, highlighting each stage's results and lessons learned while keeping the main goal, in focus. This approach provides a steady buildup of solid foundations required to accomplish the goal. The aim is not to implement a fully functional and sustainable solution able to demonstrate the goal as is, since today's technology may not even be up to the challenge-the way the brain works is impossible to reliably model solely using data provided by sensors attached to the scalp. On the contrary, the aim is to explore the possibilities in this field of brainwave processing using machine learning and draw conclusions, only to be used as input for further research ahead. Therefore, the final embodied solution does not directly address the main goal, however, it hints that future improvements may lead to it.
The article is structured in 6 sections. The next section covers a couple of previous similar achievements in the field. Section 3 presents the equipment used for the experiments of this research and section 4 illustrates the step by step nature of this work, from initial proof of concept to the final python pipeline. The possible applications of the sustainable algorithm to various domains, including neuro-marketing, neuro-management, and criminal investigation were discussed in Section 5. Chapters 6 highlight the conclusions and future work.

Previous works
Brain activity processing is not a new topic: extensive research has been conducted over the last 50 years. Relevant achievements to the topic have been done given the vastness of the field and the endless amount of possibilities it provides. Each research is unique in its own way, making it tough to find similarities between them or to apply the same principles to another's particular case, as shall be shown further. Nevertheless, we gathered a set of worth-mentioning projects which acted as sources of inspiration or provided a starting point for the research under consideration. Nishimoto et al. (2011) provided impressive results in terms of decoding brain activity into videos. Suppes et al. (1997) proved also that advanced research in the field of neuroscience was taking place even before the year 2000. This study attempts to recognize seven different spoken words using a high-end research grade scalp sensor setup, very similar to the OpenBCI setup used in the research at hand. Following the previous paper, two years later, the same team of scientists published what is perhaps the most relevant article for this work investigating the invariance of brain-wave representations of simple visual images and their names (Suppes et al., 1999). Pasley et al. (2012) dug deep into the possibility of speech prediction based on brainwave data collected by sensors. The authors were able to reconstruct continuous auditory representations from measured neural signals.
The year 2012 adds another piece to the brainwave object recognition puzzle, this time using a basic, consumer-grade, single-sensor headset (Neurosky Mindwave). In this context, application of BCI technology for color prediction using brainwaves has been done by Angelovski et al. (2012). Based on a set of three colors, making use of the Haar-Wavelet transformation and machine learning algorithms like Rotation and Random Forests, prediction accuracies exceeding 50% have been gained. Building upon the previous papers and partially answering the open question, Mehta et al. (2014) attained 100% accuracy for a binary color classification experiment, applying machine learning techniques.

Set-up system
This section presents the equipment used to develop the final solution, emphasizing a summarized spec sheet for each headset. All consumer-grade EEG headsets function in the same fashion: They read brain activity using electrodes kept in contact with the scalp. Performance-related (fidelity) differences between those are caused by factors like Number of sensors/electrodes, type of sensors, interface (USB, Bluetooth) and ADC (analogto-digital) bits. Therefore, the algorithm will receive the same type of raw voltage data from the electrodes, regardless of the headset used or the number of sensors. Several types of equipment have been tested and used in different stages of the research:  Neurosky Mindwave Mobile can be considered as the cheapest and the most ubiquitous headset on the market -1 electrode, 12 ADC bits, 512Hz sampling rate, and Bluetooth connectivity. This headset was only used during the pre-initiation phase of the research; no results are based on data gathered by it.  Emotiv Insight provides a headset that uses dry sensors -5 electrodes +2 references, 10-20 placement, 14 ADC bits, 128Hz sampling rate and Bluetooth 4.0 connectivity. It is a more userfriendly and cheaper alternative to OpenBCI, but also less powerful. This headset was used for the biggest part of the research, serving its purpose extremely well. However, is not put to work in the final version of the experiment.  OpenBCI Ultracortex Mark IV is the latest and greatest system on the consumer market, a headset which performs on par with bulky and expensive professional solutions -35 sensor locations, 10-20 placement, 16 sensor-acquisition +2 references, 125-250Hz sampling rate and Radio (RFDuino) or Bluetooth Low Energy connectivity. This headset has been used for the last portion of the research, after waiting on the preorder for almost a year and a half (Fig. 2).

Fig. 2:
Fully assembled research kit worn by subject (Ear clip= Reference electrode-one for each ear) The provided API tools available at the time were used for the experiments in this paper. Proof of reliability of the tools is the vast amount of already published papers using the same equipment. Further, Frey (2016) demonstrated the capabilities of the OpenBCI headset as being on par with g. USBamp, a professional, established research-grade equipment from g.tec (https://www.gtec.at/).

Research journey and experimental results
A set of experiments have been conducted as it will be further presented. Different methods were used in order to find the best one, the results will be shown and discussed for all of them to emphasize our findings.

AdaBoost experiments
AdaBoost is a machine learning meta-algorithm and is considered to be the first practical boosting algorithm proposed by Freund and Schapire (1996). AdaBoost is about classification problems and the purpose of it is to convert a set of weak classifiers into a strong one. This technique was applied in our first experiments. In the first step of the experiments, the Emotiv Insight headset has been used. The tools and technologies involved in the first experiments are:  Emotiv SDK Community edition  Emotiv Insight SDK Lite v1.1.1.0 (no longer in existence today)  A C++ program interfacing with Emotiv SDK's API to gather live data samples from the headset  IrfanView command line API to display full screen images  Matlab R2016a  Balu Toolbox Matlab An offline approach has been chosen for this implementation; therefore, it can be divided into two separate phases as follows:

Data acquisition via the C++ program
Step1: Read the configuration file with the line format of {second of image display, image path, threshold/label associated with the image}. Each of the 10 images has been displayed for 10 seconds, resulting in a 100 second total duration for the experiment.
Step 2: Start displaying the set of images while logging data into a CSV file. Each row of the output file, besides the 25 data points provided by the electrodes, the timestamp and threshold are included.

Training and testing via the Matlab program using the splitting technique
Step 1: Given the CSV file from the previous step, the data is split into test and train data by a 1/3 ratio (2/3 train and 1/3 test).
Step 3: Test the trained classifier over the test data and compute precision.
The total execution time of the Matlab app is around 6 minutes (48k rows of data). In this version of the implementation, no preprocessing methods have been applied (see Fig. 3 for the image set used for this method). The prediction has been compared using different combinations of channels. Fig. 4 presents the AdaBoost M. 2 method overview. As Fig. 5 illustrates the prediction accuracy is low, a little bit above random guessing. The alpha channel test performs marginally better; further research will be conducted in this direction.
As expected, the data is way too noisy (muscular contractions, environmental noise, useless extra data, etc.) to be used as is (i.e. with no preprocessing). Isolating data from the visual cortex electrode, assuming that other sensors do not gather useful info, does not make any difference. It also seems that the raw data has to be processed in advance in order to remove the noise. Even though the headset performs some noise fine-tuning by default, it looks like this does not suffice. Consequently, more research has to be done in order to develop a strategy that will successfully elaborate the signals in the interest of correctly determining the brain waves pattern and successfully "read the user's mind".

SVM experiments
A Support Vector Machine (SVM) is a supervised machine learning algorithm that can be used for both classification and regression purposes. This technique has been successfully used in text classification tasks, sentiment analysis, spam detection, image recognition, areas of handwritten digit recognition, such as postal automation services. The equipment, the tools and the method we used, in this case, are further described. For this set of experiments, the Emotiv Insight headset has been used as well. The same tooling as in the AdaBoost M. 2 experiment, except for the Balu Toolbox, was considered. Again, the method used was fairly similar to the AdaBoost M. 2 experiment, with the following differences (an overview of the method is illustrated in Fig. 6):  Instead of 10 categories of images, two were used: beach and graveyard  Instead of a single image per category, there were 60 images this time  Instead of a single subject being tested, 4 were present this time  Instead of showing an image for 10s, the chosen duration for an image was 5s  An extra step of eliminating consecutive duplicate data points was introduced, which reduced the amount of data by a factor of roughly 250  Instead of using an AdaBoost classifier, SVM was selected Fig. 7, Fig. 8 and Fig. 9 illustrate the precision accuracy for different subjects, using different combinations of filtered bands and channels. The best results varied between 82% and 100% overall subjects. For testing the proposed solution, 4 subjects have been used as volunteers for our experiments.  This experiment's results were encouraging, given the high precision achieved and the fact that the highest influence on the precision seems to root from the Pz electrode. Even though the experiment was pretty basic, using just two categories and no preprocessing at all, it proves an extremely valuable hypothesis, also presented in the literature (Mehta et al., 2014). Thus the parietal and occipital lobepositioned electrodes (i.e., Pz) carry the relevant information in experiments based on visual stimuli since the visual cortex is located in the occipital lobe.  The experiment has been repeated 10 times for each of the 4 test subjects (Alex, Diana, Andrei, Daniel). The plotted values constitute a mean accuracy for predicting the displayed image overall 10 repetitions.

Final solution
For these experiments, OpenBCI Ultracortex Mark IV headset has been used. We also involved python for the machine learning and signal processing implementation, due to its superior documentation, speed of execution, number of features, libraries, syntax and "open-sources", compared to Matlab. Libraries used are NumPy-for signal processing mostly, SciPy, Scikit-Learn-for machine learning, Pandas-for handling data frames and Matplotlib-for plotting.
Python was also used for data acquisition and we wrote a small tool designed to make the experimentation smoother. The EEG Unified Logger has a Tkinter GUI and is compatible with both the Emotiv Insight and the final version of the experiment.
OpenBCI Ultracortex Mark IV does the job of the text configuration file used in the previous experiment, but using a GUI-receives the image path, image display interval and subject name as input and outputs a CSV file containing the data.
The data transferred between the headset's board and Python has been provided by OpenBCI's NodeJS SDK, which is the most stable and well-maintained solution provided by the manufacturer. The Python script received messages through a ZeroMQ TCP port from the NodeJS program. In the process of developing the solution, we ran into a problem in the SDK, which only surfaced for the 16-channel implementation; we promptly submitted a pull request to the repo fixing the issue (Fig. 11). In this approach, we proceed to reposition the electrodes. Given the conclusions of the last implementation, we have considered the decision to change the default location of the 16 electrodes above the parietal and occipital lobes (Fig. 12). Electrodes 1 and 2 are flat-they cannot be placed over hair, hence their unchanged position. The complexity of this version has drastically increased, compared to previous ones. After many iterations, testing different combinations and adjusting parameters, the following sequence of operations has proved to be the most efficient for our type of experiment:  Preprocessing implies the following steps: -Remove DC offset -Resample (100Hz) -Discard noisy channels (ratio= 1.1) -Detrend -Bandpass filter (1, 50Hz) -Min-Max scaling (0, 1)  Feature selection is realized by Principle Component Analysis technique (2 components)  Training implies a soft voting classifier: -Nearest neighbors (100 neighbors); weight= 3 -AdaBoost (decision tree estimator); weight= 2 -Random forest (10 trees, entropy criterion); weight= 1 All of the following experiments respect the same scheme described in Fig. 13:  They do not use the splitting approach, instead, they use separate recordings of identical length for training and testing, resulting in a ratio of 0.5  They are executed on data from a single subject, using the sequence of operations previously described  Prediction is evaluated based on the mean category prediction.
Six colors are used as the set of stimuli: Black, white, blue, red, green, yellow. The experiment has been repeated 10 times with different durations for stimuli (between 1 and 20 seconds), 2 seconds resulting in the highest precision.
In the offline simple 3D scenes experiment, five simple 3D scenes were used as the set of stimuli: lake, violin, people, car, and cows. Similar to the previous one, the experiment has been repeated 10 times with different durations for stimuli, 3 seconds resulting in the highest precision.
For the online colors experiment, the same 6 colors are used as stimuli as in the offline colors experiments. It also follows the same principle as the offline version, the only difference being that the two sessions are acquisition live, one after the other and that the images are shown in a random order every time.  A couple of random spikes, but the main trend follows the expected prediction  Computing the mean category precision always produces 100% accuracy Fig. 15 plots the raw precision of one iteration of the offline 3d scenes experiment, as in the previous case, this experiment also produces similar output again and again:  Much more spikes, but still a discernable trend following the expected prediction  Computing the mean category precision almost always produces 83.3% precision (one wrong prediction), only rarely falling to 66.6% (two wrong predictions).  In the experiments from this paper, prediction accuracy refers to the number of correct guesses of the scene/color the subject is seeing.
The (final) preprocessing, feature selection and training method described in this research proves to be highly efficient in classifying visual-stimuli-based brainwaves. The present research has also combined methods, built on conclusions provided by many works in the literature and even answered open questions of others. As a conclusion, a single test subject has participated in this experiment (Alex), but it (the experiment) has been repeated 50+ times yielding identical precision results, given the smoothing.

Applying brainwave object recognition to neuro-management and marketing and other domains
The most promising everyday life applications of outstanding object recognition are centered on the idea of identifying points of interest in a scene.

Neuro-marketing and neuro-management
Perhaps one of the most promising application perspectives lies in the neural management and marketing domains. The present method can be viewed as an extractor of the subconscious manifestation of human decision-making behavior. By outputting the point of interest in a complex image, we can make educated guesses as to what might be of interest to a particular person in an image.
Particularly, an algorithm can be designed around the current platform that can learn valuable information about a subject's tastes, preferences, and inclinations without providing it with conscious input from the subject. This can prove to be an invaluable neuro-marketing technique that can be applied to many domains.

UX design
This approach regards user experience design and refers to the process design to create products that provide meaningful and relevant experiences to users. The method presented in this research can be applied on multiple subjects when using user interface for the first time, be it a physical one (the interior of a vehicle, a house) or a virtual one (a website, a mobile application). It can provide the answer to questions like: "Which is the main point of interest of the average user?" or "where is the user tempted to look or go?".

Criminal investigation
A possible adaptation of the algorithm could determine whether or not a subject has seen or not a certain or object or any other information related to a crime or other type of event. It can also be used on witnesses to extract details they may otherwise not remember consciously.

Medicine (psychology, psychiatry)/social care
Being able to pinpoint outstanding objects in pictures in another way than by simply tracking eye movements can prove to be a very useful tool in diagnosing and treating individuals with certain conditions or disorders. Any field that can benefit from identifying points of interest inside a scene or a picture is a potential candidate for applying this technology.

Conclusions and future work
As is the case with any scientific work, the future seems to hold much more than the present and the past. A set of steps already planned in perspective are described below:  Gather data from more subjects and rerun the current experiments to confirm the result. The main deficit of the final implementation has been the lack of test subjects, a result of the small room for headset size adjustments of the OpenBCI Ultracortex Mark IV. Certifying the exposed results with more subjects is definitely the first mandatory step forward.
 Devise other, perhaps more complex and sustainable experiments. Proving that the method described works for more types of experiments is crucial for validating its scientific value.  Further, improve prediction accuracy. A short analysis of the past shows us that investing more time in this subject produces progressively better results. Pursuing this assumption, further research should also bring improvements.  Strive to truly achieve the goal stated in the introduction. Even though the goal stated in the introduction chapter is closer to science fiction than to reality, at least so it seems with the current technology, there is always room for growth. Who knows what the near future could bring us?
The acquisition, preprocessing, dimensionality reduction, real-time training and testing pipeline described in the present research prove itself as an efficient combination for predicting individual scenes and colors, achieving an online prediction accuracy of 100% for a set of six colors and an offline prediction accuracy 83.3% for a set of five scenes.
The incremental nature of the research has played a crucial role in the final outcome, contributing with knowledge and determining decisions like switching to Python and repositioning electrodes, all of which leading to a successful and sustainable result. In order to achieve its goal of pinpointing outstanding objects presented with new scenes, it requires further research and development. The impressive results of this method and promising potential of the technology will definitely fuel the next steps in this endeavor.
Moreover, given the complexity of the human brain and its inner processes, being able to extract a simple output based on an input may be a big step in inviting more machine learning and artificial intelligence research to take place in this area.

Funding
Financially supporting body(s): This work has been funded by University Politehnica of Bucharest, through the "ARUT Grants" Program, UPB-GNaC. Identifier: GNaC 2018, Contract: 16/06.02.2019-RM-CYBERSEC-Risk management at the organization and individual level in the context of cyber security.