Brain Mapping

Part 1 : 2D

In collaboration with:

Charlotte Maschke

Biosignal Interaction and Personhood Technology Lab.

Master’s student in the Integrated Program in Neuroscience at McGill University, she applies techniques from the area of artificial intelligence to data, recorded from the human brain. Her work gravitates around the concept of consciousness and how we can quantify it.

The search for neural correlates of consciousness

Imagine you are in bed about to sleep, but then you hear someone talking. Scared, you want to open your eyes and lean forward to see who is there, but your body is not moving. You try to call for help but cannot voice any sound. You are helplessly stuck with your physical body with no way of communicating to the outside world. Days, months then years past, you hear your loved ones discussing the possibility to let you go. You are stuck with yourself and cannot care for your own needs.

This terrifying scenario illustrates the contrast between consciousness and responsiveness or in other words: the potential of a conscious mind to be trapped inside an unresponsive body. For patients in a disorder of consciousness, this scenario might be reality. A “disorder of consciousness” is the general term to describe patients in a coma, vegetative state or minimally conscious state which are mostly caused by traumatic brain injury. While most of these patients’ vital functions often recover, the communication between the brain and the body might be irreparable - resulting in a conscious mind trapped inside a body that is unable to move. The most relevant question for the next of kin - whether or not the patient is consciously aware, perceiving, feeling and thinking - is one of the hardest questions in recent neuroscience research. How to determine whether someone is conscious if they are not able to communicate? This project will engage the theories of consciousness and how to visualise brain states taking inspiration from the unicellular organism that is slime mould.

Slime mould model exploration with functional connectivity as an attractive force towards the brain areas. The areas are represented by the starting points of the slime moulds.

Exploratory 3D flock like behaviours representation of the data

The neuroscientist Adrian Owen first demonstrated that it is possible to communicate with patients who suffer from consciousness disorders and appears to be physically unresponsive. The idea is simple: if a patient’s body is not able to answer your question, you would need to learn to read a patient’s thoughts. Patients were asked to imagine playing tennis or walking around their house, and the scientists recorded their brain activity using an fMRI scanner. Based on the knowledge of which specific parts of the brain become active when the patient imagines playing tennis (or any other kinds of arm movement) the scientists were able to record the patient's answer to simple yes or no questions. This research demonstrated for the first time the possibility to communicate with physically unresponsive patients. It raises many ethical concerns as a result and proves that the scenario illustrated at the beginning of this article is more than just a thought experiment – it could be the actual reality for many patients with such conditions.

In 2006, Owen's work opened the floodgate to research on consciousness. Still, consciousness remains the biggest mysteries in today’s science and continue to challenge us to go beyond a mechanistic understanding of the brain and consider the fundamental ethical and philosophical questions on the identity of humankind.

The Theories of Consciousness:

David Chalmers narrowed down the questions encompassing the consciousness research to two problems - the easy one and the hard one. The former implies computational or neural mechanisms explanation for behaviours, as for the hard question of why all that is associated with subjective experience (“Hard Problem of Consciousness — David Chalmers”). As it is, the problem is open and a lot of great minds are debating it, the main theories defended are dualism, materialism and panpsychism.

Dualism

People who embrace dualism try to prove that consciousness is an entity entirely separated from the body. They attempt to address the hard problem of mind and body relations with quantum mechanics.

Materialism

On the complete opposite is the materialist point of view which holds the belief that what you see is what you get, and that everything can be explained in terms of data. As for consciousness, it is made of higher level brain activity caused by lower level processes. “Materialist have the theoretical obligation to explain how subjective qualities could be accounted for in terms of objective quantities” (Goff p171). In this context, the project data is based on EEG readings that are electrical signals extracted from the brain and could be said to represent consciousness.

Panpsychism

And in the middle, there is panpsychism, which implies that all things are conscious, to some level, and the combination of many particles or cells makes for a higher state of consciousness, where they have to solve the combination problem. An example of possible conscious state at the most basic level is slime mould which are seemingly unconscious but can learn (Moskvitch).

Consciousness: Our Research:

Despite many schools of thoughts, opinions and theories about consciousness, most of the recent neuroscience community agrees on one fact: the inexistence of “the consciousness area”. Scientists tried for many years to locate one point in the brain that is responsible for consciousness, similar to how they locate the area for vision or area for auditory perception. However, it turned out to be much harder to locate consciousness. Instead of an accumulation of cells or a particular place in the brain, consciousness is nowadays assumed to emerge from highly complex tempo-spatial computation patterns within the whole brain. However, the high dimensionality of such patterns underscore the limitation of classic neuroscientific methods. This is where artistic approach could come into the play.

Early data visualization is based on the Boid algorithm by Ben Eater. Using the Cartesian x and y plane, each dot represents a brain area. The color pallet indicates two experimental states, with blue referring to the anesthetized and orange to the conscious brain. As for the data, it is represented by the items moving in space and interacting with each other, giving an early sketch of a small data set with 16,430 data points.

The project offers an alternative way of looking at brain activity through the methods of art. Using AI (artificial intelligence) to reduce the dimensionality of the data and create visualisations of spatio-temporal activity patterns in the human brain. By doing so, it puts special emphasis on the crucial role of time and dynamics . Thus this project takes a new perspective on the brain’s dynamic computation by finding ways inspired by the to visualize high dimensional dynamic patterns emerging from neural activity.

Artistic Research:

The aim is to capture the patterns of the brain’s dynamic principles in an artistic representation. The project visually characterises a healthy brain’s dynamic pattern, then compares it to brain patterns in altered states of consciousness, such as anesthesia or disorders of consciousness. Thus, challenge the question of whether these patterns can really be the physical principle of human consciousness.

Here we have the first experimentation to see the limits of data agent I can have on-screen. This is due to every agent communicating their position and velocity to each other to calculate coherence, separation, alignment from their visual range. To improve the performance we could include spatial acceleration structures.

The main problem is the high dimensionality of the data which contradicts most intuitive visualization approaches. Every single one of the 105 electrodes is able to record the brain activity 250 times per second . Over a recording period of five minutes, this results in about 7,875,000 data points. To stay as far away from any conventional data visualization, the project draws inspiration from nature. A first exploration was inspired by Theodoros Papatheodorou and Refik Anadol, to represent each data point as an agent moving in space where their speed is changing depending on the EEG reading. This approach resulted in a really interesting fluid bird-flock like behaviour, representing the complexity and dynamics of brain activity in a rather unconventional way. But this model quickly showed its limitations as we cannot see the temporal change or the connections between different areas. In the next steps, we increasingly focused on the brain’s functional connectivities (i.e. the communication between brain areas) as this approach has shown a lot of promise in recent research of consciousness. It especially focuses on evolution through time. As a result, we found inspirations in trees, leafs, thunder and slime mould as they all share similar patterns and behaviours.

Slime Mould:

To visualise the data, a slime mould or Physarum Polycephalum model was created as a visual inspiration, as it presents some fascinating behaviour, renowned for its ability to network, to learn, to survive adversity, and to navigate complex systems in an efficient and equitable way. It has been used to accurately map routes to major cities as well as agglomeration of dark matter, thus, it was used on a smaller scale to map the connectivity between the different brain areas.

Early slime mould model exploration, without data.

To better reproduce an accurate model of the slime mould, a sample was obtained for close observation and deep understanding its behavior. This further led to exploration ants like activity or stigmergy , as the pheromone trail they leave behind in order to find their way to food and back to the nest.

First we can observe that it spreads by attempting to cover most of the surface with fractal paths in search for food. Those paths are then connected to a larger one for nutrients, very much like a tree or a neuron. What get interesting is when the paths make contact with each other they start to merge, since it is a single cell organism. If it spreads in a direction for too long without finding food (or new information), that section is slowly cut off from nutrients and disappears.

Boris, the slime mould.

Multiple slime mould models, interaction test

Looking closely at active slime mould, directional transfer of liquid happens to promote growth and expansion. If they find food, the shortest paths connecting the two pieces of food will expand in diameter due to the amount of nutrients being passed through. For data visualization, those conducts can be used. With brain areas for food, the fluid as the direction of communication, and the size of the paths for the intensity of connection giving an indication of time, the temporal brain activity can be mapped.

Code:

Here, in the terminal, we have an example of the data pipeline problem I am still fixing. It affect the visuals where particles drastically change speed.

The code is based on the stigmergy model by Jeff Jones (“Characteristics of pattern formation and evolution in approximations of physarum transport networks”) and presented in my class with Andy lomas. To gather the data and use it, a pipeline of multiple programming languages is utilised, starting by collecting the data from the EEG, it is then cleaned and tidied using Python as the electrodes are agglomerated to the area they scan (from 100 electrodes to 10 areas) and their connectivity is extracted. Then, using R, every area gets a dataset composed of their connectivity, this is the “easy” technique found to deploy the connectivity by area in the Opneframeworks sketch. Where each area has a dataset with its connectivity with all the other areas and the particles of each have their own area they are attracted to depending on the level of connectivity between their starting area and target area.

Improvement:

Talking with Josef Luis Plez, a Master’s student in Mathematics making beautiful creative code, and Andy Lomas we agreed that the project needs to be using the GPU instead of the CPU. Making the transition means that more data can be visualized, since the CPU works like a single pipe where all the agents are waiting to pass one by one through it, the GPU operates as multiple small pipes where the information is all computed at the same time. This technique enables Josef to have up to 10,000,00 particles in real time and at once without experiencing frame drops. Another approach to take is to convert the work into voxel, thus making an efficient three dimensional rendering. Overall, great improvement can come from using the GPU for the visual graphics while at the same time increasing the amount of data that can be accessed. This could make possible representing two brain states at the same time and offering an alternative and informative way to explore brain data.

Conclusion:

GPU VS CPU data processing visualisation,

both taking from The Book of Shaders

Comparison of brain states baseline, in orange, and anesthesia, in blue of a healthy patient. There is obviously more activity under anesthesia but it is too soon to say for sure due to the data pipeline problem, which is further demonstrated in the left video as, after 30 second the connectivity change and displays intense movement which should not happen since the data range from 0 to 1.

This project tackles the challenging problem of rendering consciousness and big data, it looked at the materialistic approach to gather data as a base for consciousness and panpsychism with the learning unicellular slime mould for a visual interpretation of the data. I made renders that compare brain states and some ways to visualize consciousness. After this exploration, the aim is to find the best setting and then expand the model by adding more data using shaders and the GPU and translating the project in 4D with voxels. Hopefully, as the project advances the more data can be used, the more information can be extracted and the more capable the system will be.

Bibliography:

Blackmore, Susan. Conversations on Consciousness: What the Best Minds Think about the Brain, Free Will, and What It Means to Be Human. Oxford University Press, 2007.

Chalmers, David J. “Facing Up to the Problem of Consciousness.” Journal of Consciousness Studies, 1995, http://consc.net/papers/facing.html. Accessed 22 04 2021.

“Characteristics of pattern formation and evolution in approximations of physarum transport networks.” Artificial Life, 01 03 2010, https://uwe-repository.worktribe.com/output/980579.

Goff, Philip. Galileo's Error Foundations for a New Science of Consciousness. Ebury Publishing, 2019.

“Hard Problem of Consciousness — David Chalmers.” Serious Science Youtube, 05 07 2016,

https://www.youtube.com/watch?v=C5DfnIjZPGw&ab_channel=SeriousScience. Accessed 22 04 2022.

Luis Plez, Josef. “Josef Luis Plez.” Josef Luis Plez, 2021, https://josefluispelz.com/. Accessed 14 04 2021.

Moskvitch, Katia. “Slime Molds Remember — but Do They Learn?” Quanta Magazine, 09 07 2018,

https://www.quantamagazine.org/slime-molds-remember-but-do-they-learn-20180709/.

Owen, Adrian. Into the Gray Zone: A Neuroscientist Explores the Border Between Life and Death. Faber & Faber, 2017.

Qualcomn. “AI Debate: How far can the AI revolution go?” youtube, 2018, https://www.youtube.com/watch?v=J_KvCok99io&ab_channel=Qualcomm. Accessed 16 April 2021.

Sacks, Oliver. The River of Consciousness. United States, Vintage Books, 2017.

Sage, Jenson. “Physarum.” Sage Jenson, 02 2019, https://sagejenson.com/physarum. Accessed 10 04 2021.

Vivo, Patricio Gonzalez. The Book of Shaders. Patricio Gonzalez Vivo, 2015. The Book of Shaders, https://thebookofshaders.com/01/.