3D Virtual Environments for AIs
Train, test and interact in a safe environment.
A lot of the currently perceived problems with AIs, I believe, stem from fear of the unknown. What if the AI runs rampant? What if the autonomous driving car runs you over? And what if a super-intelligent AI somehow decides that we should all be replaced with paperclips? Virtual 3D environments provide a safe and convenient place to solve some of these problems and can also serve as research tools. In this article, I will provide a modest starting point.
The AI elephant in the room as of this writing is the advent of ChatGPT,
which has accelerated the AI schedule, as AGI now seems possible within our
Lifetime. Other more advanced AIs, such as super-intelligent and sentient
ones (my lifelong interest), are also being more seriously considered,
but I will delve more into this topic later in the article.
However, it's important to note that virtual environments do not magically
solve all AI-related problems, with the looming question of jobs being
replaced by AIs being the big one.— — — — — — — — — — — ⭐️ SUBSCRIBE TO MEDIUM ! ⭐️ — — — — — — — — — — —
So, where does one start? Well, most AIs these days have some sort of iterative process, where the results of a machine learning pipeline get scored against targets, with features/data being the inputs. These inputs and outputs usually have real-world equivalents, such as correct sentences from a text corpus, objects to be detected from annotated images, and streets to navigate from multiple computer vision tasks. However, they will usually not generalize outside of their manicured and purpose-built domain. Here’s, for instance, a common training log in Keras.
As AIs become more and more complex, these metrics start becoming less clear as interactions multiply, and multiple metrics need to be accounted for. This is fine for narrow interactions and AIs, but may not be ideal for future, more complex ones…
We also inhabit a 3d world that is for the most part absent in out current
AIs, a 3D Virtual one can serve as a nurturing place to train and test new
forms of AIs.A small convenient world
For the initial setup I opted for a basic room like the ones I’ve been using in my illustrations where I need to somehow convey the subject of an AI in a virtual world:
And thanks to the miracle of modern web development it’s fairly easy to get started and make something accesible/open source, a web based playground in this case:
You can find the repo here:
https://github.com/KenoLeon/3D-Virtual-AI-Environments
And the above room (done with the awesome three.js) here:
ROOM 01, BASIC, NO AI ( you can use keybord contorls wasd/qr
for movement/rotation )Autonomy
At this point, there is no AI behavior implemented for the cube in the test room. A simple example of such behavior would be to make the cube move in a Brownian motion pattern around the room, which is a rudimentary form of autonomy some single cell organisms exhibit:
ROOM 02, Brownian Motion
Notes: In nature the observed brownian motion in microscopic organisms
is usually a combination of their own propulsion via flagellum and
currents, so a mix of environment ( which we are not simulating here) and
self propulsion ( which we are), the distinction is interesting but better
not get bogged down with details right now...So it moves ! But it’s still far from being an AI, there’s some more interesting interactions with the environment though as the cube bounces when it collides with the walls.
What even is an AI ?
A shallow definition of AI might require our cube to perform a specific
task, but this would only qualify as a simple algorithm or narrow AI.
When it comes to creating general or sentient AIs, we don't yet know
how to do it (or if there are multiple ways), and when it comes to
Human-level AIs, we have some idea but still face significant challenges.
To define human-level AI, we need to specify the human tasks that
the AI should be able to perform, but some tasks like conversing with
humans don't easily apply here. So, we need to find some common ground...As this is a simple 3d Environment for a simple AI (for now), I opted to follow once more the biological path and give the cube a basic existence starting with a limited lifespan, rewards/goals and things to avoid.
Death, no taxes
AIs don’t necessarily need a finite lifespan (or taxes, yet ?) , but as a lot of what drives us is our innate drive to well, not die, it serves as a general goal and helps with iteration, another similar concept is health, which we can also incorporate :
ROOM 03, Health & Mortality
Notes: The cube has now a health-bar that ticks down to it's demise, and
once it runs out of time a cross appears on the floor on the last location
the cube was.
While this might seem morbid and useless at first, it is needed to more
generally replace things like loss functions, epochs ( the time element)
and can come in handy when training to measure performance against any
given task.Negative and Positive elements, Rewards and Punishment.
In the real world, both humans and animals are shaped by the interplay of positive and negative elements, which can be beneficial or harmful. This interplay is often abstracted into the direction that an AI algorithm is numerically performing, such as log loss approaching zero being seen as positive, and the opposite as negative. Additionally, rewards and punishments, whether internal or external, also derive from the environment and its elements. In 3D virtual environments, we can replicate this interplay by introducing positive and negative objects, such as nutrients and toxic substances, to extend or decrease the health of an AI-controlled object, respectively. This adds another layer of complexity to the AI’s decision-making process, allowing it to learn and adapt based on the consequences of its actions in the environment.
ROOM 04, Positive and Negative elements.
Here the cube loses health when it touches the red pyramid and increases
it when touching the green cross,all the while the health is ticking down
to it's death, while morbid this behavior should be familiar to us humans. A matter of perception.
Yet the cube is not an AI and sadly we will not create one here today as this is mostly the setup part and we are making a template of sorts, but we can add one last crucial element to this setup, the ability to sense the environment.
ROOM 05, Visual Perception.
As vision is our main way of perceiving the world it seems fitting to give
our cube the ability to visually perceive the world, the upper right window
gives you the point of view of the cube ( which now has a symbolic eye ).
This is just the start though, everything from perceiving objects through
virtual touch ( collisions in game development parlance) to very advanced
perceptions like proprioception is both possible and probably needed.Where’s the AI ?
I originally wanted to provide a few examples of AIs interacting with the room, solving puzzles or just surviving, maybe even connect something like chatGPT , OpenCV and other cutting edge AIs, but realistically each one of these cases needs a dedicated discussion, hopefully this serves as an introduction and you can see the possibilities and have a place to start though.
3D development is hard, double so for AIs !
You might have noticed that we have just created a Videogame of sorts
(a very basic and boring one). I bring this up because there's a lot of
bugs an improvements in the code (it runs at different frame rates on
different browsers, the health-bar can grow beyond it's bounds, etc,etc).
If you've ever played a videogame and stuck around for the credits, you
might have noticed there's usually hundreds if not thousands of people
involved, this modest code base and post took about 2 weeks, even with
the help of coding AIs, so yeah a lot of hard work is needed, but
the alternative ( training and testing AIs in the real world) is equally
difficult mainly due to hardware issues.
⚠️ A final comment on development is that when coding a videogame
you "cheat" a lot, so instead of writing a system that generates
behavior you usually script or hard code the behavior, when writing
for AIs we can't do this, which leads to even more challenges ! The future
While 3D virtual environments are nothing new, their use as testing beds for AIs is something of a new thing, basic and purpose built ones do exist in car, engineering and robotics companies to name a few and every ML pipeline could be considered a virtual AI environment.
Where a general 3D environment might prove useful is in the development and use of more complex AIs from the general kind and even theoretical ones like conscious/sentient ones, but as mentioned there is a clear alternative in the form of the real world, if anything I hope the virtual route serves to assuage and prevent some of the negative AI related unknowns that trouble us at times.
Thanks for reading, and hope you are equally excited about the future AI opportunities and challenges. We will have.
