[author] [author_image timthumb=’on’][/author_image] [author_info]Адв. Гинка Христова е член на Софийската адвокатска колегия (САК), както и член на Дисциплинарния съд към САК до март 2021г. Към настоящия момент е член на Американската адвокатска колегия и на САК, тя е специалист по дигитално право, работи повече от 10 години в областта на авторското право, като от 4 години е асоцииран преподавател към университета Харвард и организира курса CopyrighX, воден от проф. Уилям Фишър за България. Магистър по право, адв. Христова, продължава обучението си в Световната органозация по интелектиална собственост и университета Харвард в областта на интелектуалната собственост и Интернет правото. Като председател на Международен правен център -ILAC, адв. Христова работи в областта на защита на личните данни с Министерство на правосъдието и редица други публични и частни организации, член е на Работна група към Министерство на културата за изработване на ЗИД на ЗАПСП във връзка с имплементиране на Директива 2019-790 и Работна група по транспонирането на Директива 2019-789. Автор е на монографията на френски език „Киберпрестъпността-проблеми и решения“ и по настоящем е кандидат докторант към УНИБИТ. [/author_info] [/author]
AI & VIDEOGAMES, A LONG-LASTING RELATIONSHIP
- The early years (1980’s)
The gaming industry[1] has been pioneering the use of AI since the beginning of the personal computer age. The mass-adoption by households of the first personal computers with graphic capabilities (IBM PC and Apple II) coincides with the emergence of the video games industry, which was taking advantage of this booming market by transposing one of the oldest human activities, playing games, onto this brand-new medium.
Within a couple of years (1981-1983), due to a spectacularly large and fast adhesion from the public to this new proposition[2], this had become such a success that it gave birth, in Europe, to a whole subsegment of the PC industry: the games-oriented personal computers (Amstrad, Oric), while simultaneously a brand-new industry branched out of the nascent personal computer industry: the industry of pocket games[3] and personal gaming consoles (Sega, Nintendo, Atari).
In the western world ownership of a game-oriented pc or a gaming console became as commonplace as ownership of a second car, and gaming arcades mushroomed in the wake of the rapid development of shopping malls. Soon, from coast to coast in Northern America, all across western Europe and in Japan, every teenager and young adult was playing video games.
This, in turn led to an explosion of the number of gaming studios, which were founded and staffed by self-taught individuals[4] who were keen to adopt “out-of-the-box” approaches when addressing game design related issues such as player interface and NPC[5] behavior.
As a result, the 1980’s decade will see a flurry of gaming titles pioneering the use of what could be considered as ancestors of Large Language Models and Expert Systems.
As early as 1982, “The Hobbit” offers to PC players an interface in natural language (English) in which the players are presented with a text describing their environment and featuring keywords (available items, characters) and are prompted to enter any sentence of their choice (up to 255 characters) to describe the actions they want to perform.
The underlying proprietary text interpretation technology then analyzes the player’s prompt thanks to a dictionary supporting a few hundred English verbs and adjectives, in order to determine the intention of the player and apply the corresponding action.
Shooters are evolving, and iconic titles such as Castle Wolfenstein and Doom innovate by breaking-off from the then current standard NPC behavior.
By that time, most of the shooters were programming the NPCs by determining patrol cycles (each cycle corresponding to a series of movement on the map) and breaking from these cycles only in order to respond to specific player inputs (shooting when detecting the player or returning fire when shot at by the player) and by entering into a pre-determined action cycle (usually standing in a predetermined place while shooting at the player, and reaching a predetermined covered spot when shot at, and alternating between these two based upon the players actions).
Shooters such as Doom and Castle Wolfenstein lead the way to the popularization of the use of decision trees amongst game developers. The NPC targets now reacts to the unfolding of events in a much more complex way, adapting their strategy at every step based upon the player’s actions and being able to switch their strategy at any moment instead of having to go through a full action cycle before being able to perform a change.
- 3D open worlds: a revolution (mid 1990’s – mid 2000’s)
The continuous increase in computing power of the personal computers combines with the decrease of the price of RAM and the advent of high-resolution screens to provide the gaming industry with the advanced graphical capabilities it needs to generate more immersive and realistic game worlds. 2D becomes a thing of the past.
At the same time, the gamers’ expectations are evolving, triggered by the emergence of innovative games in the late 1980’s such as the groundbreaking Legend of Zelda (1986).
With an ever-increasing time spent by individual at playing games, there is a strong demand for games with higher lifetime (gradually increasing from the initial couple of hours to today’s standard of 70 hours+ of gameplay for a RPG/Adventure game for an example), replay ability[6] and non-linear gameplay[7].
The industry response to these expectations has been the development of 3D Open world game engines, which provide a realistic environment in which each level can be entirely explored in three dimensions and interacted freely with by the player[8].
The sheer size of these worlds quickly made it impossible to entirely design them by hand[9], paving the way to the wide adoption of procedural generation of environments.
All of this in turn generated a whole new level of constraints for the game developers:
- First, contrary to a linear game which allows full predictability of the players’ sequence of actions which are entirely constrained by the level design, non-linear games imply that the players’ sequence of actions in a given level is entirely unpredictable from the level design.
- Second, the large number of NPCs required to populate an open world, and the absence of relevant spatial limitation allowing to limit efficiently the radius in which a given NPC has to move[10] prohibits any possibility to restitute realistic NPC interactions with the terrain map through programming, as it used to be the case in linear 2D (or even 3D) level design.
- Third, the massive recourse to environment design through procedural generation, which implies that the game developers do not have full knowledge of the environment which is created unless they examine it in detail manually[11], made the two precedent problems even more acute.
The critical issue of pathfinding, which is the computation performed by the game engine of the path which can be taken by a NPC to go from point A to point B on the map so that the NPC’s movement appears realistic if observed by the player has been solved by multiple ways, including the use of neural networks in order to determine the optimal path.
Expert systems have also been used in order to subtly introduce limitations to the exploration of the game world by the players without appearing as a breach of the non-linearity of the gameplay. For an example, a trick commonly used by level designers is to discourage the players from exploring a given zone of the map (for an example in order to preserve the game’s storytelling, should it imply that this zone must be accessed only at a later stage) by generating encounters with hostile NPCs when the player is approaching the zone, which is managed by complex Expert Systems performing rule-based monitoring of the player’s exploration of the level.
- The arms race in the industry (mid 2000’s – today)
From the beginning of the 2000’s onwards, the gaming industry has become a behemoth, the gaming companies commanding capitalizations, turnovers and valuations similar to blue chips[12], and the budgets allocated to game development and the revenue of the games by AAA studios are rivaling the ones of Hollywood productions[13], which themselves, more than often, are developing franchises based upon video games[14].
With such high stakes, the competition between the studios is raging, and the use of AI is now massive across the industry, as the leading providers of game engines have all incorporated AI technologies[15] which are used all across the board to lower costs and time to market of the games.
However, the holy grail of the gaming industry lies elsewhere, in the use of AI to achieve more intimate knowledge and understanding of the player, in order to provide a more personalized and immersive gameplay experience, allowing to attract the player to spend more time in the game and to continue playing the game longer.
This endeavor is mission-critical for the gaming industry which is suffering from a poor player retention issue, driving costs up, due to increased marketing expenses necessary to acquire and retain users.
For an example, the largest segment of the gaming industry, which is the RPG/Action-Adventure segment faces the paradox of being both one of the largest (17.2 billion US dollars revenue in 2022) and fastest growing (7.6% projected 2022-2027 CAGR) segments in the industry [16] while simultaneously facing a continuous degradation of its average revenue per game per user (-20% between 2017 and 2022).
In a nutshell, these figures show that more and more players are spending more and more money on games, but they are also more and more volatile, and quickly switch from one game to another.
This is a critical issue, as license sales are only about 1 billion US dollars out of the 17.2 billion US dollars total revenue, the bulk of which being generated by subscriptions and in-game advertising, both of which suppose recurring users.
Title II – GAMES HAVE ALWAYS COLLECTED PERSONAL DATA
Of course, the gaming industry has interest, like any other industry, to collect personal data from players for marketing purposes, and in-game collection of such data is increasingly implemented by game development studios, but this collection of data is of lesser importance for the studios that the in-game collection of the personal data which is related to the behavior of the player.
As discussed previously[17], since the dawn of the video games, a large part of the gaming experience has been provided through the constant monitoring of the player’s behavior in order to select the most appropriate reaction to the player’s actions.
What started as the “simple” prediction of the player’s next move and intentions from a statistic collection of the player’s previous gameplay has now taken a whole new level.
In a game such as the critically acclaimed Left4Dead (a multiplayer zombie game), the proprietary AI engine designated as “The Director”[18] entirely manages game dramatics, pacing and difficulty by allowing the game to adapt its content dynamically to the behavior of the players, both taken individually and as a team. As a key aspect of this game is the cohesion of the team of survivors facing the zombies, all behavior which is considered in line with this policy is rewarded (better survival chances due to weapons and ammunition drops, bonuses on fighting skills tests, etc.) and any contrary behavior is punished (increased number and strength of opponents, maluses…). “The Director” also monitors player’s input through the controller which is correlated with the then current in-game situation and action on screen in order to determine the player’s stress level (“Survivor Intensity” according to the terminology used by “The Director”) and determine whether to increase or decrease the pace of action.
One might question the fact that the monitoring of in-game behavior of a player controlling an avatar in a game which consists essentially at shooting hordes of zombies can be considered as collection of personal data of a player, but it is exactly that, and even more: it is a way to hack into the player’s brain.
It is an established fact that the gaming industry has long been extensively using whatever information it can collect through the game controller[19] from the physical posture, gestures and movement of the player in the real world in correlation with the produced actions of its avatar in game in order to guesstimate the emotional state of the player[20]
What is little less known is that the most sophisticated games are also working backwards, by influencing the player into performing specific actions in the physical world which will modify its emotional status.
As early as in the late 2000’s and early 2010’s, Dr Katherine Isbister’s work as the research director of the Game Innovation Lab of Polytechnic Institute of New York University (NYU-Poly) was focusing on how to infer the emotional and physical state of individuals from their interaction with controllers, and even how to influence their emotional state by leading them, through the provided content on screen, to perform certain actions in the physical world (type of interactions with the controller), for an example perform hand movements which are known for having the effect to reduce stress used in tai-chi[21].
A perfect example of application of these techniques can be found in Quantic Dream’s “Heavy Rain”. As the father of a kidnapped girl, the player has to engage, under the instructions of the kidnapper, into a session of driving at fast speed on a crowded road and in the wrong lane. This sequence, which is entirely useless from the storytelling perspective (as the gameplay makes it so that the player will systematically fail to perform what is requested from him) has for unique goal to force the player into performing a series of hand gestures which have for effect to increase significantly his stress level, in order to prepare the player for the next game sequence.
Quantic Dream is a studio which specializes in publishing critically acclaimed titles all having in common to be based upon brain-hacking techniques. These do not necessarily require the use of the controller, and can be based only upon monitoring in-game behavior and decisions of the player for assessing the player’s emotional state and proposed content for influencing this state. Quantic Dream’s “Detroit: Become Human” is fully based on the creation of a real empathy between the player and his avatar in game, a robot which is gradually developing self-conscience and emotions, through subtle psychological manipulation of the player.
The same nudging techniques can be found in the multi-awarded action-adventure game “The Last Of Us”, which fantastic commercial success is not to be attributed to its action-adventure game dynamics, but to its ability to bring the player (playing a zombie apocalypse survivor who lost his daughter) to engage emotionally with the virtual character of the player’s avatar protegee, a little orphaned girl, in order to lead the player to take the right decision (save the little girl, who is immune to the virus which turned humanity into zombies and could provide a cure, but needs to be killed for that purpose) into the final sequence of the game.
In order to optimize the couple identification of the emotional status of the player / influence on the emotional status, the developers of “Until Dawn” (a horror game) have gone as far as to train their AI on scores of players which were monitored using medical apparatus for monitoring heartbeat, pupils’ dilation, blood pressure levels, bodily heat, body stance and fingers movement while playing the beta version of the game.
As a result, the game assesses with perfect accuracy the stress level of the player and make sure it is adjusted to the game developer’s intended level through a combination of light, sound and atmosphere effects and the triggering (or to the contrary the pausing) of events.
Title 3 – THE FUTURE IS NOW: NEXT-GEN GAMES AND PERSONAL DATA
Virtual reality is happening right now, along with its multiple dedicated user interfaces which are all imperceptibly collecting personal data related to the emotional and physical data:
- VR googles are measuring the players head wobbling, eye movement and pupils’ dilation to perform their basic functions of rendering a 3D immersive view, data which can also be used to infer the then current physical and emotional state of the player.
- Microsoft Kinect uses infrared sensors to render a 3D image of the player’s body in order to analyze its stance, which is used to perform body movement recognition but which can also be used to infer the player’s psychological state.
- VR gloves allowing to perform virtual manipulations, which are also measuring bodily heat and blood pressure in the same way a smart watch does, are now available to be used as gaming interfaces.
- Walking treadmills, though still expensive, are also available and are increasingly deployed in gaming arcades. It is only a matter of time before they become common in households, and that their data collection ability is improving.
- Speech to text and text-to-speech technologies, allowing the player to interact with the game environment through speech, has been here for a long time. It was not used because of the lack of efficiency of Large Language Models, but the recent improvements brought by Open AI (Chat-GPT) and others is now making its implementation a hot topic in the gaming industry. Voice recognition software has the ability to provide additional emotional status analysis capabilities.
At the same time, generative AI offers to studios the possibility to generate gaming content on the fly, bringing them closer to the achievement of their quest for the holy grail of the game industry: the ability to generate entertainment content which is fully adapted to the expectations of the player and the experience the player wants to have, through measurement of the player’s physical and emotional state, including the satisfaction / frustration level.
This collection and exploitation of data which has all the characteristics of being highly sensitive personal data, as it is psychological and physiological data, is happening under the radar, as most people do not understand exactly what is happening under the hood of a video game. Also, the fact that they are happening in-game, for the purpose of improving the player’s experience, and are not necessarily tied to the identification of the player might be the reason why this is not considered as being a legal issue for now.
The upcoming Metaverse era will probably bring this issue to the front, as Metaverses are nothing more than massively multiplayer online next-gen game environments and are based upon the same technologies as open world next-gen games.
There is no doubt that the actors of the Metaverse industry will take advantage of the considerable experience of the gaming industry in the areas of player monitoring and player nudging[22] in order to provide commercial content which is tailored to their users’ expectation, and it is highly probable that the personal medical data which constitutes the psychological and physiological state of the user will be cross-referenced with other personal data, such as data collected upon account registration (age, sex), and with the user commercial profile (registered preferences, satisfaction surveys, financial and geographical data), to refine further the results in order to provide the said tailored content.
[1] In this article, gaming industry refers exclusively to the specific subsegment of the video games development and publishing.
[2] The IT industry would have to wait until the 2000’s and the boom of the internet to relive such fast and large adoption of a technology by the general public.
[3] Miniaturized portable consoles dedicated to a single game, the most famous of them being Donkey Kong.
[4] Mass-teaching of IT in universities did not exist at that time, and the programmers were mainly high-level math graduates which were preempted by the mastodonts of software development (Control Data, IBM) for catering the needs of the more serious, and also booming, business information technology sector.
[5] Non-player character, i.e a character which is a part of the game environment and is controlled by the computer.
[6] Which requires the coexistence of multiple possible alternate paths which the player can follow to achieve the game’s objectives.
[7] To the opposite of the historical level design, which constrained the player into following a predetermined sequence of events / actions / objectives before making the next one available and the next and so on until level completion, non-linear gameplay makes available multiple sequences of events / actions / objectives to the player at the same time, allowing the player to freely alternate between them in a “hop-in / hop-out” mode until level completion.
[8] Based upon a complex physics model and a terrain model which encompasses the physical characteristics of the terrain (water, mud, rocks, etc.) to provide a realistic simulation of the environment and its response to players’ interactions.
[9] If the first open-worlds were relatively small in size, standards of the market such as the game world of The Witcher III, Grand Theft Auto V, Cyberpunk 2077 or Red Dead Redemption 2 feature the full 3D modeling of environments which, in real life, would correspond to surfaces of several square kilometers (over 30 square kilometers for Red Dead Redemption 2).
[10] For an example, in Red Dead Redemption 2, critically acclaimed for the realism of its wildlife simulation and its hunting system, the players can hunt an animal across extremely large distances.
[11] Which is never done by game studios, as it would entirely defeat the purpose of procedural generation, which is to save time and costs by avoiding manual intervention in the level design process.
[12] Companies such as Electronic Arts, Activision, Sega or Nintendo have been valued in the billions of dollars by that time, and this valuation did not stop to increase. Recently, in 2022, Microsoft finalized the acquisition of Blizzard, the developer of World of Warcraft for 68.7 billion US dollars in cash.
[13] Many budgets are in the 100-200 million US dollars range, and the largest budget ever for a game is for a game still under development, Star Citizen, with a current budget of about 580 million US dollars, dwarfing the budget of the most expensive movie ever made, Star Wars: The Force Awakens (533 million US dollars).
[14] From Tomb Raider to The Witcher, there are countless examples of adaptation of games by the movie industry.
[15] Unreal Engine has been providing its Blueprints technology (graphical environment for codeless design of rule-based systems and decision trees and implementation in-game) for a few years now, and Unity recently announced the availability of dedicated generative AI modules, Unity Sentis and Unity Muse.
[16] https://www.statista.com/outlook/dmo/app/games/adventure-games/worldwide#revenuet
[17] See the last paragraph of Title 1 – Section A “the early years” hereinabove
[18] https://left4dead.fandom.com/wiki/The_Director
[19] From the couple mouse + keyboard to dedicated game controllers with motion detection capabilities like the Wii controller or Microsoft Kinect
[20] https://gizmodo.com/how-wii-and-kinect-hack-your-emotions-5691513
[21] Work for which she was awarded the prestigious Humboldt Research Fellowship in 2011 https://engineering.nyu.edu/news/nyu-polys-katherine-isbister-awarded-humboldt-research-fellowship
[22] Especially that major actors, such as Microsoft, are already deeply invested in the gaming industry
[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]