Algorithmic Innovations in Multi-Agent Reinforcement Learning: A Pathway for Smart Cities

Abstract

The concept of smart cities has emerged as an instrumental solution to tackle the intricate challenges associated with urbanization in the twenty-first century. Among the myriad of issues that smart cities aim to address, key concerns such as efficient traffic management, sustainable energy usage, resilient infrastructure, and enhanced public safety are at the forefront. Notably, the implementation of multi-agent reinforcement learning (MARL) has garnered significant attention for its potential role in realizing the vision of smart cities. This chapter serves as an exploration of the frontiers of algorithmic innovation within MARL and its direct applicability to address the complex challenges of urban smart grid systems. The integration of MARL principles is vital in comprehensively modeling the intricate, interdependent urban systems that underpin the smart city framework. Particularly, we emphasize the relevance of MARL in providing adaptive solutions to the intricate dynamics of the urban smart grid, ensuring effective management of energy resources and demand-side management.

1. Introduction

The rapid growth of urbanization across the globe presents a profound challenge for the design and management of modern cities. As the world’s population continues to gravitate towards urban areas, there is an ever-increasing demand for cities to operate efficiently, sustainably, and intelligently. Smart cities, a vision that combines advanced technologies with data-driven decision-making, offer a promising solution to address these complex urban challenges [1, 2]. At the heart of the smart city concept [3, 4, 5, 6] lies the ability to optimize and coordinate various urban systems in real-time, from traffic management [7] and energy distribution [8] to public safety and transportation [7]. Achieving this level of sophistication necessitates cutting-edge technologies, and among these, artificial intelligence (AI) stands out as a crucial enabler. Within the AI domain, multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for modeling and solving complex decision-making problems involving multiple autonomous agents [9]. MARL represents a fusion of reinforcement learning (RL) [10] and multi-agent systems (MAS) [11, 12] making it well-suited for addressing the intricate and interconnected challenges found in urban environments [3, 5, 6, 11, 13].

This chapter embarks on a journey into the realm of algorithmic innovation within the context of MARL, elucidating its potential as a transformative pathway for the development of smart cities. The motivation behind this exploration is rooted in the pressing need to harness the capabilities of MARL to enhance the quality of urban life. The modern cityscape is a dynamic and ever-evolving ecosystem, characterized by its complex web of interactions between diverse entities such as vehicles, buildings, infrastructure, and, most crucially, its residents. Managing these interactions optimally is a monumental task, one that has traditionally required extensive human intervention and resource allocation. Incidentally, the advent of MARL has opened new possibilities. By endowing agents, whether they are autonomous vehicles, energy management systems, or emergency response units, with the ability to learn and adapt to their environments through interactions, we can pave the way for more efficient, sustainable, and resilient urban systems. The implications extend far beyond mere automation; they encompass the potential to create cities that are responsive, data-driven, and capable of continuously optimizing their operations in real-time. The objective pursued in this chapter is at least twofold: (1) Comprehend MARL fundamentals: we will establish a solid foundation by elucidating the fundamentals of reinforcement learning and multi-agent systems, providing insights into the unique challenges and complexities of MARL in the framework of smart cities. (2) Explore algorithmic innovations: we will delve into the state-of-the-art algorithms and techniques that have emerged in the field of MARL, emphasizing their relevance and potential in the context of smart city development.

In pursuit of our goals, we will direct our attention towards the challenge of implementing intelligent automation for energy management within urban settings, employing a comprehensive multi-scalar approach that spans from individual households to the entire cityscape. It is crucial to acknowledge that the energy consumption in both residential and commercial buildings is steadily on the rise. This upward trajectory can be attributed, in part, to the escalating proliferation of household appliances, which in turn drives a substantial surge in domestic electricity demand. Consequently, effective planning for residential energy utilization becomes imperative for achieving optimal energy management within the framework of urban electricity distribution networks: smart grid. This intricate challenge is effectively addressed through the application of MARL models. The rest of the chapter is structured as follows: Section 2 focuses on the context and overview of the smart city. The Section 3 introduces the fundamentals of MARL, providing a basis for understanding its application in smart cities.

In Section 4, materials and methods, we present a comprehensive exposition on the multi-agent formalization for reinforcement learning pertaining to the automated energy management within urban buildings. This discussion is contextualized within the purview of smart grids, and it extensively expounds upon the methodological nuances underlying the modeling of diverse agents, both at the micro-level of individual buildings and at the macro-level of the entire city infrastructure. Building upon this framework, Section 5 is dedicated to an in-depth exploration of scenario testing implementation and subsequent results analysis. Here, we dissect the findings and their implications, shedding light on the intricate dynamics of urban energy management and the role played by MARL models. Finally, we conclude by emphasizing the significant potential that Collaborative Intelligence embodies in reshaping our urban landscapes, thereby contributing to the growing body of knowledge in the realm of MARL’s pivotal role in shaping the cities of the future.

2. Background and smart cities overview

Across the globe, cities are confronting and surmounting multifaceted challenges by embracing a rich tapestry of inventive concepts [3, 4, 5, 6]. This diverse spectrum of initiatives encompasses novel energy and transportation paradigms [7], groundbreaking innovations in residential construction [3], the proliferation of shared services [11], digitization of administrative processes [4], and a myriad of other pioneering endeavors. These groundbreaking initiatives are a collaborative effort involving cities, established corporations, nimble startups, non-profit organizations, and engaged citizens, all working collectively to advance innovative solutions. In recent years, a significant portion of these initiatives, particularly those harnessing the power of emerging information and communication technologies, have coalesced under the encompassing banner of “smart city” endeavors.

A smart city is one that methodically applies digital technologies to optimize resource utilization, elevate the quality of life for its residents, and bolster the regional economy’s competitiveness in a sustainable fashion. It’s a holistic approach that deploys intelligent solutions across various facets of urban life, encompassing infrastructure, energy management, housing, mobility, services, and security. These solutions are rooted in integrated sensor technology, seamless connectivity, data-driven analytics, and self-sustaining value-added processes.

However, it’s important to recognize that the path to a truly smart city is not without its challenges and complexities. Smart city projects often unfold as intricate, costly, occasionally chaotic, and time-consuming endeavors. They present a unique set of demands, necessitating two distinct competencies: an astute understanding of the ramifications of embedding digital technologies within the fabric of urban development, and the capacity to conceive integrative solutions that transcend traditional departmental boundaries. Confronted by these formidable demands (Figure 1), many decision-makers and implementers find themselves at an impasse, unsure of where to commence or how to navigate the labyrinthine journey ahead. Consequently, a significant reservoir of untapped potential languishes on the precipice of realization.

Figure 1.

General framework of smart city. Source: Adapted from Ref. [13].

2.1 The essence of smart cities

Smart cities epitomize the methodical harnessing of digital technologies’ boundless potential, seamlessly interwoven with a holistic embrace of users, inhabitants, and all stakeholders alike. At its core, the smart city endeavor is driven by a grand vision: to realize urban landscapes characterized by optimized resource utilization, elevated living standards, and a sustainable boost in competitiveness. To achieve these paramount goals, a predominantly digital transformation across the domains of infrastructure, energy management, housing, mobility, services, and security becomes not just preferable, but imperative. Central to this transformation lies the concept of a city’s “digital shadow” [14], a foundational element that serves as the bedrock for the city’s digital evolution. It’s important to distinguish this concept from the notion of a “digital twin” [15]. This digital shadow is the vessel through which intelligent solutions are infused into the urban fabric. Rooted in integrated sensor technologies, seamless connectivity, adept data analysis, and self-sustaining value-added processes, the digital shadow marks the inception of a profound change. The transformation towards an intelligent, interconnected urban ecosystem commences with the transformation of physical products, processes, and services into their digital counterparts. As these entities evolve into intelligent, autonomous, and integrated entities, a digital shadow emerges, propelling ecological and social betterment into the forefront. At the vanguard of this transformation stands the Internet of Things (IoT), serving as the conduit between the digital realm and the physical world [16]. Notably, emerging technologies such as Blockchain are poised to revolutionize secure transactions and identity management within cities, fostering trust and transparency [17, 18, 19, 20].

Powering this transformative voyage is the bedrock of modern data analytics, often encapsulated within the realm of Artificial Intelligence (AI). Through sophisticated algorithms and machine learning, AI sifts through vast datasets to uncover intricate patterns, autonomously refining systems with minimal human intervention. For instance, the intricate road system in Los Angeles has learned to optimize traffic flows through this autonomous self-improvement process, a testament to the potential unleashed by AI-driven urban innovations [11, 15, 16].

However, the digital shadow, in its nascent form, remains neutral and devoid of purpose. Therefore, it is imperative for self-learning systems to be deeply attuned to their surroundings, attentively considering the needs and aspirations of city residents. The digital shadow, while a pivotal prerequisite for smart cities, stands incomplete on its own. It is only through a laser-focused commitment to the city’s holistic milieu that it can fulfill its true potential.

2.2 Main constituents of smart city perspective evolution

In the annals of urban development, the year 2008 marked a transformative juncture when IBM introduced the visionary “smarter planet and smart city concept” worldwide. This groundbreaking concept took root in select cities, driven by the transformative potential of Information Communication Technology (ICT). Leading the charge, nations like Japan, Singapore, and China embarked on the ambitious journey of crafting smart cities, underpinned by the prowess of ICT [17].

Over the past decade, the landscape of smart cities has undergone a remarkable metamorphosis, with ICT serving as the catalyst for change. These cities have evolved to encompass a spectrum of characteristics that together define the essence of “smart cities.” These constituents are encapsulated in the following pillars: “smart governance,” “smart education,” “smart living,” “smart mobility,” “smart environment,” “smart energy,” “smart healthcare,” and “smart citizens” [21, 22]. It is through the harmonious convergence of these elements that the symphony of a smart city comes to life. The smart city narrative continues to unfold, with technology acting as an enabler for greater intelligence and efficiency. Pioneering technologies such as the IoT, AI, and the transformative potential of big data analytics have emerged as formidable allies in the quest for smarter urban landscapes [22, 23, 24, 25]. These technologies are the building blocks of a new urban paradigm, where data-driven insights, automation, and connectivity redefine the way cities operate and flourish. Each of these key components plays a unique role, interlocking with others to create a tapestry of innovation and progress. For a detailed breakdown of these components by domain, please refer to Table 1.

Key domain	Sectors and services associated with domains	References
Governance	Online citizens portals, efficient and fast public services, effective resource management, innovative planning approaches, public asset management, e-services, connecting people through social media.	[4, 20, 23, 26]
Education	Smart infrastructure with closed-circuit television surveillance, GPS tracking of school busses, smart learning through video conferencing lectures, teacher–students management solutions, virtual labs.	[4, 20, 23, 26]
Living	Public security tools, safety alarms at public places at panic situations, community network management, safety of senior citizens.	[20, 21, 23, 26]
Mobility	A smart toll collection system, community carpooling system, charging point for electric vehicles, smart parking system, connected and autonomous vehicles	[2, 3, 20, 21, 23, 26]
Environment	Traffic management, vehicle monitoring, water quality management, air quality management, smart water storage and purification system Wastewater management, pollution sensors, disaster management, green and clean environment, smart waste management.	[3, 20, 21, 23, 26]
Energy	Smart meters, efficient utilization of energy subsystem, energy distribution through sensors.	[3, 20, 23, 26]
Healthcare	E-health records, diagnostic analytics portals, emergency medical services	[20, 23, 26]
Citizenship	Privacy and security of citizens, social engagement of the people, raising awareness of smart solutions, community interactions	[3, 20, 21, 23, 26]

Table 1.

Main required components of smart cities.

The advent of smart cities has ushered in a paradigm shift in urban planning and management. At its core, this transformation embraces the concept of agent-based modeling (ABM), a powerful computational framework that treats various urban entities as intelligent agents. These agents, which can represent a wide spectrum of urban systems, including traffic signals, public transportation networks, energy grids, and waste management systems, interact with one another and their environment, making informed decisions to optimize their respective functions. In the context of smart cities, MARL allows agents to adapt to the ever-changing dynamics of urban environments, making decisions that lead to more efficient, sustainable, and responsive city operations [25, 26, 27].

3. Fundamentals of multi-agent reinforcement learning

In recent times, MARL has emerged as a focal point of research, particularly in the realm of multi-agent systems operating in expansive, large-scale environments. This surge in interest can be attributed to its remarkable success, particularly in the domain of strategic games […]. At its core, Reinforcement Learning (RL) draws inspiration from the mechanisms of animal learning in psychology […]. It embodies a trial-and-error learning process where an agent strives to learn an action policy that maximizes cumulative rewards over time through its interactions with an environment […]. Urban environments, on the other hand, are characterized by unparalleled complexity and dynamism. They comprise a multitude of interconnected components and features, perpetually influencing one another.

3.1 Reinforcement learning basics

RL stands as a prominent and widely recognized subset of machine learning methods, specifically tailored to the art of acquiring the skill to accomplish specific tasks by engaging in dynamic interactions with the surrounding environment. This pivotal task often hinges on the presence of a reward mechanism, serving as a beacon guiding the intelligent agent towards optimal performance. In essence, RL casts the intelligent agent in the role of a decision-maker, requiring it to navigate a spectrum of situations by selecting actions strategically. The accumulation of rewards based on these actions functions as a compass, guiding the agent towards more proficient decision-making in the future. The overarching goal is to amass rewards as efficiently as possible over an extended period, ultimately steering the agent’s behavior towards a state of optimal performance in the long term. The focal point of RL revolves around the quest to uncover a control policy capable of achieving predefined objectives. In this context, RL takes on the formalized structure of a Markov decision process [28], serving as the bedrock upon which iterative learning is built. Within each iteration, a RL agent carefully chooses an action (a_t) based on the prevailing policy and the current state (s_t). This selected action is then executed within the given environment, ushering in a transition to a new state (s_t+1) and the concomitant bestowal of a reward (r_t+1) as elucidated in Figure 2.

Figure 2.

RL paradigm in a Markov decision process for an agent.

Through its continuous interaction with the environment, the policy undergoes iterative refinement via RL methodologies, all aimed at maximizing the cumulative long-term reward. To compute the optimal policy, a diverse array of techniques is at the RL practitioner’s disposal, with value-based and policy-based approaches emerging as the most prominent contenders [29, 30]. When confronted with intricate challenges, such as those outlined in Table 1, neural networks emerge as a formidable tool, empowering RL to effectively predict the optimal policy or value function [31]. Yet, the scope of RL extends beyond individual agents tackling isolated problems. In scenarios marked by complexity and interdependency, as underscored in Table 1, RL seamlessly extends its domain to encompass multiple agents coexisting within the same environment. Here, these agents can engage in collaborative or competitive interactions, ushering in the realm of MARL. In the following, we will delve deeper into the intricacies of MARL and explore how these algorithmic innovations are poised to chart a transformative pathway for smart cities, where a consortium of learning agents collaboratively engages with urban environments.

3.2 From bellman equation to iterative policy evaluation and improvement

The quest to understand and approximate the value function of a given policy π has long been a central pursuit in the realm of RL. At its heart, this endeavor hinges on unraveling the intricate relationship between the value of a state and the values of its successor states. This foundational relationship is encapsulated by the state value function, which adheres to the following Eq. (1):

$$v_{\pi }\left(s_t\right)=\sum _a\pi \left(a_t|s_t\right)\sum _s{P}_s^a(r\left(s_t,a_t,s_{t+1}\right)+\mu v_{\pi }\left(s_{t+1}\right)$$E1

This equation elegantly expresses the interplay between the value of a state and the expected values of its future states. It serves as the cornerstone upon which numerous methods for computing and learning the state value function v_π are constructed. Not to be outdone, the state-action value function qπ also adheres to a recursive relationship known as the Bellman equation [32]:

$$q_{\pi }\left(s_t,a_t\right)=\sum _{a_{t+1}}{P}_s^a((r\left(s_t,a_t,s_{t+1}\right)+\mu \sum _{a_{t+1}}\pi \left(a_{t+1}|s_{t+1}\right)q_{\pi }\left(s_{t+1},a_{t+1}\right)$$E2

The Bellman equation for (q_π) embodies a similar spirit, delineating the value of a state-action pair (s, a) as a function of the rewards, transitions, and values of the ensuing states. The specific challenge of computing the state value function v_π for a given policy, termed policy evaluation, takes center stage. To tackle this challenge, we adopt an iterative approach, bypassing direct computational methods in favor of a more computationally efficient strategy and we assume complete knowledge of the environment’s dynamics, implying familiarity with the transition probabilities for each element (s_t, a_t). Below, we present the pseudo-code outlining the iterative policy evaluation algorithm:

The pursuit of computing the value function for a given policy is not merely an intellectual exercise; rather, it serves as a pivotal stepping stone towards the enhancement and optimization of existing policies. Consider a deterministic policy π for which the associated value function v_π has been diligently calculated.

This foundational knowledge becomes a catalyst for the creation of an improved policy, denoted as π′. So, how does one transition from π to π′? The process unfolds as follows:

$${\pi }^{\prime }\left(s_t\right)={\mathit{\text{argmax}}}_a{q}_{\pi }\left(s_t,a_t\right)$$E3

The Policy Improvement Theorem [33], a fundamental result in the realm of RL, underpins the process of policy refinement. According to this theorem, the new policy π′ derived from an existing policy π is inherently superior or, at the very least, equivalent in performance. This transformative procedure, which entails evolving an old policy into a more optimal one by aligning it with the insights garnered from the value function, is formally recognized as “policy improvement.” The steps to execute this method are concisely outlined below:

3.3 CityLearn: enabling algorithms implementation and execution

In this section, we present the fundamental implementation tools and aspects of our research through CityLearn [34]. Eqs (1)–(3) are operationalized through the pseudo-coded algorithms presented in Tables 2 and 3, forming the core of our agent coordination policies. In the context of MARL, these policies are the controllers. Leveraging the OpenAI Gym standard, CityLearn serves as a platform for deploying MARL algorithms for urban energy management, load-shaping, and demand response [34, 35]. CityLearn operates primarily in a decentralized control mode, which we utilized for our MARL controllers. To ensure reproducibility, we outline its four key functionalities: (1) Facilitation of MARL implementations. (2) Full customizability of the reward function, enabling the use of a multi-output function for MARL applications. (3) Modular and open-source design, accommodating diverse energy system classes, including options for user-created classes, particularly for city-scale energy systems. (4) Provision for users to generate their datasets (e.g., weather data, building energy demand, EV schedules) and integrate them into CityLearn. Further guidelines on dataset creation can be accessed through the GitHub repository [36].

Initialize v(s) arbitrarily for all s ∈ S
Repeat until convergence:
	Δ ← 0
	For each s ∈ S:
		v_old ← v(s)
		v(s) ←${\sum }_a\pi \left(a_t|s_t\right){\sum }_s{P}_s^a(r\left(s_t,a_t,s_{t+1}\right)+\mu v_{\pi }\left(s_{t+1}\right)$
		Δ ← max(Δ, |v_old - v(s)|)
	Until Δ < θ (a small positive threshold)

Table 2.

Iterative policy evaluation algorithm.

policy-stable ← true
For each s ∈ S do
	old-action← π(s)
	π(s) ← ${\mathit{\text{argmax}}}_a{q}_{\pi }\left(s_t,a_t\right)$
	if old-action $\ne$π(s) then
		policy-stable← false
	end
end
if policy-stable = true
		Stop and return π
end
else
		Go to Table 2.
end

Table 3.

Iterative policy improvement algorithm.

4. Materials and methods

MAS serve as the linchpin in a diverse array of applications within smart cities. These applications encompass a broad spectrum of technologies, including integration with the IoT and autonomous systems. Examples of such autonomous systems are intelligent robots, unmanned aerial, underwater, or surface vehicles, self-driving cars, and advanced transportation and healthcare systems. In these contexts, MAS agents engage in distributed interactions to fulfill specific tasks and objectives. In the ideal smart city scenario, MAS should exhibit robust, decentralized, and collaborative behaviors. These systems are expected to make intelligent and cognitive decisions while devising efficient solutions to data-driven challenges. Consequently, they facilitate decentralized information management and decision-making processes [37] that underpin the dynamics of smart cities. The remainder of this chapter is devoted to resolving the problem of efficient management of energy systems predicament within urban settings, employing the MARL paradigm. This section aims to dissect the complexities associated with energy resource allocation and optimization, elucidating the role of MARL in enabling efficient and effective urban smart grid management strategies.

4.1 Multi-agent model design for urban smart grid

One of the primary catalysts for grid decarbonization is the seamless integration of renewable energy systems (RES) into the grid’s supply chain. Within the residential domain, the Home Energy Management System (HEMS) proves to be a highly effective tool for automating energy management (Figure 3) as illustrated.

Figure 3.

HEMS agent at the heart of urban smart grid.

In the realm of MARL for smart cities, the selection of appropriate sensors plays a pivotal role in the acquisition of accurate and high-quality data [38]. As technology continues to advance, an extensive array of sensors has become available for gathering geospatial data within urban environments [39, 40, 41, 42]. In recent years, mobile sensors such as smartphones and tablets have gained significant popularity [43, 44]. The concept of demand response (DR) is central to an effective energy management strategy (Figure 4), offering consumers and prosumers the ability to provide the grid with the much-needed flexibility. This is achieved by reducing energy consumption through load management, shifting energy consumption to off-peak times, or generating and storing energy when grid conditions are favorable. In return, consumers and prosumers typically experience a reduction in their energy bills.

Figure 4.

Multi-agent coordination in demand response within smart grid (taken from the CityLearn GitHub repository).

In fact, HEMS functions as a distributed intelligent agent that empowers users to partake in local energy trading and realize energy savings at the household level. By acting as a strategic agent, HEMS contributes to holistic load management, ensuring that the energy needs of each household are met in an efficient and cost-effective manner. The HEMS system operates as an intelligent agent [45, 46] that crafts optimal energy plans for home appliances, considering individual customer energy consumption plans and comfort requirements. By adapting and learning from customer interactions, these agents become adept at predicting and optimizing individual customer decision-making patterns, thereby ensuring effective and efficient urban energy management.

4.2 MARL coordination policy assessment

Coordination policy takes the principles of single-agent RL and extends them to facilitate distributed decision-making at a larger scale.

• Multi-agent training for decentralized execution: In MARL, the challenge lies in training multiple agents to execute strategies in a decentralized manner [47]. To model this, we formulate the problem as a partially observable Markov decision problem within the framework of Deep Reinforcement Learning (DRL) [48]. However, when the environment is dynamic, the learning task becomes considerably more challenging. In such cases, each agent within a multi-agent system often perceives all other learning agents as part of the environment, creating a non-stationary scenario. This non-stationarity can lead to suboptimal policies, as they are developed in a distributed fashion and may not be sufficiently robust.

• Addressing non-stationarity through centralized training with decentralized execution: To mitigate the non-stationarity challenge, various approaches have been proposed [49, 50], focusing on centralized training with decentralized execution. These approaches encompass different strategies: (1) Shared Critic Networks: For actor-critic algorithms, like Multi-Agent Deep Deterministic Policy Gradient (MADDPG) [49], they involve training on an ensemble of policies that encourage more robust multi-agent policies. The shared critic network plays a crucial role in coordinating the actions of multiple agents. (2) Q-value Mixing (QMIX): QMIX [51] utilizes a network to estimate joint action values, ensuring monotonicity per agent. This guarantees tractability and consistency between centralized and decentralized policies, fostering better coordination. (3) Inter-Agent Communication: Some methods leverage communication between agents to enhance scalability. For instance, Differentiable Inter-Agent Learning (DIAL) [50] and CommNet [52] employ deep neural networks to learn end-to-end communication protocols. This is particularly useful in complex environments with partial observability. Agents in these approaches exchange information and backpropagate error derivatives, combining centralized learning with decentralized execution to improve overall performance. (4) Macro-Actions: Another approach, as introduced in [53], involves creating abstractions known as macro-actions. These abstractions help improve scalability by allowing agents to reason and plan at a higher level, thus reducing the complexity of decision-making.

5. Experimental settings and results

CityLearn stands out as a comprehensive and user-friendly OpenAI Gym environment tailored for the seamless implementation of MARL techniques within the intricate domain of urban energy systems. This innovative platform is designed to bring about significant transformations in the aggregated electricity demand curve by orchestrating the energy storage capabilities of a diverse array of buildings within a district [34, 35]. The primary goal of CityLearn is to simplify and standardize the evaluation of RL agents, thereby serving as an invaluable benchmarking tool for a wide spectrum of algorithms.

5.1 Key features of CityLearn

Key features of CityLearn are: (1) Diverse Energy Models: At the heart of CityLearn lies an extensive library of energy models, encompassing vital components such as air-to-water heat pumps, electric heaters, chilled water (CHW) systems, domestic hot water (DHW) heaters, and electricity energy storage devices. These models enable the simulation of a wide range of building energy systems, reflecting the rich heterogeneity found in real-world urban environments. (2) Building Energy Systems: Each building in the simulation is equipped with an air-to-water heat pump for cooling and an electric heater for DHW heating. Additionally, buildings have the flexibility to incorporate various combinations of CHW, DHW, and electricity storage devices. These components work collaboratively to offset cooling, DHW heating, and electrical loads drawn from the grid. (3) Storage Capacity: CityLearn introduces the concept of storage capacity, defined as multiples of the hours that storage devices can meet the maximum annual cooling and DHW demand when fully charged. This parameterization allows for precise control and optimization of energy storage strategies. (4) Grid Interaction: The framework encompasses a dynamic interaction between buildings and the main grid. Besides the energy storage devices, other electric equipment, and appliances (non-shiftable loads) also draw electricity from the grid. For sustainable energy practices, CityLearn supports the integration of Photovoltaic (PV) systems within buildings, enabling them to generate their own electricity and reduce dependency on the grid.

CityLearn has emerged as a versatile tool with a wide array of applications. Researchers have harnessed its capabilities to explore incentive-based Demand Response (DR) mechanisms [54], coordinate energy management across multiple buildings [55], and conduct rigorous benchmarking of MARL algorithms [36, 56]. Its flexibility, coupled with its diverse energy models, makes CityLearn an indispensable asset for advancing our understanding of urban energy systems and shaping the future of sustainable urban development.

5.2 Simulation setup for smart grid

In our research endeavor, we leverage the sophisticated CityLearn environment as a crucial testbed to rigorously assess the performance of our MARL algorithm in orchestrating the actions of multiple agents, as detailed in Section 3.2 of this chapter. Our primary objective is to delve into the behavioral dynamics of these agents concerning varying durations of offline training, relying on predefined policies. We postulate that an extended offline training period equips the agents with expert-level knowledge of optimal actions, thereby leading to enhanced overall performance when they transition to an online setting. Within the simulated smart city environment, each individual building is endowed with its dedicated Reinforcement Learning – Deep Deterministic Policy Gradient (RL-DDPG) controller. The number of these controllers aligns with the total count of buildings under consideration, a scenario featuring ten buildings in this specific case study. Notably, we explore the performance of two distinct RL controllers: the conventional DDPG controller and its multi-agent variant, known as MADPG.

Here are some key insights into our simulation setup:

• Controller configuration: DDPG Controller: In the DDPG setup, each agent operates in relative isolation, possessing knowledge solely of its own states and actions. This encapsulation reflects a scenario where agents are not privy to the states or actions of their peers, operating independently.

• MADPG controller: In contrast, the MADPG controller extends the agents’ awareness to include not only their own states and actions but also those of every other agent in the system. This elevated level of information sharing fosters collaborative decision-making among the agents.

• State representation: As states, our framework incorporates key variables, including the hour of the day, outdoor temperature, and the state of charge within each Chilled Water (CHW) storage tank. These states provide valuable context and information for the agents to make informed decisions.

• Action space: The controllers’ actions pertain to the energy storage and release strategies they employ during hourly intervals. These actions are pivotal in optimizing energy utilization and management within the smart city. Here is a dataset sample that captures the MARL implementation details within the CityLearn environment in Table 4.

Building ID	Controller	Hour of the day	Outdoor temperature (°C)	CHW state (%)	Action taken
1	DDPG	15:00	28	60	Store
2	DDPG	14:00	25	40	Release
3	DDPG	12:00	30	75	Store
4	DDPG	11:00	20	55	Release
5	DDPG	10:00	27	70	Store
6	MADPG	18:00	29	50	Release
7	MADPG	17:00	24	45	Store
8	MADPG	16:00	26	80	Release
9	MADPG	13:00	22	65	Store
10	MADPG	09:00	31	35	Release

Table 4.

Dataset sample that captures the MARL implementation.

Building ID specifies the unique identifier for each building within the simulation, facilitating the differentiation and tracking of individual buildings. Controller denotes the type of controller utilized in each building, indicating whether it is a DDPG or MADPG controller. The distinction between the two controllers is crucial as it influences the decision-making process and the overall behavior of the building within the smart city environment. Hour of the Day represents the specific hour of the day at which the action is being taken. It provides insights into the temporal dynamics of the decision-making process, showcasing how the controllers operate at different times of the day. Outdoor Temperature signifies the outdoor temperature in Celsius, which is a crucial environmental factor impacting the energy management decisions of the buildings. It highlights the influence of external conditions on the operational strategies implemented by the controllers. CHW represents the state of the CHW storage tank within each building, depicted as a percentage. It indicates the current level of charge within the storage tank, which directly influences the decisions regarding whether to store or release energy. Action Taken specifies the action chosen by the controller in response to the current state of the CHW storage tank. The actions are categorized as “Store” or “Release,” reflecting the decisions made by the controllers to either store or release energy based on the prevailing conditions and objectives.

This information is instrumental in understanding the efficacy of the MARL algorithms and their implications for optimizing energy utilization and management within the smart city context. Our study capitalizes on the CityLearn environment to conduct a meticulous investigation into the performance of MARL algorithms, with a particular focus on the impact of offline training durations.

We assess the efficacy of both individualized and collaborative decision-making strategies, shedding light on the potential advantages of multi-agent coordination in urban energy systems.

5.3 Policy controller results

In our comprehensive evaluation of controllers for smart grid management within the smart city context, we employed the total cooling cost as the primary performance metric to assess their effectiveness. Figure 5 provides a visual representation of our findings, where we compared the performance of two Reinforcement Learning – Deep Deterministic Policy Gradient (RL-DDPG) controllers against two rule-based controllers (RBC).

Figure 5.

MARL policy evaluation for energy distribution.

Here are the key aspects of our evaluation and observations:

5.3.1 Controller comparison

• DDPG and MADPG controllers: We subjected both the DDPG and MADPG controllers, representing advanced machine learning-driven decision-making, to a rigorous assessment. These controllers were tasked with optimizing cooling costs within the simulated environment.

• Rule-based controllers (RBC): To provide a benchmark for comparison, we introduced two rule-based controllers. The first RBC followed a standard rule-based approach, activating cooling when the water temperature in the tank reached a predefined maximum threshold. The second RBC was manually tuned to minimize total electricity costs, representing a more refined rule-based strategy.

5.3.2 Performance outcomes

• DDPG and MADPG outperform standard RBC: Notably, both the DDPG and MADPG controllers demonstrated superior performance compared to the standard RBC. This improvement was achieved without necessitating explicit system modeling and maintained the adaptive potential inherent to Reinforcement Learning (RL) methods.

• Manually optimized RBC: Interestingly, the manually optimized RBC, which was relatively straightforward to fine-tune due to the uniformity of energy systems across all buildings, outperformed both the DDPG and MADPG controllers.

The study findings, as depicted in Figure 5, indicate that within this specific environment, the MADPG controller did not exhibit a substantial advantage over the DDPG controller. In essence, sharing information among the agents did not yield significant performance improvements. This observation raises the question of whether coordination efforts are necessary in this context, considering that similar savings can be achieved without explicit coordination. We acknowledge that these results may vary in more complex environments characterized by diverse energy systems and potentially differing optimal policies for individual buildings. Further research is warranted to explore the dynamics of coordination in such intricate settings.

5.4 Evaluation metrics comparison across key algorithms

To comprehensively uncover the origins of algorithmic innovations in the pursuit of optimizing multi-agent coordination policies for addressing challenges within smart cities, it is imperative to undertake a global assessment of the performance of various algorithms employed during this research endeavor. This segment presents a thorough juxtaposition of key evaluation metrics for the Deep Deterministic Policy Gradient (DDPG) algorithm, the Multi-Agent Deterministic Policy Gradient (MADDPG) algorithm, the Rule-Based Controller (RBC) algorithm, and the Manual Optimized Rule-Based Controller (MORBC) algorithm. The comparison encompasses crucial metrics, including energy cost savings, network stability, computational complexity, adaptability, scalability, decision-making intricacy, real-time responsiveness, and implementation complexity. By scrutinizing these metrics, we can elucidate the strengths and limitations inherent in each approach as applied to the simulation of smart grid management through CityLearn. Table 5 offers a succinct overview of our comprehensive evaluation findings.

Metric	Key algorithms
	DDPG	MADDPG	RBC	MORBC
Energy cost savings	0.921	0.853	0.652	0.881
Grid stability	0.854	0.908	0.603	0.781
Computational complexity	0.702	0.605	0.852	0.753
Adaptability	0.801	0.952	0.605	0.607
Scalability	0.903	0.851	0.804	0.801
Decision-making complexity	0.705	0.953	0.601	0.604
Real-time responsiveness	0.854	0.906	0.60	0.850
Implementation complexity	0.751	0.801	0.853	0.870

Table 5.

Innovation metrics comparison across key algorithms.

By contextualizing the evaluation within the parameters of a smart grid experimental environment, these results demonstrate insight into the performance capabilities of each algorithm, helping to highlight algorithmic innovation strategies in policy implementation. Robust multi-agent coordination adapted to efficient and sustainable smart projects.

6. Conclusion and future directions

MARL stands as a transformative force with the potential to revolutionize urban environments and address the multifaceted challenges that smart cities face. By facilitating decentralized decision-making processes, fostering cooperation, and introducing elements of healthy competition among agents, MARL paves the way for more efficient resource allocation and an enhanced quality of life for the denizens of urban landscapes. While the road ahead is not without its hurdles, the ongoing dedication to research and development in the field of MARL is indispensable in realizing the boundless possibilities that smart cities can offer in the future.

This chapter serves as a resounding call to action, resonating not only with researchers and technologists but also with urban visionaries, policymakers, and the architects of tomorrow’s cities. It urges us to unite and harness the collective intelligence of autonomous agents, coupled with the adaptability and resilience embedded in reinforcement learning. Together, we can drive our cities forward into a future where they not only navigate the complexities of urban life but thrive as shining examples of sustainability and innovation.

In our evaluation, we have illuminated the promise of RL-based controllers in the realm of smart grid management. This has underscored the capacity of RL methods to surpass conventional rule-based strategies, particularly as the intricacy of systems escalates. Moreover, our findings beckon further exploration, especially in understanding the role of coordination within the intricate and multifaceted energy systems that characterize modern smart cities.

As we venture into the dynamic realm of smart city development, these results serve as a guiding beacon, urging us to embrace MARL as an essential instrument in sculpting the future of our urban environments.