__Mobility load balancing in LTE/5G__

Given below is a high-level description of an example of how reinforcement learning can be used with NetSim 5G

- Load transfer from an overloaded cell to an under-loaded neighbouring cell
- The default association in NetSim is Max RSS i.e., the UE associates itself to the BS from which it sees max signal strength

- Now, let us denote c_ij as the instantaneous rate of a UE and is theoretically a log function of SINR

- And let R_ij be the long-term rate and y_ij is the fraction of resources allocated, to UE_i by BS_j

- Note that the max RSS association does not balance the load between BSs. The load balancing problem can be solved from optimization theory for a fixed topology
- Now let's say the SINR changes with time due to user mobility
- Then RL can be used to decide a "load-aware" UE-BS association i.e., the association is not based on max RSS
- We explore, Markov decision process/Q-learning based (model-free) RL
- At state s_t RL agent selects action a_t by following policy π and receives reward r(s_t, a_t).
- The MDP has value function V^π (s), and action value function Q^π (s, a) where α (0≤α≤1) is the discount factor
- We assume that the update interval (epoch) ≫ LTE/5G frame length of 10ms

- State: UE SINRs (γ_1,…, γ_N ), based on the current association at time t
- Action:
- Association x_ij (indicator variable showing association of UEi_ to BS_j)
- Resource allocation y_ij (equals 1/sum_j x_ij, i.e., the reciprocal of the number of UEs associated with a BS. This is exact for Round robin, and on average for PFS)

__Variations__

- Consider GBR and Non GBR users
- Split between GBR PRB usage and Non GBR PRB usage
- Time-varying network traffic
- Use deep neural networks to approximate the Q and Value functions
- Additional constraint: Minimum throughput per user (i.e., minimum SINR , γ, to all users)
- Objective: Latency minimization

__Other examples__

- Association based on logical cell boundaries using Cell individual offset (CIO)
- Power control in multi-tier HetNets

Attachment

A toy example (4 slide PDF) showing RL for 5G is attached