An introduction to federated learning

The core problem

Classical machine learning relies on a simple assumption: all the data is available in one place. We gather the data, train a model, and deploy it. In practice, this assumption quickly breaks down.

Imagine a hospital that wants to train an early cancer detection model from scans. The data is sensitive, subject to the GDPR, and each institution refuses, rightly so, to centralize it. As a result, each hospital has too little data to train a good model on its own, yet sharing the data is impossible.

This is exactly the problem federated learning aims to solve.

What is federated learning?

Federated learning is a distributed training paradigm introduced by Google in 2017. The core idea: the model goes to the data, not the other way around.

Here is how a standard round of federated training works:

1. A central server sends the current global model to a subset of clients.
2. Each client (hospital, phone, institution) trains the model locally on its own data.
3. The clients send back their updates (gradients or weights), not their data.
4. The server aggregates these updates to improve the global model.
5. Repeat until convergence.

The data never leaves the client. That is the fundamental promise of the paradigm.

The FedAvg algorithm

The reference algorithm is FedAvg (Federated Averaging), proposed by McMahan et al. (2017). Its aggregation rule is elegantly simple: the global update is the average of the local updates, weighted by the size of each client's dataset.

Formally, at each round $t$, the server selects a subset $\mathcal{S}_t \subset [K]$ of clients and sends them the current model $w^t$. Each client $k$ locally computes:

$w_k^{t+1} = w^t - \eta \, \nabla \mathcal{L}_k(w^t)$

then the server aggregates with an average weighted by the local dataset sizes $n_k$:

$$w^{t+1} = \sum_{k \in \mathcal{S}_t} \frac{n_k}{n} \, w_k^{t+1} \quad \text{where} \quad n = \sum_{k \in \mathcal{S}_t} n_k$$

# Simplified FedAvg pseudocode
for round t in 1..T:
    clients = sample(all_clients, fraction=C)

    for each client k in clients:
        w_k = local_SGD(w_global, data_k, epochs=E)

    # Weighted aggregation
    w_global = sum(n_k / n * w_k for k in clients)
    # where n_k = number of samples on client k, n = total

The beauty of FedAvg lies in its generality: it works with any differentiable model. In practice, the number of rounds $T$, the participation rate $C$ and the number of local epochs $E$ are the main levers.

The challenge of non-IID data

Classical statistics assumes that data is i.i.d. (independent and identically distributed). In federated learning, this assumption is systematically violated.

Each client's data reflects its own usage, demographics and environment. A phone used in Paris has a very different data distribution from a phone used in Lomé.

"Non-IID data is arguably the most fundamental challenge in federated learning." — Li et al., 2020

This heterogeneity causes several concrete problems:

• Client drift: the local updates $w_k$ move away from the global optimum $w^*$, because each client optimizes for its own distribution $\mathcal{D}_k$.
• Slow or unstable convergence: averaging gradients that pull in very different directions converges poorly.
• Uneven performance: the global model can work very well on average but poorly for some clients.

Beyond FedAvg: a few answers

Since 2017, the community has proposed many variants to address these problems. Here are a few that mark out my research:

• FedProx adds a proximal term $\frac{\mu}{2}\|w - w^t\|^2$ to the local loss, which limits drift. The client minimizes: $h_k(w) = \mathcal{L}_k(w) + \frac{\mu}{2}\|w - w^t\|^2$
• SCAFFOLD uses control variates to correct the bias introduced by the variance of local gradients.
• FedNova normalizes local updates to account for the number of steps each client performs.
• Per-FedAvg / pFedMe are meta-learning approaches that give each client a personalized model, not just a global one.

Limits and open questions

Federated learning is far from a solved problem. A few major tensions:

• Communication: sending model updates at every round is expensive. On slow networks or with large models, it is a real bottleneck.
• Actual privacy: sharing gradients is not harmless. Gradient inversion attacks can reconstruct private data from the updates.
• Robustness: malicious clients can send poisoned updates to degrade the global model.
• Fairness: minimizing the average loss can hurt minority clients. How can we guarantee a model that is fair to everyone?

Conclusion

Federated learning is not a silver bullet: it is a trade-off. A trade-off between performance and privacy, between centralization and distribution, between a global model and local personalization.

What makes it fascinating is that it forces us to rethink deep assumptions of machine learning: what is a "good" model when the data is inherently disparate? How do we guarantee fairness in a decentralized system? These questions, at the intersection of mathematics, computer science and ethics, are the ones that guide my research.

In a future post, I will present a comparative benchmark of FedAvg and FedProx on MNIST data partitioned in a non-IID way, with code, curves, and concrete conclusions.