We would like to thank Vinnova, Sweden’s innovation agency, for funding this project.

TL;DR: Vertical Federated Learning allows clients with different but complementary features for the same samples to collaborate and train a machine learning model without sharing their data.

FEDn now implements split learning as an approach for the vertical case. Split learning enables hospitals, banks, and other organizations to leverage all available information while maintaining patient or customer data privacy.

Horizontal Federated Learning

When one talks about federated learning, one usually refers to horizontal federated learning (horizontal FL). In horizontal FL, every client holds data that has the same features, but the individual samples (records) differ between clients.

Example: Hospitals might collect the same information for each patient, such as age, blood pressure, BMI, etc. Hospital 1 might have records of one group of patients, while hospital 2 has records for different patients.

Vertical Federated Learning

Vertical federated learning (Vertical FL) applies to situations where different clients hold information on the same set of individuals, but with different features. While the samples (e.g., patients or customers) are the same across client datasets, the available features differ between clients. Usually, it is also the case that only one client has access to the labels.

Examples:

  • Hospital 1 might store age and blood pressure about a patient, while hospital 2 stores other information about the same patient. Combining these features can improve disease prediction.
  • Bank 1 might store transaction data about customers, while bank 2 collects credit card usage details for the same customers. Combining this information can improve credit score prediction.

Vertical FL is used when parties have different, but complementary features for the same data samples. Vertical FL allows them to build more accurate predictive models while preserving data privacy.

A concrete example: The Pima Indians Diabetes Database dataset

Consider two hospitals that store complementary data about the same patients. They want to train a machine learning model that predicts whether a patient has diabetes or not. As each hospital stores useful features to predict diabetes, they want to use all available information about each patient. But due to privacy reasons, they are not allowed to share the data with each other.

Using the Pima Indians Diabetes Database dataset, we split the dataset across two hospitals (clients). Each hospital stores a subset of the available features.

Centralized case

If the hospitals do not collaborate, each trains its own small neural network based only on its available features. This typically results in lower test accuracies (e.g., 0.68 and 0.7).

Collaboration through vertical FL

The two hospitals collaboratively train a model using all patient data via FEDn’s implementation of split learning. In this scenario, the test accuracy increases (to 0.72) because the hospitals leverage more features. While it remains slightly below a fully centralized training case (0.74), it significantly improves prediction accuracy over individual training.

Vertical FL in FEDn

Why standard FL fails in the vertical case

In the vertical case, clients store different features, and usually, only one client has the labels. Therefore:

  1. Loss cannot be calculated on the client side.
  2. Local model parameters cannot be updated.
  3. No parameter updates can be sent to the server for aggregation.
  4. Aggregating models trained on different features is not mathematically meaningful.

Split Learning

FEDn supports split learning with label sharing (also referred to as “SplitNN”). This implementation assumes labels are shared with the server. A neural network is split across participating clients and the server:

  1. Forward Pass: Clients perform a forward pass using their local network and send the resulting embedding to the server.
  2. Server Processing: The server concatenates the embeddings and continues the forward pass. Since the server holds the labels, it calculates the loss.
  3. Backward Pass: The server performs backpropagation and sends the obtained gradients back to the clients.
  4. Client Update: Clients continue backpropagation and update the parameters of their local partial neural networks.

This process is repeated for a set number of training rounds.

How to use Split Learning with FEDn

The diabetes example is implemented in the FEDn repository: splitlearning_diabetes.