---
date: '2024-12-14'
description: machine learning approach combining multiple classifiers through bagging, random forests, and boosting to improve performance via voting procedures.
id: ensemble learning
modified: 2026-06-05 15:08:26 GMT-04:00
tags:
  - ml
title: ensemble learning
created: '2024-12-14'
published: '2024-12-14'
pageLayout: default
slug: thoughts/ensemble-learning
permalink: https://aarnphm.xyz/thoughts/ensemble-learning.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
idea: train multiple classifier and then combine them to improve performance.

aggregate their decisions via voting procedure.

Think of boosting, decision tree.

## bagging

_using non-overlapping training subset creates truly independent/diverse classifiers_

bagging is essentially bootstrap aggregating where we do random sampling with replacement.

## random forests

bagging but with random subspace methods [^random-subspace]

[^random-subspace]: The idea of training each classifier using a random subset of the feature sets. Also known as feature bagging

### decision tree

- handle categorical features

> \[!note\] Note
>
> can overfit easily with deeper tree.

## boosting

a greedier approach for reducing bias where we “pick base classifiers incrementally”.

we will train “weaker learner” and thus it can combined to become “stronger learner”.