Decision trees are a supervised learning algorithm often used in machine learning. Trees are a common analogy in everyday life. Shaped by a combination of roots, trunk, branches, and leaves, trees often symbolize growth. In machine learning, a decision tree is an algorithm that can create both classification and regression models.
The decision tree is so named because it starts at the root, like an upside-down tree, and branches off to demonstrate various outcomes. Because machine learning is based on the notion of solving problems, decision trees help us to visualize these models and adjust how we train them. What is a decision tree? A decision tree is a supervised learning algorithm that is used for classification and regression modeling. Regression is a method used for predictive modeling, so these trees are used to either classify data or predict what will come next.
Decision trees look like flowcharts, starting at the root node with a specific question of data, that leads to branches that hold potential answers. The branches then lead to decision (internal) nodes, which ask more questions that lead to more outcomes. This goes on until the data reaches what’s called a terminal (or “leaf”) node and ends.
In machine learning, there are four main methods of training algorithms: supervised, unsupervised, reinforcement learning, and semi-supervised learning. A decision tree helps us visualize how a supervised learning algorithm leads to specific outcomes. Types of decision trees in machine learning Decision trees in machine learning can either be classification trees or regression trees. Together, both types of algorithms fall into a category of “classification and regression trees” and are sometimes referred to as CART. Their respective roles are to “classify” and to “predict.”
- Classification trees Classification trees determine whether an event happened or didn’t happen. Usually, this involves a “yes” or “no” outcome.
We often use this type of decision-making in the real world. Here are a few examples to help contextualize how decision trees work for classification: Decision tree terminology These terms come up frequently in machine learning and are helpful to know as you embark on your machine learning journey:
Root node: The topmost node of a decision tree that represents the entire message or decision
Decision (or internal) node: A node within a decision tree where the prior node branches into two or more variables
Leaf (or terminal) node: The leaf node is also called the external node or terminal node, which means it has no child—it’s the last node in the decision tree and furthest from the root node
Splitting: The process of dividing a node into two or more nodes. It’s the part at which the decision branches off into variables
Pruning: The opposite of splitting, the process of going through and reducing the tree to only the most important nodes or outcomes
Example 1: How to spend your free time after work What you do after work in your free time can be dependent on the weather. If it is sunny, you might choose between having a picnic with a friend, grabbing a drink with a colleague, or running errands. If it is raining, you might opt to stay home and watch a movie instead. There is a clear outcome. In this case, that is classified as whether to “go out” or “stay in.” 2. Regression trees Regression trees, on the other hand, predict continuous values based on previous data or information sources. For example, they can predict the price of gasoline or whether a customer will purchase eggs (including which type of eggs and at which store).
This type of decision-making is more about programming algorithms to predict what is likely to happen, given previous behavior or trends.
Example 1: Housing prices in Colorado Regression analysis could be used to predict the price of a house in Colorado, which is plotted on a graph. The regression model can predict housing prices in the coming years using data points of what prices have been in previous years. This relationship is a linear regression since housing prices are expected to continue rising. Machine learning helps us predict specific prices based on a series of variables that have been true in the past.