Python Statistics

Statistics


  Machine Learning in Python

Table of Contents

Central Tendency

Mean

One of the most basic properties of a data set is it’s mean. Mean is a measure of the Central Tendency of data. In layman terms, if you were to pick one point in a dataset that is representative of the entire dataset, it would be the mean. Also called “average” in common vocabulary, calculating the mean is really simple. Just add up all the values in a particular variable of a dataset and divide it by the total number of values in it.

For example, if there are 5 Uber drivers with 5 different ratings, what is their mean ?

from statistics import mean

ratings = [4.5, 3.9, 4.6, 4.8, 3.9]

mean(ratings)
4.34

Median

Mean does not always represent the average of the data. For normal distributions it typically does. However, for many other data distributions, mean does not represent the average. For example, take the US income distribution data. Since the number of housholds is huge, the data has been bucketed into income brackets for easy analysis.

income_bracket    households
10-15k            5700000
15-20k            5620000
20-25k            5930000
25-30k            5500000
30-35k            5780000
35-40k            5340000
40-45k            5380000
45-50k            4730000
50-60k            9210000
60-75k           11900000
75-100k          14700000
100-125k         10300000
125-150k          6360000
150-200k          6920000
200k-plus         7600000

from statistics import mean, median

income = [5700000,5620000, 5930000, 5500000, 5780000, 5340000, 5380000, 
          4730000, 9210000, 11900000, 14700000, 10300000, 6360000, 6920000,7600000 ]

print ( "mean =", mean(income) )
print ( "median =", median(income) )
mean = 7398000
median = 5930000

As you can see, depending on the data distribution, mean and median could be totally different from each other.

Mode

Mode is just the highest value in the dataset. If the dataset follows a gaussian distribution, it is the peak of the histogram. Think of mode of a dataset as the most commonly occuring number.

from statistics import mode

ages = [12,13,14,11,12,13,15,10,13 ]

mode(ages)

13

Mean vs Median vs Mode

The reason why these terms exist is that, there is no one way to define the Central Tendency of a dataset. It depends on the nature of the distribution that the dataset conforms to. For example, if the dataset is a normal distribution, then most of the time the mean, median and mode are pretty close together. The relationship is beautifully visualized in this wikipedia diagram.

For a normal distribution, the 3 parameters ( mean, median, mode ) are pretty close together. For a skewed distribution, they are pretty staggered.

Spread

Standard Deviation

While mean represents the “Central” or “Average” value of a dataset, variance represents how spread out the data is. For example, look at the 3 histograms below. They represent histograms of GRE scores among 3 different groups.

All these 3 graphs represent 10K rows of GRE scores from 3 different groups. All 3 of them have the same mean – 300. However, they are different, right ? What I want you to focus on is the shape of the distribution, not the height. Specifically, look at the x-axis. In the first plot, the data is focussed, pretty much around the average(300) mark. In the second plot, it is a bit spread out ( hence the reduction in size ) and the third plot is pretty spread out.

The green line represents the mean and the red line represents the Standard Deviation or σ ( represented by the Greek symbol sigma ). It is a measure of how spread out the distribution is. The more spread out the distribution is, the more flatter the bell curve is.

You can also think of Standard Deviation as a measure of uncertianity in Data Science. Imagine a drug trail working on reducing blood pressure. Say, on an average 3 different trails produce an average reduction of 30 points. However, trail 1 has very low standard deviation. Obviously, you would want to go with the first drug – because the third drug is more uncertain about the result.

from statistics import stdev

income = [5700000,5620000, 5930000, 5500000, 5780000, 5340000, 5380000, 
          4730000, 9210000, 11900000, 14700000, 10300000, 6360000, 6920000,7600000 ]

stdev(income)

2884498.470890811

Variance

Standard Deviation is actually a derivative of variance. So, variance is calculated first. However, to illustrate that variance or standard deviation represents spread on the plot, we have learnt standard deviation first. However, we have to understand how variance is calculated, because this is the basis for calculating standard deviation .

Let’s calculate variance of a small dataset in excel to understand the process better.

variance_calculation

What does Standard Deviation represent

Now that we know the basic definitions of Variance and Standard Deviation, let’s look at what it really represents.

How far from the mean

Wikipedia has a nice picture to show this in the context of Gaussian distribution.

How spread out

Another way to look at the Standard Deviation is , how spread out the distribution is. Once again, there is a nice representation of this in Wikipedia.

The more the Standard Deviation , the more the distribution is spread out.

Why 2 different metrics to measure spread

Why are we using 2 different metircs (Variance & Standard Deviation) to measure the same parameter – spread ?

For one, you can only calculate Standard Deviation from _Mean. Also, Variance is used to measure how far the data is spread out, while Standard Deviation is used to measure how the data differs from its mean. However, since Standard Deviation is in the same metric as the data, it is used more often than Variance.

%d bloggers like this: