How did you eliminate outliers in R


  R Interview Questions

What is an outlier ?

Outliers are data points that fall way out of the normal data range. For example, let’s take 3 sensors measuring temperature in Chicago on a winter day.

s_1 = round(rnorm(7,5,5),1)
s_2 = round(rnorm(7,5,5),1)
s_3 = round(rnorm(7,5,5),1)

Say one of the readings in one of the sensors malfunctioned.

# Inject a faulty recording
s_3[7] = -100

Let’s visualize the data

# Temperature in Chicago on a wintery day
s_data = c( s_1 , s_2, s_3 )
# Let's visualize
hist(s_data, breaks = 10)

As you can see, the distribution is highly skewed. The outlier is disturbing the nature of the distribution.

There are a couple of methods to remove outliers, and we are going to use the most intuitive and simplest method – Tukey’s method of removing anything > 1.5 IQR.

# Use Tukey's boxplot to get outlisers ( > 1.5 IQR )
> b_before = boxplot(s_data)

See, what is close to a normal distribution ( more apparent after the cleanse ) looks so skewed because of the outlier.

# Get the outlisers
> b_before$out 
> b_before$out
[1]   13.1 -100.0

And the outliers are given in the out variable of the boxplot.

Let’s remove the outliers.

# Remove the outliers
> s_data_normalize = ifelse(s_data %in% b_before$out, NA, s_data)

Let’s do another boxplot now and see if things changed.

# box plot again - Looking much better. 
b_after = boxplot(s_data_normalize)

You can see that the new histogram is much better as well.

> hist(s_data_normalize,breaks=20)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.