Suggest a number of clusters

Computes a cluster-quality diagnostic across a range of k and draws the diagnostic curve, with the suggested k highlighted. Three criteria are available: the average silhouette width (maximised), the total within-cluster sum of squares "elbow" (the point of maximum curvature), and the gap statistic (the smallest k whose gap is within one standard error of the next, using the Tibshirani et al. heuristic).

Usage

k_diagnostic(
  data,
  k_range = NULL,
  method = c("silhouette", "wss", "gap"),
  cols = NULL,
  scale = TRUE,
  nstart = 10,
  B = 50,
  title = NULL
)

Arguments

data

A data frame or numeric matrix.

k_range

Integer vector of candidate cluster counts to evaluate. The default, NULL, searches 1:8 for "gap" and "wss" and 2:8 for "silhouette". Candidates the chosen criterion cannot evaluate are skipped with a message naming them: k = 1 under "silhouette" (a lone cluster has no neighbouring cluster to compare against, so the width is undefined), and any k at or above the number of observations under every criterion.

method

Diagnostic: "silhouette", "wss" (within sum of squares elbow) or "gap".

cols

When data is a data frame, the numeric columns to use.

scale

Whether to scale variables to unit variance first. Columns with (near-)zero variance cannot be scaled and are dropped with a message, as in correlation_heatmap().

nstart

Number of random starts for stats::kmeans().

B

Number of reference bootstrap samples for the gap statistic.

title

Plot title.

Value

A ggplot2::ggplot object showing the diagnostic value against k, with the suggested k marked and named in the subtitle. The underlying computation is attached as attributes: attr(p, "k_table") (a data frame with one row per k), attr(p, "suggested") (the chosen k) and attr(p, "method").

Details

Only the gap statistic can conclude that the data hold no clusters, by suggesting k = 1, and only when the search starts at 1, as it does by default. The elbow is the candidate farthest from the straight line joining the first and last points of the curve, so it is never the first or last candidate. A range therefore has to extend one step beyond any k the elbow should be able to choose.

References

Rousseeuw PJ (1987). “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.” Journal of Computational and Applied Mathematics, 20, 53–65. doi:10.1016/0377-0427(87)90125-7 .

Tibshirani R, Walther G, Hastie T (2001). “Estimating the number of clusters in a data set via the gap statistic.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(2), 411–423. doi:10.1111/1467-9868.00293 .

Examples

p <- k_diagnostic(crop_yield, k_range = 2:6,
                  cols = c("rainfall", "fertiliser", "soil_ph", "yield"))
attr(p, "suggested")
#> [1] 2