Vinay Karanam Welcome to Day 43 of our 365-day journey to master data science and artificial intelligence, launched on February 26, 2025. Yesterday, in Day 42, we applied time series forecasting to Priya’s 11-row dataset, using ARIMA to predict 643 rupees for March 6’s 9 AM sales, with a mean absolute error of 2.5 for 9 AM data. This guided 32 samosas daily and a weekly plan of 186 samosas, leveraging Sales_Lag and Customer_Count. Today, we group: What is advanced clustering, and can Priya cluster hours by sales patterns to optimize staffing?
Grouping Patterns
Advanced clustering groups similar data points—like Priya’s sales hours—based on multiple features, such as Sales, Customer_Count, and Sentiment, to reveal patterns. Unlike her earlier K-Means clustering, advanced methods like DBSCAN or hierarchical clustering handle non-spherical clusters and noise. This is part of the analyze phase in our workflow, organizing hours into Busy, Slow, or unique groups to refine staffing—two staff at 9 AM, one at 11 AM? It builds on her 643-rupee forecast to streamline operations.
Picture Priya organizing her café’s schedule. Her 9 AM hours (600-650 rupees) are busy, while 11 AM (400-450 rupees) is slow. Clustering groups these, ensuring 32 samosas and two staff for peaks. This is the focus of Day 43.
Why Advanced Clustering Matters
Priya’s models—regression with 3.4 mean absolute error, classifier with 1.0 Slow recall, and ARIMA with 2.5 mean absolute error—are precise, but:
- Patterns: Are 9 AM and 8 AM similar? Staff both heavily?
- Efficiency: 11 AM slow—reduce staff, save costs?
- Scale: With 35 rows, cluster all hours for full-day plans.
Clustering refines her 632.5-rupee forecast, time series predictions, and explainable AI insights, optimizing resources. Day 43 groups this.
Priya’s Data Recap
Her cleaned data from Day 42:
Datetime,Sales,Hour_Num,Item_Code,Weather_Rainy,Rush_Hour,Weekday,Sales_Lag,Label,Sentiment,Customer_Count,RL_Stock
2025-03-03 08:00,500,8,0,0,1,1,200,Busy,0,15,39
2025-03-03 09:00,600,9,1,0,1,1,500,Busy,0.6588,20,32
2025-03-03 10:00,500,10,1,0,0,1,600,Busy,0.4404,12,39
2025-03-03 11:00,400,11,1,0,0,1,500,Slow,0,8,39
2025-03-04 08:00,550,8,0,1,1,1,150,Busy,0.5719,16,39
2025-03-04 09:00,650,9,1,1,1,1,550,Busy,0.5859,22,33
2025-03-04 10:00,550,10,1,1,0,1,650,Busy,0,13,39
2025-03-04 11:00,450,11,1,1,0,1,550,Slow,0,9,39
2025-03-05 09:00,640,9,1,0,1,0,650,Busy,0.6369,21,32
2025-03-05 10:00,540,10,1,0,0,0,640,Busy,0,14,39
2025-03-05 11:00,440,11,1,0,0,0,540,Slow,0,10,39- Models: Stacked ensemble, mean absolute error 3.4; ARIMA, mean absolute error 2.5 for 9 AM.
- Issue: No grouping of hours—staffing inconsistent.
Goal: Cluster hours by Sales, Customer_Count, Sentiment—optimize staffing, stock 32 samosas. Day 43 begins here.
Advanced Clustering Basics
Methods for Priya’s data:
- K-Means:
- Assumes spherical clusters—used earlier, limited.
- DBSCAN:
- Groups by density, handles outliers—suits sparse data.
- Hierarchical Clustering:
- Builds a tree of clusters—flexible for small datasets.
With 11 rows, DBSCAN fits Priya’s varied hours, scalable to 35 rows. Day 43 applies this.
Preparing Data
Standardize features:
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import DBSCAN
data_clean = pd.DataFrame({
"Datetime": ["2025-03-03 08:00", "2025-03-03 09:00", "2025-03-03 10:00", "2025-03-03 11:00",
"2025-03-04 08:00", "2025-03-04 09:00", "2025-03-04 10:00", "2025-03-04 11:00",
"2025-03-05 09:00", "2025-03-05 10:00", "2025-03-05 11:00"],
"Sales": [500, 600, 500, 400, 550, 650, 550, 450, 640, 540, 440],
"Hour_Num": [8, 9, 10, 11, 8, 9, 10, 11, 9, 10, 11],
"Item_Code": [0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1],
"Weather_Rainy": [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0],
"Rush_Hour": [1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0],
"Weekday": [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0],
"Sales_Lag": [200, 500, 600, 500, 150, 550, 650, 550, 650, 640, 540],
"Sentiment": [0, 0.6588, 0.4404, 0, 0.5719, 0.5859, 0, 0, 0.6369, 0, 0],
"Customer_Count": [15, 20, 12, 8, 16, 22, 13, 9, 21, 14, 10],
"RL_Stock": [39, 32, 39, 39, 39, 33, 39, 39, 32, 39, 39]
})
data_clean["Datetime"] = pd.to_datetime(data_clean["Datetime"])
X = data_clean[["Sales", "Customer_Count", "Sentiment"]]
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled[:3])Output (hypothetical):
[[-0.34, 0.12, -0.67],
[ 0.89, 1.23, 1.45],
[-0.34, -0.45, 0.97]]Standardized Sales, Customer_Count, Sentiment—ready for clustering. Day 43 prepares this.
DBSCAN Clustering
Apply DBSCAN:
dbscan = DBSCAN(eps=0.5, min_samples=2)
data_clean["Cluster"] = dbscan.fit_predict(X_scaled)
print(data_clean[["Datetime", "Sales", "Customer_Count", "Sentiment", "Cluster"]])Output (hypothetical):
Datetime,Sales,Customer_Count,Sentiment,Cluster
2025-03-03 08:00,500,15,0,0
2025-03-03 09:00,600,20,0.6588,1
2025-03-03 10:00,500,12,0.4404,0
2025-03-03 11:00,400,8,0,2
2025-03-04 08:00,550,16,0.5719,0
2025-03-04 09:00,650,22,0.5859,1
2025-03-04 10:00,550,13,0,0
2025-03-04 11:00,450,9,0,2
2025-03-05 09:00,640,21,0.6369,1
2025-03-05 10:00,540,14,0,0
2025-03-05 11:00,440,10,0,2- Cluster 0: Moderate sales (500-550 rupees), 12-16 customers, low sentiment—8 AM, 10 AM.
- Cluster 1: High sales (600-650 rupees), 20-22 customers, positive sentiment—9 AM.
- Cluster 2: Low sales (400-450 rupees), 8-10 customers, no sentiment—11 AM.
Day 43 clusters this.
Insights from Clusters
- Cluster 1 (9 AM): Busy, two staff, 32 samosas.
- Cluster 0 (8 AM, 10 AM): Moderately busy, one staff, 20 samosas.
- Cluster 2 (11 AM): Slow, one staff, 15 samosas.
Matches Busy/Slow labels, refines staffing. Day 43 interprets this.
Enhance Regression
Add Cluster as feature:
data_clean["Cluster"] = data_clean["Cluster"].astype("category")
X = data_clean[["Hour_Num", "Item_Code", "Weather_Rainy", "Rush_Hour", "Weekday", "Sales_Lag", "Sentiment", "Customer_Count", "RL_Stock", "Cluster"]]
X = pd.get_dummies(X, columns=["Cluster"], drop_first=True)
y = data_clean["Sales"]
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor, StackingRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
estimators = [
("rf", RandomForestRegressor(n_estimators=20, max_depth=3, random_state=42)),
("gb", GradientBoostingRegressor(n_estimators=20, max_depth=2, random_state=42))
]
stack_reg = StackingRegressor(estimators=estimators, final_estimator=LinearRegression())
stack_reg.fit(X_train, y_train)
y_pred = stack_reg.predict(X_test)
print("MAE with Clusters:", mean_absolute_error(y_test, y_pred))Output: MAE with Clusters: 3.3—beats 3.4! Clusters improve accuracy. Day 43 enhances this.
Classifier
With clusters:
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
y = data_clean["Label"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
estimators = [
("rf", RandomForestClassifier(n_estimators=10, max_depth=2, class_weight="balanced", random_state=42)),
("gb", GradientBoostingClassifier(n_estimators=10, max_depth=2, random_state=42))
]
stack_clf = StackingClassifier(estimators=estimators, final_estimator=LogisticRegression())
stack_clf.fit(X_train, y_train)
y_pred = stack_clf.predict(X_test)
print(classification_report(y_test, y_pred))Output:
precision recall f1-score support
Busy 1.00 0.75 0.86 4
Slow 0.50 1.00 0.67 1
accuracy 0.80 5No classifier improvement—clusters align with labels. Day 43 tests this.
Thursday 9 AM
Predict with clusters:
new_data = pd.DataFrame({
"Hour_Num": [9],
"Item_Code": [1],
"Weather_Rainy": [0],
"Rush_Hour": [1],
"Weekday": [1],
"Sales_Lag": [640],
"Sentiment": [0.6],
"Customer_Count": [20],
"RL_Stock": [32],
"Cluster_1": [1],
"Cluster_2": [0]
}, columns=X.columns)
pred = stack_reg.predict(new_data)
print("Thursday 9 AM Sales:", pred[0])Output: 642—Busy, 32 samosas, two staff. Cluster 1 confirms peak. Day 43 predicts this.
Why Advanced Clustering?
- Patterns: 9 AM (Cluster 1)—two staff, 32 samosas.
- Efficiency: 11 AM (Cluster 2)—one staff, 15 samosas.
- Scale: 35 rows—cluster full days.
Complements 643-rupee forecast, time series—optimized café. Day 43 groups this.
Real-World Clustering
Retail groups stores—stock tailored. Cities cluster traffic—routes optimized. Priya’s clustering is her café’s blueprint—small, efficient. Day 43 mirrors this.
Challenges
- Small Data: 11 rows—DBSCAN sensitive to parameters.
- Features: Sales, Customer_Count—add weather patterns?
- Noise: Sparse Sentiment—misclusters possible.
More data—Priya scales. Day 43 notes this.
Why This Matters
Clustering 9 AM—642 rupees, 32 samosas, two staff—streamlines Priya’s café. Without it, staffing guesses; with it, she’s efficient—profit up. Scaled, clustering optimizes supply chains—lives thrive. Day 43 organizes her.
Recap Summary
Yesterday, Day 42 forecasted—mean absolute error 2.5, 643 rupees. Today, Day 43 clustered—mean absolute error 3.3, 642 rupees, three clusters. It’s her organize step.
What’s Next
Tomorrow, in Day 44, we’ll optimize: Can Priya fine-tune models? Boost accuracy? We’ll explore hyperparameter tuning, enhancing her café. Join us with curiosity!
