# ============================================
# Title: Midterm Project — Data Wrangling and Cleansing
# Author: Abhishekdave9 (original dataset)
# Date: 22 April 2026
# Modified By: Hali Sarah Parsons
# Description: This notebook loads, inspects, and applies five data
# transformation and cleansing steps to the Digital Habits
# vs. Mental Health dataset sourced from Kaggle. The goal
# is to produce a clean, analysis-ready dataset.
# Source: https://www.kaggle.com/datasets/abhishekdave9/digital-habits-vs-mental-health-dataset
# ============================================
import pandas as pd
import numpy as np
Load the Dataset¶
# Load the raw dataset from the uploaded Excel file
df = pd.read_excel('digital_habits_vs_mental_health.xlsx')
print(f'Shape: {df.shape}')
print(f'Columns: {list(df.columns)}')
df.head()
Shape: (100000, 6) Columns: ['screen_time_hours', 'social_media_platforms_used', 'hours_on_TikTok', 'sleep_hours', 'stress_level', 'mood_score']
| screen_time_hours | social_media_platforms_used | hours_on_TikTok | sleep_hours | stress_level | mood_score | |
|---|---|---|---|---|---|---|
| 0 | 12.0 | 4 | 7.1 | 10.0 | 7 | 9 |
| 1 | 11.3 | 1 | 6.5 | 10.0 | 9 | 10 |
| 2 | 10.8 | 2 | 6.4 | 10.0 | 7 | 10 |
| 3 | 10.8 | 5 | 6.2 | 10.0 | 7 | 10 |
| 4 | 11.3 | 4 | 6.1 | 10.0 | 5 | 8 |
# Initial data quality overview — dtypes, null counts, duplicate count
print('--- Data Types ---')
print(df.dtypes)
print('\n--- Null Values ---')
print(df.isnull().sum())
print('\n--- Duplicate Rows ---')
print(f'{df.duplicated().sum()} duplicate rows found')
print('\n--- Descriptive Statistics ---')
df.describe()
--- Data Types --- screen_time_hours float64 social_media_platforms_used int64 hours_on_TikTok float64 sleep_hours float64 stress_level int64 mood_score int64 dtype: object --- Null Values --- screen_time_hours 0 social_media_platforms_used 0 hours_on_TikTok 0 sleep_hours 0 stress_level 0 mood_score 0 dtype: int64 --- Duplicate Rows --- 1865 duplicate rows found --- Descriptive Statistics ---
| screen_time_hours | social_media_platforms_used | hours_on_TikTok | sleep_hours | stress_level | mood_score | |
|---|---|---|---|---|---|---|
| count | 100000.000000 | 100000.000000 | 100000.000000 | 100000.000000 | 100000.000000 | 100000.00000 |
| mean | 6.003505 | 2.997820 | 2.401433 | 6.988489 | 6.179590 | 9.06076 |
| std | 1.987073 | 1.414452 | 1.079111 | 1.466330 | 2.045409 | 1.27847 |
| min | 1.000000 | 1.000000 | 0.200000 | 3.000000 | 1.000000 | 2.00000 |
| 25% | 4.700000 | 2.000000 | 1.600000 | 6.000000 | 5.000000 | 8.00000 |
| 50% | 6.000000 | 3.000000 | 2.300000 | 7.000000 | 6.000000 | 10.00000 |
| 75% | 7.300000 | 4.000000 | 3.100000 | 8.000000 | 8.000000 | 10.00000 |
| max | 12.000000 | 5.000000 | 7.200000 | 10.000000 | 10.000000 | 10.00000 |
Transformation 1 — Replace Headers with Readable Column Names¶
The original column names use underscores and abbreviations that are not immediately readable. Replacing them with cleaner, more descriptive labels makes the dataset easier to work with and communicate to others.
# Transformation 1: Rename columns to human-readable labels
column_rename_map = {
'screen_time_hours': 'Daily Screen Time (hrs)',
'social_media_platforms_used': 'Platforms Used',
'hours_on_TikTok': 'TikTok Usage (hrs)',
'sleep_hours': 'Sleep Duration (hrs)',
'stress_level': 'Stress Level (1-10)',
'mood_score': 'Mood Score (1-10)'
}
df.rename(columns=column_rename_map, inplace=True)
print('Columns after renaming:')
print(list(df.columns))
df.head()
Columns after renaming: ['Daily Screen Time (hrs)', 'Platforms Used', 'TikTok Usage (hrs)', 'Sleep Duration (hrs)', 'Stress Level (1-10)', 'Mood Score (1-10)']
| Daily Screen Time (hrs) | Platforms Used | TikTok Usage (hrs) | Sleep Duration (hrs) | Stress Level (1-10) | Mood Score (1-10) | |
|---|---|---|---|---|---|---|
| 0 | 12.0 | 4 | 7.1 | 10.0 | 7 | 9 |
| 1 | 11.3 | 1 | 6.5 | 10.0 | 9 | 10 |
| 2 | 10.8 | 2 | 6.4 | 10.0 | 7 | 10 |
| 3 | 10.8 | 5 | 6.2 | 10.0 | 7 | 10 |
| 4 | 11.3 | 4 | 6.1 | 10.0 | 5 | 8 |
Transformation 2 — Remove Duplicate Rows¶
The initial inspection found 1,865 duplicate rows in the dataset. Duplicate records can skew summary statistics and inflate sample counts. Removing them ensures each observation represents a unique respondent or data point.
# Transformation 2: Drop duplicate rows
rows_before = len(df)
df.drop_duplicates(inplace=True)
df.reset_index(drop=True, inplace=True)
rows_after = len(df)
print(f'Rows before: {rows_before}')
print(f'Rows after: {rows_after}')
print(f'Duplicates removed: {rows_before - rows_after}')
print(f'Duplicates remaining: {df.duplicated().sum()}')
Rows before: 100000 Rows after: 98135 Duplicates removed: 1865 Duplicates remaining: 0
Transformation 3 — Identify and Flag Outliers in Screen Time¶
Screen time values range from 1.0 to 12.0 hours per day. Using the IQR method, values that fall more than 1.5 × IQR above the third quartile or below the first quartile are flagged as potential outliers. Rather than dropping these rows, a boolean flag column is added so the outliers can be reviewed or filtered downstream without permanently removing data.
# Transformation 3: Flag outliers in Daily Screen Time using IQR method
col = 'Daily Screen Time (hrs)'
# Calculate IQR bounds
Q1 = df[col].quantile(0.25)
Q3 = df[col].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
print(f'Q1: {Q1}, Q3: {Q3}, IQR: {IQR}')
print(f'Lower bound: {lower_bound}, Upper bound: {upper_bound}')
# Add a flag column — True means the row is an outlier
df['Screen Time Outlier'] = (df[col] < lower_bound) | (df[col] > upper_bound)
print(f'\nOutlier rows flagged: {df["Screen Time Outlier"].sum()}')
df[df['Screen Time Outlier'] == True].head()
Q1: 4.7, Q3: 7.4, IQR: 2.7 Lower bound: 0.6499999999999995, Upper bound: 11.450000000000001 Outlier rows flagged: 299
| Daily Screen Time (hrs) | Platforms Used | TikTok Usage (hrs) | Sleep Duration (hrs) | Stress Level (1-10) | Mood Score (1-10) | Screen Time Outlier | |
|---|---|---|---|---|---|---|---|
| 0 | 12.0 | 4 | 7.1 | 10.0 | 7 | 9 | True |
| 15 | 12.0 | 3 | 5.3 | 10.0 | 7 | 10 | True |
| 25 | 11.8 | 2 | 5.2 | 10.0 | 5 | 10 | True |
| 33 | 12.0 | 5 | 5.1 | 10.0 | 7 | 10 | True |
| 44 | 12.0 | 4 | 4.9 | 10.0 | 5 | 9 | True |
Transformation 4 — Add a Derived Categorical Column: Sleep Quality Category¶
Sleep duration is currently stored as a continuous numeric value. Adding a categorical label — based on widely referenced sleep guidelines — makes the data more interpretable and creates a useful grouping variable for downstream analysis. Categories follow the National Sleep Foundation's recommended ranges for adults and teenagers.
# Transformation 4: Add Sleep Quality Category derived from Sleep Duration (hrs)
def categorize_sleep(hours):
"""Categorize sleep duration into quality labels based on NSF guidelines."""
if hours < 5:
return 'Severely Insufficient'
elif hours < 6:
return 'Insufficient'
elif hours < 7:
return 'Below Recommended'
elif hours <= 9:
return 'Recommended'
else:
return 'Excessive'
# Apply the function to create a new categorical column
df['Sleep Quality Category'] = df['Sleep Duration (hrs)'].apply(categorize_sleep)
print('Sleep Quality Category value counts:')
print(df['Sleep Quality Category'].value_counts())
df[['Sleep Duration (hrs)', 'Sleep Quality Category']].head(10)
Sleep Quality Category value counts: Sleep Quality Category Recommended 41532 Below Recommended 24039 Insufficient 15609 Excessive 8478 Severely Insufficient 8477 Name: count, dtype: int64
| Sleep Duration (hrs) | Sleep Quality Category | |
|---|---|---|
| 0 | 10.0 | Excessive |
| 1 | 10.0 | Excessive |
| 2 | 10.0 | Excessive |
| 3 | 10.0 | Excessive |
| 4 | 10.0 | Excessive |
| 5 | 10.0 | Excessive |
| 6 | 10.0 | Excessive |
| 7 | 10.0 | Excessive |
| 8 | 10.0 | Excessive |
| 9 | 10.0 | Excessive |
Transformation 5 — Round Continuous Columns to Consistent Decimal Places¶
Screen time and TikTok usage values contain varying decimal precision across rows. Standardizing both to one decimal place improves consistency and readability without losing meaningful precision — the difference between 10.800000 and 10.8 carries no analytical significance here.
# Transformation 5: Standardize decimal precision on float columns
df['Daily Screen Time (hrs)'] = df['Daily Screen Time (hrs)'].round(1)
df['TikTok Usage (hrs)'] = df['TikTok Usage (hrs)'].round(1)
df['Sleep Duration (hrs)'] = df['Sleep Duration (hrs)'].round(1)
print('Decimal precision standardized on float columns.')
df[['Daily Screen Time (hrs)', 'TikTok Usage (hrs)', 'Sleep Duration (hrs)']].head(10)
Decimal precision standardized on float columns.
| Daily Screen Time (hrs) | TikTok Usage (hrs) | Sleep Duration (hrs) | |
|---|---|---|---|
| 0 | 12.0 | 7.1 | 10.0 |
| 1 | 11.3 | 6.5 | 10.0 |
| 2 | 10.8 | 6.4 | 10.0 |
| 3 | 10.8 | 6.2 | 10.0 |
| 4 | 11.3 | 6.1 | 10.0 |
| 5 | 11.4 | 6.0 | 10.0 |
| 6 | 10.0 | 5.9 | 10.0 |
| 7 | 9.8 | 5.8 | 10.0 |
| 8 | 10.9 | 5.6 | 10.0 |
| 9 | 10.8 | 5.6 | 10.0 |
Final Cleansed Dataset — Top 5 Records¶
# Final cleansed dataset — top 5 records
print(f'Final dataset shape: {df.shape}')
print(f'Columns: {list(df.columns)}')
df.head()
Final dataset shape: (98135, 8) Columns: ['Daily Screen Time (hrs)', 'Platforms Used', 'TikTok Usage (hrs)', 'Sleep Duration (hrs)', 'Stress Level (1-10)', 'Mood Score (1-10)', 'Screen Time Outlier', 'Sleep Quality Category']
| Daily Screen Time (hrs) | Platforms Used | TikTok Usage (hrs) | Sleep Duration (hrs) | Stress Level (1-10) | Mood Score (1-10) | Screen Time Outlier | Sleep Quality Category | |
|---|---|---|---|---|---|---|---|---|
| 0 | 12.0 | 4 | 7.1 | 10.0 | 7 | 9 | True | Excessive |
| 1 | 11.3 | 1 | 6.5 | 10.0 | 9 | 10 | False | Excessive |
| 2 | 10.8 | 2 | 6.4 | 10.0 | 7 | 10 | False | Excessive |
| 3 | 10.8 | 5 | 6.2 | 10.0 | 7 | 10 | False | Excessive |
| 4 | 11.3 | 4 | 6.1 | 10.0 | 5 | 8 | False | Excessive |
# Final summary statistics on the cleansed dataset
df.describe()
| Daily Screen Time (hrs) | Platforms Used | TikTok Usage (hrs) | Sleep Duration (hrs) | Stress Level (1-10) | Mood Score (1-10) | |
|---|---|---|---|---|---|---|
| count | 98135.000000 | 98135.000000 | 98135.000000 | 98135.000000 | 98135.000000 | 98135.000000 |
| mean | 6.017178 | 2.998268 | 2.407408 | 6.978098 | 6.196698 | 9.048607 |
| std | 1.991190 | 1.414731 | 1.081405 | 1.469633 | 2.048782 | 1.283805 |
| min | 1.000000 | 1.000000 | 0.200000 | 3.000000 | 1.000000 | 2.000000 |
| 25% | 4.700000 | 2.000000 | 1.600000 | 6.000000 | 5.000000 | 8.000000 |
| 50% | 6.000000 | 3.000000 | 2.300000 | 7.000000 | 6.000000 | 10.000000 |
| 75% | 7.400000 | 4.000000 | 3.100000 | 8.000000 | 8.000000 | 10.000000 |
| max | 12.000000 | 5.000000 | 7.200000 | 10.000000 | 10.000000 | 10.000000 |
Written Analysis — Dataset, Transformations, and Ethical Implications¶
Dataset Overview
The dataset used for this project is the Digital Habits vs. Mental Health dataset, sourced from Kaggle (Dave, 2024). It contains 100,000 rows and six columns — daily screen time in hours, number of social media platforms used, hours spent on TikTok specifically, sleep duration, a self-reported stress level on a 1–10 scale, and a mood score on the same scale. The subject matter connects directly to my BU 360 term project, which is building a custom Named Entity Recognition (NER) system to extract entities from screen-time discourse. That project works with text; this dataset provides a quantitative view of the same domain. Together they represent two different angles on the same public health conversation.
Transformation Steps
Five transformations were applied to the raw dataset. First, all six column headers were renamed to human-readable labels with units included — for example, screen_time_hours became Daily Screen Time (hrs). This makes the dataset immediately interpretable without needing to reference separate documentation. Second, 1,865 duplicate rows were identified and removed using pandas' drop_duplicates(). Retaining duplicates would inflate row counts and skew descriptive statistics. Third, outliers in the daily screen time column were identified using the interquartile range method and flagged with a boolean column rather than dropped. Flagging instead of deleting preserves the original data while marking rows that may need special handling in analysis. Fourth, a new categorical column — Sleep Quality Category — was derived from the continuous sleep duration column using labels aligned with National Sleep Foundation guidelines (Severely Insufficient, Insufficient, Below Recommended, Recommended, Excessive). Adding a categorical grouping variable makes the sleep data more useful for segmentation. Fifth, the three float columns were rounded to one decimal place to standardize precision across the dataset.
Ethical Implications
Several ethical considerations apply here. On sourcing: the dataset was downloaded from Kaggle, a public data repository. The original Kaggle listing does not provide detailed documentation of how the data was collected — whether through a survey, a simulation, or another method — which limits the ability to fully verify its credibility or confirm informed consent was obtained from any participants. This is a real limitation worth acknowledging, and it points to a broader issue in public datasets: provenance is often unclear even when the file itself is freely available. For this assignment, the dataset is treated as appropriate for academic practice; for any production use it would require additional verification.
On legal and regulatory considerations: because the dataset includes stress and mood scores alongside digital behavior, it edges into health-adjacent data. In a production setting, data of this type — particularly if linked to identifiable individuals — could fall under HIPAA or other health data protections depending on how it was collected. No such identifiers exist in this dataset, so that risk is low here but worth flagging conceptually.
On transformation risks: removing duplicates assumed that duplicate rows represent data entry errors rather than genuinely repeated observations from different individuals with identical values. That assumption may not always hold. The outlier flagging step assumed that IQR-based bounds are the appropriate threshold for this variable, which is a judgment call. The sleep categorization step applied adult-focused guidelines from the National Sleep Foundation; if the dataset includes adolescents, the recommended ranges differ and the labels could misclassify some rows.
To mitigate these risks, the outlier rows were flagged rather than deleted, preserving the original records for review. The sleep categories are labeled clearly and their source is documented, so any analyst using the column downstream can evaluate whether the thresholds are appropriate for their specific use case.
References
Dave, A. (2024). Digital habits vs. mental health dataset [Data set]. Kaggle. https://www.kaggle.com/datasets/abhishekdave9/digital-habits-vs-mental-health-dataset
McGregor, S. E. (2022). Practical Python data wrangling and data quality. O'Reilly Media. https://www.oreilly.com/library/view/practical-python-data/9781492091493/
National Sleep Foundation. (2023). How much sleep do you really need? https://www.thensf.org/how-many-hours-of-sleep-do-you-really-need/