Results · Analysis #8 — Data Scientist Agent 2.0

Objective not fulfilled

The objective was not met. F1 of 0.4098 is too low to rely on for decisions. Success criterion: A model that clearly beats the naive baseline on F1 and behaves sensibly on the outcome the objective cares about.

Predictive quality
F1 of 0.4098 is too low to rely on for decisions.
Beats the naive baseline
The trivial baseline scores 0.3012 and the chosen model scores 0.4098 - an improvement of 0.1086.
Catches the minority class
It identifies just 15.4% of the 'Premium' cases, so it is close to useless for the decision this objective implies.
Stable across folds
5-fold cross-validation gives 0.4097 (± 0.0482) against a test score of 0.4098, so the result is consistent.

Best model

Logistic Regression

of 6 that trained

F1

0.4098

baseline 0.301

Data quality

93/100

before cleaning

Features used

22

22 engineered, 5 removed

Dataset overview

Rows500

Columns11

Numeric columns3

Categorical columns5

Date columns3

Duplicate rows0

Missing cells398 (7.24%)

Size in memory0.23 MB

Columns the agent found

first 25 shown

Column	Detected as	Type	Missing	Unique	Examples
`CustomerID`	numeric	int64	0.0%	500	1, 2, 3
`Name`	categorical	object	4.0%	477	Mark Cruz, Mr. Christopher Maldonado DDS, Robert Black
`DateOfBirth`	datetime	object	6.0%	465	09-13-1979, 04-27-2004, 06-22-1973
`SignupDate`	datetime	object	0.0%	272	2020-05-04, 2020-08-21, 2020-10-09
`LastPurchaseDate`	datetime	object	0.0%	277	2021-03-22, 2021-07-09, 2021-06-09
`PurchaseAmount`	numeric	float64	10.0%	449	495.03, 734.67, 662.94
`Currency`	categorical	object	0.0%	3	GBP, USD, EUR
`LoyaltyPoints`	numeric	float64	6.0%	364	501.0, 958.0, 144.0
`Region`	categorical	object	26.2%	8	West, South, East
`Email`	categorical	object	12.0%	433	m.christopher@gmail.com, r.black@gmail.com, d.burnett@yahoo.com
`SubscriptionType`	categorical	object	15.4%	3	Standard, Basic, Premium

Data quality

score 93/100

medium 7.24% of all cells are missing (398 cells).
low 1 identifier-like column(s) detected and excluded from modelling: Name.
medium High-cardinality categorical column(s) that will be grouped before encoding: Email.

Cleaning summary

3 action(s) taken

Before cleaning

Rows500

Columns11

Missing cells398

Duplicates0

After cleaning

Rows500

Columns10

Missing cells378

Duplicates0

What was done

Unified category casing. Merged case-variant categories (for example 'Male' and 'male') in 1 column(s).
Parsed date columns. Converted 3 text column(s) into real datetime columns so that calendar features can be derived from them.
Dropped identifier columns. Looks like a row identifier. Identifiers are unique per row and would let a model memorise rows instead of learning a pattern.

Remaining missing values are imputed inside the modelling pipeline (numeric columns with the median, categorical columns with the most frequent value). The imputers are fitted on the training split only, so no information from the test set leaks into training.

Problem framing and features

split before engineering, so nothing leaks

Problem type and target

Problem typeclassification

Target columnSubscriptionType

Confidence100%

MetricF1

Training rows338

Test rows85

The target is non-numeric with 3 distinct classes.

Using F1 as recommended by the AI analyst for this objective.

Stratified 80/20 split with random_state=42, so both splits keep the same class balance.

Features kept

CustomerIDPurchaseAmountCurrencyLoyaltyPointsRegionEmailDateOfBirth_yearDateOfBirth_monthDateOfBirth_dayDateOfBirth_weekdaySignupDate_daySignupDate_weekdaySignupDate_quarterLastPurchaseDate_monthLastPurchaseDate_dayLastPurchaseDate_weekdayCustomerID_per_PurchaseAmountCustomerID_per_LoyaltyPointsPurchaseAmount_per_LoyaltyPointsCustomerID_minus_PurchaseAmountCustomerID_minus_LoyaltyPointsPurchaseAmount_minus_LoyaltyPoints

Strongest signal against the target

DateOfBirth_year1.0

LastPurchaseDate_day0.5969

CustomerID0.4665

CustomerID_per_PurchaseAmount0.4176

PurchaseAmount0.353

PurchaseAmount_per_LoyaltyPoints0.3408

Engineered features

22

Feature	Type	Why
`DateOfBirth_year`	datetime part	A model cannot read a raw timestamp, so DateOfBirth was split into its calendar parts, which let it learn seasonal and trend patterns.
`DateOfBirth_month`	datetime part	Calendar 'month' taken from DateOfBirth.
`DateOfBirth_day`	datetime part	Calendar 'day' taken from DateOfBirth.
`DateOfBirth_weekday`	datetime part	Calendar 'weekday' taken from DateOfBirth.
`DateOfBirth_quarter`	datetime part	Calendar 'quarter' taken from DateOfBirth.
`SignupDate_year`	datetime part	A model cannot read a raw timestamp, so SignupDate was split into its calendar parts, which let it learn seasonal and trend patterns.
`SignupDate_month`	datetime part	Calendar 'month' taken from SignupDate.
`SignupDate_day`	datetime part	Calendar 'day' taken from SignupDate.
`SignupDate_weekday`	datetime part	Calendar 'weekday' taken from SignupDate.
`SignupDate_quarter`	datetime part	Calendar 'quarter' taken from SignupDate.
`LastPurchaseDate_year`	datetime part	A model cannot read a raw timestamp, so LastPurchaseDate was split into its calendar parts, which let it learn seasonal and trend patterns.
`LastPurchaseDate_month`	datetime part	Calendar 'month' taken from LastPurchaseDate.
`LastPurchaseDate_day`	datetime part	Calendar 'day' taken from LastPurchaseDate.
`LastPurchaseDate_weekday`	datetime part	Calendar 'weekday' taken from LastPurchaseDate.
`LastPurchaseDate_quarter`	datetime part	Calendar 'quarter' taken from LastPurchaseDate.
`CustomerID_per_PurchaseAmount`	ratio	Ratio of two of the most informative numeric columns. Ratios capture relative size, which raw values on their own cannot.
`CustomerID_minus_PurchaseAmount`	difference	Gap between CustomerID and PurchaseAmount; differences often matter more than either value alone.
`CustomerID_per_LoyaltyPoints`	ratio	Ratio of two of the most informative numeric columns. Ratios capture relative size, which raw values on their own cannot.
`CustomerID_minus_LoyaltyPoints`	difference	Gap between CustomerID and LoyaltyPoints; differences often matter more than either value alone.
`PurchaseAmount_per_LoyaltyPoints`	ratio	Ratio of two of the most informative numeric columns. Ratios capture relative size, which raw values on their own cannot.
`PurchaseAmount_minus_LoyaltyPoints`	difference	Gap between PurchaseAmount and LoyaltyPoints; differences often matter more than either value alone.
`Email (grouped)`	rare category grouping	290 rare category value(s) in Email were merged into 'Other'. Encoding each of them would create many near-empty columns that only add noise.

Removed features

5

Feature	Reason
`SignupDate_year`	Only one distinct value, so it cannot explain any variation.
`LastPurchaseDate_year`	Only one distinct value, so it cannot explain any variation.
`DateOfBirth_quarter`	Correlated 0.972 with DateOfBirth_month, which carries the same information and scores higher against the target.
`SignupDate_month`	Correlated 0.973 with SignupDate_quarter, which carries the same information and scores higher against the target.
`LastPurchaseDate_quarter`	Correlated 0.975 with LastPurchaseDate_month, which carries the same information and scores higher against the target.

Models trained and compared

ranked on F1, not on accuracy by default

How the shortlist was chosen

The AI layer suggested model names; the agent resolved each one against a fixed library of hand-written models and refused anything that did not fit.

Logistic RegressionDecision Tree ClassifierRandom Forest ClassifierExtra Trees ClassifierK-Nearest Neighbors ClassifierBaseline (Most Frequent Class)

Model	Accuracy	Precision	Recall	F1	ROC-AUC	Time (s)
Logistic Regression best	0.4353	0.4305	0.4353	0.4098	0.5129	0.072
Decision Tree Classifier	0.4118	0.4012	0.4118	0.3951	0.4847	0.039
Random Forest Classifier	0.3765	0.3556	0.3765	0.3226	0.4306	0.489
Extra Trees Classifier	0.3647	0.2853	0.3647	0.3077	0.4213	0.366
K-Nearest Neighbors Classifier	0.3765	0.3408	0.3765	0.3432	0.4632	0.054
Baseline (Most Frequent Class) baseline	0.4706	0.2215	0.4706	0.3012	0.5000	0.032

The selected model

Logistic Regression

F1 0.4098

Logistic Regression achieved the best F1 of 0.4098, ahead of Decision Tree Classifier (0.3951) by 0.0147. F1 was chosen because it matches what the objective is asking for.

Linear, fast and highly interpretable - the sensible starting point.

Accuracy 0.4353

Precision 0.4305

Recall 0.4353

F1 0.4098

Roc Auc 0.5129

Naive baseline 0.3012

5-fold cross-validation 0.4097 ± 0.0482

Performance on each class

a high overall score can still hide a class the model never finds

Class	Rows in test set	Correctly identified
`Basic`	40	65.0%
`Premium`	13	15.4%
`Standard`	32	28.1%

What drives the predictions

Absolute model coefficients

The size of each feature's coefficient after scaling. Larger means a one-unit change in that feature moves the prediction more.

#	Feature	Share
1	`CustomerID_minus_LoyaltyPoints`	12.15%
2	`LoyaltyPoints`	10.29%
3	`PurchaseAmount`	7.39%
4	`CustomerID_minus_PurchaseAmount`	5.99%
5	`Region_east`	5.66%
6	`CustomerID_per_LoyaltyPoints`	5.5%
7	`CustomerID`	5.36%
8	`Region_north`	4.02%
9	`Currency_USD`	3.96%
10	`LastPurchaseDate_weekday`	3.9%

Insights and next steps

Key insights

Cleaning removed 0 row(s) and 1 column(s) that could not contribute to a reliable model.
The winning model's decisions are driven mainly by CustomerID_minus_LoyaltyPoints (12%), LoyaltyPoints (10%), PurchaseAmount (7%).
The model is weakest on the 'Premium' class, catching only 15% of those cases. If that class is the one the business cares about, this is the number to improve.
The objective was not met. F1 of 0.4098 is too low to rely on for decisions. Success criterion: A model that clearly beats the naive baseline on F1 and behaves sensibly on the outcome the objective cares about.

Recommended next steps

Collect more data. 500 rows is a small sample, and the score above could move considerably on a larger one.
Try adjusting the decision threshold rather than the model. Moving it away from 0.5 trades precision for recall and often fixes a model that is accurate but never flags the class you care about.
If the classes are imbalanced, resampling the training data or class weighting is usually a bigger win than switching algorithms.
Tune Logistic Regression with a grid or randomised search. This run used sensible defaults with no hyperparameter search, so there is headroom left.
Review the 5 removed feature(s) with a domain expert. Automated selection is statistical; it does not know which columns matter to the business.
Before deploying, re-test on data from a later time period than the training data. A random split cannot tell you whether the pattern holds next quarter.

The built-in analyst's reasoning

'SubscriptionType' scored highest as a target candidate because last column in the file; 3 distinct classes. The objective wording suggests a classification task. Identifier and constant columns were excluded because they cannot generalise. This recommendation was produced by the built-in rule-based analyst rather than Gemini.

Success criterion it set: A model that clearly beats the naive baseline on F1 and behaves sensibly on the outcome the objective cares about.

Caveats from this run

Gemini was not used for this run: google-genai package is not installed.
77 row(s) had no value for 'SubscriptionType' and were removed. A missing label cannot be imputed without inventing the answer.

The written report goes further than this dashboard

Executive summary, every decision explained, the full metric tables, limitations and recommended next steps — written for technical and non-technical readers at once.

Open the full report