scikit-learn 모델 학습 에러 해결 가이드

scikit-learn은 Python의 대표적인 머신러닝 라이브러리지만, 모델 학습 과정에서 다양한 에러가 발생할 수 있습니다. 이 글에서는 흔한 에러 패턴과 해결 방법을 다룹니다.

Environment

$ python --version
Python 3.11.5

$ pip show scikit-learn
Name: scikit-learn
Version: 1.3.2

$ pip show numpy
Name: NumPy
Version: 1.26.2

$ pip show pandas
Name: pandas
Version: 2.1.4

Problem: ValueError: Input Contains NaN

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Sample data with missing values
data = pd.DataFrame({
    'age': [25, 30, None, 35, 40],
    'income': [50000, 60000, 75000, None, 90000],
    'target': [0, 1, 0, 1, 0]
})

X = data[['age', 'income']]
y = data['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = RandomForestClassifier()
model.fit(X_train, y_train)

에러 출력:

ValueError: Input contains NaN

Analysis: 데이터 품질 문제 분석

데이터의 결측값을 분석합니다:

import pandas as pd
import numpy as np

# Check for NaN values
print("NaN count per column:")
print(data.isnull().sum())

print("\nNaN percentage:")
print(data.isnull().mean() * 100)

# Check data types
print("\nData types:")
print(data.dtypes)

# Statistical summary
print("\nStatistical summary:")
print(data.describe())

출력:

NaN count per column:
age        1
income     1
target     0

NaN percentage:
age       20.0
income    20.0
target     0.0

Data types:
age       float64
income    float64
target      int64

Solution: 데이터 전처리 및 에러 해결

방법 1: SimpleImputer로 결측값 처리

import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

data = pd.DataFrame({
    'age': [25, 30, None, 35, 40, 28, 33, None, 45, 50],
    'income': [50000, 60000, 75000, None, 90000, 55000, None, 85000, 95000, 100000],
    'target': [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
})

X = data[['age', 'income']]
y = data['target']

# Split first
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Impute missing values
imputer = SimpleImputer(strategy='mean')
X_train_imputed = imputer.fit_transform(X_train)
X_test_imputed = imputer.transform(X_test)

# Now training works
model = RandomForestClassifier(random_state=42)
model.fit(X_train_imputed, y_train)

print(f"Training score: {model.score(X_train_imputed, y_train):.2f}")
print(f"Test score: {model.score(X_test_imputed, y_test):.2f}")

방법 2: Pipeline으로 처리

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

# Create pipeline
pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(random_state=42))
])

# Cross-validation works with missing data handling
scores = cross_val_score(pipeline, X, y, cv=5, scoring='accuracy')
print(f"CV Accuracy: {scores.mean():.2f} (+/- {scores.std():.2f})")

방법 3: DataFrameMapper로 열별 처리

from sklearn_pandas import DataFrameMapper
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler

# Different strategies per column
mapper = DataFrameMapper([
    (['age'], SimpleImputer(strategy='median'), {'alias': 'age_imputed'}),
    (['income'], SimpleImputer(strategy='mean'), {'alias': 'income_imputed'}),
])

X_transformed = mapper.fit_transform(X)
print(f"Shape after transformation: {X_transformed.shape}")

Other Common Errors and Solutions

ValueError: Unknown label type

# Problem
y = ['low', 'medium', 'high', 'low', 'high']
model.fit(X, y)  # May fail with some estimators

# Solution: Label encoding
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
y_encoded = le.fit_transform(y)
model.fit(X, y_encoded)

ConvergenceWarning

# Problem: Model not converging
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=100)
model.fit(X_train, y_train)
# ConvergenceWarning: lbfgs failed to converge

# Solution: Increase max_iter or scale data
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

model = LogisticRegression(max_iter=1000)
model.fit(X_train_scaled, y_train)

DataConversionWarning

import warnings
from sklearn.utils.validation import DataConversionWarning

# Suppress specific warning
warnings.filterwarnings('ignore', category=DataConversionWarning)

# Or ensure correct data types
import numpy as np
X = np.array(X, dtype=np.float64)

Lessons Learned

1. 데이터 품질 확인: 모델 학습 전에 반드시 결측값, 이상치, 데이터 타입을 확인하세요.

2. Pipeline 활용: 전처리와 모델을 Pipeline으로 묶으면 일관된 처리가 가능합니다.

3. Train/Test Split 먼저: 데이터 분할 후 전처리해야 data leakage를 방지할 수 있습니다.

4. 에러 메시지 읽기: scikit-learn의 에러 메시지는 문제의 원인을 명확히 보여줍니다.

5. Cross-validation 활용: 파이프라인과 함께 cross-validation을 사용하면 더 안정적인 성능 평가가 가능합니다.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.