Python UnicodeDecodeError 해결

Introduction

UnicodeDecodeError is one of the most common and confusing errors in Python 3. It occurs when Python tries to decode a byte sequence using an encoding that cannot represent certain bytes. The error message includes cryptic details like 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte.

This error is especially common when working with files that were created on different operating systems, processing data from external APIs, or handling CSV files exported from Excel. I encountered this repeatedly while building a data pipeline that processed Korean financial data exported from various banking systems in different encodings.

Environment

Python 3.12.3

Problem

Error 1: Reading a file with wrong encoding

# File is actually EUC-KR encoded, but we try UTF-8
with open("korean_data.csv", "r") as f:
    content = f.read()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 15: 
invalid start byte

Error 2: Decoding bytes without specifying encoding

data = b'\xc0\xc1'  # Raw bytes
decoded = data.decode()  # Defaults to 'utf-8'

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: 
invalid start byte

Error 3: Reading binary file as text

with open("image.png", "r") as f:
    content = f.read()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: 
invalid start byte

Error 4: CSV with mixed encodings

import csv

with open("mixed_encoding.csv", "r") as f:
    reader = csv.reader(f)
    for row in reader:
        print(row)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 42: 
invalid start byte

Analysis

Python 3 uses Unicode strings internally but reads bytes from files and networks. The decoding process converts bytes to strings using a specific encoding. When bytes do not match the expected encoding, UnicodeDecodeError occurs.

Common encodings and their characteristics:

• UTF-8: Variable-length encoding. ASCII-compatible. Most common on modern systems.
• UTF-16: 2 or 4 bytes per character. Used by Windows internally.
• EUC-KR: Fixed-width Korean encoding. Common in legacy Korean systems.
• CP949: Microsoft's extension of EUC-KR. Covers more Korean characters.
• ISO-8859-1: Single-byte encoding for Western European languages.
• CP1252: Windows variant of ISO-8859-1.

Why the error occurs:

# UTF-8 encoding rules:
# 0xxxxxxx: ASCII characters (0x00-0x7F)
# 110xxxxx 10xxxxxx: 2-byte sequence (0xC0-0xDF, then 0x80-0xBF)
# 1110xxxx 10xxxxxx 10xxxxxx: 3-byte sequence

# When Python encounters 0xC0 followed by something that is NOT 0x80-0xBF,
# it raises UnicodeDecodeError because the byte sequence is invalid UTF-8

Solution

Fix 1: Detect encoding before reading

# Install chardet: pip install chardet
import chardet

def detect_encoding(file_path):
    with open(file_path, "rb") as f:
        raw_data = f.read()
    result = chardet.detect(raw_data)
    return result["encoding"], result["confidence"]

encoding, confidence = detect_encoding("korean_data.csv")
print(f"Detected encoding: {encoding} (confidence: {confidence:.2%})")

# Read with detected encoding
with open("korean_data.csv", "r", encoding=encoding) as f:
    content = f.read()

Fix 2: Use error handling modes

# Skip invalid bytes
with open("data.csv", "r", encoding="utf-8", errors="ignore") as f:
    content = f.read()

# Replace invalid bytes with Unicode replacement character
with open("data.csv", "r", encoding="utf-8", errors="replace") as f:
    content = f.read()

# Use 'surrogateescape' for round-trip fidelity
with open("data.csv", "rb") as f:
    raw = f.read()
content = raw.decode("utf-8", errors="surrogateescape")

Fix 3: Try multiple encodings

def read_with_fallback(file_path, encodings=None):
    if encodings is None:
        encodings = ["utf-8", "cp949", "euc-kr", "cp1252", "iso-8859-1"]
    
    for encoding in encodings:
        try:
            with open(file_path, "r", encoding=encoding) as f:
                return f.read(), encoding
        except UnicodeDecodeError:
            continue
    
    raise ValueError(f"Could not decode {file_path} with any encoding")

content, used_encoding = read_with_fallback("korean_data.csv")
print(f"Read successfully with {used_encoding}")

Fix 4: Handle binary data correctly

# Read binary files in binary mode
with open("image.png", "rb") as f:
    binary_content = f.read()

# Process binary data with struct or specific libraries
import struct
header = struct.unpack(">I", binary_content[:4])

Fix 5: For subprocess output

import subprocess

# subprocess returns bytes, not strings
result = subprocess.run(["ls", "-la"], capture_output=True)

# Decode with proper encoding
output = result.stdout.decode("utf-8")

# Or let Python handle it
output = result.stdout.decode("utf-8", errors="replace")

Lessons Learned

• Always specify encoding explicitly when opening files — do not rely on the system default.
• Use chardet to detect encoding when you do not know it beforehand.
• Use errors="replace" or errors="ignore" when you need to process data regardless of encoding issues.
• Read binary files in binary mode ("rb") — never try to read them as text.
• Keep a list of common encodings for your domain (e.g., EUC-KR for Korean systems) and try them in order.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.