Node.js Buffer.from() 인코딩 에러

Introduction

Buffer is Node.js's way of handling binary data. Encoding issues with Buffer.from() can lead to corrupted data, garbled text, and mysterious Unicode errors. After working with international text processing, I have compiled the most common encoding pitfalls.

Environment

node --version
# v20.11.0

# Encoding support in Node.js
node -e "console.log(Buffer.encodings)"

Problem

Converting between strings and Buffers with incorrect encoding produces corrupted data:

// Korean text processing
const koreanText = '안녕하세요 العالم';

// WRONG: Using default encoding assumptions
const buf1 = Buffer.from(koreanText, 'latin1');
console.log(buf1.toString('utf8'));
// Output: 깐벹emodel_charsetcharsetcharsetcharsetcharsetcharset

// Correct approach
const buf2 = Buffer.from(koreanText, 'utf8');
console.log(buf2.toString('utf8'));
// Output: 안녕하세요 العالم

The error does not always throw an exception - it silently corrupts your data.

Analysis

Different encodings use different byte sequences for the same characters:

Character: '안' (Korean)
UTF-8:     0xEC 0x95 0x88 (3 bytes)
UTF-16:    0xC5 0xB0 (2 bytes)
Latin1:    0xC5 (1 byte - truncated!)

When you encode with one encoding and decode with another, the byte sequences do not match, producing garbage.

The most common encoding mistakes:

// Mistake 1: Forgetting encoding parameter
Buffer.from('hello'); // UTF-8 by default - usually fine
Buffer.from('hello', 'binary'); // Different result!

// Mistake 2: Using 'binary' instead of 'latin1'
Buffer.from('hello', 'binary'); // Works but misleading name

// Mistake 3: Base64 encoding confusion
const encoded = Buffer.from('hello').toString('base64');
const decoded = Buffer.from(encoded, 'base64').toString('utf8');
// This is correct, but mixing up the order corrupts data

Solution

Solution 1: Always specify encoding explicitly

// GOOD: Explicit encoding everywhere
const text = '안녕하세요';
const buffer = Buffer.from(text, 'utf8');
const backToString = buffer.toString('utf8');

// For Base64
const base64 = Buffer.from(text, 'utf8').toString('base64');
const fromBase64 = Buffer.from(base64, 'base64').toString('utf8');

Solution 2: Handle encoding errors

function safeBufferFrom(data, inputEncoding = 'utf8', outputEncoding = 'utf8') {
  try {
    if (Buffer.isEncoding(inputEncoding)) {
      return Buffer.from(data, inputEncoding).toString(outputEncoding);
    }
    throw new Error(`Unsupported encoding: ${inputEncoding}`);
  } catch (err) {
    console.error(`Encoding error: ${err.message}`);
    return null;
  }
}

// Usage
const result = safeBufferFrom('안녕하세요', 'utf8', 'base64');
console.log(result); // 7JDq7YOo7Yqk

Solution 3: Detect encoding automatically

function detectEncoding(buffer) {
  // Check for BOM (Byte Order Mark)
  if (buffer[0] === 0xEF && buffer[1] === 0xBB && buffer[2] === 0xBF) {
    return 'utf8';
  }
  if (buffer[0] === 0xFF && buffer[1] === 0xFE) {
    return 'utf16le';
  }
  if (buffer[0] === 0xFE && buffer[1] === 0xFF) {
    return 'utf16be'; // Not directly supported, needs swapping
  }
  
  // Check for valid UTF-8
  let isValidUtf8 = true;
  let i = 0;
  while (i < buffer.length) {
    if (buffer[i] <= 0x7F) {
      i++;
    } else if (buffer[i] >= 0xC0 && buffer[i] <= 0xDF) {
      if (buffer[i + 1] === undefined || (buffer[i + 1] & 0xC0) !== 0x80) {
        isValidUtf8 = false;
        break;
      }
      i += 2;
    } else if (buffer[i] >= 0xE0 && buffer[i] <= 0xEF) {
      if (buffer[i + 1] === undefined || buffer[i + 2] === undefined ||
          (buffer[i + 1] & 0xC0) !== 0x80 || (buffer[i + 2] & 0xC0) !== 0x80) {
        isValidUtf8 = false;
        break;
      }
      i += 3;
    } else {
      isValidUtf8 = false;
      break;
    }
  }
  
  return isValidUtf8 ? 'utf8' : 'latin1';
}

// Usage
const sample = Buffer.from('안녕하세요');
console.log(detectEncoding(sample)); // 'utf8'

Solution 4: Stream encoding

const fs = require('fs');
const { Transform } = require('stream');

const encodingConverter = new Transform({
  transform(chunk, encoding, callback) {
    try {
      // Convert from detected encoding to UTF-8
      const text = chunk.toString('utf8');
      callback(null, text);
    } catch (err) {
      callback(err);
    }
  }
});

fs.createReadStream('file.txt')
  .pipe(encodingConverter)
  .pipe(fs.createReadStream('file-utf8.txt'));

Lessons Learned

1. Always specify encoding explicitly - Do not rely on defaults for critical data
2. Use 'utf8' for text - It is the standard for modern applications
3. Handle encoding errors gracefully - Corrupted data is worse than missing data
4. Test with international characters - ASCII-only testing misses encoding bugs
5. Use streams for large files - Buffer.from() loads entire file into memory

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.