aiohttp 비동기 HTTP 클라이언트 에러 해결

aiohttp는 Python의 강력한 비동기 HTTP 라이브러리이지만, async/await 패턴을 올바르게 이해하지 않으면 various 에러가 발생할 수 있습니다.

Environment

$ python --version
Python 3.11.5

$ pip show aiohttp
Name: aiohttp
Version: 3.9.1

$ pip show aiosignal
Name: aiosignal
Version: 1.3.1

Problem: RuntimeWarning: coroutine was never awaited

import aiohttp
import asyncio

async def fetch_url(url):
    session = aiohttp.ClientSession()
    response = await session.get(url)
    data = await response.text()
    await session.close()
    return data

# Wrong: Not using await
result = fetch_url("https://httpbin.org/get")
print(result)

출력:

RuntimeWarning: coroutine 'fetch_url' was never awaited
result = None

Analysis: async/await 동작 원리

import asyncio

async def example():
    print("Start")
    await asyncio.sleep(1)
    print("End")
    return "Done"

# Wrong ways to call async function
coroutine = example()  # Returns coroutine object, doesn't execute
print(coroutine)  # 

# Correct way
async def main():
    result = await example()
    print(result)

asyncio.run(main())

Solution: 올바른 aiohttp 사용법

기본 클라이언트 사용

import aiohttp
import asyncio

async def fetch_url(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

async def main():
    # Single request
    html = await fetch_url("https://httpbin.org/get")
    print(f"Response length: {len(html)}")

asyncio.run(main())

에러 처리 포함

import aiohttp
import asyncio
from aiohttp import ClientError, ClientTimeout

async def safe_fetch(url, timeout=10):
    timeout_obj = ClientTimeout(total=timeout)
    
    try:
        async with aiohttp.ClientSession(timeout=timeout_obj) as session:
            async with session.get(url) as response:
                if response.status == 200:
                    return await response.json()
                else:
                    return {"error": f"HTTP {response.status}"}
    except ClientError as e:
        return {"error": f"Client error: {e}"}
    except asyncio.TimeoutError:
        return {"error": "Request timed out"}
    except Exception as e:
        return {"error": f"Unexpected error: {e}"}

async def main():
    result = await safe_fetch("https://httpbin.org/get")
    print(result)

asyncio.run(main())

동시 요청 처리

import aiohttp
import asyncio

async def fetch_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for url in urls:
            task = asyncio.create_task(fetch_with_session(session, url))
            tasks.append(task)
        
        results = await asyncio.gather(*tasks, return_exceptions=True)
        return results

async def fetch_with_session(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    urls = [
        "https://httpbin.org/delay/1",
        "https://httpbin.org/delay/2",
        "https://httpbin.org/get",
    ]
    
    results = await fetch_all(urls)
    for i, result in enumerate(results):
        if isinstance(result, Exception):
            print(f"URL {i}: Error - {result}")
        else:
            print(f"URL {i}: Success - {len(result)} bytes")

asyncio.run(main())

Semaphore로 동시성 제한

import aiohttp
import asyncio

async def fetch_limited(urls, max_concurrent=10):
    semaphore = asyncio.Semaphore(max_concurrent)
    
    async def fetch_with_limit(session, url):
        async with semaphore:
            async with session.get(url) as response:
                return await response.text()
    
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_with_limit(session, url) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

async def main():
    urls = [f"https://httpbin.org/get?i={i}" for i in range(100)]
    results = await fetch_limited(urls, max_concurrent=10)
    print(f"Fetched {len(results)} URLs")

asyncio.run(main())

Advanced: 재시도 로직 구현

import aiohttp
import asyncio
import random

class RetryClient:
    def __init__(self, max_retries=3, backoff_factor=0.5):
        self.max_retries = max_retries
        self.backoff_factor = backoff_factor
        self.session = None
    
    async def __aenter__(self):
        self.session = aiohttp.ClientSession()
        return self
    
    async def __aexit__(self, exc_type, exc_val, exc_tb):
        if self.session:
            await self.session.close()
    
    async def fetch(self, url):
        for attempt in range(self.max_retries):
            try:
                async with self.session.get(url) as response:
                    if response.status < 500:
                        return await response.json()
                    
                    # Server error, retry
                    if attempt < self.max_retries - 1:
                        wait_time = self.backoff_factor * (2 ** attempt)
                        wait_time += random.uniform(0, 0.5)
                        await asyncio.sleep(wait_time)
                        
            except aiohttp.ClientError as e:
                if attempt < self.max_retries - 1:
                    await asyncio.sleep(self.backoff_factor)
                else:
                    raise
        
        raise Exception(f"Failed after {self.max_retries} retries")

async def main():
    async with RetryClient(max_retries=3) as client:
        result = await client.fetch("https://httpbin.org/get")
        print(result)

asyncio.run(main())

Lessons Learned

1. await 필수: async 함수는 반드시 await를 사용하여 호출해야 합니다.

2. Session 관리: async with를 사용하여 ClientSession을 안전하게 관리하세요.

3. 에러 처리: 네트워크 에러, 타임아웃 등을 처리하는 로직을 반드시 포함하세요.

4. 동시성 제한: Semaphore를 사용하여 너무 많은 동시 요청을 방지하세요.

5. 재시도 로직: 일시적인 에러에 대비하여 재시도 메커니즘을 구현하세요.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.