Setting Timeout in the OpenAI Python Library

This is a guide on how to set timeouts when calling the OpenAI API using the OpenAI Python Library.

Introduction
Setting Timeout in the OpenAI Python Library
- How to: Using the request_timeout Parameter
- A Pitfall: The timeout Parameter
Conclusion
References

Introduction

While using the OpenAI Python Library to call the ChatCompletion API, I wanted to set a timeout.

Since I couldn't immediately figure out how to do this from the reference documentation, I'm writing this guide for future reference.

# Working environment
# openai version
0.27.8

Note: This article was translated from my original post.

Setting Timeout in the OpenAI Python Library

How to: Using the request_timeout Parameter

You can set a timeout using the request_timeout parameter.

# Code example
response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Hello world"},
    ],
    request_timeout=20,  # Set timeout (in seconds)
)

If you look carefully, this request_timeout parameter is actually mentioned in the README.

All endpoints have a .create method that supports a request_timeout param. This param takes a Union[float, Tuple[float, float]] and will raise an openai.error.Timeout error if the request exceeds that time in seconds

Ref. GitHub - openai/openai-python: The official Python library for the OpenAI API

It appears that the request_timeout parameter can be used with any method that has a .create endpoint, not just the ChatCompletion API.

When a timeout occurs, an openai.error.Timeout error is raised.

A Pitfall: The timeout Parameter

Looking at the openai source code, it appears that a timeout parameter might also work for ChatCompletion.

class ChatCompletion(EngineAPIResource):
    engine_required = False
    OBJECT_NAME = "chat.completions"

    @classmethod
    def create(cls, *args, **kwargs):
        """
        Creates a new chat completion for the provided messages and parameters.

        See https://platform.openai.com/docs/api-reference/chat/create
        for a list of valid parameters.
        """
        start = time.time()
        timeout = kwargs.pop("timeout", None)

        while True:
            try:
                return super().create(*args, **kwargs)
            except TryAgain as e:
                if timeout is not None and time.time() > start + timeout:
                    raise

                util.log_info("Waiting for model to warm up", error=e)