refactor(openai): move endpoint hooks to new internal file, add utils file#5865
Merged
Conversation
93722ba to
d3a19df
Compare
d3a19df to
296784d
Compare
emmettbutler
previously approved these changes
May 18, 2023
emmettbutler
approved these changes
May 19, 2023
majorgreys
approved these changes
May 19, 2023
15 tasks
Yun-Kim
added a commit
that referenced
this pull request
Jun 15, 2023
## Tips for reviewers **note: 99% of the line changes in this PR are non-code, in the form of snapshot files, riot lockfiles, test data (audio/image files), and test cassette files used to mock request/responses with the OpenAI API, which store binary image/audio file data in `.yaml` files that Github counts as lines of code.** The major files that reviewers should focus on are: - `ddtrace/contrib/openai/patch.py` - outlines which endpoint hooks are added, minor changes to overall patching to minimize coupling with endpoint hook information. - `ddtrace/contrib/openai/_endpoint_hooks.py` - implementation of each endpoint hook, as well as minor refactor to minimize code duplication between endpoint hooks - `tests/contrib/openai/test_openai.py`. - shows how each endpoint is expected to be used. ## Summary This PR adds tracing support for the remaining endpoints not implemented by #5488. This includes: - Model (list, retrieve) - Edits (create) - Images (create, edit, create variation) - Audio (transcribe, translate) - Files (list, create, delete, retrieve, retrieve contents) - Fine-tunes (list, create, retrieve, cancel, list events, delete fine tuned model) - Moderations (create) Changes in this PR include: - Modifying the design of the endpoint hooks to minimize duplicate code, and adding new endpoint hooks for the above endpoints. - Ensuring rate-limit related metrics are tagged as numeric metrics rather than string tags. - Ensure user's OpenAI API key is set as a tag in multiple use scenarios (set in env var, set as attribute in `openai`, directly added as a request param) - Adding and correcting names of attributes extracted from `openai` (e.g. `organization`, `api_type`) - Simplifying and separating patch logic from endpoint-specific tracing logic (only operationID is propagated from endpoint hook level to patch level) - Moving some global integration-level tags to request-level (e.g. `openai.request.model`, `openai.request.endpoint`, added `openai.request.method`) ## OpenAI Integration Design ### Overall Patch Implementation **note**: the overall patch implementation has not changed; this is simply a refresher on how the overall integration works. The openai integration features patching via generators - each of the openAI API's endpoint methods (ex: `openai.Completions.create()`) is wrapped by `_patched_endpoint(endpoint_hook, ...)`, which itself takes a corresponding endpoint hook as an argument. A rough description of what happens when `_patched_endpoint()` is called is as follows: - `_patched_endpoint(endpoint_hook, ...)` starts a trace and starts a generator `_traced_endpoint(endpoint_hook, ...)` - `_traced_endpoint(endpoint_hook, ...)` starts `endpoint_hook().handle_request(...)`, which performs endpoint-specific tracing logic, and yields back to `_patched_endpoint()` once the request has been processed by the endpoint hook. - `_patched_endpoint()` runs the underlying openai API method, then yields back the response/error back to `_traced_endpoint()`, which in turn yields the response/error back to the `endpoint_hook`. - `endpoint_hook()` performs the endpoint-specific tracing logic for the response/error, then finishes the trace. ### Endpoint Hook Design The internal implementation design of the endpoint hooks have been modified (building off of the refactor from #5865) to minimize code duplication. This means that each endpoint hook now stores the following: - `_request_arg_params` and `_request_kwarg_params`, which are tuples containing the arg/kwarg signature of the underlying openAI API endpoint method. These tuples are used in `_EndpointHook._record_request(...)` to add request arg/kwarg parameters as span tags/metrics. - `ENDPOINT_NAME`, `REQUEST_TYPE`, `OPERATION_ID` constants, which reflect the base endpoint name (e.g. `/completions`), http request type (e.g. `POST`), and operation ID as specified by the openAI API specifcations (e.g. `createCompletion`). The operationID is used as the span resource name, and the endpoint/request_type values are added as span tags. - Each endpoint hook also features (optional) a `_record_request(...)` and `_record_response(...)` to add any endpoint-specific span tagging logic. ## Testing The testing for this PR involves snapshot testing for each endpoint, as well as `vcrpy` cassettes used to mock request/responses to OpenAI. ## Checklist - [x] Change(s) are motivated and described in the PR description. - [x] Testing strategy is described if automated tests are not included in the PR. - [x] Risk is outlined (performance impact, potential for breakage, maintainability, etc). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) are followed. - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)). - [x] OPTIONAL: PR description includes explicit acknowledgement of the performance implications of the change as reported in the benchmarks PR comment. ## Reviewer Checklist - [x] Title is accurate. - [x] No unnecessary changes are introduced. - [x] Description motivates each change. - [x] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Testing strategy adequately addresses listed risk(s). - [x] Change is maintainable (easy to change, telemetry, documentation). - [x] Release note makes sense to a user of the library. - [x] Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment. --------- Co-authored-by: Munir Abdinur <munir.abdinur@datadoghq.com> Co-authored-by: Federico Mon <federico.mon@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR refactors the openAI integration In order to more easily support multiple endpoints other than the existing
Completions/ChatCompletions/Embeddings.This PR makes the following refactors to the OpenAI integration:
_EndpointHookclass and its subclasses that we use to trace each endpoint are contained in their own internalddtrace.contrib.openai._endpoint_hooks.pyfile._EndpointHookbase class'shandle_request()is no longer an abstract method, and instead now contains two refactored abstract methods_pre_response()and_post_response()which happen before and after the response/error is returned from the openAI API. This allows more granularity and clarity on what the traced endpoint subclasses are implementing, especially since many endpoints will be able to share the same code for_pre_response()or_post_response()._default_name, which is the default name used for theopenai.response.operationtag._request_tag_attrs, which is a list of request params that will be automatically tagged by_EndpointHook._record_request()._est_tokenshelper function has been moved into addtrace.contrib.openai._utils.pyfile. More helper functions are expected to be contained in this new openai utils file when additional endpoints will be implemented.No functionality has been changed so no additional testing is required.
Checklist
Reviewer Checklist