Reviewing files that changed from the base of the PR and between 8d94a68 and 7f1c1c2.

• README.md (1 hunks)
• supadata/__init__.py (2 hunks)
• supadata/types.py (7 hunks)
• supadata/youtube.py (5 hunks)
• tests/test_client.py (12 hunks)

README.md (1)

68-87: Good job on the documentation updates!

The new examples for YouTube functionality are clear, concise, and follow the existing documentation style. They effectively demonstrate all the new endpoints added to the SDK.

supadata/types.py (1)

131-219: Well-structured and thoroughly documented data classes

The new data classes for YouTube entities are well-designed with appropriate types and comprehensive documentation. The optional description field in YoutubePlaylist is a good approach for handling potentially missing data.

tests/test_client.py (1)

272-441: Great test coverage for new YouTube endpoints

The tests for the new YouTube functionality are well-structured and thorough. They test both successful API calls and error handling scenarios for each endpoint. The mock responses are realistic and the assertions validate all important aspects of the returned objects.

supadata/youtube.py (6)

3-14: Imports look good.

No issues identified. The necessary classes and types are properly imported.

---

20-20: Constructor type annotations.

Great job specifying the callable signature for the request handler, ensuring clarity about expected arguments and return types.

---

28-30: Transcript method updates.

All parameter types, request parameters, and chunk handling look correct and consistent with the rest of the code.

Also applies to: 44-45, 58-60

---

66-67: Translate method updates.

The type hints and request structure are properly aligned, and chunk transformations are handled similarly to the transcript method. No issues found.

Also applies to: 81-85, 94-96, 99-100

---

101-112: New video method signature and docstring.

Everything appears well-structured: typed parameter and docstring are clear, raising SupadataError for failed requests.

---

127-129: Consider handling invalid date strings.

Wrap datetime.fromisoformat(...) in a try-except block or verification logic to prevent unhandled exceptions if upload_date is malformed.

Reviewing files that changed from the base of the PR and between 42f775e and 23e2a76.

• supadata/types.py (7 hunks)
• supadata/youtube.py (5 hunks)

supadata/types.py (6)

4-5: Import additions look good.

The added imports for datetime and additional typing options support the new YouTube-related classes and improve type hinting.

---

53-73: The updated Scrape class attributes are well documented.

The addition of description, og_url, and count_characters attributes enhances the data model with proper type annotations and clear documentation.

---

88-101: Good addition of description attribute to CrawlPage class.

The addition of the description attribute to the CrawlPage class is consistent with the Scrape class update and properly documented.

---

130-142: Well-structured TypedDict for YouTube channel base information.

Using TypedDict for the channel information that will be nested in other classes is a good design pattern.

---

174-197: Well-structured YoutubeChannel class.

The YoutubeChannel class has appropriate attributes with clear type annotations and comprehensive documentation.

---

199-219: YoutubePlaylist class looks good.

The implementation includes all necessary attributes with appropriate types. The optional description field with a default value of None is a good design choice.

supadata/youtube.py (8)

3-14: Imports properly include the new types and required modules.

The updated imports properly include the required modules and the new YouTube-related types defined in the types.py file.

---

20-20: Improved type annotation for request_handler.

The explicit typing for the request_handler parameter enhances code clarity and API documentation.

---

28-30: Improved formatting and type hints for existing methods.

The reformatting of the transcript and translate methods improves readability while maintaining functionality. The explicit type hints provide better documentation.

Also applies to: 66-67, 81-85

---

101-117: Video method implementation looks good.

The method properly handles the API interaction and datetime conversion for the uploaded_date field. The method signature includes appropriate type annotations.

---

118-133: Channel method implementation is well structured.

The implementation is straightforward and appropriate, with clear parameter and return type annotations.

---

134-158: Channel_videos method handles pagination correctly.

The method validates the limit parameter and constructs the query parameters appropriately. The return type annotation is accurate.

---

159-175: Playlist method properly handles date conversion.

The method correctly extracts and converts the ISO format date string to a datetime object before creating the YoutubePlaylist instance.

---

200-208: Robust limit validation method with clear error messages.

The _validate_limit method provides comprehensive validation with appropriate error messages for invalid inputs.

Reviewing files that changed from the base of the PR and between 23e2a76 and 277431a.

• README.md (1 hunks)
• supadata/__init__.py (2 hunks)
• supadata/types.py (7 hunks)
• supadata/youtube.py (5 hunks)
• tests/test_client.py (12 hunks)

• supadata/init.py

README.md (1)

68-87: Added documentation is clear and concise

The new examples clearly demonstrate how to use the newly added YouTube endpoint functions: video metadata, channel metadata, channel videos, playlist metadata, and playlist videos. The examples follow the existing pattern and are easy to understand.

supadata/types.py (10)

4-5: Good import organization

Correctly importing datetime and expanding the typing imports to support the new classes.

---

7-14: Comprehensive import list

All necessary types are imported for the new YouTube functionality while maintaining the existing imports.

---

56-64: Updated Scrape class docstring is complete

The docstring for the Scrape class has been properly updated to include the new fields: description, og_url, and count_characters.

---

70-72: New fields added to Scrape class

The new fields match what's described in the docstring and follow the existing pattern.

---

89-96: Updated CrawlPage class docstring

The CrawlPage docstring has been updated to include the new description attribute.

---

101-101: Added description field to CrawlPage

The description field has been added to the CrawlPage class as documented.

---

130-142: Well-structured YoutubeChannelBaseDict

The YoutubeChannelBaseDict TypedDict is well-structured and documented. It will be reused in other YouTube-related classes, which is good for consistency.

---

144-170: Complete YoutubeVideo class implementation

The YoutubeVideo class is well-documented with all the necessary fields for video metadata. The use of proper types for each field is excellent.

---

173-196: Comprehensive YoutubeChannel class

The YoutubeChannel class is well-structured with all the necessary fields for channel data, and includes proper typing.

---

198-218: Well-implemented YoutubePlaylist class

The YoutubePlaylist class follows the same pattern as the other YouTube classes, with proper typing and documentation. Good use of Optional for the description field that might not always be available.

tests/test_client.py (12)

3-18: Appropriate imports for new classes

The imports have been updated to include the necessary imports for the new YouTube classes: YoutubeChannel, YoutubePlaylist, and YoutubeVideo. The datetime import is added to handle date fields in these classes.

---

172-180: Updated test for crawl results

Added description field to the test to match updated CrawlPage class structure.

---

193-194: Updated test for second page response

Properly updated the test for the second page of crawl results to include the description field.

---

272-303: Comprehensive test for the video endpoint

This test covers all the necessary assertions for the YoutubeVideo class, including proper conversion of date strings to datetime objects. The test uses realistic mock data that closely resembles an actual API response.

---

305-324: Error handling test for invalid video ID

Good test for handling invalid video IDs. The test verifies that the appropriate error is raised and that the error fields are populated correctly.

---

326-352: Thorough test for the channel endpoint

The test covers all attributes of the YoutubeChannel class and uses realistic mock data. All assertions are properly set up to verify the correct behavior.

---

355-374: Test for invalid channel ID

Proper test for handling an invalid channel ID request. It verifies that the appropriate error is raised with the correct error details.

---

376-399: Good playlist endpoint test

The test for the playlist endpoint is comprehensive, covering all attributes of the YoutubePlaylist class including the datetime conversion.

---

401-421: Proper test for invalid playlist ID

The test correctly checks error handling for invalid playlist IDs.

---

424-441: Test for channel videos endpoint

Good test for retrieving video IDs from a channel. The test verifies that the response is a list and contains the expected video IDs.

---

443-463: Test for invalid channel ID when retrieving videos

Proper test for handling an invalid channel ID when retrieving videos. It validates that the appropriate error is raised with the correct error details.

---

465-483: Test for playlist videos endpoint

Good test for retrieving video IDs from a playlist. The test verifies that the response is a list and contains the expected video IDs.

supadata/youtube.py (16)

3-14: Appropriate imports

The import statements have been updated to include the necessary imports for the new YouTube functionality. Good use of typing for function signatures.

---

20-20: Improved type annotation for init

Added a more specific type annotation for the request_handler parameter, which improves code readability and helps with IDE autocompletion.

---

28-30: Better formatting for transcript method signature

Improved the formatting of the method signature for better readability.

---

44-45: Clean code formatting

Reorganized the params dictionary creation for better readability.

---

58-61: Consistent code formatting

Improved the formatting of the list comprehension for better readability.

---

66-66: Better type annotation for translate method

Improved the type annotation for the translate method parameters.

---

81-85: Improved request formatting

Reorganized the request call for better readability.

---

94-97: Consistent formatting

Improved the formatting of the list comprehension to match the style used earlier.

---

101-117: Well-implemented video method

The video method is well-implemented with proper typing, docstring, and error handling. The method correctly transforms the uploaded_date from ISO format to a datetime object.

---

118-133: Well-implemented channel method

The channel method is well-implemented with proper typing, docstring, and error handling.

---

134-158: Well-structured channel_videos method

The channel_videos method is well-implemented with proper typing, docstring, error handling, and parameter validation. Good use of the _validate_limit helper method.

---

159-175: Well-implemented playlist method

The playlist method is well-implemented with proper typing, docstring, and error handling. The method correctly transforms the last_updated field from ISO format to a datetime object.

---

176-199: Good implementation of playlist_videos method

The playlist_videos method follows the same pattern as channel_videos, with proper typing, docstring, error handling, and parameter validation.

---

200-208: Well-implemented _validate_limit helper

The _validate_limit helper method is properly implemented with good error handling and appropriate limit validation. The error messages are clear and informative.

---

114-114:

Details

❓ Verification inconclusive

Potential key error in video method

The code assumes the response will always contain an "upload_date" key, but if this key is missing or named differently in the API response, it would cause a KeyError.

Consider using a more defensive approach:

- uploaded_time = datetime.fromisoformat(response.pop("upload_date"))
+ uploaded_time = datetime.fromisoformat(response.pop("upload_date", None) or datetime.now().isoformat())

Or handle the potential KeyError explicitly:

- uploaded_time = datetime.fromisoformat(response.pop("upload_date"))
+ try:
+     uploaded_time = datetime.fromisoformat(response.pop("upload_date"))
+ except KeyError:
+     # Log warning or use a default value
+     uploaded_time = datetime.now()

---

🏁 Script executed:

#!/bin/bash
# Check if the API response always includes upload_date
rg -A 5 -B 5 "upload_date|uploadDate" --include "*.py"

Length of output: 498

---

Action: Verify API Response Field and Implement Defensive Handling
After attempting to search for “upload_date” via a ripgrep command we encountered an error (incorrect use of the --include flag). This means we haven’t yet confirmed whether the API response always provides the “upload_date” field. Without definitive evidence that the key is always present in all API responses, it’s prudent to adopt a more defensive coding approach.

Suggested next steps:

• Manually verify (or re-run with a corrected command) that the API response always includes “upload_date” (or “uploadDate”) across all relevant endpoints. For example, using:

  rg -A5 -B5 --glob "*.py" "upload_date|uploadDate"

• Update the code in supadata/youtube.py (around line 114) to either provide a default fallback value or explicitly catch a potential KeyError. For example:

- uploaded_time = datetime.fromisoformat(response.pop("upload_date"))
+ try:
+     uploaded_time = datetime.fromisoformat(response.pop("upload_date"))
+ except KeyError:
+     # Log warning or handle as needed (using current datetime as fallback)
+     uploaded_time = datetime.now()

or

- uploaded_time = datetime.fromisoformat(response.pop("upload_date"))
+ uploaded_time = datetime.fromisoformat(response.pop("upload_date", None) or datetime.now().isoformat())

Please verify these changes within the context of your API responses to ensure they handle all cases correctly.

---

172-172:

Details

❓ Verification inconclusive

Potential key error in playlist method

Similar to the video method, there's a risk of KeyError if "last_updated" is missing from the response.

Consider using a more defensive approach:

- last_updated = datetime.fromisoformat(response.pop("last_updated"))
+ try:
+     last_updated = datetime.fromisoformat(response.pop("last_updated"))
+ except KeyError:
+     # Log warning or use a default value
+     last_updated = datetime.now()

---

🏁 Script executed:

#!/bin/bash
# Check if the API response always includes last_updated
rg -A 5 -B 5 "last_updated|lastUpdated" --include "*.py"

Length of output: 500

---

Action Required: Confirm and Fix Potential KeyError in Playlist Method
There’s a risk of a KeyError in the playlist method if the API response is missing the "last_updated" key. Although we attempted to verify the presence of this key with a search, the execution error with the rg --include flag means we couldn’t conclusively determine if every API response always provides this field. Please verify the API responses manually or via an updated search (for example, using --glob "*.py" instead of --include) to ensure that the key is reliably present. In the meantime, adopting a defensive coding approach is recommended. For instance, you can update the code in supadata/youtube.py (around line 172) as follows:

-        last_updated = datetime.fromisoformat(response.pop("last_updated"))
+        try:
+            last_updated = datetime.fromisoformat(response.pop("last_updated"))
+        except KeyError:
+            # Log warning or use a default value
+            last_updated = datetime.now()

Please confirm the API behavior and adjust the handling as necessary.

Reviewing files that changed from the base of the PR and between aae525e and 616f41d.

• README.md (1 hunks)
• supadata/__init__.py (2 hunks)
• supadata/types.py (7 hunks)
• supadata/youtube.py (5 hunks)
• tests/test_client.py (12 hunks)

• README.md
• supadata/init.py

supadata/types.py (4)

129-141: Well-structured TypedDict for channel base information.

This is a good foundation for reusing channel information across different types like YoutubeVideo and YoutubePlaylist. The TypedDict approach is the right choice for this scenario.

---

144-171: Comprehensive video type with complete attributes.

The YoutubeVideo class correctly captures all necessary attributes for a YouTube video, including metadata and engagement metrics. The documentation is thorough and all fields are properly typed.

---

173-196: Well-defined YouTube channel dataclass.

The YoutubeChannel class has a complete set of attributes needed to represent YouTube channels. All fields have appropriate types and the documentation clearly describes each attribute.

---

198-219: Good implementation of playlist dataclass with optional description.

The YoutubePlaylist class is well-structured with appropriate types. Making the description optional is a good choice, as not all playlists have descriptions.

tests/test_client.py (7)

272-303: Well-structured test for YouTube video retrieval.

The test thoroughly validates the video method by checking the return type and verifying that all properties from the mock response are correctly mapped to the corresponding fields in the YoutubeVideo object.

---

305-324: Error handling test is correctly implemented.

This test properly verifies that invalid video ID requests raise a SupadataError with the appropriate error details. The difference between mock response error messages and assertions is expected due to the gateway error handling in Supadata._handle_gateway_error as explained in previous comments.

---

326-353: Good test for YouTube channel retrieval.

The test thoroughly validates all fields of the returned YoutubeChannel object against the mock response, ensuring that the channel method works as expected.

---

355-374: Error handling test is correctly implemented.

This test properly verifies that invalid channel ID requests raise a SupadataError with the appropriate error details. As previously discussed, the difference in error details is due to the gateway error handling in the client.

---

376-399: Thorough test for YouTube playlist retrieval.

The test effectively validates the playlist method by checking all properties of the returned YoutubePlaylist object, including proper handling of the datetime conversion for the last_updated field.

---

424-441: Good test for channel videos retrieval using namespace approach.

The test correctly validates the namespace-based approach for retrieving channel videos (client.youtube.channel.videos), which aligns with rafalzawadzki's feedback in the PR comments about using dot notation for namespace access.

---

465-483: Good test for playlist videos retrieval using namespace approach.

The test correctly validates the namespace-based approach for retrieving playlist videos (client.youtube.playlist.videos), which follows the same pattern as the channel videos implementation, maintaining consistency.

supadata/youtube.py (7)

20-20: Good addition of type hints to the constructor.

Adding type hints to the request_handler parameter improves code documentation and provides better IDE support. The Callable[[str, str, Any], Dict[str, Any]] type accurately describes the expected function signature.

---

103-118: Well-implemented video metadata retrieval.

The video method correctly handles the YouTube video data retrieval, including the conversion of the upload date string to a datetime object. The method has good error handling, proper type hints, and comprehensive documentation.

---

120-132: Good implementation of namespace properties.

The channel and playlist properties implement the dot notation approach as requested in the PR feedback, allowing users to access features like channel.videos instead of using channel_videos. The lazy initialization pattern used here is efficient.

---

134-142: Proper limit validation.

The _validate_limit method provides a reusable validation function for limits across different endpoints, with appropriate error messages and handling of edge cases, including None values.

---

144-190: Well-structured Channel class implementation.

The _Channel inner class properly encapsulates channel-related functionality, with both metadata retrieval and video listing capabilities. The use of __call__ to allow direct invocation is a nice design pattern.

---

189-189: Consistent use of snake_case for API response keys.

Using video_ids key is correct here. As noted in the retrieved learnings, the Supadata client automatically converts camelCase keys from the API response to snake_case via the _camel_to_snake method, so even though the API returns videoIds, it will be accessible as video_ids in the code.

---

236-236: Consistent use of snake_case for API response keys.

Using video_ids key is correct here, following the same pattern as in the Channel class. The _camel_to_snake method handles the conversion from the API's camelCase to snake_case internally.