Overview

This pull request refactors the metadata retrieval process used for telemetry, centralizing the logic within the pkg/config package. It removes the reliance on the external Firestore-based caching tool and instead fetches data directly using the GitHub API, backed by a robust local file-caching mechanism.

Process of Getting Standard Data (In Detail)

This process dynamically retrieves this data with the following steps:

1. Local Caching (getOrFetchCachedList):
   Before making any network requests, the system attempts to read the required data from a local JSON cache file located in the user's cache directory (e.g., standard_modules_<version>.json). If the cache hits and is successfully parsed, the data is returned immediately.
2. GitHub Git Trees API:
   If there is a cache miss, the system falls back to fetching the full file tree for the current toolkit version directly from the GitHub Git Trees API (https://api.github.com/repos/GoogleCloudPlatform/cluster-toolkit/git/trees/...).
3. Filtering for Modules and Examples:
   The retrieved repository tree is then parsed to identify standard paths:
   • Modules: It looks for .tf and .pkr.hcl files within the modules/ and community/modules/ directories to build the list of standard modules.
   • Examples: It identifies .yaml and .yml files within the examples/ and community/examples/ directories.
4. Fetching Blueprint Names via Worker Pool:
   To extract blueprint names efficiently, the system sets up a bounded concurrent worker pool (capped at 10 workers). These workers concurrently fetch the raw content of the identified example YAML files (https://raw.githubusercontent.com/...) and unmarshal only the blueprint_name field.
5. Cache Population:
   Once the standard modules, examples, or blueprint names are successfully fetched, the results are marshaled into JSON format and saved to the local user cache directory to speed up subsequent CLI executions.

Summary of Changes

• Added Local Caching and API Fetching: Implemented config.GetPredefinedModules(), config.GetPredefinedExampleFiles(), and config.GetStandardBlueprintNames() to handle local file-based caching and GitHub API fallbacks.
• Concurrency Improvements: Added a concurrent worker pool to fetch and parse raw YAML files for blueprint extraction without hitting GitHub rate limits sequentially.
• Telemetry Collector Updates: Updated pkg/telemetry/collector.go to use the new native config.GetPredefinedModules() functions instead of Viper.
• Cleanup: Deleted tools/cache_metadata/main.go and .github/workflows/cache-metadata.yml as this functionality is now handled natively within the toolkit.
• Testing: Added comprehensive unit tests in pkg/config/config_test.go using a mock HTTP transport (mockTransport) to simulate cache hits, network failures, and GitHub API responses.