-
-
Notifications
You must be signed in to change notification settings - Fork 202
Optimize & Minify DevRank Data Schema #9059
Copy link
Copy link
Closed
Labels
aienhancementNew feature or requestNew feature or requestperformancePerformance improvements and optimizationsPerformance improvements and optimizations
Description
The current users.json dataset is ~6MB for ~6k users (~1KB/user). Scaling to 100k users would result in an unmanageable ~100MB payload.
We need to drastically optimize the data structure to reduce file size by ~70% while maintaining functionality.
Optimization Strategy:
- Short Keys (Mapping): Map verbose keys to short codes (e.g.,
login->l,total_contributions->tc). - Avatar IDs: Store only the integer ID for avatars (User & Orgs), not full URLs.
- Years Array: Convert the
yearsobject map to a sequential array starting fromfirst_year. - Org Simplification: Limit orgs to top 5 and store as a compact tuple
[login, id]. - Smart Formatting: Use a "One-Record-Per-Line" format (valid JSON array, but each object is minified on a single line) to balance compression with git-diffability.
Tasks:
- Update
DevRank.model.Contributorwith field mappings and reconstruction logic. - Update
DevRank.services.Updaterto persist data in this new format. - Migrate existing
users.jsonto the new schema.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
aienhancementNew feature or requestNew feature or requestperformancePerformance improvements and optimizationsPerformance improvements and optimizations