{"id":93,"date":"2023-05-22T18:57:36","date_gmt":"2023-05-22T17:57:36","guid":{"rendered":"https:\/\/livingdevops.com\/?p=93"},"modified":"2025-04-03T16:57:52","modified_gmt":"2025-04-03T15:57:52","slug":"google-cloud-resource-manager-listing-projects-python","status":"publish","type":"post","link":"https:\/\/livingdevops.com\/gcp\/google-cloud-resource-manager-listing-projects-python\/","title":{"rendered":"How to List All Projects in Google Cloud Organization Using Python and gcloud"},"content":{"rendered":"\n
GCP Project: Listing all projects under a Google Cloud organization with Python<\/h2>\n\n\n\n
When working on Google Cloud Platform, we often need to list all GCP projects under an organization. This is a common task for cloud administrators, and there are two main approaches to do this: using the gcloud<\/code> command-line tool or Python with the Resource Manager API.<\/p>\n\n\n\n
In this comprehensive guide, I’ll walk you through both methods step-by-step, focusing on how to efficiently retrieve a complete list of all projects across your entire GCP organization.<\/p>\n\n\n\n
But before we start, let me talk about GCP Org Hierarchy, which is important to understand before we write the code.<\/p>\n\n\n\n<\/figure>\n\n\n\n
GCP Resource Hierarchy Structure<\/h3>\n\n\n\n\n
Organization<\/strong> (Blue) – The top-level node and root of the hierarchy\n
\n
Represents an entire company or organization<\/li>\n\n\n\n
Allows centralized visibility and control<\/li>\n<\/ul>\n<\/li>\n\n\n\n
Folders<\/strong> (Orange) – Grouping mechanism for organizing resources\n
\n
Can be nested (like Team A folder inside Development)<\/li>\n\n\n\n
Often represent departments, teams, or environments<\/li>\n\n\n\n
Create logical boundaries for applying policies<\/li>\n<\/ul>\n<\/li>\n\n\n\n
Projects<\/strong> (Green) – Containers for Google Cloud resources\n
\n
All resources must belong to a project<\/li>\n\n\n\n
Projects can exist directly under an organization or within folders<\/li>\n\n\n\n
Provide isolation boundaries for billing and access control<\/li>\n<\/ul>\n<\/li>\n\n\n\n
Resources<\/strong> (Red) – The actual cloud services\n
Always belong to a project<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n
Key Points About the Hierarchy<\/h3>\n\n\n\n
\n
Inheritance<\/strong>: IAM policies and organization policies flow down the hierarchy<\/li>\n\n\n\n
Flexibility<\/strong>: Projects can be directly under the organization or nested in folders<\/li>\n\n\n\n
Isolation<\/strong>: Projects provide isolation for resources and billing<\/li>\n\n\n\n
Organization<\/strong>: The diagram shows common patterns like having separate folders for Development, Production, and Infrastructure<\/li>\n<\/ul>\n\n\n\n
Using gcloud to List All Projects in an Organization<\/h2>\n\n\n\n
The quickest way to list all projects in your GCP organization is using the gcloud<\/code> command-line tool. Let’s start with this approach.<\/p>\n\n\n\n
# List all projects in the organization\ngcloud projects list --filter=\"parent.id:[ORGANIZATION_ID]\" --format=\"table(projectId,name,projectNumber)\"\n<\/code><\/pre>\n\n\n\n
Replace [ORGANIZATION_ID]<\/code> with your actual organization ID (without the “organizations\/” prefix).<\/p>\n\n\n\n
This command will list all projects that are directly under your organization, but it won’t show projects nested inside folders. To get a complete list including all projects in folders, we need a different approach.<\/p>\n\n\n\n
Why Python is Better for Listing All Projects in a GCP Organization<\/h2>\n\n\n\n
While the gcloud<\/code> command is simple, it has limitations when dealing with complex organizational structures. Here’s why Python with the Resource Manager API is better:<\/p>\n\n\n\n\n
Complete hierarchy traversal<\/strong>: Python can recursively go through all folders and subfolders<\/li>\n\n\n\n
Customizable output<\/strong>: You can format and filter the results exactly as needed<\/li>\n\n\n\n
Automation-friendly<\/strong>: Easy to integrate with other systems and scheduled tasks<\/li>\n\n\n\n
Complex filtering<\/strong>: Apply advanced filters based on any project property<\/li>\n<\/ol>\n\n\n\n
Using Python to List All Projects in a GCP Organization<\/h2>\n\n\n\n
Now, let’s dive into the Python approach, which will give us a complete view of all projects across the entire organization hierarchy.<\/p>\n\n\n\n
Step 1: Create a Service Account with the Right Permissions<\/h3>\n\n\n\n
First, we need to create a service account with appropriate permissions to view the organization structure:<\/p>\n\n\n\n
# Set the project in cloud shell\ngcloud config set project someproject\n\n# Create a service account\ngcloud iam service-accounts create rm-svc \\\n --description=\"Service account for listing GCP projects\" \\\n --display-name=\"resource-manager-list\"\n\n# Assign organization-level permission to the service account\ngcloud organizations add-iam-policy-binding \\ \nORGANIZATION_ID --member=\"serviceAccount:rm-svc@someproject.iam.gserviceaccount.com\" \\\n --role=\"roles\/resourcemanager.folderViewer\"\n\n# Create a key for the service account\ngcloud iam service-accounts keys create rm-svc.json --iam-account \\\nrm-svc@someproject.iam.gserviceaccount.com\n<\/code><\/pre>\n\n\n\n
The key file will look something like this (with actual values instead of placeholders):<\/p>\n\n\n\n
Note<\/strong>: There are many better ways to run the code than creating keys, such as using short-lived credentials, but I want to keep it simple for demo purposes.<\/p>\n<\/blockquote>\n\n\n\n
Step 2: Authenticate as the Service Account<\/h3>\n\n\n\n
Next, we’ll impersonate the service account to run our Python code:<\/p>\n\n\n\n
Step 3: Install the Required Python Library<\/h3>\n\n\n\n
We need to install the Python client for the Resource Manager:<\/p>\n\n\n\n
# Assuming you already have Python and pip installed\npip install google-cloud-resource-manager \n<\/code><\/pre>\n\n\n\n
Step 4: Write the Python Script to List All GCP Projects<\/h3>\n\n\n\n
Create a file named list_all_gcp_projects.py<\/code> with the following code:<\/p>\n\n\n\n
from google.cloud import resourcemanager_v3\n\ndef get_folders(\n parent_id = \"organizations\/ORGANIZATION_ID\",\n folders = None):\n \"\"\"\n Recursively retrieves all folders in the organization.\n \n This function walks through the entire folder hierarchy starting\n from the organization and collects all folder IDs, including nested folders.\n \n Args:\n parent_id: The starting point (organization ID or folder ID)\n folders: Accumulator list for recursion\n \n Returns:\n List containing all folder IDs in the organization\n \"\"\"\n if folders is None:\n folders = []\n\n # Create a client for accessing folders\n client = resourcemanager_v3.FoldersClient()\n \n # Prepare the request to list folders under the parent\n request = resourcemanager_v3.ListFoldersRequest(\n parent=parent_id,\n )\n\n # Get all folders under the parent\n page_result = client.list_folders(request=request)\n \n # For each folder found, add it to our list and then search its subfolders\n for folder in page_result:\n folders.append(folder.name)\n # Recursive call to find subfolders\n get_folders(parent_id=folder.name, folders=folders)\n \n return folders\n\ndef search_projects(folder_id):\n \"\"\"\n Finds all projects under a specific folder.\n \n Args:\n folder_id: The folder ID to search under\n \n Returns:\n List of project objects found in the folder\n \"\"\"\n # Create a client for accessing projects\n client = resourcemanager_v3.ProjectsClient()\n\n # Create a query to find projects under the specified folder\n query = f\"parent:{folder_id}\"\n request = resourcemanager_v3.SearchProjectsRequest(query=query)\n \n # Get all projects under the folder\n page_result = client.search_projects(request=request)\n \n # Collect all projects into a list\n search_result = []\n for project in page_result:\n search_result.append(project)\n \n return search_result\n\ndef search_org_projects(org_id):\n \"\"\"\n Finds all projects directly under the organization (not in folders).\n \n Args:\n org_id: The organization ID to search under\n \n Returns:\n List of project objects found directly under the organization\n \"\"\"\n # Create a client for accessing projects\n client = resourcemanager_v3.ProjectsClient()\n\n # Create a query to find projects directly under the organization\n query = f\"parent:{org_id}\"\n request = resourcemanager_v3.SearchProjectsRequest(query=query)\n \n # Get all projects under the organization\n page_result = client.search_projects(request=request)\n \n # Collect all projects into a list\n search_result = []\n for project in page_result:\n search_result.append(project)\n \n return search_result\n\ndef list_all_projects():\n \"\"\"\n Lists all projects in the entire organization.\n \n This function:\n 1. Gets all projects directly under the organization\n 2. Gets all folders in the organization\n 3. Gets all projects in each folder\n 4. Combines the results and filters for active projects\n \n Returns:\n List of all active project IDs in the organization\n \"\"\"\n active_projects = []\n organization_id = \"organizations\/ORGANIZATION_ID\"\n \n # Step 1: Get projects directly under the organization\n print(f\"Searching for projects directly under the organization...\")\n org_projects = search_org_projects(organization_id)\n for project in org_projects:\n if str(project.state) == \"State.ACTIVE\":\n active_projects.append(project.project_id)\n \n print(f\"Found {len(active_projects)} projects directly under the organization\")\n \n # Step 2 & 3: Get all folders and the projects within them\n print(f\"Searching for folders in the organization...\")\n folders = get_folders(parent_id=organization_id, folders=None)\n print(f\"Found {len(folders)} folders to check\")\n \n # Step 4: Find projects in each folder\n for i, folder in enumerate(folders):\n print(f\"Searching folder {i+1}\/{len(folders)}: {folder}\")\n folder_projects = search_projects(folder)\n \n for project in folder_projects:\n if str(project.state) == \"State.ACTIVE\":\n active_projects.append(project.project_id)\n \n return active_projects\n\nif __name__ == \"__main__\":\n print(\"Starting search for all GCP projects in the organization...\")\n projects = list_all_projects()\n print(f\"\\nSummary: Found {len(projects)} active projects in the organization\")\n print(\"\\nProject list:\")\n for project in projects:\n print(f\"- {project}\")\n<\/code><\/pre>\n\n\n\n
\n
Important<\/strong>: Replace ORGANIZATION_ID<\/code> with your actual Google Cloud Organization ID in two places in the script.<\/p>\n<\/blockquote>\n\n\n\n
Step 5: Run the Script<\/h3>\n\n\n\n
Now we can run our Python script to list all GCP projects:<\/p>\n\n\n\n
Starting point<\/strong>: It begins with the organization and recursively explores all folders<\/li>\n\n\n\n
Recursion<\/strong>: For each folder found, it calls itself to find sub-folders<\/li>\n\n\n\n
Accumulation<\/strong>: It builds a list of all folder IDs during the process<\/li>\n<\/ul>\n\n\n\n
This recursive approach ensures we discover all folders regardless of how deeply nested they are in the organization.<\/p>\n\n\n\n
2. The Project Search Functions<\/h3>\n\n\n\n
We have two project search functions:<\/p>\n\n\n\n
def search_projects(folder_id):\n # Find projects in a folder\n \ndef search_org_projects(org_id):\n # Find projects directly under the organization\n<\/code><\/pre>\n\n\n\n
These functions:<\/p>\n\n\n\n
\n
Create a Resource Manager client<\/li>\n\n\n\n
Build a query to find projects under a specific parent<\/li>\n\n\n\n
Execute the query and collect the results<\/li>\n<\/ul>\n\n\n\n
The key difference is that one searches folders, while the other searches directly under the organization.<\/p>\n\n\n\n
3. The Main List Function<\/h3>\n\n\n\n
The list_all_projects()<\/code> function brings everything together:<\/p>\n\n\n\n
def list_all_projects():\n<\/code><\/pre>\n\n\n\n
This function:<\/p>\n\n\n\n
\n
Finds projects directly under the organization<\/li>\n\n\n\n
Gets all folders in the organization<\/li>\n\n\n\n
Searches each folder for projects<\/li>\n\n\n\n
Filters out any non-active projects<\/li>\n\n\n\n
Returns a complete list of active project IDs<\/li>\n<\/ol>\n\n\n\n
4. The Output<\/h3>\n\n\n\n
The script provides detailed progress information:<\/p>\n\n\n\n
\n
Shows how many projects are directly under the organization<\/li>\n\n\n\n
Reports the total number of folders found<\/li>\n\n\n\n
Shows progress as it searches each folder<\/li>\n\n\n\n
Provides a final summary and complete project list<\/li>\n<\/ul>\n\n\n\n
Enhancing the Script for More Information<\/h2>\n\n\n\n
The basic script gives us a list of project IDs, but what if we want more details? Let’s enhance it to include more information:<\/p>\n\n\n\n
def list_projects_with_details():\n \"\"\"\n Lists all projects with additional details.\n \n Returns:\n Dictionary of project details by project ID\n \"\"\"\n project_details = {}\n organization_id = \"organizations\/ORGANIZATION_ID\"\n \n # Get projects directly under the organization\n org_projects = search_org_projects(organization_id)\n for project in org_projects:\n if str(project.state) == \"State.ACTIVE\":\n project_details[project.project_id] = {\n \"name\": project.display_name,\n \"number\": project.project_number,\n \"parent\": \"organization\",\n \"create_time\": project.create_time.isoformat(),\n \"labels\": dict(project.labels)\n }\n \n # Get all folders\n folders = get_folders(parent_id=organization_id, folders=None)\n folder_names = {}\n \n # Get folder names for better reporting\n client = resourcemanager_v3.FoldersClient()\n for folder_path in folders:\n folder_id = folder_path.split('\/')[-1]\n request = resourcemanager_v3.GetFolderRequest(name=folder_path)\n folder = client.get_folder(request=request)\n folder_names[folder_path] = folder.display_name\n \n # Get projects in each folder\n for folder in folders:\n folder_projects = search_projects(folder)\n for project in folder_projects:\n if str(project.state) == \"State.ACTIVE\":\n project_details[project.project_id] = {\n \"name\": project.display_name,\n \"number\": project.project_number,\n \"parent\": f\"folder:{folder_names.get(folder, folder)}\",\n \"create_time\": project.create_time.isoformat(),\n \"labels\": dict(project.labels)\n }\n \n return project_details\n<\/code><\/pre>\n\n\n\n
Exporting Results to CSV for Reporting<\/h2>\n\n\n\n
Let’s add a function to export our results to a CSV file for easy reporting:<\/p>\n\n\n\n
import csv\nfrom datetime import datetime\n\ndef export_to_csv(project_details):\n \"\"\"\n Export project details to a CSV file\n \n Args:\n project_details: Dictionary of project details\n \"\"\"\n timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n filename = f\"gcp_projects_{timestamp}.csv\"\n \n with open(filename, 'w', newline='') as csvfile:\n fieldnames = [\"project_id\", \"name\", \"number\", \"parent\", \"create_time\", \"labels\"]\n writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n \n writer.writeheader()\n for project_id, details in project_details.items():\n row = {\"project_id\": project_id}\n row.update(details)\n # Convert labels dictionary to string\n row[\"labels\"] = \", \".join([f\"{k}={v}\" for k, v in details[\"labels\"].items()])\n writer.writerow(row)\n \n print(f\"Exported project data to {filename}\")\n\n# Using it in the main function\nif __name__ == \"__main__\":\n print(\"Getting detailed information for all GCP projects...\")\n project_details = list_projects_with_details()\n print(f\"Found {len(project_details)} active projects\")\n \n # Export to CSV\n export_to_csv(project_details)\n \n # Print summary\n print(\"\\nProject summary:\")\n for project_id, details in project_details.items():\n print(f\"- {project_id} ({details['name']}): {details['parent']}\")\n<\/code><\/pre>\n\n\n\n
Common Issues and Troubleshooting<\/h2>\n\n\n\n
When listing projects across a GCP organization, you might encounter these common issues:<\/p>\n\n\n\n
1. Permission Denied Errors<\/h3>\n\n\n\n
If you see Permission Denied<\/code> errors, check:<\/p>\n\n\n\n
\n
The service account has the correct roles (resourcemanager.folderViewer<\/code> and resourcemanager.projectViewer<\/code>)<\/li>\n\n\n\n
You’ve activated the service account correctly<\/li>\n\n\n\n
The organization policy allows the service account to access the resources<\/li>\n<\/ul>\n\n\n\n
2. Missing Projects<\/h3>\n\n\n\n
If projects are missing from your results:<\/p>\n\n\n\n
\n
Ensure the script recursively searches all folders (our implementation does this)<\/li>\n\n\n\n
Check if you’re filtering out projects that aren’t in the “ACTIVE” state<\/li>\n\n\n\n
Verify that the service account has access to all folders<\/li>\n<\/ul>\n\n\n\n
3. Rate Limiting<\/h3>\n\n\n\n
For very large organizations:<\/p>\n\n\n\n
\n
Add delays between API calls using time.sleep()<\/code><\/li>\n\n\n\n
Implement exponential backoff for retries<\/li>\n\n\n\n
Consider paginating results for better performance<\/li>\n<\/ul>\n\n\n\n
Practical Applications for GCP Project Listing<\/h2>\n\n\n\n
Here are some real-world use cases for listing all projects in your GCP organization:<\/p>\n\n\n\n
\n
Resource Inventory<\/strong>: Create a complete inventory of all your GCP resources<\/li>\n\n\n\n
Cost Management<\/strong>: Identify orphaned or unused projects to reduce costs<\/li>\n\n\n\n
Security Audits<\/strong>: Ensure all projects comply with security policies<\/li>\n\n\n\n
Compliance Reporting<\/strong>: Generate reports for compliance requirements<\/li>\n\n\n\n
Project Governance<\/strong>: Enforce naming conventions and organizational structure<\/li>\n<\/ol>\n\n\n\n
Conclusion: Choosing the Right Approach<\/h2>\n\n\n\n
We’ve explored two methods for listing all projects in a Google Cloud organization:<\/p>\n\n\n\n
\n
Using gcloud<\/strong>: Simple but limited to direct children of the organization<\/li>\n\n\n\n
Using Python<\/strong>: More complex but provides a complete view of all projects<\/li>\n<\/ol>\n\n\n\n
For most real-world scenarios, the Python approach is superior because it:<\/p>\n\n\n\n
\n
Handles complex organizational structures with nested folders<\/li>\n\n\n\n
Provides detailed information about each project<\/li>\n\n\n\n
Can be automated and integrated with other systems<\/li>\n\n\n\n
Allows for custom filtering and processing of results<\/li>\n<\/ul>\n\n\n\n
With the Python script provided, you can now efficiently list all projects across your entire GCP organization, regardless of how complex your hierarchy is.<\/p>\n\n\n\n
We can use the Resource Manager API to do so much more than this – from managing project hierarchies to implementing custom governance policies. The possibilities are vast!<\/p>\n\n\n\n
GCP Project: Listing all projects under a Google Cloud organization with Python When working on Google Cloud Platform, we often need to list all GCP projects under an organization. This is a common task for cloud administrators, and there are two main approaches to do this: using the gcloud command-line tool or Python with the […]<\/p>\n","protected":false},"author":1,"featured_media":95,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,7],"tags":[],"class_list":["post-93","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-gcp","category-python"],"blocksy_meta":[],"_links":{"self":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts\/93","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/comments?post=93"}],"version-history":[{"count":7,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts\/93\/revisions"}],"predecessor-version":[{"id":755,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts\/93\/revisions\/755"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/media\/95"}],"wp:attachment":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/media?parent=93"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/categories?post=93"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/tags?post=93"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"gcp](\"https:\/\/livingdevops.com\/wp-content\/uploads\/2023\/05\/gcp-org-hirearcy-1024x755.jpg\")
<\/figure>\n\n\n\n