Resource Metadata + Secrets

Problem statement

Developers want to see additional metadata about resources in Coder. This could be an instance type, the number of cores an instance has, or a link to the cloud resource itself (say in a GCP dashboard).

As a developer, I want to see information about my workspace, both information that is dynamic (e.g external IP) and the most relevant aspects of the template (no. CPUs).

Definition of done

We should add arbitrary key/value pairs that allow template authors to expose additional metadata. This should be added in our Terraform Provider, allowing customers to expose arbitrary values from existing resources that are provisioned. The key/value pairs would be available from the CLI and the dashboard.

---

Implementation notes (@bpmct)

Dashboard (figma):

Template:

resource "null_resource" "about" {}

resource "coder_metadata" "about_info" {
  resource_id = null_resource.about.id
  pair {
    key = "type"
    value = null
  }
  pair {
    key = "quicklinks"
    value = "[onboarding guide](https://mywiki.com/onboarding), [troubleshooting](https://mywiki.com/troubleshooting)"
  }
}

resource "kubernetes_pod" "main-dev" {
  # ...
}

resource "coder_metadata" "dev_info" {
  resource_id = kubernetes_pod.main-dev.id
  pair {
    key = "api_key"
    value = "asdfjkl"
    sensitive: true
  }
}

CLI:

$ coder metadata my-workspace
RESOURCE_NAME      KEY                 VALUE
about              workspace-id        acf2a70d-6347-4b8c
about              quicklinks          [onboarding guide](https://mywiki.com/onboarding), [troubleshooting](https://mywiki.com/troubleshooting)
main-dev           type             kubernetes_pod
main-dev           api_key             (hidden)

Some values are sensitive. Use `coder metadata myworkspace <key>` to fetch individual values.

In the future, perhaps resource_id = "global" or global = true could be used to display metadata workspace-wide in another panel if the hack is widely used.

---

Related:

• #1325
• #1321

I'd like to give customers more help here. Are there certain common items we can discover automatically? Or perhaps naming conventions that tell us to pull out particular variables?

Also a limitation (perhaps) of the above is around units/types and naming. For instance we measure RAM and disk in GB, but they're separate concepts. And they may want to measure CPU in a strict count.

I don't think we can easily assume which parts of the resource object are most relevant to the user.

I wouldn't assume in terms of limitations on what can be relevant. But if we make it zero effort to see how much e.g. S3 space is in use, I think that'd be good. And we should be able to do it with no coupling worth noting.

We can't really make arbitrary API calls (like getting the s3 bucket used) to these clouds. That would couple us heavily with the provisioner environment. So, we're limited to what terraform is exposing as variables.

If we want some "default metadata" to expose, we can just put those blocks in our example templates. That way the user can easily discover how to disable them and how to expose more metadata.

Sorry, "use" was the wrong word. I meant "allocated," which is something we can know from just our local information. We're targeting quotas to work on workspaces not in them, so that should be fine.

I don't want to give up on actual usage one day; I think it's pretty doable, but that's not what I meant above.

Sorry, "use" was the wrong word. I meant "allocated," which is something we can know from just our local information. We're targeting quotas to work on workspaces not in them, so that should be fine.

Ok, in that case I think we should include default metadata blocks in our examples.

I think the way we should provide access to the secrets in the workspace is via a command coder env get <key name> (run in the workspace that has access to the values). That is flexible enough to work on multiple operating systems in a consistent way and gives users a starting point where they can then store the values in anything else they want.

I believe this can be done in a satisfactory manner with no web UI. The template files and the above coder command should do it pretty well. Perhaps there's future work that would justify a web UI, but I think to start we don't need any of CLI, API, or web UI.

Are you suggesting that on top of the original post's suggestion or instead of?

Also environment secrets can already be injected without any new feature, which is why I left the environment injection outside of my suggestion:

resource "twilio_iam_api_key" "api_key" {
  account_sid   = "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  friendly_name = "Test API Key"
}

resource "coder_agent" "dev" {
  # ...
  env = {
    TWILIO_API_SECRET = "${twilio_iam_api_key.api_key.secret}"
  }
}

If you're suggesting that we add a coder env command in addition to environment variables, check out how Vercel approaches the problem.

TL;DR they don't have a vercel env get command, probably because it leads to an anti-pattern in scripts where you call out to vercel instead of the environment. This breaks the repo when running in CI or another environment without the coder command present. Also: there is no compatibility risk with scripts? All shell interpreters and most programming languages abstract environment variables to a portable interface.

Perhaps env isn't the right command name. I did want to have them call out to coder though, as that provides flexibility for e.g. things that change/rotate or are super sensitive. I don't think environment variables are the primary or only way we should inject these things; I'm too paranoid for that. Also in general I think a pull model is better than a push model.

Vercel provides a vercel env pull which produces a .env file. That could be used to satisfy rotations and "pull" while still discouraging any deep integration of your software with the coder command. Either way, it should be coder secrets or and not coder env to avoid confusion with the env block in the coder_agent resource.

I'm okay with it not being env, but not sold on secrets because, well, they might want to use this for things that are not secrets. values? variables? parameters has a potential conflict with template parameters. I kind of hate myself, but maybe coder workspace context?

I'm fine with semantics similar to the above. I was assuming (without saying so) that you could do e.g. coder workspace context --all --format=json > variables.json, which only differs from the above in small details.

This should just be coder metadata <workspace-name> [item-name]. Secrets isn't a solid use case, as environment variables can be directly mounted and it's not a great pattern to call Coder to access sensitive data. "sensitive" metadata is fine, but the primary purpose of this issue is to see key/value pairs from the dashboard and CLI.

This should just be coder workspace metadata.

The workspace commands are at the root, so I assume you mean coder metadata?

Secrets isn't a solid use case, as environment variables can be directly mounted and it's not a great pattern to call Coder to access sensitive data. "sensitive" metadata is fine, but the primary purpose of this issue is to see key/value pairs from the dashboard and CLI.

This issue is just suggesting helping the user access the existing secrets functionality. We already support secrets like this. The proposal is purely about convenience.

The issue description has a good template schema, but is missing some UX details, hence the discussion above. Here are some suggestions/mocks.

Assigning metadata to a resource:

# EDIT: see comments below for updated schema

resource "kubernetes_pod" "main-dev" {
   # ...
}
resource "coder_metadata" "api-key" {
   value       = "whatever"
   sensitive   = "true"
   resource_id = kubernetes_pod.main-dev.id
}

CLI:

$ coder metadata my-workspace
RESOURCE_NAME      KEY                 VALUE
home_volume        size                32G
kubernetes_pod     api_key             (hidden)

Some values are sensitive. Use `coder metadata myworkspace <key>` to fetch individual values.

Dashboard (figma):

@ammar - Correct on both points. Edited to a root-level command.

I like that mock up. Any open question is whether the metadata should be associated with resources or with the workspace as a whole?

The value should be able to be a link. What do we think about making that markdown?

I was thinking we could embed multiple values inside a metadata block:

resource "coder_metadata" "dev" {
  resource_id = "asdasd"
  values = {
    frog: "testing",
    memory: "something else"
  }
}

This metadata could be expanded to include additional properties down the line, like icon.

I think this issue conflates two different things. I see resource metadata as an aid to introspection of the workspace: tell me about the workspace I'm in, like /proc/cpuinfo. The parameters etc. not about the workspace but for use by stuff running in the workspace.

I think this issue conflates two different things. I see resource metadata as an aid to introspection of the workspace: tell me about the workspace I'm in, like /proc/cpuinfo. The parameters etc. not about the workspace but for use by stuff running in the workspace.

Ah, I think we should create a separate issue for that. I consider "metadata" more static, where activity/usage metrics change during runtime and cannot be given values during "build time," unless it's some command to run on the agent to fetch the latest value at an interval.

This would also be tied to an agent, not a resource (although all agents can only run on one resource)

The value should be able to be a link. What do we think about making that markdown?

I was thinking we could embed multiple values inside a metadata block:

resource "coder_metadata" "dev" {
  resource_id = "asdasd"
  values = {
    frog: "testing",
    memory: "something else"
  }
}

This metadata could be expanded to include additional properties down the line, like icon.

I like this format and see the value in adding support for markdown links. I see the coder_metadata has the dev name. I'm assuming, with that schema, "dev" wouldn't be rendered anywhere user-facing, and there is only 1 coder_metadata object per resource @kylecarbs?

I think activity/usage metrics are yet another category separate from the two stated above.

I'm not really sure what additional features you're looking for as a part of this issue. Exposing key/value pairs within the shell of the resource itself can be achieved with this feature and via environment variables on the coder agent.

Any open question is whether the metadata should be associated with resources or with the workspace as a whole?

After a chat with @kylecarbs, we settled on metadata associated with resources.

While slightly hacky, an admin can still display "workspace-wide" metadata in a fun way with null_resource. This uses the new schema @kylecarbs proposed as well (single metadata block per resource, markdown link support):

Dashboard (figma):

Template:

resource "null_resource" "about" {}

resource "coder_metadata" "about_info" {
  resource_id = null_resource.about.id
  pair {
    key = "type"
    value = null
  }
  pair {
    key = "quicklinks"
    value = "[onboarding guide](https://mywiki.com/onboarding), [troubleshooting](https://mywiki.com/troubleshooting)"
  }
}

resource "kubernetes_pod" "main-dev" {
  # ...
}

resource "coder_metadata" "dev_info" {
  resource_id = kubernetes_pod.main-dev.id
  pair {
    key = "api_key"
    value = "asdfjkl"
    sensitive: true
  }
}

CLI:

$ coder metadata my-workspace
RESOURCE_NAME      KEY                 VALUE
about              workspace-id        acf2a70d-6347-4b8c
about              quicklinks          [onboarding guide](https://mywiki.com/onboarding), [troubleshooting](https://mywiki.com/troubleshooting)
main-dev           type             kubernetes_pod
main-dev           api_key             (hidden)

Some values are sensitive. Use `coder metadata myworkspace <key>` to fetch individual values.

In the future, perhaps resource_id = "global" or global = true could be used to display metadata workspace-wide in another panel if the hack is widely used.

@bpmct I loved your mocks and I think they are good to go. I just made a "real size" version of it.

@dwahler I would love to know your thoughts.

I realized the single-block-per-resource was missing an indicator of whether metadata is sensitive. I added that to the Terraform in this comment under coder_metadata.dev_info.values.api_key { sensitive: true, value: "asfdjkl" }

I like this. A few comments:

Possibly a dumb question, but why do we need to explicitly create a null_resource, as opposed to just denoting "global" metadata fields by leaving the resource_id unset? Is it just to provide the option of creating multiple null resources for different groups of metadata fields?

Likewise, I'm not sure why we need to make users explicitly set type: null when we could just automatically hide the type field for null_resources. Unless that's too magical?

If the values parameter is a map, then AFAIK we have no way of detecting and maintaining the original ordering of map entries, so we'd have to just sort them alphabetically or something. Probably not a dealbreaker but still worth mentioning.

The UI mockup shows metadata fields attached to workspace resources, but do we also want to support them on coder_agent resources, or is that out of scope?

Possibly a dumb question, but why do we need to explicitly create a null_resource, as opposed to just denoting "global" metadata fields by leaving the resource_id unset? Is it just to provide the option of creating multiple null resources for different groups of metadata fields?

This decision was pretty arbitrary. I assumed it would be more complex to add handling for global metadata + add a new frontend component for global metadata. If this is minimal to handle and we have a frontend component in mind (cc @BrunoQuaresma), I see the value in supporting global metadata as a first-class feature.

Likewise, I'm not sure why we need to make users explicitly set type: null when we could just automatically hide the type field for null_resources. Unless that's too magical?

Hiding the type for null_resource seems like a sane default but is slightly magical. I'm flexible here, your call.

I think it would still be nice to support type: null for other resources. For example, a developer may not care that their 10gb disk is a kubernetes_persistant_volume_claim so an admin might hide this for simplicity.

If the values parameter is a map, then AFAIK we have no way of detecting and maintaining the original ordering of map entries, so we'd have to just sort them alphabetically or something. Probably not a dealbreaker but still worth mentioning.

To my knowledge, resources, variables, and workspace apps are already listed alphabetically so I think it's OK for now. Down the road, perhaps we could do api_key: {sensitive: true, order: 0, value: "asdasd"} to indicate an order.

The UI mockup shows metadata fields attached to workspace resources, but do we also want to support them on coder_agent resources, or is that out of scope?

I can see the use case for this, such as in the multi-service example, but think I think agent metadata should be out of scope unless it's trivial to add from a backend perspective.

Should I interpret the above to mean that this is now scoped to be just about showing which resources are allocated to a workspace and which secrets are needed to in support?

Thanks @bpmct, that makes sense.

Another thing that occurs to me is that the current design implies that we're showing the type field in the UI as though it was an "automatic" metadata field. but on the other hand we're not showing it in the output of coder metadata. If type is going to be treated specially, then we should definitely mention that in the docs as an exception to the rule that metadata keys can be arbitrary strings.

Should I interpret the above to mean that this is now scoped to be just about showing which resources are allocated to a workspace and which secrets are needed to in support?

Not sure I fully understand this. We'll discuss offline

Another thing that occurs to me is that the current design implies that we're showing the type field in the UI as though it was an "automatic" metadata field. but on the other hand we're not showing it in the output of coder metadata. If type is going to be treated specially, then we should definitely mention that in the docs as an exception to the rule that metadata keys can be arbitrary strings.

Agreed we should document the special behavior of type. Now that you bring it up, it should probably show up in coder metadata too, why not 🤷🏼

Is metadata the best name for this feature? There may be some confusion with parameters. metadata is a pretty broad term.

@kylecarbs and I chatted a bit and landed on a better solution for key/value pairs using Terraform's block syntax. Updated the comment in this issue

-  values = {
-    type: null, # hides "type: null_resource" which would otherwise appear by default
-    api_key: { sensitive: true, value: "asdfjkl" }, # since key/values are nested, I imagine we'd have to indicate sensitivity here and support `map(any)` for coder_metadata.values 
-    quicklinks: "[onboarding guide](https://mywiki.com/onboarding), [troubleshooting]= (https://mywiki.com/troubleshooting)"
-  } 
+  pair {
+    key = "type"
+    value = null
+  }
+  pair {
+    key = "api_key"
+    value = "asdfjkl"
+    sensitive: true
+  }
+  pair {
+    key = "quicklinks"
+    value = "[onboarding guide](https://mywiki.com/onboarding), [troubleshooting](https://mywiki.com/troubleshooting)"
+  }

Also opens the door for other metadata besides key/value pairs down the road :)

Is metadata the best name for this feature? There may be some confusion with parameters. metadata is a pretty broad term.

Hmm. How about annotations? Seems to be pretty in-line with how Kubernetes treats these things as annotations are designed to be human-readable or ingested by third-party services:

https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

(for the record, I think metadata is also fine)

IMO annotations is a great term for things like the quicklinks example above, but it's not such a good fit for something like an instance ID, where the value is derived from the output of the Terraform execution rather than being directly added by the template author.

Some other options that come to mind: properties, attributes, info_fields, details.

(and I'm OK with metadata too, so maybe this is just bikeshedding)

@dwahler you're not bikeshedding at all. Feature naming is very important, esp. for marketing which we don't think about enough. I think annotations is pretty good as it doesn't compete with existing concepts in terraform. I would love a deeper market investigation from @bpmct on the best name for this feature.

IMO annotations is a great term for things like the quicklinks example above, but it's not such a good fit for something like an instance ID, where the value is derived from the output of the Terraform execution rather than being directly added by the template author.

Some other options that come to mind: properties, attributes, info_fields, details.

(and I'm OK with metadata too, so maybe this is just bikeshedding)

Great point. I like info_fields and details as I can see how the others would be confused with parameters. I can do a bit more investigation too, as Ammar suggested.

To my knowledge, we haven't had any user requests for this feature. It does unlock some valuable use cases:

• display CPU cores to workspace user
• display links to relevant docs to workspace user
• display necessary values from Terraform-generated resources (e.g API key) for consumption elsewhere
• hide resource type (unnecessary complexity in dashboard)
• display the instance type to the workspace user
• down the road: custom icons for resources

I think it's still valuable to define the feature & schema but I suggest we don't prioritize this in coming weeks, especially as we focus on "blocking" issues. For now, I consider it a "nice-to-have"

cc @misskniss @tjcran

Where a subset of this may relate to demonstrated customer needs is with respect to resource quotas.

After a bit more research, I think metadata is still the best name for this feature. As Ammar mentioned, it is a broad term but the feature itself will likely expand to support more than simply key/value pairs in future iterations so the term is "encompassing".

I also think it is distinct enough from parameters.

Also spoke with @kylecarbs some more and since it's on the roadmap let's keep it moving! Over the next week or so, we might adjust as the roadmap changes but spoke with @dwahler and @BrunoQuaresma and seems like we're good to go.

Thanks @bpmct. What is the expected timeline for an MVP into main?

For the initial backend implementation, here's what I'm thinking:

• To keep things simple, for now let's assume that metadata is always associated with a resource, which might be a null_resource
• Metadata values are always either strings or null (integers etc. would be supported in templates, but stringified)
• The provisioner API will not directly return coder_metadata resources in job results (since they're an implementation detail of the Terraform provisioner) and instead extracts their key/value data and returns it along with the "target" resource
• Metadata is stored in a new workspace_resource_metadata DB table and included as a field of the WorkspaceResource object that we return from /api/v2/workspacebuilds/{id}/resources:

@@ -28,6 +28,12 @@ type WorkspaceResource struct {
        Type       string                      `json:"type"`
        Name       string                      `json:"name"`
        Agents     []WorkspaceAgent            `json:"agents,omitempty"`
+       Metadata   []WorkspaceResourceMetadata `json:"metadata,omitempty"`
 }
 
+type WorkspaceResourceMetadata struct {
+       Key       string `json:"key"`
+       Value     string `json:"value"`
+       Sensitive bool   `json:"sensitive,omitempty"`
+	// in future: Icon? Order? Type? LastUpdated?
+}

• The API response includes both "explicit" metadata fields returned from the provisioner, and "implicit" ones like type, with explicit taking precedence and null-valued fields omitted from the response

How does that sound? If I don't run into any unexpected snags in the implementation, I expect to be able to get the backend and CLI parts of this done within the next week.

That seems perfect to me!

@ammario @bpmct @kylecarbs we now just need to weigh this against the other roadmap items for August, but this seems like a solid candidate.

@tjcran This is currently in this sprint, the July roadmap, and being worked on. We can consider prioritizing against the August roadmap if necessary

I've gotten the core of this (provisioner and API only) implemented in PRs coder/terraform-provider-coder#34 and #3242.

Unfortunately, I hit a noticeable snag, which is that the syntax described in the requirements:

resource "coder_metadata" "dev_info" {
  resource_id = kubernetes_pod.main-dev.id
  # ...
}

may not actually be valid Terraform code. If the target resource is something like a pod whose lifecycle is tied to the workspace start/stop, then it has a count attribute which may be either 0 or 1, and you get an error like this when trying to directly reference its id field:

Error: Missing resource instance key Because kubernetes_pod.main-dev has "count" set, its attributes must be accessed on specific instances.

The error is from Terraform itself, not from our provider, and I haven't yet found anything in the language that would allow us to make the original syntax work. As a workaround, you can do this instead, but it's kind of ugly:

resource "coder_metadata" "dev_info" {
  resource_id = try(kubernetes_pod.main-dev[0].id, "")
  # ...
}

Any suggestions for better ways to deal with this are welcome.

@dwahler that seems reasonable to me. Template admins could add a count to the metadata to avoid the try function 🤷🏼

# main.tf
resource "aws-instance" "dev" {
    count = count = data.coder_workspace.me.start_count
    # ... 
}

resource "coder_metadata" "dev_info" {
+  count = data.coder_workspace.me.start_count

-   resource_id = try(kubernetes_pod.main-dev[0].id, "")
+   resource_id = kubernetes_pod.main-dev[0].id
  # ...
}

Another thing that comes to mind is using Terraform's depends_on to establish the relation, but I think that introduces some more problems:

• user may try to add multiple resources to depends_on or legitimately need it to depend on a resource
• to my knowledge, depends_on works for the resource as a whole. no way to have separate metadata for a unique "count" of a resource

Note: I'll write the docs for this once the frontend comes in and close the issue once that is complete.