Keeping user and group separate complicates the structure, I think:

1. There's a lot of fields with two mentions of "user", e.g. user.audit.user.id. The additional, second user doesn't really add any new information though, we already know this is user information from the first. It could just be user.audit.id and still be unique and understandable.
2. Not nesting the top-level group under user creates an inconsistency: Top-level group.id is its own object, but then all other group.id fields are nested under user. - begging the question of why the top-level one is not nested as well, as user.group.id. The other disadvantage is readability and searchability, e.g. in the JSON output user.id and user.group.id will be just a few rows apart and are clearly connected since they are part of the same top-level object, whereas user.id and group.id are in completely different parts of the event often dozens of lines apart and it's no longer easily possible to find them or know that they are connected.
   I think we don't need to dismiss the top-level group object completely though. It could be used to hold group information when the group is a standalone object, e.g. when a group has been changed. In this case though, all groups incl. the top-level group are just attributes of the user ("the user's group`, "the effective users's group", etc.).

In essence, what I'm proposing is that the user object be defined like:

"user": {
  "id": <real UID or primary SID>,
  "name": <id name>,
  
  "group": {
    "id": <real GID or primary group SID>
    "name": <id name>
  }

  "effective": <user object>,
  "audit": <user object>,
  etc.
}

@cwurm Yes, I see what you mean. And to complete the part related to the group a little more, you're proposing this?

"user": {
  "id": <real UID or primary SID>,
  "name": <id name>,
  
  "group": {
    "id": <real GID or primary group SID>
    "name": <id name>
    "audit": { "id": "...", "name": "..." }
    "effective": { "id": "...", "name": "..." }
    ...
  }
  "effective": <user object>,
  "audit": <user object>,
  etc.
}

@webmat I saw group as a part of the user object, so all of them would have one:

"user": {
  "id": <real UID or primary SID>,
  "name": <id name>,
  
  "group": {
    "id": <real GID or primary group SID>
    "name": <id name>
  }

  "effective": {
    "id": "...",  // effective user ID
    "name": "..." // effective user name
    "group": { "id": "...", "name": "..." } // effective user group ID and name
  },

  "audit": <user object>, // same as "effective", incl. a group object
  etc.
}

I've updated #9963 to the nested user format. Looks like this: https://github.com/elastic/beats/blob/b5319d7ed873f2d08c493781a85c963af04cfeb2/x-pack/auditbeat/module/system/process/_meta/data.json

Ah, gotcha. Yes, this makes sense. I'll be going with this as well.

@MikePaquette What do you think of this way of nesting user/group data, when reporting on the entities at play to determine effective rights? (See Christoph's comment above)

When I try to move to this structure, I'm getting a mapping exception, because currently ECS defines user.group as a string.

I think we should actually consider making this change to ECS for 1.0.0 GA, and have user.group become a place where the group field set can be reused. The current user.group string field actually goes against our principle of not reusing the same name in a different way elsewhere.

To clarify, I would not go ahead and add all of these nested values for user/group IDs and names part of ECS for GA just yet. I'm proposing changing only this one field, user.group.

I created elastic/ecs#304 to discuss this. We could get this changed and re-imported this week.

In the meantime, I'll work around this and take this PR as far as it can go.

Rebased, and temporarily introduced field name user.group_.id, for the purpose of the review.

Let's see if we decide to go ahead with elastic/ecs#304, to see what we do next.

Rebased over #10275. Once #10275 is merged, this will need a final rebase.

Blocked by #10306.

Ready for final review.

New tonight:

• migrate a few more fields that map to address and port (source and destination)
• migrate cmd (command line string) to process.args (array)
• map tty and terminal to user.terminal, not process.terminal
• [BUG] fix a mistake with the module's main fields.yml file: it must end with the opening of the field group for each fileset. I had added global fields after it, which meant the fileset fields were nested completely in the wrong place.

Ok, here's the latest development. The idea is to move forward a bit without introducing potential bugs. We can keep making things better later (even after 7.0, I would think).

• I fixed the issue with fields.yml causing weirdness in the docs
• I undid the tty => user.terminal transition. Only terminal is mapped to user.terminal
• I undid a network address/port mapping that was bugging me anyway. What bugged me: assuming "remote" was the "source". So I got rid of the mapping of the pair: addr, rport, laddr, lport
• All fields for which I undid the migration are now documented, however.

I think it's good enough like this. The rest is incremental improvements that may require more in depth tought.