Allow authorization engines as an extension by jaymode · Pull Request #37785 · elastic/elasticsearch

jaymode · 2019-01-23T19:58:14Z

Authorization engines can now be registered by implementing a plugin,
which also has a service implementation of a security extension. Only
one extension may register an authorization engine and this engine will
be used for all users except reserved realm users and internal users.

Authorization engines can now be registered by implementing a plugin, which also has a service implementation of a security extension. Only one extension may register an authorization engine and this engine will be used for all users except reserved realm users and internal users.

elasticmachine · 2019-01-23T19:58:15Z

Pinging @elastic/es-security

lcawl · 2019-01-23T20:58:23Z

@jaymode Is there overlap between this new content and the existing info here?: https://www.elastic.co/guide/en/elastic-stack-overview/master/custom-roles-provider.html

Should " Implementing an authorization engine" and "Implementing a custom roles provider" both fall under "Customizing authorization"?

jaymode · 2019-01-23T22:22:19Z

Should " Implementing an authorization engine" and "Implementing a custom roles provider" both fall under "Customizing authorization"?

I think there is overlap and combining the two makes sense. I will look at how to best merge them together

alpar-t · 2019-01-24T12:52:33Z

x-pack/qa/security-example-authorization-engine/build.gradle

@@ -0,0 +1,46 @@
+apply plugin: 'elasticsearch.esplugin'


Maybe this should live in plugins/examples .
That way we will automatically test it as stand alone project which would alert us if some required dependency is not published. Assuming it's ok for the example code to live outside x-pack directory.

I think we need to answer two questions:

Are we ok having more dependencies on x-pack code outside of the x-pack directory?

Are we ok making this code ASL 2.0 licensed?

For 2, I do not see an issue since the previous example was ASL 2.0 licensed. I think there may be more concern about 1; @rjernst @jasontedor any thoughts.

I don't think depending on/using xpack outside the xpack directory is a problem, when the code in question will be ASL 2.0 licensed anyways. I think that is better from a visibility standpoint, and also clearer, since having ASL code under the x-pack dir could be easily overlooked. So putting this example (and probably moving the existing xpack examples) to plugins/examples makes sense to me.

@atorok @rjernst I moved the example added here; can one of you take a look at that part?

I'll open up a separate change for the existing example.

thanks @jaymode looks good.

x-pack/docs/en/security/authorization/custom-authorization.asciidoc

lcawl · 2019-01-25T18:08:40Z

FYI I've created elastic/stack-docs#191 to pull the new custom-authorization.asciidoc file into the Stack Overview.

x-pack/docs/en/security/authorization/custom-authorization.asciidoc

lcawl · 2019-01-25T18:22:04Z

x-pack/docs/en/security/authorization/custom-authorization.asciidoc

+custom_roles_provider.s3_roles_provider.region: us-east-1
+custom_roles_provider.s3_roles_provider.secret_key: xxx
+custom_roles_provider.s3_roles_provider.access_key: xxx
+----------------------------------------


Is this example a repetition of the one below on purpose? If there's a different example we could use, I think that would be more helpful.

that was an oops

lcawl · 2019-01-25T18:23:57Z

x-pack/docs/en/security/authorization/custom-authorization.asciidoc

+[[using-security-extension]]
+==== Using the security extension
+
+To use a security extension:


If this set of steps is specific to implementing an authorization engine, I'd recommend something like this:

Suggested change

To use a security extension:

To use the authorization engine:

This applies to both.

commit 05b3a571658312fa4259b652d9b8478ecec278e7 Author: jaymode <jay@elastic.co> Date: Thu Jan 31 15:48:48 2019 -0700 Move request interceptors to AuthorizationService This change moves the RequestInterceptor iteration from the action filter to the AuthorizationService. This is done to remove the need for the use of a role within the request interceptors and replace it with the AuthorizationEngine. The AuthorizationEngine interface was also enhanced with a new method that is used to determine if a users permission on one index is a subset of their permissions on a list of indices or aliases. Additionally, this change addresses some leftover cleanups. commit 0e1c191 Merge: 3280607 b7de8e1 Author: jaymode <jay@elastic.co> Date: Thu Jan 31 08:56:45 2019 -0700 Merge branch 'master' into security_authz_engine commit 3280607 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Tue Jan 29 14:17:37 2019 -0700 Allow authorization engines as an extension (elastic#37785) Authorization engines can now be registered by implementing a plugin, which also has a service implementation of a security extension. Only one extension may register an authorization engine and this engine will be used for all users except reserved realm users and internal users. commit d628008 Author: jaymode <jay@elastic.co> Date: Tue Jan 29 10:06:09 2019 -0700 fix RBACEngine after restricted indices changes commit 5074683 Merge: 74f2e99 3c9f703 Author: jaymode <jay@elastic.co> Date: Tue Jan 29 08:09:39 2019 -0700 Merge branch 'master' into security_authz_engine commit 74f2e99 Merge: 7846ee8 899dfc3 Author: jaymode <jay@elastic.co> Date: Fri Jan 25 15:02:07 2019 -0700 Merge branch 'master' into security_authz_engine commit 7846ee8 Merge: b9a2c81 a81931b Author: jaymode <jay@elastic.co> Date: Thu Jan 24 07:52:08 2019 -0700 Merge branch 'master' into security_authz_engine commit b9a2c81 Author: jaymode <jay@elastic.co> Date: Tue Jan 22 09:48:11 2019 -0700 Fix resolving restricted indices after merging commit d98a77a Merge: 83cde40 5c1a1f7 Author: jaymode <jay@elastic.co> Date: Tue Jan 22 09:09:23 2019 -0700 Merge branch 'master' into security_authz_engine commit 83cde40 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Tue Jan 22 08:03:19 2019 -0700 Add javadoc to the AuthorizationEngine interface (elastic#37620) This commit adds javadocs to the AuthorizationEngine interface aimed at developers of an authorization engine. Additionally, some classes were also moved to the core project so that they are ready to be exposed once we allow authorization engines to be plugged in. commit 9a240c6 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Thu Jan 17 19:33:35 2019 -0700 Encapsulate request, auth, and action name (elastic#37495) This change introduces a new class called RequestInfo that encapsulates the common objects that are passed to the authorization engine methods. By doing so, we give ourselves a way of adding additional data without breaking the interface. Additionally, this also reduces the need to ensure we pass these three parameters in the same order everywhere for consistency. commit 6278eab Merge: c555a44 4351a5e Author: jaymode <jay@elastic.co> Date: Thu Jan 17 07:51:32 2019 -0700 Merge branch 'master' into security_authz_engine commit c555a44 Merge: 1362ab6 ecf0de3 Author: jaymode <jay@elastic.co> Date: Wed Jan 16 10:24:33 2019 -0700 Merge branch 'master' into security_authz_engine commit 1362ab6 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Wed Jan 16 10:23:45 2019 -0700 Replace AuthorizedIndices class with a List (elastic#37328) This change replaces the AuthorizedIndices class with a simple list. The change to a simple list does remove the lazy loading of the authorized indices in favor of simpler code as the loading of this list is now an asynchronous operation that is delegated to the authorization engine. commit 0246442 Merge: 8ccdc19 a2a40c5 Author: jaymode <jay@elastic.co> Date: Tue Jan 15 10:49:12 2019 -0700 Merge branch 'master' into security_authz_engine commit 8ccdc19 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Mon Jan 7 13:43:22 2019 -0700 Introduce asynchronous RBACEngine (elastic#36245) In order to support the concept of different authorization engines, this change begins the refactoring of the AuthorizationService to support this. Previously, the asynchronous work for authorization was performed by the AsyncAuthorizer class, but this tied the authorization service to a role based implementation. In this change, the authorize method become asynchronous and delegates much of the actual permission checking to an AuthorizationEngine. The pre-existing RBAC permission checking has been abstracted into the RBACEngine. The majority of calls to AuthorizationEngine instances are asynchronous as the underlying implementation may need to make network calls that should not block the current thread, which are often network threads. This change is meant to be built upon. The basic concepts are introduced without proper documentation, plumbing to enable other AuthorizationEngine types, and some items we may want to refactor. For example, the AuthorizedIndices class is lazily loaded but this might actually be something we want to make asynchronous. We pass a lot of the same arguments to the various methods and it would be prudent to wrap these in a class; this class would provide a way for us to pass additional items needed by future enhancements without breaking the interface and requiring updates to all implementations. See elastic#32435

commit 3e60a91 Author: jaymode <jay@elastic.co> Date: Mon Feb 4 12:34:23 2019 -0700 add licensing for authorization engine commit 1c9a8e1 Author: jaymode <jay@elastic.co> Date: Mon Feb 4 12:17:54 2019 -0700 fix inconsistency in parameter name/type commit 34aa55a Author: Jay Modi <jaymode@users.noreply.github.com> Date: Mon Feb 4 11:45:01 2019 -0700 Authorization engines evaluate privileges for APIs (elastic#38219) This commit moves the evaluation of privileges from a few transport actions into the authorization engine. The APIs are used by other applications for making decisions and if a different authorization engine is used that is not role based, we should still allow these APIs to work. By moving this evaluation out of the transport action, the transport actions no longer have a dependency on roles. commit 54d7b4c Merge: e5615d2 715e581 Author: jaymode <jay@elastic.co> Date: Mon Feb 4 08:06:14 2019 -0700 Merge branch 'master' into security_authz_engine commit e5615d2 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Mon Feb 4 07:53:37 2019 -0700 Move request interceptors to AuthorizationService (elastic#38137) This change moves the RequestInterceptor iteration from the action filter to the AuthorizationService. This is done to remove the need for the use of a role within the request interceptors and replace it with the AuthorizationEngine. The AuthorizationEngine interface was also enhanced with a new method that is used to determine if a users permission on one index is a subset of their permissions on a list of indices or aliases. Additionally, this change addresses some leftover cleanups. commit 0e1c191 Merge: 3280607 b7de8e1 Author: jaymode <jay@elastic.co> Date: Thu Jan 31 08:56:45 2019 -0700 Merge branch 'master' into security_authz_engine commit 3280607 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Tue Jan 29 14:17:37 2019 -0700 Allow authorization engines as an extension (elastic#37785) Authorization engines can now be registered by implementing a plugin, which also has a service implementation of a security extension. Only one extension may register an authorization engine and this engine will be used for all users except reserved realm users and internal users. commit d628008 Author: jaymode <jay@elastic.co> Date: Tue Jan 29 10:06:09 2019 -0700 fix RBACEngine after restricted indices changes commit 5074683 Merge: 74f2e99 3c9f703 Author: jaymode <jay@elastic.co> Date: Tue Jan 29 08:09:39 2019 -0700 Merge branch 'master' into security_authz_engine commit 74f2e99 Merge: 7846ee8 899dfc3 Author: jaymode <jay@elastic.co> Date: Fri Jan 25 15:02:07 2019 -0700 Merge branch 'master' into security_authz_engine commit 7846ee8 Merge: b9a2c81 a81931b Author: jaymode <jay@elastic.co> Date: Thu Jan 24 07:52:08 2019 -0700 Merge branch 'master' into security_authz_engine commit b9a2c81 Author: jaymode <jay@elastic.co> Date: Tue Jan 22 09:48:11 2019 -0700 Fix resolving restricted indices after merging commit d98a77a Merge: 83cde40 5c1a1f7 Author: jaymode <jay@elastic.co> Date: Tue Jan 22 09:09:23 2019 -0700 Merge branch 'master' into security_authz_engine commit 83cde40 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Tue Jan 22 08:03:19 2019 -0700 Add javadoc to the AuthorizationEngine interface (elastic#37620) This commit adds javadocs to the AuthorizationEngine interface aimed at developers of an authorization engine. Additionally, some classes were also moved to the core project so that they are ready to be exposed once we allow authorization engines to be plugged in. commit 9a240c6 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Thu Jan 17 19:33:35 2019 -0700 Encapsulate request, auth, and action name (elastic#37495) This change introduces a new class called RequestInfo that encapsulates the common objects that are passed to the authorization engine methods. By doing so, we give ourselves a way of adding additional data without breaking the interface. Additionally, this also reduces the need to ensure we pass these three parameters in the same order everywhere for consistency. commit 6278eab Merge: c555a44 4351a5e Author: jaymode <jay@elastic.co> Date: Thu Jan 17 07:51:32 2019 -0700 Merge branch 'master' into security_authz_engine commit c555a44 Merge: 1362ab6 ecf0de3 Author: jaymode <jay@elastic.co> Date: Wed Jan 16 10:24:33 2019 -0700 Merge branch 'master' into security_authz_engine commit 1362ab6 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Wed Jan 16 10:23:45 2019 -0700 Replace AuthorizedIndices class with a List (elastic#37328) This change replaces the AuthorizedIndices class with a simple list. The change to a simple list does remove the lazy loading of the authorized indices in favor of simpler code as the loading of this list is now an asynchronous operation that is delegated to the authorization engine. commit 0246442 Merge: 8ccdc19 a2a40c5 Author: jaymode <jay@elastic.co> Date: Tue Jan 15 10:49:12 2019 -0700 Merge branch 'master' into security_authz_engine commit 8ccdc19 Author: Jay Modi <jaymode@users.noreply.github.com> Date: Mon Jan 7 13:43:22 2019 -0700 Introduce asynchronous RBACEngine (elastic#36245) In order to support the concept of different authorization engines, this change begins the refactoring of the AuthorizationService to support this. Previously, the asynchronous work for authorization was performed by the AsyncAuthorizer class, but this tied the authorization service to a role based implementation. In this change, the authorize method become asynchronous and delegates much of the actual permission checking to an AuthorizationEngine. The pre-existing RBAC permission checking has been abstracted into the RBACEngine. The majority of calls to AuthorizationEngine instances are asynchronous as the underlying implementation may need to make network calls that should not block the current thread, which are often network threads. This change is meant to be built upon. The basic concepts are introduced without proper documentation, plumbing to enable other AuthorizationEngine types, and some items we may want to refactor. For example, the AuthorizedIndices class is lazily loaded but this might actually be something we want to make asynchronous. We pass a lot of the same arguments to the various methods and it would be prudent to wrap these in a class; this class would provide a way for us to pass additional items needed by future enhancements without breaking the interface and requiring updates to all implementations. See elastic#32435

For some users, the built in authorization mechanism does not fit their needs and no feature that we offer would allow them to control the authorization process to meet their needs. In order to support this, a concept of an AuthorizationEngine is being introduced, which can be provided using the security extension mechanism. An AuthorizationEngine is responsible for making the authorization decisions about a request. The engine is responsible for knowing how to authorize and can be backed by whatever mechanism a user wants. The default mechanism is one backed by roles to provide the authorization decisions. The AuthorizationEngine will be called by the AuthorizationService, which handles more of the internal workings that apply in general to authorization within Elasticsearch. In order to support external authorization services that would back an authorization engine, the entire authorization process has become asynchronous, which also includes all calls to the AuthorizationEngine. The use of roles also leaked out of the AuthorizationService in our existing code that is not specifically related to roles so this also needed to be addressed. RequestInterceptor instances sometimes used a role to ensure a user was not attempting to escalate their privileges. Addressing this leakage of roles meant that the RequestInterceptor execution needed to move within the AuthorizationService and that AuthorizationEngines needed to support detection of whether a user has more privileges on a name than another. The second area where roles leaked to the user is in the handling of a few privilege APIs that could be used to retrieve the user's privileges or ask if a user has privileges to perform an action. To remove the leakage of roles from these actions, the AuthorizationService and AuthorizationEngine gained methods that enabled an AuthorizationEngine to return the response for these APIs. Ultimately this feature is the work included in: #37785 #37495 #37328 #36245 #38137 #38219 Closes #32435

For some users, the built in authorization mechanism does not fit their needs and no feature that we offer would allow them to control the authorization process to meet their needs. In order to support this, a concept of an AuthorizationEngine is being introduced, which can be provided using the security extension mechanism. An AuthorizationEngine is responsible for making the authorization decisions about a request. The engine is responsible for knowing how to authorize and can be backed by whatever mechanism a user wants. The default mechanism is one backed by roles to provide the authorization decisions. The AuthorizationEngine will be called by the AuthorizationService, which handles more of the internal workings that apply in general to authorization within Elasticsearch. In order to support external authorization services that would back an authorization engine, the entire authorization process has become asynchronous, which also includes all calls to the AuthorizationEngine. The use of roles also leaked out of the AuthorizationService in our existing code that is not specifically related to roles so this also needed to be addressed. RequestInterceptor instances sometimes used a role to ensure a user was not attempting to escalate their privileges. Addressing this leakage of roles meant that the RequestInterceptor execution needed to move within the AuthorizationService and that AuthorizationEngines needed to support detection of whether a user has more privileges on a name than another. The second area where roles leaked to the user is in the handling of a few privilege APIs that could be used to retrieve the user's privileges or ask if a user has privileges to perform an action. To remove the leakage of roles from these actions, the AuthorizationService and AuthorizationEngine gained methods that enabled an AuthorizationEngine to return the response for these APIs. Ultimately this feature is the work included in: elastic#37785, elastic#37495, elastic#37328, elastic#36245, elastic#38137, elastic#38219 Closes elastic#32435

For some users, the built in authorization mechanism does not fit their needs and no feature that we offer would allow them to control the authorization process to meet their needs. In order to support this, a concept of an AuthorizationEngine is being introduced, which can be provided using the security extension mechanism. An AuthorizationEngine is responsible for making the authorization decisions about a request. The engine is responsible for knowing how to authorize and can be backed by whatever mechanism a user wants. The default mechanism is one backed by roles to provide the authorization decisions. The AuthorizationEngine will be called by the AuthorizationService, which handles more of the internal workings that apply in general to authorization within Elasticsearch. In order to support external authorization services that would back an authorization engine, the entire authorization process has become asynchronous, which also includes all calls to the AuthorizationEngine. The use of roles also leaked out of the AuthorizationService in our existing code that is not specifically related to roles so this also needed to be addressed. RequestInterceptor instances sometimes used a role to ensure a user was not attempting to escalate their privileges. Addressing this leakage of roles meant that the RequestInterceptor execution needed to move within the AuthorizationService and that AuthorizationEngines needed to support detection of whether a user has more privileges on a name than another. The second area where roles leaked to the user is in the handling of a few privilege APIs that could be used to retrieve the user's privileges or ask if a user has privileges to perform an action. To remove the leakage of roles from these actions, the AuthorizationService and AuthorizationEngine gained methods that enabled an AuthorizationEngine to return the response for these APIs. Ultimately this feature is the work included in: #37785, #37495, #37328, #36245, #38137, #38219 Closes #32435

jaymode added >non-issue :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC labels Jan 23, 2019

jaymode requested a review from tvernum January 23, 2019 19:58

fix checkstyle

73fa5ad

tvernum approved these changes Jan 24, 2019

View reviewed changes

alpar-t reviewed Jan 24, 2019

View reviewed changes

jaymode added 4 commits January 24, 2019 08:19

Merge branch 'security_authz_engine' into pluggable_authz_engine

f561192

fix authz bug in example

a82db80

move example plugin

d1d8a86

combine docs

741375d

jaymode requested review from alpar-t and lcawl January 25, 2019 17:23

lcawl reviewed Jan 25, 2019

View reviewed changes

x-pack/docs/en/security/authorization/custom-authorization.asciidoc Outdated Show resolved Hide resolved

lcawl mentioned this pull request Jan 25, 2019

[DOCS] Replace use of custom-roles-provider.asciidoc elastic/stack-docs#191

Merged

lcawl reviewed Jan 25, 2019

View reviewed changes

x-pack/docs/en/security/authorization/custom-authorization.asciidoc Outdated Show resolved Hide resolved

lcawl reviewed Jan 25, 2019

View reviewed changes

jaymode added 5 commits January 25, 2019 15:00

apply feedback from @lcawl

39d651b

Merge branch 'security_authz_engine' into pluggable_authz_engine

8b27e9f

Merge branch 'security_authz_engine' into pluggable_authz_engine

e8fe606

add missing notconsole

9e149f1

Merge branch 'security_authz_engine' into pluggable_authz_engine

3de2f0a

jaymode merged commit 3280607 into elastic:security_authz_engine Jan 29, 2019

jaymode deleted the pluggable_authz_engine branch January 29, 2019 21:17

This was referenced Feb 1, 2019

Allow custom authorization with an authorization engine #38225

Closed

Allow custom authorization with an authorization engine (6.x) #38228

Merged

Allow custom authorization with an authorization engine #38358

Merged

	To use a security extension:
	To use the authorization engine:

Conversation

jaymode commented Jan 23, 2019

Uh oh!

elasticmachine commented Jan 23, 2019

Uh oh!

lcawl commented Jan 23, 2019

Uh oh!

jaymode commented Jan 23, 2019

Uh oh!

alpar-t Jan 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaymode Jan 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rjernst Jan 24, 2019

Choose a reason for hiding this comment

Uh oh!

jaymode Jan 25, 2019

Choose a reason for hiding this comment

Uh oh!

alpar-t Jan 28, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lcawl commented Jan 25, 2019

Uh oh!

Uh oh!

lcawl Jan 25, 2019

Choose a reason for hiding this comment

Uh oh!

jaymode Jan 25, 2019

Choose a reason for hiding this comment

Uh oh!

lcawl Jan 25, 2019

Choose a reason for hiding this comment

Uh oh!

jaymode Jan 25, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

alpar-t Jan 24, 2019 •

edited

Loading

jaymode Jan 24, 2019 •

edited

Loading