Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce 'scope' field in key value model #2669

Merged
merged 35 commits into from
May 18, 2016

Conversation

lakshmi-kannan
Copy link
Contributor

@lakshmi-kannan lakshmi-kannan commented May 4, 2016

TODO:

  • GET on user scoped variables ** Hits V1 APIs with ?scope= query param.
  • LIST on scoped variables ** Hits V1 APIs with ?scope= query param.
  • V2 APIs for scoped key value access
  • Backward compatibility tests
    • Migration script if needed
    • st2-packages changes to include migration script. DECIDED TO WRITE DOCS FOR MIGRATION
  • KeyValueLookup should work with user scopes
  • Tests
  • Docs --> Decided to write this after we support resolving jinja resolution.
  • CLI shows 400: Bad request but doesn't tell what is bad about the request
  • Changelog

Prefix based RBAC will be in another PR. Trying to scope things down.

CLI examples:

GET

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key set company brocade     [ruby-1.9.3p484]
+------------------+---------+
| Property         | Value   |
+------------------+---------+
| name             | company |
| value            | brocade |
| expire_timestamp |         |
+------------------+---------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key get company             [ruby-1.9.3p484]
+------------------+---------+
| Property         | Value   |
+------------------+---------+
| name             | company |
| value            | brocade |
| secret           | False   |
| encrypted        | False   |
| scope            | system  |
| expire_timestamp |         |
+------------------+---------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key set boss lakstorm --scope=user
+------------------+----------+
| Property         | Value    |
+------------------+----------+
| name             | boss     |
| value            | lakstorm |
| expire_timestamp |          |
+------------------+----------+

## XXX: Get needs to be fixed so asking for something without scope "user" doesn't return it. 

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key get boss                [ruby-1.9.3p484]
+------------------+----------+
| Property         | Value    |
+------------------+----------+
| name             | boss     |
| value            | lakstorm |
| secret           | False    |
| encrypted        | False    |
| scope            | user     |
| expire_timestamp |          |
+------------------+----------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key set manas grump --scope=nsfw
ERROR: 400 Client Error: Bad Request
MESSAGE: Invalid scope "nsfw"! Allowed scopes are ['system', 'user']. for url: http://127.0.0.1:9101/v1/keys/manas

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$

LIST

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key list --scope=user       [ruby-1.9.3p484]
+------+-------+--------+-----------+------------------+
| name | value | secret | encrypted | expire_timestamp |
+------+-------+--------+-----------+------------------+
+------+-------+--------+-----------+------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key list --scope=system     [ruby-1.9.3p484]
+------+-------+--------+-----------+------------------+
| name | value | secret | encrypted | expire_timestamp |
+------+-------+--------+-----------+------------------+
| foo  | bar   | False  | False     |                  |
+------+-------+--------+-----------+------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key set boss lakstorm --scope=user
+------------------+----------+
| Property         | Value    |
+------------------+----------+
| name             | boss     |
| value            | lakstorm |
| expire_timestamp |          |
+------------------+----------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key list --scope=user       [ruby-1.9.3p484]
+------+----------+--------+-----------+------------------+
| name | value    | secret | encrypted | expire_timestamp |
+------+----------+--------+-----------+------------------+
| boss | lakstorm | False  | False     |                  |
+------+----------+--------+-----------+------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$

v2 API examples

PUT

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$ http PUT "http://localhost:9101/v2/keys/user/company" < kv.json
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 195
Content-Type: application/json; charset=UTF-8
Date: Sat, 07 May 2016 01:07:40 GMT
X-Request-Id: 24183b1b-30f1-45e3-a4ab-4460a7fa620d

{
    "encrypted": false,
    "id": "572d3fdcd9d7ed14d88a3e23",
    "name": "company",
    "scope": "user",
    "secret": false,
    "uid": "key_value_pair:user:company",
    "value": "BROcade"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$

DELETE

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$ http DELETE "http://localhost:9101/v2/keys/user/company"
HTTP/1.1 204 No Content
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 4
Date: Sat, 07 May 2016 01:08:02 GMT
X-Request-Id: 673981e0-c1f0-4117-80cf-7d539567e64d



(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$

GET

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$ http "http://localhost:9101/v2/keys/user/foo"
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive


  1 {
Content-Length: 187
Content-Type: application/json; charset=UTF-8
Date: Sat, 07 May 2016 01:06:02 GMT
X-Request-Id: 1a1bfc27-c54f-4ac6-8ab9-5107954c2dd2

{
    "encrypted": false,
    "id": "572d3dd7d9d7ed0c17a9857c",
    "name": "foo",
    "scope": "user",
    "secret": false,
    "uid": "key_value_pair:user:foo",
    "value": "userbar"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$ http "http://localhost:9101/v2/keys/system/foo"
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 187
Content-Type: application/json; charset=UTF-8
Date: Sat, 07 May 2016 01:06:07 GMT
X-Request-Id: 493f459f-b9df-4fc6-a200-d122298e61d6

{
    "encrypted": false,
    "id": "572d3dbed9d7ed0c17a9857b",
    "name": "foo",
    "scope": "system",
    "secret": false,
    "uid": "key_value_pair:system:foo",
    "value": "bar"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$

v2 APIs with secrets

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$ http PUT "http://localhost:9101/v2/keys/user/company" < kv_secret.json
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 300
Content-Type: application/json; charset=UTF-8
Date: Sat, 07 May 2016 01:10:30 GMT
X-Request-Id: 33cc6994-f140-42a9-a3b4-31211d79220c

{
    "encrypted": true,
    "id": "572d4086d9d7ed14d88a3e24",
    "name": "company",
    "scope": "user",
    "secret": true,
    "uid": "key_value_pair:user:company",
    "value": "0083E1033A54F9FAB3336FDA3F6FE0A658A02A21DC6BA530A5FB062B13999A85418906DEE8D62CFD13BDF64863013DBEDAB998C742D4767A10"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$ http "http://localhost:9101/v2/keys/user/company"
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 300
Content-Type: application/json; charset=UTF-8
Date: Sat, 07 May 2016 01:10:45 GMT
X-Request-Id: 45f12a60-8593-4ae1-800d-befa8590f8c0

{
    "encrypted": true,
    "id": "572d4086d9d7ed14d88a3e24",
    "name": "company",
    "scope": "user",
    "secret": true,
    "uid": "key_value_pair:user:company",
    "value": "0083E1033A54F9FAB3336FDA3F6FE0A658A02A21DC6BA530A5FB062B13999A85418906DEE8D62CFD13BDF64863013DBEDAB998C742D4767A10"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$ http "http://localhost:9101/v2/keys/user/company?decrypt=true"
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 194
Content-Type: application/json; charset=UTF-8
Date: Sat, 07 May 2016 01:10:53 GMT
X-Request-Id: 623d82d8-bc63-45e1-a4a7-4c0743e93c72

{
    "encrypted": false,
    "id": "572d4086d9d7ed14d88a3e24",
    "name": "company",
    "scope": "user",
    "secret": true,
    "uid": "key_value_pair:user:company",
    "value": "BROcade"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$

doc['value'] = symmetric_decrypt(KeyValuePairAPI.crypto_key, model.value)
encrypted = False

scope = getattr(model, 'scope', SYSTEM_KV_PREFIX)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally view scope and key prefix as two different things and they don't need to be the same (scope is a user visible feature and key prefix for a particular scope is an implementation detail).

In any case, I would have those scopes:

  • user - User scopes, keys are prefixed with some common prefix for this scope (user..fooor_user..foo`)
  • global - Anything which doesn't have a prefix (so any other existing key). We could explicitly prefix keys with a global scope, but the question is how to handle the keys without the prefix then.

Or do you want to introduce also a concept of no scope?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like "user" and "global" scopes as well. I even anticipate more scopes like "environment" etc. Our current story for key value store is everything is global and you can "refer" to a variable via {{system.foo}} even though "foo" is the actual key name. So for backward compatibility, I am setting the scope to "system" and don't want to introduce another name "global". If you guys think this is a bad idea, we should talk. /cc: @manasdk

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we did design this from scratch we would have the scope property and assign it to system/global by default. The other choice would be to have {{system.foo}} and {{system.user.foo}} where system is still a prefix and what follows is possibly a scope i.e. user or something else in the future if we allow for user configurable scopes. I actually do prefer system transitioning to a scope although it is really from the scope perspective better named global. I wish I had been prescient and made space for this when I added ability to look up kv-store values in rules, actions etc.

Taking this forward it would mean that when {{system.foo}} and {{user.foo}} we now grab 2 top level words i.e. system and user for the context addressable space. Not ideal but not really much worse that what we have already. I feel we should only grab {{$user.foo}} or some such obscure token where 2 purposes are served -

  1. $ suggest it will be replaced
  2. $user is less-likely to appear in an addressable context

from_model_kwargs=from_model_kwargs,
**kwargs
)
kvp_db = kvp_db[0] if kvp_db else None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why/how would get_one by name+scope return multiple values?

I feel this code needs to be in the service layer where there is a method get_one(scope, name)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why/how would get_one by name+scope return multiple values?

It is not returning more than one value. It simply returns an array with one item which is both counter intuitive and blows up CLI.


name = me.StringField(required=True, unique=True)
value = me.StringField()
secret = me.BooleanField(default=False)
expire_timestamp = me.DateTimeField()
scope = me.StringField(default=SYSTEM_KV_PREFIX)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens to old data in DB with new code? Still works?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I captured it in a TODO item. Backward compatibility. Yet to be done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what happens when I read code and dont read the description :P

@manasdk
Copy link
Contributor

manasdk commented May 5, 2016

@lakshmi-kannan What does st2 key list show? Would there be a scope column?

@lakshmi-kannan
Copy link
Contributor Author

@manasdk Not yet

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●●)$ st2 key list                    [ruby-1.9.3p484]
+------+-------+--------+-----------+------------------+
| name | value | secret | encrypted | expire_timestamp |
+------+-------+--------+-----------+------------------+
| foo  | bar   | False  | False     |                  |
+------+-------+--------+-----------+------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (issue_167/user_scoped_vars●)$

@@ -242,6 +242,25 @@ def _get_from_model_kwargs_for_request(self, request):
"""
return self.from_model_kwargs

def _get_one_by_scope_and_name(self, scope, name, from_model_kwargs=None):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@manasdk FYI

Lakshmi Kannan added 4 commits May 12, 2016 17:20
* master: (62 commits)
  Update affected tests.
  Update affected test.
  Remove some unecessary duplicated variable declarations.
  Update a base RBAC tests class to insert a user with no permissions and role assignments.
  Add RBAC tests for runner types API endpoint.
  Update "_get_by_name_or_id" method so it's consistent with "_get_by_ref_or_id" and throws if a resource is not found.
  Update changelog.
  Remove now old and unused assertion methods.
  Update more RBAC related assertion methods and update existing tests to use those new methods.
  Add more developer-friendly assertion methods and update affected code to utilize those new methods.
  Fix lint.
  Add new assertion methods for asserting RBAC permissions grants and update affected tests to use those new methods.
  Add new _user_doesnt_have_resource_db_permissions assertion method which is the inverse of the existing one and update tests to use it.
  Add a test case for disabled runner.
  Update more affected tests.
  Update affected tests - add missing required "name" attribute to the RunnerTypeDB objects.
  Use getattr, weak typing ftl.
  Add RBAC resolver tests for runner type resolver.
  Add a test case for disabling and re-enabling a runner.
  Only allow "enabled" attribute of the runner to be updated via the API.
  ...
@lakshmi-kannan lakshmi-kannan changed the title WIP: Introduce 'scope' field in key value model Introduce 'scope' field in key value model May 13, 2016
* master:
  Add a test case for it.
  Update changelog.
  Allow register-setup-virtualenvs flag to be used in combination with register-all in the `st2-register-content script.
  Add missing abspath call.
  Update changelog.# On branch pack_test_fixture_loading_utils
  Add new "get_fixture_content" method to all the pack resource test classes.

Conflicts:
	CHANGELOG.rst
"""
instance = self.access.get_by_scope_and_name(scope=scope, name=name)
if not instance:
return instance
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_get_one_* methods usually return 404 (pecan.abort) if a object is not found (and _get_by_* throw StackStormDBObjectNotFoundError if an object is not found, and _get_one methods usually call _get_by methods).

We should do the same here for consistency and also add corresponding test case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also seems like this is kinda key value pair controller specific so maybe it should live in base key value pair controller or similar?

Copy link
Contributor Author

@lakshmi-kannan lakshmi-kannan May 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You made that change (StackStormDBObjectNotFoundError) already. I'll wait for you to merge that and make the corresponding change here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also seems like this is kinda key value pair controller specific so maybe it should live in base key value pair controller or similar?

I had it in KeyValuePairController and manas said move it to some base layer. Geez. Ignoring.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed StackStormDBObjectNotFoundError - feef877

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lesson -> don't listen to manas :P

j/k

@Kami
Copy link
Member

Kami commented May 16, 2016

I've added some in-line comments.

Overall, I'm fine with other changes, but as mentioned on Slack, I would remove v2 stuff from this PR.

I think we all need to get on the same page and have a consistent story for the v2 API so we don't need to deprecate v2 yet again in the near future when we plan to work on an improved API.

Also, what's our plan for RBAC here? Will this be done separately, or?

@manasdk
Copy link
Contributor

manasdk commented May 16, 2016

I would remove v2 stuff from this PR.

What is wrong with it? is the v2 API bad already?

@Kami
Copy link
Member

Kami commented May 16, 2016

@manasdk The problem is that it's a one off-thing.

We need a consistent story and plan for it (that means how paths are going to look like, responses, etc.). I'm fine with adding it as soon as we have a consistent story and plan for it, because then we can make sure this new v2 API follows all of those conventions.

Tomorrow I plan to start a discussion thread on v2 API and we can continue discussion about that there.

@manasdk
Copy link
Contributor

manasdk commented May 16, 2016

The problem is that it's a one off-thing.

Is it ad-hoc? Isn't this exactly the reason i.e. we are changing a model therefore need to have a new endpoint. I am fine with the discussion driving the impl and then coming around to picking up changes here but why would you suggest reverting the changes?

@Kami
Copy link
Member

Kami commented May 16, 2016

@manasdk Yes, and the same end result can be achieved using v1 API so I see no point in rushing v2 API until we have a clear picture on how we want it to look.

Lakshmi Kannan added 6 commits May 17, 2016 19:59
* master: (28 commits)
  register ACTIONEXECUTIONSTATE_XCHG
  register LIVEACTION_STATUS_MGMT_XCHG exchange by default
  For consistency, make sure that _get_by method doesn't throw.
  Fix assertion method name.
  Add tests for config schema API endpoints.
  Fix lint.
  Update packs.uninstall action to also delete corresponding ConfigSchemaDB object.
  Now that it's a top-level endpoint it makes sense to have "get all" / list endpoint.
  Don't use pack cache in the tests.
  Add tests for pack and config schema registrar.
  Update changelog.
  Add config schema fixture for dummy_pack_1.
  Fix lint, add missing file.
  Add API endpoints RBAC tests for pack config schema.
  Fix RBAC for Pack Config Schema.
  Add "get_by_pack" method and fix resource registrar.
  Add comment and assertion to detect programmer errors.
  Fix "_get_by_pack_ref" method to throw if the resource doesn't exist.
  Update affected test - this user is now added as part of the base class.
  Fix dummy pack metadata.
  ...

Conflicts:
	CHANGELOG.rst
@lakshmi-kannan
Copy link
Contributor Author

@manasdk @Kami : Removed v2 APIs here 67d7ed9

@lakshmi-kannan
Copy link
Contributor Author

lakshmi-kannan commented May 18, 2016

@Kami: > Also, what's our plan for RBAC here? Will this be done separately, or?

You should read the description sometimes before complaining everywhere :P

@lakshmi-kannan lakshmi-kannan merged commit 483b2a6 into master May 18, 2016
@lakshmi-kannan lakshmi-kannan deleted the issue_167/user_scoped_vars branch May 18, 2016 03:39
@@ -33,9 +34,12 @@ class RootController(BaseRootController):

def __init__(self):
v1_controller = v1_root.RootController()
v2_controller = v2_root.RootController()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably forgot to remove this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants