Last active
August 18, 2016 22:34
Revisions
-
whitlockjc revised this gist
Aug 18, 2016 . 1 changed file with 3 additions and 4 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -35,8 +35,8 @@ document and it is used as part of [OpenID Connect Discovery][oidc-discovery], s ## OpenID Connect Discovery Instead of giving up, we decided to look into what it takes to implement OIDC Discovery starting with the `$OIDC_ISSUER_URL/.well-known/openid-configuration`. Reading the [Obtaining OpenID Provider Configuration Information][oidc-obtaining-provider-configuration-info] portion of the OIDC Discovery specification, we learned that `$OIDC_ISSUER_URL/.well-known/openid-configuration` is used by OIDC clients to obtain the OpenID Provider Configuration. So now that we understand the structure of the URL that Kubernetes was @@ -133,8 +133,7 @@ After this was up, we re-ran `kubectl get pods` and this time we got another err `JWT claims invalid: invalid claim value: 'iss'. expected=$ISSUER_URL, found=$ISSUER_URL/oauth/token., crypto/rsa: verification error` _(notice the extra `/oauth/token`)_ **Note:** At this point I would like to point out that while progress is being made, we were beginning to think we would continue down this rabbit hole forever. The good news was this error was easy to understand and that is based on the OpenID Provider Metadata documentation, the `iss` claim value **MUST MATCH** the `issuer` value of the OpenID Provider Metadata document. Unfortunately, this is -
Aug 18, 2016 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -138,7 +138,7 @@ we were going to continue down this rabbit hole. The good news was this error was easy to understand and that is based on the OpenID Provider Metadata documentation, the `iss` claim value **MUST MATCH** the `issuer` value of the OpenID Provider Metadata document. Unfortunately, this is not something you can toggle within UAA which led to a [pull request][uaa-pulls-425]. The purpose of this PR was to get the ball rolling on fixing this officially and that PR contains the exact changes we made to our custom UAA server to fix this. Once we deployed the new version of UAA with the PR changes made, lo and behold `kubectl get pods` worked as expected. -
Aug 18, 2016 . 1 changed file with 3 additions and 3 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -8,7 +8,7 @@ useful to you as well. **Note:** This post provides background on the process we took and how we successfully wired things up. If you do not care about this and just want to know the steps required to use UAA, and possibly other OAuth 2.0 providers, as an OIDC provider for Kubernetes, please skip to the [Cliffs Notes](#cliffs_notes) section. ## Kubernetes Authentication @@ -96,7 +96,7 @@ kubectl config use-context $CONTEXT_NAME Here is an example: **Note:** It is only a coincidence that we use `kube-solo-secure` for all names in the examples below. That is not a requirement and is done purely to make cleaning things up simpler. ``` @@ -152,7 +152,7 @@ of a handful of steps that workaround UAA's lack of OIDC support. If that is th 1. Wait until UAA officially supports OIDC 2. Use [dex][]'s [UAA support][dex-uaa-support] ## Cliffs Notes<a name="cliffs_notes"></a> The explanation above discusses how we got to a working deployment of UAA being used for Kubernetes authentication via OIDC. To summarize the things that were required, here is a bulleted list of the steps: -
Aug 17, 2016 . 1 changed file with 1 addition and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -171,6 +171,7 @@ long as it matches the `jwks_uri` property in your OpenID Provider Metadata docu the URLs mentioned in steps 2 and 4 7. Update the Kubernetes API Server options to have the `--oidc-client-id` option set to the UAA OAuth client application created in step 5 8. Update the Kubernetes API Server options for OIDC as you need not related to steps 6 and 7 ## Conclusion -
Aug 17, 2016 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,201 @@ # Kubernetes Authentication with UAA Recently at Apigee we have started using [Kubernetes][kubernetes] and while working on securing access to it, we learned a few things that we felt could be useful to other Kubernetes consumers. This post will discuss how we were able to use CloudFoundry's [UAA][uaa] as an [OpenID Connect Provider][oidc] for Kubernetes authentication. If you are not using UAA but you are using an [OAuth2][oauth2] provider for your authentication needs, stick around because this post could be useful to you as well. **Note:** This post provides background on the process we took and how we successfully wired things up. If you do not care about this and just want to know the steps required to use UAA, and possibly other OAuth 2.0 providers, as an OIDC provider for Kubernetes, please skip to [The Cliffs Notes](#the_cliffs_notes) section. ## Kubernetes Authentication When starting down the path of securing access to Kubernetes, our ultimate goal was to use our existing single-sign-on solution. Upon evaluating the [different options][kubernetes-authn] Kubernetes offers for cluster authentication, there was only one option that seemed plausible and that was the OpenID Connect _(OIDC)_ authentication provider. OIDC is basically a _"simple identity layer on top of the OAuth 2.0 protocol"_ and it just so happens that UAA is an OAuth 2.0 provider with _limited_ OIDC support. Even with only limited OIDC support, this seemed like as good a place as to start. Our first step was to see how far we could get by configuring Kubernetes to use our UAA server as an OIDC provider. To do this we passed the following command-line options to the Kubernetes API Server _(based on the [OpenID Connect Tokens](kubernetes-oidc) section of the Kubernetes Authentication guide)_: * `--oidc-issuer-url`: This tells Kubernetes where your OIDC server is * `--oidc-client-id`: This tells Kubernetes the OAuth client application to use But during startup, the API Server failed to start and we saw errors like this: `Failed to fetch provider config, trying again in 3s: invalid character '<' looking for beginning of value`. After some digging in the Kubernetes sources, and the [go-oidc][] sources, we found out that upon start, the Kubernetes API Server expects to find a document located at `$OIDC_ISSUER_URL/.well-known/openid-configuration`. What kind of file is this and what are its contents? After some Googling around, we found out this document is an _OpenID Provider Metadata_ document and it is used as part of [OpenID Connect Discovery][oidc-discovery], something UAA itself does not support. ## OpenID Connect Discovery Instead of giving up when we found out that UAA does not implement OIDC Discovery, we decided to look into what it takes to implement OIDC Discovery starting with the `$OIDC_ISSUER_URL/.well-known/openid-configuration`. Reading the [Obtaining OpenID Provider Configuration Information][oidc-obtaining-provider-configuration-info] portion of the OIDC Discovery specification, we learned that `$OIDC_ISSUER_URL/.well-known/openid-configuration` is used by OIDC clients to obtain the OpenID Provider Configuration. So now that we understand the structure of the URL that Kubernetes was looking for, we now need to understand the structure of this document. As expected, the OIDC Discovery specification explains this in the [OpenID Provider Metadata][oidc-provider-metadata] section. Since this post is not an OpenID Connect tutorial, I will instead point you to a few public OpenID Providers for reference: * `https://accounts.google.com` _(Documented [here][google-oidc-docs])_: <https://accounts.google.com/.well-known/openid-configuration> * `https://login.salesforce.com` _(Documented [here][salesforce-oidc-docs])_: <https://login.salesforce.com/.well-known/openid-configuration> Based on the OIDC Discovery specification and the various examples we found online from public OpenID Connect Providers, we felt confident that we could create an OpenID Provider Metadata document for our UAA server and that was our next step. **Note:** Since UAA does not support OpenID Connect Discovery, we had to serve the OpenID Provider Metadata document ourselves so you will likely need to solve this as well. ## JSON Web Tokens and Signing Once we had our OpenID Provider Metadata document served at `$OIDC_ISSUER_URL/.well-known/openid-configuration` we restarted the API Server and this time, there were no errors related to OIDC and the API Server started successfully. The next step was to get a token and attempt to authenticate to Kubernetes using said token. Of course, depending on your environment _how_ you get your token will change but for UAA users, you could use the [uaac][] to do this like so: ``` # Set the uaac target (The UAA server location) uaac target $OIDC_ISSUER_URL # Get a user token from UAA uaac token authcode get # Print the uaac contexts uaac contexts ``` The last command will print out your uaac contexts and one should match your target server. Once you find it, the token you need is the `access_token` property. Once we have the token from UAA, we need to create/update our Kubernetes client _(`kubectl`)_ context to contain our newly-retrieved token like so: ``` # Create a new kubectl cluster configuration kubectl config set-cluster $CLUSTER_NAME --server=$K8S_SERVER_URL --certificate-authority=$K8S_CA_CRT # Configure a context user (This user IS NOT the username used to authenticate to Kubernetes, that is in your token) kubectl config set-context $CONTEXT_NAME --cluster=$CLUSTER_NAME --user=$USER_NAME # Configure the context user to use the token we just retrieved kubectl config set-credentials $USER_NAME --token=$TOKEN # Configure kubectl to use the context we just created kubectl config use-context $CONTEXT_NAME ``` Here is an example: **Note:** It is only a coincedence that we use `kube-solo-secure` for all names in the examples below. That is not a requirement and is done purely to make cleaning things up simpler. ``` kubectl config set-cluster kube-solo-secure --server=kube-solo-secure --certificate-authority=/tmp/ca.crt --embed-certs kubectl config set-context kube-solo-secure --cluster=kube-solo-secure --user=kube-solo-secure kubectl config set-credentials kube-solo-secure --token="$TOKEN" kubectl config use-context kube-solo-secure ``` Each of these commands above should have output a value of `[cluster|context|user] "kube-solo-secure" set.`, except for the `kubectl config use-context` command which should have output `switched to context "kube-solo-secure".`. Once this was done, we were ready to see how much further this got us so we ran `kubectl get pods` and unfortunately, we got this error: `error: you must be logged in to the server (the server has asked for the client to provide credentials)` Looking into the API Server logs we saw the following error: `Unable to authenticate the request due to an error: [oidc: failed syncing KeySet: illegal base64 data at input byte 19, crypto/rsa: verification error]` After a great deal of research and digging around, we found out that [JSON Web Keys][json-web-keys] document, whose location is set via the `jwks_uri` in the OpenID Provider Metadata document, was invalid and that's when we ran into our first incompatibility with UAA's OIDC support. ## UAA's Incompatibility with OIDC JSON Web Keys _(JWS)_ are used to verify JSON Web Tokens _(JWT)_ and the structure of a JWS mandates that the modulus used to verify signatures is to be `base64url` encoded but the modules _(the `n` property of the JWS provided by UAA)_ was only `base64` encoded. So UAA is not encoding their JWS appropriately per the JWS specification. This led to us [filing a bug][uaa-issues-424] and coming up with a workaround. Much like the need for us to host our own `/.well-known/openid-configuration` document alongside UAA, we also created a new version of the JWS file _(`/token_keys`)_ at _(`/k8s_token_keys`)_ and updated our OpenID Provider Metadata document to have the `jwks_uri` use the new document. After this was up, we re-ran `kubectl get pods` and this time we got another error: `JWT claims invalid: invalid claim value: 'iss'. expected=$ISSUER_URL, found=$ISSUER_URL/oauth/token., crypto/rsa: verification error` _(notice the extra `/oauth/token`)_ **Note:** At this point I would like to point out that while progress is being made, it was around this time we thought we were going to continue down this rabbit hole. The good news was this error was easy to understand and that is based on the OpenID Provider Metadata documentation, the `iss` claim value **MUST MATCH** the `issuer` value of the OpenID Provider Metadata document. Unfortunately, this is not something you cannot toggle within UAA which led to a [pull request][uaa-pulls-425]. The purpose of this PR was to get the ball rolling on fixing this officially and that PR contains the exact changes we made to our custom UAA server to fix this. Once we deployed the new version of UAA with the PR changes made, lo and behold `kubectl get pods` worked as expected. ## Alternatives We realize that the information above might not sit well with some. Building a custom version of UAA to help it implement OIDC just for Kubernetes authentication might seem like a bit much, not to mention that that alone is just one of a handful of steps that workaround UAA's lack of OIDC support. If that is the case, you have two options: 1. Wait until UAA officially supports OIDC 2. Use [dex][]'s [UAA support][dex-uaa-support] ## The Cliffs Notes<a name="the_cliffs_notes"></a> The explanation above discusses how we got to a working deployment of UAA being used for Kubernetes authentication via OIDC. To summarize the things that were required, here is a bulleted list of the steps: 1. Patch UAA _(Using this PR: https://github.com/cloudfoundry/uaa/pull/425)_ and rebuild to avoid `/oauth/token` from being appended to your `iss` claim 2. Create a version of `$UAA_SERVER/token_keys` that has the `n` properties `base64url` encoded instead of just `base64` encoded 3. Create an OpenID Provider Metadata document based on the [OpenID Provider Metadata][oidc-provider-metadata] and/or the examples linked to above 4. Serve your OpenID Provider Metadata and JWS documents at `$UAA_SERVER/.well-known/openid-configuration` and `$UAA_SERVER/k8s_token_keys` respectively _(The latter URL is just an example and you can use whatever path you want so long as it matches the `jwks_uri` property in your OpenID Provider Metadata document)_ 5. Create an OAuth client application in UAA that has the appropriate scope _(`openid`)_ 6. Update the Kubernetes API Server options to have the `--oidc-issuer-url` option set to the `$UAA_SERVER` portion of the URLs mentioned in steps 2 and 4 7. Update the Kubernetes API Server options to have the `--oidc-client-id` option set to the UAA OAuth client application created in step 5 ## Conclusion In the end, our goals were met and we were able to successfully use our single-sign-on solution for Kubernetes authentication. While it would be ideal if UAA supported OIDC and we could just point Kubernetes to UAA and call it good, he steps above are easy to repeat, safe and have allowed us to get what we need right now. [auth0-oidc-config]: https://sample.auth0.com/.well-known/openid-configuration [dex]: https://github.com/coreos/dex [dex-uaa-support]: https://github.com/coreos/dex/pull/542 [go-oidc]: https://github.com/coreos/go-oidc [google-oidc-docs]: https://developers.google.com/identity/protocols/OpenIDConnect [google-oidc-config]: https://accounts.google.com/.well-known/openid-configuration [json-web-keys]: https://tools.ietf.org/html/draft-ietf-jose-json-web-key-41 [kubectl-config-set-cluster]: http://kubernetes.io/docs/user-guide/kubectl/kubectl_config_set-cluster/ [kubernetes]: http://kubernetes.io/ [kubernetes-authn]: http://kubernetes.io/docs/admin/authentication/ [kubernetes-oidc]: http://kubernetes.io/docs/admin/authentication/#openid-connect-tokens [oauth2]: http://oauth.net/2/ [oidc]: http://openid.net/connect/ [oidc-discovery]: https://openid.net/specs/openid-connect-discovery-1_0.html [oidc-obtaining-provider-configuration-info]: https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfig [oidc-provider-metadata]: https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderMetadata [salesforce-oidc-docs]: https://help.salesforce.com/apex/HTViewHelpDoc?id=remoteaccess_using_openid_discovery_endpoint.htm [uaa-issues-424]: https://github.com/cloudfoundry/uaa/issues/424 [uaa-pulls-425]: https://github.com/cloudfoundry/uaa/pull/425 [uaa]: https://github.com/cloudfoundry/uaa [uaac]: https://github.com/cloudfoundry/cf-uaac