User Authentication#
Subsystem Goal#
This subsystem is responsible for authenticating users using their Virginia Tech identity and group memberships. That way, all platform-related actions occur only after two-factor authentication and align with proper account lifecycle management.
Components in Use#
- Kubernetes OIDC - validates OIDC tokens and impersonates the user against the underlying K8s API
- VT Gateway Service - OIDC protocol integration to VT Enterprise Directory
- Headlamp - a Kubernetes GUI that performs the OIDC flow and provides a webpage with easy copy/paste commands to configure a user's kubeconfig file
Background#
User Impersonation Basics#
Kubernetes has several methods to authenticate users, but are limited in their ability to connect to external identity providers, such as VT's identity systems. To support many different use cases, Kubernetes provides user impersonation. With this approach, a small K8s API can perform authentication or validate tokens and then pass along pertinent details when forwarding the request to the underlying Kubernetes API. If the user isn't authenticated, no forward is made. It is this process we are leveraging.
When proxy requests are performed, both the username and group details are able to
be specified. If a user named johnny.bravo
is in groups groupA
, groupB
, and
groupC
, the following headers are sent during the impersonation:
Impersonate-User: johnny.bravo
Impersonate-Group: groupA
Impersonate-Group: groupB
Impersonate-Group: groupC
From there, normal RoleBinding
and ClusterRoleBinding
definitions can be used to
grant access based on user (not preferred) or groups (preferred). The following
will grant access to a platform-tenant
Role to the namespace tenant-a
to all members
in the groupA
group.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: platform-tenant-access
namespace: tenant-a
subjects:
- kind: Group
name: groupA
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: platform-tenant
apiGroup: rbac.authorization.k8s.io
Group Namespacing
In most impersonation scenarios, it is better to namespace the groups. In our
implementation, since we are using OIDC (more on that shortly), the groups
passed through impersonation are prefixed with oidc:
. Therefore, all bindings
would use a name of oidc:<ed-group-name>
.
Granting Tokens to Users#
Now that we understand how impersonation works, how do we actually obtain or grant tokens to our users? Fortunately, the VT Gateway Service is an OAuth service that also supports OIDC (read about OAuth and OIDC here or here). By leveraging this service, a user is able to authenticate and obtain a JWT that contains many claims about the user, including their group memberships (read more about JWTs here). An example of the claims from a VT Gateway user token is below:
{
"sub": "ABCDEF135872635876====",
"email_verified": true,
"iss": "https://gateway.login.vt.edu",
"active": true,
"ttl": 86400,
"aud": "882bfd93-b97b-4ed5-abf5-9ce4065fa10e",
"scope": [],
"name": "Johnny Bravo",
"groupMembershipUugid": [
"it.platform.clusters.dvlp.admins",
"it.platform.roles.admin"
],
"exp": 1644702214,
"iat": 1644615814,
"jti": "abcdef1234567890987654321",
"email": "example@vt.edu"
}
So, if we have a simple app that can issue credentials, we can configure an API proxy to validate and trust those credentials and then forward the request. Headlamp provides this functionality! The entire flow looks like this:
Phew! That certainly seems a little complicated. But, it's really slick and mostly invisible to our users (unless they watch all of the redirects going on). There are other steps involved there, such as the Identity Provider actually sending the user off to VT's CAS service to actually perform the authentication. But, that's beyond our control. We simply care that we get a token issued by VT's identity services.
How it's Configured#
Since this subsystem is hard to replicate locally, we'll first explain how it works in the deployed platform. Then, we'll deploy a modified version to get a hands-on understanding of how the RBAC system works.
Configuring Kubernetes OIDC#
AWS Config - Uses the aws_eks_identity_provider_config resourse from the EKS module. EKS-A Config - Uses the OIDCConfig API provided by EKS-A.
Deploying it Yourself#
Configuring Tenant Access#
Remember that sample-tenant
namespace we created during the GitOps
section? Let's create a few basic roles that grant read-only access to tenant users.
For simplicity, any user in the sample-tenant
group will be granted read-only
access.
-
Run the following command to grant read-only access to any user in the
sample-tenant
group:cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: readonly-tenant namespace: sample-tenant labels: platform.it.vt.edu/ed-group: sample-tenant subjects: - kind: Group name: oidc:sample-tenant apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: view apiGroup: rbac.authorization.k8s.io EOF
The key part here is the
subject
that indicates any user that auths with a group ofoidc:sample-tenant
will have theview
role. -
We're going to create a new kubeconfig context to test out our tenant user. Run the following:
-
Now, go to the Mock OIDC Provider and generate a token. You can provide any email, but be sure to add the
sample-tenant
group. -
Copy the kubectl command run it. You now have a configured token to use against the endpoint!
Testing out Tenant Access#
Now that we have a kube context configured, let's use it and test things out!
-
Run the following command to change our kubectl context to use the newly create tenant context:
In order for this to work with minikube you may need to run the below commands with a tag of
-
Now, we should be able to query the pods in the
sample-tenant
namespace and get results!And we should see something similar to this:
Hooray! We have access!
-
Now, let's try to run a new pod!
We should see an error looking something like this:
Error from server (Forbidden): pods is forbidden: User "oidc:test@vt.edu" cannot create resource "pods" in API group "" in the namespace "sample-tenant"
We have read-only access!
-
If we try to query any other namespace, we should be denied too!
And we should see a similar error:
-
When you're done testing things out, feel free to revert back to your default context to regain your admin rights. On Docker Desktop, you can run the following:
Wrapping Up#
Hopefully, you got a taste of how the user auth works. We leverage the central
VT auth services by obtaining tokens with claims about the user and create various
RBAC configurations to authorize members of groups to access specific resources.
On the platform, we have a secondary ClusterRole
that can grant exec permission
to a tenant as well. It's simple another RoleBinding
.
Later, we'll talk about how the Landlord chart helps us simplify the management of these resources. They're all very repetitive, with only minor adjustments. So, stay tuned!
What's next?#
Now that we have user authentication and authorization plugged in, it's time to explore how we add additional value on the platform. We'll do a little bit of exploring with log forwarding first!
Go to the Log Forwarding subsystem now!
Common Troubleshooting Notes#
Why is a tenant unable to view their resources?
-
As a platform admin, first validate the expected group is configured in the landlord tenant config.
-
Ensure there are
RoleBinding
objects in the tenant's namespace.If there are none, look at the landlord helm chart and try to figure out why it failed to deploy.
-
Look at the RoleBindings and validate one exists with the correct ED group.
You should see one with a Subject of
Group
and a Name ofoidc:<ed-group-name>
. If not, check the landlord again. -
If the RoleBinding exists and the user is still unable to see the namespace, that means the group is not in the user's JWT. Validate they have added the
mw-gateway
as a Service Viewer to the ED Group.
How do I audit requests made by a user?
All requests made against the K8s API are logged and sent to CloudWatch. This can be helpful to see both the requests made by a user, but also the details about a particular request (the groups that were part of the request impersonation and why a particular request was authorized).
-
Log into the proper AWS account and go to CloudWatch.
-
Find the log group named
/aws/eks/<cluster-identifier>/cluster
. -
Find a log stream that begins with
kube-apiserver-audit
. -
Perform a search using the following pattern:
How do I grant exec access to a tenant?
The RBAC config in the landlord supports multiple groups and levels of permissions for each group. Simply update the landlord config to grant them access.