# Tutorial: Standalone Envoy The tutorial shows how Envoy’s External [authorization filter](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html) can be used with OPA as an authorization service to enforce security policies over API requests received by Envoy. The tutorial also covers examples of authoring custom policies over the HTTP request body. ## Overview In this tutorial we'll see how to use OPA as an External Authorization service for the Envoy proxy. We'll do this by: * Running a local Kubernetes cluster * Creating a simple authorization policy in Rego and serving it via the Bundle API * Deploying a sample application with Envoy and OPA sidecars * Run some sample requests to see the policy in action Note that other than the HTTP client and bundle server, all components are co-located in the same pod. ## Running a local Kubernetes cluster To start a local Kubernetes cluster to run our demo, we'll be using [kind](https://kind.sigs.k8s.io/). info If you haven't used `kind` before, you can find installation instructions in the [project documentation](https://kind.sigs.k8s.io/#installation-and-usage). Create a cluster with the following command: ``` $ kind create cluster --name opa-envoy --image kindest/node:v1.33.1Creating cluster "opa-envoy" ... ✓ Ensuring node image (kindest/node:v1.33.1) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾... ``` Once the cluster is created, make sure your `kubectl` context is set to connect to the new cluster: ``` $ kubectl cluster-info --context kind-opa-envoyKubernetes control plane is running at ...CoreDNS is running at ...... ``` Listing the cluster nodes, should show something like this: ``` $ kubectl get nodesNAME STATUS ROLES AGE VERSIONopa-envoy-control-plane Ready control-plane 2m35s v1.33.1 ``` ## Creating & Serving our Policy Bundle This tutorial assumes you have some Rego knowledge, in summary the policy below does the following: * Checks that the JWT token is valid * Checks that the action is allowed based on the token payload `role` and the request path * Guests have read-only access to the `/people` endpoint, admins can create users too as long as the name is not the same as the admin's name. ``` # policy.regopackage envoy.authzimport input.attributes.request.http as http_requestdefault allow := falseallow if { is_token_valid action_allowed}is_token_valid if { token.valid now := time.now_ns() / 1000000000 token.payload.nbf <= now now < token.payload.exp}action_allowed if { http_request.method == "GET" token.payload.role == "guest" glob.match("/people", ["/"], http_request.path)}action_allowed if { http_request.method == "GET" token.payload.role == "admin" glob.match("/people", ["/"], http_request.path)}action_allowed if { http_request.method == "POST" token.payload.role == "admin" glob.match("/people", ["/"], http_request.path) lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub)}token := {"valid": valid, "payload": payload} if { [_, encoded] := split(http_request.headers.authorization, " ") [valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})} ``` Create a file called `policy.rego` with the above content and store it in a ConfigMap: ``` kubectl create configmap authz-policy --from-file policy.rego ``` Now that the policy is stored in a ConfigMap, we can spin up an HTTP server to make it available as a Bundle to OPA when it's making decisions for our application: ``` # bundle-server.yaml---apiVersion: apps/v1kind: Deploymentmetadata: name: bundle-server labels: app: bundle-serverspec: replicas: 1 selector: matchLabels: app: bundle-server template: metadata: labels: app: bundle-server spec: initContainers: - name: opa-builder image: openpolicyagent/opa:latest args: - "build" - "--bundle" - "/opt/policy/" - "--output" - "/opt/output/bundle.tar.gz" volumeMounts: - name: index mountPath: /opt/output/ - name: policy mountPath: /opt/policy/ containers: - name: bundle-server image: nginx:1.25 ports: - containerPort: 80 name: http volumeMounts: - name: index mountPath: /usr/share/nginx/html volumes: - name: index emptyDir: {} - name: policy configMap: name: authz-policy---apiVersion: v1kind: Servicemetadata: name: bundle-serverspec: selector: app: bundle-server ports: - protocol: TCP port: 80 targetPort: http ``` Create a file called `bundle-server.yaml` with the above content and apply it to the cluster: ``` kubectl apply -f bundle-server.yaml ``` Once the deployment is running, we can check that the bundle is available by running: ``` kubectl port-forward service/bundle-server 8080:80 ``` Before checking that the bundle has been generated correctly and is available to download: ``` $ curl -I localhost:8080/bundle.tar.gzHTTP/1.1 200 OK... ``` You may now exit the port-forwarding session, the bundle server will only be accessed from inside the cluster from now on. ## Deploying an application with Envoy and OPA sidecars In this tutorial, we are manually configuring the Envoy proxy sidecar to intermediate HTTP traffic from clients and our application. Envoy will consult OPA to make authorization decisions for each request by sending `CheckRequest` messages over a gRPC connection. We will use the following Envoy configuration to achieve this. In summary, this configures Envoy to: * Listen on port `8000` for HTTP traffic * Consult OPA for authorization decisions at 127.0.0.1:9191 & deny failing requests * Forward requests to the application at 127.0.0.1:8080 if ok. ``` # envoy.yamlstatic_resources: listeners: - address: socket_address: address: 0.0.0.0 port_value: 8000 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager codec_type: auto stat_prefix: ingress_http route_config: name: local_route virtual_hosts: - name: backend domains: - "*" routes: - match: prefix: "/" route: cluster: service http_filters: - name: envoy.ext_authz typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz transport_api_version: V3 with_request_body: max_request_bytes: 8192 allow_partial_message: true failure_mode_allow: false grpc_service: google_grpc: target_uri: 127.0.0.1:9191 stat_prefix: ext_authz timeout: 0.5s - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router clusters: - name: service connect_timeout: 0.25s type: strict_dns lb_policy: round_robin load_assignment: cluster_name: service endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 port_value: 8080admin: access_log_path: "/dev/null" address: socket_address: address: 0.0.0.0 port_value: 8001layered_runtime: layers: - name: static_layer_0 static_layer: envoy: resource_limits: listener: example_listener_name: connection_limit: 10000 overload: global_downstream_max_connections: 50000 ``` Create a `ConfigMap` containing the above configuration by running: ``` kubectl create configmap proxy-config --from-file envoy.yaml ``` Our application will be configured using a `Deployment` and `Service`. There are a few things to note: * the pods have an `initContainer` that configures the `iptables` rules to redirect traffic to the Envoy proxy. * the `demo-test-server` container is a simple user store using in-memory state. * the `envoy` container is configured to use the `proxy-config` `ConfigMap` we created earlier. * The OPA container is configured to download policy bundles from the in-cluster bundle server (`bundle-server.default.svc.cluster.local`). ``` # app.yamlkind: DeploymentapiVersion: apps/v1metadata: name: example-app labels: app: example-appspec: replicas: 1 selector: matchLabels: app: example-app template: metadata: labels: app: example-app spec: initContainers: - name: proxy-init image: openpolicyagent/proxy_init:v8 # Configure the iptables bootstrap script to redirect traffic to the # Envoy proxy on port 8000. Envoy will be running as 1111, and port # 8282 will be excluded to support OPA health checks. args: ["-p", "8000", "-u", "1111", "-w", "8282"] securityContext: capabilities: add: - NET_ADMIN runAsNonRoot: false runAsUser: 0 containers: - name: app image: openpolicyagent/demo-test-server:v1 ports: - containerPort: 8080 - name: envoy image: envoyproxy/envoy:v1.26.3 volumeMounts: - readOnly: true mountPath: /config name: proxy-config args: - "envoy" - "--config-path" - "/config/envoy.yaml" env: - name: ENVOY_UID value: "1111" - name: opa image: openpolicyagent/opa:latest-envoy-static args: - "run" - "--server" - "--addr=localhost:8181" - "--diagnostic-addr=0.0.0.0:8282" - "--set=services.default.url=http://bundle-server" - "--set=bundles.default.resource=bundle.tar.gz" - "--set=plugins.envoy_ext_authz_grpc.addr=:9191" - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow" - "--set=decision_logs.console=true" - "--set=status.console=true" - "--ignore=.*" livenessProbe: httpGet: path: /health?plugins scheme: HTTP port: 8282 initialDelaySeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: /health?plugins scheme: HTTP port: 8282 initialDelaySeconds: 1 periodSeconds: 3 volumes: - name: proxy-config configMap: name: proxy-config---apiVersion: v1kind: Servicemetadata: name: example-appspec: selector: app: example-app ports: - protocol: TCP port: 80 targetPort: 8080 ``` Deploy the application and Kubernetes Service to the cluster with: ``` kubectl apply -f app.yaml ``` Check that everything is working by listing the pod (make sure that all three pods are running ok). ``` $ kubectl get podsNAME READY STATUS RESTARTS AGEbundle-server-5d7bfffdb6-bgn86 1/1 Running 0 1mexample-app-74b4bc88-5d4wh 3/3 Running 0 1m ``` ## See the Policy in Action Run a shell inside the cluster to use for testing. We will use this in-cluster shell for the rest of the tutorial. ``` kubectl run curl --restart=Never -it --rm --image curlimages/curl:8.1.2 -- sh ``` Set two tokens for two users, Alice and Bob with different permissions. As defined by our policy: ``` export ALICE_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8"export BOB_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiYWRtaW4iLCAic3ViIjogIlltOWkifQ.5qsm7rRTvqFHAgiB6evX0a_hWnGbWquZC0HImVQPQo8" ``` ### Listing People Send a request to list people. This should succeed for both Alice and Bob. ``` curl -i -H "Authorization: Bearer $ALICE_TOKEN" http://example-app/people ``` ``` HTTP/1.1 200 OKcontent-type: application/jsondate: Tue, 18 Jul 2023 15:22:25 GMTcontent-length: 96x-envoy-upstream-service-time: 14server: envoy[{"id":"1","firstname":"John","lastname":"Doe"},{"id":"2","firstname":"Jane","lastname":"Doe"}] ``` And for Bob: ``` curl -i -H "Authorization: Bearer $BOB_TOKEN" http://example-app/people ``` ``` HTTP/1.1 200 OK...omitted... ``` ### Creating People Send a request to create a new user. This should fail for Alice but not Bob: ``` curl -i -H "Authorization: Bearer $ALICE_TOKEN" \ -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \ -X POST http://example-app/people ``` ``` HTTP/1.1 403 Forbiddendate: Tue, 18 Jul 2023 15:25:28 GMTserver: envoycontent-length: 0 ``` And for Bob, the request is permitted and the user is saved with an ID ``` curl -i -H "Authorization: Bearer $BOB_TOKEN" \ -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \ -X POST http://example-app/people ``` ``` HTTP/1.1 200 OKcontent-type: application/jsondate: Tue, 18 Jul 2023 15:28:20 GMTcontent-length: 51x-envoy-upstream-service-time: 11server: envoy{"id":"498081","firstname":"Foo","lastname":"Bar"} ``` ### Creating People: Conflict Our policy also blocks users from creating users with the same name, test that functionality with this request: ``` curl -i -H "Authorization: Bearer $BOB_TOKEN" \ -d '{"firstname":"Bob", "lastname":"Bar"}' -H "Content-Type: application/json" \ -X POST http://example-app/people ``` ``` HTTP/1.1 403 Forbiddendate: Tue, 18 Jul 2023 15:31:48 GMTserver: envoycontent-length: 0 ``` ## Shutting Down Exit the in-cluster shell by typing `exit`. Delete the cluster by running: ``` $ kind delete cluster --name opa-envoyDeleting cluster "opa-envoy" ...Deleted nodes: ["opa-envoy-control-plane"] ``` ## Wrap Up Congratulations on finishing the tutorial ! This tutorial showed how to use OPA as an External authorization service to enforce custom policies by leveraging Envoy’s External authorization filter. This tutorial also showed a sample OPA policy that returns a `boolean` decision to indicate whether a request should be allowed or not. Envoy's external authorization filter allows optional response headers and body to be sent to the downstream client or upstream. An example of a rule that returns an object that not only indicates if a request is allowed or not but also provides optional response headers, body and HTTP status that can be sent to the downstream client or upstream can be seen [in the opa-envoy-plugin documentation](https://github.com/open-policy-agent/opa-envoy-plugin#example-policy-with-object-response).