Venil Noronha (VMware) Service Mesh Day 2019
So the good parts about this approach is that you get isolation. That is now if service B goes down, your system is still going to partially respond. Um, and that’s good. You also get scalability. That is now you can increase the number of replicas for each of your services. Introduce a load balancer in front of them and your system can be independently scaled. Your development and deployment can be, uh, Much more speedier now because these services are not a monolith. And finally you can pick the language of your choice to implement these services. You just need to take care that the rest API works according to your contract.
We have a service named user service. We have a function called find user, which takes a find user request object as input and returns a find user response object as output. And both, both of these messages have strongly typed fields within. So you get type safety and protobufs gives you, uh, forward and backward compatibility. For example, if you rename one of these fields on the server end, the client is still going to respond because these fields are not referenced by name anymore. They are referenced by their position within the message. Protobufs when transferred over a gRPC are encoded as base 64 strings, which gives a better performance than json. And finally, the entire API here is written using the protobuf IDL. So you have a well defined contract definition. Now that we are happy with the gRPC and protobufs, we can replace the rest APIs for services A and B with gRPC.
However, the Web UI can only speak http. So the communication between the front end and the backend cannot happen now over gRPC. And one easy solution for this is to implement an http server, expose the rest API and map it to one of the gRPC backends. This is good if you have just a couple of services, but if you have a lot of microservices, maintenance would be a nightmare. So let’s take a look at a different approach here. For that quickly, service meshes. On the right side you see green boxes, which are your applications and the purple boxes are your proxies. Now the communication between these green boxes, that is services don’t happen directly, but it goes via this proxy. By doing that you get um, analytics of your requests and responses. These proxies can also modify your requests and responses as well. And finally you can do fancy routing with these.
So essentially it’s a dedicated infrastructure layer that handles service communication and they can manage complex topology of services, uh, with the help of these network proxies. They’re pretty lightweight and deployed alongside application processes. And your application don’t actually need to be aware that there is a proxy in front of them. If you look carefully at a single deployment, this is what it will look like. We have a service, a gRPC end point, and we have the proxy process in front of it. Both are two different processes. And the good part here is, uh, on my proxy, which serves as their default proxy for Istio has this a filter called gRPC web, which takes an http requests from the client and transcodes it into gRPC requests and it then converts the response back into http. Now if you can imagine, we can replace the http server that we built earlier with this proxy.
And when we deployed the previous, uh, architecture on Istio, this is a what it would look like. So we have services A and B and both of these are proxies for the transcoding. And we also have proxies in front of the web UI and the database. So now we understand why we need these proxies in front of services A and B. That is to transcode http requests to gRPC. But why do we need these proxies in front of the Web UI and the database? So for the web UI it can tell you, uh, the request size, the response size, the request duration, and other metrics. It can also do layer seven routing. And for the database it can tell you what tables are being accessed and what operations are being performed, and it can also do RBAC. So it gives you this generic transcoding for services A and B and it gives you observability for entire mesh.
And here are all the benefits of having a service mesh. Now we don’t need an external service registry to make your services discoverable. You also get robustness frameworks like rate-limiting, retries and circuit braking. You also get load balancing and test infrastructure so you can inject falls and delays and observe your system under such circumstances. Your services are dynamically configurable. That is you do not need to bring any of your services down in order to configure these proxies so you get better uptime. Istio by default gives you a whole set of dashboards so we can visualize these metrics. It can also do tracing and you can visualize your service mesh and finally using a service mesh, you can secure the communication happening between your services. That is by enabling mutual TLS and you can also do policies on your services. To showcase a gRPC web with Istio, I built a small web app, I call it the emoji web app. It’s available at this URL and the code that I’m going to show you here may not compile - just a heads up there.
Here’s the implementation of my server. It’s very simple. So we have this type call server with the insert emojis function. I’m using a package that I found on Github and when I pass in a string with embedded keywords in it, this returns a string with the Emoji icons in it. We take the output, output text and return it as response. In the main function, I create a simple gRPC server, register our implementation and finally start listening to request on Port 9000 we have the server ready and when I run it, this is what is going to do.
It’s going to wait for requests on Port 9000. To test the API, I created a simple client using Go LAN again. I have two flags, server and texts. The server is the address of the server and the text field. will take the input from the user and send it back to the, um, send it to the backend to observe the response. In the main function, I create a connection to the, to the backend. And then I use the connection to create a new instance of the service client. And finally I take the user input, create a new request, send it to the backend and print out the response.
We can go ahead and deploy these configurations. The first thing I deployed here is the server. I’m using the Istio cuttle ccommand to manually inject the proxy. The next thing I deploy is the web UI. And again, I will use the Isito cuttle command to inject the proxy here. We can also automate this. And, finally I deploy the gateway and virtual service configuration? Now that everything is up and running, we can fetch the pods running in the system. And as you notice both the services are up and running and we have two containers running within each pod. This is because one represents the service itself and the other represents the proxy. We can now go ahead and access the ingress gateway on the browser. This is the div that we created using the html that I showed earlier. And now when I type in characters, you see these request flowing to the backend and the text is going to be updated in real time.
Notice that the end point is insert emojis. The status code is 200 and the protocol is http 1.1, but gRPC is based on http 2. That is the state code is not a gRPC. But uh, http to be specific and let’s have a look at the requests in more detail. So this is the whole end point here. That is proto dot emoji service slash insert emojis. Um, we also see a bunch of headers specific to gRPC web. Now these are the response headers. And you can also notice that the server here is is Istio - envoy, which is the proxy itself.
You’ll see similar headers in the request object and finally the payload here is a base 64 string. Okay, so this is all good. Uh, and our gRPC web application is working pretty good on Istio but apart from this feature is still gives us a whole bunch of dashboards as I mentioned earlier. Let’s have a look at a few. Here is Grafana. We can see the request volume and the success rate for the web UI deployment here. The success rate is basically non five xx responses from this particular instance. We can also load a similar dashboard for the server instance.
And it also shows us a whole bunch of other details. For example, we can see the response code for this particular instance. You can also see the request duration, the request size and the response size. Next, let’s have a look at the tracing dashboard. Here is Jaeger and we are trying to find traces on the server deployment.
Now when I’m click on a particular trace, is it’s going to tell me how the request flowed through the system. In this case a request went through the ingress gateway, then it reached the server end point and when I click on the server instance, it’s going to show me all the details about the request. So you can see that the end point here is a proto dot emoji service slash insert emojis. The http method was “Get” and you can also see the user agent here, which is a Mozilla, that is the browser. However, the protocol here is http 2. Now if you remember the browser actually sent http 1.1 request and now here we see http 2, which is because the proxy has upgraded the request from http 1.1 to http 2 in order to satisfy the gRPC requirement.
We can use this dashboard to figure out slow services and failing services. And finally, let’s look at Kiali, which we chose as our service graph. This graph is generated real time based on the traffic flowing through these proxies and we can click on a single instance here and find more details about the response codes and also the load on the system at that point in time. So in conclusion, protobufs let you define API contracts and data models using the IDL and it also gives you forward and backward compatibility.
gRPC is based on http 2 and it also generates client stops for you so you don’t need to do it yourself. And since it’s based on http 2, you get better network performance and also you get a better performance than json because it encodes objects into base 64. gRPC web lets you take the protobuf and gRPC experience onto browsers and it needs a proxy like Envoy for that. Istio gives you Envoy and a whole bunch of dashboards for metrics for tracing and service graph. And Istio is also extensible so you can create your own adapters and wire in the backend that you want. In my case, I created a plug in for wavefront. So you can visualize the same metrics on wavefront now, get rid of your http. Questions? Yea?
So the question was that gRPC support bi-directional streaming, gRPC web does not do that. Um, and do we have plans to support bi-directional streaming with gRPC web? So the answer is no, because the browser cannot support the same use case. I was speaking to the maintainer of gRPC web last week and he mentioned about this problem that you cannot do this with browsers, but if Chrome decides to actually enable that, I think it’s going to be possible. Any other questions? Yes?
So the question is, if I used a customized version of Envoy, or is it straight from Istio? The answer is it’s straight from Istio. So when we name a particular port as gRPC web dash something, it’s going to automatically enabled the gRPC web filter. So you don’t need a custom Envoy for that.
oh, okay. So for the previous question, I think you can do partial streaming with web sockets along with gRPC or gRPC web. Any other questions? All right, thank you.