Connecting to Cloud Foundry Kafka Service from your localhost using Kafkacat

So we are developing a cloud foundry application  and one of the backing services we use is a Kafka Service. Our backing service uses SASL for authentication and uses a self signed certificate issued by the owner of the backing service.

Now we all know that while deploying and debugging  the application on localhost while connecting with Cloudfoundry backing is sometimes a necessary evil and depending on your network setup it could sometimes be tricky.

I managed to make Kafkacat (https://github.com/edenhill/kafkacat) connect to the backing service from my local macbook. Here is how you can achieve it.

Here is how my service looked like in the VCAP_SERVICES variable.


Now there are a couple of challenges for me to work with service using Kafkacat.

• The username and password given in the VCAP_SERVICES variable don't work directly with Kafkacat. We have to generate a Token using the token services URL given in the json and then provide that authentication token as password to the utility. 
• The broker IPs and the zookeeper IPs (10.254 series of IPs in the json) are unreachable from my local machine.
• The SSL communication between kafkacat and kafka service is encrypted by a certificate which is issued by the rootCA which is available on  https://kafka-service-broker.example.com/certs/rootCA.crt

Let's solve these issues one by one. 

1. In order to create the token to be provided to the kafka service as a password you can use the following script. The envrionment variables USERNAME and PASSWORD are set to the values which you found in the VCAP_SERVICES. The output of this command can be set in another environment variable TOKEN which we will use in the final steps.    
2. You should store all the BROKERS in an environment variable called BROKERS. 
3. In order to make the kafkacat work, we need to make sure that we are able to reach the 3 broker IPs given in the environment variables from the localhost. We will do this with the help of virtual interfaces and ssh tunnels on the macbook. I didn't manage to find the right way to do it using bash so I am listing down the steps you need to take from UI (with screenshots). 
• Open the network option in your System Preferences and from the bottom wheel select "Manage Virtual Interfaces" .  
  
• In the next dialog select Add VLAN option to add a VLAN. 
  
• Give the VLAN the name vlan0 and select an interface. (You need to select the interface you are connected to else it doesn't work). 
  
• Press create->done and ensure that VLAN appears in the network dialog, as shown below. 
  
• Now go ahead and change the Configure IPv4 from DHCP to manually and assign one of the IP address which we had in the VCAP_SERVICES to this vlan interface. 
  
• After this go ahead and duplicate vlan0 2 times. The option is available as shown below. 
  
• Make sure that you enter the other 2 IPs in the duplicated interfaces. After this step my dialog looks like the following. Please note that IP is different for all 3 interfaces. For me vlan0 => 10.254.33.21, vlan1 => 10.254.33.22, vlan2 => 10.254.33.23 now. 
  
4. After the above step, the macbook now starts responding to all the 3 broker IPs. Now we need to ensure that any request on these interfaces is tunneled to the real Kafka brokers hosted in Cloud Foundry. To do that I establish 3 SSH tunnels with the commands below. 
5. In order to provide the rootCA.crt to kafkacat, I download the certificate to the local machine in a folder from where I intend to use kafkacat. I use curl  to do this. 
6. You can install kafkacat on mac using brew
7. Now you can start using kafkacat with its normal commands, however since it's a SASL setup, you need to provide some parameters to make it work. Here is how I use kafkacat to list all topics.  You can use the same format to execute any other kafkacat commands you are interested in. You can also use your java application running your localhost to now start connecting to the remote kafka service on cloudfoundry. 

Happy coding

!Abhishek