Google Protocol Buffer Jetty Web Server Integration

Google Protocol Buffer Jetty Web Server Integration

You must have seen and worked with many web servers where you would have pass data from web page to back end servers. As this is common for any web application. Think if you get requirement to communicate machine to machine, immediately you will start googling the solution and unfortunately you won’t find any complete example as me when I was working on this requirement for one of my recent project.

My requirement was:

• Write application in Java which will exposed as web service without using any sever
• It should work same as any web service works
• Data serialization over the wire should backward compatible
• Performance should be faster as compare to other technology as of date
• You should do very less boiler plate coding
• Easy to understand

After researching many option finally I decided to use Jetty Web server with Google Protocol Buffers.

What is Jetty Web server: Jetty is Java HTTP Web server with Java Servlet container. You might have seen web Servers are usually associated with only serving documents, Jetty is often used for machine to machine communications, within big software frameworks. Jetty is free and open source project as part of Eclipse Foundation. The web server is used in products such as Apache ActiveMQ, Alfresco, Apache Geronimo, Apache Maven, Apache Spark, Google App Engine, Eclipse, FUSE, Twitter Streaming API etc.. Jetty is also server in open source projects such as Lift, Eucalyptus, Red5, Hadoop. Jetty supports latest Java Servlet API which include JSP support as well as protocols HTTP/2 and WebSocket.

What is Google Protocol Buffers: Protocol Buffers is a technique of serializing structured data into byte streaem. It is very useful in developing programs which communicate each other over on a wire or for storing the data. Method involves an interface description language that describes structure of data and Program which generates source code from that description for generating or parsing a stream of bytes that represents structured data.

Google designed and developed Protocol Buffers for use internally and has provided a code generator for multiple languages. Actually the design goals for Protocol Buffers emphasized simplicity and performance and it was designed to be smaller and faster than XML. It is very similar to Apache Thrift protocol (Facebook has used it) except that the public Protocol Buffers implementation does not include concrete RPC protocol stack to use for the defined services. It also provide backward compatibility.

Now lets jump on actual implementation:

Tools needed:

• Eclipse latest version
• JDK 1.8
• Google Protocol Buffers exe which is already include in the project for download
• Maven 3.2.5
• Jetty dependent jars (already included in pom.xml)

Steps:

• First create Maven project name: JettyGoogleProtoBuffIntegration below It shows complete project structure for reference. I have put completed project for download in the bottom:

• pom.xml where all project dependencies has been included:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.javahonk.jettygoogleprotobuffintegration</groupId>
	<version>1.0-SNAPSHOT</version>		
	<artifactId>JettyGoogleProtoBuffIntegration</artifactId>
	
	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
		
		<protobuf.version>2.6.1</protobuf.version>
		<protoc-jar.version>2.6.1.3</protoc-jar.version>
		<org.apache.log4j.version>2.1</org.apache.log4j.version>
		<org.springframework.version>4.1.5.RELEASE</org.springframework.version>
		<commons.logging.version>1.2</commons.logging.version>
		<apache.commons.lang.version>3.3.2</apache.commons.lang.version>
		<commons.collections.version>3.2.1</commons.collections.version>
		<junit.version>4.12</junit.version>
		
	</properties>

	<dependencies>

		<!-- Apache HTTP client --> 
		<dependency>
			<groupId>org.apache.httpcomponents</groupId>
			<artifactId>httpclient</artifactId>
			<version>4.3.6</version>
		</dependency>

		<!-- google proto buff -->
		<dependency>
			<groupId>com.google.protobuf</groupId>
			<artifactId>protobuf-java</artifactId>
			<version>2.6.1</version>
		</dependency>

		<!-- Jetty/Jersey dependency -->
		<dependency>
			<groupId>com.sun.jersey</groupId>
			<artifactId>jersey-server</artifactId>
			<version>1.9</version>
		</dependency>


		<dependency>
			<groupId>com.sun.jersey</groupId>
			<artifactId>jersey-json</artifactId>
			<version>1.9</version>
		</dependency>

		<dependency>
			<groupId>org.codehaus.jackson</groupId>
			<artifactId>jackson-core-asl</artifactId>
			<version>1.8.8</version>
		</dependency>


		<dependency>
			<groupId>org.codehaus.jackson</groupId>
			<artifactId>jackson-mapper-asl</artifactId>
			<version>1.8.8</version>
		</dependency>

		<dependency>
			<groupId>org.eclipse.jetty</groupId>
			<artifactId>jetty-server</artifactId>
			<version>9.2.9.v20150224</version>
		</dependency>

		<dependency>
			<groupId>org.eclipse.jetty</groupId>
			<artifactId>jetty-servlet</artifactId>
			<version>9.2.9.v20150224</version>
		</dependency>

		<dependency>
			<groupId>commons-lang</groupId>
			<artifactId>commons-lang</artifactId>
			<version>2.5</version>
		</dependency>
		
		<!-- JUNIT -->
		<dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
        </dependency>

		<!-- Spring -->
		<!-- Core utilities used by other modules. Define this if you use Spring 
			Utility APIs (org.springframework.core.*/org.springframework.util.*) -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-core</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Expression Language (depends on spring-core) Define this if you use 
			Spring Expression APIs (org.springframework.expression.*) -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-expression</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Bean Factory and JavaBeans utilities (depends on spring-core) Define 
			this if you use Spring Bean APIs (org.springframework.beans.*) -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-beans</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Aspect Oriented Programming (AOP) Framework (depends on spring-core, 
			spring-beans) Define this if you use Spring AOP APIs (org.springframework.aop.*) -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-aop</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Application Context (depends on spring-core, spring-expression, spring-aop, 
			spring-beans) This is the central artifact for Spring's Dependency Injection 
			Container and is generally always defined -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-context</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Various Application Context utilities, including EhCache, JavaMail, 
			Quartz, and Freemarker integration Define this if you need any of these integrations -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-context-support</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- JDBC Data Access Library (depends on spring-core, spring-beans, spring-context, 
			spring-tx) Define this if you use Spring's JdbcTemplate API (org.springframework.jdbc.*) -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-jdbc</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Object-to-Relation-Mapping (ORM) integration with Hibernate, JPA, 
			and iBatis. (depends on spring-core, spring-beans, spring-context, spring-tx) 
			Define this if you need ORM (org.springframework.orm.*) -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-orm</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Web application development utilities applicable to both Servlet and 
			Portlet Environments (depends on spring-core, spring-beans, spring-context) 
			Define this if you use Spring MVC, or wish to use Struts, JSF, or another 
			web framework with Spring (org.springframework.web.*) -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-web</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Support for testing Spring applications with tools such as JUnit and 
			TestNG This artifact is generally always defined with a 'test' scope for 
			the integration testing framework and unit testing stubs -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-test</artifactId>
			<version>${org.springframework.version}</version>
			<scope>test</scope>
		</dependency>
		
		<!-- Spring Aspects -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-aspects</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>
		
		<!-- Spring Tx -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-tx</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>

		<!-- Spring JMS -->
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-jms</artifactId>
			<version>${org.springframework.version}</version>
		</dependency>
		
		<!-- Log4j -->
        <dependency>
        	<groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>${org.apache.log4j.version}</version>
		</dependency>
        
        <dependency>
        	<groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${org.apache.log4j.version}</version>
		</dependency>
		
		<!-- commons-io -->		
		<dependency>	
			<groupId>commons-io</groupId>
  		  	<artifactId>commons-io</artifactId>
  		  	<version>2.4</version>
		</dependency>

	</dependencies>
	
</project>

• First create Google Protocol Buffers template and generate java classes. As you see above structure its inside script folder:
• javaHonkTest.proto: This is protocol buffers template class with two string parameter:

option java_package = "com.javahonk.proto";
option java_outer_classname = "JavaHonkTestInfoProto";

message JavaHonkTestInfo {
  required string key = 1;
  required string executedBy = 2;
}

message CompositeJavaHonkTestInfo {
  repeated JavaHonkTestInfo javaHonkTestInfo = 1;
}

• I have included other sample class (addressbook.proto, javaHonkDateRangeRequest.proto) as well to better understand proto template but we will use in our example only JavaHonkTestInfoProto.
• Now to generate java class from it use below command mentioned in readme.txt and you will have to execute this command from ..JettyApplication\scripts\protobuff because in this folder you have protoc.exe file which generate Java class:

 #Generating java files for Google protobuff
 
 protoc.exe  -I=./ --java_out=../../src/main/java ./message/addressbook.proto
 
 protoc.exe  -I=./ --java_out=../../src/main/java ./message/javaHonkDateRangeRequest.proto
 
 protoc.exe  -I=./ --java_out=../../src/main/java ./message/javaHonkTest.proto

• Once you execute above command then refresh your project you will below folder structure generate with below classes:

• Create server-context.xml for spring to load:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:jdbc="http://www.springframework.org/schema/jdbc"
       xmlns:tx="http://www.springframework.org/schema/tx"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.1.xsd
		http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc-4.1.xsd
		http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-4.1.xsd
		http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-4.1.xsd">

    <context:annotation-config />
    <context:component-scan base-package="com.javahonk" />
    
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
		<property name="locations">
			<list>
				<value>javahonk.properties</value>			
			</list>
		</property>
		<property name="ignoreUnresolvablePlaceholders" value="true"/>		
	</bean>

    <bean  class="com.javahonk.service.MessageHandler" />
    <bean class="com.javahonk.service.JettyService" />
    <bean id="jettyServer" class="org.eclipse.jetty.server.Server"/>
    
    <bean id="javaHonkProcessor" class="com.javahonk.adapter.JavaHonkProcessor"/>
    
    <bean id= "urlToAdaptorMap" class="java.util.HashMap">
        <constructor-arg>
            <map key-type="java.lang.String" value-type="com.javahonk.adaptor.ServerMessageProcessor">
                <entry key="${javahonkTest}" value-ref="javaHonkProcessor" />                
            </map>
        </constructor-arg>
    </bean>
    
</beans>

• javahonk.properties file to keep all application specific properties:

#jetty server properties
web.port=8080

#URLs
javahonkTest=/app/javahonk/test

• Create class JettyService.java to load and start server on port 8080 which is mentioned in properties file and you can change it to any port:

package com.javahonk.service;

import javax.annotation.PostConstruct;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.eclipse.jetty.server.Connector;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;

public class JettyService {

	@Value("${web.port}")
	private int port;
	
	@Autowired
	private MessageHandler handler; 

	@Autowired
	Server server;

	private static final Logger logger = LogManager.getLogger(MessageHandler.class.getName());
	
	public void setPort(int port) {
		this.port = port;
	}
	
	public void setMessageHandler(MessageHandler handler) {
		this.handler = handler;
	}
	
	@PostConstruct
    public void init() throws Exception {

		logger.info("starting jetty server at port {}", port);
		ServerConnector connector = new ServerConnector(server);
        connector.setPort(port);
        server.setConnectors(new Connector[] { connector });
        server.setHandler(handler);
        if (server.isRunning()) {
        	logger.error("Jetty server already running.");
        }
        else
        {
        	server.start();
        }
    }
	
}

• Define MessageHandler.java class which will process request from client send back response:

package com.javahonk.service;

import java.io.IOException;
import java.util.Map;

import javax.annotation.Resource;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.HttpMethod;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.handler.AbstractHandler;

import com.javahonk.adapter.ServerMessageProcessor;

public class MessageHandler extends AbstractHandler {

    private static final Logger logger = LogManager.getLogger(MessageHandler.class.getName());

    @Resource(name="urlToAdaptorMap")
    Map<String, ServerMessageProcessor> urlToAdaptorMap;


    public void handle(String target, Request baseRequest,
                       HttpServletRequest request, HttpServletResponse response)
            throws IOException, ServletException {

        logger.info("HTTP remote box- " + request.getRemoteAddr());
        logger.info("HTTP request- " + request.getPathInfo());

        String path = request.getPathInfo();

        if (request.getMethod().equals(HttpMethod.POST)) {
            try {
                if (urlToAdaptorMap.containsKey(path)) {
                    urlToAdaptorMap.get(path).handleRequest(request, response);
                    response.setStatus(HttpServletResponse.SC_OK);
                }
                baseRequest.setHandled(true);
            } catch (Exception ex) {
                logger.error("Error :", ex);
                response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
            }
        } else {
            logger.warn("Received request on unsupported url {}", path);
            response.setStatus(HttpServletResponse.SC_SERVICE_UNAVAILABLE);
        }
    }
}

• Create interface ServerMessageProcessor.java and define one method so that any class who implements this will include it to process the request. If you want to use it then just create seperate classed in use it:

package com.javahonk.adapter;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

public interface ServerMessageProcessor {

    public void handleRequest(HttpServletRequest request, HttpServletResponse response) throws IOException;

}

• JavaHonkProcessor.java is main class which process the request and send back response:

package com.javahonk.adapter;

import java.io.IOException;
import java.util.List;

import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.io.IOUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

import com.javahonk.proto.JavaHonkDateRangeRequestProto;
import com.javahonk.proto.JavaHonkDateRangeRequestProto.CompositeJavaHonkDateRangeRequestInfo;
import com.javahonk.proto.JavaHonkDateRangeRequestProto.JavaHonkDateRangeRequestInfo;

public class JavaHonkProcessor implements ServerMessageProcessor {

    private static final Logger logger = LogManager.getLogger(JavaHonkProcessor.class.getName());

    public void handleRequest(HttpServletRequest request, HttpServletResponse response) throws IOException {

        byte[] message = IOUtils.toByteArray(request.getInputStream());

        List<JavaHonkDateRangeRequestInfo> compositeSwapResetInfoList = CompositeJavaHonkDateRangeRequestInfo
				.parseFrom(message).getJavaHonkDateRangeRequestInfoList();

		for (JavaHonkDateRangeRequestInfo javaHonkDateRangeRequestInfo : compositeSwapResetInfoList) {
			System.out.println(javaHonkDateRangeRequestInfo.getEndDate());
		}

        logger.info("Successfully parsed message {}", compositeSwapResetInfoList.toString());

        ServletOutputStream outputStream = response.getOutputStream();
        
        JavaHonkDateRangeRequestProto.CompositeJavaHonkDateRangeRequestInfo.Builder compositeJavaHonkDateRangeRequestInfo 
		= JavaHonkDateRangeRequestProto.CompositeJavaHonkDateRangeRequestInfo.newBuilder();
	
		JavaHonkDateRangeRequestInfo.Builder javaHonkDateRangeRequestInfo = JavaHonkDateRangeRequestInfo.newBuilder();
		javaHonkDateRangeRequestInfo.setStartDate("20160628");
		javaHonkDateRangeRequestInfo.setEndDate("20160630");
				
		compositeJavaHonkDateRangeRequestInfo.addJavaHonkDateRangeRequestInfo(javaHonkDateRangeRequestInfo);
		compositeJavaHonkDateRangeRequestInfo.addJavaHonkDateRangeRequestInfo(javaHonkDateRangeRequestInfo);
	
		outputStream.write(compositeJavaHonkDateRangeRequestInfo.build().toByteArray());
    }    

}

• HTTPPostSendTestData.java: This is test client class which will post the data in byte array where data is created using Google Protocol Buffers and it will also process the response from server and prints out the output:

package com.javahonk.test;

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.List;

import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import org.apache.commons.io.IOUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ByteArrayEntity;
import org.apache.http.impl.client.HttpClientBuilder;

import com.javahonk.proto.JavaHonkDateRangeRequestProto;
import com.javahonk.proto.JavaHonkDateRangeRequestProto.CompositeJavaHonkDateRangeRequestInfo;
import com.javahonk.proto.JavaHonkDateRangeRequestProto.JavaHonkDateRangeRequestInfo;

public class HTTPPostSendTestData {

	public static void main(String[] args) {
		postHttpAsString("http://localhost:8080/app/javahonk/test", buildData());		
	}

	private static void postHttpAsString(String url, byte[] message) {

		try {
			HttpClient httpClient = HttpClientBuilder.create().build();
			HttpPost postRequest = new HttpPost(url);

			ByteArrayEntity entity = new ByteArrayEntity(message);
			entity.setContentType(MediaType.APPLICATION_OCTET_STREAM);
			postRequest.setEntity(entity);

			HttpResponse response = httpClient.execute(postRequest);

			if (response.getStatusLine().getStatusCode() != Response.Status.OK.getStatusCode()) {
				throw new RuntimeException("Failed : HTTP error code : " + response.getStatusLine().getStatusCode());
			}
			
			byte[] value = IOUtils.toByteArray(response.getEntity().getContent());

	        List<JavaHonkDateRangeRequestInfo> javahonkTestInfo = CompositeJavaHonkDateRangeRequestInfo
					.parseFrom(value).getJavaHonkDateRangeRequestInfoList();

			for (JavaHonkDateRangeRequestInfo javaHonkDateRangeRequestInfo : javahonkTestInfo) {
				System.out.println(javaHonkDateRangeRequestInfo);				
			}

		} catch (MalformedURLException e) {

			e.printStackTrace();

		} catch (IOException e) {

			e.printStackTrace();

		}
	}
	
	private static byte[] buildData(){
		
		JavaHonkDateRangeRequestProto.CompositeJavaHonkDateRangeRequestInfo.Builder compositeJavaHonkDateRangeRequestInfo 
		= JavaHonkDateRangeRequestProto.CompositeJavaHonkDateRangeRequestInfo.newBuilder();
	
		JavaHonkDateRangeRequestInfo.Builder javaHonkDateRangeRequestInfo = JavaHonkDateRangeRequestInfo.newBuilder();
		javaHonkDateRangeRequestInfo.setStartDate("20160628");
		javaHonkDateRangeRequestInfo.setEndDate("20160630");
				
		compositeJavaHonkDateRangeRequestInfo.addJavaHonkDateRangeRequestInfo(javaHonkDateRangeRequestInfo);
		compositeJavaHonkDateRangeRequestInfo.addJavaHonkDateRangeRequestInfo(javaHonkDateRangeRequestInfo);
	
		return compositeJavaHonkDateRangeRequestInfo.build().toByteArray();
	}

}

• Finally create Java main JavaHonkMainApp.java class to load and start the application:

package com.javahonk;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.AbstractApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class JavaHonkMainApp {

    private static final Logger logger = LogManager.getLogger(JavaHonkMainApp.class.getName());

    private static String[] configLocation = {"server-context.xml"};

    public static void main(String[] args) {

        logger.info("Starting JavaHonkMainApp...");

        setUncaughtException();

        ApplicationContext context = getApplicationContext();

        registerShutdownHook(context);
    }

    private static void setUncaughtException() {
        Thread.currentThread().setUncaughtExceptionHandler((t, e) -> logger.error("Thread {} threw uncaught exception ", t.getName(), e));
    }

    protected static ApplicationContext getApplicationContext() {
        return new ClassPathXmlApplicationContext(configLocation);
    }

    private static void registerShutdownHook(ApplicationContext context) {
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            logger.info("JavaHonkMainApp Exiting.");
            ((AbstractApplicationContext) context).close();
        }));
    }
}

• If everything setup correctly you will below on your console without any exception:

• Now right click HTTPPostSendTestData.java which will post data to Jetty and process it response you will below responses:

• That’s it. For more information you could visit this Google Protocol Buffer and Jetty Web Server.

Download code: JettyGoogleProtoBuffIntegration