Publishing Application Events using Kafka and Spring Transaction Events

Many applications have a need to publish application level events based on some operations happening in the system.  Here is one requirement that I had recently. Basically in my tutorial system, if there is new user created, I need to publish a UserCreated event. Here is how we can accomplish this using Transaction Events provided by Springframework.

First I create a event pojo that will be used to publish the event.

	package in.springframework.blog.tutorials.events;
	import lombok.Builder;
	import lombok.Data;
	@Builder
	@Data
	public class UserCreatedEvent {
	private long userId;
	private String username;
	private String fullname;
	private String status;
	}

view raw UserCreatedEvent.java hosted with ❤ by GitHub

Now we create an event listener.

	package in.springframework.blog.tutorials.listeners;
	import in.springframework.blog.tutorials.events.UserCreatedEvent;
	import in.springframework.blog.tutorials.utils.MyConstants;
	import lombok.extern.log4j.Log4j2;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.beans.factory.annotation.Qualifier;
	import org.springframework.kafka.core.KafkaTemplate;
	import org.springframework.stereotype.Component;
	import org.springframework.transaction.event.TransactionPhase;
	import org.springframework.transaction.event.TransactionalEventListener;
	import org.springframework.util.concurrent.ListenableFutureCallback;
	@Component
	@Log4j2
	public class UserCreatedEventListener {
	@Autowired
	@Qualifier(MyConstants.FIRST_TOPIC_TEMPLATE_NAME)
	private KafkaTemplate firstKafkaTemplate;
	@TransactionalEventListener(phase= TransactionPhase.BEFORE_COMMIT)
	void handleBeforeCommit(UserCreatedEvent userCreatedEvent) {
	log.error("Before Commit " + userCreatedEvent);
	sendMessage(userCreatedEvent, "BeforeCommit");
	}
	@TransactionalEventListener(phase= TransactionPhase.AFTER_COMMIT)
	void handleAfterCommit(UserCreatedEvent userCreatedEvent) {
	log.error("After Commit " + userCreatedEvent);
	sendMessage(userCreatedEvent, "AfterCommit");
	}
	@TransactionalEventListener(phase= TransactionPhase.AFTER_ROLLBACK)
	void handleAfterRollback(UserCreatedEvent userCreatedEvent) {
	log.error("After Rollback " + userCreatedEvent);
	sendMessage(userCreatedEvent, "AfterRollback");
	}
	@TransactionalEventListener(phase= TransactionPhase.AFTER_COMPLETION)
	void handleAfterCompletion(UserCreatedEvent userCreatedEvent) {
	log.error("After Completion " + userCreatedEvent);
	sendMessage(userCreatedEvent, "AfterCompletion");
	}
	private void sendMessage(UserCreatedEvent userCreatedEvent, String status) {
	userCreatedEvent.setStatus(status);
	firstKafkaTemplate.send(MyConstants.FIRST_TOPIC, userCreatedEvent.toString()).addCallback(new ListenableFutureCallback() {
	@Override
	public void onFailure(Throwable throwable) {
	log.info(String.format("Message '%s' failed!", userCreatedEvent.toString()));
	}
	@Override
	public void onSuccess(Object o) {
	log.info(String.format("Message '%s' sent successfully!", userCreatedEvent.toString()));
	}
	});
	}
	}

view raw UserCreatedEventListener.java hosted with ❤ by GitHub

As you can see within the event listener, I have a handler method that is a hook to a particular phase in transaction commit cycle. At different phases of commit cycle, we are publishing an event to Kafka topic 'FirstTopic'

Now we create an event publisher.

	package in.springframework.blog.tutorials.publishers;
	import in.springframework.blog.tutorials.entities.User;
	import in.springframework.blog.tutorials.events.UserCreatedEvent;
	import lombok.extern.log4j.Log4j2;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.context.ApplicationEventPublisher;
	import org.springframework.stereotype.Component;
	@Component
	@Log4j2
	public class UserEventPublisher {
	@Autowired
	private ApplicationEventPublisher applicationEventPublisher;
	public void publishUserCreatedEvent(final User user) {
	log.info(String.format("Publishing event for user %s with id %d", user.getUsername(), user.getId()));
	applicationEventPublisher.publishEvent(UserCreatedEvent.builder()
	.userId(user.getId())
	.username(user.getUsername())
	.fullname(user.getFullname())
	.build()
	);
	}
	}

view raw UserEventPublisher.java hosted with ❤ by GitHub

The event publisher just constructs the event object and publishes the event.

We add a createUser method within the user service and mark it @Transactional. This is important to make sure any exceptions coming out of createUser transaction are caught and propagated appropriately.

	package in.springframework.blog.tutorials.services;
	import in.springframework.blog.tutorials.entities.Tenant;
	import in.springframework.blog.tutorials.entities.User;
	import in.springframework.blog.tutorials.mappers.UserPojoMapper;
	import in.springframework.blog.tutorials.pojos.Role;
	import in.springframework.blog.tutorials.pojos.UserPojo;
	import in.springframework.blog.tutorials.publishers.UserEventPublisher;
	import in.springframework.blog.tutorials.repositories.TenantRepository;
	import in.springframework.blog.tutorials.repositories.UserRepository;
	import in.springframework.blog.tutorials.utils.TutorialRequestContext;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.security.crypto.password.PasswordEncoder;
	import org.springframework.stereotype.Service;
	import org.springframework.transaction.annotation.Transactional;
	import java.util.Optional;
	@Service
	public class UserService {
	@Autowired
	private UserRepository userRepository;
	@Autowired
	private TenantRepository tenantRepository;
	@Autowired
	private PasswordEncoder passwordEncoder;
	@Autowired
	private UserPojoMapper userToPojoMapper;
	@Autowired
	private UserEventPublisher userEventPublisher;
	public Iterable<User> findAll() {
	return userRepository.findAll();
	}
	public Optional<User> findByUsername(String username) {
	return userRepository.findUserByTenantAndUsername(TutorialRequestContext.currentTenant.get(), username);
	}
	public Optional<User> retrieveUser(String idOrUserNameOrEmail) {
	Optional<Tenant> optionalTenant = tenantRepository.findById(TutorialRequestContext.currentTenant.get().getId());
	try {
	Long id = Long.parseLong(idOrUserNameOrEmail);
	Optional<User> optionalUser = userRepository.findById(id);
	if (optionalUser.isPresent()) {
	return optionalUser;
	}
	}
	catch(NumberFormatException e) {
	}
	Optional<User> optionalUser = userRepository.findUserByEmail(idOrUserNameOrEmail);
	if (optionalUser.isPresent()) {
	return optionalUser;
	}
	optionalUser = userRepository.findUserByUsernameAndTenant(optionalTenant.get(),
	idOrUserNameOrEmail);
	return optionalUser;
	}
	public User save(User user) {
	return userRepository.save(user);
	}
	public void delete(Long id) {
	userRepository.deleteById(id);
	}
	@Transactional
	public Optional<UserPojo> createUser(UserPojo userPojo) {
	User user = userToPojoMapper.convert(userPojo);
	user.setMask(Role.USER.getMask());
	user.setTenant(TutorialRequestContext.currentTenant.get());
	user.setPassword(passwordEncoder.encode(userPojo.getPassword()));
	User storedUser = save(user);
	storedUser.setPassword(null);
	userEventPublisher.publishUserCreatedEvent(storedUser);
	return Optional.of(userToPojoMapper.convert(storedUser));
	}
	}

view raw UserService.java hosted with ❤ by GitHub

 

Now within the user registration, we add a publish within a transaction. Please be careful that the code block that is publishing the event is within a transaction otherwise these don't perform any operations. 

	package in.springframework.blog.tutorials.endpoints;
	import in.springframework.blog.tutorials.entities.TutorialClientDetails;
	import in.springframework.blog.tutorials.exceptions.EntityAlreadyExistsException;
	import in.springframework.blog.tutorials.pojos.UserPojo;
	import in.springframework.blog.tutorials.services.ClientService;
	import in.springframework.blog.tutorials.services.UserService;
	import in.springframework.blog.tutorials.utils.MyConstants;
	import lombok.extern.log4j.Log4j2;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.dao.DataIntegrityViolationException;
	import org.springframework.http.MediaType;
	import org.springframework.security.access.prepost.PreAuthorize;
	import org.springframework.web.bind.annotation.RequestBody;
	import org.springframework.web.bind.annotation.RequestMapping;
	import org.springframework.web.bind.annotation.RequestMethod;
	import org.springframework.web.bind.annotation.RestController;
	import javax.websocket.server.PathParam;
	import java.util.Optional;
	@Log4j2
	@RestController
	@RequestMapping(MyConstants.REGISTRATION_ENDPOINT)
	public class RegistrationEndpoint {
	@Autowired
	private UserService userService;
	@Autowired
	private ClientService clientService;
	@RequestMapping(value = "/user", method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
	public Optional<UserPojo> createUser(@PathParam(value="tenant") String tenant,
	@RequestBody UserPojo userPojo) {
	try {
	return userService.createUser(userPojo);
	}
	catch(DataIntegrityViolationException divx) {
	throw new EntityAlreadyExistsException(String.format("User %s or email %s already exists", userPojo.getUsername(), userPojo.getEmail()));
	}
	}
	@RequestMapping(value = "/client", method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
	@PreAuthorize("hasAnyAuthority('ADMIN', 'TENANT_ADMIN')")
	public Optional<TutorialClientDetails> getClient(@RequestBody TutorialClientDetails clientDetails) {
	return clientService.create(clientDetails);
	}
	}

view raw RegistrationEndpoint.java hosted with ❤ by GitHub

Look at the createUser method that is implementing /user request mapping and method POST. The penultimate line in the method is publishing the event. Now let's test this code.

Just to test the code, we also run a kafka-console-consumer listening to the topic to which our code is publishing the message.

First we try to create a user that is already existing.

$ curl --request POST   --url http://localhost:8081/tenant1/registration/user   \
 --header 'authorization: Basic c3VwZXJzZWNyZXRjbGllbnQ6c3VwZXJzZWNyZXRjbGllbnQxMjM='   \
 --header 'cache-control: no-cache'   \
 --header 'content-type: application/json'   \
 --header 'postman-token: 71180383-2b1f-482e-2fcc-2a23d045b205'   \
 --header 'x-tenant: tenant1'   \
 --data '{"username": "admin8","password": "admin1234","audience" : "self"}' | python -m json.tool
 {
    "error": "Conflict",
    "message": "",
    "path": "/tenant1/registration/user",
    "status": 409,
    "timestamp": 1596015557641
}

Now we try to create a user that doesn't exist.

$ curl --request POST   --url http://localhost:8081/tenant1/registration/user \
--header 'authorization: Basic c3VwZXJzZWNyZXRjbGllbnQ6c3VwZXJzZWNyZXRjbGllbnQxMjM='\
--header 'cache-control: no-cache' \
--header 'content-type: application/json'\
--header 'postman-token: 71180383-2b1f-482e-2fcc-2a23d045b205' \
--header 'x-tenant: tenant1' \
--data '{"username": "admin9","password": "admin1234","audience" : "self"}' | python -m json.tool
{
    "createdAt": 1596015670301,
    "createdBy": "UnAuthenticated",
    "id": 21,
    "mask": 1,
    "tenantId": 1,
    "updatedAt": 1596015670301,
    "updatedBy": "UnAuthenticated",
    "username": "admin9"
}

We can also check our Kafka consumer that we ran earlier.

$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic FirstTopic
UserCreatedEvent(userId=20, username=admin8, fullname=null, status=BeforeCommit)
UserCreatedEvent(userId=20, username=admin8, fullname=null, status=AfterRollback)
UserCreatedEvent(userId=20, username=admin8, fullname=null, status=AfterCompletion)
UserCreatedEvent(userId=21, username=admin9, fullname=null, status=BeforeCommit)
UserCreatedEvent(userId=21, username=admin9, fullname=null, status=AfterCommit)
UserCreatedEvent(userId=21, username=admin9, fullname=null, status=AfterCompletion)

The first three events are for the failed transaction which rolled back and the last three are for the successful transaction.

Building a multitenant service

If you build any production service that is valuable, very soon you find a customer who wants to use it but wants it white-labeled for him. You also very quickly find out that there is a need to data isolation, i.e. different customers want their data to be kept in their own databases. Because of these reasons, it is always a good idea to design a service keeping multi-tenancy in mind.

In this tutorial we will look at some of the important features required for a multi-tenant service and how do we leverage springframework to deliver a service that is truly enabled for a modern multi-tenant service.

Here are couple of the important aspects of a multi-tenant service.

1. Everybody gets their own endpoints
2. Everybody can be given their own databases

Let's design how our endpoint URLs would look like keeping tenancy in mind. We add a discriminator in the URL that identified a tenant. For example our OAuth URLs would become something like below.

http://example.com/tenant1/oauth/token
http://example.com/tenant1/oauth/check_token

To accomplish, we define our RequestMapping with an embedded tenant variable that becomes part of each of the URLs.

 public static final String BASE_ENDPOINT = "/{tenant}";
 public static final String USER_ENDPOINT = BASE_ENDPOINT + "/user";
 public static final String REGISTRATION_ENDPOINT = BASE_ENDPOINT + "/registration";

As we can see, the first part of the URL defines the tenant. It is not mandatory but I also design the service so that a header is expected that also defines the tenant. This is just to guard against some client mistakenly calling a wrong tenant because the same tenant has to be added in two places. We will use following header.

X-tenant:tenant1

Every incoming request into the system needs to know the tenant and the authenticated user that is part of the request. We define a sequence of filters for this purpose. Since these filters need to be called in a particular order, we define the order as follow.

public static final int TENANT_HEADER_PRECEDENCE = Ordered.HIGHEST_PRECEDENCE;
public static final int SEED_DATA_PRECEDENCE = TENANT_HEADER_PRECEDENCE - 1;
public static final int TENANT_PRECEDENCE = SEED_DATA_PRECEDENCE - 1;
public static final int USER_PRECEDENCE = Ordered.LOWEST_PRECEDENCE;

As you can see we are going to define four filters which will perform specific functions.

1. TenantHeader filter will extract tenant header from the incoming request, match the URL piece with the header and set it in a ThreadLocal variable.
2. SeedData filter is only require to create some seed data to make the service usable. We need a default tenant in the system so that we can start making some requests. This doesn't do anything most of the time.
3. Tenant filter will extract the tenant object and set it into another ThreadLocal variable.
4. User filter is the last in the precedence and will extract the current authenticated user and will store it into a ThreadLocal.

We are using following TutorialRequestContext class to store these thread local variables.

	package in.springframework.blog.tutorials.utils;
	import in.springframework.blog.tutorials.entities.Tenant;
	import in.springframework.blog.tutorials.entities.User;
	public class TutorialRequestContext {
	public static final ThreadLocal<Tenant> currentTenant = new ThreadLocal<>();
	public static final ThreadLocal<User> currentUser = new ThreadLocal<>();
	public static final ThreadLocal<String> currentTenantDiscriminator = new ThreadLocal<>();
	}

view raw TutorialRequestContext.java hosted with ❤ by GitHub

Now let's look at these filters one by one.

	package in.springframework.blog.tutorials.filters;
	import in.springframework.blog.tutorials.exceptions.InvalidTenantDiscriminatorException;
	import in.springframework.blog.tutorials.utils.MyConstants;
	import in.springframework.blog.tutorials.utils.TutorialRequestContext;
	import org.codehaus.jackson.map.ObjectMapper;
	import org.springframework.beans.factory.annotation.Value;
	import org.springframework.core.annotation.Order;
	import org.springframework.http.HttpStatus;
	import org.springframework.http.MediaType;
	import org.springframework.stereotype.Component;
	import org.springframework.web.context.WebApplicationContext;
	import org.springframework.web.context.support.WebApplicationContextUtils;
	import javax.servlet.*;
	import javax.servlet.http.HttpServletRequest;
	import javax.servlet.http.HttpServletResponse;
	import java.io.File;
	import java.io.IOException;
	import java.util.HashMap;
	import java.util.Map;
	@Component
	@Order(MyConstants.TENANT_HEADER_PRECEDENCE)
	public class TenantHeaderFilter implements Filter {
	public static final String TENANT_HEADER = "X-tenant";
	private static final String defaultClient = "supersecretclient";
	private static final String defaultSecret = "supersecretclient123";
	@Value("${tutorial.default.tenant.discriminator:default}")
	private String defaultDiscriminator;
	@Override
	public void doFilter(ServletRequest request,
	ServletResponse response,
	FilterChain chain) throws IOException, ServletException {
	HttpServletRequest httpServletRequest = (HttpServletRequest)request;
	HttpServletResponse httpServletResponse = (HttpServletResponse) response;
	String[] dirs = httpServletRequest.getRequestURI().split(File.separator);
	String tenantDiscriminatorInPath = dirs[1];
	WebApplicationContext webApplicationContext
	= WebApplicationContextUtils.getWebApplicationContext(request.getServletContext());
	String tenantDiscriminator = httpServletRequest.getHeader(TENANT_HEADER);
	if (tenantDiscriminatorInPath.equals(tenantDiscriminator)) {
	TutorialRequestContext.currentTenantDiscriminator.set(tenantDiscriminator);
	chain.doFilter(request, response);
	}
	else {
	httpServletResponse.setStatus(HttpStatus.FORBIDDEN.value());
	httpServletResponse.setContentType(MediaType.APPLICATION_JSON_VALUE);
	Map<String, Object> errorDetail = new HashMap<>();
	errorDetail.put("message", String.format("The header X-tenant should be same as the tenant discriminator in URL. Currently are %s and %s ", tenantDiscriminator, tenantDiscriminatorInPath));
	ObjectMapper mapper = new ObjectMapper();
	mapper.writeValue(httpServletResponse.getWriter(), errorDetail);
	throw new InvalidTenantDiscriminatorException(String.format("The header X-tenant should be same as the tenant discriminator in URL. Currently are %s and %s ", tenantDiscriminator, tenantDiscriminatorInPath));
	}
	}
	}

view raw TenantHeaderFilter.java hosted with ❤ by GitHub

This filter just extracts the X-tenant header stores in the thread local.

	package in.springframework.blog.tutorials.filters;
	import in.springframework.blog.tutorials.entities.Tenant;
	import in.springframework.blog.tutorials.services.ClientService;
	import in.springframework.blog.tutorials.services.TenantService;
	import in.springframework.blog.tutorials.utils.MyConstants;
	import org.springframework.core.annotation.Order;
	import org.springframework.stereotype.Component;
	import org.springframework.web.context.WebApplicationContext;
	import org.springframework.web.context.support.WebApplicationContextUtils;
	import javax.servlet.*;
	import javax.servlet.http.HttpServletRequest;
	import java.io.IOException;
	@Component
	@Order(MyConstants.SEED_DATA_PRECEDENCE)
	public class SeedDataInitializationFilter implements Filter {
	public static final String TENANT_HEADER = "X-tenant";
	private static final String defaultClient = "supersecretclient";
	private static final String defaultSecret = "supersecretclient123";
	@Override
	public void doFilter(ServletRequest request,
	ServletResponse response,
	FilterChain chain) throws IOException, ServletException {
	HttpServletRequest httpServletRequest = (HttpServletRequest)request;
	WebApplicationContext webApplicationContext
	= WebApplicationContextUtils.getWebApplicationContext(request.getServletContext());
	TenantService tenantService = webApplicationContext.getBean(TenantService.class);
	ClientService clientService = webApplicationContext.getBean(ClientService.class);
	if (tenantService.count() == 0) {
	Tenant tenant = new Tenant();
	tenant.setDescription("Default tenant for the system");
	tenant.setDiscriminator("default");
	tenant.setName("Default Tenant");
	tenant.setDefaultValue(true);
	}
	chain.doFilter(request, response);
	}
	}

view raw SeedDataInitializationFilter.java hosted with ❤ by GitHub

This filter checks that there should be atleast one tenant in the system, if not found, it inserts a default tenant. This is required so that we can use rest calls.

	package in.springframework.blog.tutorials.filters;
	import in.springframework.blog.tutorials.entities.Tenant;
	import in.springframework.blog.tutorials.exceptions.InvalidTenantDiscriminatorException;
	import in.springframework.blog.tutorials.services.TenantService;
	import in.springframework.blog.tutorials.utils.MyConstants;
	import in.springframework.blog.tutorials.utils.TutorialRequestContext;
	import org.codehaus.jackson.map.ObjectMapper;
	import org.springframework.beans.factory.annotation.Value;
	import org.springframework.core.annotation.Order;
	import org.springframework.http.HttpStatus;
	import org.springframework.http.MediaType;
	import org.springframework.stereotype.Component;
	import org.springframework.web.context.WebApplicationContext;
	import org.springframework.web.context.support.WebApplicationContextUtils;
	import javax.servlet.*;
	import javax.servlet.http.HttpServletRequest;
	import javax.servlet.http.HttpServletResponse;
	import java.io.IOException;
	import java.util.HashMap;
	import java.util.Map;
	import java.util.Optional;
	@Component
	@Order(MyConstants.TENANT_PRECEDENCE)
	public class TenantFilter implements Filter {
	public static final String TENANT_HEADER = "X-tenant";
	private static final String defaultClient = "supersecretclient";
	private static final String defaultSecret = "supersecretclient123";
	@Value("${tutorial.default.tenant.discriminator:default}")
	private String defaultDiscriminator;
	@Override
	public void doFilter(ServletRequest request,
	ServletResponse response,
	FilterChain chain) throws IOException, ServletException {
	HttpServletRequest httpServletRequest = (HttpServletRequest)request;
	HttpServletResponse httpServletResponse = (HttpServletResponse)response;
	WebApplicationContext webApplicationContext
	= WebApplicationContextUtils.getWebApplicationContext(request.getServletContext());
	TenantService tenantService = webApplicationContext.getBean(TenantService.class);
	String tenantDiscriminator = TutorialRequestContext.currentTenantDiscriminator.get();
	Optional<Tenant> optionalTenant = tenantService.retrieveTenantsByDiscriminator(tenantDiscriminator);
	if (!optionalTenant.isPresent()) {
	httpServletResponse.setStatus(HttpStatus.FORBIDDEN.value());
	httpServletResponse.setContentType(MediaType.APPLICATION_JSON_VALUE);
	Map<String, Object> errorDetail = new HashMap<>();
	errorDetail.put("message", String.format("Tenant %s doesn't exist", tenantDiscriminator));
	ObjectMapper mapper = new ObjectMapper();
	mapper.writeValue(httpServletResponse.getWriter(), errorDetail);
	throw new InvalidTenantDiscriminatorException(String.format("Tenant %s doesn't exist", tenantDiscriminator));
	}
	else {
	TutorialRequestContext.currentTenant.set(optionalTenant.get());
	}
	chain.doFilter(request, response);
	}
	}

view raw TenantFilter.java hosted with ❤ by GitHub

This filter extract the complete Tenant object from the database and stores it into the thread local.

	package in.springframework.blog.tutorials.filters;
	import in.springframework.blog.tutorials.entities.User;
	import in.springframework.blog.tutorials.services.UserService;
	import in.springframework.blog.tutorials.utils.MyConstants;
	import in.springframework.blog.tutorials.utils.TutorialRequestContext;
	import org.springframework.core.annotation.Order;
	import org.springframework.security.core.context.SecurityContextHolder;
	import org.springframework.stereotype.Component;
	import org.springframework.web.context.WebApplicationContext;
	import org.springframework.web.context.support.WebApplicationContextUtils;
	import javax.servlet.*;
	import java.io.IOException;
	import java.util.Optional;
	@Component
	@Order(MyConstants.USER_PRECEDENCE)
	public class CurrentUserExtractionFilter implements Filter {
	public static final String TENANT_HEADER = "X-tenant";
	@Override
	public void doFilter(ServletRequest servletRequest,
	ServletResponse servletResponse,
	FilterChain filterChain) throws IOException, ServletException {
	WebApplicationContext webApplicationContext
	= WebApplicationContextUtils.getWebApplicationContext(servletRequest.getServletContext());
	UserService userService = webApplicationContext.getBean(UserService.class);
	if (SecurityContextHolder.getContext().getAuthentication() != null) {
	Object principal = SecurityContextHolder.getContext().getAuthentication().getPrincipal();
	String username = null;
	if (principal instanceof org.springframework.security.core.userdetails.User) {
	username = ((org.springframework.security.core.userdetails.User)principal).getUsername();
	}
	else if (principal instanceof String) {
	username = (String)principal;
	}
	Optional<User> optionalUser = userService.findByUsername(username);
	if (optionalUser.isPresent()) {
	TutorialRequestContext.currentUser.set(optionalUser.get());
	}
	}
	filterChain.doFilter(servletRequest, servletResponse);
	}
	}

view raw CurrentUserExtractionFilter.java hosted with ❤ by GitHub

This filter extract currently authenticate user and populates it in the thread local.

In the last tutorial, we used the spring provided default ClientDetailsService, we create our own service in this tutorial to make sure we can have a schema that we like. To do that we need a entity, TutorialClientDetails and a service TutorialClientDetailsService.

	package in.springframework.blog.tutorials.entities;
	import lombok.Getter;
	import lombok.Setter;
	import org.springframework.data.jpa.domain.support.AuditingEntityListener;
	import org.springframework.security.core.GrantedAuthority;
	import org.springframework.security.oauth2.provider.ClientDetails;
	import javax.persistence.*;
	import java.util.Collection;
	import java.util.HashMap;
	import java.util.Map;
	import java.util.Set;
	import java.util.stream.Collectors;
	@Getter
	@Setter
	@Entity
	@Table(name = "clients",
	uniqueConstraints =
	{
	@UniqueConstraint(name = "uq_clientId", columnNames = {"clientId"})
	})
	@EntityListeners(AuditingEntityListener.class)
	public class TutorialClientDetails implements ClientDetails {
	public String getClientId() {
	return clientId;
	}
	public void setClientId(String clientId) {
	this.clientId = clientId;
	}
	public Set<String> getResourceIds() {
	return resourceIds;
	}
	public void setResourceIds(Set<String> resourceIds) {
	this.resourceIds = resourceIds;
	}
	public String getClientSecret() {
	return clientSecret;
	}
	public void setClientSecret(String clientSecret) {
	this.clientSecret = clientSecret;
	}
	public Set<String> getScope() {
	return scope;
	}
	public void setScope(Set<String> scope) {
	this.scope = scope;
	}
	public Set<String> getAuthorizedGrantTypes() {
	return authorizedGrantTypes;
	}
	public void setAuthorizedGrantTypes(Set<String> authorizedGrantTypes) {
	this.authorizedGrantTypes = authorizedGrantTypes;
	}
	public Set<String> getWebServerRedirectUri() {
	return webServerRedirectUri;
	}
	public void setWebServerRedirectUri(Set<String> webServerRedirectUri) {
	this.webServerRedirectUri = webServerRedirectUri;
	}
	public void setAuthorities(Set<GrantedAuthority> authorities) {
	this.authorities = authorities;
	}
	public int getAccessTokenValidity() {
	return accessTokenValidity;
	}
	public void setAccessTokenValidity(int accessTokenValidity) {
	this.accessTokenValidity = accessTokenValidity;
	}
	public int getRefreshTokenValidity() {
	return refreshTokenValidity;
	}
	public void setRefreshTokenValidity(int refreshTokenValidity) {
	this.refreshTokenValidity = refreshTokenValidity;
	}
	public void setAdditionalInformation(Map<String, String> additionalInformation) {
	this.additionalInformation = additionalInformation;
	}
	public boolean isAutoApprove() {
	return autoApprove;
	}
	public void setAutoApprove(boolean autoApprove) {
	this.autoApprove = autoApprove;
	}
	@Override
	public boolean isSecretRequired() {
	return clientSecret != null;
	}
	@Override
	public boolean isScoped() {
	return scope != null && !scope.isEmpty();
	}
	@Override
	public Set<String> getRegisteredRedirectUri() {
	return webServerRedirectUri;
	}
	@Override
	public Collection<GrantedAuthority> getAuthorities() {
	return authorities.stream().collect(Collectors.toSet());
	}
	@Override
	public Integer getAccessTokenValiditySeconds() {
	return accessTokenValidity;
	}
	@Override
	public Integer getRefreshTokenValiditySeconds() {
	return refreshTokenValidity;
	}
	@Override
	public boolean isAutoApprove(String s) {
	return autoApprove;
	}
	public Map<String, Object> getAdditionalInformation() {
	Map<String, Object> returnMap = new HashMap<>();
	additionalInformation.forEach((k,v) -> returnMap.put(k, v));
	return returnMap;
	}
	@Id
	private String clientId;
	@ElementCollection(fetch = FetchType.EAGER)
	private Set<String> resourceIds;
	private String clientSecret;
	@ElementCollection(fetch = FetchType.EAGER)
	private Set<String> scope;
	@ElementCollection(fetch = FetchType.EAGER)
	private Set<String> authorizedGrantTypes;
	@ElementCollection(fetch = FetchType.EAGER)
	private Set<String> webServerRedirectUri;
	@ElementCollection(fetch = FetchType.EAGER)
	private Set<GrantedAuthority> authorities;
	private int accessTokenValidity;
	private int refreshTokenValidity;
	@ElementCollection(fetch = FetchType.LAZY)
	private Map<String, String> additionalInformation;
	private boolean autoApprove;
	}

view raw TutorialClientDetails.java hosted with ❤ by GitHub

This entity just implements the ClientDetails interface

	package in.springframework.blog.tutorials.services;
	import in.springframework.blog.tutorials.entities.TutorialClientDetails;
	import in.springframework.blog.tutorials.repositories.ClientRepository;
	import lombok.extern.log4j.Log4j2;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.security.crypto.password.PasswordEncoder;
	import org.springframework.security.oauth2.provider.ClientDetails;
	import org.springframework.security.oauth2.provider.ClientDetailsService;
	import org.springframework.security.oauth2.provider.ClientRegistrationException;
	import org.springframework.stereotype.Service;
	import java.util.List;
	import java.util.Optional;
	@Service
	@Log4j2
	public class TutorialClientDetailsService implements ClientDetailsService {
	@Autowired
	private ClientRepository clientRepository;
	@Autowired
	private PasswordEncoder passwordEncoder;
	@Override
	public ClientDetails loadClientByClientId(String clientId) throws ClientRegistrationException {
	Optional<TutorialClientDetails> optionalClientDetails = clientRepository.findById(clientId);
	if (optionalClientDetails.isPresent()) {
	return optionalClientDetails.get();
	}
	throw new ClientRegistrationException(String.format("%s client doesn't exist!", clientId));
	}
	}

view raw TutorialClientDetailsService.java hosted with ❤ by GitHub

This service needs to implement a method loadClientByClientId.

Now that all the foundation work is in place, we get down to making our service multitenant. The first things that we need to do is to remove all the old dataSources that we had defined. We will now define a routing data source that will choose the right data source based on the tenant that we are taking to. This is where our tenant discriminator thread local will be usedful. Take a look at the following class.

	package in.springframework.blog.tutorials.configs;
	import in.springframework.blog.tutorials.utils.TutorialRequestContext;
	import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;
	public class TutorialMultitenantDataSource extends AbstractRoutingDataSource {
	@Override
	protected Object determineCurrentLookupKey() {
	if (TutorialRequestContext.currentTenantDiscriminator.get() != null) {
	return TutorialRequestContext.currentTenantDiscriminator.get();
	}
	else {
	return "unknown";
	}
	}
	}

view raw TutorialMultitenantDataSource.java hosted with ❤ by GitHub

Here we implement a method determineCurrentLookupKey which uses thread local to identify the current tenant and returns the key.

	package in.springframework.blog.tutorials.configs;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.boot.autoconfigure.domain.EntityScan;
	import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
	import org.springframework.boot.context.properties.ConfigurationProperties;
	import org.springframework.boot.jdbc.DataSourceBuilder;
	import org.springframework.context.ApplicationContext;
	import org.springframework.context.annotation.Bean;
	import org.springframework.context.annotation.Configuration;
	import org.springframework.context.annotation.Primary;
	import org.springframework.core.io.Resource;
	import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
	import org.springframework.orm.hibernate5.LocalSessionFactoryBean;
	import org.springframework.orm.jpa.JpaTransactionManager;
	import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
	import org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter;
	import org.springframework.transaction.PlatformTransactionManager;
	import org.springframework.transaction.annotation.EnableTransactionManagement;
	import javax.sql.DataSource;
	import java.io.File;
	import java.io.FileInputStream;
	import java.io.IOException;
	import java.util.HashMap;
	import java.util.Map;
	import java.util.Properties;
	@Configuration
	@EnableTransactionManagement
	@EnableJpaRepositories(
	entityManagerFactoryRef = "multiEntityManager",
	transactionManagerRef = "multiTransactionManager",
	basePackages = {"in.springframework.blog.tutorials.repositories"})
	@EntityScan(basePackages = {"in.springframework.blog.tutorials.entities"})
	public class TutorialMultitenantConfig {
	@Autowired
	private DataSourceProperties dataSourceProperties;
	@Autowired
	private ApplicationContext applicationContext;
	private final String TENANT_PROPERTIES_RESOURCE = "classpath:tenants/*.properties";
	@Primary
	@Bean(name = "dataSource")
	@ConfigurationProperties(
	prefix = "spring.datasource"
	)
	public DataSource dataSource() {
	Map<Object, Object> resolvedDataSources = new HashMap<>();
	try {
	Resource[] resources = applicationContext.getResources(TENANT_PROPERTIES_RESOURCE);
	for (Resource resource : resources) {
	File propertyFile = resource.getFile();
	Properties tenantProperties = new Properties();
	DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create(this.getClass().getClassLoader());
	try {
	tenantProperties.load(new FileInputStream(propertyFile));
	String tenantId = tenantProperties.getProperty("spring.datasource.name");
	// Assumption: The tenant database uses the same driver class
	// as the default database that you configure.
	dataSourceBuilder.driverClassName(dataSourceProperties.getDriverClassName())
	.url(tenantProperties.getProperty("spring.datasource.url"))
	.username(tenantProperties.getProperty("spring.datasource.username"))
	.password(tenantProperties.getProperty("spring.datasource.password"));
	if (dataSourceProperties.getType() != null) {
	dataSourceBuilder.type(dataSourceProperties.getType());
	}
	resolvedDataSources.put(tenantId, dataSourceBuilder.build());
	} catch (IOException e) {
	// Ooops, tenant could not be loaded. This is bad.
	// Stop the application!
	e.printStackTrace();
	return null;
	}
	}
	// Create the final multi-tenant source.
	// It needs a default database to connect to.
	// Make sure that the default database is actually an empty tenant database.
	// Don't use that for a regular tenant if you want things to be safe!
	TutorialMultitenantDataSource dataSource = new TutorialMultitenantDataSource();
	dataSource.setDefaultTargetDataSource(defaultDataSource());
	dataSource.setTargetDataSources(resolvedDataSources);
	// Call this to finalize the initialization of the data source.
	dataSource.afterPropertiesSet();
	return dataSource;
	}
	catch (IOException e) {
	}
	return defaultDataSource();
	}
	/**
	* Creates the default data source for the application
	* @return
	*/
	private DataSource defaultDataSource() {
	DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create(this.getClass().getClassLoader())
	.driverClassName(dataSourceProperties.getDriverClassName())
	.url(dataSourceProperties.getUrl())
	.username(dataSourceProperties.getUsername())
	.password(dataSourceProperties.getPassword());
	if(dataSourceProperties.getType() != null) {
	dataSourceBuilder.type(dataSourceProperties.getType());
	}
	return dataSourceBuilder.build();
	}
	@Bean(name = "multiEntityManager")
	public LocalContainerEntityManagerFactoryBean multiEntityManager() {
	LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
	em.setDataSource(dataSource());
	em.setPackagesToScan("in.springframework.blog.tutorials.entities");
	HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
	em.setJpaVendorAdapter(vendorAdapter);
	em.setJpaProperties(hibernateProperties());
	return em;
	}
	@Bean(name = "multiTransactionManager")
	public PlatformTransactionManager multiTransactionManager() {
	JpaTransactionManager transactionManager
	= new JpaTransactionManager();
	transactionManager.setEntityManagerFactory(
	multiEntityManager().getObject());
	return transactionManager;
	}
	@Primary
	@Bean(name = "dbSessionFactory")
	public LocalSessionFactoryBean dbSessionFactory() {
	LocalSessionFactoryBean sessionFactoryBean = new LocalSessionFactoryBean();
	sessionFactoryBean.setDataSource(dataSource());
	sessionFactoryBean.setPackagesToScan("in.springframework.blog.tutorials.entities");
	sessionFactoryBean.setHibernateProperties(hibernateProperties());
	return sessionFactoryBean;
	}
	private Properties hibernateProperties() {
	Properties properties = new Properties();
	properties.put("hibernate.show_sql", true);
	properties.put("hibernate.format_sql", true);
	return properties;
	}
	}

view raw TutorialMultitenantConfig.java hosted with ❤ by GitHub

The bulk of smartness lies in the class TutorialMultitenantConfig, specifically the dataSource bean that returns the default dataSource for the system. What we are assuming that within our resource directory, we will have a tenants subdirectory and within that we will have one properties file per tenant. The property file will look like below.

 

	spring.datasource.name=tenant1
	spring.datasource.url=jdbc:mysql://localhost:3306/tenant1?useSSL=false&allowPublicKeyRetrieval=true
	spring.datasource.username=tenant1
	spring.datasource.password=tenant1123
	spring.datasource.driver-class-name=com.mysql.jdbc.Driver
	spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect

view raw temamt1.properties hosted with ❤ by GitHub

Here we have added usual spring data source properties alongwith a name property which will be the name of the tenant. This name will be matched with the name in the URL and the header.

In the class TutorialMultitenantConfig, look at the definition of variable TENANT_PROPERTIES_RESOURCE, this basically looks up for all the *.properties file in the tenant directory.  Lines 61 through 74, create a data source object for each of the tenants and store these in a map with key being the tenant name. We remember determineCurrentLookupKey method which returned the name of current tenant, that return value is used to fetch appropriate data source object for the request being processed. Line 88 defines a default data source that is used if there is no data source present for the given tenant.

Now our multi tenanted service is ready and it is time to test. The first thing that we need to do is create couple of more databases with exact same schema as the first database. Please keep in mind that this has to be done manually even if you have defined ddl-auto property. Just take a mysqldump of the first database and import in two other databases.

We also need to make sure that the system atleast has one tenant defined in each of the databases. This is required in order to make sure we are able to use the rest calls. The best approach is to have a default tenant and then take a dump so that default tenant is also copied in each of the databases. We have created an admin user which has ADMIN role. We call the user endpoint that will return the list of all the users.

$ curl --request GET \
>   --url http://localhost:8081/tenant1/user \
>   --header 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI5Mzc1MzksInVzZXJfbmFtZSI6ImFkbWluIiwiYXV0aG9yaXRpZXMiOlsiVVNFUiIsIkFETUlOIl0sImp0aSI6IjRmOTdiNThkLTI1NTEtNDA4Yi04ZWM4LWUzZGZmZWQ3MTQ2NiIsImNsaWVudF9pZCI6InN1cGVyc2VjcmV0Y2xpZW50Iiwic2NvcGUiOlsicmVhZCIsImNvZGUiLCJ3cml0ZSJdfQ.fWt_H-ORHP44xOIljoqMfVIeGkqJGQqUBMj8paVxAPM' \
>   --header 'cache-control: no-cache' \
>   --header 'content-type: application/json' \
>   --header 'postman-token: 76b3525e-bc3b-e629-91b6-a75253f657d8' \
>   --header 'x-tenant: tenant1' |python -m json.tool

[
    {
        "createdAt": 1592936869000,
        "createdBy": "UnAuthenticated",
        "email": "admin@springframework.in",
        "fullname": "Spring Tutorial Admin",
        "grantedAuthorities": [
            "ADMIN",
            "USER"
        ],
        "id": 6,
        "mask": 1,
        "tenantId": 1,
        "updatedAt": 1592936869000,
        "updatedBy": "UnAuthenticated",
        "username": "admin"
    },
    {
        "createdAt": 1592904896000,
        "createdBy": "UnAuthenticated",
        "email": "defaultuser@defaultadmin.com",
        "fullname": "Default User",
        "grantedAuthorities": [],
        "id": 2,
        "mask": 0,
        "tenantId": 1,
        "updatedAt": 1592904896000,
        "updatedBy": "UnAuthenticated",
        "username": "defaultuser"
    },
    {
        "createdAt": 1592904900000,
        "createdBy": "UnAuthenticated",
        "email": "vinay@avasthi.com",
        "fullname": "Vinay Avasthi",
        "grantedAuthorities": [],
        "id": 3,
        "mask": 1,
        "tenantId": 1,
        "updatedAt": 1592904900000,
        "updatedBy": "UnAuthenticated",
        "username": "vavasthi"
    }
]

Now we run the same query in tenant2

$ curl --request GET   --url http://localhost:8081/tenant2/user   --header 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI5MzgzMDcsInVzZXJfbmFtZSI6ImFkbWluIiwiYXV0aG9yaXRpZXMiOlsiVVNFUiIsIkFETUlOIl0sImp0aSI6IjJmOWVkOTYxLTk5MjktNDI3Zi1iZGU4LTc5NGIwYWNhMGYzNiIsImNsaWVudF9pZCI6InN1cGVyc2VjcmV0Y2xpZW50Iiwic2NvcGUiOlsicmVhZCIsImNvZGUiLCJ3cml0ZSJdfQ.KNzjB_JqJDaVI5vZNK-OcXDgvM5Uwt4I8tsVCazerpU'   --header 'cache-control: no-cache'   --header 'content-type: application/json'   --header 'postman-token: b1a9651c-e4c0-7b2e-c7b3-e0b3e06fa40e'   --header 'x-tenant: tenant2' |python -m json.tool
[
    {
        "createdAt": 1592904896000,
        "createdBy": "UnAuthenticated",
        "email": "defaultuser@defaultadmin.com",
        "fullname": "Default User",
        "grantedAuthorities": [],
        "id": 2,
        "mask": 0,
        "tenantId": 1,
        "updatedAt": 1592904896000,
        "updatedBy": "UnAuthenticated",
        "username": "defaultuser"
    },
    {
        "createdAt": 1592904900000,
        "createdBy": "UnAuthenticated",
        "email": "vinay@avasthi.com",
        "fullname": "Vinay Avasthi",
        "grantedAuthorities": [],
        "id": 3,
        "mask": 1,
        "tenantId": 1,
        "updatedAt": 1592904900000,
        "updatedBy": "UnAuthenticated",
        "username": "vavasthi"
    },
    {
        "createdAt": 1592938002000,
        "createdBy": "UnAuthenticated",
        "email": "ut2@springframework.in",
        "fullname": "User in T2",
        "grantedAuthorities": [],
        "id": 6,
        "mask": 1,
        "tenantId": 1,
        "updatedAt": 1592938002000,
        "updatedBy": "UnAuthenticated",
        "username": "userint2"
    },
{
        "createdAt": 1592937749000,
        "createdBy": "UnAuthenticated",
        "email": "admin@springframework.in",
        "fullname": "Springframework Tenant2 Administrator",
        "grantedAuthorities": [
            "ADMIN",
            "USER"
        ],
        "id": 5,
        "mask": 1,
        "tenantId": 1,
        "updatedAt": 1592937749000,
        "updatedBy": "UnAuthenticated",
        "username": "admin"
    }
]

As we can see, we are seeing two totally different sets of users which are stored in two totally different sets of databases.

Complete code for this blog post is available in my github repository here. 

OAuth2 and JWT Tokens Part 1

In this blog post we look at how do we make our spring server become an OAuth2 Authorization server and start producing JWT tokens.

The first step that we take is disable all the filters that we had added earlier. We don't want to authorize using old style REST calls.  

Now, the first step is to enable our authorization server. We create a new java configuration file that extends AuthorizationServerConfigurerAdapter.

	package in.springframework.blog.tutorials.configs;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.beans.factory.annotation.Qualifier;
	import org.springframework.context.annotation.Bean;
	import org.springframework.context.annotation.Configuration;
	import org.springframework.context.annotation.Primary;
	import org.springframework.security.authentication.AuthenticationManager;
	import org.springframework.security.core.userdetails.UserDetailsService;
	import org.springframework.security.crypto.password.PasswordEncoder;
	import org.springframework.security.crypto.scrypt.SCryptPasswordEncoder;
	import org.springframework.security.oauth2.config.annotation.configurers.ClientDetailsServiceConfigurer;
	import org.springframework.security.oauth2.config.annotation.web.configuration.AuthorizationServerConfigurerAdapter;
	import org.springframework.security.oauth2.config.annotation.web.configuration.EnableAuthorizationServer;
	import org.springframework.security.oauth2.config.annotation.web.configurers.AuthorizationServerEndpointsConfigurer;
	import org.springframework.security.oauth2.config.annotation.web.configurers.AuthorizationServerSecurityConfigurer;
	import org.springframework.security.oauth2.provider.client.JdbcClientDetailsService;
	import org.springframework.security.oauth2.provider.token.DefaultTokenServices;
	import org.springframework.security.oauth2.provider.token.ResourceServerTokenServices;
	import org.springframework.security.oauth2.provider.token.TokenStore;
	import org.springframework.security.oauth2.provider.token.store.JwtAccessTokenConverter;
	import org.springframework.security.oauth2.provider.token.store.JwtTokenStore;
	import javax.sql.DataSource;
	@Configuration
	@EnableAuthorizationServer
	public class TutorialAuthServerConfig extends AuthorizationServerConfigurerAdapter {
	@Autowired
	private AuthenticationManager authenticationManager;
	@Autowired
	@Qualifier("dataSource")
	private DataSource dataSource;
	@Autowired
	private UserDetailsService userDetailsService;
	@Override
	public void configure(AuthorizationServerEndpointsConfigurer endpoints) throws Exception {
	endpoints.tokenStore(tokenStore())
	.accessTokenConverter(accessTokenConverter())
	.authenticationManager(authenticationManager)
	.userDetailsService(userDetailsService);
	}
	@Override
	public void configure(AuthorizationServerSecurityConfigurer oauthServer) {
	oauthServer.tokenKeyAccess("permitAll()").checkTokenAccess("permitAll()");
	}
	@Bean
	public TokenStore tokenStore() {
	return new JwtTokenStore(accessTokenConverter());
	}
	@Override
	public void configure(ClientDetailsServiceConfigurer clients) throws Exception {
	JdbcClientDetailsService jdbcClientDetailsService = new JdbcClientDetailsService(dataSource);
	jdbcClientDetailsService.setPasswordEncoder(userPasswordEncoder());
	clients.withClientDetails(jdbcClientDetailsService);
	}
	@Bean
	public JwtAccessTokenConverter accessTokenConverter() {
	JwtAccessTokenConverter converter = new JwtAccessTokenConverter();
	converter.setSigningKey("123");
	return converter;
	}
	@Bean
	@Primary
	public ResourceServerTokenServices tokenServices() {
	DefaultTokenServices defaultTokenServices = new DefaultTokenServices();
	defaultTokenServices.setTokenStore(tokenStore());
	defaultTokenServices.setSupportRefreshToken(true);
	return defaultTokenServices;
	}
	@Bean
	public PasswordEncoder userPasswordEncoder() {
	return new SCryptPasswordEncoder();
	}
	}

view raw TutorialAuthServerConfig.java hosted with ❤ by GitHub

In the configuration , we autowire AuthenticationManager, DataSource and a UserDetailsService. There are two types of auth tokens in OAuth2. The first one are the usual tokens that are requested by users by providing their username and password. The second set of tokens are not tied to individual users, these are called client tokens and could be used for services talking to each other.

We configure ClientDetailsService to use Jdbc client service. We add a password encoder in the configuration and provide a data source that would be used to store persistent data. In our example, we are using SCryptPasswordEncoder. Here we are using spring provided Jdbc client service but one could implement ClientDetailsService and ClientRegistrationService interfaces and provide their own custom implementation for client service.

The next steps is to enable web security and provide an authentication manager bean for performing authentication.

	package in.springframework.blog.tutorials.configs;
	import org.springframework.context.annotation.Bean;
	import org.springframework.context.annotation.Configuration;
	import org.springframework.security.authentication.AuthenticationManager;
	import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
	import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
	@Configuration
	@EnableWebSecurity
	public class TutorialWebServerSecurityConfig extends WebSecurityConfigurerAdapter {
	@Override
	@Bean
	public AuthenticationManager authenticationManagerBean() throws Exception {
	return super.authenticationManagerBean();
	}
	}

view raw TutorialWebServerSecurityConfig.java hosted with ❤ by GitHub

We have seen earlier that we used a UserDetailsService to configure the authorization server. We are not going to use the spring provided service but write our own. 

	package in.springframework.blog.tutorials.services;
	import in.springframework.blog.tutorials.entities.User;
	import in.springframework.blog.tutorials.exceptions.NotFoundException;
	import in.springframework.blog.tutorials.pojos.TutorialGrantedAuthority;
	import in.springframework.blog.tutorials.pojos.TutorialUserDetails;
	import in.springframework.blog.tutorials.repositories.UserRepository;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.security.core.GrantedAuthority;
	import org.springframework.security.core.userdetails.UserDetails;
	import org.springframework.security.core.userdetails.UserDetailsService;
	import org.springframework.stereotype.Service;
	import org.springframework.transaction.annotation.Transactional;
	import java.util.HashSet;
	import java.util.Optional;
	import java.util.Set;
	@Service
	@Transactional
	public class TutorialUserDetailService implements UserDetailsService {
	@Autowired
	private UserRepository userRepository;
	@Override
	public UserDetails loadUserByUsername(String username) throws NotFoundException {
	Optional<User> user = userRepository.findUserByUsername(username);
	if (user != null) {
	TutorialUserDetails userDetails = new TutorialUserDetails();
	userDetails.setUserName(user.get().getUsername());
	userDetails.setPassword(user.get().getPassword());
	Set<GrantedAuthority> authorities = new HashSet<GrantedAuthority>();
	for (String authority : user.get().getGrantedAuthorities()) {
	authorities.add(new TutorialGrantedAuthority(authority));
	}
	userDetails.setGrantedAuthorities(authorities);
	return userDetails;
	}
	throw new NotFoundException(username);
	}
	}

view raw TutorialUserDetailService.java hosted with ❤ by GitHub

In the UserDetailsService, we need to provide our own implementation for a method loadByUsername. In this method, we basically load the user entity from our database and create an object of type UserDetails and return it. The object also requires a list of GrantedAuthority. 

UserDetails is an interface provided in springframework, we create our own class that implements the interface.

	package in.springframework.blog.tutorials.pojos;
	import lombok.Getter;
	import lombok.Setter;
	import org.springframework.security.core.GrantedAuthority;
	import org.springframework.security.core.userdetails.UserDetails;
	import java.util.Collection;
	import java.util.Set;
	@Getter
	@Setter
	public class TutorialUserDetails implements UserDetails {
	private String userName;
	private String password;
	private boolean accountNonLocked;
	private boolean accountMonExpired;
	private boolean credentialsNonExpired;
	private Set<GrantedAuthority> grantedAuthorities;
	@Override
	public Collection<? extends GrantedAuthority> getAuthorities() {
	return grantedAuthorities;
	}
	@Override
	public String getPassword() {
	return password;
	}
	@Override
	public String getUsername() {
	return userName;
	}
	@Override
	public boolean isAccountNonExpired() {
	return true;
	}
	@Override
	public boolean isAccountNonLocked() {
	return true;
	}
	@Override
	public boolean isCredentialsNonExpired() {
	return true;
	}
	@Override
	public boolean isEnabled() {
	return true;
	}
	public void setUserName(String userName) {
	this.userName = userName;
	}
	public Set<GrantedAuthority> getGrantedAuthorities() {
	return grantedAuthorities;
	}
	public void setGrantedAuthorities(Set<GrantedAuthority> grantedAuthorities) {
	this.grantedAuthorities = grantedAuthorities;
	}
	public void setPassword(String password) {
	this.password = password;
	}
	}

view raw TutorialUserDetails.java hosted with ❤ by GitHub

Similar to UserDetails, GrantedAuthority is also an interface provided by springframework, we implement that interface to provide our concrete implementation of GrantedAuthority. For a very simple understanding, GrantedAuthority is like a user role. We can use this role later to provide access control on endpoints.

	package in.springframework.blog.tutorials.pojos;
	import lombok.AllArgsConstructor;
	import lombok.Getter;
	import lombok.NoArgsConstructor;
	import org.springframework.security.core.GrantedAuthority;
	import java.io.Serializable;
	@Getter
	@AllArgsConstructor
	@NoArgsConstructor
	public class TutorialGrantedAuthority implements GrantedAuthority, Serializable {
	private String authority;
	}

view raw TutorialGrantedAuthority.java hosted with ❤ by GitHub

These changes will make the server ready for OAuth2 service. We still have a testing nightmare. Because now all our endpoints are behind this authentication filter, there is no way for us to create new clients and users. We could directly insert values in database, but we still have to worry about how to encrypt the password before inserting into the data. To get out of this situation, I add two endpoints, one for handling clients and another for handling users. These endpoints need to be configured so that they don't go through the authentication service. These need to be removed before the service goes into production.

We define a pojo that mimics the OauthClientDetails schema in the database. It looks like below.

	package in.springframework.blog.tutorials.entities;
	import lombok.Getter;
	import lombok.Setter;
	import javax.persistence.Column;
	import javax.persistence.Entity;
	import javax.persistence.Id;
	import javax.persistence.Table;
	@Getter
	@Setter
	@Entity
	@Table(name = "oauth_client_details")
	public class OauthClientDetails {
	@Id
	@Column(name = "client_id")
	private String clientId;
	@Column(name = "resource_ids")
	private String resourceIds;
	@Column(name = "client_secret")
	private String clientSecret;
	private String scope;
	@Column(name = "authorized_grant_types")
	private String authorizedGrantTypes;
	@Column(name = "web_server_redirect_uri")
	private String webServerRedirectUri;
	private String authorities;
	@Column(name = "access_token_validity")
	private int accessTokenValidity;
	@Column(name = "refresh_token_validity")
	private int refreshTokenValidity;
	@Column(name = "additional_information")
	private String additionalInformation;
	@Column(name = "autoapprive")
	private String autoApprove;
	}

view raw OauthClientDetails.java hosted with ❤ by GitHub

Now we add a repository for handling OauthClientDetails table.

	package in.springframework.blog.tutorials.repositories;
	import in.springframework.blog.tutorials.entities.OauthClientDetails;
	import org.springframework.data.jpa.repository.JpaRepository;
	public interface ClientRepository extends JpaRepository<OauthClientDetails, String> {
	}

view raw ClientRepository.java hosted with ❤ by GitHub

We also add a service layer for Client handling.

	package in.springframework.blog.tutorials.services;
	import in.springframework.blog.tutorials.entities.OauthClientDetails;
	import in.springframework.blog.tutorials.exceptions.NotFoundException;
	import in.springframework.blog.tutorials.repositories.ClientRepository;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.security.crypto.password.PasswordEncoder;
	import org.springframework.stereotype.Service;
	import java.util.Optional;
	@Service
	public class ClientService {
	@Autowired
	private ClientRepository clientRepository;
	@Autowired
	private PasswordEncoder passwordEncoder;
	public Iterable<OauthClientDetails> findAll() {
	return clientRepository.findAll();
	}
	public Optional<OauthClientDetails> findById(String clientId) {
	return clientRepository.findById(clientId);
	}
	public Optional<OauthClientDetails> update(String clientId, OauthClientDetails clientDetails) {
	Optional<OauthClientDetails> optionalOauthClientDetails
	= clientRepository.findById(clientId);
	if (!optionalOauthClientDetails.isPresent()) {
	throw new NotFoundException(String.format("%s is not a valid client id.", clientId));
	}
	OauthClientDetails storedClientDetails = optionalOauthClientDetails.get();
	storedClientDetails.setClientSecret(passwordEncoder.encode(clientDetails.getClientSecret()));
	return Optional.of(clientRepository.save(storedClientDetails));
	}
	}

view raw ClientService.java hosted with ❤ by GitHub

We already had a UserRepository, we change it to encrypt the password before use store the password.

	package in.springframework.blog.tutorials.repositories;
	import in.springframework.blog.tutorials.entities.Tenant;
	import in.springframework.blog.tutorials.entities.User;
	import org.springframework.data.jpa.repository.JpaRepository;
	import org.springframework.data.jpa.repository.Query;
	import org.springframework.data.repository.query.Param;
	import java.util.Optional;
	public interface UserRepository extends JpaRepository<User, Long> {
	@Query("SELECT u FROM User u where u.email=:email")
	Optional<User> findUserByEmail(@Param("email") String email);
	@Query("SELECT u FROM User u where u.email=:emailOrUsername or u.username = :emailOrUsername")
	Optional<User> findUserByEmailOrUsername(@Param("emailOrUsername") String emailOrUsername);
	@Query("SELECT u FROM User u where u.username=:username")
	Optional<User> findUserByUsername(@Param("username") String username);
	@Query("SELECT u FROM User u where u.authToken=:authToken")
	Optional<User> findUserByAuthToken(@Param("authToken") String AuthToken);
	@Query("SELECT u FROM User u where u.tenant=:tenant and (u.username = :emailOrUsername or u.email = :emailOrUsername)")
	Optional<User> findUserByTenantAndUsername(@Param("tenant") Tenant tenant,
	@Param("emailOrUsername") String emailOrUsername);
	}

view raw UserRepository.java hosted with ❤ by GitHub

We also add a service layer for User.

	package in.springframework.blog.tutorials.services;
	import in.springframework.blog.tutorials.entities.User;
	import in.springframework.blog.tutorials.repositories.UserRepository;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.stereotype.Service;
	import java.util.Optional;
	@Service
	public class UserService {
	@Autowired
	private UserRepository userRepository;
	public Iterable<User> findAll() {
	return userRepository.findAll();
	}
	public Optional<User> retrieveUser(String idOrUserNameOrEmail) {
	try {
	Long id = Long.parseLong(idOrUserNameOrEmail);
	Optional<User> optionalUser = userRepository.findById(id);
	if (optionalUser.isPresent()) {
	return optionalUser;
	}
	}
	catch(NumberFormatException e) {
	}
	Optional<User> optionalUser = userRepository.findUserByEmail(idOrUserNameOrEmail);
	if (optionalUser.isPresent()) {
	return optionalUser;
	}
	optionalUser = userRepository.findUserByUsername(idOrUserNameOrEmail);
	return optionalUser;
	}
	public User save(User user) {
	return userRepository.save(user);
	}
	public void delete(User user) {
	userRepository.delete(user);
	}
	}

view raw UserService.java hosted with ❤ by GitHub

Now that we have all the layers required, we add the endpoint for Client.

	package in.springframework.blog.tutorials.endpoints;
	import in.springframework.blog.tutorials.entities.OauthClientDetails;
	import in.springframework.blog.tutorials.services.ClientService;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.http.MediaType;
	import org.springframework.web.bind.annotation.*;
	import java.util.Optional;
	@RestController
	@RequestMapping("/oauth/client")
	public class ClientEndpoint {
	@Autowired
	private ClientService clientService;
	@RequestMapping(method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
	public Iterable<OauthClientDetails> getClients() {
	return clientService.findAll();
	}
	@RequestMapping(value = "/{clientId}", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
	public Optional<OauthClientDetails> getClient(@PathVariable("clientId") String clientId) {
	return clientService.findById(clientId);
	}
	@RequestMapping(value = "/{clientId}", method = RequestMethod.PUT, produces = MediaType.APPLICATION_JSON_VALUE)
	public Optional<OauthClientDetails> getClient(@PathVariable("clientId") String clientId,
	@RequestBody OauthClientDetails clientDetails) {
	return clientService.update(clientId, clientDetails);
	}
	}

view raw ClientEndpoint.java hosted with ❤ by GitHub

The next thing to modify is the endpoint layer for user.

	package in.springframework.blog.tutorials.endpoints;
	import in.springframework.blog.tutorials.entities.User;
	import in.springframework.blog.tutorials.pojos.Role;
	import in.springframework.blog.tutorials.pojos.UserChangePassword;
	import in.springframework.blog.tutorials.services.UserService;
	import in.springframework.blog.tutorials.utils.RequestContext;
	import lombok.extern.log4j.Log4j2;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.http.MediaType;
	import org.springframework.security.access.prepost.PostFilter;
	import org.springframework.security.core.context.SecurityContextHolder;
	import org.springframework.security.crypto.password.PasswordEncoder;
	import org.springframework.web.bind.annotation.*;
	import javax.persistence.EntityNotFoundException;
	import java.util.Optional;
	@Log4j2
	@RestController
	@RequestMapping("/user")
	public class UserEndpoint {
	@Autowired
	private UserService userService;
	@Autowired
	private PasswordEncoder passwordEncoder;
	@RequestMapping(method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
	@PostFilter("hasAuthority('ADMIN') or filterObject.authToken == authentication.name")
	public Iterable<User> getUsers() {
	log.error(SecurityContextHolder.getContext().getAuthentication().getName());
	log.error(SecurityContextHolder.getContext().getAuthentication().getPrincipal());
	Iterable<User> users = userService.findAll();
	return users;
	}
	@RequestMapping(value = "/{idOrUserNameOrEmail}", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
	public Optional<User> getUser(@PathVariable("idOrUserNameOrEmail") String idOrUserNameOrEmail) {
	return userService.retrieveUser(idOrUserNameOrEmail);
	}
	@RequestMapping(method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
	// @PreAuthorize(MyConstants.ANNOTATION_ROLE_NEWUSER)
	public Optional<User> createUser(@RequestHeader(value="username") String username, @RequestBody User user) {
	user.setMask(Role.USER.ordinal());
	user.setUsername(username);
	user.setTenant(RequestContext.currentTenant.get());
	user.setPassword(passwordEncoder.encode(user.getPassword()));
	User storedUser = userService.save(user);
	storedUser.setPassword(null);
	return Optional.of(storedUser);
	}
	@RequestMapping(value = "/{idOrUserNameOrEmail}", method = RequestMethod.DELETE, produces = MediaType.APPLICATION_JSON_VALUE)
	public Optional<User> deleteUser(@PathVariable("idOrUserNameOrEmail") String idOrUserNameOrEmail) {
	Optional<User> optionalUser = userService.retrieveUser(idOrUserNameOrEmail);
	if (optionalUser.isPresent()) {
	userService.delete(optionalUser.get());
	return optionalUser;
	}
	throw new EntityNotFoundException(String.format("%s user not found.", idOrUserNameOrEmail));
	}
	@RequestMapping(value = "/changePassword/{idOrUserNameOrEmail}",
	method = RequestMethod.PATCH,
	produces = MediaType.APPLICATION_JSON_VALUE)
	public Optional<User> changePassword(@PathVariable("idOrUserNameOrEmail") String idOrUserNameOrEmail,
	@RequestBody UserChangePassword ucp) {
	Optional<User> optionalUser = userService.retrieveUser(idOrUserNameOrEmail);
	if (optionalUser.isPresent()) {
	User storedUser = optionalUser.get();
	// if (passwordEncoder.matches(ucp.getOldPassword(), storedUser.getPassword())) {
	storedUser.setPassword(passwordEncoder.encode(ucp.getNewPassword()));
	// }
	storedUser = userService.save(storedUser);
	storedUser.setPassword(null);
	return Optional.of(storedUser);
	}
	throw new EntityNotFoundException(String.format(String.format("User with id %s is not found", idOrUserNameOrEmail)));
	}
	@RequestMapping(value = "/{idOrUserNameOrEmail}",
	method = RequestMethod.PATCH,
	produces = MediaType.APPLICATION_JSON_VALUE)
	public Optional<User> patchUser(@PathVariable("idOrUserNameOrEmail") String idOrUserNameOrEmail,
	@RequestBody User user) {
	Optional<User> optionalUser = userService.retrieveUser(idOrUserNameOrEmail);
	if (optionalUser.isPresent()) {
	User storedUser = optionalUser.get();
	if (user.getEmail() != null) {
	storedUser.setEmail(user.getEmail());
	}
	if (user.getFullname() != null) {
	storedUser.setFullname(user.getFullname());
	}
	if (user.getPassword() != null) {
	storedUser.setPassword(user.getPassword());
	}
	return Optional.of(userService.save(storedUser));
	}
	throw new EntityNotFoundException(String.format(String.format("User with id %s is not found", idOrUserNameOrEmail)));
	}
	}

view raw UserEndpoint.java hosted with ❤ by GitHub

Now we are ready with our code for create new clients and users. We still have the small issue because if we hit these endpoints, we will get unauthorized error. So we need to put these in the exception list. For this we go to our SecurityConfiguration that we had defined in one of the earlier tutorials.

	package in.springframework.blog.tutorials.configs;
	import in.springframework.blog.tutorials.pojos.Role;
	import in.springframework.blog.tutorials.providers.TokenAuthenticationProvider;
	import in.springframework.blog.tutorials.providers.UsernamePasswordAuthenticationProvider;
	import in.springframework.blog.tutorials.utils.MyConstants;
	import org.springframework.context.annotation.Bean;
	import org.springframework.context.annotation.Configuration;
	import org.springframework.core.annotation.Order;
	import org.springframework.scheduling.annotation.EnableScheduling;
	import org.springframework.security.authentication.AuthenticationProvider;
	import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
	import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity;
	import org.springframework.security.config.annotation.web.builders.HttpSecurity;
	import org.springframework.security.config.annotation.web.builders.WebSecurity;
	import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
	import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
	import org.springframework.security.config.http.SessionCreationPolicy;
	import org.springframework.security.web.AuthenticationEntryPoint;
	import javax.servlet.http.HttpServletResponse;
	/**
	* Created by vinay on 1/27/16.
	*/
	@Configuration
	@EnableWebSecurity
	@EnableScheduling
	@EnableGlobalMethodSecurity(prePostEnabled = true)
	@Order(99)
	public class SecurityConfiguration extends WebSecurityConfigurerAdapter {
	@Override
	protected void configure(AuthenticationManagerBuilder auth) throws Exception {
	auth.authenticationProvider(tokenAuthenticationProvider());
	auth.authenticationProvider(usernamePasswordAuthenticationProvider());
	}
	@Override
	public void configure(WebSecurity web) throws Exception {
	web.ignoring().antMatchers("/manage/health");
	web.ignoring().antMatchers("/manage/info");
	web.ignoring().antMatchers("/webjars/**");
	web.ignoring().antMatchers("/error");
	web.ignoring().antMatchers("/swagger-ui.html");
	web.ignoring().antMatchers("/v2/api-docs/**");
	web.ignoring().antMatchers("/swagger-resources/**");
	web.ignoring().antMatchers("/oauth/client/**");
	web.ignoring().antMatchers("/user/**");
	}
	@Override
	protected void configure(HttpSecurity http) throws Exception {
	http.csrf().disable().
	sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS).
	and().
	authorizeRequests().
	antMatchers(actuatorEndpoints()).hasAuthority(Role.ADMIN.name()).
	anyRequest().authenticated().
	and().
	anonymous().disable().
	exceptionHandling().authenticationEntryPoint(unauthorizedEntryPoint());;
	/* http.addFilterBefore(new TenantHeaderFilter(), BasicAuthenticationFilter.class);
	http.addFilterBefore(new AuthenticationParameterExtractionFilter(), BasicAuthenticationFilter.class);
	http.addFilterBefore(new SufficientAuthenticationParametersPresentFilter(), BasicAuthenticationFilter.class);
	http.addFilterBefore(new AuthenticationFilter(authenticationManager()), BasicAuthenticationFilter.class);*/
	}
	private String[] actuatorEndpoints() {
	return new String[]{
	MyConstants.AUTOCONFIG_ENDPOINT,
	MyConstants.BEANS_ENDPOINT,
	MyConstants.CONFIGPROPS_ENDPOINT,
	MyConstants.ENV_ENDPOINT,
	MyConstants.MAPPINGS_ENDPOINT,
	MyConstants.METRICS_ENDPOINT,
	MyConstants.SHUTDOWN_ENDPOINT
	};
	}
	@Bean
	public AuthenticationEntryPoint unauthorizedEntryPoint() {
	return (request, response, authException) -> response.sendError(HttpServletResponse.SC_UNAUTHORIZED);
	}
	@Bean
	public AuthenticationProvider tokenAuthenticationProvider() {
	return new TokenAuthenticationProvider();
	}
	@Bean
	public AuthenticationProvider usernamePasswordAuthenticationProvider() {
	return new UsernamePasswordAuthenticationProvider();
	}
	}

view raw SecurityConfiguration.java hosted with ❤ by GitHub

Look at the method public void configure(WebSecurity web) throws Exception.  We have added following two lines to ignore evaluation of authentication filters for two families of URL.

        web.ignoring().antMatchers("/oauth/client/**");
        web.ignoring().antMatchers("/user/**");

Of course this is very dangerous and we have added it only for testing purpose. In a future tutorial we will have a more elegant solution for this.

Now we are ready. We can test the service with following curl or equivalent command.

curl -X POST \
  http://localhost:8081/oauth/token \
  -H 'authorization: Basic YW5kcm9pZC1jbGllbnQ6YW5kcm9pZC1zZWNyZXQ=' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/x-www-form-urlencoded' \
  -H 'postman-token: 3a349bc0-1230-adbe-4b79-9b938728a101' \
  -d 'grant_type=password&password=mypassword&username=myuser&client_id=my-client&client_secret=my-secret'

This is the response that we get

{
    "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjI5NTgsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiOGExZDYxN2ItZDU4OC00Nzc5LThlOTQtYTBiNWZkYzcxOTg2IiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl19.n7hCnBdnjMC8vuCsHxkOznfd06iJctGNeypx2fXWla4",
    "token_type": "bearer",
    "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMwNTgsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiNGU4ODg3OTYtZWZiZS00ZDc5LTg1YmMtN2EzNzFhM2I4Yzg0IiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl0sImF0aSI6IjhhMWQ2MTdiLWQ1ODgtNDc3OS04ZTk0LWEwYjVmZGM3MTk4NiJ9.xuElTWYSLscgdcqj0t-4t6prJbVOfHVqM331UUjfPBQ",
    "expires_in": 99,
    "scope": "code",
    "jti": "8a1d617b-d588-4779-8e94-a0b5fdc71986"
}

We can get the refresh token by calling the same endpoint with grant_type refresh_token.

curl -X POST \
>   http://localhost:8081/oauth/token \
>   -H 'authorization: Basic YW5kcm9pZC1jbGllbnQ6YW5kcm9pZC1zZWNyZXQ=' \
>   -H 'cache-control: no-cache' \
>   -H 'content-type: application/x-www-form-urlencoded' \
>   -H 'postman-token: 0adf261f-ed1e-85b3-67a5-21430fee8b38' \
>   -d 'grant_type=refresh_token&refresh_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMxOTAsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiNGMwNjczOTQtZjllNS00NDVjLTg5MzItMmRiMDM4N2U2ZjIxIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl0sImF0aSI6ImE1ZmY5MjhiLWQ1YjEtNDQ5Yy04N2I4LTU3ODgwYzY1NjM3NiJ9.RS2K7N5XCHQNov02WNu1QK1AqOvgc7MuNzeCydt7Ajs'

We get following response.

{
    "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMxNzQsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiY2E1OWU3NzktZDRkZS00NDU1LWFlNjItNDQ2NTAxYzUxZjhmIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl19.WgVgpBWuDmdJUk3UG8MTKOBsXn5zbEGO8gyqg2akN8o",
    "token_type": "bearer",
    "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMxOTAsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiNGMwNjczOTQtZjllNS00NDVjLTg5MzItMmRiMDM4N2U2ZjIxIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl0sImF0aSI6ImNhNTllNzc5LWQ0ZGUtNDQ1NS1hZTYyLTQ0NjUwMWM1MWY4ZiJ9.3zDqHLSwVLq_Dg8r4Ppf5tfvzxwT_7FEwdTV67l3VYQ",
    "expires_in": 99,
    "scope": "code",
    "jti": "ca59e779-d4de-4455-ae62-446501c51f8f"
}

We can verify a token by accessing oauth/check_token endpoint and providing the token.

curl --request POST \
>   --url http://localhost:8081/oauth/check_token \
>   --header 'authorization: Basic YW5kcm9pZC1jbGllbnQ6YW5kcm9pZC1zZWNyZXQ=' \
>   --header 'cache-control: no-cache' \
>   --header 'content-type: application/x-www-form-urlencoded' \
>   --header 'postman-token: ef035580-4cba-a7a1-deaa-dcf5f59e41fc' \
>   --data token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjM2MTEsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiODZhZjZhYmUtZjA4YS00YzIzLThkYzYtYjY2ZmM3ZWQ0YWRiIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl19.Kn284yuxxHSxVtEc3D8H5YjVjPi0yN6oX1hDwEx5bOo

We get the following response.

{
    "client_id": "android-client",
    "exp": 1592723611,
    "jti": "86af6abe-f08a-4c23-8dc6-b66fc7ed4adb",
    "scope": [
        "code"
    ],
    "user_name": "myuser"
}

This is all about enabling JWT OAuth tokens with any spring server. We will continue this tutorial in next part with some more important details. The complete code for the working server is here.

Auditing in MySQL

One of the important requirements in many RDBMS based workloads is to have audit log where any row in any table is stamped with who changed it and  when was it changed? To do this in every place where the table is modified is extremely cumbersome. In this blog post we look at how we can enable the springframework based auditing framework to perform this activity.

Before we do that, we need to upgrade the versions for our dependencies since I have been using this project for quite sometime. Also I decided to use lombok logging annotation and removed all the dependencies on log4j.

Here are the modifications to the pom.xml for setting dependencies.

	<?xml version="1.0" encoding="UTF-8"?>
	<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>in.springframework.blog</groupId>
	<artifactId>tutorials</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>
	<name>tutorials</name>
	<description>Demo project for Spring Boot</description>
	<parent>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-parent</artifactId>
	<version>2.3.0.RELEASE</version>
	<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
	<java.version>1.8</java.version>
	<swagger.version>2.9.2</swagger.version>
	<gson.version>2.8.6</gson.version>
	<log4j.version>2.13.3</log4j.version>
	</properties>
	<dependencies>
	<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-web</artifactId>
	<exclusions>
	<exclusion>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-tomcat</artifactId>
	</exclusion>
	</exclusions>
	</dependency>
	<dependency>
	<groupId>org.springframework.kafka</groupId>
	<artifactId>spring-kafka</artifactId>
	</dependency>
	<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	</dependency>
	<dependency>
	<groupId>mysql</groupId>
	<artifactId>mysql-connector-java</artifactId>
	</dependency>
	<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-data-jpa</artifactId>
	</dependency>
	<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-jetty</artifactId>
	</dependency>
	<dependency>
	<groupId>org.eclipse.jetty</groupId>
	<artifactId>jetty-deploy</artifactId>
	</dependency>
	<dependency>
	<groupId>org.eclipse.jetty</groupId>
	<artifactId>jetty-rewrite</artifactId>
	</dependency>
	<dependency>
	<groupId>org.eclipse.jetty</groupId>
	<artifactId>jetty-util</artifactId>
	</dependency>
	<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-security</artifactId>
	</dependency>
	<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-expression</artifactId>
	</dependency>
	<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-configuration-processor</artifactId>
	<optional>true</optional>
	</dependency>
	<dependency>
	<groupId>org.springframework.security</groupId>
	<artifactId>spring-security-web</artifactId>
	</dependency>
	<dependency>
	<groupId>org.springframework.security</groupId>
	<artifactId>spring-security-config</artifactId>
	</dependency>
	<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-actuator</artifactId>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core -->
	<dependency>
	<groupId>org.apache.logging.log4j</groupId>
	<artifactId>log4j-core</artifactId>
	<version>${log4j.version}</version>
	</dependency>
	<dependency>
	<groupId>com.google.code.gson</groupId>
	<artifactId>gson</artifactId>
	<version>${gson.version}</version>
	</dependency>
	<dependency>
	<groupId>io.springfox</groupId>
	<artifactId>springfox-swagger2</artifactId>
	<version>${swagger.version}</version>
	</dependency>
	<dependency>
	<groupId>io.springfox</groupId>
	<artifactId>springfox-swagger-ui</artifactId>
	<version>${swagger.version}</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.projectlombok/lombok -->
	<dependency>
	<groupId>org.projectlombok</groupId>
	<artifactId>lombok</artifactId>
	<scope>provided</scope>
	</dependency>
	<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-test</artifactId>
	<scope>test</scope>
	</dependency>
	<dependency>
	<groupId>org.springframework.security</groupId>
	<artifactId>spring-security-test</artifactId>
	<scope>test</scope>
	</dependency>
	</dependencies>
	<build>
	<plugins>
	<plugin>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-maven-plugin</artifactId>
	</plugin>
	<plugin>
	<groupId>pl.project13.maven</groupId>
	<artifactId>git-commit-id-plugin</artifactId>
	<version>2.2.1</version>
	</plugin>
	</plugins>
	</build>
	</project>

view raw pom-0506.xml hosted with ❤ by GitHub

Now we look at our User entity. We need to add auditing fields to this table. Since in a realistic project one would have multiple entities, we create a abstract entity base class with all the audit fields. We also move the primary key to the abstract base class. When we do that, hibernate provides multiple ways of mapping the entities to the database schema. We want all the parent attributes to be stored in a single table with the child class attribute. To accomplish that, we need to add @MappedSuperClass to the base class.

	package in.springframework.blog.tutorials.user.domain;
	import lombok.Getter;
	import lombok.Setter;
	import org.springframework.data.annotation.CreatedBy;
	import org.springframework.data.annotation.CreatedDate;
	import org.springframework.data.annotation.LastModifiedBy;
	import org.springframework.data.annotation.LastModifiedDate;
	import javax.persistence.*;
	import java.util.Date;
	@Getter
	@Setter
	@MappedSuperclass
	public abstract class AbstractBaseEntity {
	@Id
	@GeneratedValue(strategy= GenerationType.AUTO)
	private Long id;
	@CreatedBy
	private Long createdBy;
	@LastModifiedBy
	private Long updatedBy;
	@CreatedDate
	private Date createdAt;
	@LastModifiedDate
	private Date updatedAt;
	}

view raw AbstractBaseEntity.java hosted with ❤ by GitHub

As we can see, we have added five attributes in the base class. One is the id for all our entities and rest four will be used for auditing purposes.

At this time we also add another layer to our code. Currently all the Endpoints directly call the Repository layer, this causes a problem if we want to write functions that can be reused across different endpoints. An example of this need is retrieveUser method that takes an argument that could be a username or a email. Currently this method lies in the Endpoint layer as a private method. This is a useful method in many different contexts, so we create a new UserService layer and move this method there.

	package in.springframework.blog.tutorials;
	import in.springframework.blog.tutorials.user.domain.User;
	import in.springframework.blog.tutorials.user.repository.UserRepository;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.stereotype.Service;
	import java.util.Optional;
	@Service
	public class UserService {
	@Autowired
	private UserRepository userRepository;
	public Iterable<User> findAll() {
	return userRepository.findAll();
	}
	public Optional<User> retrieveUser(String idOrUserNameOrEmail) {
	try {
	Long id = Long.parseLong(idOrUserNameOrEmail);
	Optional<User> optionalUser = userRepository.findById(id);
	if (optionalUser.isPresent()) {
	return optionalUser;
	}
	}
	catch(NumberFormatException e) {
	}
	Optional<User> optionalUser = userRepository.findUserByEmail(idOrUserNameOrEmail);
	if (optionalUser.isPresent()) {
	return optionalUser;
	}
	optionalUser = userRepository.findUserByUsername(idOrUserNameOrEmail);
	return optionalUser;
	}
	public User save(User user) {
	return userRepository.save(user);
	}
	public void delete(User user) {
	userRepository.delete(user);
	}
	}

view raw UserService.java hosted with ❤ by GitHub

Now, let's get to the original task of enabling auditing. First we define a Auditing Config as below.

	package in.springframework.blog.tutorials.configs;
	import in.springframework.blog.tutorials.RequestContext;
	import in.springframework.blog.tutorials.user.domain.User;
	import org.springframework.context.annotation.Bean;
	import org.springframework.context.annotation.Configuration;
	import org.springframework.data.domain.AuditorAware;
	import org.springframework.data.jpa.repository.config.EnableJpaAuditing;
	import java.util.Optional;
	@Configuration
	@EnableJpaAuditing(auditorAwareRef = "auditorAware")
	public class EntityAuditConfig {
	@Bean
	public AuditorAware<Long> auditorAware() {
	return new AuditorAware<Long>() {
	@Override
	public Optional<Long> getCurrentAuditor() {
	return Optional.of(RequestContext.currentUser.get().getId());
	}
	};
	}
	}

view raw EntityAuditConfig.java hosted with ❤ by GitHub

We had earlier defined a ThreadLocal that is used by the auditAware method defined above to extract currently logged in user and return its userId. As we can see the audit fields in the AbstractBaseEntity expects a Long for @createdBy and  @LastModifiedBy fields. The EntityAuditConfig also has annotation @EnableJpaAuditing which is required.

At this point we also add a new endpoint called ProfileEndpoint which can be used to manage the entity that represents a user profile. This entity currently only contains a url.

Now if we perform any operation on any of the endpoint, we will see the auditing fields automatically populated. Give it a spin. It is a life saver in many productions applications. I have had situations where users changed their passwords, forgot them and then complained saying that they have been hacked. 

The complete code for this and previous posts can be found at my github repository. This tutorial changes are under v1.5.

Hadoop and Spark Locally

As a continuation of the last post, we now look at how to make it deployable in a proper Spark/Hadoop cluster. We will not go into the details of the setup of these clusters themselves but more into how do we make sure a program that we developed earlier could run as a job in a cluster.

We will continue with the setup in our local machine. I am using a Mac so the instructions are with respect to that but most instructions would be common to any other platform.

If Spark is processing data from a database and writing into a hive, pretty much what we did in the last post would work. The problem arises if some of the data being processed exists as flat files. If we want to submit our jobs to a Spark cluster, we can not use local files because the jobs are not running in the local file system. 

The best approach is to either use a hdfs cluster or deploy a single node hdfs on your machine. Here I am enumerating the steps to set up a single node hdfs cluster on a Mac OS X machine.

• Download the Hadoop distribution for your machine here.
• Hadoop distribution is available in the form of a .tar.gz file and you can expand it in some directory on your machine. The expansion will create a directory of the form hadoop-x.y.z assuming your Hadoop version is x.y.z. set the environment variable HADOOP_HOME to the full pathname of this Hadoop directory.
• Add $HADOOP_HOME/bin to the PATH variable.
• Now we need to update the configuration files for hadoop.

$ cd $HADOOP_HOME/etc/hadoop
$ vi core-site.xml

We update the file with the following properties.

	<property>
	<name>fs.default.name</name>
	<value><a href="hdfs://localhost:9000">hdfs://localhost:9000</a></value>
	</property>

view raw hadoop-core-site.xml hosted with ❤ by GitHub

$ vi hdfs-site.xml

We update the file with the following properties.

	<property>
	<name>dfs.replication</name>
	<value>1</value>
	</property>
	<property>
	<name>dfs.permission</name>
	<value>false</value>

view raw hadoop-hdfs-site.xml hosted with ❤ by GitHub

$ vi mapred-site.xml

We update the file with the following properties.

	<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
	</property>

view raw hadoop-mapred-site.xml hosted with ❤ by GitHub

$ vi yarn-site.xml

We update the file with the following properties.

	<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
	</property>
	<property>
	<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
	<value>org.apache.hadoop.mapred.ShuffleHandler</value>
	</property>

view raw hadoop-yarn-site.xml hosted with ❤ by GitHub

Now we start the Hadoop.

$ cd $HADOOP_HOME
$ sbin/start-all.sh

Now we can access files stored into the HDFS in our spark jobs.

	SparkSession sparkSession = SparkSession
	.builder()
	.appName("SparkFileSumApp")
	.config("spark.driver.host", "localhost")
	.config("spark.driver.memory", "8g")
	.master("local[*]")
	.getOrCreate();
	Dataset<Row> dataframe = sparkSession
	.read()
	.format("com.crealytics.spark.excel")
	.option("dataAddress", "mySheet!A3:K58760")
	.option("useHeader", "true")
	.option("inferSchema", "false")
	.load("hdfs://localhost:9000/user/myuser/data/det-assembly.xls")

view raw SparkJob.java hosted with ❤ by GitHub

The next post will go more into the details of how to process files in spark.

Hive and Spark

In this blog post, we take a slight deviation from core issues related to the spring framework and look at an issue that spring programmers might face regularly. Recently I was looking with Spark and found a need to read the data from MySQL, do some processing and write it back to a hive instance. We will look at this issue in this post.
We start by making sure our hive instance is backed by a database. To do this, we do the following.

$ cp $HIVE_HOME/conf/hive-default.xml.template $HIVE_HOME/conf/hive-site.xml
$ vi gnu/apache-hive-3.1.2-bin/conf/hive-site.xml

We edit the hive-site.xml file and make sure it is configured as below.

	<property>
	<name>javax.jdo.option.ConnectionURL</name>
	<value>jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true</value>
	</property>
	<property>
	<name>javax.jdo.option.ConnectionDriverName</name>
	<value>com.mysql.cj.jdbc.Driver</value>
	</property>
	<property>
	<name>javax.jdo.option.ConnectionUserName</name>
	<value>hive</value>
	</property>
	<property>
	<name>javax.jdo.option.ConnectionPassword</name>
	<value>hive123</value>
	</property>
	<property>
	<name>datanucleus.fixedDatastore</name>
	<value>false</value>
	</property>
	<property>
	<name>hive.exec.scratchdir</name>
	<value>/tmp/hive</value>
	</property>
	<property>
	<name>hive.exec.local.scratchdir</name>
	<value>/Data/appData/hive/tmpData</value>
	</property>
	<property>
	<name>hive.downloaded.resources.dir</name>
	<value>/Data/appData/hive/tmpData</value>
	</property>
	<property>
	<name>hive.querylog.location</name>
	<value>/Data/appData/hive/tmpData</value>
	</property>
	<property>
	<name>hive.metastore.warehouse.dir</name>
	<value>/apps/hive/warehouse</value>
	</property>
	<property>
	<name>hive.metastore.uris</name>
	<value>thrift://localhost:9083</value>
	</property>
	<property>
	<name>hive.cli.print.current.db</name>
	<value>true</value>
	</property>
	<property>
	<name>hive.cli.print.header</name>
	<value>true</value>
	</property>

view raw hive-site.xml hosted with ❤ by GitHub

We can configure the values to suit our needs. But make sure the MySQL username, password, database URL exists and has relevant permissions.
Now we create another database in MySQL which will contain the data that we need to process. I am calling this database mystuff, with username mystuff, password mystuff123. We need to run following commands in MySQL to make sure everything exists and permissions are appropriate.

create database mystuff;
Query OK, 1 row affected (0.00 sec)

mysql> create user mystuff@localhost identified by 'mystuff123';
Query OK, 0 rows affected (0.00 sec)

mysql> create user mystuff@'%' identified by 'mystuff123';
Query OK, 0 rows affected (0.01 sec)

mysql> grant all on mystuff.* to mystuff@localhost;
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on mystuff.* to mystuff@'%';
Query OK, 0 rows affected (0.00 sec)

Now we create a plain java project in IntelliJ with the following pom.xml file.

	<?xml version="1.0" encoding="UTF-8"?>
	<project>
	<modelVersion>4.0.0</modelVersion>
	<groupId>spark-reporting</groupId>
	<artifactId>spark-reporting-main</artifactId>
	<version>1.0-SNAPSHOT</version>
	<properties>
	<maven.compiler.source>1.8</maven.compiler.source>
	<maven.compiler.target>1.8</maven.compiler.target>
	</properties>
	<dependencies>
	<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
	<dependency>
	<groupId>org.apache.spark</groupId>
	<artifactId>spark-core_2.12</artifactId>
	<version>2.4.4</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
	<dependency>
	<groupId>org.apache.spark</groupId>
	<artifactId>spark-sql_2.12</artifactId>
	<version>2.4.4</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
	<dependency>
	<groupId>org.apache.spark</groupId>
	<artifactId>spark-hive_2.12</artifactId>
	<version>2.4.5</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-spark -->
	<dependency>
	<groupId>org.apache.hbase</groupId>
	<artifactId>hbase-spark</artifactId>
	<version>2.0.0-alpha4</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/com.oracle.jdbc/ojdbc6 -->
	<dependency>
	<groupId>oracle</groupId>
	<artifactId>jdbc</artifactId>
	<version>11.2.0.4</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
	<dependency>
	<groupId>mysql</groupId>
	<artifactId>mysql-connector-java</artifactId>
	<version>8.0.19</version>
	</dependency>
	</dependencies>
	<repositories>
	<repository>
	<id>ICM</id>
	<url>http://maven.icm.edu.pl/artifactory/repo/</url>
	</repository>
	</repositories>
	<build>
	<pluginManagement>
	<plugins>
	<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-compiler-plugin</artifactId>
	<version>3.8.1</version>
	</plugin>
	</plugins>
	</pluginManagement>
	</build>
	</project>

view raw spark-pom.xml hosted with ❤ by GitHub

Now to look at the problem at hand. We have a table in MySQL with the following structure.

mysql> desc mydata;
+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| id    | int          | YES  |     | NULL    |       |
| k     | varchar(10)  | YES  |     | NULL    |       |
| v     | varchar(255) | YES  |     | NULL    |       |
+-------+--------------+------+-----+---------+-------+
3 rows in set (0.00 sec)

We want to flatten this table such that each key gets converted to a column for each id. So assume our current data is as below.

mysql> select * from mydata;
+------+-------+-------------------+
| id   | k     | v                 |
+------+-------+-------------------+
|    1 | NAME  | John Doe.         |
|    1 | EMAIL | jd@example.com    |
|    2 | NAME  | Jane Doe          |
|    2 | EMAIL | janed@example.com |
+------+-------+-------------------+
2 rows in set (0.00 sec)

We want to load this data and convert it into a flattened table that has three columns i.e. id, name. email. Then we want to populate this into table person into hive with flattened data.

hive (default)> select * from person;
OK
person.id person.email person.name
1 jd@example.com.  John Doe
2 janed@example.com Jane Doe
Time taken: 0.094 seconds, Fetched: 2 row(s)

The following code will perform the above conversion.

	import org.apache.spark.sql.Dataset;
	import org.apache.spark.sql.Row;
	import org.apache.spark.sql.SaveMode;
	import org.apache.spark.sql.SparkSession;
	import static org.apache.spark.sql.functions.first;
	public class MySqlToHive {
	public static void main(String[] args) {
	SparkSession hiveSession = SparkSession
	.builder()
	.appName("interfacing spark sql to hive metastore without configuration file")
	.config("hive.metastore.uris", "thrift://localhost:9083") // replace with your hivemetastore service's thrift url
	.master("local[*]")
	.enableHiveSupport() // don't forget to enable hive support
	.getOrCreate();
	SparkSession sparkSession = SparkSession
	.builder()
	.appName("SparkFileSumApp")
	.config("spark.driver.host", "localhost")
	.config("spark.driver.memory", "8g")
	.master("local[*]")
	.getOrCreate();
	String mysqlUrl = "jdbc:mysql://localhost/mystuff?createDatabaseIfNotExist=true";
	String query = "(select * from mydata) mydata";
	Dataset<Row> dataframe = sparkSession
	.read()
	.format("jdbc")
	.option("url", mysqlUrl)
	.option("driver", "com.mysql.cj.jdbc.Driver")
	.option("dbtable",query)
	.option("user","mystuff")
	.option("password","mystuff123")
	.load();
	dataframe.persist();
	Dataset<Row> resultSet = dataframe
	.groupBy("id")
	.pivot("k")
	.agg(first("v"));
	resultSet.persist();
	Dataset<Row> df = resultSet.toDF();
	System.out.println("WRITING HIVE");
	hiveSession.createDataFrame(df.rdd(), df.schema())
	.write()
	.mode(SaveMode.Append)
	.saveAsTable("person");
	System.out.println("Number of rows are :" + resultSet.count());
	sparkSession.close();
	}
	}

view raw MySqlToHive.java hosted with ❤ by GitHub

Line numbers 11 through 17 create a SparkSession for hive operations. The key instruction here is enableHiveSupport. Lines 20 through 26 create a SparkSession that will be used for MySQL operations. Lines 29 through 37 load the complete contents of the table. Lines 39 through 42 will group the results by the id and pivot the table on the field K. Line 46 through 49 creates a data frame for hiveSession and writes the contents in a table with named person.

Spring and more Kafka

In the last post, we saw how to integrate Kafka with Spring Boot application. The post was a very simple implementation of Kafka. The real world is much more complex. You have to deal with multiple topics, you need multiple partitions. In this post, we explore more details of a spring boot application with Kafka.

Let's start with the fact that we have multiple topics that we are dealing with. The very first thing that we need to do is to separate out the properties for each of the topics.

	spring.kafka.topic1.bootstrap-servers=localhost:9092
	spring.kafka.topic1.consumer.group-id=firstGroupId
	spring.kafka.topic2.bootstrap-servers=localhost:9092
	spring.kafka.topic2.consumer.group-id=secondGroupId

view raw application-kafka-multiple.properties hosted with ❤ by GitHub

We can no longer use default KafkaTemplate, we will have to create our own. The best way is to define a KafkaProducer. To create a KafkaProducer, we need to create KarkaProducerConfig. Here are two sets of KafkaProducerConfig class, one each for the topic.

	package in.springframework.blog.tutorials.configs;
	import in.springframework.blog.tutorials.Constants;
	import org.apache.kafka.clients.producer.ProducerConfig;
	import org.apache.kafka.common.serialization.StringSerializer;
	import org.springframework.beans.factory.annotation.Value;
	import org.springframework.context.annotation.Bean;
	import org.springframework.kafka.core.DefaultKafkaProducerFactory;
	import org.springframework.kafka.core.KafkaTemplate;
	import org.springframework.kafka.core.ProducerFactory;
	import org.springframework.stereotype.Component;
	import java.util.HashMap;
	import java.util.Map;
	@Component
	public class FirstTopicProducerConfig {
	@Value("${spring.kafka.topic1.bootstrap-servers:localhost}")
	private String bootstrapAddress;
	public ProducerFactory<String, String> producerFactory() {
	Map<String, Object> configProps = new HashMap<>();
	configProps.put(
	ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
	bootstrapAddress);
	configProps.put(
	ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
	StringSerializer.class);
	configProps.put(
	ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
	StringSerializer.class);
	return new DefaultKafkaProducerFactory<>(configProps);
	}
	@Bean(name = Constants.FIRST_TOPIC_TEMPLATE_NAME)
	public KafkaTemplate<String, String> kafkaTemplate() {
	return new KafkaTemplate<>(producerFactory());
	}
	}

view raw FirstTopicProducerConfig.java hosted with ❤ by GitHub

	package in.springframework.blog.tutorials.configs;
	import in.springframework.blog.tutorials.Constants;
	import org.apache.kafka.clients.producer.ProducerConfig;
	import org.apache.kafka.common.serialization.StringSerializer;
	import org.springframework.beans.factory.annotation.Value;
	import org.springframework.context.annotation.Bean;
	import org.springframework.kafka.core.DefaultKafkaProducerFactory;
	import org.springframework.kafka.core.KafkaTemplate;
	import org.springframework.kafka.core.ProducerFactory;
	import org.springframework.stereotype.Component;
	import java.util.HashMap;
	import java.util.Map;
	@Component
	public class SecondTopicProducerConfig {
	@Value("${spring.kafka.topic2.bootstrap-servers:localhost}")
	private String bootstrapAddress;
	public ProducerFactory<String, String> producerFactory() {
	Map<String, Object> configProps = new HashMap<>();
	configProps.put(
	ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
	bootstrapAddress);
	configProps.put(
	ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
	StringSerializer.class);
	configProps.put(
	ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
	StringSerializer.class);
	return new DefaultKafkaProducerFactory<>(configProps);
	}
	@Bean(name = Constants.SECOND_TOPIC_TEMPLATE_NAME)
	public KafkaTemplate<String, String> kafkaTemplate() {
	return new KafkaTemplate<>(producerFactory());
	}
	}

view raw SecondTopicProducerConfig.java hosted with ❤ by GitHub

Looking at the producer config, we can observe that we create two beans which are qualified by names and return an appropriate KafkaTemplate which is later used to send the message. Similar to producer config, we need to create consumer config. Consumer config is used by the listener to listen for the message.

	package in.springframework.blog.tutorials.configs;
	import org.apache.kafka.clients.consumer.ConsumerConfig;
	import org.apache.kafka.common.serialization.StringDeserializer;
	import org.springframework.beans.factory.annotation.Value;
	import org.springframework.context.annotation.Bean;
	import org.springframework.context.annotation.Configuration;
	import org.springframework.kafka.annotation.EnableKafka;
	import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
	import org.springframework.kafka.core.ConsumerFactory;
	import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
	import java.util.HashMap;
	import java.util.Map;
	@EnableKafka
	@Configuration
	public class FirstTopicConsumerConfig {
	@Value("${spring.kafka.topic1.bootstrap-servers:localhost}")
	private String bootstrapAddress;
	@Value("${spring.kafka.topic1.group-id:firstgroup}")
	private String groupId;
	public ConsumerFactory<String, String> consumerFactory() {
	Map<String, Object> props = new HashMap<>();
	props.put(
	ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
	bootstrapAddress);
	props.put(
	ConsumerConfig.GROUP_ID_CONFIG,
	groupId);
	props.put(
	ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
	StringDeserializer.class);
	props.put(
	ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
	StringDeserializer.class);
	return new DefaultKafkaConsumerFactory<>(props);
	}
	@Bean(name="firstTopicListenerContainerFactory")
	public ConcurrentKafkaListenerContainerFactory<String, String>
	kafkaListenerContainerFactory() {
	ConcurrentKafkaListenerContainerFactory<String, String> factory
	= new ConcurrentKafkaListenerContainerFactory<>();
	factory.setConsumerFactory(consumerFactory());
	return factory;
	}
	}

view raw FirstTopicConsumerConfig.java hosted with ❤ by GitHub

	package in.springframework.blog.tutorials.configs;
	import org.apache.kafka.clients.consumer.ConsumerConfig;
	import org.apache.kafka.common.serialization.StringDeserializer;
	import org.springframework.beans.factory.annotation.Value;
	import org.springframework.context.annotation.Bean;
	import org.springframework.context.annotation.Configuration;
	import org.springframework.kafka.annotation.EnableKafka;
	import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
	import org.springframework.kafka.core.ConsumerFactory;
	import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
	import java.util.HashMap;
	import java.util.Map;
	@EnableKafka
	@Configuration
	public class SecondTopicConsumerConfig {
	@Value("${spring.kafka.topic2.bootstrap-servers:localhost}")
	private String bootstrapAddress;
	@Value("${spring.kafka.topic2.group-id:secondgroup}")
	private String groupId;
	public ConsumerFactory<String, String> consumerFactory() {
	Map<String, Object> props = new HashMap<>();
	props.put(
	ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
	bootstrapAddress);
	props.put(
	ConsumerConfig.GROUP_ID_CONFIG,
	groupId);
	props.put(
	ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
	StringDeserializer.class);
	props.put(
	ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
	StringDeserializer.class);
	return new DefaultKafkaConsumerFactory<>(props);
	}
	@Bean(name="secondTopicListenerContainerFactory")
	public ConcurrentKafkaListenerContainerFactory<String, String>
	kafkaListenerContainerFactory() {
	ConcurrentKafkaListenerContainerFactory<String, String> factory
	= new ConcurrentKafkaListenerContainerFactory<>();
	factory.setConsumerFactory(consumerFactory());
	return factory;
	}
	}

view raw SecondTopicConsumerConfig.java hosted with ❤ by GitHub

As we can see in each of the consumer configs, we create a bean which returns a ConcurrentKafkaListenerContainerFactory. The beans are qualified by a name so that we can use an appropriate container factory for receiving messages.
We also modify the MyTopicMessage and add a member variable topicName that will help us distinguish the topic to which the message needs to be sent.

	package in.springframework.blog.tutorials.pojos;
	import in.springframework.blog.tutorials.Constants;
	public class MyTopicMessage {
	public MyTopicMessage() {
	}
	public MyTopicMessage(String message, Constants.TOPIC_NAME topicName) {
	this.message = message;this.topicName = topicName;
	}
	public String getMessage() {
	return message;
	}
	public void setMessage(String message) {
	this.message = message;
	}
	public Constants.TOPIC_NAME getTopicName() {
	return topicName;
	}
	public void setTopicName(Constants.TOPIC_NAME topicName) {
	this.topicName = topicName;
	}
	private String message;
	private Constants.TOPIC_NAME topicName;
	}

view raw MyTopicMessage.java hosted with ❤ by GitHub

We also modify the endpoint so that the message can be sent to the appropriate topic.

	package in.springframework.blog.tutorials.endpoints;
	import in.springframework.blog.tutorials.Constants;
	import in.springframework.blog.tutorials.MyConstants;
	import in.springframework.blog.tutorials.pojos.MyTopicMessage;
	import lombok.extern.slf4j.Slf4j;
	import org.apache.log4j.Level;
	import org.apache.log4j.Logger;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.beans.factory.annotation.Qualifier;
	import org.springframework.http.MediaType;
	import org.springframework.kafka.core.KafkaTemplate;
	import org.springframework.security.access.prepost.PreAuthorize;
	import org.springframework.util.concurrent.ListenableFutureCallback;
	import org.springframework.web.bind.annotation.RequestBody;
	import org.springframework.web.bind.annotation.RequestMapping;
	import org.springframework.web.bind.annotation.RequestMethod;
	import org.springframework.web.bind.annotation.RestController;
	@RestController
	@RequestMapping("/send")
	public class MyTopicMessageSenderEndpoint {
	@Autowired
	@Qualifier(Constants.FIRST_TOPIC_TEMPLATE_NAME)
	private KafkaTemplate firstKafkaTemplate;
	@Autowired
	@Qualifier(Constants.SECOND_TOPIC_TEMPLATE_NAME)
	private KafkaTemplate secondKafkaTemplate;
	Logger logger = Logger.getLogger(MyTopicMessageSenderEndpoint.class);
	@RequestMapping(method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
	@PreAuthorize(MyConstants.ANNOTATION_ROLE_USER)
	public String sendMessage(@RequestBody MyTopicMessage message) {
	switch(message.getTopicName()) {
	case FirstTopic:
	firstKafkaTemplate.send(Constants.FIRST_TOPIC, message.getMessage()).addCallback(new ListenableFutureCallback() {
	@Override
	public void onFailure(Throwable throwable) {
	logger.log(Level.INFO, String.format("Message '%s' failed!", message.getMessage()));
	}
	@Override
	public void onSuccess(Object o) {
	logger.log(Level.INFO, String.format("Message '%s' sent successfully!", message.getMessage()));
	}
	});
	break;
	case SecondTopic:
	secondKafkaTemplate.send(Constants.SECOND_TOPIC, message.getMessage()).addCallback(new ListenableFutureCallback() {
	@Override
	public void onFailure(Throwable throwable) {
	logger.log(Level.INFO, String.format("Message '%s' failed!", message.getMessage()));
	}
	@Override
	public void onSuccess(Object o) {
	logger.log(Level.INFO, String.format("Message '%s' sent successfully!", message.getMessage()));
	}
	});
	break;
	}
	return "Success!";
	}
	}

view raw MyTopicMessageSenderEndpoint.java hosted with ❤ by GitHub

Now we modify the Listener to integrate everything so that the appropriate message can be received. Look at @KafkaListener annotation, we pass on the ListenerContainerFactory as an argument to receive a message from a queue.

	package in.springframework.blog.tutorials.listeners;
	import org.apache.logging.log4j.LogManager;
	import org.apache.logging.log4j.Logger;
	import org.springframework.kafka.annotation.KafkaListener;
	import org.springframework.kafka.support.KafkaHeaders;
	import org.springframework.messaging.handler.annotation.Header;
	import org.springframework.messaging.handler.annotation.Payload;
	import org.springframework.stereotype.Component;
	@Component
	public class KafkaMessageListener {
	Logger logger = LogManager.getLogger(KafkaMessageListener.class);
	@KafkaListener(topics = "FirstTopic", containerFactory = "firstTopicListenerContainerFactory")
	public void processFirstTopicMessage(@Payload String content,
	@Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition,
	@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
	logger.info(String.format("Received %s message for partition %d ", topic, partition) + content);
	}
	@KafkaListener(topics = "SecondTopic", containerFactory = "secondTopicListenerContainerFactory")
	public void processSecondTopicMessage(@Payload String content,
	@Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition,
	@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
	logger.info(String.format("Received %s message for partition %d ", topic, partition) + content);
	}
	}

view raw KafkaMessageListener.java hosted with ❤ by GitHub

Now we can test the server. The flow of the service is as below.

1. The message is posted to the endpoint as a POST request
2. The @RestController receives the message and based on the topicName in the request, it sends to message to the topic with the same name.
3. The listener receives the message from the appropriate queue.

$ curl -X POST \
>   'http://localhost:8081/send?token=3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
>   -H 'Content-Type: application/json' \
>   -H 'Postman-Token: 15fbe075-9c80-4af9-a797-6b5e0979fd1b' \
>   -H 'cache-control: no-cache' \
>   -H 'token: 3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
>   -d '{
> "message" : "This is my message!",
> "topicName" : "FirstTopic"
> }'
Success!

The message receipt is indicated in the server log.

2019-04-12 14:50:47.142  INFO 51396 --- [ntainer#0-0-C-1] i.s.b.t.listeners.KafkaMessageListener   : Received FirstTopic message for partition 0 This is my message!

Similarly, we can send a message to SecondTopic.

$ curl -X POST   'http://localhost:8?token=3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c'   -H 'Content-Type: application/json'   -H 'Postman-Token: 15fbe075-9c80-4af9-a797-6b5e0979fd1b'   -H 'cache-control: no-cache'   -H 'token: 3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c'   -d '{
"message" : "This is another message!",
"topicName" : "SecondTopic"
}'
Success!

The message receipt is indicated in the server log.

2019-04-12 14:53:14.972  INFO 51396 --- [ntainer#1-0-C-1] i.s.b.t.listeners.KafkaMessageListener   : Received SecondTopic message for partition 0 This is another message!

The complete code base for this tutorial can be found at my github repository at v1.4.

Kafka and Spring

Kafka has become a very popular platform and is being used as a stream, journal and even eventing system. In this post, we explore how to integrate Kafka with spring framework application. First, we add the Kafka bootstrap server details in the application.properties file.

	spring.kafka.bootstrap-servers=localhost:9092
	spring.kafka.consumer.group-id=ofs

view raw application-kafka.properties hosted with ❤ by GitHub

Let's also add dependencies in pom.xml.

	<dependency>
	<groupId>org.springframework.kafka</groupId>
	<artifactId>spring-kafka</artifactId>
	</dependency>
	<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	</dependency>

view raw pom-kafka.xml hosted with ❤ by GitHub

Now, for each Kafka topic, we create a listener class. The listener class provides a callback method that is called when any message is retrieved on that topic.

	package in.springframework.blog.tutorials.listeners;
	import in.springframework.blog.tutorials.Constants;
	import org.apache.logging.log4j.LogManager;
	import org.apache.logging.log4j.Logger;
	import org.springframework.beans.factory.annotation.Value;
	import org.springframework.kafka.annotation.KafkaListener;
	import org.springframework.stereotype.Component;
	@Component
	public class MyTopicKafkaListener {
	Logger logger = LogManager.getLogger(MyTopicKafkaListener.class);
	@KafkaListener(topics = Constants.MY_TOPIC)
	public void processMessage(String content) {
	logger.info("Received Promise message " + content);
	}
	}

view raw MyTopicKafkaListener.java hosted with ❤ by GitHub

Now we create an endpoint through which we inject a message in the queue. The message is sent to the queue and is retrieved by the listener.

	package in.springframework.blog.tutorials.endpoints;
	import in.springframework.blog.tutorials.Constants;
	import in.springframework.blog.tutorials.MyConstants;
	import in.springframework.blog.tutorials.pojos.MyTopicMessage;
	import org.springframework.beans.factory.annotation.Autowired;
	import org.springframework.http.MediaType;
	import org.springframework.kafka.core.KafkaTemplate;
	import org.springframework.security.access.prepost.PreAuthorize;
	import org.springframework.web.bind.annotation.RequestBody;
	import org.springframework.web.bind.annotation.RequestMapping;
	import org.springframework.web.bind.annotation.RequestMethod;
	import org.springframework.web.bind.annotation.RestController;
	@RestController
	@RequestMapping("/send")
	public class MyTopicMessageSenderEndpoint {
	@Autowired
	private KafkaTemplate kafkaTemplate;
	@RequestMapping(method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
	@PreAuthorize(MyConstants.ANNOTATION_ROLE_USER)
	public String createUser(@RequestBody MyTopicMessage message) {
	kafkaTemplate.send(Constants.MY_TOPIC, message.getMessage());
	return "Message sent successfully!.";
	}
	}

view raw MyTopicMessageSenderEndpoint.java hosted with ❤ by GitHub

We autowire a KafkaTemplate instance that is used to send the message to the queue.

$ curl -X POST \
>   'http://localhost:8081/send?token=3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
>   -H 'Content-Type: application/json' \
>   -H 'Postman-Token: e281e3c5-0dae-4bb7-ac8d-6555f66a18c6' \
>   -H 'cache-control: no-cache' \
>   -H 'token: 3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
>   -d '{
> "message" : "This is my message!"
> }'
Message sent successfully!.

The receipt of message is indicated in the spring server log.

2019-04-10 14:28:03.969  INFO 31091 --- [ntainer#0-0-C-1] i.s.b.t.listeners.MyTopicKafkaListener   : Received Promise message This is my message!

Spring Boot and Docker Containers

With microservices based deployment, the first step is to dockerize your software. In our case, we want to create docker images for each of our microservices so that we can orchestrate them better. I have decided to use container registry provided by Google to build and upload the images.

	<properties>
	<docker.image.prefix>rae</docker.image.prefix>
	</properties>
	<dependencyManagement>
	<dependencies>
	<!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-dependencies -->
	<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-dependencies</artifactId>
	<version>Greenwich.RELEASE</version>
	<type>pom</type>
	<scope>import</scope>
	</dependency>
	</dependencies>
	</dependencyManagement>
	<dependencies>
	<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-config-server</artifactId>
	</dependency>
	</dependencies>
	<build>
	<plugins>
	<plugin>
	<groupId>com.spotify</groupId>
	<artifactId>dockerfile-maven-plugin</artifactId>
	<version>1.4.10</version>
	<executions>
	<execution>
	<id>default</id>
	<goals>
	<goal>build</goal>
	<goal>push</goal>
	</goals>
	</execution>
	</executions>
	<configuration>
	<repository>gcr.io/${docker.gcp.project}/${docker.image.prefix}/${project.artifactId}</repository>
	<tag>${project.version}</tag>
	<buildArgs>
	<JAR_FILE>${project.build.finalName}.jar</JAR_FILE>
	</buildArgs>
	</configuration>
	</plugin>
	</plugins>
	</build>

view raw pom-docker.xml hosted with ❤ by GitHub

The first thing we need to do is to create a dependency to spring-cloud-dependencies pom. Then we add a dependency to spring-cloud-config-server. Then we add a dockerfile-maven-plugin. Now we need to keep in mind that in the configuration for dockerfile-maven-plugin we need to provide the repository. If can see that our repository starts with gcr.io/. This makes sure that the image after creation is pushed to the container registry hosted by Google. If you want to have some other registry, you need to provide it in the form hostname:portnumber.
Now we can issue the following command the docker image would be build and pushed to google registry.

$ mvn deploy -DskipTests
[INFO] 
[INFO] --- dockerfile-maven-plugin:1.4.10:push (default) @ customer ---
[INFO] Using Google application default credentials
[INFO] loaded credentials for user account with clientId=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com
[INFO] The push refers to repository [gcr.io/myproject/rae/customer]
[INFO] Image 967d96afcc46: Preparing
[INFO] Image 36e051842720: Preparing
[INFO] Image d1646aaa6540: Preparing
[INFO] Image 19382582b926: Preparing
[INFO] Image 41715d8d7d2b: Preparing
[INFO] Image f3a38968d075: Preparing
[INFO] Image a327787b3c73: Preparing
[INFO] Image 5bb0785f2eee: Preparing
[INFO] Image f3a38968d075: Waiting
[INFO] Image a327787b3c73: Waiting
[INFO] Image 5bb0785f2eee: Waiting
[INFO] Image 36e051842720: Layer already exists
[INFO] Image 41715d8d7d2b: Layer already exists
[INFO] Image d1646aaa6540: Layer already exists
[INFO] Image 19382582b926: Layer already exists
[INFO] Image 967d96afcc46: Pushing
[INFO] Image a327787b3c73: Layer already exists
[INFO] Image 5bb0785f2eee: Layer already exists
[INFO] Image f3a38968d075: Layer already exists
[INFO] Image 967d96afcc46: Pushed
[INFO] 0.0.1-SNAPSHOT: digest: sha256:f6bad4811f867dd75225797bee684ea43c0ddaf2b83de1b419a9f75e9a3941bc size: 2001
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47.638 s
[INFO] Finished at: 2019-03-10T19:58:27+05:30
[INFO] Final Memory: 55M/857M
[INFO] ------------------------------------------------------------------------

We can see below that the image is now pushed to google container registry.

$ docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
gcr.io/myproject/rae/customer   0.0.1-SNAPSHOT      a566e2f28705        19 seconds ago      518MB
gcr.io/myproject/rae/customer                       8c61d1a5aef4        13 minutes ago      518MB

12. Adding GIT release information

In the previous post, we saw how to enable actuator endpoints on our spring server. Once we have done that, it is a good idea to add GIT release information to the server in order to get the information related to the currently deployed server at runtime.
We add the git-commit-id-plugin to our pom.xml.

   <plugin>
    <groupid>pl.project13.maven</groupid>
    <artifactid>git-commit-id-plugin</artifactid>
    <version>2.2.1</version>
   </plugin>

The next things to do is to create a git.properties file in the resource directory of the project.

	# git.properties
	git.tags=${git.tags}
	git.branch=${git.branch}
	git.dirty=${git.dirty}
	git.remote.origin.url=${git.remote.origin.url}
	git.commit.id=${git.commit.id}
	git.commit.id.abbrev=${git.commit.id.abbrev}
	git.commit.id.describe=${git.commit.id.describe}
	git.commit.id.describe-short=${git.commit.id.describe-short}
	git.commit.user.name=${git.commit.user.name}
	git.commit.user.email=${git.commit.user.email}
	git.commit.message.full=${git.commit.message.full}
	git.commit.message.short=${git.commit.message.short}
	git.commit.time=${git.commit.time}
	git.closest.tag.name=${git.closest.tag.name}
	git.closest.tag.commit.count=${git.closest.tag.commit.count}
	git.build.user.name=${git.build.user.name}
	git.build.user.email=${git.build.user.email}
	git.build.time=${git.build.time}
	git.build.host=${git.build.host}
	git.build.version=${git.build.version}

view raw git.properties hosted with ❤ by GitHub

The git-commit-id is added by the maven plugin so it is a good idea to build the project using maven.

$ mvn clean package -DskipTests

Now we can run the server and check the /manage/info endpoint using curl command.

$ curl -X GET http://localhost:9091/manage/info
{"git":{"branch":"master","commit":{"id":"6fb94c0","time":1549868235.000000000}}