Wednesday, January 23, 2019

9. Role based data access

Many times we have a need to return data from an endpoint based on the role. Springframework provider easy mechanism for us to be able to do that. Let's take an example. In the previous example, we want to add a GET method in UserEndpoint that returns all the users. For any safe system, we want to return all the users if the role is ADMIN but if the role is USER then we want to return only that particular user. We don't want to write multiple methods for that purpose.
    @RequestMapping(method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
    @PostFilter("hasAuthority('ADMIN') or filterObject.authToken ==")
    public Iterable getUsers() {
        Iterable users = userRepository.findAll();
        return users;
The method is described above. As we can see it is an extremely simple method, it calls findAll method on the repository which will return all the valid user records and returns an Iterable collection back. The interesting aspect of this method is in the @PostFilter annotation. Let's try to understand the annotation. The first condition is hasAuthority('ADMIN'). It implies that if the authenticated role is ADMIN then return the records as it is. The next bit of the filter condition uses an object called filterObject. This is an automatically defined expression that we can use in @PostAuthorize filter. A complete list of all the expressions exists here. The expression filterObject is used for each element of a collection that is returned from the endpoint. Since we know that when this endpoint is called, we would only be authenticating using authToken, the name of authentication in security context is set to the token itself. We can verify this in the code in AuthenticationFilter class.
            else {

                Optional token = getOptionalHeader(httpRequest,"token");
                TokenPrincipal authTokenPrincipal = new TokenPrincipal(token);

The code fragment described above creates a TokenPrincipal object with the token as its name. That is the reason we have the condition filterObject.authToken == Taking the complete condition of @PostFilter we can see that the condition implies that return everything unconditionally if the role is ADMIN otherwise return the users with authToken as the currently authenticated user.
We have two users defined on the system. One has a role of the user and the other has a role of admin. Here are the examples of what happens when we call the endpoint with both of these users.
The first example is with a user with role USER.
$ curl -X GET "http://localhost:8081/user" -H "accept: application/json" -H "token: 3d47912d-73a0-4c4c-95e6-0486273d6221-28fa4f38-0f1b-4740-8e1d-3228288de631" | python -m json.tool
        "authToken": "3d47912d-73a0-4c4c-95e6-0486273d6221-28fa4f38-0f1b-4740-8e1d-3228288de631",
        "email": "",
        "expiry": 1548345909000,
        "fullname": "User",
        "id": 2,
        "mask": 1,
        "password": "User123",
        "username": "user"

The second example is with a user with role ADMIN.
$ curl -X GET "http://localhost:8081/user" -H "accept: application/json" -H "token: a137dd09-11e4-4dcf-a141-0b235d39a505-60d43bf4-3674-4248-be1d-c2669f14589f" | python -m json.tool
        "authToken": "3d47912d-73a0-4c4c-95e6-0486273d6221-28fa4f38-0f1b-4740-8e1d-3228288de631",
        "email": "",
        "expiry": 1548345909000,
        "fullname": "User",
        "id": 2,
        "mask": 1,
        "password": "User123",
        "username": "user"
        "authToken": "a137dd09-11e4-4dcf-a141-0b235d39a505-60d43bf4-3674-4248-be1d-c2669f14589f",
        "email": "",                                                                                         
        "expiry": 1548348263000,                                                                                                     
        "fullname": "Administrator",                                                                                                 
        "id": 3,                                                                                                                     
        "mask": 4,                                                                                                                   
        "password": "Admin123",                                                                                                      
        "username": "admin"                                                                                                          

As we can see above, the call with the role USER only returns the object related to that particular user while the call with the role returns all the users present in the system. This is how we can achieve role based object access without writing multiple endpoints.

Tuesday, January 22, 2019

8. That little matter of creating a user

Now that our simplified authentication system is in place, we are faced with the little matter of how to create a new user. Since we don't have a username, password, or token we can't really create a new user.
To accomplish that, we need to make some modifications to our AuthenticationFilter.  In the doFilter method, we add a special if block to take care of user creation.
            if (httpRequest.getRequestURI().toString().equals("/user") && httpRequest.getMethod().equals("POST")) {

                Optional username = getOptionalHeader(httpRequest,"username");
                UsernamePasswordPrincipal usernamePasswordPrincipal = new UsernamePasswordPrincipal(username, username, true);
            else if (httpRequest.getRequestURI().toString().equals("/authenticate") && httpRequest.getMethod().equals("POST")) {

                Optional username = getOptionalHeader(httpRequest,"username");
                Optional password = getOptionalHeader(httpRequest,"password");
                UsernamePasswordPrincipal usernamePasswordPrincipal = new UsernamePasswordPrincipal(username, password);

As we can see the code fragment, if the call is made to /user endpoint with POST method, we look for a username header and trigger spring authentication. We also need to make a change in the UsernamePasswordPrincipal to take a flag that would tell us if the user is a new user or existing user.
Now that we have modified the principal and filter, we need to handle this in the provider. The provider that gets invoked for username and password authentication is UsernamePasswordAuthenticationProvider.
As we can see in the provider's authenticate method, we have added an if block that checks if this is a new user. In case it is a new user, we authenticate this user with a role NEWUSER. The create user endpoint is only allowed to be called for a role NEWUSER.
Now that we have stitched the path for authentication of a new user to create user endpoint, we can see the endpoint itself.
    @RequestMapping(method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
    public Optional createUser(@RequestHeader(value="username") String username, @RequestBody User user) {
        User storedUser =;
        return Optional.of(storedUser);
As we can see, we have a @PreAuthorize added with NEWUSER role. We also set the role of the user to the USER. Before returning the response, we set the password to null.
This fixes our service to allow the creation of a new user. The complete code is available tagged as v1.1 for this and previous posts.

7. Enabling Swagger

Before we proceed any further, let's enable swagger on our service so that testing it becomes easy. The first step is to add springfox dependencies.

Now that we have dependencies added, we need to define a swagger configuration. Most of our endpoint will need to have a token passed as HTTP header, we need to configure our swagger configuration to facilitate that.

We create a swagger configuration file as defined above. The globalOperationParameters clause dictates to swagger that each endpoint will have a parameter named token of type header and is mandatory. We know there is two endpoint that will not have a token header but we can just pass some dummy values. We can also make it non-mandatory if we so wish.

We also modify authentication endpoint to take a username and password header for authentication endpoint. That is defined in the AuthenticateEndpoint as above. Look at the @RequestHeader parameter passed in the login method of the endpoint.

Also, make sure that that we had defined in one of the previous posts a list of swagger related URLs were made part of an exception list so that these do not go through authentication filter.
Now if we rebuild the server and run it and visit http://localhost:8081/swagger-ui.html we see the swagger page with all the API endpoints listed there. Here is the screenshot of the swagger UI as seen on visiting above URL.

Friday, January 4, 2019

6. Introducing Spring Security

If the web services that we are building require any level of authentication and authorization, it is better to understand Spring security context. Services that are properly built with security context can implement a better level of security at each method and endpoint level.
We want to handle all the security, authentication and, authorization related code in a single block so that managing it becomes easier.
The first step that we need to do is to add security dependencies in the pom.xml.
For authentication and authorization to make sense, we need to have a concept of a Role within the system. The roles are required to authorize the users for specific purposes. We define a set of roles in the form of a bitmask defined as an enum.

Now that we have defined Roles, we need to add few fields in our User class. We add a field mask for holding allowed roles and two different fields for authToken and expiry. We also add an index in the User entity.

We also need to add a class that implements GrantedAuthority interface. An authority in the spring security system is represented by a string and we use the Role enum name as the authority in the system.

We probably need to have two different types of authentication and authorization within the server. The first authentication and authorization will work with username and password and the second one will work with the token. Spring security requires two different entities to be defined for authentication and authorization. We need a Principal and a Provider class each for username and token authentication and authorization. Let's first look at the class required for authentication using username and password.  A principal is nothing but the abstraction of credentials that is flowing through spring security. We define it as below.

A provider is a class that provides the authentication and authorization functionality for a particular type. The username and password provider is defined as below. As we can see, the provider class extracts the credentials from the Principal and then makes sure the password is correct for the username. It also populates the list of roles granted to the user in the form of Authority.

Similarly, we define a Token principal. This principal just contains the token value that is received from the user.

We also define a token provider. The token provider looks up the user from the database by querying it from token and authenticates the user. In this example, we are making an implicit assumption that a token will be unique across all the users. If that were not to be the case, we will also have to pass the username or some other identifier for the user along with the token during authentication.

We also define a class RequestContext that will contain a set of thread local variables so that once authenticated, we will not need to look up the user details from the database. We will only need to hit the database in cases where we need to update the record in the User table.
We can see in both provider classes, after successful authentication of the user, the User is set into the thread local so that we can use it within the thread boundaries of that request.
Now the providers and principals are in place, we need to start using them. Rather than using them on a case to case basis, the best way is to define a filter which applies on each request and authentication and authorization can be done on all requests.

AuthenticationFilter provides the doFilter method which is called for each incoming request. As we can see within the doFilter method, if we receive a POST request to /authenticate URL we assume that to be a username and password authentication and we process it as a username and password authentication, in all other cases, it is assumed to be a token-based authentication. All the three parameters, username, password, and token are expected to be passed as headers.
Now we need to plug everything in so that it is called by the system. For that, we need to define a security configuration.
The SecurityConfiguration class is defined by extending WebSecurityConfigurerAdapter class. We override a few sets of methods. The first method defines a set of authentication providers that are available to the system.
    protected void configure(AuthenticationManagerBuilder auth) throws Exception {

The second method defines a set of URLs that would be ignored by the security system.
    public void configure(WebSecurity web) throws Exception {

The third method defines the security rules for all the requests.
    protected void configure(HttpSecurity http) throws Exception {

        http.addFilterBefore(new AuthenticationFilter(authenticationManager()), BasicAuthenticationFilter.class);

The configuration first disables CSRF and then sets a session management policy. After that, it declares that all the actuator endpoints need to be authenticated by a user having the ADMIN roles. Then all other users need to be authenticated. With this, every request coming into the system will be automatically authenticated and authorized. Now we can annotate our endpoints with specific roles so that unauthorized users can not make the calls. For example, we add the following annotation to our endpoint that we had defined earlier.

This annotation implies that a user with the role of USER can only call this endpoint. Any other user would not be able to call this endpoint.

This is also our AuthenticationEndpoint. This will need to be called when a user wants to login using username and password. This now provides us with a functioning service with spring security enabled. In future posts, we will provide more details on it.  The complete code for this post is available in github repository with version 1.0. Click here to download it. 

Wednesday, December 19, 2018

5. Replacing tomcat with jetty

By default when we run spring boot application, it runs a tomcat server. I have personally found tomcat too heavyweight and I prefer to use Jetty for writing a webservices backend.
2018-12-19 10:54:08.403  INFO 16029 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)

2018-12-19 10:54:08.424  INFO 16029 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]

2018-12-19 10:54:08.424  INFO 16029 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/9.0.12

To replace tomcat with Jetty, we need to make a couple of changes.  The first step is to add Jetty dependencies in the pom.xml.

Since springframework boot bom automatically includes a dependency on tomcat in spring-boot-starter-web, we specifically need to exclude it. So we change


The next step is to add a customizer for jetty which will allow us to set some parameters related to Jetty.
As we can see the customizer uses a set of variables that we can override in file to modify the behavior of the server.

2018-12-19 11:06:44.577  INFO 16219 --- [           main] o.e.jetty.server.AbstractConnector       : Started ServerConnector@42ebece0{HTTP/1.1,[http/1.1]}{}
2018-12-19 11:06:44.580  INFO 16219 --- [           main] o.s.b.web.embedded.jetty.JettyWebServer  : Jetty started on port(s) 8081 (http/1.1) with context path ''
Now the logs clearly show that in place of tomcat, it is running jetty.

Wednesday, November 28, 2018

4. More on CrudRepository

In the previous post, we used the CrudRepository interface. We only used the methods already provided by the interface. In this post, we look at how we can extend the interface with custom queries. Let's go back to our UserEndpoint. In the GET method, we can currently get the details of a user given his id. This is an impractical scenario. Most users' would not know what their id is. It is an internal identifier generated by our service and it makes no sense for our users.
Let's say we want to extend the existing GET/DELETE interface to allow query by id, username, and email address. Since we intend to query the table using the username and email columns, we need some kind of index on those columns. Since these fields are expected to be unique, we put a unique constraint in the @Table annotation.

When this change is deployed, the corresponding changes in database schema would be as below.
mysql> desc user;
| Field    | Type         | Null | Key | Default | Extra |
| id       | bigint(20)   | NO   | PRI | NULL    |       |
| email    | varchar(255) | YES  | UNI | NULL    |       |
| fullname | varchar(255) | YES  |     | NULL    |       |
| password | varchar(255) | YES  |     | NULL    |       |
| username | varchar(255) | YES  | UNI | NULL    |       |
5 rows in set (0.01 sec)
mysql> show create table user\G
*************************** 1. row ***************************
       Table: user
Create Table: CREATE TABLE `user` (
  `id` bigint(20) NOT NULL,
  `email` varchar(255) DEFAULT NULL,
  `fullname` varchar(255) DEFAULT NULL,
  `password` varchar(255) DEFAULT NULL,
  `username` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uq_email` (`email`),
  UNIQUE KEY `uq_username` (`username`)
1 row in set (0.00 sec)

As we can see from the schema both the columns have unique constraints attached to them. The next step is to add methods in the to query the user for a given username or email.

Now that we have all the required changes in place, we modify the UserEndpoint to handle the changes. We first create a private method retrieveUser which will retrieve a user from repository given an id, username or email.

In this method, we take an argument of type String, we first assume it to be the id if id fails we try with email and then with the username.
$ curl -X GET http://localhost:8080/user/
{"id":6,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":""}
Here is the complete UserEndpoint class after modification.

Tuesday, November 27, 2018

3. All the methods in endpoint

In the previous post, we defined a single method for an endpoint. In a typical web application, each endpoint will have multiple methods. To build an example of a complete endpoint, we need to have a persistence layer that can support these endpoints. To enable MySQL support for spring framework, we need to update pom.xml file with following dependencies.

Now we need to define data sources for MySQL. The first step is to create a database and user in the MySQL database. To do that log in to MySQL using the following command.
$ mysql -u root
Now run following set of commands to create a database named tutorial with username tutorial and password tutorial123.

Now we can initialize a data source in spring framework by putting properties in the file.

As we can see in the properties, the value ddl-auto is set to update. This will cause the database schema to be updated based on the definition of entity objects. This is a good practice while developing the system but normally should not be used while in production.
Let's say we want to define a simple endpoint that supports an entity called User. We define the entity in Java as below.

As we can see, the key to an entity class is the annotation @Entity.  You can also add an optional annotation @Table which will allow you control over the name of the table that gets created within the database. By default, if you omit the @Table annotation, the table name would be automatically generated.
The next step is to generate a Repository interface for accessing the entity from the database. We typically create a subclass of CrudRepository.

An empty repository interface provides sufficient functionality for this example. We will see how to add more functionality to repository class in a  different post. The Javadoc of CrudRepository provides details of all the methods that are readily available without any addition to the interface.
Let's run the server to see whether the server can connect to the database. As we run the server, we check what happened to our database.
$ mysql -u tutorial -ptutorial123 tutorial
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 8.0.12 Homebrew

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables;
| Tables_in_tutorial |
| hibernate_sequence |
| user               |
2 rows in set (0.01 sec)

mysql> desc user;
| Field    | Type         | Null | Key | Default | Extra |
| id       | bigint(20)   | NO   | PRI | NULL    |       |
| email    | varchar(255) | YES  |     | NULL    |       |
| fullname | varchar(255) | YES  |     | NULL    |       |
| password | varchar(255) | YES  |     | NULL    |       |
| username | varchar(255) | YES  |     | NULL    |       |
5 rows in set (0.01 sec)


The database has the user table created mapping our entity class to a table. Now we have confirmed that the server is able to connect to our database server. Now we go back to the old endpoint class that we created in the previous post and create a similar endpoint class called UserEndpoint. In this class, we will define all the methods required for managing users.

$ curl -X POST \
>   http://localhost:8080/user \
>   -H 'Content-Type: application/json' \
>   -H 'Postman-Token: d0298724-86d2-49da-a0c0-067ed9e18e1e' \
>   -H 'cache-control: no-cache' \
>   -d '{
> "fullname":"John Doe",
> "username":"jdoe",
> "password":"JohnDoe123",
> "email":""
> }'
{"id":1,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":""}
We can also verify the record in the database.
$ mysql -u tutorial -ptutorial123 tutorial -e "select * from user"
mysql: [Warning] Using a password on the command line interface can be insecure.
| id | email               | fullname | password   | username |
|  1 | | John Doe | JohnDoe123 | jdoe     |
Now we can perform other operations on User record.
$ curl -X GET http://localhost:8080/user/1
{"id":1,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":""}

curl -X DELETE http://localhost:8080/user/1
{"id":1,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":""}

Now we can run the database query again and see that the record has vanished.
mysql> select * from user;
Empty set (0.00 sec)

Now we create the user again by running the POST command. Since we are using database autoincrement, the id of the user will be 2 now.
Now we change the email of the user using PATCH command.
curl -X PATCH   http://localhost:8080/user/2   -H 'Content-Type: application/json'   -H 'Postman-Token: d0298724-86d2-49da-a0c0-067ed9e18e1e'   -H 'cache-control: no-cache'   -d '{
{"id":2,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":""}

Here we have it. All the endpoints are functioning properly backed by MySQL as the database.

Tuesday, November 20, 2018

2. Adding an Endpoint

One of the primary reasons why one would be using spring framework would be to define rest based web services.  In this post, we will see how do we add an endpoint to our server. First, download and install your favorite IDE. I prefer IntelliJ IDEA but you can pick the IDE that you are comfortable with.
Let's open the project that we created in the previous blog post in the IDE. The source root for the project is <Project Name>/src/main/java. Once you open that directory, you will see the package of the project that you created. Within that, you will see the main for the spring application. The only thing that we change for now is to add scanBasePackages parameter in @SpringBootApplication annotation.

For now, we can leave the main as it is. The next step that we want to do is to add an endpoint. Create a new java class in your project. I call it HelloWorldEndpoint. To define an endpoint, we need to do the following things. The endpoint needs to have a path relative to the server root. We also need to know what operations are needed to be defined.
We will discuss the whole details of spring security later. So for now please remove security dependencies from your pom.xml. Following lines should be removed from pom.xml

It helps to look at a REST endpoint as an object with operations used to define their lifestyle state changes.

  • Path defines a class
  • Method GET is for reading the state of that object
  • Method POST is for creating a new object of that class
  • Method DELETE is for deleting an instance of that class
  • Method PUT is for updating the object
  • Method PATCH is for selectively updating the values.
Looking at our example, we are going to create a HelloWorld endpoint which will be used to manage the state of class Hello. Here is what class Hello looks like.

Now we look at our endpoint class. The class contains a single method GET which returns a temporarily created instance of Hello class.

Now we build the project.
$mvn clean package -DskipTests

And we can run the project.
$mvn spring-boot:run

Since maven build creates a single jar for each project. We can also run the project with the java command.
$java -jar target/tutorials-0.0.1-SNAPSHOT.jar

Most IDEs would allow you to build and run the project from within the IDE. You can do that as well. Now that the server is running we can test the endpoint that we had written. You can use tools like Postman but since it is a simple endpoint, we can just test it with curl command.
curl -X GET http://localhost:8080/hello

As we can see the JSON that is returned corresponds to the temporary object that we created and returned. Of course, this is too simplistic and we will look at doing something more substantial in the next blog post. 

Monday, November 19, 2018

1. Starting a springframework project

The best way to start a spring framework project is to use the spring initializer to do that. Visit Spring Initializer website. Provider values for group and artifact and select the dependencies that you want to initialize with. To start with you can start with minimum dependencies and then add them later on. These are just maven dependencies.

Here I have selected Web and Security to start with. Now we click on Generate Project. It downloads a zip file which is the starting project. We unzip the file and then we run following command.
$ mvn clean package -DskipTests

 It will download the dependencies and then compile the project. The finally built jar file is in the tutorials/target directory and is named tutorials-0.0.1-SNAPSHOT.jar. The name is created using the version number defined in the pom.xml file generated. We can run the project by running following command.
$ java -jar target/tutorials-0.0.1-SNAPSHOT.jar

It will run the project and the following output is displayed. The last line tells us that the project launch is successful.

So here it is. Our first springframework project.