tag:blogger.com,1999:blog-26714482012123102312024-02-19T17:39:03.798+05:30Learnings in SpringframeworkHere I collate my learnings of using spring framework over the years.vavasthihttp://www.blogger.com/profile/17794185844812005239noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-2671448201212310231.post-82394821919339066862022-08-22T19:21:00.007+05:302022-08-22T19:21:58.087+05:30Story of a funny linked list<p> Recently I was presented with this problem, for a moment I was completely blanked on how to solve this problem. I had to think for some time and finally found a solution. Let's first look at the problem.</p><p><br /></p><blockquote><p>Let's assume we have a simple singly linked list. The only modification is that each node of the linked list also has another link, let's call it <i>other</i>. For some of the elements of the list, the link <i>other</i> contains a pointer to some other element in the list. Now, we have to write a clone function that makes a deep copy of the list.</p></blockquote>
<script src="https://gist.github.com/vavasthi/c2f83c888f176fefce7d238f6c1c7963.js"></script>
<p>The salient method is the <i>clone</i> method in the class. The challenge is that <i>other</i> pointer can come in any other. A node earlier in the list can point to a node later in the list and vice versa. So we have to basically run two passes on the list.</p><p>In the first pass, we make a regular copy of the list and create two hash-maps. The first hash-map contains <i>other</i> pointer as the key and the old and new node as the value pair. The second hash map contain a mapping of each node and its corresponding new map. </p><p>Once we complete the simple copy of the list, we traverse through the first map and find all corresponding new nodes for corresponding old nodes to which the other pointer is pointing to and then we retrofit the list.</p><p>So that's how I was able to solve the problem of funny linked list.</p>vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-24741012652660817442020-08-25T12:03:00.006+05:302020-08-25T12:06:39.537+05:30Database Schema Version Control<p> Most of the times when we have a service in production, we might make changes to the database schema. We want to make sure of two things here. </p><p></p><ol style="text-align: left;"><li>Database changes are version controlled</li><li>Changes to actual database are done in a controlled fashion. </li></ol><div>I am not very comfortable with some CI/CD job going on its own and making changes to database schema. Deployments that need schema changes are done as part of planned maintenance cycles and need to be monitored by actual humans.</div><div><br /></div><div>So, what do I expect from a solution that takes care of schema versioning.</div><div><br /></div><div><ul style="text-align: left;"><li>The service should not be allowed to start if the schema is not in sync with the in-memory entities and code that is designed to use them</li><li>An easy mechanism should be provided for migration of schema version. This is generally done through different phases of development like dev/stage/test and then to production., We want to automate it in the sense that there should be scripts that are tested and validated through previous phases of development lifecycle but still has an option of actual user triggering it in production.</li></ul><div>With above two goals, let's try to define a database schema versioning system. I am going to use <a href="https://flywaydb.org/">flywaydb</a> as the primary tool that does it with configurations that make it useful for my usecase. First let's define a configuration file that takes defines database connectivity option for my database.</div></div><div><script src="https://gist.github.com/vavasthi/58fe7b1f5bb7e92acffaa5f362581a3e.js"></script><br /></div><div>You would remember since ours is a multi-tenant application, we have three databases and we have define configurations for all the three databases.</div><div><script src="https://gist.github.com/vavasthi/9b988339198dc801228a515ea74c6d97.js"></script><br /></div><div><script src="https://gist.github.com/vavasthi/3a48a39a28d2a738652120762a6063aa.js"></script><br /></div><div><br /></div><div>Here we have all the three database configurations defined. Now let's make changes to pom.xml for dependencies.<br /></div><div><script src="https://gist.github.com/vavasthi/9ea2d35f857503c40b09ac904cafb0ee.js"></script><br /></div><div>If you are just creating your application, then your life is easy, just create the initial <i>sql file</i> and add it to your <i>src/resources/db/migration</i> directory. Be sure to name it something like V1.0__Initial_Schema.sql. You need to make sure the first character is V, U, or R for version, undo, repeatable respectively. After that you have a version number, you can follow any version number strategy but it should be in the form of the dotted notation of <i>ver.major.minor</i>. For more details on this, please <a href="https://flywaydb.org/documentation/migrations#sql-based-migrations">read here</a> on how this can be effectively used. What we need to understand is that flyway will consider a version 1.1 to be a version later than version 1.0.<br /></div><div>Now that we have the existing SQL stored in the file, we can baseline our current schema so that any future changes could be incorporated using this mechanism.</div><div><pre class="code">$ mvn flyway:baseline -Dflyway.configFiles=./tutorials/config/default.config</pre><pre class="code">[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< in.springframework.blog:tutorials >------------------
[INFO] Building tutorials 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:6.5.5:baseline (default-cli) @ tutorials ---
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[INFO] Flyway Community Edition 6.5.5 by Redgate
[INFO] Database: jdbc:mysql://localhost:3306/tutorial (MySQL 8.0)
[INFO] Creating Schema History table `tutorial`.`flyway_schema_history` with baseline ...
[INFO] Successfully baselined schema with version: 1.0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.285 s
[INFO] Finished at: 2020-08-25T10:14:53+05:30
[INFO] ------------------------------------------------------------------------
</pre><br />Make sure the pathname of the configuration file is appropriate for your scenario. Also like in my case, since I have multiple databases due to multi-tenancy, I will need to do this for each of the databases. Once done, I can check the status of schema using the <i>info</i> command.</div><div><pre class="code">$ mvn flyway:info -Dflyway.configFiles=./tutorials/config/default.config
<br /></pre><pre class="code">[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< in.springframework.blog:tutorials >------------------
[INFO] Building tutorials 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:6.5.5:info (default-cli) @ tutorials ---
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[INFO] Flyway Community Edition 6.5.5 by Redgate
[INFO] Database: jdbc:mysql://localhost:3306/tutorial (MySQL 8.0)
[INFO] Schema version: 1.0
[INFO]
[INFO] +----------+---------+----------------+----------+---------------------+----------+
| Category | Version | Description | Type | Installed On | State |
+----------+---------+----------------+----------+---------------------+----------+
| | 1.0 | Base Migration | BASELINE | 2020-08-25 10:13:12 | Baseline |
+----------+---------+----------------+----------+---------------------+----------+
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.247 s
[INFO] Finished at: 2020-08-25T10:13:26+05:30
[INFO] ------------------------------------------------------------------------
</pre>Now our existing database is baselined and we can start using the flyway versioning mechanism. We still have one issue to resolve, since we want to make sure our application doesn't start till the time we have appropriate code backing appropriate schema. The first step is to disable any kind of migration being started by our application. So we define a null migration strategy that will not do anything.</div><div><script src="https://gist.github.com/vavasthi/657b411694abd79d92f93628a1f49bdf.js"></script><br /></div><div>This will ensure that no automatic migration is executed. We still want to make sure that the application exits if the schema can't be validated. For this we need to go to the method where we are creating the datasource. We add the validation of the datasource there.</div><div><br /></div><div>In our example, since we are using Multi-tenant datasource, we add this validation there. </div><div><script src="https://gist.github.com/vavasthi/2b362659b4aa38c70690b6ddb1c43f7c.js"></script><br /></div><div>Look at <i>validateDatasource</i> method, it takes a dataSource as input and raises a SpringApplication exit if the database can't be validated. Now, let's test it. We will take our entity <i>Profile</i> and add a new field in, let's call it <i>maritalStatus</i>. </div><div><br /></div><div>We now create a <i>sql</i> file that alters the table.</div><div><script src="https://gist.github.com/vavasthi/4158c85c43161256119d95f0071f3f0b.js"></script><br /></div><div><br /></div><div>Now if we try to run this application, let's see what happens.</div><div> </div><div><pre class="code">2020-08-25 11:53:54.652 ERROR 40041 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Unsatisfied dependency expressed through method 'flywayInitializer' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.flywaydb.core.Flyway]: Factory method 'flyway' threw exception; nested exception is org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'dataSource': Could not bind properties to 'DataSource' : prefix=spring.datasource, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@ba2f4ec has been closed already
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
</pre><br /></div><div>Now let's run the migration for this sql.</div><div><pre class="code">$ mvn flyway:migrate -Dflyway.configFiles=./tutorials/config/default.config</pre><pre class="code">
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< in.springframework.blog:tutorials >------------------
[INFO] Building tutorials 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:6.5.5:migrate (default-cli) @ tutorials ---
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[INFO] Flyway Community Edition 6.5.5 by Redgate
[INFO] Database: jdbc:mysql://localhost:3306/tutorial (MySQL 8.0)
[INFO] Successfully validated 3 migrations (execution time 00:00.017s)
[INFO] Current version of schema `tutorial`: 1.1
[INFO] Migrating schema `tutorial` to version 1.2 - Add Marital Status
[INFO] Successfully applied 1 migration to schema `tutorial` (execution time 00:00.075s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.323 s
[INFO] Finished at: 2020-08-25T11:56:25+05:30
[INFO] ------------------------------------------------------------------------
<br /></pre><pre class="code">$ mvn flyway:info -Dflyway.configFiles=./tutorials/config/default.config
<br /></pre><pre class="code">[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< in.springframework.blog:tutorials >------------------
[INFO] Building tutorials 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:6.5.5:info (default-cli) @ tutorials ---
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[INFO] Flyway Community Edition 6.5.5 by Redgate
[INFO] Database: jdbc:mysql://localhost:3306/tutorial (MySQL 8.0)
[INFO] Schema version: 1.2
[INFO]
[INFO] +-----------+---------+---------------------------+----------+---------------------+----------+
| Category | Version | Description | Type | Installed On | State |
+-----------+---------+---------------------------+----------+---------------------+----------+
| | 1.0 | Base Migration | BASELINE | 2020-08-25 10:15:00 | Baseline |
| Versioned | 1.1 | Remove AuthToken FromUser | SQL | 2020-08-25 10:49:27 | Success |
| Versioned | 1.2 | Add Marital Status | SQL | 2020-08-25 11:56:25 | Success |
+-----------+---------+---------------------------+----------+---------------------+----------+
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.226 s
[INFO] Finished at: 2020-08-25T11:56:30+05:30
[INFO] ------------------------------------------------------------------------
</pre>As we can see now, the database schema is migrated. So if we make sure all the database migration SQL files are properly made part of the project, we can eliminate schema mismatch issues to a large extent. We also have an option to make it fully automated or a single command migration with manual verification.</div><div>The hibernate ORMs are just in-memory representations of actual database schema and the schema is owned by the database, ORM changes have to be done by the programmers.</div><div><br /></div><div>The complete code supporting this tutorial is available at my <a href="https://github.com/vavasthi/springblog/tree/v1.8">spring blog</a> tagged as v1.8 </div><p></p>vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-20518839282242659962020-07-29T15:20:00.000+05:302020-07-29T15:20:13.158+05:30Publishing Application Events using Kafka and Spring Transaction EventsMany applications have a need to publish application level events based on some operations happening in the system. Here is one requirement that I had recently. Basically in my tutorial system, if there is new user created, I need to publish a UserCreated event. Here is how we can accomplish this using Transaction Events provided by Springframework.<div>First I create a event pojo that will be used to publish the event.</div><div>
<script src="https://gist.github.com/vavasthi/c64c1f61bc3d2b0ba0bfa9f4d90cc7bb.js"></script><br /></div><div>Now we create an event listener.</div><div><script src="https://gist.github.com/vavasthi/1956fcc2eeb356595201dae267e9cdec.js"></script><br /></div><div>As you can see within the event listener, I have a handler method that is a hook to a particular phase in transaction commit cycle. At different phases of commit cycle, we are publishing an event to Kafka topic 'FirstTopic'</div><div><br /></div><div>Now we create an event publisher.</div><div><script src="https://gist.github.com/vavasthi/d4794a3fa1d8abc46513b5b443f8bde8.js"></script>The event publisher just constructs the event object and publishes the event.</div><div>We add a <i>createUser</i> method within the user service and mark it <i>@Transactional</i>. This is important to make sure any exceptions coming out of <i>createUser</i> transaction are caught and propagated appropriately.</div><div><script src="https://gist.github.com/vavasthi/c98bf68f36aa65ee6c867f184c5c0c52.js"></script><br /></div><div> </div><div>Now within the user registration, we add a publish within a transaction. Please be careful that the code block that is publishing the event is within a transaction otherwise these don't perform any operations. </div><div><script src="https://gist.github.com/vavasthi/ec4a385e78f2e03551694ce28c819c51.js"></script>Look at the <i>createUser</i> method that is implementing <i>/user</i> request mapping and method <i>POST</i>. The penultimate line in the method is publishing the event. Now let's test this code.</div><div><br /></div><div>Just to test the code, we also run a <i>kafka-console-consumer</i> listening to the topic to which our code is publishing the message.</div><div>First we try to create a user that is already existing.</div><div><br />
<pre class="code">$ curl --request POST --url http://localhost:8081/tenant1/registration/user \
--header 'authorization: Basic c3VwZXJzZWNyZXRjbGllbnQ6c3VwZXJzZWNyZXRjbGllbnQxMjM=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 71180383-2b1f-482e-2fcc-2a23d045b205' \
--header 'x-tenant: tenant1' \
--data '{"username": "admin8","password": "admin1234","audience" : "self"}' | python -m json.tool
{
"error": "Conflict",
"message": "",
"path": "/tenant1/registration/user",
"status": 409,
"timestamp": 1596015557641
}
</pre><pre class="code"><br /></pre>Now we try to create a user that doesn't exist.<pre class="code">$ curl --request POST --url http://localhost:8081/tenant1/registration/user \
--header 'authorization: Basic c3VwZXJzZWNyZXRjbGllbnQ6c3VwZXJzZWNyZXRjbGllbnQxMjM='\
--header 'cache-control: no-cache' \
--header 'content-type: application/json'\
--header 'postman-token: 71180383-2b1f-482e-2fcc-2a23d045b205' \
--header 'x-tenant: tenant1' \
--data '{"username": "admin9","password": "admin1234","audience" : "self"}' | python -m json.tool
{
"createdAt": 1596015670301,
"createdBy": "UnAuthenticated",
"id": 21,
"mask": 1,
"tenantId": 1,
"updatedAt": 1596015670301,
"updatedBy": "UnAuthenticated",
"username": "admin9"
}
</pre><div>We can also check our Kafka consumer that we ran earlier.</div><div><pre class="code">$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic FirstTopic
UserCreatedEvent(userId=20, username=admin8, fullname=null, status=BeforeCommit)
UserCreatedEvent(userId=20, username=admin8, fullname=null, status=AfterRollback)
UserCreatedEvent(userId=20, username=admin8, fullname=null, status=AfterCompletion)
UserCreatedEvent(userId=21, username=admin9, fullname=null, status=BeforeCommit)
UserCreatedEvent(userId=21, username=admin9, fullname=null, status=AfterCommit)
UserCreatedEvent(userId=21, username=admin9, fullname=null, status=AfterCompletion)
</pre></div><div>The first three events are for the failed transaction which rolled back and the last three are for the successful transaction.<br /></div></div>vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-65717676835521111402020-06-24T00:24:00.000+05:302020-06-24T00:24:19.373+05:30Building a multitenant serviceIf you build any production service that is valuable, very soon you find a customer who wants to use it but wants it white-labeled for him. You also very quickly find out that there is a need to data isolation, i.e. different customers want their data to be kept in their own databases. Because of these reasons, it is always a good idea to design a service keeping multi-tenancy in mind.<div>In this tutorial we will look at some of the important features required for a multi-tenant service and how do we leverage <i>springframework</i> to deliver a service that is truly enabled for a modern multi-tenant service.</div><div>Here are couple of the important aspects of a multi-tenant service.</div><div><ol style="text-align: left;"><li>Everybody gets their own endpoints</li><li>Everybody can be given their own databases</li></ol><div>Let's design how our endpoint URLs would look like keeping tenancy in mind. We add a discriminator in the URL that identified a tenant. For example our OAuth URLs would become something like below.</div></div><div><pre class="code">http://example.com/tenant1/oauth/token
http://example.com/tenant1/oauth/check_token<br /></pre></div><div>To accomplish, we define our <i>RequestMapping</i> with an embedded tenant variable that becomes part of each of the URLs.</div><div><div><br /></div><div><pre class="code"> public static final String BASE_ENDPOINT = "/{tenant}";
public static final String USER_ENDPOINT = BASE_ENDPOINT + "/user";
public static final String REGISTRATION_ENDPOINT = BASE_ENDPOINT + "/registration";</pre></div><div><br /></div><div><br /></div><div><br /></div><div>As we can see, the first part of the URL defines the tenant. It is not mandatory but I also design the service so that a header is expected that also defines the tenant. This is just to guard against some client mistakenly calling a wrong tenant because the same tenant has to be added in two places. We will use following header.</div><div><div><pre class="code">X-tenant:tenant1<br /></pre></div></div><div>Every incoming request into the system needs to know the tenant and the authenticated user that is part of the request. We define a sequence of filters for this purpose. Since these filters need to be called in a particular order, we define the order as follow.</div><div><br /></div><div><div><pre class="code">public static final int TENANT_HEADER_PRECEDENCE = Ordered.HIGHEST_PRECEDENCE;
public static final int SEED_DATA_PRECEDENCE = TENANT_HEADER_PRECEDENCE - 1;
public static final int TENANT_PRECEDENCE = SEED_DATA_PRECEDENCE - 1;
public static final int USER_PRECEDENCE = Ordered.LOWEST_PRECEDENCE;</pre><br />
As you can see we are going to define four filters which will perform specific functions.</div></div><div><ol style="text-align: left;"><li>TenantHeader filter will extract tenant header from the incoming request, match the URL piece with the header and set it in a ThreadLocal variable.</li><li>SeedData filter is only require to create some seed data to make the service usable. We need a default tenant in the system so that we can start making some requests. This doesn't do anything most of the time.</li><li>Tenant filter will extract the tenant object and set it into another ThreadLocal variable.</li><li>User filter is the last in the precedence and will extract the current authenticated user and will store it into a ThreadLocal.</li></ol><div>We are using following TutorialRequestContext class to store these thread local variables.</div></div><div><script src="https://gist.github.com/vavasthi/4ac810c8f85f33fe624fbe58a9fe5c4b.js"></script><br /></div><div>Now let's look at these filters one by one.</div><div><script src="https://gist.github.com/vavasthi/7527dc98a7aa1e69e7c999a47d272e88.js"></script><br />This filter just extracts the X-tenant header stores in the thread local.</div><div><script src="https://gist.github.com/vavasthi/27ddb8cc9bbce40bd791103f781b4973.js"></script><br />This filter checks that there should be atleast one tenant in the system, if not found, it inserts a default tenant. This is required so that we can use rest calls.</div><div><script src="https://gist.github.com/vavasthi/24d969c989a7f46a6e28b7dc65229672.js"></script><br />This filter extract the complete Tenant object from the database and stores it into the thread local.</div><div><script src="https://gist.github.com/vavasthi/89aeb1f8aff448eb5303aa3bcc7d5856.js"></script><br />This filter extract currently authenticate user and populates it in the thread local.</div><div>In the last tutorial, we used the spring provided default <i>ClientDetailsService</i>, we create our own service in this tutorial to make sure we can have a schema that we like. To do that we need a entity, <i>TutorialClientDetails</i> and a service <i>TutorialClientDetailsService</i>.<br /><script src="https://gist.github.com/vavasthi/b42e22a83f54e596288fef6f493387c8.js"></script><br />This entity just implements the <i>ClientDetails</i> interface</div></div><div><script src="https://gist.github.com/vavasthi/0aee7732ab67ea7dda5d5e34706b9254.js"></script><br />This service needs to implement a method <i>loadClientByClientId</i>.</div><div><br /></div><div>Now that all the foundation work is in place, we get down to making our service multitenant. The first things that we need to do is to remove all the old dataSources that we had defined. We will now define a routing data source that will choose the right data source based on the tenant that we are taking to. This is where our tenant discriminator thread local will be usedful. Take a look at the following class.</div><div><script src="https://gist.github.com/vavasthi/40241d4019343cbe4bdea2633d6cbe51.js"></script><br />Here we implement a method <i>determineCurrentLookupKey</i> which uses thread local to identify the current tenant and returns the key.</div><div><script src="https://gist.github.com/vavasthi/944b86cc1456898f8de776d48d0e5205.js"></script><br />The bulk of smartness lies in the class <i>TutorialMultitenantConfig,</i> specifically the dataSource bean that returns the default dataSource for the system. What we are assuming that within our resource directory, we will have a tenants subdirectory and within that we will have one properties file per tenant. The property file will look like below.</div><div> </div><div><script src="https://gist.github.com/vavasthi/4a6db8d6aa4491926dd38d067bc29b55.js"></script><br />Here we have added usual spring data source properties alongwith a name property which will be the name of the tenant. This name will be matched with the name in the URL and the header.</div><div>In the class <i>TutorialMultitenantConfig,</i> look at the definition of variable <i>TENANT_PROPERTIES_RESOURCE</i>, this basically looks up for all the <i>*.properties</i> file in the tenant directory. Lines 61 through 74, create a data source object for each of the tenants and store these in a map with key being the tenant name. We remember <i>determineCurrentLookupKey</i> method which returned the name of current tenant, that return value is used to fetch appropriate data source object for the request being processed. Line 88 defines a default data source that is used if there is no data source present for the given tenant.</div><div><br /></div><div>Now our multi tenanted service is ready and it is time to test. The first thing that we need to do is create couple of more databases with exact same schema as the first database. Please keep in mind that this has to be done manually even if you have defined ddl-auto property. Just take a mysqldump of the first database and import in two other databases.</div><div>We also need to make sure that the system atleast has one tenant defined in each of the databases. This is required in order to make sure we are able to use the rest calls. The best approach is to have a default tenant and then take a dump so that default tenant is also copied in each of the databases. We have created an admin user which has ADMIN role. We call the user endpoint that will return the list of all the users.</div><div><pre class="code">$ curl --request GET \
> --url http://localhost:8081/tenant1/user \
> --header 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI5Mzc1MzksInVzZXJfbmFtZSI6ImFkbWluIiwiYXV0aG9yaXRpZXMiOlsiVVNFUiIsIkFETUlOIl0sImp0aSI6IjRmOTdiNThkLTI1NTEtNDA4Yi04ZWM4LWUzZGZmZWQ3MTQ2NiIsImNsaWVudF9pZCI6InN1cGVyc2VjcmV0Y2xpZW50Iiwic2NvcGUiOlsicmVhZCIsImNvZGUiLCJ3cml0ZSJdfQ.fWt_H-ORHP44xOIljoqMfVIeGkqJGQqUBMj8paVxAPM' \
> --header 'cache-control: no-cache' \
> --header 'content-type: application/json' \
> --header 'postman-token: 76b3525e-bc3b-e629-91b6-a75253f657d8' \
> --header 'x-tenant: tenant1' |python -m json.tool
[
{
"createdAt": 1592936869000,
"createdBy": "UnAuthenticated",
"email": "admin@springframework.in",
"fullname": "Spring Tutorial Admin",
"grantedAuthorities": [
"ADMIN",
"USER"
],
"id": 6,
"mask": 1,
"tenantId": 1,
"updatedAt": 1592936869000,
"updatedBy": "UnAuthenticated",
"username": "admin"
},
{
"createdAt": 1592904896000,
"createdBy": "UnAuthenticated",
"email": "defaultuser@defaultadmin.com",
"fullname": "Default User",
"grantedAuthorities": [],
"id": 2,
"mask": 0,
"tenantId": 1,
"updatedAt": 1592904896000,
"updatedBy": "UnAuthenticated",
"username": "defaultuser"
},
{
"createdAt": 1592904900000,
"createdBy": "UnAuthenticated",
"email": "vinay@avasthi.com",
"fullname": "Vinay Avasthi",
"grantedAuthorities": [],
"id": 3,
"mask": 1,
"tenantId": 1,
"updatedAt": 1592904900000,
"updatedBy": "UnAuthenticated",
"username": "vavasthi"
}
]
</pre><br /></div><div>Now we run the same query in tenant2<br /><pre class="code">$ curl --request GET --url http://localhost:8081/tenant2/user --header 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI5MzgzMDcsInVzZXJfbmFtZSI6ImFkbWluIiwiYXV0aG9yaXRpZXMiOlsiVVNFUiIsIkFETUlOIl0sImp0aSI6IjJmOWVkOTYxLTk5MjktNDI3Zi1iZGU4LTc5NGIwYWNhMGYzNiIsImNsaWVudF9pZCI6InN1cGVyc2VjcmV0Y2xpZW50Iiwic2NvcGUiOlsicmVhZCIsImNvZGUiLCJ3cml0ZSJdfQ.KNzjB_JqJDaVI5vZNK-OcXDgvM5Uwt4I8tsVCazerpU' --header 'cache-control: no-cache' --header 'content-type: application/json' --header 'postman-token: b1a9651c-e4c0-7b2e-c7b3-e0b3e06fa40e' --header 'x-tenant: tenant2' |python -m json.tool
[
{
"createdAt": 1592904896000,
"createdBy": "UnAuthenticated",
"email": "defaultuser@defaultadmin.com",
"fullname": "Default User",
"grantedAuthorities": [],
"id": 2,
"mask": 0,
"tenantId": 1,
"updatedAt": 1592904896000,
"updatedBy": "UnAuthenticated",
"username": "defaultuser"
},
{
"createdAt": 1592904900000,
"createdBy": "UnAuthenticated",
"email": "vinay@avasthi.com",
"fullname": "Vinay Avasthi",
"grantedAuthorities": [],
"id": 3,
"mask": 1,
"tenantId": 1,
"updatedAt": 1592904900000,
"updatedBy": "UnAuthenticated",
"username": "vavasthi"
},
{
"createdAt": 1592938002000,
"createdBy": "UnAuthenticated",
"email": "ut2@springframework.in",
"fullname": "User in T2",
"grantedAuthorities": [],
"id": 6,
"mask": 1,
"tenantId": 1,
"updatedAt": 1592938002000,
"updatedBy": "UnAuthenticated",
"username": "userint2"
},
{
"createdAt": 1592937749000,
"createdBy": "UnAuthenticated",
"email": "admin@springframework.in",
"fullname": "Springframework Tenant2 Administrator",
"grantedAuthorities": [
"ADMIN",
"USER"
],
"id": 5,
"mask": 1,
"tenantId": 1,
"updatedAt": 1592937749000,
"updatedBy": "UnAuthenticated",
"username": "admin"
}
]
</pre>
</div>
<div>As we can see, we are seeing two totally different sets of users which are stored in two totally different sets of databases.</div><div><br /></div><div>Complete code for this blog post is available in my github repository <a href="#" id="https://github.com/vavasthi/springblog/tree/v1.7" name="https://github.com/vavasthi/springblog/tree/v1.7">here</a>. </div>vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-66253757211896409762020-06-21T12:49:00.001+05:302020-06-21T12:49:25.838+05:30OAuth2 and JWT Tokens Part 1In this blog post we look at how do we make our spring server become an OAuth2 Authorization server and start producing JWT tokens.<div><br /></div><div>The first step that we take is disable all the filters that we had added earlier. We don't want to authorize using old style REST calls. </div><div><br /></div><div>Now, the first step is to enable our authorization server. We create a new java configuration file that extends <i>AuthorizationServerConfigurerAdapter.</i></div><div><i><br /><script src="https://gist.github.com/vavasthi/1690798d99a1ae74cfc589b045d2d13a.js"></script></i></div><div>In the configuration , we autowire <i>AuthenticationManager, DataSource </i>and a<i> UserDetailsService</i>. There are two types of auth tokens in OAuth2. The first one are the usual tokens that are requested by users by providing their username and password. The second set of tokens are not tied to individual users, these are called client tokens and could be used for services talking to each other.</div><div><br /></div><div>We configure <i>ClientDetailsService</i> to use Jdbc client service. We add a password encoder in the configuration and provide a data source that would be used to store persistent data. In our example, we are using <i>SCryptPasswordEncoder</i>. Here we are using spring provided Jdbc client service but one could implement <i>ClientDetailsService</i> and <i>ClientRegistrationService</i> interfaces and provide their own custom implementation for client service.</div><div><br /></div><div>The next steps is to enable web security and provide an authentication manager bean for performing authentication.</div><div><br /><script src="https://gist.github.com/vavasthi/b0877b510ba07779a6022153bef0e83f.js"></script></div><div>We have seen earlier that we used a <i>UserDetailsService</i> to configure the authorization server. We are not going to use the spring provided service but write our own. </div><div><br /><script src="https://gist.github.com/vavasthi/2b78f5c987d2c237e668074bdad71b10.js"></script></div><div>In the <i>UserDetailsService,</i> we need to provide our own implementation for a method <i>loadByUsername</i>. In this method, we basically load the user entity from our database and create an object of type <i>UserDetails</i> and return it. The object also requires a list of <i>GrantedAuthority</i>. </div><div><br /></div><div><i>UserDetails</i> is an interface provided in <i>springframework</i>, we create our own class that implements the interface.</div><div><br /><script src="https://gist.github.com/vavasthi/617e2eaa6769f09509e58eb673746749.js"></script></div><div>Similar to <i>UserDetails</i>, <i>GrantedAuthority</i> is also an interface provided by <i>springframework</i>, we implement that interface to provide our concrete implementation of <i>GrantedAuthority</i>. For a very simple understanding, <i>GrantedAuthority</i> is like a user role. We can use this role later to provide access control on endpoints.</div><div><br /><script src="https://gist.github.com/vavasthi/95b02ee4bdc7c0569ff704d66b962c2c.js"></script></div><div>These changes will make the server ready for OAuth2 service. We still have a testing nightmare. Because now all our endpoints are behind this authentication filter, there is no way for us to create new clients and users. We could directly insert values in database, but we still have to worry about how to encrypt the password before inserting into the data. To get out of this situation, I add two endpoints, one for handling clients and another for handling users. These endpoints need to be configured so that they don't go through the authentication service. These need to be removed before the service goes into production.</div><div><br /></div><div>We define a pojo that mimics the <i>OauthClientDetails</i> schema in the database. It looks like below.</div><div><br /><script src="https://gist.github.com/vavasthi/7a898ce78b88a0c34f819b0b7a5feb65.js"></script></div><div>Now we add a repository for handling <i>OauthClientDetails table.</i></div><div><i><br /><script src="https://gist.github.com/vavasthi/8e22c0a88fbbc7470460ee3a16716a48.js"></script></i></div><div>We also add a service layer for Client handling.</div><div><br /><script src="https://gist.github.com/vavasthi/032b970aa9c2d1dda5bbb347514ddea7.js"></script></div><div><br /></div><div>We already had a <i>UserRepository</i>, we change it to encrypt the password before use store the password.</div><div><br /><script src="https://gist.github.com/vavasthi/2768fb7dae4f8f09cde2ab014be775cc.js"></script></div><div>We also add a service layer for User.</div><div><br /><script src="https://gist.github.com/vavasthi/9313c24c979aa4ffa599f5f1d317381a.js"></script></div><div>Now that we have all the layers required, we add the endpoint for Client.</div><div><br /><script src="https://gist.github.com/vavasthi/131b1522844950e4eb48ef72e307448c.js"></script></div><div><br />The next thing to modify is the endpoint layer for user.</div><div><br /><script src="https://gist.github.com/vavasthi/d61b14a6b1a5787cf77c5e28d38c4412.js"></script></div><div>Now we are ready with our code for create new clients and users. We still have the small issue because if we hit these endpoints, we will get unauthorized error. So we need to put these in the exception list. For this we go to our <i>SecurityConfiguration</i> that we had defined in one of the earlier tutorials.</div><div><br /><script src="https://gist.github.com/vavasthi/120f6ad1747ebd9c9641be14f756607b.js"></script></div><div>Look at the method <i>public void configure(WebSecurity web) throws Exception</i>. We have added following two lines to ignore evaluation of authentication filters for two families of URL.</div><div><pre class="code"> web.ignoring().antMatchers("/oauth/client/**");
web.ignoring().antMatchers("/user/**");
</pre>Of course this is very dangerous and we have added it only for testing purpose. In a future tutorial we will have a more elegant solution for this.</div><div><br /></div><div>Now we are ready. We can test the service with following curl or equivalent command.</div><div><br /><pre class="code">curl -X POST \
http://localhost:8081/oauth/token \
-H 'authorization: Basic YW5kcm9pZC1jbGllbnQ6YW5kcm9pZC1zZWNyZXQ=' \
-H 'cache-control: no-cache' \
-H 'content-type: application/x-www-form-urlencoded' \
-H 'postman-token: 3a349bc0-1230-adbe-4b79-9b938728a101' \
-d 'grant_type=password&password=mypassword&username=myuser&client_id=my-client&client_secret=my-secret'</pre><br />This is the response that we get</div><div><br /><pre class="code">{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjI5NTgsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiOGExZDYxN2ItZDU4OC00Nzc5LThlOTQtYTBiNWZkYzcxOTg2IiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl19.n7hCnBdnjMC8vuCsHxkOznfd06iJctGNeypx2fXWla4",
"token_type": "bearer",
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMwNTgsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiNGU4ODg3OTYtZWZiZS00ZDc5LTg1YmMtN2EzNzFhM2I4Yzg0IiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl0sImF0aSI6IjhhMWQ2MTdiLWQ1ODgtNDc3OS04ZTk0LWEwYjVmZGM3MTk4NiJ9.xuElTWYSLscgdcqj0t-4t6prJbVOfHVqM331UUjfPBQ",
"expires_in": 99,
"scope": "code",
"jti": "8a1d617b-d588-4779-8e94-a0b5fdc71986"
}
</pre></div><div>We can get the refresh token by calling the same endpoint with grant_type refresh_token.</div><div><br /><pre class="code">curl -X POST \
> http://localhost:8081/oauth/token \
> -H 'authorization: Basic YW5kcm9pZC1jbGllbnQ6YW5kcm9pZC1zZWNyZXQ=' \
> -H 'cache-control: no-cache' \
> -H 'content-type: application/x-www-form-urlencoded' \
> -H 'postman-token: 0adf261f-ed1e-85b3-67a5-21430fee8b38' \
> -d 'grant_type=refresh_token&refresh_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMxOTAsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiNGMwNjczOTQtZjllNS00NDVjLTg5MzItMmRiMDM4N2U2ZjIxIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl0sImF0aSI6ImE1ZmY5MjhiLWQ1YjEtNDQ5Yy04N2I4LTU3ODgwYzY1NjM3NiJ9.RS2K7N5XCHQNov02WNu1QK1AqOvgc7MuNzeCydt7Ajs'</pre></div><div>We get following response.<br /><pre class="code">{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMxNzQsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiY2E1OWU3NzktZDRkZS00NDU1LWFlNjItNDQ2NTAxYzUxZjhmIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl19.WgVgpBWuDmdJUk3UG8MTKOBsXn5zbEGO8gyqg2akN8o",
"token_type": "bearer",
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjMxOTAsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiNGMwNjczOTQtZjllNS00NDVjLTg5MzItMmRiMDM4N2U2ZjIxIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl0sImF0aSI6ImNhNTllNzc5LWQ0ZGUtNDQ1NS1hZTYyLTQ0NjUwMWM1MWY4ZiJ9.3zDqHLSwVLq_Dg8r4Ppf5tfvzxwT_7FEwdTV67l3VYQ",
"expires_in": 99,
"scope": "code",
"jti": "ca59e779-d4de-4455-ae62-446501c51f8f"
}</pre>We can verify a token by accessing <i>oauth/check_token</i> endpoint and providing the token.</div><div><br /><pre class="code">curl --request POST \
> --url http://localhost:8081/oauth/check_token \
> --header 'authorization: Basic YW5kcm9pZC1jbGllbnQ6YW5kcm9pZC1zZWNyZXQ=' \
> --header 'cache-control: no-cache' \
> --header 'content-type: application/x-www-form-urlencoded' \
> --header 'postman-token: ef035580-4cba-a7a1-deaa-dcf5f59e41fc' \
> --data token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTI3MjM2MTEsInVzZXJfbmFtZSI6InZhdmFzdGhpIiwianRpIjoiODZhZjZhYmUtZjA4YS00YzIzLThkYzYtYjY2ZmM3ZWQ0YWRiIiwiY2xpZW50X2lkIjoiYW5kcm9pZC1jbGllbnQiLCJzY29wZSI6WyJjb2RlIl19.Kn284yuxxHSxVtEc3D8H5YjVjPi0yN6oX1hDwEx5bOo</pre></div><div>We get the following response.<br /><pre class="code">{
"client_id": "android-client",
"exp": 1592723611,
"jti": "86af6abe-f08a-4c23-8dc6-b66fc7ed4adb",
"scope": [
"code"
],
"user_name": "myuser"
}</pre>This is all about enabling JWT OAuth tokens with any spring server. We will continue this tutorial in next part with some more important details. The complete code for the working server is <a href="https://github.com/vavasthi/springblog/tree/v1.6" target="_blank">here</a>.</div><div><br /></div>vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-75619277852629858282020-06-05T13:15:00.001+05:302020-06-05T13:15:34.124+05:30Auditing in MySQLOne of the important requirements in many RDBMS based workloads is to have audit log where any row in any table is stamped with <i>who changed it</i> and <i> when was it changed</i>? To do this in every place where the table is modified is extremely cumbersome. In this blog post we look at how we can enable the springframework based auditing framework to perform this activity.<div><br /></div><div>Before we do that, we need to upgrade the versions for our dependencies since I have been using this project for quite sometime. Also I decided to use <i>lombok</i> logging annotation and removed all the dependencies on log4j.</div><div><br /></div><div>Here are the modifications to the <i>pom.xml</i> for setting dependencies.</div><div><br /></div><div><br /></div><div><script src="https://gist.github.com/vavasthi/aeb4914fda1ac435fb2aaf26050be4c7.js"></script>Now we look at our <i>User</i> entity. We need to add auditing fields to this table. Since in a realistic project one would have multiple entities, we create a abstract entity base class with all the audit fields. We also move the primary key to the abstract base class. When we do that, hibernate provides multiple ways of mapping the entities to the database schema. We want all the parent attributes to be stored in a single table with the child class attribute. To accomplish that, we need to add <i>@MappedSuperClass</i> to the base class.</div><div><script src="https://gist.github.com/vavasthi/617084dd2c03f59b552547219d8059bd.js"></script><br /></div><div>As we can see, we have added five attributes in the base class. One is the id for all our entities and rest four will be used for auditing purposes.</div><div>At this time we also add another layer to our code. Currently all the <i>Endpoints</i> directly call the <i>Repository</i> layer, this causes a problem if we want to write functions that can be reused across different endpoints. An example of this need is <i>retrieveUser</i> method that takes an argument that could be a <i>username</i> or a <i>email</i>. Currently this method lies in the <i>Endpoint</i> layer as a private method. This is a useful method in many different contexts, so we create a new <i>UserService</i> layer and move this method there.</div><div><br /></div><div><script src="https://gist.github.com/vavasthi/cb0a6104fbe0eac7beeb918336893803.js"></script><br /></div><div>Now, let's get to the original task of enabling auditing. First we define a Auditing Config as below.</div><div><script src="https://gist.github.com/vavasthi/3d284bdf1fd672a4e45f9da99f50493c.js"></script><br /></div><div>We had earlier defined a <i>ThreadLocal</i> that is used by the <i>auditAware</i> method defined above to extract currently logged in user and return its userId. As we can see the audit fields in the AbstractBaseEntity expects a Long for @<i>createdBy</i> and <i> @LastModifiedBy</i> fields. The <i>EntityAuditConfig</i> also has annotation @<i>EnableJpaAuditing which is </i>required<i>.</i></div><div>At this point we also add a new endpoint called ProfileEndpoint which can be used to manage the entity that represents a user profile. This entity currently only contains a url.</div><div>Now if we perform any operation on any of the endpoint, we will see the auditing fields automatically populated. Give it a spin. It is a life saver in many productions applications. I have had situations where users changed their passwords, forgot them and then complained saying that they have been hacked. </div><div>The complete code for this and previous posts can be found at my <a href="#" id="https://github.com/vavasthi/springblog/tree/v1.5" name="https://github.com/vavasthi/springblog/tree/v1.5">github</a> repository. This tutorial changes are under v1.5.</div>vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-22620746235515536942020-02-24T13:26:00.003+05:302020-02-24T13:26:57.246+05:30Hadoop and Spark Locally<div dir="ltr" style="text-align: left;" trbidi="on">
As a continuation of the <a href="https://www.springframework.in/2020/02/hive-and-spark.html">last</a> post, we now look at how to make it deployable in a proper Spark/Hadoop cluster. We will not go into the details of the setup of these clusters themselves but more into how do we make sure a program that we developed earlier could run as a job in a cluster.<br />
<div>
We will continue with the setup in our local machine. I am using a Mac so the instructions are with respect to that but most instructions would be common to any other platform.</div>
<div>
If Spark is processing data from a database and writing into a hive, pretty much what we did in the last post would work. The problem arises if some of the data being processed exists as flat files. If we want to submit our jobs to a Spark cluster, we can not use local files because the jobs are not running in the local file system. </div>
<div>
The best approach is to either use a hdfs cluster or deploy a single node hdfs on your machine. Here I am enumerating the steps to set up a single node hdfs cluster on a Mac OS X machine.</div>
<div>
<ul style="text-align: left;">
<li>Download the Hadoop distribution for your machine <a href="https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz">here</a>.</li>
<li>Hadoop distribution is available in the form of a <i>.tar.gz</i> file and you can expand it in some directory on your machine. The expansion will create a directory of the form hadoop-x.y.z assuming your Hadoop version is x.y.z. set the environment variable HADOOP_HOME to the full pathname of this Hadoop directory.</li>
<li>Add $HADOOP_HOME/bin to the PATH variable.</li>
<li>Now we need to update the configuration files for hadoop.</li>
</ul>
<pre class="code" style="background-color: lightgrey; color: #333333; font-size: 13px; white-space: pre-wrap;">$ cd $HADOOP_HOME/etc/hadoop
$ vi core-site.xml</pre>
<br />
We update the file with the following properties.<br />
<script src="https://gist.github.com/vavasthi/2f5fe52e9b4c64af546d07191241a2a8.js"></script><br />
<pre class="code" style="background-color: lightgrey; color: #333333; font-size: 13px; white-space: pre-wrap;">$ vi hdfs-site.xml</pre>
<br />
We update the file with the following properties.
<script src="https://gist.github.com/vavasthi/d52d7b31eb8e02988e20cf0bd87d06fa.js"></script><br />
<pre class="code" style="background-color: lightgrey; color: #333333; font-size: 13px; white-space: pre-wrap;">$ vi mapred-site.xml</pre>
<br />
We update the file with the following properties.
<script src="https://gist.github.com/vavasthi/69d645030ba9594ef25330be461ddf9f.js"></script><br />
<pre class="code" style="background-color: lightgrey; color: #333333; font-size: 13px; white-space: pre-wrap;">$ vi yarn-site.xml</pre>
<br />
We update the file with the following properties.
<script src="https://gist.github.com/vavasthi/9dd5da907ba549c22d0eaebcf872bc79.js"></script><br />
Now we start the Hadoop.
<br />
<pre class="code" style="background-color: lightgrey; color: #333333; font-size: 13px; white-space: pre-wrap;">$ cd $HADOOP_HOME
$ sbin/start-all.sh</pre>
Now we can access files stored into the HDFS in our spark jobs.<br />
<script src="https://gist.github.com/vavasthi/82c9ecd6f8ec72cbf97f376f01da49b3.js"></script>The next post will go more into the details of how to process files in spark.</div>
</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-44249652516047087352020-02-18T14:45:00.001+05:302020-02-18T15:03:52.144+05:30Hive and Spark<div dir="ltr" style="text-align: left;" trbidi="on">
In this blog post, we take a slight deviation from core issues related to the spring framework and look at an issue that spring programmers might face regularly. Recently I was looking with Spark and found a need to read the data from MySQL, do some processing and write it back to a hive instance. We will look at this issue in this post.<br />
We start by making sure our hive instance is backed by a database. To do this, we do the following.<br />
<pre class="code">$ cp $HIVE_HOME/conf/hive-default.xml.template $HIVE_HOME/conf/hive-site.xml
$ vi gnu/apache-hive-3.1.2-bin/conf/hive-site.xml</pre>
<br />
We edit the hive-site.xml file and make sure it is configured as below.<br />
<script src="https://gist.github.com/vavasthi/071676fe2f306e7e48293beb73236d73.js"></script>We can configure the values to suit our needs. But make sure the MySQL username, password, database URL exists and has relevant permissions.<br />
Now we create another database in MySQL which will contain the data that we need to process. I am calling this database mystuff, with username mystuff, password mystuff123. We need to run following commands in MySQL to make sure everything exists and permissions are appropriate.
<br />
<pre class="code">create database mystuff;
Query OK, 1 row affected (0.00 sec)
mysql> create user mystuff@localhost identified by 'mystuff123';
Query OK, 0 rows affected (0.00 sec)
mysql> create user mystuff@'%' identified by 'mystuff123';
Query OK, 0 rows affected (0.01 sec)
mysql> grant all on mystuff.* to mystuff@localhost;
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on mystuff.* to mystuff@'%';
Query OK, 0 rows affected (0.00 sec)
</pre>
<br />
Now we create a plain java project in IntelliJ with the following pom.xml file.<br />
<script src="https://gist.github.com/vavasthi/3d4350bdc22bedcd561bbb3ffee4737a.js"></script>
Now to look at the problem at hand. We have a table in MySQL with the following structure.
<br />
<pre class="code">mysql> desc mydata;
+-------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| id | int | YES | | NULL | |
| k | varchar(10) | YES | | NULL | |
| v | varchar(255) | YES | | NULL | |
+-------+--------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
</pre>
<br />
We want to flatten this table such that each key gets converted to a column for each id. So assume our current data is as below.
<br />
<pre class="code">mysql> select * from mydata;
+------+-------+-------------------+
| id | k | v |
+------+-------+-------------------+
| 1 | NAME | John Doe. |
| 1 | EMAIL | jd@example.com |
| 2 | NAME | Jane Doe |
| 2 | EMAIL | janed@example.com |
+------+-------+-------------------+
2 rows in set (0.00 sec)
</pre>
We want to load this data and convert it into a flattened table that has three columns i.e. id, name. email. Then we want to populate this into table person into hive with flattened data.
<br />
<pre class="code">hive (default)> select * from person;
OK
person.id person.email person.name
1 jd@example.com. John Doe
2 janed@example.com Jane Doe
Time taken: 0.094 seconds, Fetched: 2 row(s)
</pre>
<br />
The following code will perform the above conversion.
<script src="https://gist.github.com/vavasthi/4b93d7f297f86f5c59ad0d636ab40c71.js"></script>
<br />
Line numbers 11 through 17 create a SparkSession for hive operations. The key instruction here is enableHiveSupport. Lines 20 through 26 create a SparkSession that will be used for MySQL operations. Lines 29 through 37 load the complete contents of the table. Lines 39 through 42 will group the results by the id and pivot the table on the field K.
Line 46 through 49 creates a data frame for hiveSession and writes the contents in a table with named person.
</div>
vavasthihttp://www.blogger.com/profile/17794185844812005239noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-58365217088452142802019-04-12T14:55:00.001+05:302019-04-12T15:01:45.913+05:30Spring and more Kafka<div dir="ltr" style="text-align: left;" trbidi="on">
In the <a href="https://www.springframework.in/2019/04/kafka-and-spring.html">last</a> post, we saw how to integrate Kafka with Spring Boot application. The post was a very simple implementation of Kafka. The real world is much more complex. You have to deal with multiple topics, you need multiple partitions. In this post, we explore more details of a spring boot application with Kafka.<br />
<div>
Let's start with the fact that we have multiple topics that we are dealing with. The very first thing that we need to do is to separate out the properties for each of the topics.<br />
<script src="https://gist.github.com/vavasthi/49442471a0e24cd01912023bc785e3f3.js"></script><br />
We can no longer use default <i>KafkaTemplate</i>, we will have to create our own. The best way is to define a KafkaProducer. To create a <i>KafkaProducer</i>, we need to create <i>KarkaProducerConfig</i>. Here are two sets of <i>KafkaProducerConfig</i> class, one each for the topic.<br />
<script src="https://gist.github.com/vavasthi/72ffb670aa331699eb106198bb71fa54.js"></script><br />
<script src="https://gist.github.com/vavasthi/9cca3fed434db91ad3be6fd6a3d92909.js"></script><br /></div>
Looking at the producer config, we can observe that we create two beans which are qualified by names and return an appropriate <i>KafkaTemplate</i> which is later used to send the message. Similar to producer config, we need to create consumer config. Consumer config is used by the listener to listen for the message.<br />
<script src="https://gist.github.com/vavasthi/65d8bd9adb6858914b4127dd26391401.js"></script><br />
<script src="https://gist.github.com/vavasthi/76c54675a8cc132aaaa7ef7fbeeff1dd.js"></script><br />
As we can see in each of the consumer configs, we create a bean which returns a <i>ConcurrentKafkaListenerContainerFactory</i>. The beans are qualified by a name so that we can use an appropriate container factory for receiving messages.<br />
We also modify the <i>MyTopicMessage</i> and add a member variable <i>topicName</i> that will help us distinguish the topic to which the message needs to be sent.<br />
<script src="https://gist.github.com/vavasthi/0d99b847cc702405410139f60d157f5f.js"></script><br />
We also modify the endpoint so that the message can be sent to the appropriate topic.<br />
<script src="https://gist.github.com/vavasthi/4ee1bee3dba7b412750db70be1b7c125.js"></script><br />
Now we modify the Listener to integrate everything so that the appropriate message can be received. Look at <i>@KafkaListener</i> annotation, we pass on the ListenerContainerFactory as an argument to receive a message from a queue.
<script src="https://gist.github.com/vavasthi/6a606143a492f8c5cb4bf5a99d7948ee.js"></script><br />
Now we can test the server. The flow of the service is as below.
<br />
<br />
<ol style="text-align: left;">
<li>The message is posted to the endpoint as a <i>POST</i> request</li>
<li>The <i>@RestController</i> receives the message and based on the <i>topicName</i> in the request, it sends to message to the topic with the same name.</li>
<li>The listener receives the message from the appropriate queue.</li>
</ol>
<div>
<pre class="code">$ curl -X POST \
> 'http://localhost:8081/send?token=3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
> -H 'Content-Type: application/json' \
> -H 'Postman-Token: 15fbe075-9c80-4af9-a797-6b5e0979fd1b' \
> -H 'cache-control: no-cache' \
> -H 'token: 3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
> -d '{
> "message" : "This is my message!",
> "topicName" : "FirstTopic"
> }'
Success!
</pre>
<br />
The message receipt is indicated in the server log.<br />
<pre class="code">2019-04-12 14:50:47.142 INFO 51396 --- [ntainer#0-0-C-1] i.s.b.t.listeners.KafkaMessageListener : Received FirstTopic message for partition 0 This is my message!
</pre>
Similarly, we can send a message to <i>SecondTopic.</i><br />
<pre class="code">$ curl -X POST 'http://localhost:8?token=3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' -H 'Content-Type: application/json' -H 'Postman-Token: 15fbe075-9c80-4af9-a797-6b5e0979fd1b' -H 'cache-control: no-cache' -H 'token: 3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' -d '{
"message" : "This is another message!",
"topicName" : "SecondTopic"
}'
Success!
</pre>
<br />
The message receipt is indicated in the server log.<br />
<pre class="code">2019-04-12 14:53:14.972 INFO 51396 --- [ntainer#1-0-C-1] i.s.b.t.listeners.KafkaMessageListener : Received SecondTopic message for partition 0 This is another message!
</pre>
The complete code base for this tutorial can be found at my <a href="https://github.com/vavasthi/springblog/tree/v1.4">github repository at v1.4</a>.
</div>
</div>
vavasthihttp://www.blogger.com/profile/17794185844812005239noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-27644302454092457512019-04-10T14:30:00.000+05:302019-04-10T14:30:11.822+05:30Kafka and Spring<div dir="ltr" style="text-align: left;" trbidi="on">
Kafka has become a very popular platform and is being used as a stream, journal and even eventing system. In this post, we explore how to integrate Kafka with spring framework application. First, we add the Kafka bootstrap server details in the <i>application.properties</i> file.<br />
<script src="https://gist.github.com/vavasthi/21b6743b78301cceb551cc7165bbd72d.js"></script><br />
Let's also add dependencies in pom.xml.<br />
<script src="https://gist.github.com/vavasthi/7c0896a5ac3d3a98e11a614f95f10b39.js"></script><br />
Now, for each Kafka topic, we create a listener class. The listener class provides a callback method that is called when any message is retrieved on that topic.<br />
<script src="https://gist.github.com/vavasthi/fe5e18132e512e03c03fabae52f12b45.js"></script><br />
Now we create an endpoint through which we inject a message in the queue. The message is sent to the queue and is retrieved by the listener.<br />
<br />
<script src="https://gist.github.com/vavasthi/8093a5204869a92b59ba159851d61f9e.js"></script>We autowire a KafkaTemplate instance that is used to send the message to the queue.<br />
<br />
<pre class="code">$ curl -X POST \
> 'http://localhost:8081/send?token=3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
> -H 'Content-Type: application/json' \
> -H 'Postman-Token: e281e3c5-0dae-4bb7-ac8d-6555f66a18c6' \
> -H 'cache-control: no-cache' \
> -H 'token: 3193fa24-a0ba-451b-83ff-eb563c3fd43b-cdf12811-7e41-474b-8fa6-e8fefd4a738c' \
> -d '{
> "message" : "This is my message!"
> }'
Message sent successfully!.
</pre>
The receipt of message is indicated in the spring server log.<br />
<br /><pre class="code">
2019-04-10 14:28:03.969 INFO 31091 --- [ntainer#0-0-C-1] i.s.b.t.listeners.MyTopicKafkaListener : Received Promise message This is my message!
</pre>
</div>
vavasthihttp://www.blogger.com/profile/17794185844812005239noreply@blogger.com1tag:blogger.com,1999:blog-2671448201212310231.post-33327350811038883002019-03-10T20:01:00.002+05:302019-03-10T20:01:47.513+05:30Spring Boot and Docker Containers<div dir="ltr" style="text-align: left;" trbidi="on">
With microservices based deployment, the first step is to <i>dockerize</i> your software. In our case, we want to create docker images for each of our microservices so that we can orchestrate them better. I have decided to use container registry provided by Google to build and upload the images.<br />
<br />
<script src="https://gist.github.com/vavasthi/a01c60aa3fbcb44b9cef871fe3416039.js"></script>The first thing we need to do is to create a dependency to <i>spring-cloud-dependencies</i> pom. Then we add a dependency to <i>spring-cloud-config-server.</i> Then we add a <i>dockerfile-maven-plugin.</i> Now we need to keep in mind that in the configuration for <i>dockerfile-maven-plugin</i> we need to provide the repository. If can see that our repository starts with <i>gcr.io/</i>. This makes sure that the image after creation is pushed to the container registry hosted by Google. If you want to have some other registry, you need to provide it in the form <i>hostname:portnumber</i>.<br />
Now we can issue the following command the docker image would be build and pushed to google registry.<br />
<br />
<pre class="code">$ mvn deploy -DskipTests
[INFO]
[INFO] --- dockerfile-maven-plugin:1.4.10:push (default) @ customer ---
[INFO] Using Google application default credentials
[INFO] loaded credentials for user account with clientId=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com
[INFO] The push refers to repository [gcr.io/myproject/rae/customer]
[INFO] Image 967d96afcc46: Preparing
[INFO] Image 36e051842720: Preparing
[INFO] Image d1646aaa6540: Preparing
[INFO] Image 19382582b926: Preparing
[INFO] Image 41715d8d7d2b: Preparing
[INFO] Image f3a38968d075: Preparing
[INFO] Image a327787b3c73: Preparing
[INFO] Image 5bb0785f2eee: Preparing
[INFO] Image f3a38968d075: Waiting
[INFO] Image a327787b3c73: Waiting
[INFO] Image 5bb0785f2eee: Waiting
[INFO] Image 36e051842720: Layer already exists
[INFO] Image 41715d8d7d2b: Layer already exists
[INFO] Image d1646aaa6540: Layer already exists
[INFO] Image 19382582b926: Layer already exists
[INFO] Image 967d96afcc46: Pushing
[INFO] Image a327787b3c73: Layer already exists
[INFO] Image 5bb0785f2eee: Layer already exists
[INFO] Image f3a38968d075: Layer already exists
[INFO] Image 967d96afcc46: Pushed
[INFO] 0.0.1-SNAPSHOT: digest: sha256:f6bad4811f867dd75225797bee684ea43c0ddaf2b83de1b419a9f75e9a3941bc size: 2001
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47.638 s
[INFO] Finished at: 2019-03-10T19:58:27+05:30
[INFO] Final Memory: 55M/857M
[INFO] ------------------------------------------------------------------------
</pre>
We can see below that the image is now pushed to google container registry.<br />
<pre class="code">$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/myproject/rae/customer 0.0.1-SNAPSHOT a566e2f28705 19 seconds ago 518MB
gcr.io/myproject/rae/customer <none> 8c61d1a5aef4 13 minutes ago 518MB
<none><none><none><none><none><none><none><none>
</none></none></none></none></none></none></none></none></none></pre>
</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-37246507114774606852019-02-12T14:38:00.002+05:302019-02-12T14:38:54.277+05:3012. Adding GIT release information<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post, we saw how to enable actuator endpoints on our spring server. Once we have done that, it is a good idea to add GIT release information to the server in order to get the information related to the currently deployed server at runtime.<br />
We add the <i>git-commit-id-plugin</i> to our <i>pom.xml</i>.<br />
<br />
<pre class="code"> <plugin>
<groupid>pl.project13.maven</groupid>
<artifactid>git-commit-id-plugin</artifactid>
<version>2.2.1</version>
</plugin>
</pre>
The next things to do is to create a git.properties file in the resource directory of the project.<br />
<script src="https://gist.github.com/vavasthi/99e9eb3e91650bc4b3ac2eb6327a2e9e.js"></script><br/>
The git-commit-id is added by the maven plugin so it is a good idea to build the project using maven.<br />
<br />
<pre class="code">$ mvn clean package -DskipTests
</pre>
Now we can run the server and check the <i>/manage/info</i> endpoint using <i>curl</i> command.<br />
<pre class="code">$ curl -X GET http://localhost:9091/manage/info
{"git":{"branch":"master","commit":{"id":"6fb94c0","time":1549868235.000000000}}
</pre>
<div>
<br /></div>
</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-6152426499068460252019-02-12T14:30:00.001+05:302019-02-12T14:37:18.204+05:3011. Spring Actuators<div dir="ltr" style="text-align: left;" trbidi="on">
Spring provides actuators that are a helpful set of tools to debug the application on runtime. Here is how to enable them. We first add actuator dependency in the pom.xml<br />
<pre class="code"><dependency>
<groupid>org.springframework.boot</groupid>
<artifactid>spring-boot-starter-actuator</artifactid>
</dependency></pre>
<br />
Now we define a prefix for all the actuator endpoints. We add the following line into the <i>application.properties</i> file. This enables all the actuator endpoints. We can enable specific endpoints by adding a comma delimited list of endpoints. We also deploy management endpoint on a separate port so that we can block its access from something like ELB.<br />
<pre class="code">management.endpoints.web.base-path=/manage
management.server.port=9091
management.endpoints.web.exposure.include=*
</pre>
Since we already have a security filter defined, we need to exempt health and info endpoint from security check. We add the following URLs int he SecurityConfiguration configure method.<br />
<pre class="code">@Override
public void configure(WebSecurity web) throws Exception {
web.ignoring().antMatchers("/manage/health");
web.ignoring().antMatchers("/manage/info");
web.ignoring().antMatchers("/webjars/**");
web.ignoring().antMatchers("/error");
web.ignoring().antMatchers("/swagger-ui.html");
web.ignoring().antMatchers("/v2/api-docs/**");
web.ignoring().antMatchers("/swagger-resources/**");
}
</pre>
Here we have added paths related to <i>error</i>, <i>actuator</i>, and <i>swagger.</i><br />
This enables actuator endpoints for our server. We can query these endpoints and following is the sample response.<br />
<br />
<pre class="code">$ curl -X GET http://localhost:9091/manage/health
{"status":"UP"}
</pre>
</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-76989151381343907702019-02-11T12:32:00.000+05:302019-02-11T12:32:01.481+05:3010. Application with multiple datasources<div dir="ltr" style="text-align: left;" trbidi="on">
Many times it is a practical requirement to have multiple databases for a single application. These databases could be at different locations on the cloud and different entities in your application may be dealing with these databases.<br />
Multi-tenancy is a great requirement when multiple data sources are needed. Many tenants may insist on having their own databases. Here we present how we can configure the spring application to interact with multiple databases.<br />
The first step is to look at our <i>application.properties</i> file. We have a list of properties defined for the default dataSource which we will need to replicate for our second database. Let's assume we are going to use two data sources, the first one is called the<i> user</i> and the second one is called <i>other</i>.<br />
<script src="https://gist.github.com/vavasthi/bf7cb57dc42805b2c7bf5cf3ab92901b.js"></script><br />
As we can see above, we have replicated all the data source properties and given it a new prefix, <i>other</i>. Now, these two databases could have completely independent settings. They could point to totally different databases. Each of these properties could be configured to completely different settings. Here we have just changed the name of the <i>database</i>, <i>username</i>, <i>password</i>.<br />
To make this work. we will have to split the repositories and the entity objects for each of the data sources. Here we create the following hierarchy of packages for each of the data sources.<br />
<pre class="code">src/main/java
- in.springframework.blog.tutorials
- user
- domain
- repository
- other
- domain
- repository
</pre>
As we can see for each of the data sources, we have a domain package that would contain the entities and a repository package that would contain the repository class. This is needed so that each of the entity managers only searches for its own classes.<br />
Now we need to define a configuration of each of the data sources that we have defined. The first data source will contain <i>the user</i> table and will also be the primary data source.<br />
<script src="https://gist.github.com/vavasthi/80a3c5076706478c321ce15a205e6390.js"></script><br />
As we can see in the class above, it uses all the properties prefixed with <i>spring.datasource</i> and scans directories related to the user data source. Now let's look at the configuration for <i>other</i> data source.<br />
<script src="https://gist.github.com/vavasthi/158a4e5cf43df2e4b9e842ddeb9f0a70.js"></script><br />
The things to note in the <i>other</i> data source is that the <i>@Primary</i> annotation doesn't exist because we can have only one set of primary beans of a type. Also, the directories to be searched are for the <i>other</i> data source's <i>domain</i> and <i>repository</i> objects.<br />
At this time we also move the old <i>User</i> and <i>UserRepository</i> classes to their respective subdirectories. We also create <i>Other</i> and <i>OtherRepository</i> classes in their respective subdirectories. We also change <i>CrudRepository</i> to <i>JpaRepository</i> in each of the repository classes.<br />
Now, our application is set up to use two different data sources and we can verify that by running the application. Since we have set <i>ddl-auto</i> property to <i>update, </i>it should create a new schedule when the application is run.<br />
<pre class="code">$ mysql -u tutorial -ptutorial123 tutorial
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 115
Server version: 8.0.12 Homebrew
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show tables;
+--------------------+
| Tables_in_tutorial |
+--------------------+
| hibernate_sequence |
| user |
+--------------------+
2 rows in set (0.00 sec)
mysql> desc user;
+------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+-------+
| id | bigint(20) | NO | PRI | NULL | |
| email | varchar(255) | YES | UNI | NULL | |
| fullname | varchar(255) | YES | | NULL | |
| password | varchar(255) | YES | | NULL | |
| username | varchar(255) | YES | UNI | NULL | |
| auth_token | varchar(255) | YES | UNI | NULL | |
| expiry | datetime | YES | | NULL | |
| mask | bigint(20) | NO | | NULL | |
| authToken | varchar(255) | YES | | NULL | |
+------------+--------------+------+-----+---------+-------+
9 rows in set (0.00 sec)
</pre>
<br />
<pre class="code">$ mysql -u other -pother123 other
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 116
Server version: 8.0.12 Homebrew
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show tables;
+--------------------+
| Tables_in_other |
+--------------------+
| hibernate_sequence |
| other |
+--------------------+
2 rows in set (0.00 sec)
mysql> desc other
-> ;
+-----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| id | bigint(20) | NO | PRI | NULL | |
| otherData | varchar(255) | YES | | NULL | |
+-----------+--------------+------+-----+---------+-------+
2 rows in set (0.00 sec)</pre>
The source code for this tutorial is available at git repository as <a href="https://github.com/vavasthi/springblog/tree/v1.2">v1.2</a>.</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com1tag:blogger.com,1999:blog-2671448201212310231.post-64673803202259072582019-01-23T22:36:00.000+05:302019-01-23T22:36:27.393+05:309. Role based data access<div dir="ltr" style="text-align: left;" trbidi="on">
Many times we have a need to return data from an endpoint based on the role. Springframework provider easy mechanism for us to be able to do that. Let's take an example. In the previous example, we want to add a <i>GET</i> method in <i>UserEndpoint</i> that returns all the users. For any safe system, we want to return all the users if the role is <i>ADMIN</i> but if the role is <i>USER</i> then we want to return only that particular user. We don't want to write multiple methods for that purpose.<br />
<pre class="code"> @RequestMapping(method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
@PostFilter("hasAuthority('ADMIN') or filterObject.authToken == authentication.name")
public Iterable<user> getUsers() {
Iterable<user> users = userRepository.findAll();
return users;
}
</user></user></pre>
The method is described above. As we can see it is an extremely simple method, it calls <i>findAll</i> method on the repository which will return all the valid user records and returns an <i>Iterable</i> collection back. The interesting aspect of this method is in the <i>@PostFilter</i> annotation. Let's try to understand the annotation. The first condition is <i>hasAuthority('ADMIN')</i>. It implies that if the authenticated role is <i>ADMIN</i> then return the records as it is. The next bit of the filter condition uses an object called <i>filterObject</i>. This is an automatically defined expression that we can use in <i>@PostAuthorize</i> filter. A complete list of all the expressions exists <a href="https://docs.spring.io/spring-security/site/docs/current/reference/html/authorization.html#el-common-built-in">here</a>. The expression <i>filterObject</i> is used for each element of a collection that is returned from the endpoint. Since we know that when this endpoint is called, we would only be authenticating using <i>authToken</i>, the name of authentication in security context is set to the token itself. We can verify this in the code in <i>AuthenticationFilter</i> class.<br />
<pre class="code"> else {
Optional<string> token = getOptionalHeader(httpRequest,"token");
TokenPrincipal authTokenPrincipal = new TokenPrincipal(token);
processTokenAuthentication(authTokenPrincipal);
}
</string></pre>
The code fragment described above creates a TokenPrincipal object with the token as its name. That is the reason we have the condition <i>filterObject.authToken == authentication.name</i>. Taking the complete condition of <i>@PostFilter</i> we can see that the condition implies that <b>return everything unconditionally if the role is <i>ADMIN</i> otherwise return the users with authToken as the currently authenticated user.</b><br />
We have two users defined on the system. One has a role of the user and the other has a role of admin. Here are the examples of what happens when we call the endpoint with both of these users.<br />
The first example is with a user with role <i>USER</i>.<br />
<pre class="code">$ curl -X GET "http://localhost:8081/user" -H "accept: application/json" -H "token: 3d47912d-73a0-4c4c-95e6-0486273d6221-28fa4f38-0f1b-4740-8e1d-3228288de631" | python -m json.tool
[
{
"authToken": "3d47912d-73a0-4c4c-95e6-0486273d6221-28fa4f38-0f1b-4740-8e1d-3228288de631",
"email": "user@springframework.in",
"expiry": 1548345909000,
"fullname": "User",
"id": 2,
"mask": 1,
"password": "User123",
"username": "user"
}
]
</pre>
<br />
The second example is with a user with role <i>ADMIN</i>.<br />
<pre class="code">$ curl -X GET "http://localhost:8081/user" -H "accept: application/json" -H "token: a137dd09-11e4-4dcf-a141-0b235d39a505-60d43bf4-3674-4248-be1d-c2669f14589f" | python -m json.tool
[
{
"authToken": "3d47912d-73a0-4c4c-95e6-0486273d6221-28fa4f38-0f1b-4740-8e1d-3228288de631",
"email": "user@springframework.in",
"expiry": 1548345909000,
"fullname": "User",
"id": 2,
"mask": 1,
"password": "User123",
"username": "user"
},
{
"authToken": "a137dd09-11e4-4dcf-a141-0b235d39a505-60d43bf4-3674-4248-be1d-c2669f14589f",
"email": "admin@springframework.in",
"expiry": 1548348263000,
"fullname": "Administrator",
"id": 3,
"mask": 4,
"password": "Admin123",
"username": "admin"
}
]
</pre>
As we can see above, the call with the role USER only returns the object related to that particular user while the call with the role returns all the users present in the system. This is how we can achieve role based object access without writing multiple endpoints.</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-41339070700877137192019-01-22T22:21:00.000+05:302019-01-22T22:26:07.503+05:308. That little matter of creating a user<div dir="ltr" style="text-align: left;" trbidi="on">
Now that our simplified authentication system is in place, we are faced with the little matter of how to create a new user. Since we don't have a <i>username</i>, <i>password</i>, or <i>token</i> we can't really create a new user.<br />
To accomplish that, we need to make some modifications to our <i>AuthenticationFilter</i>. In the <i>doFilter</i> method, we add a special <i>if</i> block to take care of user creation.<br />
<pre class="code"> if (httpRequest.getRequestURI().toString().equals("/user") && httpRequest.getMethod().equals("POST")) {
Optional<string> username = getOptionalHeader(httpRequest,"username");
UsernamePasswordPrincipal usernamePasswordPrincipal = new UsernamePasswordPrincipal(username, username, true);
processUsernameAuthentication(usernamePasswordPrincipal);
}
else if (httpRequest.getRequestURI().toString().equals("/authenticate") && httpRequest.getMethod().equals("POST")) {
Optional<string> username = getOptionalHeader(httpRequest,"username");
Optional<string> password = getOptionalHeader(httpRequest,"password");
UsernamePasswordPrincipal usernamePasswordPrincipal = new UsernamePasswordPrincipal(username, password);
processUsernameAuthentication(usernamePasswordPrincipal);
}
</string></string></string></pre>
<script src="https://gist.github.com/vavasthi/d9bcd32888eabdcb858affb6ecdce4b5.js"></script><br />
As we can see the code fragment, if the call is made to <i>/user</i> endpoint with <i>POST</i> method, we look for a <i>username</i> header and trigger spring authentication. We also need to make a change in the <i>UsernamePasswordPrincipal</i> to take a flag that would tell us if the user is a new user or existing user.<br />
<script src="https://gist.github.com/vavasthi/8e5478aba994368410f7fc15db37f0c2.js"></script>Now that we have modified the principal and filter, we need to handle this in the provider. The provider that gets invoked for username and password authentication is <i>UsernamePasswordAuthenticationProvider</i>.<br />
As we can see in the provider's <i>authenticate</i> method, we have added an if block that checks if this is a new user. In case it is a new user, we authenticate this user with a role <i>NEWUSER</i>. The create user endpoint is only allowed to be called for a role <i>NEWUSER</i>.<br />
Now that we have stitched the path for authentication of a new user to create user endpoint, we can see the endpoint itself.<br />
<pre class="code"> @RequestMapping(method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
@PreAuthorize(MyConstants.ANNOTATION_ROLE_NEWUSER)
public Optional<user> createUser(@RequestHeader(value="username") String username, @RequestBody User user) {
user.setMask(Role.USER.ordinal());
User storedUser = userRepository.save(user);
storedUser.setPassword(null);
return Optional.of(storedUser);
}
</user></pre>
As we can see, we have a <i>@PreAuthorize</i> added with <i>NEWUSER</i> role. We also set the role of the user to <i>the USER</i>. Before returning the response, we set the password to null.<br />
This fixes our service to allow the creation of a new user. The <a href="https://github.com/vavasthi/springblog/tree/v1.1">complete</a> code is available tagged as v1.1 for this and previous posts.<br />
<script src="https://gist.github.com/vavasthi/8cb8e218f0afca39df9e1acb3e76a81d.js"></script><br /></div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-5068756456024139342019-01-22T21:40:00.001+05:302019-01-22T21:40:25.640+05:307. Enabling Swagger<div dir="ltr" style="text-align: left;" trbidi="on">
Before we proceed any further, let's enable swagger on our service so that testing it becomes easy. The first step is to add springfox dependencies.<br />
<script src="https://gist.github.com/vavasthi/9a3a82c8ea98324a3bc9972e933483b3.js"></script><br />
Now that we have dependencies added, we need to define a swagger configuration. Most of our endpoint will need to have a token passed as HTTP header, we need to configure our swagger configuration to facilitate that.<br />
<script src="https://gist.github.com/vavasthi/ad2765f27e9a68f09e923ff13fc2219b.js"></script><br />
We create a swagger configuration file as defined above. The <i>globalOperationParameters</i> clause dictates to swagger that each endpoint will have a parameter named <i>token </i>of type <i>header</i> and is mandatory. We know there is two endpoint that will not have a <i>token</i> header but we can just pass some dummy values. We can also make it non-mandatory if we so wish.<br />
<script src="https://gist.github.com/vavasthi/31a624f3bbf93581569249b983107cf4.js"></script><br />
We also modify authentication endpoint to take a <i>username</i> and <i>password</i> header for authentication endpoint. That is defined in the <i>AuthenticateEndpoint</i> as above. Look at the <i>@RequestHeader</i> parameter passed in the <i>login</i> method of the endpoint.<br />
<pre class="code"> web.ignoring().antMatchers("/swagger-ui.html");
web.ignoring().antMatchers("/v2/api-docs/**");
web.ignoring().antMatchers("/swagger-resources/**");
</pre>
<br />
Also, make sure that <i>SecurityConfiguration.java</i> that we had defined in one of the previous posts a list of swagger related URLs were made part of an exception list so that these do not go through authentication filter.<br />
Now if we rebuild the server and run it and visit <i>http://localhost:8081/swagger-ui.html</i> we see the swagger page with all the API endpoints listed there. Here is the screenshot of the swagger UI as seen on visiting above URL.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi65v4gi1h8DBqr_yJ6NJX0j0S94Qx04XdYfhePw5cKZT7M1xsaxhIcWMv64c5L1wFB5cpT4lgFkgCEjxHGCzXyVxXOxdowARKGP37VdS4ohOZbQE7hZD2CLryEs0I4-X4nIERlw8sLfujF/s1600/a.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="849" data-original-width="987" height="550" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi65v4gi1h8DBqr_yJ6NJX0j0S94Qx04XdYfhePw5cKZT7M1xsaxhIcWMv64c5L1wFB5cpT4lgFkgCEjxHGCzXyVxXOxdowARKGP37VdS4ohOZbQE7hZD2CLryEs0I4-X4nIERlw8sLfujF/s640/a.png" width="640" /></a></div>
<br /></div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-85919391525188501492019-01-04T19:55:00.002+05:302019-01-04T20:03:19.343+05:306. Introducing Spring Security<div dir="ltr" style="text-align: left;" trbidi="on">
If the web services that we are building require any level of authentication and authorization, it is better to understand Spring security context. Services that are properly built with security context can implement a better level of security at each method and endpoint level.<br />
We want to handle all the security, authentication and, authorization related code in a single block so that managing it becomes easier.<br />
The first step that we need to do is to add security dependencies in the pom.xml. <script src="https://gist.github.com/vavasthi/4a2be1f9b8efffe2c3e00a66e4fb7b66.js"></script><br />
For authentication and authorization to make sense, we need to have a concept of a Role within the system. The roles are required to authorize the users for specific purposes. We define a set of roles in the form of a bitmask defined as an enum.<br />
<script src="https://gist.github.com/vavasthi/802c9e57c43d63ed266cf092fd5e5ad5.js"></script><br />
Now that we have defined Roles, we need to add few fields in our <i>User</i> class. We add a field <i>mask</i> for holding allowed roles and two different fields for <i>authToken</i> and <i>expiry</i>. We also add an index in the User entity.<br />
<script src="https://gist.github.com/vavasthi/a264c8f311bce35f5b73c9c6f04e00d5.js"></script><br />
We also need to add a class that implements GrantedAuthority interface. An authority in the spring security system is represented by a string and we use the Role enum name as the authority in the system.<br />
<script src="https://gist.github.com/vavasthi/b9f1405a5184aaae9098d9713093c457.js"></script><br />
We probably need to have two different types of authentication and authorization within the server. The first authentication and authorization will work with username and password and the second one will work with the token. Spring security requires two different entities to be defined for authentication and authorization. We need a Principal and a Provider class each for username and token authentication and authorization. Let's first look at the class required for authentication using username and password. A principal is nothing but the abstraction of credentials that is flowing through spring security. We define it as below.<br />
<script src="https://gist.github.com/vavasthi/613430b9d834ceb7272fc87bb1d4848e.js"></script><br />
A provider is a class that provides the authentication and authorization functionality for a particular type. The username and password provider is defined as below. As we can see, the provider class extracts the credentials from the Principal and then makes sure the password is correct for the username. It also populates the list of roles granted to the user in the form of Authority.<br />
<script src="https://gist.github.com/vavasthi/269e3916080512eb4eb9c4e6f36ef10b.js"></script><br />
Similarly, we define a Token principal. This principal just contains the token value that is received from the user.<br />
<script src="https://gist.github.com/vavasthi/984d35a8a1bc9377a32b90efffc84473.js"></script><br />
We also define a token provider. The token provider looks up the user from the database by querying it from token and authenticates the user. In this example, we are making an implicit assumption that a token will be unique across all the users. If that were not to be the case, we will also have to pass the username or some other identifier for the user along with the token during authentication.<br />
<script src="https://gist.github.com/vavasthi/33a3437708e8ec0ba7296d00daf68330.js"></script><br />
We also define a class RequestContext that will contain a set of thread local variables so that once authenticated, we will not need to look up the user details from the database. We will only need to hit the database in cases where we need to update the record in <i class="">the User</i> table.<script src="https://gist.github.com/vavasthi/e8d62160c49d5483c71c226d7dc94fd4.js"></script><br />
We can see in both provider classes, after successful authentication of the user, the User is set into the thread local so that we can use it within the thread boundaries of that request.<br />
Now the providers and principals are in place, we need to start using them. Rather than using them on a case to case basis, the best way is to define a filter which applies on each request and authentication and authorization can be done on all requests.<br />
<script src="https://gist.github.com/vavasthi/808675f82ba69b5c1292bea40f114a30.js"></script><br />
AuthenticationFilter provides the <i>doFilter</i> method which is called for each incoming request. As we can see within the doFilter method, if we receive a <i>POST</i> request to <i>/authenticate</i> URL we assume that to be a username and password authentication and we process it as a username and password authentication, in all other cases, it is assumed to be a token-based authentication. All the three parameters, username, password, and token are expected to be passed as headers.<br />
Now we need to plug everything in so that it is called by the system. For that, we need to define a security configuration.<br />
<script src="https://gist.github.com/vavasthi/5bbdf2c0a5254a77ff6f339c9ff38096.js"></script> The <i>SecurityConfiguration</i> class is defined by extending <i>WebSecurityConfigurerAdapter</i> class. We override a few sets of methods. The first method defines a set of authentication providers that are available to the system.<br />
<pre class="code"> @Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
auth.authenticationProvider(tokenAuthenticationProvider());
auth.authenticationProvider(usernamePasswordAuthenticationProvider());
}
</pre>
<br />
The second method defines a set of URLs that would be ignored by the security system.<br />
<pre class="code"> @Override
public void configure(WebSecurity web) throws Exception {
web.ignoring().antMatchers("/manage/health");
web.ignoring().antMatchers("/swagger-ui.html");
web.ignoring().antMatchers("/webjars/**");
web.ignoring().antMatchers("/v2/api-docs/**");
web.ignoring().antMatchers("/error");
web.ignoring().antMatchers("/swagger-resources/**");
}
</pre>
<br />
The third method defines the security rules for all the requests.<br />
<pre class="code"> @Override
protected void configure(HttpSecurity http) throws Exception {
http.csrf().disable().
sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS).
and().
authorizeRequests().
antMatchers(actuatorEndpoints()).hasAuthority(Role.ADMIN.name()).
anyRequest().authenticated().
and().
anonymous().disable().
exceptionHandling().authenticationEntryPoint(unauthorizedEntryPoint());;
http.addFilterBefore(new AuthenticationFilter(authenticationManager()), BasicAuthenticationFilter.class);
}
</pre>
<br />
The configuration first disables <i>CSRF</i> and then sets a session management policy. After that, it declares that all the actuator endpoints need to be authenticated by a user having the ADMIN roles. Then all other users need to be authenticated. With this, every request coming into the system will be automatically authenticated and authorized. Now we can annotate our endpoints with specific roles so that unauthorized users can not make the calls. For example, we add the following annotation to our endpoint that we had defined earlier.<br />
<pre class="code">@PreAuthorize("hasAuthority('USER')")
</pre>
<br />
This annotation implies that a user with the role of USER can only call this endpoint. Any other user would not be able to call this endpoint.<br />
<script src="https://gist.github.com/vavasthi/b3cb0110679b7a077aa050813ec372b4.js"></script><br />
This is also our AuthenticationEndpoint. This will need to be called when a user wants to login using username and password. This now provides us with a functioning service with spring security enabled. In future posts, we will provide more details on it. The complete code for this post is available in github repository with version 1.0. Click <a href="https://github.com/vavasthi/springblog/releases/tag/v1.0">here</a> to download it. </div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-41876610841718161152018-12-19T11:14:00.002+05:302018-12-19T11:14:36.634+05:305. Replacing tomcat with jetty<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
By default when we run spring boot application, it runs a tomcat server. I have personally found tomcat too heavyweight and I prefer to use Jetty for writing a webservices backend.<br />
<pre class="code">2018-12-19 10:54:08.403 INFO 16029 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2018-12-19 10:54:08.424 INFO 16029 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2018-12-19 10:54:08.424 INFO 16029 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/9.0.12</pre>
<br />
<div>
To replace tomcat with Jetty, we need to make a couple of changes. The first step is to add Jetty dependencies in the pom.xml.<br />
<pre class="code"> <dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-deploy</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-rewrite</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
</dependency></dependency></pre>
<br /></div>
Since springframework boot bom automatically includes a dependency on tomcat in spring-boot-starter-web, we specifically need to exclude it. So we change<br />
<pre class="code"> <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
</pre>
<br />
to<br />
<pre class="code"> <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
</pre>
<div>
<br />
<script src="https://gist.github.com/vavasthi/948d61e25999f52e2d20b5c85edd1390.js"></script><br /></div>
The next step is to add a customizer for jetty which will allow us to set some parameters related to Jetty.<br />
As we can see the customizer uses a set of variables that we can override in <i>application.properties</i> file to modify the behavior of the server.<br />
<br /></div>
<pre class="code">2018-12-19 11:06:44.577 INFO 16219 --- [ main] o.e.jetty.server.AbstractConnector : Started ServerConnector@42ebece0{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2018-12-19 11:06:44.580 INFO 16219 --- [ main] o.s.b.web.embedded.jetty.JettyWebServer : Jetty started on port(s) 8081 (http/1.1) with context path ''
</pre>
Now the logs clearly show that in place of tomcat, it is running jetty.
</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-1017205028963917302018-11-28T10:47:00.000+05:302018-11-28T10:47:21.185+05:304. More on CrudRepository<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post, we used the <i>CrudRepository</i> interface. We only used the methods already provided by the interface. In this post, we look at how we can extend the interface with custom queries. Let's go back to our UserEndpoint. In the GET method, we can currently get the details of a user given his <i>id</i>. This is an impractical scenario. Most users' would not know what their <i>id</i> is. It is an internal identifier generated by our service and it makes no sense for our users.<br />
Let's say we want to extend the existing GET/DELETE interface to allow query by id, username, and email address. Since we intend to query the table using the username and email columns, we need some kind of index on those columns. Since these fields are expected to be unique, we put a unique constraint in the @Table annotation.<br />
<script src="https://gist.github.com/vavasthi/891461b7a6721e266712794432e52077.js"></script><br />
When this change is deployed, the corresponding changes in database schema would be as below.<br />
<pre class="code">mysql> desc user;
+----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+-------+
| id | bigint(20) | NO | PRI | NULL | |
| email | varchar(255) | YES | UNI | NULL | |
| fullname | varchar(255) | YES | | NULL | |
| password | varchar(255) | YES | | NULL | |
| username | varchar(255) | YES | UNI | NULL | |
+----------+--------------+------+-----+---------+-------+
5 rows in set (0.01 sec)
mysql> show create table user\G
*************************** 1. row ***************************
Table: user
Create Table: CREATE TABLE `user` (
`id` bigint(20) NOT NULL,
`email` varchar(255) DEFAULT NULL,
`fullname` varchar(255) DEFAULT NULL,
`password` varchar(255) DEFAULT NULL,
`username` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uq_email` (`email`),
UNIQUE KEY `uq_username` (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
</pre>
<br />
As we can see from the schema both the columns have unique constraints attached to them. The next step is to add methods in the UserRepository.java to query the user for a given username or email.<br />
<script src="https://gist.github.com/vavasthi/3b39721367222e9ba8489b1eb1fadd0d.js"></script><br />
Now that we have all the required changes in place, we modify the <i>UserEndpoint</i> to handle the changes. We first create a private method <i>retrieveUser</i> which will retrieve a user from repository given an id, username or email.<br />
<script src="https://gist.github.com/vavasthi/f16f9f48016645b4d5f197ce6d52621a.js"></script><br />
In this method, we take an argument of type String, we first assume it to be the id if id fails we try with email and then with the username.<br />
<pre class="code">$ curl -X GET http://localhost:8080/user/jdoe@example.com
{"id":6,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":"jdoe@example.com"}</pre>
Here is the complete UserEndpoint class after modification.<br />
<script src="https://gist.github.com/vavasthi/14b27041c70032305d628fa3723371ea.js"></script><br />
<br />
<br /></div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-59022691743238140312018-11-27T14:21:00.002+05:302018-11-27T14:21:38.091+05:303. All the methods in endpoint<div dir="ltr" style="text-align: left;" trbidi="on">
In the previous post, we defined a single method for an endpoint. In a typical web application, each endpoint will have multiple methods. To build an example of a complete endpoint, we need to have a persistence layer that can support these endpoints. To enable MySQL support for spring framework, we need to update pom.xml file with following dependencies.<br />
<br />
<br />
Now we need to define data sources for MySQL. The first step is to create a database and user in the MySQL database. To do that log in to MySQL using the following command.<br />
<pre class="tr_bq code">$ mysql -u root</pre>
Now run following set of commands to create a database named <i>tutorial</i> with username <i>tutorial</i> and password <i>tutorial123</i>.<br />
<br />
<script src="https://gist.github.com/vavasthi/f52ab44f6b921c1620fbd3dfb4100705.js"></script>Now we can initialize a data source in spring framework by putting properties in the <i>application.properties</i> file.<br />
<script src="https://gist.github.com/vavasthi/75f887272fd889d7cc7f0484ab9a536b.js"></script><br />
As we can see in the properties, the value <i>ddl-auto</i> is set to update. This will cause the database schema to be updated based on the definition of entity objects. This is a good practice while developing the system but normally should not be used while in production.<br />
Let's say we want to define a simple endpoint that supports an entity called <i>User.</i> We define the entity in Java as below.<br />
<script src="https://gist.github.com/vavasthi/a2df463bfc79ecf071c49d9da7c5e11e.js"></script><br />
As we can see, the key to an entity class is the annotation <i>@Entity</i>. You can also add an optional annotation <i>@Table</i> which will allow you control over the name of the table that gets created within the database. By default, if you omit the <i>@Table</i> annotation, the table name would be automatically generated.<br />
The next step is to generate a Repository interface for accessing the entity from the database. We typically create a subclass of CrudRepository.<br />
<script src="https://gist.github.com/vavasthi/d4a38d924da9e4f53e13e5145e5e9671.js"></script><br />
An empty repository interface provides sufficient functionality for this example. We will see how to add more functionality to repository class in a different post. The <a href="https://docs.spring.io/spring-data/commons/docs/current/api/org/springframework/data/repository/CrudRepository.html">Javadoc</a> of CrudRepository provides details of all the methods that are readily available without any addition to the interface.<br />
Let's run the server to see whether the server can connect to the database. As we run the server, we check what happened to our database.<br />
<pre class="tr_bq code">$ mysql -u tutorial -ptutorial123 tutorial
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 8.0.12 Homebrew
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show tables;
+--------------------+
| Tables_in_tutorial |
+--------------------+
| hibernate_sequence |
| user |
+--------------------+
2 rows in set (0.01 sec)
mysql> desc user;
+----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+-------+
| id | bigint(20) | NO | PRI | NULL | |
| email | varchar(255) | YES | | NULL | |
| fullname | varchar(255) | YES | | NULL | |
| password | varchar(255) | YES | | NULL | |
| username | varchar(255) | YES | | NULL | |
+----------+--------------+------+-----+---------+-------+
5 rows in set (0.01 sec)
mysql>
</pre>
<br />
The database has the user table created mapping our entity class to a table. Now we have confirmed that the server is able to connect to our database server. Now we go back to the old endpoint class that we created in the previous post and create a similar endpoint class called UserEndpoint. In this class, we will define all the methods required for managing users.<br />
<script src="https://gist.github.com/vavasthi/c65f41ec54de88a06e1013cf269cfb19.js"></script><br />
<br />
<pre class="code">$ curl -X POST \
> http://localhost:8080/user \
> -H 'Content-Type: application/json' \
> -H 'Postman-Token: d0298724-86d2-49da-a0c0-067ed9e18e1e' \
> -H 'cache-control: no-cache' \
> -d '{
> "fullname":"John Doe",
> "username":"jdoe",
> "password":"JohnDoe123",
> "email":"johndoe@example.com"
> }'
{"id":1,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":"johndoe@example.com"}
</pre>
We can also verify the record in the database.<br />
<pre class="code">$ mysql -u tutorial -ptutorial123 tutorial -e "select * from user"
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+---------------------+----------+------------+----------+
| id | email | fullname | password | username |
+----+---------------------+----------+------------+----------+
| 1 | johndoe@example.com | John Doe | JohnDoe123 | jdoe |
+----+---------------------+----------+------------+----------+</pre>
Now we can perform other operations on User record.<br />
<pre class="code">$ curl -X GET http://localhost:8080/user/1
{"id":1,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":"johndoe@example.com"}</pre>
<br />
<pre class="code">curl -X DELETE http://localhost:8080/user/1
{"id":1,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":"johndoe@example.com"}</pre>
<br />
Now we can run the database query again and see that the record has vanished.<br />
<pre class="code">mysql> select * from user;
Empty set (0.00 sec)</pre>
<br />
Now we create the user again by running the POST command. Since we are using database autoincrement, the id of the user will be 2 now.<br />
Now we change the email of the user using PATCH command.<br />
<pre class="code">curl -X PATCH http://localhost:8080/user/2 -H 'Content-Type: application/json' -H 'Postman-Token: d0298724-86d2-49da-a0c0-067ed9e18e1e' -H 'cache-control: no-cache' -d '{
"email":"jdoe@example.com"
}'
{"id":2,"fullname":"John Doe","username":"jdoe","password":"JohnDoe123","email":"jdoe@example.com"}</pre>
<br />
Here we have it. All the endpoints are functioning properly backed by MySQL as the database.</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-86902030990616196202018-11-20T10:34:00.001+05:302018-11-20T21:17:52.920+05:302. Adding an Endpoint<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
One of the primary reasons why one would be using spring framework would be to define rest based web services. In this post, we will see how do we add an endpoint to our server. First, download and install your favorite IDE. I prefer IntelliJ IDEA but you can pick the IDE that you are comfortable with.<br />
Let's open the project that we created in <a href="https://www.springframework.in/2018/11/starting-springframework-project.html">the previous</a> blog post in the IDE. The source root for the project is <Project Name>/src/main/java. Once you open that directory, you will see the package of the project that you created. Within that, you will see the main for the spring application. The only thing that we change for now is to add <i>scanBasePackages</i> parameter in <i>@SpringBootApplication</i><b style="font-style: italic;"> </b>annotation.<br />
<script src="https://gist.github.com/vavasthi/b6f09ef7a7a39b6eb0882e6b1d029879.js"></script><br />
For now, we can leave the main as it is. The next step that we want to do is to add an endpoint. Create a new java class in your project. I call it HelloWorldEndpoint. To define an endpoint, we need to do the following things. The endpoint needs to have a path relative to the server root. We also need to know what operations are needed to be defined.<br />
We will discuss the whole details of spring security later. So for now please remove security dependencies from your pom.xml. Following lines should be removed from pom.xml<br />
<script src="https://gist.github.com/vavasthi/1e2c77b5b0832fb6e171714f2d644105.js"></script><br/>
It helps to look at a REST endpoint as an object with operations used to define their lifestyle state changes.<br />
<br />
<ul style="text-align: left;">
<li>Path defines a class</li>
<li>Method GET is for reading the state of that object</li>
<li>Method POST is for creating a new object of that class</li>
<li>Method DELETE is for deleting an instance of that class</li>
<li>Method PUT is for updating the object</li>
<li>Method PATCH is for selectively updating the values.</li>
</ul>
<div>
Looking at our example, we are going to create a HelloWorld endpoint which will be used to manage the state of class Hello. Here is what class Hello looks like.</div>
<div>
<script src="https://gist.github.com/vavasthi/3e782e90f7f0ad1fa565ab2fbce3d045.js"></script><br /></div>
<div>
Now we look at our endpoint class. The class contains a single method GET which returns a temporarily created instance of Hello class.<br />
<script src="https://gist.github.com/vavasthi/439aaab8b2473ff2288dbf16fa2f3839.js"></script><br />
Now we build the project.<br />
<pre class="tr_bq code">$mvn clean package -DskipTests</pre>
</div>
</div>
<br />
And we can run the project.<br />
<pre class="tr_bq code">$mvn spring-boot:run</pre>
<br />
Since maven build creates a single jar for each project. We can also run the project with the java command.<br />
<pre class="tr_bq code">$java -jar target/tutorials-0.0.1-SNAPSHOT.jar</pre>
<br />
Most IDEs would allow you to build and run the project from within the IDE. You can do that as well. Now that the server is running we can test the endpoint that we had written. You can use tools like Postman but since it is a simple endpoint, we can just test it with curl command.<br />
<pre class="tr_bq code">curl -X GET http://localhost:8080/hello
{"message":"Hello","name":"World"}</pre>
As we can see the JSON that is returned corresponds to the temporary object that we created and returned. Of course, this is too simplistic and we will look at doing something more substantial in the next blog post. </div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0tag:blogger.com,1999:blog-2671448201212310231.post-21836132929451352122018-11-19T21:14:00.001+05:302018-11-20T13:33:56.702+05:301. Starting a springframework project<div dir="ltr" style="text-align: left;" trbidi="on">
The best way to start a spring framework project is to use the spring initializer to do that. Visit <a href="https://start.spring.io/">Spring Initializer</a> website. Provider values for group and artifact and select the dependencies that you want to initialize with. To start with you can start with minimum dependencies and then add them later on. These are just maven dependencies.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiLWWjPMSXKRDgJqknmaZgMRMLqGM1atk84iiFXPEu0nDzvsbxf0tNNjDZ80xH93SxpjA2FDhaiNY-PsB2QWVlDFQcm8UBNlfwvj8vL37_DCzba5cocfM3AHm0eYTxtnvpBR6icBbs6GKy/s1600/startio.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="848" data-original-width="1417" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiLWWjPMSXKRDgJqknmaZgMRMLqGM1atk84iiFXPEu0nDzvsbxf0tNNjDZ80xH93SxpjA2FDhaiNY-PsB2QWVlDFQcm8UBNlfwvj8vL37_DCzba5cocfM3AHm0eYTxtnvpBR6icBbs6GKy/s400/startio.png" width="400" /></a></div>
<br />
Here I have selected Web and Security to start with. Now we click on Generate Project. It downloads a zip file which is the starting project. We unzip the file and then we run following command.<br />
<pre class="tr_bq code">
$ mvn clean package -DskipTests</pre><br />
It will download the dependencies and then compile the project. The finally built jar file is in the tutorials/target directory and is named tutorials-0.0.1-SNAPSHOT.jar. The name is created using the version number defined in the pom.xml file generated. We can run the project by running following command.<br />
<pre class="tr_bq code">
$ java -jar target/tutorials-0.0.1-SNAPSHOT.jar</pre><br />
It will run the project and the following output is displayed. The last line tells us that the project launch is successful.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtiG1nQbgWEGgD3-acGyBYIzAXkEoTu13ACl18RnxqjWZfymyqi6XDkm1G2tSbBKlbsZsuxFnjOY4z3-E2D2fphI4NJeqSfQZf06RyXGDpShKFZJisp9eszJDS6QdV4ByrPKx4GJ3mkwjL/s1600/projectrun.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="955" data-original-width="956" height="398" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtiG1nQbgWEGgD3-acGyBYIzAXkEoTu13ACl18RnxqjWZfymyqi6XDkm1G2tSbBKlbsZsuxFnjOY4z3-E2D2fphI4NJeqSfQZf06RyXGDpShKFZJisp9eszJDS6QdV4ByrPKx4GJ3mkwjL/s400/projectrun.png" width="400" /></a></div>
<br />
So here it is. Our first springframework project.</div>
vavasthihttp://www.blogger.com/profile/02718490805722952012noreply@blogger.com0