Posts

Components of Big Data - Hadoop System

Image
In this blog i will explain important components which are part of Hadoop System. I will give very brief overview of these components. Below diagram shows very high level components in Hadoop system. Master Node (MN) Name Node (NN) It is a daemon process runs on Master Node. Takes care of reading the data file to be analyzed. Splits the data file based on block size configured, default is 64MB and 128 MB. Distributes the split data file across multiple Data Node. Maintains the index file to keep track of where the data has been distributed. Think this as "Table of Content" in a book. It provides input to Job Tracker for location of the data files in Data Node. This is one part of HDFS system in Hadoop. Job Tracker (JT) Job tracker is also a daemon process. This is part of Processing Engine of Hadoop system. It is responsible for running the program which will analyze the data and produce results. Job Tracker communicates with NN to identify the location of the the data file. ...

Load Testing of Rest Webservice Using SOAP UI

Image
SOAP UI provides easy integration with different web services like SOAP and REST along with feature of Load Testing. Below I will show how to setup SOAP UI for doing load testing. To start with you need to have SOAP UI installed in the system. In the example I will consume REST webservice. Web Service end point is http://localhost:9081/UserService/users Lets start with the load testing setup now. Start SOAP UI tool and setup the REST webservice call and do a round of testing to ensure that tool is able to connect to the service and is able to get the response. Following screens shows the same. Setup REST Project Create New Project Right click on the Project and click on create REST Project. After providing the URL in above alert box, You will get below windows where you can see the various options like Method Endpoint Resource Input parameter Once you provide all the necessary configuration required for your webservice do test and once you see the expected response in the output window...

Basic Authentication Setup Java Web Application

Image
This blog shows how to setup Basic Authentication in Web application. I am using Spring based Web Service to demonstrate the same. To start please ensure that you have spring application configured properly. On top of that i will show what changes needs to be be made to enable Basic Authentication. Securing application is one of the important activity which developer and designer has to keep in mind while designing. Basic authentication can be one of the basic security mechanism which can be enabled to secure web application or web service. Basic Authentication security is where application will expect the consumer to pass User and password in request header. In case if these values are not passed then spring framework will throw back Unauthorized error code. Following are the steps which needs to be followed. Step1: Add the spring security jar files in the application. Following are the jar files. spring-security-core.jar spring-security-config.jar spring-security-web.jar Step2: Add f...

GIT vs SVN

GIT vs SVN In the DevOps world, selection of right SCM tool is a significant decision. Since these tools are kind of driving force behind the process, a wrong SCM selection may lead to inefficiencies throughout your delivery pipeline. I personally have used CVS and SVN in most part of my career. Those projects were not really Agile projects. While we had some kind of CI and CD, there were no end to end DevOps process in place. So we were able live with it. Recently I started using GIT - both private GIT as well as public GITHUB and BIT BUCKET.  Also in the recent assignments, we don't just go with what is provided by customer. We demanded the right SCM. We study the customer needs and recommend centralized or distributed SCM.  In this blog, I will  compare the capabilities of GIT and SVN at very basic level. This blog does not cover the detailed scenarios like when GIT performs better over SVN and vice versa or how GIT's way of working on the content instead of file helps...

Mockito setup in Spring application

Mockito is used widely for application code Unit Testing. In this blog I will show the dependencies which are required to configure Mockito in Spring Web application. In the existing Spring based application, following dependent jar files needs to be added. asm-1.0.2.jar hamcrest-all-1.3.jar json-path-1.2.0.jar json-smart-2.1.0.jar junit-4.11.jar mockito-core-1.9.5.jar objenesis-2.1.jar after adding these jar file you are all set to start writing the Mockito based JUnit test cases. Few of the useful plugin for Eclipse IDE that will help in analyzing code coverage can be installed. The plugin which I used for code coverage is "EclEmma". This can be installed from Eclipse market place which gives report on percentage of code line which has been covered by Unit Testing.  In subsequent blogs i will show some sample Mockito Unit Test cases classes.

Use Swagger UI for showing interactive API documentation

Image
Following Swagger post I showed how to configure Swagger with Spring project . In this blog I will demonstrate using Swagger UI for showing interactive documentation. Swagger UI is part of the Swagger project and it allows you to generate, visualize and consume API. It is package which basically consist of html, css and javascript, and parses the json output generated from application code scanning. This scanning and configuration was explained in following post. Swagger UI Setup can be downloaded from " here ". Copy the files to a Dynamic Web project in eclipse and paste the content from "dist" folder to WebContent as shown below. Here I am using tomcat server to host this UI application. By doing so it will be easy to share the API application with other user. It shows following UI.  Please note that by default swagger will load the petstore Api details. I will cover customizing them in next point.  Swagger UI presents very nice interactive tool, using which users...

Reading log file path from environment Variable in logback

In this short blog i will be showing how to read the log folder location from environment variable. In general log path for an application is one time configuration. So why to keep multiple property files for each environments. Better approach could be using environment variable on server, which can be configured once. Here i am using Tomcat 7.x as a server. For Logback using following jars. logback-classic-1.1.3.jar logback-core-1.1.3.jar First lets start by creating environment variable in tomcat server. For this you need to open context.xml file located at TOMCAT_HOME/conf folder. Open with text editor and add following line in the file. <Environment name="LOG_PATH" type="java.lang.String" value="/logs"/> you are done with creation of environment variable which will be accessible via jndi lookup. next will be open logback.xml file which contains application logging configurations. Use JNDI lookup to retrieve the environment variable. Logback pro...