MongoDB Tip 2 : Do check for errors

MongoDb stores the data in documents which are BSON format. BSON is a binary format just like the JSON format.

Operational speed in MongoDb is quite higher then SQL DB’s. The reason lies in the way MongoDB stores the data and retrieves the same.

By default most drivers do asynchronous, ‘unsafe’ writes – this means that the driver does not return an error directly, similar to INSERT DELAYED with MySQL.

If you want to know if something succeeded, you can check for errors using getLastError.

For cases where in case of something wrong you want to throw an error, it’s simple in most drivers to enable “safe” queries which are synchronous.

This helps as a bridge for those who are comparatively new to MongoDB.

 

MongoDB Tip 1 : Things to consider while selecting a shard key

Sharding is a phenomenon used by Mongo DB for the scaling purpose.

While selecting a shard key for sharding we should consider following points:

Distribution of reads and writes

One important aspect is the distribution of reads and writes. If you’re always using a single machine, then that machine will have a high probability of write locks,which will ultimately make the write cycles longer in time. It doesn’t matter how many machines you have in total, as all the writes will go to the same place. This is why you should never use the constant increasing primary id's or a time stamp as the shard key.

Similarly, if all of your reads are going to the same replica set, then you’d better hope that your working set fits into RAM on one machine.

Size of chunks

Another important aspect to be considered is the chunk size. MongoDB splits large size chunks into smaller ones where the shard keys are different. Large number of documents with the same shard key you end up with huge chunks which create problems while transferring the data among shards.

ALL ABOUT TIMEZONE: PART 2

After going through the time zone concepts in Part 1.We will be implementing the concepts technically. So let’s give it a start.

We need to create a table in our database for several time zone and user related entries.

TABLE 1: dbo.tblTimeZone

This table will be a master table containing all the info about different time zones with following columns.

  • TimeZoneID: primary key.
  • TimeZoneName: stores name of the time zones i.e. IST, BST, EST, MST, and CST and so on.
  • LagInHours: This column will contain the difference from UTC in hours.
  • LagInMinutes: This column will contain the difference from UTC in minutes.
  • DST: This will be a Boolean column that will tell us whether the time zone follows DST or not.
  • LagInHoursWithDST: This column will contain the difference from UTC in hours including DST if it follows DST.
  • LagInMinutesWithDST: This column will contain the difference from UTC in minutes including DST if it follows DST.
  • DSTStartDateTime: Date time from which the DST will be starting.
  • DSTEndDateTime: Date time from which the DST will be ending.
  • DSTYear: Year for which the DST is under action.

 

This table will be filled with different entries of time zones, DST column will be true if the time zone follows DST and then only LagInHoursWithDST and LagInMinutesWithDST will be having entries for a DSTYear.

Every year a new entry will be made for all the time zone that follows DST with new DSTYear.

Let me show you a table having entries for few time zones so that you can get a better idea.

WP2

So as we discussed in the previous post. Denver and Arizona both fall in MST but one follows DST and the other does not, for such cases we will be making entries in the time zone table in the above manner.

You can also see for different years DST starts at different time so for handling that we just need to refer the current year from DSTYear column of the time zone table.

TABLE 2: dbo.tblUsers

This table will contain the relationship of users with the time zone to which they belong. From front end it can be handled in various ways during user creation. Web applications do provide a module for the user creation. While creating a user put an extra dropdown that will contain all the time zones and this way we can map the users with the time zones.

In backend the tblUsers will look like below.

WP3

We can see that in this table we will be specifying a user against a time zone.

Method:

We will need a method in the application for converting the dates based on time zone for saving as well as displaying purpose.

For saving to the database we will convert the date time fields to UTC from the user time zone and for displaying after bringing the dates from the database we will convert them to the user time zone.

It will be good to create a utility method like GetDesiredDateTime.

This method will expect 3 parameters:

  • DateTime under action.
  • Current time zone.
  • Desired time zone.

Signature:

GetDesiredDateTime(dateInput, CurrentTimeZone, desiredTimezone)

 

Like for saving a date time field which is in IST to UTC, we will use this method as: GetDesiredDateTime(“20-10-2014 19:00:00”,”IST”,”UTC”)

And this should result in “20-10-2014 13:30:00”.

Similarly GetDesiredDateTime(“20-10-2014 15:30:00”,”UTC”,”IST”) will result in “20-10-2014 21:00:00”.

The application will first retrieve the current logged in user credentials and will fetch time zone out of it. If the user is not registered or in case this is a reporting system then the user time zone can also be fetched from the browser and should be stored in some kind of session variable.

Internally this method will call a procedure and this procedure will do following steps.

While converting date time from User Time Zone to UTC:

Find out that the current time zone (input field) follows DST or not from dbo.tblTimeZone.

If Yes then check for the current year. If not then take usual lag hours and minutes.

After finding the current year, check if the current date time is lying between the DSTStartDateTime and DSTEndDateTime.

If yes then take the lag hours and minutes of the DST. If not then take usual lag hours and minutes.

Finally you will be having the lag hours and minutes. Now you just need to subtract these hour and minutes to the input date time field and this will give you the UTC date time that you can save to the data base.

While converting date time from UTC to User Time Zone:

Find the lag hours and minutes from the desired time zone in exactly the same way as above and finally just add them to the input date time field.

Graphical Representation:

WP4

Instead of subtract do addition when converting from UTC to logged in user time zone.

So this is how we can handle time zone in web applications.

Again with different requirements of different applications we may have to change few logics but the core substrate remains the same.

Thanks for giving it a read.

ALL ABOUT TIMEZONE: PART 1

In this post we will be discussing about time zone, its effect on applications, way to efficiently implement the time zone and much more. So let’s start.

What Time zone actually is: As Wikipedia says “A time zone is a region that has a uniform standard time for legal, commercial, and social purposes. It is convenient for areas in close commercial or other communication to keep the same time, so time zones tend to follow the boundaries of countries and their subdivisions. ”

So basically different sub regions follow different time zones like India follows IST , Britain follow BST and like these there are many more. We consider UTC as the base time zone and define others, in terms of hours lag, on a negative to positive scale of UTC. Like IST (UTC + 5:30 hours).

 

What do developers care about?

When we are developing an application which is going to cater a large set of users located across different regions of world, we surely need to handle time zone in a very effective way. Even if we are targeting a single country we have to take time zone into account as even in a single country we can have more than one time zone.

Date time in applications is used to record the exact moment of the occurrence of events or transactions. All the functionalities which are dependent on date times should be built taking time zone effects in mind.

 

Scenarios:

Suppose few end users sitting in different time zones are navigating through the volume or managerial reports which contain some date time entries. Respective users should be able to see the date times as per their time zones. If a task has created date time as 25 Oct 9 pm for an IST user then it should be shown as 25 Oct 3:30 pm to a UTC user, so that the users have an exact knowledge of the occurrence of events.

If a particular task is bound to SLA and an end user doesn’t get its correct origination time because he/she is sitting in a different time zone then it may create a hell lot of problem.

Solutions:

We have to take care of the date time fields in an application at two points to efficiently implement the time zone functionality.

  • Handling all the transaction related to the date time and saving them to the back end.
  • Showing the date time fields to an end user.

Transaction with Date time fields:

For transactions with date time fields we have to take care of time zone. It may be implemented by making all the transaction in a desired time zone and then finally saving the fields in the back end after converting them to UTC.

What is desirable!

Selecting a time zone for the transactions depends on the business logic that an application serves. We assign a time zone to an end user based on the needs of the application.

Suppose there is a very simple app whose end users may be sitting anywhere on the earth. We may directly find the time zone of the end user through its browser and make all the transactions with the user time zone and then finally convert the date time fields in UTC and save it to the database.

But what if the structure of the application differentiates its users based on some facts other than their individual time zone. For example suppose there is an application that divides the whole process based on regions like NASA and EMEAC or like any other way suppose the application has few groups say randomly A ,B and C. Now all the End users that has to deal with group A will come under IST, all in B will come under EST and those under C will come under MST. So in this case all the transaction will occur considering the group time zone. For example users create a task in group A so the task creation dates time will be according to IST even if the user is sitting in Arizona which follows Mountain Standard Time (MST).

Showing the date time fields to the end user:

It’s relatively an easy task we need to find out the time zone of the logged in user and convert the date fields retrieved from database from UTC to the user’s time zone.

Is time zone completed conceptually?

Yes almost except DST (Daylight Saving Time)

So what is DST?

As Wiki says it is the practice of advancing clocks during summer months (that feature more daylight) so that people get up earlier in the morning and experience more daylight in the evening. Typically, users of DST adjust clocks forward one hour near the start of spring and adjust them backward in the autumn.

For more info on it do visit DST section of www.timeanddate.com.

The list of DST for various countries around the globe for the year 2014 can be found in this link:  http://www.timeanddate.com/time/dst/2014.html

Plain meaning and effect on time zone: Suppose in London the clocks are moved forward by one hour from 30 Mar 2014 2 Am to 26 Oct 2014 1 Am.

sdaf

It means in 2014 London will be following GMT or UTC + 1 hour from 30 MAR 2 Am to 26 OCT 1 Am and outside this frame of time it follow UTC.

Conceptually it may be easily understood but technically there are many questions that do arise in mind like:

  • How to handle DST along with the time zone concept.
  • Will the clocks be moved 1 hour every time?
  • Are these dates constant or these dates vary every year and if they do vary then how come a developer will get to know that from when to when the DST will be applied.
  • Does every city or country inside a time zone follows DST or not and if not then how we are going to handle it for individual countries?

Technically for implementing of complete time zone concept we will be discussing everything later on but for few conceptual clearing following points will help:

  • DST can be easily handled alongside time zone. Whenever we will be converting the date time fields from UTC to a time zone say BST or EST, we will put an extra check on the current date time and if the time zone is under DST at the current time then we will take into account of that extra one hour added because of the DST.
  • No the clocks will not always be moved by one hour. In past there are years when the clocks are moved from 30 minutes to even 2 hours.
  • These dates are not hard coded dates and they may vary every year. An application is not built for just one year every year so we have to take care of DST very year for our application. Ideal case would be if we can create a configurable system that can detect the start of DST but unfortunately there isn’t any pattern that can be applied for the complete world. Some say that it will start from 2nd Sunday of March and will end on first Sunday of November but this is not right in context of all the countries.
  • Every country does not follow DST like India does not follow. Even countries /cities within the same time zone do not necessarily follow DST. For example Denver and Arizona lies under MST (Mountain Standard Time) but Arizona does not follow any DST whereas Denver does follow it.

It’s always good to have knowledge of the challenges that one is going to face while accomplishing a task. So now we have adequate knowledge of time zone plus DST and a challenge to implement it technically.

Will soon be coming up with the technical implementation of the Time zone concepts.

 

Kendo Ui Tip 6 : Customized View through Querying Datasource

Creating a customized view is highly desirable in web applications .

Kendo UI too provides a way to query the datasource through query method.

Available operations are paging, sorting, filtering, grouping. If data is not available or remote operations are enabled, data is requested through the transport. Otherwise operations are executed over the available data.

Example :

View from page 2 with 10 results and data sorted by EmployeeID

dataSource.query({page: 2, pageSize : 10 , sort :{ field : “EmployeeID”, dir : “asc”} }):

This is how we can query datasource and can create a customized view.


					

Kendo Ui Tip 5 : Retrieving Kendo Grid State

In Kendo UI we can retrieve the state of a Kendo grid and can use the same for many purposes like suppose at a point of time you need to cache the state of the grid so that when you come back to the page you can see the grid in exactly the same form in which you left it.

Finding a grid state is quite easy. Once your grid is created create the object of your kendo grid and find the it’s datasource.

         var grid= $(‘#yourGridID’).data(“kendoGrid”);
         var dataSource = grid.dataSource;
From the datasource we can create the grid state like below.
   var gridState = {
                    page: dataSource.page(),
                    pageSize: dataSource.pageSize(),
                    sort: dataSource.sort(),
                    group: dataSource.group(),
                    filter: dataSource.filter()
                 };
This gridState take accounts of following aspects of the kendo grid:
  • The page size of the grid.
  • The selected page of the grid.
  • The current state of sorting i.e the columns with whom the grid is sorted currently.
  • The current state of grouping i.e the columns with whom the grid is grouped currently.
  • The current state of filtering i.e the columns with whom the grid is filtered currently.
This is how we can retrieve the state of  a kendo grid.

Performance in MongoDB

Performance is a key and very vital feature of an application. Ageing of applications , dataload and high throughput are the factors which effect the performance of an application and can bring it down steeply. In this post we will be discussing the Performance aspect in MongoDB.

Increase in datasets and size of data in the server can bring the performance of an application down because of few factors like increase in query execution time which will directly increase the time for which the server remains busy.

Many times we have heard that MongoDB is faster then relation databases. It’s because of the unique way in which Mongo can store the datasets in different data servers and can effectively query them. Mongo uses Sharding to achieve this.

Sharding :

Database Sharding is a highly scalable approach for improving the throughput and overall performance of high-transaction, large database-centric business applications.

Sharding is a  unique way of horizontal scaling in which we distribute the data over different servers known as shards. Each shard is a single independent database which can be used to store the datasets. The collection of these shards form a logical database. Chunking down big database into small shards will ensure high availability and data consistency.

How It Works :

In MongoDB, to retrieve the whole entity, Mongo instance follow below procedure:

  •  One index lookup on the collection (assuming the entity is fetched by id)
  • Retrieve the contents of one database page (the actual binary json document)

Possible question arise in mind that how do a Mongo Instance knows that which shard out of many will be containing the relevant dataset which contains the information it is looking for.

When the sharding is done and a cluster of shards is created , all the metadata about the distribution of the datasets is stored in Config servers.

So the Mongo Instance also known as Query routers uses the metadata stored in the config servers to target operations to specific shards.

WP

So this is how we handle Scaling in MongoDB.

Benefits :

Through this partitioning there is a ensured enhancement in the usability of RAM.

Horizontal scaling or sharding automatically handles the load balancing between the shards(Will be next thing).

Load Balancing : 

Initially created shards are created with equal sizes but as the chunk size increases there are possibilities of unequal load among shards. May be one shard grows huge in size in comparison to other shards.

This is big problem because if chunk size of a shard increases to a big size then complete concept of sharding will be of no use as only one shard will be getting handling the most execution and others will be relatively free .To overcome such situation MongoDB has a process of  maintaining a balance between shards.

A balancer is a entity of MongoDB which is responsible to maintain the balance between shards by transferring chunks from a crowded shard to a less crowded shard.

Balancer transfers a big chunk from a shard(Origin Shard) to another less crowded or relatively free shard(Destination shard). It transfers the documents of Origin Shard chunk to destination shard chunk. Destination shard takes care of the changes done to the documents and implement them to the documents after the migration is done.

What happens to the mapping ? As i told earlier that the information about the address of a document that a MongoDB instance is looking for is stored in the config servers, Now as and when a chunk gets transferred from one shard to another the routing will go off.

Balancer only handles this situation as well , once the migration is done balancer modifies the Metadata ,related to the location of the chunk ,stored in the config server so that it remains relevant.

 

This is how the performance is scaled in MongoDB.