High Availability Architectures (3/4) – Performances
In the previous posts of this serie we’ve addressed the availability and scalability aspects of high availability systems. In this (rather lengthy) one we’ll focus on the performance side of things.
Again, performance is something to be contextually defined quite early in the project. For instance a requirement such as “3s response time” is not precise enough. “3s response time with 200 simultaneous users” is a valid requirement.
History
Performance has been a common issue for the last decade or so with the emergence of multi-tier IT systems. It has not been such a problem in the past or rather it has been addressed as the core issue and fixed once for all during the mainframe years (65-89). It also has been skipped during the client-servers years (90’s) as no real software architecture was in place. Back then, no-one would react if SQL code was found in the presentation layer.
Nowadays, the standard 5 layer software architecture (Client, Application, Business, Integration and Resources) has naturally emerged as the de facto solution. The basic principle is to ensure for isolation between layers to provide the software architecture with a greater modularity and robustness. With the considerable litterature available, architecture software development constraints are much more pregnant – and that’s an excellent thing.
However, such complex software architecture requires to be extremely cautious from the very early stage of architecture and design as a laidback appoach can prove to be quite expensive in terms of performance.
Usual Suspects
From experience, when it comes to performance, the usual suspects in this 5 layers architecture are :
- Integration tier
- Database tier : Execution plans, indexes, table schemes, DBA early in the process
- Business and Application tier : Algorithm, transactions, resource connections and APIs
Integration tier
Java Enterprise frameworks (Hibernate, Entity Beans) have been developed to abstract data persistence from the application developer. On one hand that’s a good thing : application developers can then focus on their own business problems and forget about SQL.
As a result these frameworks usage may not optimised and it could happen that application developers lose control on what they actually do, the number of objects in memory etc … The solution is to keep a fine grained control on these frameworks and the SQL that is actually generated and executed behind the scene. A close collaboration with a experience DBA (Database Administrator) is strongly recommended here, from the early steps of design.
For information, it’s worth mentionning that big internet giant such as Amazon for instance just don’t use such frameworks in order not to lose control on piece of processing that actually is critical in terms of performance. Read this article on that very subject.
Databases tier
Another drawback is that SQL not being at the center of development concerns, we may end up with applications that are not really optimised from the database perspective : database schema and queries execution plans are inappropriate and inefficient.
Then again it is recommended to have a DBA involved from the very early stage of the project (architecture and design) to validate the database schema and suggest execution plan to optimise the database usage.
Stored Procedures
At first, stored procedures may be considered as antiquities inheritated from the client/server years, souding like heretic software components from the 5 layer perspective.
However, from a genuine performance perspective they still are top drawer solutions. Data handling on the server allow to save a lot of time for applications and on the network.
The aim here is obviously not to develop the whole business layer as sored procedures but, rather, to optimise a few heavy process. Besides it proves to be an excellent antidote to the “DB request machin-gun” anti-pattern.
Application and business tier
There are a few standards Java dos and don’t to improve algorithm path length and CPU/memory usage.
The most important ones though regard sessions, transactions, APIs and resource connection
- Avoid stateful component whenever possible (EJB Stateful sessions are performance killers). Whenever handling sessions (e.g HTTP Session) make sure they don’t contains too many objects.
- Transactions must be kept as short as possible.
- Resource connections are very precious asset. There use should be optimised to the actual need and kept as short as possible.
- Whenever developing services API, always develop both unitary treatment service (i.e perform one action for one given item) and multiple treatments interface (i.e execute multiple request with multiple inputs and return multiple replies). This basic principle allows to save a lot of time by avoiding machin-gun request to the service.
Lastly, whenever building a new enterprise system using platform such as JEE or .Net, we need to keep in mind that these were made for real time process, not for long process. So if the application you are developing takes times (i.e above 5 sec), this component must not be deployed on the application server : it will uses precious resources for long process that are not used in the meatime to serve real-time services. Such a component should be deployed as a stand-alone server instead.
The Asynchronous trick
This is a great tool to load balance. Whenever addressing long process issue, it always is worth considering deploying it as an asynchonous service. This is a very natural and elegant load balancer mechanism that does not impact real time service QOS.
The biggest problem here is to convinced the business to accept to implement some asynchronous solution for rather peripheral services (e.g Reporting).
JMS is the natural solution whenever developing asynchronous solution in the Java world. One has still to be careful when choosing a JMS implementation.
A basic rule of thumb is that whenever you’re dealing with transactional messages with high availability and robustness concerns, you should rule out JMS implementation without persistence mechanism. JBoss JMS for instance is not as mature as JBoss Servlet or EJB implementation. Today the best solution on the market for JMS is SonicMQ.
Related posts:
Tags: Amazon, Database, EJB, HA Architecture, Hibernate, JBoss, JMS, Sonic MQ, SQL, Stored Procedures
Cecil, can you elaborate on why performance issues happen i) in the integration tier; ii) in the database tier – rather than on upper layouts? Thx for your clarifications man.
I certainly can Jeremy.
Basically : the bottleneck in terms of performances but also development and complexity in any enterprise application is the Resource Access and the I/Os. In our 5 software layers architecture, the access to the I/O is the integration layer while the database actually is the resource access.
The main objective of development frameworks for enterprise systems, such as EJBs and Hibernate, is to save the developers from being bothered with SQL and actual Database Physical implementation.
The aim is for application developers to only focus on the business part of the development so that they can bring a real added value to the development. This type of proposal sounds like heaven choir to management ears : “developers are more productive and can really focus on business problems”.
The problem is that these frameworks are extremely heavy (Entity EJBs are a disaster in that respect) or abstract too much complexity (Hibernate) to the user.
While developing it’s great : you save a lot of time as the gory details (SQL request, DB Tables implementation) are hidden by the framework. It all works well on the developer local version of the server.
The problem appears when the solution is tested on a server with many concurrent users.
The actual interface development with the database is completely delegated to these tools. You end up with a layer of software automatically generated that hardly is optimised. You could have (i’ve seen this) thousands of objects incorrectly read from the DB flooding your memory and making the application slow as hell for all users for a mere delete. You also get SQL requests that are not optimised, tables that are not properly indexed etc …
Médéric told us that whenever he is asked for consulting mission regarding Performances, experience has shown that these always are the main suspects, in that order. Integraation as common mistakes are made there and database as this is the place where to search for potential improvements.
Application problems may be in the form of multiple requests (DB, Network, etc …) while a single global request would have done the trick. Standard development design patterns for distributed applications must be applied here (Facade etc …). He gave us an example of an application doing 19 networks call to a Web Services. Changing the API of the service to have the correct granularity have saved 80% of them reducing the average request time from 25s down to 3s.
Thank you for your time Cecil. As usual, you explanations just made everything crystal clear to me.
Very good article and explanation. Can you write about wordpress performance as well?
Hey WP I almost believed it !
Okay it’s Xmas Time so I wont flag as spam, but it’s just for this time.
[...] In the previous episodes of this serie, we’ve addressed the Availability, Scalability and Performances aspects of HA Architecture. In this one we’ll concentrate on the future of these architectures [...]