Share via

BizTalk Server: Performance Tuning & Optimization

Introduction

Performance tuning is a bit more art than science. Not every single tuning technique solves every problem, and very often techniques interact. The approach should always be methodical, including baseline and differential testing particularly when justifying capital expenditures.

Performance limitations should be viewed as a series of roadblocks. Removing one roadblock doesn’t clear the road, but simply makes it easier to drive. Not until all of the roadblocks have been removed does performance truly jump.

A word of caution. It’s important to understand each tuning technique, both what it does and why it works. Running application of the included scripts, registry files, etc. is not recommended and could lead to serious consequences for the target system. It is not recommended at all to change the settings to the production server directly. It is always recommended to perform the changes in UAT or DEV and then perform the changes in production in controlled manner following change process of your enterprise.

In this article we are trying to keep everything in one place which are related to performance tuning and optimization for Microsoft BizTalk Server. Most of the concepts are valid across BizTalk Server versions, however it is always recommended to refer the product documentation of applicable version. 

Few Problems 

At some point of time, we may face issues in our BizTalk farm and then we think about ways to identify and resolve those issues and we start searching for BizTalk Performance Tuning and Optimization options. Those issues could be one of following or somewhere similar: 

  • BizTalk is performing slower,when we drop the messages or during high volume of messages.
  • Large no of messages are causing memory errors in BizTalk farm.
  • Prolonged BizTalk Throttling.
  • BizTalk receive location is not picking up the files.
  • BizTalk Database Jobs are taking too much time to complete.
  • Response time of message processing is increasing. There is a high latency in environment.
  • It seems BizTalk is not able to process any message, The messages or service instance are in Active/Running State, however nothing is happening.
  • SOAP Adapter is taking long time to call a Web Service.
  • Orchestration/Receive Port is taking  long time to debatch a large message.
  • The BizTalk production farm has suddenly started behaving in a strange manner.
  • High Memory/CPU utilization of the BizTalk servers.

Few Requirements

It is true, at some point of time, we start getting queries, thought or requirement similar to below and then we look for options to scale or optimize the BizTalk Server:

  • Looking for creating scalable BizTalk solutions.
  • Trying to achieve a particular latency for a BizTalk application.
  • Requirement to process 1000 messages in 1 second by a BizTalk application.
  • Requirement of processing large messages in BizTalk Server.
  • Planning to change the performance throughput.
  • Trying to determine/test maximum sustainable throughput ( MST ) of a BizTalk farm.
  • Trying to figure if BizTalk farm can support next 20+ applications which are going to be deployed in the future.
  • How should I decide if My BizTalk farm needs to be tuned or optimized?
  • How should I decide if BizTalk production farm needs to be scaled-up or scaled-out?

What Should We Do

First Few Steps

If  you have anything similar in the list above (Few Problems), then we should start with doing below things:

  • Check SQL Server Agent and BizTalk Server Jobs Status of the BizTalk SQL Server Instance - The agent should always be running and all the BizTalk Jobs should be enabled from the day one in your production Farm, If there are any issues refer this in-depth Wiki , "BizTalk Server:Databases: Survival Guide".
  • The BizTalk Host instance that has the "allow Host tracking" option set is stopped - If the host is stopped, then it should be started without any delay. As a practice there should always one dedicated "Tracking Host" should be running in BizTalk Farm. Please refer "Configuring a Dedicated Tracking Host" MSDN page for more details.
  • Record count of the Spool table- the row count in the tables should not be too much or it should not keep growing. Check the record count of the Host Instance Queue tables. Refer "Understanding Host Queue Tables" article for more details.
  • Free space of the Data and log file drives on SQL Server, as there should be enough free space in the log and data files drive on SQL Server.
  • Size of BizTalkDTADb database - Need to make sure the size is not too much of this database, It is a standard practice to keep an eye on the size of this database. Refer the following articles:
  • Excessive Disk I/O Latency (on SQL Server) - If the incoming rate of data into the MessageBox database is faster than which the system can process and move the data to the BizTalkDTADb database, backlog can build up in the MessageBox database. If the backlog continues to grow unabated, this is a very serious problem and system performance will degrade over time. One way to alleviate this problem is to install faster disks and/or upgrade the hardware to help ensure that the system is capable of recovering from any message backlogs experienced over time.

Review of Solution Design

Start reviewing the application design and architecture. Sometime a design change may have a huge impact over the application performance. The following should be the starting reference points while reviewing the design and code of BizTalk solutions:

  • Code review of Custom Pipeline component.
  • Are we using XMLDom to load a very large message in pipeline ?
  • What is the method used to read the values in an incoming message , Use streaming while reading the message or use XpathReader ?
  • Code of Orchestration?
  • Design ( use of functoid or inline xslt) of MAP used in transforming for large messages.
  • Are we using Enterprise Integration Patterns or not while designing enterprise solutions?
  • What kind of Custom Code we are using?
  • Can we use inline send port instead of normal send ports to achieve low latency solutions?
  • How we are calling Business Rule engine in our BizTalk Orchestrations?
  • How many scopes have been used in orchestration?
  • What kind of instrumentation or tracing is used?
  • Use of BAM API's or TPE.

References for Code Review/Solution Design

Following articles might be useful while reviewing the solution architecture and the BizTalk Code.

Review Everything Else!

It is also very crucial to review BizTalk environment including SQL and everything else which is your control. The following could be some initial points to start with:

  1. Hardware configurations of the BizTalk Server Box, and yes its not only about looking RAM and CPU there are other things as well like antivirus settings, hyper-threading etc. It is very important to decide these things while designing the farm. Please see below reference links for more details.
  2. Hardware and SAN configuration details of SQL Server instance. Choosing right RAID level of SAN disks really matters!!
  3. Network settings can be crucial. Check for instance this article from Sandro Pereira.
  4. Load balancing of BizTalk Web Services really matters while designing low latency solutions. Need to check options of IIS tuning.
  5. Setting BizTalk databases and log file drive size really matters need to review them. See below references for more details.

References

Below are few references, They have tons of information on performance optimization and tuning of BizTalk.

MSDN

MSDN Social

  • MSDN Social Forum for the discussion with Community members - Please refer this forum for posting a new Question or reading about ongoing discussions.
  • MSDN Social Forum - All threads related to BizTalk Performance Tuning & Optimization

Tools

There are lots of tools described in "Performance optimization guide" by Microsoft and "BizTalk Server 2010:Tools" wiki, however below are few tools which are very common and it is good to be familiar with them:

Blogs

Following are few community blog posts which might be helpful while working with tuning & optimization.

Hot Fixes

Following are few product hot fixes which may be helpful, But it is strongly recommended to read the KB article first and understand if they are absolutely necessary for the product.

  • KB2627804 - Hotfix that improves the performance of the tracking feature in BizTalk Server
  • KB970070  - Support for Locked Pages on SQL Server 2008 R2 Standard Edition x64, on SQL Server 2005 Standard Edition 64-bit systems, and on SQL Server 2008 Standard Edition 64-bit systems
  • KB944426  - Orphaned cache instances may be built in the Instances and Hosts Queue tables of the BizTalkMsgBoxDb database in BizTalk Server 2006 and in BizTalk Server 2006 R2
  • KB943165  - A CPU usage spike occurs on all the BizTalk hosts after you install security bulletin MS07-040 on a server that is running Microsoft BizTalk Server 2006 or Microsoft BizTalk Server 2006 R2
  • kb2387313 - FIX: Poor performance occurs when you use WCF-based BizTalk Adapter for Oracle Database on a dynamic port that is used by BizTalk Server

See Also

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.