Data Center Stories – Upgrading Bitbucket Data Center in a Complicated Configuration

Organizations across the globe are trying to increase their speed of delivery to keep up with evolving customer demands, technological evolution, and market trends. These forces are also putting increased pressure on organizations to increase their capacity to innovate and bring robust, secure, and reliable software products into the market faster.

It is only when organizations can enable speed of development that they can catch trends (consumer and enterprise) as they emerge, stay ahead of the competition, elevate quality and drive down costs.

Slow development in such a fast-paced, competitive business landscape is counter-productive, and fast development cycles lead to more innovation and lower costs.

The Challenge

Data Center plays a critical role to enable innovation at speed by becoming a powerful enabler of development teams to build better and build faster, especially in the age of DevOps.

Our client, a leader in the BFSI sector, approached us to assist with the optimization of their Bitbucket Data Center environments, specifically, their Development and Production environments. While both the environments were heavily used, they were unstable and slow and often in need of restarting.

Upon investigating the problem, we discovered that the Bitbucket version in both environments was woefully out-of-date and end-of-life. The obvious choice was to upgrade the same to the latest enterprise-supported version along with adding several improvements to the application JVM and operating system tuning.

However, we discovered many challenges hidden in the folds. These were:

Lack of uniformity in the production and development environments. The Dev environment had a relatively standard Data Center configuration, while the Production environment configuration had an unorthodox design. The Production environment employed two data centers in different geographic locations (Those were called Production and DR respectively).
The Production and the DR data center had two active Bitbucket nodes – each of which up to four nodes across both the data centers.
Both the data centers had their individual shared drive, MSSQL server, and single Elasticsearch node. The Production shared drive and the MSSQL database were mirrored to the DR shared drive and MSSQL server. However, the DR Elasticsearch server remained dormant and unused.
The DR Bitbucket nodes were connected to the Production side instead of being connected to the DR shared drive and MSSQL database. This instance would, however, be stopped in case of a disaster affecting the Production data center. In such an event, the DR nodes would have to be remapped to the DR shared drive, MSSQL database, and Elasticsearch.
Tight regulations and restrictive change management and separation of duty policies in the clients’ industry led them to employ legacy operating systems. Such systems needed several modifications and workarounds to support the upgraded application as well as the newer versions of Git that Bitbucket. This was so because an OS upgrade could not be approved within the upgrade window.
The fragmented separation-of-duties made scheduling and coordination of the upgrades difficult.
Downtime had to be minimal for the production upgrade since it was linked to critical business functions.

The Approach

The Addteq team did a thorough evaluation of all the dependencies and moving parts. After that, the team prepared a detailed upgrade plan for the production and Dev environments to the latest available Bitbucket version. We also proposed upgrading Git and Elasticsearch and automating the processes as far as possible using Ansible.

Upgrading the Dev environment was simple and was done employing the Bitbucket Data Center upgrade process. Upgrading the Production environment was more challenging because of its complexity and the need from the client-side for minimal downtime.

To upgrade the Production environment while keeping all the dependencies and client demands in mind,

We began by separating the DR environment from Production. Doing this made sure that the DR environment ran completely independently of the Production data center.
Upon assuring that the DR environment was validated and working, we revoked the write permissions in the DR environment and changed it to a read-only configuration.
The Global Load Balancer was also configured to only direct traffic to the DR environment.
On completion of this process, we proceeded to commence the upgrade process in the Production environment. During this time, the DR continued to serve a ‘read-only’ access for business-critical automated processes that needed it.
A Global Load Balancer was configured to direct traffic solely to the Production nodes once the Production-side environment was validated, and only then was the DR upgrade initiated.

We then proceeded to restore the pre-upgrade configuration of the environment when all of the upgrades were complete.

The Outcome

Despite the complex and non-standard configuration of their environment, restrictive separation of duties, and the difficult conditions of the customer’s change management process, Addteq successfully completed this complicated upgrade.

Our solution was implemented within the specified timeframe and with the least possible downtime. Our client was able to upgrade both Production and Dev environments to the latest version of Bitbucket. While doing so, we also ensured that:

The upgrade of the application nodes was automated simultaneously while implementing the modifications and workarounds on the legacy OS.
Production’s complete downtime (no application access) was under a half-hour in total during the entire process. We could do this as we employed the read-only DR data center location while the main Production data center location was upgraded.
We also made sure that all critical business functions remained unaffected and available during the upgrade. This was made possible by making the application available in the ‘read-only’ mode.
All application instability and performance issues were resolved and remediated.
The application became reliable, performance-driven, and stable by implementing the updates to the installed plugins, tuning of the operating system, and the application JVM.

Implementing an upgrade for Atlassian tools is not complicated. However, these need expert assistance when configurations become complicated, and there are legacy systems and processes at play. Being Atlassian Platinum partners gives us the expertise to solve all such complex challenges associated with these tools and deliver greater value to our clients by furnishing the right solution to suit their specific organizational narrative.