Time Warner Cable Generator Base Tank Repair
Time Warner Cable acquired a company that provides cloud and hosting services. As with all mission critical facilities, downtime is never an option. So, what does a data center do when a problem is found during routine maintenance of a backup generator tank? This case study shows how CommTank ensured the Andover, Massachusetts based facilities’ uptime by providing generator base tank repair services for the fuel supply tank without taking the generator offline.
Time Warner Cable staff discovered a leak in the primary ultra-low sulfur diesel sub-base tank located beneath generator set #3. The fuel was discovered accumulating in the secondary containment tank of generator set #3. Time Warner Cable hired a test company to test the primary tank. The result was positive for a fuel leak located within the generator base tank of generator set #3.
CommTank was contacted by Time Warner Cable to conduct an investigation into the leak to determine the exact location that ultra-low sulfur diesel entered the secondary containment basin of the #3 generator set. Time Warner Cable also requested that CommTank investigate the possibility of repairing the leak. Prior testing ruled out the fittings on top of the primary tank as the source of the leak. It was also determined that water entered into the containment basin. A video scope inserted into the narrow spaces of the basin tank did not help determine a cause for the fuel leak.
How to test a generator base tank without taking it offline?
Time Warner Cable insisted the generator and power were to remain in place and could not be removed to replace the sub-base tank. Following an evaluation of all possible repair options, we decided the most efficient and cost effective approach to the primary tank repair was to open and remove the face of the secondary containment sub base tank and extract the primary tank. This was necessary in order to investigate and fully evaluate the primary tank and eventually repair and test it. When presented with our recommendation, Time Warner Cable chose the repair option as opposed to having the manufacturer provide a new tank.
Time Warner Cable could not take the generator set out of service due to operational requirements at the facility. With this restriction, CommTank arranged a visit at the Pritchard Brown manufacturing facility in Baltimore, MD to determine a procedure for removing the primary tank while keeping the generator set in service. The solution involved mobilizing a temporary 3,000-gallon AST and placing it next to the generator set. A 150-gallon day tank was also placed inside the generator housing close to the generator supply lines. Due to motor vibration, the last 4 feet of supply line to a generator is made of flexible hose. A temporary supply and return were installed, primed and tested which allowed full operation of the generator set and maintained reliable backup power. The day tank allowed us to reduce the transition time between switching sources of fuel to and from the temporary 3,000-gallon AST.
Access to the generator base tank was blocked by an iron fence and sloping hill. CommTank removed the fence and excavated a level path in front of the generator housing. Onsite testing allowed us to perform a quick evaluation (visual corrosion on multiple surfaces and the tank would not hold pressure long.) A crane was used to lift the primary tank on to a flatbed truck. We then transported the primary tank to our facility to allow us better access for testing and repair.
The project was conducted in December and the tank was too large to fit inside our facility. CommTank crews built a temporary structure to keep snow and ice off the base tank surface while they worked it. To check for tightness and leaks our technicians applied 3psi of pressurized air to the primary tank. Several leaks were found during testing. Seventeen leaks were documented using a video recorder and tagged along the tank’s outer shell surface. Most leaks were seepage found at or near weld seams of plate joints that make up the tank’s structure. The seepage was only found while the tank was under pressure. One leak was severe enough that a steady stream of fluid was observed leaving the tank while under pressure. This leak was located on the bottom side of the tank just under the emergency vent. We believe this leak was the primary contributor to fuel oil entering the secondary containment.
How did this generator base tank leak occur?
Based on the customer’s observations, it was immediately apparent water had entered the secondary tank. CommTank’s engineer observed water entering the secondary containment by two paths. One path was a roof leak that was discovered by CommTank technicians and repaired. The second path flowed down the outside of a vent pipe, entered the generator housing and continued into the secondary containment at the connection to the base tank. This water traveled and pooled along the top surface of the primary tank until it eventually found its way into the bottom of the secondary tank. It could not be detected or cleaned out easily due to the nature of the secondary containment construction. The metal coating eventually failed as water contacted metal and created oxidation and corrosion which led to a breach in the primary tank. CommTank coordinated with Pritchard Brown (the generator enclosure manufacturer) to find a solution for the generator housing where the vent pipe entered the enclosure so this problem would not reoccur in the future.
Mission critical facilities and data centers promise ‘zero downtime’ to their clients and CommTank helped Time Warner meet that goal. Switching fuel sources to an external aboveground tank when the generator base tank has failed is a scary proposition for facility managers. However, when the project is set up in stages, the actual downtime time can be only 3-5 minutes. That exposure risk to the back-up power source is minimal. Here’s how we accomplish it.
Switching Fuel Sources - From a Base Tank to a Temporary Aboveground Storage Tank
- A temporary above ground storage tank is mobilized and installed on site
- Steel supply and return pipes are connected to the temporary storage tank and run as close to the generator as possible.
- The storage tank is filled with fuel.
- The flexible fuel hoses from the generator are disconnected and moved to the new piping from the temporary tank
- The fuel lines are primed.
- The generator is tested to insure it will run on the new fuel supply
- Repairs can now be performed on the base tank without impacting the back-up generator.
This project was originally projected to take 15 working days and ultimately took 29 working days to complete. The additional 14 days were required because of a change order for the day tank installation and extensive repair (welding, sandblasting and coating) of the primary AST. There were also lost days due to weather conditions that prevented repair work because of unusually high winds and rain.