High tech lab gets a high tech control room
New face lift improves efficiency
Nowadays, a person unhappy with his or her appearance can get an "extreme makeover" and completely redraw their features and figure with a few slices of the surgeon's scalpel. Thanks to reality shows, a studio audience can follow the patient's months-long transformation from ugly duckling to swan in the space of an hour, with the new and improved subject revealed to gasps of surprise from the audience and tearful family members at the close of the show.
Over the summer, the CEBAF accelerator control room, located in the Machine Control Center (MCC), underwent a similar transformation. But no high-priced physicians were called in, and the result was no surprise to those who knew the control room best. Laboratory staff members and external contractors rolled up their sleeves and jumped into the fray from the outset, carefully sculpting the accelerator's nerve center into a state-of-the-art, technologically advanced and ergonomically sound control room for CEBAF.
Initial construction began on the control room in 1990. In those days, computer operators saved data to large floppy disks, cellular phones were the size and weight of house bricks, and the Internet was known only to the scientists who had invented it. While the rest of the world traded in floppies for compact discs, picked up pocket-sized cell phones and learned to surf the World Wide Web, the control room largely remained a child of the year it was built.
Operations Projects Group Leader Mike Spata and Information Systems Administrator Tom Oren spearheaded the makeover effort. Spata explains that some updates were made as the need arose, "As time went on, we began to digitize and miniaturize. So we began using flat panel monitors, and more and more of our analog equipment was made digital in the field." For instance, hefty CRT monitors were replaced with trim LCDs, older computers gave way to faster, cheaper models and additional equipment was added on a piece-by-piece basis. But the same, basic foundation on which the control room was built remained in use.
The Winds of Change Were Blowing
The accelerator control room failed workplace assessments performed by JLab Occupational Health and Safety Medical Director Dr. Smitty Chandler and JLab Assessment Engineer Hugh Williams. They found that the control room didn't measure up to the standards published in the International Standards Organization (ISO) Ergonomic Design of Control Centers report and that it also failed to meet some Lab EH&S (Environmental, Health & Safety) standards. The assessments noted many issues that needed to be addressed, including an elevated background noise level that could potentially interfere with operators' ability to hear alarms, computer screens with excessive glare and a less-than-ideal floor layout that contributed to what assessors referred to as a generally stressful environment. However, three areas were of particular concern.
The first area involved ergonomics. The assessments noted that the outdated system of equipment racks accelerator staff used to mount various monitors was not optimal. "We had three tiers of displays mounted in racks to display all the systems we need to monitor. And the upper tier was too high for the operators, so the angle they needed to tilt their heads to see these monitors was too severe," Spata notes. Dr. Chandler also found that the one-size-fits-all, non-adjustable desks compounded the problem.
The air-handling system represented another opportunity for improvement. The system served a dual purpose: it cooled equipment as well as people. Air was piped into the control room via the open space between the floor system and the concrete foundation of the MCC. From there, it was forced up directly into the rack system and through other floor vents into the room. This setup provided a potential breeding ground for mold, mildew and other problems. Plus, a single air-handling system fan produced noise equivalent to two-thirds the level for which OSHA requires ear plugs. The assessments called for replacing the entire system with one that was both quieter and separate from the floor system.
Finally, the aging floor structure offered another strong impetus for change. "Fundamentally, the flooring had failed," Spata explains, "The floor tiles were galvanized steel with cement cores, and the carpet, which was well over 10 years old and truly disgusting, was integral to these tiles. So effectively, the entire floor system needed to be replaced."
Galvanized floor tiles are also theorized to be the root cause for computer server failures over the years. Firms using the tiles had discovered that the zinc used in the galvanized steel could separate from the steel, growing into so-called zinc whiskers. These microscopic hairs could flake off, get dispersed by the air handling systems into computers and potentially short out circuit traces within the servers. Plans were made to replace the entire floor system during the scheduled accelerator down period in the summer of 2003. Making the Case for a Makeover
Those plans changed when Spata and other operations staff attended the Workshop on Accelerator Operation in Japan during March 2003. There, operators attended sessions and heard talks about how other accelerators were run. They saw pictures of other control rooms and learned about design options that hadn't been available when CEBAF was first built. And they listened to what other operators, from around the world, wanted to see in a control room. For Spata, the trip sparked a new idea: instead of just replacing the carpet, perhaps they should consider more extensive renovations.
Spata assembled a team to explore remodeling options, and within months of the workshop, the team had put together an initial proposal for completely restructuring the control room. They presented their ideas during the August 2003 budget meeting. According to Associate Director of the Accelerator Division, Swapan Chattopadhyay, "They worked very hard to conceptualize it, and they were willing to be flexible in terms of cost and design." After an initial signal of interest, team members began fleshing out their ideas, exploring vendor options and calling in Jefferson Lab employees from other departments to help create a more user-friendly control room.
The core design team and other employees struggled through 10 months of choosing, reviewing and re-reviewing options. Employees not involved in the design process were called in to independently review the plans. By the time project approval was granted, the original proposal had undergone a full 13 revisions.
Finally, during the scheduled accelerator down in August 2004, accelerator control was transferred to a temporary control room set up in the conference room of Building 87, and the entire accelerator control room was gutted down to the concrete foundation. Employees from across the Lab and contractors pitched in to help renovate the room in the 21 days allotted. In the end, more than 50 people, representing nearly every division at the Lab, were involved with some aspect of renovation planning, review and implementation. This collaborative effort has resulted in one of the world's most advanced accelerator control rooms. The Control Room's New Face
When you step through the main entrance of the Machine Control Center, the first thing you notice is the control room itself. A framed-glass wall allows you to immediately see what's going on in the Continuous Electron Beam Accelerator Facility's nerve center. What's more, the original anteroom referred to as the fishbowl has been removed, so now the control room is one entirely open room. While these cosmetic differences make a dramatic impact, the truly useful changes can't be seen until you enter the room itself.
Before the renovation, a large conference-style table squatted in the center of the room, with a smattering of chairs at its perimeter and a permanent collection of logbooks piled at its center. To the right, another table housed camera monitors and safety systems. And sprawled out in a large, square horseshoe shape along the back wall, tall, blue racks of equipment and monitors towered over controllers sitting at keyboards placed on a utilitarian counter that ran the length of the racks. All vestiges of that layout are gone.
In its place are six distinct work areas. On the left side of the room, in place of the fish bowl, are four connected workstations reserved for vacuum, magnet and other accelerator specialists who may need a terminal to check CEBAF operations. These four stations are arranged in an open horseshoe pattern. Like all workstations in the room, each one has its own computer, chair and adjustable keyboard. Also on the left side of the room, a semicircular work area with a bank of computer monitors denotes the Crew Chief's domain. This area includes a wall-mounted white board and bulletin board. The Crew Chief's new location and orientation to the space is intended to emphasize that individual's role as shift supervisor with an unobstructed view of the entire facility.
The safety system operator and related equipment is situated on the right side of the room, with a clear view of three flat-screen monitors mounted on the right wall that depict camera shots of accelerator access points and an overview of the Personnel Safety System status viewable from anywhere in the room. The program deputy workstation is connected to the safety system operator's station and located adjacent to a collaborative problem-solving area in the corner of the room. Situated about 15 feet from the back wall is a workstation for accelerator operations staff, outfitted with four stations along its 11 foot length. Finally, two consoles in the center of the room are reserved for accelerator operations principal investigators who aid in program execution and provide additional expertise when solving problems.
A seamless screen of rear-projection dlp (digital light processing) cubes along the back wall has replaced the monitor-bearing blue computer racks. The screen displays all the information that the multitude of monitors in the racks formerly displayed and then some. The system is run by an independent processor, whose sole function is to arrange the information operators need in a logical format. "Previously, our view of the machine was housed in a series of 14 different displays, all showing different pieces of the control system," Spata notes.
Now, operators look to the video screen for all that information, which can be viewed comfortably from any point in the room. "Display of information is just as important as control of information. Because most of the time, your ability to solve a problem depends on your ability to know what the problem is," Chattopadhyay says.
Of the 20 or so blue racks, only three remain. Painted the color of putty, they're lined up unobtrusively along the back of the left wall and house the remaining essential analog equipment as well as some additional hardware to handle analog and video signals from the field. In addition, a new air-handling system sends currents of comfortable air through vents in a ceiling that has been raised about a foot. New lighting fixtures produce full-spectrum light, providing more natural-looking white light than fluorescent lighting. One door was removed, and most printers were relocated to the MCC copy area. The floor structure was realigned and the old cement tile system replaced with a new, carpeted system that is easily maintained. A coffee station along the right wall rounds out the room's accoutrements.
In the end, the control room renovation addressed nearly every point identified in the ergonomics assessments; and all of the work was completed within the normal accelerator down time, without causing a single lost hour of operation. It's anticipated that these fundamental changes, and others, will help accelerator operators run CEBAF more efficiently. "The problem solvers, the people who run the machine, and the people who run experiments can all have this integrated approach to problem solving and gathering information. It must mean that the machine will be up and running for more time, percentage-wise, than it was before. So the availability of useful beam for physics must go up. The machine itself hasn't gotten better, but your ability to see what is wrong and act on it is better," Chattopadhyay says.
Spata sees CEBAF's new face as the next leap forward in accelerator control room design. "We're the first accelerator control room to throw away the equipment racks wholesale and go to a more digital environment. We're leading here, and that's an exciting thing. And I think people will want to come see what we're doing and copy it," he says. The Future of CEBAF's Control Room
Now that this phase of the renovation is complete, plans are being made to continue the upgrade effort. Andrew Hutton, deputy associate director of the Accelerator Division, says this renovation lays the groundwork for future improvements in CEBAF control and operation. "We still have problems where the operator is searching through screens looking for the thing that's wrong, and that's slowing down the way we work. A computer can do that search in fractions of a second. The goal would be that the big screen would normally have rather few pieces of information, such as boxes showing that beam delivered to each Hall is OK. If something is wrong, then we would like the system to automatically display information on the piece that is not functioning properly."
Hutton's long-term aim is to make the control system a more robust system that is proactive rather than reactive. "In some cases, there are measurements that we make infrequently because they interrupt beam delivery. So one of the jobs that goes with this is upgrading our diagnostics, which will permit us to achieve real-time observation of every specification at the beam current required by the user," Hutton adds. He says that can be achieved by continually upgrading the machine with new beam diagnostics, by changing the way the control system monitors the machine and by changing the way information is displayed.
According to Spata, the renovation was made with just such future improvements in mind. "There's room for expansion. The layout is intended to support the 6 GeV (billion electron volts) machine, and it's perfectly capable of supporting a 12 GeV machine. I look at it as a long-term investment in the future of Jefferson Lab," he says.