Category Archives: Engineering

VAC – A New Beginning

Well, my old nemesis ComEd has reared its head again.

Last night, I had another beep hunt. This was a steady, rapid beep, so it only took about 30 seconds to find the problem: The alarm was going off on the UPS protecting the television and TiVo. It’s a fairly dumb UPS, so there’s no way to tell what the problem is, but I figured it was just the battery. I shut it off and unplugged it, and I plugged our television gear directly into the wall. Problem solved. I could replace the UPS later this week.

Then, about 4:30 this morning, I woke up to more beeping. This one was a little harder to localize for a reason I couldn’t quite figure out, but I eventually traced it to the UPS protecting my work computer.

This was a bad sign. One UPS alarming is probably a battery problem or some other end-of-life issue. Two UPS alarms is a problem with the electrical power coming in to the house.

I had shut down that computer for the weekend, so I just turned off the UPS to stop the alarm…and I realized that something else was still beeping somewhare in the house. That’s why I had trouble figuring out where the beep was coming from! I was hearing beeping from two sources.

The other alarm turned out to be the UPS protecting my wife’s computer. It had tripped out and turned off the computer, so I just turned off the UPS to kill the alarm. This time it worked. Silence.

It took me a minute to find my DMM and another to find a pair of probes, but with the dial set to measure AC voltage, I stuck the probe tips into an outlet in the kitchen: 138 volts.

That’s high. American residential electricity is supposed to be about 120 volts, with houses closest to the transformer getting a slightly higher voltage, and houses farthest away getting a slightly lower voltage, plus some variation during the day due to the changing load.

So it was high, but how high is too high? Surprisingly, that’s not an easy thing to find out. The ComEd site has nothing about it that I could find, and the Citizens Utility Board site is useless: Lots of stuff about rates, but nothing about the power supply specification.

I eventually found what I wanted in the ICC regulations governing electrical voltage in Illinois (emphasis mine):

Section 410.300 Voltage Regulation

a) Standard voltage. Each entity supplying electrical energy for general use shall adopt a standard service voltage of 120 volts (when measured phase to neutral) and shall maintain the service voltage within the allowable variations from that value at all times.

b) Allowable voltage variations. For service rendered at the standard service voltage, voltage variations as measured at any customer’s point of delivery shall not exceed a maximum of 127 volts nor fall below a minimum of 113 volts for periods longer than two minutes in each instance.  For service rendered at voltages other than the standard voltage value, voltage variations as measured at any customer’s point of delivery shall not exceed 10% above or below the service voltage for a longer period than two minutes in each instance.

So it was definitely way too high.

While I was writing this, the lineman showed up. He was a friendly guy, and he seemed to take the problem seriously. (I suspect ComEd gets a lot of complaints about “the voltage” from crackpots, so I’ve been trying to act as sane as possible so they take me seriously.) I let him into the basement to check the voltage at the meter, and he measured it at 139 and 140 VAC.

The UPS on my main home computer is bigger and more expensive than the other three, and it has a voltage regulation feature, so that computer is still working, and I’ve plugged my work computer into it so I can do my job. However, I’ve got most of the household electronics shut off, and I also shut off the air conditioner, because I’m concerned that the compressor will overheat on the high voltage.

So now I wait. They’ve seen the problem. Now I just need them to fix it.

Fukushima: Waiting For the Magic Bullet

Here’s something I’m wondering about. Maybe some of you can help me out, or point out where I’m going wrong…

The situation at the Fukushima Daiichi plant is just continuing to deteriorate, with damage at all three fueled reactors, containment cracks at two of them, and overheating spent fuel pools at one of the shutdown reactors, with more spent fuel problems likely.

The heat in the reactors comes from decaying fission products. The initial rate of decay is very fast, giving off a lot of heat. That fast decay uses up a lot of fission products, which means the rate of decay, and therefore heat output, drops pretty quickly. But as the decay rate drops, so does the rate at which the fission products are used up, meaning that reduction in decay heat slows down.

According to the MIT Department of Nuclear Science and Engineering’s estimates of decay heat, after the reactors were shut down, the decay rate dropped by half in the first minute, then by half again in the next half hour, then by half again about 16 hours later, and so on. Today, the reactors are probably each putting out between 6 and 12 million watts of power (depending on which reactor), but that amount isn’t going to go down much more in the near future. By the end of the month, they’ll still be putting out between 5 and 9 million watts. It will take a year for the heat output to drop to half of what it is now.

In other words, the battle at the power plants isn’t going to come to an end by itself. However, there is one thing that can stop the overheating, stop the leaks, and return the spent fuel pools to normal. It’s a magic bullet, or like Superman coming to the rescue, except it’s totally real.

The magic bullet that will solve nearly all the problems at the Fukushima Daiichi plant is kind of ironic: The power plant needs power. Since the earthquake hit, the plant has been off the Japanese power grid, and since the tsunami an hour later, all of the backup diesel generators at the plant have been inoperable. Thus, none of the cooling systems at the power plant have been working.

As soon as the plant site gets power, the reactor wouldn’t have to be cooled using fire engines for pumping. Operators could just start the Emergency Core Cooling System (or whatever the call it in Japan), which would drain hot water from the pressure vessel, cool it in a heat exchanger, and pump the cool water back over the reactor cores. Everything will cool off. The water will stop evaporating to steam, the vapor pressure will go down, and radioactive material will no longer be released because the pressure vessel and containment won’t be venting the pressurized steam.

They wouldn’t need elaborate and dangerous plans to re-fill the spent fuel pools using helicopters. They’d just turn on the pumps.

People have got to be working on this. They’ve got to be trying to reconnect the plant to the national power grid. They’ve got to be trying to repair the diesel generators at the plant site. They’ve got to be trying to bring in new generators.

Yet I haven’t been able to find news coverage of the effort to resupply the plant with power. Does anybody out there have more information about this? Or am I misunderstanding the situation?

Update: While I was writing this, CBC Canada answered it:

A new power line could soon restore electricity to cooling systems at Japan’s tsunami-damaged nuclear plant, its operator said early Thursday, a development that would reduce the threat of a meltdown.

Construction of the line is nearly complete, said Tokyo Electric Power Co. spokesman Naoki Tsunoda, and officials hoped to try it “as soon as possible.” The potential for a meltdown at the Fukushima Daiichi complex remains, however.

(Hat tip: repstock1, confirmed by MIT NSE)

Update: On the other hand, MIT MNE folks say it may not be as easy as running a power line:

The problem with the spent fuel pools is not a want of generators, but a want of electrical connection. Flooding of electrical switch gear has left the generators unable to be put into service.

So I guess they’re going to have to repair and replace some of the switch gear as well.

Update: Just to be clear, given that the reactors have been through earthquakes and a tsunami and explosions and fires and contamination and salt water, the problems aren’t going to go away instantly–calling it a magic bullet was a bit of hyperbole–but things will get a hell of a lot easier once they can get the pumps running. Everything going on right now is a holding action until that happens.

Fukushima Uncerntainty

A lot of people have been saying a lot of things about the nuclear emergency in Japan, much of it conflicting, and therefore much of it wrong. I see no reason why I shouldn’t get involved. As with almost everything else I talk about here, I claim no expertise beyond an amateur interest in the subject. Make of that what you will.

To start with, Dr Josef Oehmen, a research scientist at MIT, originally posted a pretty good description of the situation in Fukushima at the Morgsatlarge blog, explaining “Why I am not worried about Japan’s nuclear reactors.” That document has since been taken over and expanded by some folks at the MIT Department of Nuclear Science and Engineering, and it’s now available at the MIT NSE Nuclear Information Hub here, although the new authors disavow the original title.

Scott Greenfield has of course expressed a bit of skepticism:

[B]eliefs in the promise of science and safety have been certain before, and ultimately proven wrong. Afterward, everybody says how obvious is was that the work was shoddy, the systems inadequate, the science lacking. It’s always afterward that everybody knows better.

I’ve seen that a few times myself. I think the most common cause is that expert opinions are being filtered. Sometimes it’s intentional manipulation by interested parties, but sometimes it’s just the news media going with the story that’s easiest to tell.

Is anybody else old enough to remember the comet Kohoutek? It was detected in the spring of 1973, and some experts thought it would be an incredible sight in the night sky when it arrived at the end of the year–a big beautiful “Christmas Comet.” Other experts were predicting a much less spectacular show. Guess which ones got the most airtime?

When Kohoutek proved to be little more than another speck in the night sky, the news story changed. Early on, scientists had had different theories about whether Kohoutek was a new comet, which would give off a lot off a large gaseous tail when it arrived, or an old comet, which had already given off most of its gases in previous passes around the sun. Once Kohoutek arrived, scientists knew it was an old comet which couldn’t possibly have produced a big showy tail. The comet’s arrival provided actual data, which killed the alternative theories.

On the other hand, sometimes the experts are just ignorant: Nobody knew the thing could happen that way, but now that it’s happened, we understand it, and it seems obvious. Wind loading on tall buildings is a good example: Hundreds of years ago, a lot of tall church towers crumbled to the ground before people figured out that wind was rocking the tower back and forth, grinding away the mortar between the bricks. Nowadays, planning for wind loading is a routine part of structural engineering.

In the commentsJeff Gamso adds his own cynical twist:

When the experts tell me everything is under control and there’s no need to worry, my worry increases exponentially.

I know exactly what he means. Those pronouncements are never as reassuring as the speaker thinks they are. I cringe whenever I hear that authorities are trying to “reassure the public” or “calm fears.” I don’t want to be reassured, and I don’t want to be calmed. I want to be informed and, if necessary, helped. I don’t want to be manipulated into trusting some “authority” who’s obviously trying to spin a disaster.

In my admittedly limited experience, however, it’s not the actual experts who give false reassurances. Doctors will tell you when you’re really sick. Accountants will tell you when you’re really broke. Engineers will tell you when the bridge is going to fail. The people who will try to soften the news are the public information officers, the industry spokesmen, the public relations flacks, the executives, and the politicians.

Worst of all, however, are the people with political axes to grind. Thankfully, most Americans–myself included–are completely ignorant of Japanese politics, so the media was unable find clear sides to turn this into a horse-race issue. All we’ve had until now is the basic nuclear power debate, in which the anti-nuclear folks shout and point at the dangerous happenings in Japan, and the pro-nuclear folks “tut, tut” that everybody is worried about a little steam and some radiation only slightly above the background.

Lately, however, I’ve noticed that pundits have hit upon the idea of tying the Japanese nuclear emergency to the future of U.S. nuclear energy policy, thus pulling the debate into a familiar and wasteful left v.s. right argument.

All of this makes it kind of hard for those of us watching from a distance to figure out what’s going on, which is why a relatively technical source like the aformentioned MIT NSE Nuclear Information Hub is useful for people who want to figure this stuff out. The MIT folks seem fairly sure that this isn’t going to turn into a big accident, but despite what they’ve written, there are four things that worry me.

My first worry is in regard to this statement from the MIT engineers:

The entire primary loop of the nuclear reactor–the pressure vessel, pipes, and pumps that contain the coolant (water)–are housed in the containment structure. This structure is the fourth barrier to radioactive material release. The containment structure is a hermetically (air tight) sealed, very thick structure made of steel and concrete. This structure is designed, built and tested for one single purpose: To contain, indefinitely, a complete core meltdown.

It’s the “tested” part that concerns me. I’m sure the engineers did all the simulations to test the design, and I’m sure they conducted a variety of heat and pressure tests on the components of the containment, but I’m also quite sure that no one has ever subjected a containment structure to a full-scale test with a full load of reactor fuel. That would be insanely dangerous, anywhere on on Earth. So nobody knows for sure that the containment will function as planned.

That leads to my second concern:

The earthquake that hit Japan was several times more powerful than the worst earthquake the nuclear power plant was built for (the Richter scale works logarithmically; for example the difference between an 8.2 and the 8.9 that happened is 5 times, not 0.7).

Some people have taken the reactor’s survival of such a severe quake as an engineering triumph, and in a way they’re right. However, I’m pretty sure the containment structures haven’t been carefully inspected since the quake, so all we know is that the containments were still standing and still pressure-tight. But just because the containment buildings didn’t collapse and spill out their contents during the quake doesn’t mean they haven’t suffered serious damage. The real question we need to answer is whether the containments are still up to spec. Can they still to the job they were designed for? Can they still contain a nuclear meltdown?

(Some people believe that the containment of reactor 2 has sprung a leak–it lost pressure after an internal explosion, and the external radiation levels shot up.)

My third concern:

All of this, however shocking it seems to us, is part of the day-to-day training you go through as an operator.

I don’t think that’s true anymore. I think this situation is not one covered by the training manuals, as evidenced by the fact that the plant workers are making mistakes, like letting the seawater pump for reactor 2 run out of fuel, causing the core to be exposed.

Finally, there’s the question of the spent fuel pools.

When reactors are shut down, the decay heat slowly tapers off, and after a month or so, the fuel rods can be removed and placed in storage at the reactor site. They are still giving off heat, so they have to be submerged in a pool of cooling water for several years until they can be transfered to long-term storage.

These pools have to be cooled and evaporated water has to be replaced. I don’t think the fuel inside can produce enough energy to melt down, but if the water is allowed to evaporate, the fuel rods will be uncovered and heat up enough catch fire, which will damage the rods and spread radioactive materials.

The spent fuel pools at the Fukushima Daiichi plant are located inside the reactor buildings, on top of the outer containment structure. That’s the top third or so of the reactor building where the hydrogen explosions took place at reactors 1 and 3. As far as I know, none of the spent fuel pools at reactors 1, 2, and 3 have caused trouble.

The fire at reactor 4 (which had been shutdown for inspection at the time of the quake) took place near the spent fuel pools, but plant officials are now saying that the fuel was not involved. They’re talking about the need to replenish the water for the pools sometime over the next couple of days.

In any case, it looks like Fukushima Daiichi is now a member of the exclusive club that includes such names as Windscale, Brown’s Ferry, Three Mile Island, and Chernobyl.

If you want to know more, I’ve found pretty good coverage of the incident by scanning CNN, Reuters, and the New York Times. If you’d like some more technical information, there are interesting stories at World Nuclear News, and the aformentioned MIT NSE site has some good explanations.


Elevator of Death…Or Not

When it comes to elevators, our biggest fear is that the cable will snap and the elevator will fall to the bottom of the shaft. People debate whether there is anything you can do to save yourself—jumping just before you hit is the most popular suggestion. A few years ago, Mythbusters did a test, and the result was not pretty. Jumping doesn’t help.

Elevator engineers could have told you that. If our legs were powerful enough to jump hard enough to counter the speed of the fall, they would also be powerful enough to absorb the impact. Heck, if our legs were that strong, we’d all be able to jump very high. Apartment buildings up to about five stories wouldn’t need stairs—people would just jump up onto their balconies from outside.

On the other hand, I’ve long suspected that such elevator falls are extremely rare. Although hoists of various kinds go far back in history, we didn’t start putting passenger elevators in buildings until after Elisha Otis invented a mechanism to prevent elevators from falling. Modern elevators are defined by their inability to fall. My guess was that fatal falls happened less than once a year, at least in this country.

According to a fascinating article about elevators by Nick Paumgarten in the New Yorker, I’m way off, but in a good way: For a long time, there was only one known free-fall incident in a modern elevator. It happened in 1945. And it happend when a B-25 bomber struck the Empire State Building in the fog. The impact cut all the cables on two elevators, which fell all the way to the bottom. There was only one person in the elevators at the time, an elevator operator. She was hurt bad, but she survived.

(The terrorist attack on the World Trade Center—which also involved airplanes hitting buildings—killed a bunch of people in elevators, a few of whom probably died from falls down the shaft.)

If you’ve ever seen the top of an elevator, you know there are a bunch of cables holding it up. Every single one of those cables can hold the entire weight of the fully-loaded elevator. Many elevators also have brakes that lock the elevator to its rails, so even a loss of all the cables wouldn’t make it fall. Finally, there’s a hydraulic buffer at the bottom of the shaft to cushion the impact.

For all practical purposes, nobody ever dies in an elevator fall. In fact, of the 20 or 30 elevator-related deaths each year, most of them are maintenance accidents—technicians leaning too far into the shaft or getting caught between moving parts.

However, if you still want your elevator rides to have some thrills, there are always the hazards of elevator doors to worry your mind, such as people stepping blindly through doors that open into empty shafts or being strangled by scarves caught in the doors. If you want something really scary to spice up your elevator rides, sometimes the door-open safety mechanism fails and an elevator suddenly moves while people are getting in or out. The results are often gruesome, and sometimes end with the elevator passengers riding up a few floors with a severed head.

The Mystery of the Self-Unscrewing Bulb

Libby asks:

How do light bulbs unscrew themselves? This can’t be only be happening to me. My lamps are really old and rickety for the most part but I don’t think that’s the reason that I have to retighten the bulbs every once in while. Sometimes I think the bulb is out and discover it’s just not twisted in far enough. And today I changed a bulb that did burn out but it was only one twist away from unscrewing itself and leaping to the floor.

Are my lightbulbs just suicidal or is there a scientific explanation for this do you think?

This is one of those questions that stuck in my head…let me try some guesswork.

Light bulbs are held in place by friction between the threads of the bulb base and the threads of the socket. The friction forces tend to prevent movement of the bulb, but they don’t actively force the bulb into the fully-seated position (where the center electrical contacts are touching, allowing the bulb to light). So if some outside force unscrews the bulb out of the seated position for any reason, it stays there unless another outside force pushes it back.

Assume that some random force occasionally nudges the bulb. This nudge will either screw the bulb further into the socket or unscrew it out of the socket. If the bulb is already fully-seated in the socket, it can’t screw in any further. (Actually, it can move a little bit by deforming the socket and base a bit, but that takes more force than simply screwing or unscrewing.) On the other hand, the bulb can always move in the unscrewing direction. There’s nothing to stop it until it falls out.

If the random forces are not perfectly balanced, they will tend to nudge the bulb more in one direction than the other. If they tend to unscrew it, over time they will eventually loosen it enough to break the electrical contact and the bulb will go out. You’ll notice that and discover the loose bulb. But if the random forces tend to screw the bulb in, it will just stay snug in the socket and you will ignore it.

So what are these mysterious random forces that move the bulb?

Two possibilities come to mind. The first is thermal cycling. When the lamp turns on, the parts of the bulb base and socket heat up, causing them to expand. However, not all parts heat at the same rate. In addition, the parts are made from different materials which have different expansion charactistics. Thus the parts change size, but not in perfect synchronization, which creates internal stresses that push on the bulb, loosening it or tightening it.

(This used to happen with old socketed computer chips such as the CPU and RAM. Over hundreds of turn-on/turn-off cycles they would expand and contract and slowly walk themselves out of the socket until the computer died. I remember some old computer equipment where the first line of problem diagnosis was to pull all the cards and press down firmly on every single chip. It often solved the problem. Most modern computers use complex sockets designed to prevent things from creeping loose—this is why CPU modules and memory cards usually have a retainer that snaps into place.)

Once a bulb unscrews itself this way, it goes out, and therefore thermal cycling stops, which means that thermal cycling can’t turn a bulb back on again. On the other hand, it can’t unscrew a bulb beyond the point where it stops working, so this doesn’t explain how Libby found a bulb that had nearly fallen out. Some other forces must be at work.

All I can think of is random vibrations from things like people walking around, passing vehicles, and circulating air currents. Again, if these vibrations are not perfectly balanced—and they never are—then they will tend to nudge the bulb more in one direction than another. If they’re pushing in the unscrewing direction, they will nudge the bulb away from the electrical contacts and eventually out of the socket.

That’s about all I can think of to explain the self-unscrewing-bulb phenomenom.

Unless you have cats. They do stuff like this all the time ’cause they think you’ll never suspect them.

Son of VAC

Well, I thought my electrical problems were over.

About five minutes ago, however, the lights went out. Looking out the windows, I can see that the lights are out in several neighboring blocks as well. I can still see skyglow, and some lights further away, so it’s not the whole city.

(4 minutes to shutdown.)

After the first couple of minutes, the power came on for about 15 seconds and then went off again. That can’t be a good sign. It probably means that some sort of backup plan has also failed.

(2 minutes to shutdown.)

Still dark.

Until civilization returns, this is Windypundit, signing off…

Update: Power came back on about 10:20 local time.

The best part is that the UPS for the Tivo didn’t give out—although it was sounding the alarm for the last few minutes before power returned—so we got all of the Daily Show.

What? Doesn’t everyone have their DVR and cable box on a UPS?

VAC, The Final Chapter

(Links to Chapter 1, Chapter 2, and Chapter 3.)

When I first called ComEd’s customer service center to complain about the electricity in my house, I wasn’t even sure that 135 volts AC was out of the accepted range for electric service. While several consumer web sites have information about electricity costs and rules for disconnection, the web has surprisingly little about electrical service quality and voltage standards.

(Thanks to Firefly Digital Media for a comment clarifying that “135V is indeed WAY too high, and it’s not a matter of ComEd WANTING to fix it. It is a violation of standards that can threaten public safety.”)

I guess my easy-going telephone personality and my hesitancy over whether this was even a problem caused ComEd to not take me very seriously. As of Tuesday morning, a week and a half since I first brought the problem to ComEd’s attention, the household electricity was coming in at 137 Volts AC.

It was time to use my nuclear option: I let my wife handle it.

You see, I don’t like putting pressure on customer service people. I hate it when people put me on the spot for things, so I’m reluctant to do it to other people.

My wife, on the other hand, does customer service for a living. She’s worked in call centers, managed call centers, and set up call centers. Providing good customer service is a very big part of her job. So when she calls other customer service centers and they don’t solve her problem, she shows no mercy. Without making threats or being unfriendly, she makes it absolutely clear that they’re not going to get rid of her without doing exactly what she wants.

One hour after her call to ComEd, I spotted this guy out the window:

After checking our power line directly, he backed the truck up to the transformer drum that serves my building and checked something there. Then he proceeded up along the distribution lines, checking one transformer on each high-voltage line that serves each block, until he got to the last pole where the lines go underground. (Yes, I followed him.) Then he drove off to do something else.

About an hour and a half later I’m sitting at my computer when I hear the UPS click and the lights get dim. I notice these brief brownouts a couple of times a month. Except this one is lasting longer than they usually do, and although the UPS clicked like it always does, it isn’t buzzing with 60Hz hum from supplying battery power to the computer…


I pull out the DMM and check an outlet. 122 volts.

So when I called ComEd, nothing much happened for a week and a half; but when my wife calls ComEd at 9am, they fix the problem by noon.


As an aside, here’s something that makes me glad I don’t have to troubleshoot electric power distribution systems:

I’ve been measuring the household voltage by sticking the probes on my multimeter into the two slots in an ordinary outlet on the wall. Here’s a picture of the probes:

As you can see, they’re about 5 inches long and have maybe 1/8 inch of insulation. That’s plenty for measuring the household 120 volts (or so). In fact, the meter and probes are rated for 1000 volts.

Now take a look at the probes the ComEd technician was using to check the high-voltage distribution lines:

Damn, that must be a lot of voltage.