What a day
Whenna machine that complex has a massive event like this one, it has layers of built in safeties that react to keep the most important pieces from damaging themselves. And those safeties latch themselves in place and sometimes electronically finding their little cul-de-sacs and letting them out of them is a challenge at best, this machine has ten full axis of movement with several more to deal with tool changing etc..
So even once you have repaired the short, and chased the I/O link all the way down the line and made sure all the devices are playing nice with one another, you end up diging through the PLC program finding all the little safeties written in by the programmers. If you lose power, this servo brake engages, if this servo brake engages, the servo command drops to zero, if the servos go off, the doors unlock, the toolchanger door lock latches shut, etc. etc. etc.
After i got all the physical repairs made, the software hunting took me from nine thirty to six. Then, when you hear that big MCR pull in, the hydraulic pump comes on, the spindle fans come on, it’s like a rebirth. And the co-workers didn’t mind too much that I sang the first eight bars of “ode to Joy”.
Glad to have that one behind me. They don’t write procedures for things that aren’t ever supposed to happen, so making it up on the fly is never fun. And the young turk engineers just ran; pussies. I’m the one who is supposed to be old and inflexible and unable to learn new shit.
14 comments Og | Uncategorized
http://www.youtube.com/watch?v=Z5axImP5_C0
:)
Jenny
Definitely need video of you singing Ode to Joy.
“Definitely need video of you singing Ode to Joy. “
Yeah, like that’s gonna happen.
If it was easy, anyone could do it.
Tom
Yeah, lord knows. Job security, I guess.
Old warhorses will get the younger ones every time. Key element is don’t panic.
I don’t have any panic left.
ou end up diging through the PLC program finding all the little safeties written in by the programmers.
I’m surprised that the assorted safeties aren’t grouped in a class. I may be mis-understanding the language involved.
You assume there is a rhyme or reason. The PLC program, printed, is six hundred pages. Probably written by thirty people. And each deals with his own stuff, often without any communication between the group.
“printed, is six hundred pages … written by thirty people … without any communication”
(twitch)
Let me walk that ‘twitch’ back, a bit.
I understand that the way it _is_ works. That tearing it apart and building it ‘right’ would not be cost effective. Might not even be possible.
I’m a nerd, and an IT dork, and a jarhead, but I also speak a little ‘business’.
But still I have this desire to take something like that and make it _right_.
Brian: I completely understand; it is both better than and worse than you can imagine- the 600 pages are printed on both sides, single space. But; these are some of the very best machines in the industry, and that 600pages is remarkably well done.
What has happened is, this generation of hardware does not have the fusing and protection of previous generations. This means a simple short circuit can and often will take out far more than just a simple relay, it can get deep into the system, electronically. And when this happens, the software reacts to protect the machine.
Each system is written by an individual or group- and this is not code the way you think of it, probably, this is ladder logic- think boolean with timers and counters. Each segment of ladder deals with a specific subgroup of the machine. One will be the cutting tools, another will deal with tool changers, another will deal with fixturing, another with the status of the various drives. The control itself is another whole deal altogether- as are the drives; the ladder merely directs. And while there is little comm between modules there is an overseer or director who qualifies that all the sub processes are compatible, and they are. We rarely have legitimate issues.
What does happen, though, are the subgroups are intentionally told to protect themselves, and they do; once a large event such as the one that caused this mess takes place, most of the sub processes lock themselves out of operator manipulation until they are reset. Some can be reset by simple boolean operators built into the ladder; force this input, that latch clears, this module is released from it’s safety. Some are reset by forcing components to function out of order- open a door where it’s not supposed to be open, for instance, and that eliminates the latch holding that subprocess in limbo. All of this has done precisely what it was designed to do; i.e., protect the machine. Because this has not been done before, releasing those processes was a new challenge. And I’ve only just now finished copying the notes off the palm of my hand and documenting the process so the NEXT guy can figure it out.
Ladder logic I sorta-kinda savvy: reading about I have a mental picture of DOS batch language in terms of of flow.
I hadn’t extended the concept from ‘example’ to an actual real-world machine. Very cool.
” open a door where it’s not supposed to be open, for instance,”
We once had a very troublesome tape library robot. When it got into trouble the reset directions were a lot like that: ‘Hold exit door open for at _least_ 30 seconds while pressing down on the power button”
Now I understand _why_.
That sort of thing allows you to have a sort of “Backdoor” way of solving issues. Can you imagine the size of the operator panel (No less the cost!) if it had a separate pushbutton for each possible emergency? And most of those buttons would go unpushed forever.
Fanuc, Mitsubishi, all those controls have special keystrokes that you can use to get into special “Modes”. The PLC’s are written no differently.
IIRC, you live locally (Peoples Socialist Republic of ILL). If you have a chance, come to the next machine tool show. I have tried to get Ed to one of the last two to no avail (His scheduling always seems to conflict!) but you’d enjoy it. Not till 2014, but it is an extravaganza of ohshit.