Category Archives: Work

Keystone Kops

Posted on by
Reply

I’ve had a hell of a time with Verizon. We have a Verizon Government contract at work because we are a governmental entity along with being a non-profit.

To set the stage, my system support specialist wanted to drop his old phone and take his number that he likes and that everyone knows belongs to him and have that number assigned to his work iPhone 5. We asked Verizon a few times and it took a fair bit for them to come around and we eventually got it off and running, or at least so we thought. Then this email exchange happened. I have redacted the numbers and names to protect the guilty.

Verizon Government said:

Thank you for contacting the Verizon Wireless Government Customer Operations Team. We have received your request to transfer [NEW PHONE NUMBER] to account [VZ ACCOUNT NUMBER]. In order to complete your request, please have the end user ensure the device and price plan are associated with Verizon Wireless and not Alltell. We are unable to transfer an Alltell device and price plan.
—
VZ Rep said:
Andy the user on this account needs to have his rate plans changed to non share Verizon plans. Once that is done they should be able to process the transfer.
—
Verizon Government said:
We have completed your request to perform an Assumption of Liability (AOL) for mobile number [NEW PHONE NUMBER] . On January 31, 2013 we account number removed the aforementioned mobile number from [EMPLOYEE]’s private account and reassigned to [INSTITUTIONS] account number [ACCOUNT NUMBER].
—
VZ Rep said:
Please put the line [NEW PHONE NUMBER] on the AC1000 min plan code 74053 with the 10.00 hot spot. See below.
—
Verizon Government to VZ Rep:
Dear [VZ REP],
Additional information or clarification is needed so that we may process your request. We apologize for any inconvenience this may cause.

Information Needed:

• With the line of [NEW PHONE NUMBER], the requested price plan of 74053: AMERICAS CHOICE II 1000 SHARE EMAIL & DATA+N&W+IN UNL $96.14 is not valid for the simple phone of Samsung Reality SCH-u820.
—
VZ Rep to Me:
Andy are you going to cancel the line that has the iPhone? [EMPLOYEE] does not have a smart device so the line was transferred with his consumer plan.
—
Me to VZ Rep:
[VZ Rep],
[EMPLOYEE] doesn’t really care for his current device and so I’d like to have the number associated with the iPhone 5 that [EMPLOYEE] is currently is using to have that phone number. The plan will be applied to that iPhone 5 that he is already using.
—
VZ Rep to Me:
You need to cancel the number currently on the iPhone 5. Once it is canceled you can then request the device be activate on his line.
—
Me to VZ Rep:
[VZ Rep], I can do this on our website right? I’ll give it a shot later today. Thanks!
—
VZ Rep to Me:
The site will only allow you to suspend. Please send me the line that needs canceling.
—
Me to VZ Rep:
Okay, I’ll repeat myself once again. We’d like to cancel [OLD PHONE NUMBER]
and put [NEW PHONE NUMBER] in its place. Put the number on the iPhone. No new
device, use the device we have. The device that [EMPLOYEE] already has, his
iPhone 5. The device that we have already on our account, the one that
currently has [OLD PHONE NUMBER], should instead have the phone number
[NEW PHONE NUMBER]. This device already has a plan, the plan that has always
been on the device since we started our contract.
I don’t understand why I have to keep on requesting the same thing with
the same details over and over again.
—
VZ Rep to Verizon Government:
Please cancel [OLD PHONE NUMBER]. Once that is processed please take that
device and put it on mobile [NEW PHONE NUMBER]. The [NEW PHONE NUMBER] needs to be put on
the same plan that [OLD PHONE NUMBER] had – 74053 – AMERICAS CHOICE II 1000 SHARE
EMAIL & DATA+N&W+IN UNL $96.14 0408 with the 10.00 hot spot.
—
Verizon Government to VZ Rep and I:
Dear [CUSTOMER],
We have completed part of your request.
Reference Number: [REFERENCE NUMBER]
We have deactivated:
Mobile Number: [OLD PHONE NUMBER]
Effective Date: 02/09/2013 06:35:20
Contract End Date: 02/09/2013 06:35:20
Early Termination Fee:0.0

Second request to do an ESN device change to [NEW PHONE NUMBER] for the Iphone 5 [MEID], the SIM [SIM CODE] was already paired on mobile number [OLD PHONE NUMBER] and can not be used on more than one number. Please either advise a new SIM card that has not been used or we can ship you one for free, just provide a shipping address. ( NO PO BOX)
—
My response to Verizon, both Government and Rep:
Thanks, but since you tossed [OLD PHONE NUMBER] the device Iphone 5 [MEID], the SIM [SIM CODE] no longer has a number. Please set [NEW PHONE NUMBER] to this device.

Please do this, otherwise I’ll have to ask [VZ Rep] to do this again in a few days and you’ll do it anyways. Lets just cut to the chase and do it now. I’d rather not have to make this request a 15th time, literally, just to get one step closer. We’re so close. So unbearably, deliciously close!

I have faith in you all. That a task asked 15 times over two months can be done. We all know it can, so pretending that there are problems is just playing coy. Think of it this way, once we’re done I’ll stop being the worlds most screechiest wheel. I do not like being that loopy wheel on the shopping cart of life that just refuses to track with all the others.

If I can’t get to the end just with our consumer/carrier relationship then at least help me finish this. Think of it not in terms of ESN’s, plans, or SIM cards but rather in the merciful euthanization of this sprawling task from hell that shot out grubby tentacles and turned into one ugly time vampire. Don’t do it for me, do it to kill this wretched task. It must be killed and the only way to do that is to spritz it with the holy water of assigning Iphone 5 [MEID], the SIM [SIM CODE] to [NEW PHONE NUMBER].

Once we are done the angels in heaven will trumpet our glorious success.

LET US FINISH THIS. PLEASE!

Administrator’s Eyes

Posted on by
Reply

Working in IT in Higher Education for the past 14 years has taught me many key survival tactics. Life in Higher Education is special because of the unique specialness of the needs that many of my coworkers have. I don’t want to call anything specific out, I’m not here to hurt anyones feelings.

One of the first things you learn is that no matter what the patina is that people do their level best to project, right underneath it is some of the most kinky, clever, sneaky freaks you will ever meet. I hate to be picky but there is quite literally a 10 out of 10 chance that the truly kinky will be the boys. Perhaps this is higher education, perhaps not, but gentlemen, you are filthy. Damn.

When I started working in my profession I made some basic decisions which have saved my bacon more times than I care to even contemplate. First and foremost of those is cultivating “Administrator’s Eyes” which is the very state that I enter into when I help anyone with their technology. I started it as a habit and now it’s become a perception-altering meditative practice, nearly. When I am helping a client (I don’t call them customers, that’s inappropriate, they are clients) and I am sitting down where they normally sit I will focus my entire attention on the parts of the screen that contain only those pieces that enable me to render assistance. I do not let my eyes wander. It’s not out of some lofty sense of propriety that I picked up over time but more specifically battle-earned knowledge. I cannot, I will not handle the kinky freakish things that my fellow human beings get themselves into. Often times people will say “Oh, certainly nobody does that in a professional setting!” and I point them to teenage boys that spend way too much time in the bathroom with flimsy Scientific American magazines that appear to be on their last legs to keep their covers from falling off… these boys grew up into men and being a boy who grew up to be a man, I can say with authority that the only thing that honestly changed was that our hair started to thin or fall out.

It’s a habit that I recommend every IT professional adopt. It saves you from social embarrassment, even by proxy, and at the core of it stands this central question which each one of us in the IT field must eventually answer: “Can you handle the answers?” This is the first thing I consider before I even allow the questions to occur to me. Almost always the answer is no. A huge orchestra-blaring no. I can’t handle knowing anything. I can’t handle knowing usernames, passwords, websites, or anything at all beyond the thin border of a web browser. It’s not that knowing would endanger my professional life, but it would change my relationship with my clients and I simply cannot risk that. I have relationships that I must preserve, beyond everything else. I cannot perceive porn webpages, anything blah-Tube, even if it’s just online banking, trips to Amazon.com, or the stray Solitaire game being played. I have a deeply rooted and vested interest in knowing as little about my clients as I can manage beyond their presentation to me in the professional setting at work. It really is self-preservation. I do not perceive anything that would naturally be upsetting to anyone else so that the material in question does not change the fundamental relationship between IT professional and client. I suppose in a way, medical doctors take a “Do No Harm” oath, and I suppose I am advocating for IT professionals to take a similar oath “Do Not See”. Help with getting whatever it is up and working properly with sample data or bogus Lorem Ipsum if you can manage it, and even if you can’t and you have to look directly at the entire screen, once you engage the habit of “Do Not See” hard enough you might be able to pull off maintaining this state of blissful ignorance the entire way through your day.

This is something I encourage in all my assistants and people who work with me on IT tasks. I try to impress upon them that their coworkers may not be as pure as the driven snow and that through the adoption of Administrator’s Eyes they can learn a way to avoid the awkwardness that comes when you accidentally stumble onto a terabyte of stored data that people ought to keep at home, under a blanket, probably with a hot shower at the ready. It saves you from ever having to ask yourself that most torturous question “Can you handle the answers?” because I know I cannot. Therefore not only do I not ask the questions, but I don’t even consciously perceive anything that would lead me down that dark alley.

If there are any IT admins that read my blog, what are your thoughts on Administrator’s Eyes? Do you agree or do you think differently? Please comment here or on Facebook, I would love to know, as long as you’ve washed your hands in hot soapy water for a count of twenty. 🙂

PAD 2/5/13 – Call Me, Maybe, Maybe Not.

Posted on by
1

“Describe your relationship with your phone. Is it your lifeline, a buzzing nuisance, or something in between?”

I’ve never understood why people exclaim that their mobile phone is some sort of yoke or control collar that was tied to them. You don’t have to attend to it, even if your company pays for the device. Then there are people who think of mobile phones as possible risks to their privacy. For those who are that paranoid I often get to laughing, “You really think that anything you are fretting about is a risk?” You can’t conduct business without leaving a huge paper trail behind you. Instead of fearing all of that, I say that people should revel in it, nay, wallow in it. What are you protecting?

For me my iPhone is an indispensable intellectual swiss army knife. I use it for many things, work, personal, pleasure, business, you name it. It’s my camera, the loom of my social network and the device where I play Letterpress. I am addicted to it, and I am perfectly fine with the notion of being addicted to a device. I’m addicted to alcohol so what the hell am I protecting? Some image of myself that never existed? What I can’t understand is why more people don’t see the value as I do.

My iPhone is still a phone, and that I suppose will always be true but the device has become much more than just a plain old telephone. Voice is full of noise, errors… problems. English demands so much and then the immediacy and interruptibility of vocal communications just add to the pile of unpleasantness. When you get a call it’s a moment transfixed and pinned to the ground. Someone is imposing their will upon yours, taking up your time, ignoring your flow and your tasks and imposing theirs on top as if the previous did not matter. This isn’t so much a problem with an iPhone as it is a gripe I’ve harbored for a very long time about the more general telephone technology that we all use. Telephones are a lot like walk-up service at work. Knocking on my door, ringing my phone, either of those demands that I entertain a very expensive intellectual interrupt so that I can put whatever it is that I’m working on into a wait-state so that I can switch mental contexts and engage in either a face to face conversation or a telephone conversation.

Just the presence of this technology alone is bothersome, but the language brings even more awkwardness. There is no chance to plan and consider what you are about to write, no opportunities, really, to proofread and revise before sending. The pressure of speech, body language, and freudian (jungian?) slips abound. English, and the culture that surrounds it like a cloud demands a proper greeting, a discussion with turns, and a proper closing. That’s how you are supposed to conduct yourself without seeming rude, insolent, or impertinent. All of this would be fine if it wasn’t for the fact that normal human beings are fleshy water-filled bags of error just waiting to pounce. Modern discourse doesn’t value listening so people tend to talk at each other instead of talking to each other. You can’t get a word in edgewise and because you value the other person you are talking to, you let them just trample on. This creates a self-reinforcing reward for future verbal tramplings. After a few conversations it’s not really pleasurable any longer, it’s a battle. Then there is the proper closings. You don’t want to seem cold or rude so you attempt to close the conversation with some sort of closure marker like “thank you” and sometimes these events don’t actually take and you end up sending multiple passes of closure invitations to the other side. You go from feeling bad about being curt and rude to feeling bad about appearing to be mentally defective.

I have said in the past, and I will say time and time again that text beats voice all the time. Especially for technologically-tied workers like myself. When I am at work, or engaged in any activity really, I often times find myself within a flow. It’s texting and iMessage and IM’ing and email where you can strike a new playing field. Text is planned and groomed, opening and closing control symbols are cliché and common as dirt, so they aren’t a problem, and the way these messages are propagated does not necessarily break flow. In many ways, these technologies are more polite forms of communication. “I need your attention, but it isn’t life-or-death and so, since I value your time, energy, and flow I will send a queue’able message that you may defer until you are ready to accept it.” and I have said time and time again that text communication is much more respectful and gentile than face to face communications or telephone communications.

What about family calls? Yes, this is a point at which all of my arguments fly right out the window. Nothing, not even flow is more important than your family, so for that there will always be a need for telephone technology in the world. I would argue that actually FaceTime technology, which is video-augmented telephone calls are superior to plain-old-telephone-calls because there is so much more there. You see the other person, something that usually takes airfare or a long car ride to accomplish. The level of information in a FaceTime conversation I would argue is far higher than in a basic telephone conversation – you can see body language, facial expressions, so much more than can be carried by voice alone.

At least for me, my family can FaceTime call me, or call me on the phone. Everyone else really should use some text infrastructure. The only part where any of this is flexible is who you consider family? Friends and family? I draw the line at coworkers and professional contacts. If you aren’t my friend outside of work, keep your phone calls – send me an email or a text.

System Hackery

Posted on by
1

Sometimes I find ways to make life easier for my coworkers, things they couldn’t possibly understand but would benefit them anyways. A few weeks ago I discovered a series of system level adjustments to the TCP/IP stack which I thought would benefit everyone. These adjustments, just in case anyone was curious and technically so are here:

/etc/sysctl.conf

kern.ipc.maxsockbuf=4194304
kern.ipc.somaxconn=512
kern.ipc.maxsockets=2048
kern.ipc.nmbclusters=2048
net.inet.tcp.rfc1323=1
net.inet.tcp.win_scale_factor=3
net.inet.tcp.sockthreshold=16
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
net.inet.tcp.mssdflt=1500
net.inet.tcp.msl=15000
net.inet.tcp.always_keepalive=0
net.inet.tcp.delayed_ack=1
net.inet.tcp.slowstart_flightsize=4
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=50

/etc/nsmb.conf

[default]
streams=no
minauth=none
soft=yes
notify_off=yes
port445=no_netbios

I found these files online in various sites and the ultimate goal was to find any way to make the networking work better for clients on their iMacs. Most of these settings make sense if you are using a plain Ethernet system like we are here at Western, most specifically the mssdflt setting at 1500. One setting that I think was causing some issues was net.inet.tcp.delayed_ack. This setting covers how the systems handle ACK packets in TCP/IP communications. Back in the 80’s there was a dispute on how TCP/IP should function and there has been confusion and splits between two camps ever since, and this setting causes issues all depending on which number you set it to, 0 through 3. Apple sets this feature by default to 3, and I had this turned to 0 for many workstations and then I started noticing people having problems accessing SupportPress reliably. Perhaps I was overzealous in the 0 setting, so then the question becomes, how to change the setting without upsetting people?

Of course, ARD to the rescue. I created a new list of computers at work, removed the obvious ones that shouldn’t be touched such as the primary file server and such, and for the rest I copied these two files to the client workstations. Alas, these changes don’t work unless you can reboot the stations. I was able to reboot remotely several unoccupied stations but that wasn’t a real solution. I need to cover all my bases, not only change the system so that the new settings are permanent over reboot, but that they take effect now instead of later. I was on the edge of sending out an email to the group asking them to please reboot their computers during lunch and then it struck me, why not simply ship out the adjustment over ARD? ARD can send Unix commands to connected workstations and it can masquerade as root, so, why not?

The command to find out what this setting has is:

sysctl -a|grep delayed_ack

The command to make a change while the user is online is:

sysctl -w net.inet.tcp.delayed_ack=1

The users have no idea I’m mucking about underneath their systems, way down in the BSD bits, all from my office, over the wire, with them logged in doing whatever it is that they are doing. Now all the workstations have this change effective immediately and when they reboot they’ll get this change applied automatically as well.

I will have to sit back and see if 1 is indeed the setting that will help. We shall see.

Decameron

Posted on by
1

Influenza is a wildfire that is blazing through this state and my office. Many of my coworkers are out sick and at first what I thought was just the standard Influenza might be a few other things. WMU, through the health center and ultimately through the CDC pushed the 2012 Influenza vaccine shot which we later popularly discovered didn’t apparently take into account the strain that is blazing through Michigan and our office. I have talked to a few people who characterized this new flu as “Flu Type A”and I don’t know where they got that moniker from. I also heard that another virus, the Norovirus was blazing across the US, sourced from Sydney Australia. Are these tag-team illnesses or are we mistaking the Norovirus for the Influenza? For me it’s just idle speculation as the practical upshot is, I’m slowly being surrounded by sick people and eventually my resistance will falter, something will happen – either a surface I thoughtlessly touch or some aerosolized agent that I somehow come into contact with.

This has got me thinking about all the popular culture illnesses. Nothing as awe-inspiring as Captain Trips from Stephen King’s stories, but even movies like Hot Zone all lend themselves certain weight to the idea of control, quarantine, and the eventual lapse in vigilance. I haven’t gotten sick (knock on wood) and for that I’m very thankful, but something is knocking on the door and I don’t know if I’m doing enough to protect myself. Much of what I do is probably just a placebo, taking extra doses of Vitamin C, a dose of Vitamin D-3 (which I need anyways, and it probably doesn’t do anything else) drinking lots of hot tea (hot water can’t hurt) and regular drinks at night. Nothing comically appropriate like getting piss drunk every night, but a wee something regularly, wine, liquor, cider, beer. Does it help? It’s not hurting, so why not?

Beyond the things I eat and drink, vigilance visits me in what I do at work and at home. I often times worry that I’m starting to develop a germ-phobia laced with a touch of obsessive compulsive disorder. I know at least somewhat clinically that this activity of washing my hands before I eat (and sometimes afterwards) is only really a mental illness if I am paralyzed because I cannot proceed without cleaning my hands or it somehow impacts my quality of life. There is a small part of me that is concerned that all this handwashing, in hot water, for twenty seconds using rather aggressive soaps is just hastening my seasonal skin issues on my hands. The colder the weather, the drier the climate the more dry and cracked and bloody my hands get. My hands and my legs bear the worst of it, but my legs get a respite as I have them covered up almost all the time, where my hands don’t and pay the price. All this handwashing is just pushing them even harder. At what point will I have that breakpoint of diminishing returns? When will washing my hands mean nothing if I’m bleeding from the cracks from the angry skin on the back of my hands? What to do to cope? I’ve decided that Dove Soap’s line that caters to men, with their moisturizer as part of the soap may be my best effort. I’ve also got a pump bottle of moisturizing sanitizer however as I discovered tonight, sanitizer doesn’t touch Norovirus. Not that I’m really convinced that Norovirus is chewing through the office, but if it isn’t, then it’s on the heels of Influenza Type A.

This very story has played out before. It plays out whenever there are communicable outbreaks and the natural question pops up – at what point does it make more sense to just not go to work and expose yourself? At what point do you stop leaving the house? I laughingly call it the Decameron moment as the people in that book, in order to pass the time recount stories to each other and remain away from the city to avoid the plague. I can’t deny the pleasure of reading the Decameron back when I was in college and if it weren’t for the two other books that I’m currently hip-deep in reading, I would take it right up as it’s applicability in this particular situation is undeniable.

So tomorrow I’m going to have to come up with ways to protect myself at work. Bringing my own soap maybe to start would be okay, paper towels are still the best way to dry my hands as we don’t have any hot-air blowers at work. As for surfaces, it’s going to have to be Lysol and Isopropyl Alcohol as I can’t risk using Clorox on the surfaces at work. I know that Lysol and Alcohol will not likely damage the things at work, but I’m pretty sure that Clorox, even diluted would likely have unintended consequences. I will have to have faith that what I have, plus my nearly OCD handwashing and keeping my distance from people is enough. I have been dallying with the notion of pushing SupportPress down my clients figurative throats and only rendering help over Apple Remote Desktop in order to zero out the touching-of-surfaces vector of possible sickness. I haven’t gotten there yet, but it is something I am considering. I sometimes wonder if anyone has done a pathology survey in regards to electronic forms of communication and that impact on disease spread? What happens if we all switch to video links, phones, and email and shun contact with each other even more than we already have? In a lot of ways, each office could be it’s own Decameron, with people holed up, trying to avoid getting sick and passing the time.

I feel excellent. There is nothing wrong now, but it’s coming. The worst part is not knowing, or rather suspecting that something you can’t see is lying in wait for you and at the very best could make you miserable and at the very worst, kill you outright. Another bit of consideration is what the break-off point is for workplaces all around when a majority of staff is actually sick. At what point is going to work and accomplishing nothing cost more than just staying at home, claiming that you are sick when really, you’re just holed up waiting for the illness to burn past you?

Vectors Hidden In Plain Sight

Posted on by
1

virus cells
While walking back from the bathroom and sitting down in my office I looked around and noticed all the devices that I touch. We are currently witnessing a epidemic of influenza and because it’s a clear and present danger to our health I’m spending more and more time considering ways to avoid it. Obviously there are all the classic things one can do, frequent hand-washing, sanitization, Vitamin C (Placebo anyone?), Tea (Paging Dr. Placebo), supplements (Will Dr. Placebo PLEASE ANSWER THE PHONE!) and as I was sitting back considering all the ways you could acquire an active influenza virus it struck me. Much like wondering how invading Aliens were jaunting past the razorwire like it wasn’t there only to find out they were skittering along in the drop ceiling – a hidden vector of infection: Touch Devices.

Ever since Apple (and others, of course) developed tablet and phone technology in the modern sense, mostly iPads, iPhones, Nook HD’s and MacBooks people have been touching these things and not really paying much attention to what all that touching means. If you wash your hands then your hands are clean until you touch an object, then you have doubt. Did that surface that I touched harbor a virus or bacteria that could make me sick? You don’t know. Obviously life goes on merrily and has ever since these devices have been in our grubby little clutches, but still, just to think about it gave me pause. I was using the bathroom, washing my hands, then touching my iPad. Dirty, clean… dirty? I don’t know. It’s the doubt that grips me.

There is one chemical that I know will disinfect non-porous surfaces and most likely will not damage those surfaces and that’s isopropyl alcohol. So at work I have asked my S3 to follow a new protocol during these months when these viruses are on the loose and we’re in the trenches when it comes to being vectors ourselves because we touch a lot of things that others touch. So now, at work, whenever we see an iPad, an iPhone, or a MacBook we grab a microfiber cloth, wet it with alcohol and wipe down the entire surface. Each time. It’s a lot of wiping and a lot of alcohol, but what if we kill a virus that otherwise would have made the epidemic worse? Isn’t it worth the little bit of time and effort to kill a bad thing early on than have to suffer its effects once we’ve succumbed? I think so.

If you have non-porous surfaces that you touch very frequently, like we do, I strongly recommend wiping things with a rag soaked in alcohol. You may very well perform one action which could stem the tide and spare you and the people around you the danger and inconvenience of this particularly nasty influenza virus.

Problems & Puzzles

Posted on by
Reply

problem

 

I took a long while to hack at this problem and then I decided to be cheeky and post it to my door at work. If you know the answer, please keep it to yourself. If you don’t know the answer, don’t feel really bad that you can’t figure it out – it took smarter people than I a long time to get the answer. If you want to know if you are right, feel free to email me or iMessage me with your answer and I’ll let you know if you are right or not.

Infrastructure

Posted on by
Reply

Limited Options

Today the city of Kalamazoo will be turning off the water to a series of properties that includes my workplace. The general statement is that the water will be off from 9am to 4pm today, although of course, they will endeavor to not let it be off for any longer than it really has to be.

This event got me thinking about the topic of infrastructure. How much of the first world lifestyle is possible because of things none of us see or notice until they are gone. It’s a curse of ubiquity and constancy, everywhere you go there is running water, there has rarely ever been situations where running water was not available, so then, what happens when it’s gone? There are so many pieces to modern living that we all take for granted and in doing so we have become functionally dependent on these things. This is a savage dependency, without running water, electricity, fuel, and information services how can the average of us cope? That’s both fascinating and terrifying. To see how average folk would respond to the sudden loss of first-world fundamental services, to the failure of first-world infrastructure is a possibility to see how we can cope and the terror that will descend upon us when we find we cannot cope.

Politics touches on so much of our lives, and even here in terms of infrastructure it lumbers along. How much time and energy have we invested in our roads, in our electrical systems, our fuel delivery systems and our water systems? We used to before we spent all our money on wars, but now? Ever since the I-35W Bridge failed the nation turned at least one wary eye to the conditions of our infrastructure. How much of it is rotting away, needs maintenance, needs money. How can we function without it? Can we?

The government, through their primary readiness website ready.gov has resources a-plenty, but really, how many of us are ready for any of it? At work we have fire drills and tornado-readiness drills, but what about other sorts of disasters? What about infrastructure failure drills? What do we do when we have to span a day at work without some fundamental service, such as electricity, information services, fuel, or water? That’s what has captured my attention currently.

Getting back to the topic of infrastructure, if there is failure, do the systems we have have enough robust redundancy to cope with failure and can we quickly recover function from the gaping maw of failure if it should strike? It’s clear and present, when it comes to water supply that we may be running too close to just-in-time delivery for comforts sake? What about having a large container of potable water in a gravity-fed system just in case we need it? There rarely is such a thing because the system has rarely failed and we don’t feel the need for the extra design or cost. Just because something has rarely failed does not necessarily mean it is proofed against failure.

So when will the water go out? Nobody knows. Maybe at 9, but it’s still running so I doubt that. What is our plan to cope with this loss of one part of the infrastructure? We don’t have one. We don’t need one, or do we? Nobody is really clear and instead of asking, we’re just sitting around warily looking at the sinks and wondering if it will work, and if it doesn’t, where are we going to go when we need to use the bathrooms? Good questions, all.

PAD 1/7/2013 – Helplessness

Posted on by
Reply

“Helplessness: that dull, sick feeling of not being the one at the reins. When did you last feel like that –- and what did you do about it?”

I rarely have this feeling. Almost always I can either acquire control or I can find some way to escape the situation. There is one time, not really a matter of helplessness, but one of catastrophic failure that recently happened to me that I can write about. Several months ago I had a server, a Dell uber-tower that was 9 years old and suffered a total systems failure while I was actively trying to backup files on that server because I just didn’t trust the tape system to work properly. Turns out I was just a little bit psychic I guess, because half-way through my attempt to backup the machine, there was a catastrophic fault on the servers motherboard which pretty much hosed the entire machine.

I needed to get the data off the tapes and get a new server set up as quickly as I could. The realization of the failure hit me right in my gut. It’s where cancers always feel stress, like a knot in your stomach. Not nausea, but it felt just like I had been punched right in my gut and was short on air.  I arranged to get a new server up and running and got the system back on its feet but needed some of those files on the tapes in order to rescue everything. I called all around and nobody had that old technology still, so I had no choice but to resort to data rescue services. They got the tapes, and after a protracted back and forth regarding them I received the files that were written to the tapes on several DVD’s. Turns out that Backup Exec lied about making successful backups all along and that all my tapes had filenames and directory structures, but no actual data to any of the files. They were all zero kb. In many regards I had spent a lot of money for DVD’s with just the headstones of the files I needed, and no bodies.

Thankfully I had mirrored the system to another server a few months back and I was able to rescue a majority of the data that I really needed and only lost about two months of my coworkers work. It was bad, but it wasn’t heart-attack bad.

Since then, I resolved that I would never again trust any data to tapes, and thankfully when that server died it took the last tape drive with it, so there are no more tape drives in use in my office. Tapes are very 20th century things and there are better ways to store and backup data now.

I will never forget that feeling of being punched in the gut. Thankfully now the relevant systems are under contracts and warranties and letters of agreement and whatever else we could find to properly cover-my-ass and ensure that this would never again happen to me. It might happen to someone else here, but it won’t be my fault, ever again.