10:00 AM You know, this is getting old.

Yesterday, I was ordered out of the office by my boss. Why? Well, it seems I'd already been here for 40 hours for the week. When it's only Tuesday, that's a bad thing. Well, OK, I can see his point. But it's not like I was just loafing around; I worked my tail off the entire time.

Doing what? Oh, nothing much. Just a complete rebuild of the entire network without any downtime. No problem? Let's see YOU try it. <G>

I've been planning this for weeks. I knew exactly what I had available, where I had to end up, and how I needed to accomplish it. The best way would have been to start on Friday at 5:00 PM, kick everybody out, and spend a nice, relaxing weekend in air-conditioned comfort at the office, reorganizing and rebuilding and having Keri fetch me food and beer as needed.

Someday, I'm going to have to try it that way.

Instead, I started on Friday by rebuilding the system that became the router. Nothing complicated to start with; Linux Mandrake with just about every optional package stripped out, upgraded to the latest (2.2.16-3) kernel, and started loading security software; Tripwire for intrustion detection, PortSentry for active firewalling, SSH, and shut EVERYTHING off except SSH. Set up masquerading, some firewalling rules, and the basics of the network routing.

By now, it's Saturday and I start a series of processes on the Oracle database server that must be completed by Monday. Unfortunately, for this to work, the VPN to corporate must be up and running continuously, which meant I couldn't move the VPN behind the firewall, nor could I modify the server's IP settings. The process took until Sunday, with lots of remote monitoring and cussing.

Sunday noon, I went in to the office. Have I mentioned that we don't own our office space? We rent it, temporarily, while the permanent office is redecorated and rebuilt. Well, this pleasant little place doesn't run the air conditioning at night or on weekends. My office has 4 dual-processor IBM NetFinity 5600 servers in it, a few workstations, and some other heat-generating odds and ends. 92 degrees. That's how hot it was. It got hotter. Bleh.

Anyway, I reconfigured the VPN to work from behind the firewall, copied my pre-planned scripts into place, restarted the network on the router to initialize the configuration, and went around changing the IP settings on all the servers.

The VPN didn't work.

A little before 9 on Sunday night, I finally gave up and went home, convinced the problem was something simple that I was too tired to see. Went home, grumbled a lot, checked a few more things, and went to bed. For those keeping score at home, I worked about 10 hours Sunday, 8 of them at the office.

Monday, I was in the office at 6. 2 hours later, the management of the rental agency came in, and I was able to rewire the network to go through the new router, rather than just the portion in my office.

Still no VPN. Time to call RedCreek, the company that manufactured the VPN router. I won't go into the reasons why I didn't call Siebel's central IT office. Okay, no problem. Go the their website, click the support button, and...

What? No phone number?

Email only. OK, fine, fire off an email listing MY phone number and email, a brief but complete explanation of the problem. That at least gets us a toll-free support line.

Two hours later...

The support tech determines the problem must be the firewall. It's not passing the authentication packets, which are Protocol Type 50. (WTF is Protocol Type 50, you ask? I do, too.) After much back and forth, we agree that it isn't RedCreek's job to support linux, and a thorough search of the web and newsgroups, plus a few assorted mailing lists, and a last-ditch effort mail to the Daynotes Gang. Nope. Nothing. Moshe Bar comes up with some helpful tips, but in the end my final diagnosis was that something in the firewall was hosed, and that I lacked the time and expertise to find it. Other options?

Why yes, the RedCreek guy says. Put the VPN in parallel with the firewall. Why didn't I do that first?

Good question. Maybe because the documentation of said VPN says nothing about doing it that way?

Yes, well, that might be it. Anyway, it would be perfectly secure; the VPN can be configured to accept packets only from one specific IP on the outside, so there shouldn't be a security hole. And firmware updates are of course free, just in case any are found.

Well.

How many problem will this cause? Oh, well, none at all sir, just change this and this and this and move this over here...

Oh and the remote location has to reconfigure as well. Hmm. That could be a problem. Do they have to change much? Well, no, just one item.

Well... ok. We should be able to manage that. Lots of phone calls, some yelling, and only a small amount of swearing and rank-pulling later, the changes were made. The VPN worked. Bliss.

Um. Almost.

The router now needed to be rebuilt again. Well, not rebuild. But all those scripts that I'd prepared were so much wasted space; they assumed the VPN would be where it was not, doing something completely different from what it was actually doing. So THAT needed redoing. Then San Mateo called and explained that they'd given me the wrong network segment, so change to use THIS class C for the private net instead.

Uh-huh.

Reconfigure the VPN router again, the main router again, and all the servers again. Fortunately, we'd decided to put the office on DHCP, so the workstations were a snap. Everything seemed to be working, so I went home. And spent several hours logged in remotely, resetting the security and figuring out why one server wasn't responding to requests from the main Siebel network. (Short answer: Matt made a boo-boo in the routing configuration. Gateways are important; it's a good idea to put in the right address.)

So when I dragged in on Tuesday, I had already worked 34 hours that week. Around 1 that afternoon, I passed 40. Around 4, my boss realized this and kicked me out, even to having someone give me a lift. And although I did in fact think about making a post (thanks to all those who asked, yes, I am in fact alive) my exhaustion, food, and beer were just too attractive. <SEG>

But today's a new day. Time to crank up something odd on the cupholder (Wild Thing performed by Kermit, Animal, and Floyd, I think...)

9:30 PM Relatively quiet day today. Started writing the documentation for my beer-truck book, handled a couple of minor emergencies, started the backup system, and generally kept the office's world spinning on its access.

What's a beer truck book? A red binder that sits in a locked drawer of my desk; it contains all the information any competent admin would need to do the daily tasks I do, and most of the predictable emergencies. The beer truck book exists in the event that I get hit by a beer truck; I don't tell people root passwords, I don't hand out maps of the network, I don't make public lists of all the machines and their functions. That's all in the beer truck book. My boss and his boss know where it is; if for some reason they should need it, they can find it.

The backup system has been sitting there since before I was hired, but I hadn't had a chance to deal with it. Nothing complicated right now; an IBM-made external DLT tape system and Veritas Backup Exec software. I got the hardware in place and functioning, now I need to get it talking with Backup Exec and a backup schedule set up.

The emergencies were fun; about 2:30 this afternoon, network connectivity disappeared. I was in the middle of downloading a utility for the router (ironically, a script to check connectivity and measure downtime) and suddenly, the download stopped. My first thought was to check my network cable; it wouldn't have been the first time I shifted the laptop and yanked the network card dongle loose. Nope. Next I pinged the router. No joy. Great. The router's down. Right now the router is on my desk, patched in to the wiring closet with two wall drops; I turned and hit the monitor power button, checked it over. No, the router was up and running, everything as it should be - but no traffic. Ok, try to ping another server, both from the router and my laptop - nothing on either. Others started complaining at about this point, so I knew it wasn't an isolated problem.

About this time, I realized that I couldn't ping Vantas' outside router, either. (Vantas is the executive suites' company we rent our temp offices from.) And I COULD ping the VPN router (directly connected to the main router) but I COULDN'T ping the other side. Which meant there was only one possibility; the wiring closet was down, and AT LEAST our main hub and the Vantas router were dead.

A few minutes later, I learned that a breaker had blown in the wiring closet; Vantas' technician got the power restored, and a few minutes later, we were back up. For about twenty minutes. Then it went down again.

I never did learn exactly what went wrong. I have my suspicions, but there's little I can do about it, anyway. I did finish downloading the utility I was originally after. Written (rather well, I might add) in Perl, the program is called Downtime. The link is to the ftp directory where the program can be downloaded; it works by pinging one or many different addresses, and keeps a log of when and how often they are down. It can play a .wav file when the connection is lost, and another when the connection is restored. I've got it pinging (at intervals) the Vantas router, and a server on the other side of the VPN. I need to modify it a bit; right now it simply logs downtime. I need to configure sound on that box and set the sound files, and I'd also like it to email me. (I know, I know - why email if the connection is down?) The possibility I like best is to connect an old modem and have the script page me with a number code when there's a problem.

And that was my day. Hope yours was good as well. Later.

9:20 AM Well, isn't this a cheerful morning... grey skies, threat of rain later. Perfect. <G>

A day of documentation and backup-system installation lies ahead; both are boring, but have to be done. Ah well.

Chores for the weekend; first, I have to take Keri to the X-Men movie tonight. She's a big fan, and although I never really got into it (my comic books were written by Heinlein and Pournelle and Niven and Asimov) it really does look like it has the potential to be an excellent movie.

Second chore is this laptop. I have to decide if it's worth the effort of maintaining Mandrake on it; on top of that, if I do decide to keep it, I need to do some serious work on it, which I'll probably do by reinstalling. I know, I know, I shouldn't to do that - but this way, I'll also take the opportunity to resize some of the partitions. If I do keep it, I need to shrink the space Mandrake uses; I need more room in Windows for applications I need.

And that's about it. I am thinking about starting to post on weekends, since it's harder to do so on weekdays; I may or may not do that this weekend. See you all later.