Jobs that beat the caring out of you

Ok, Since Jen and Grant started this, it’s time to share some horrors…

This happened during the five or so years I was doing consulting type work with a small consulting company (which was itself a bad idea, but that’s a whole ‘nother story). Work was a tad on the sparse side at the time and I was looking for anything. Enter a logistics company that needed some integration work doing.

SSIS and ETL work for a couple months. Not ideal, but how bad could it be?

Bad. Very bad indeed, and mostly because of management. There’s one thing at least I can say about this place, they taught me how not to manage an IT department.

I’m not going to cover everything that happened at that place, just some aspects of one project. It was a package tracking system, intended to take waybill data, vehicle tracking and some other bits and pieces and make it so that any delivery/shipment could be identified as being in a warehouse or on a truck, and specifically which warehouse or truck.

First problem. It was 6 months of work at least. We had 6 weeks. Not 6 weeks to a deadline that everyone understood was going to be missed. 6 weeks to the date that the company CEO had been told this new tracking system would be in use. Nothing that can possibly go wrong there.

First problem, the project manager. He managed by gantt chart, but that’s not all that uncommon. What was less common was that he appeared to have no concept of time management at all. I worked for them 3 days a week. During one Monday afternoon project meeting, I gave the project manager an estimate of 10 days for chunk of work. I found out later that day that he’d promised it would be in production the following week Thursday. 10 calendar days from the time he was given an estimate, at a point where I’d have had 5 working days to finish it.

That got me yelled at by the head of IT.

Second problem. The project manager and BI specialist (read Excel report writer). They both repeatedly agreed on things in meetings, and then told the head of IT something completely different, something that cast them in a good light and the developers as incompetent idiots. Once is an accident, twice might be coincidence. Three times or more however…

I got into the habit of openly recording the meetings on my phone (for ‘documentation purposes’)

Third problem. The head of IT. I’d say she was a little on the side of micromanaging, but that would be like saying a Joburg thunderstorm is a tad damp. She also had a tendency to overreact, and to listen to only one side of a story before reacting.

The last straw of that particular project was the Monday when the project manager decided that the system we were working on was going into UAT for user testing (not testers, business users). It was not in any way ready and I told him that in the meeting, as did the other developer. After listening to out explanations he agreed and said he’d get another week. It might have been enough.

Next thing I know one of the other devs tells me that the head of IT wanted a word.

No, she didn’t want a word. She wanted to scream at me for over 5 minutes, at the top of her voice, in an open plan office, in front of everyone else about how irresponsible it was to suggest that the project was ready for UAT over the project manager’s recommendations, tell me how incompetent I was, how useless I was, what a terrible developer I was, that I was a liar, lazy, and that she would have me fired and ensure I never got another IT job.

I didn’t walk out. Not quite, but I did call my boss immediately afterwards. See, I didn’t work for her. I was doing the work on contract. She couldn’t fire me.

My boss at the time was the softest spoken person I know, he never raised his voice, never lost his temper, never sounded irritated no matter what. That afternoon, when he had to drop all the other work he had planned and come out to the logistics company, that afternoon I heard him angry.

Somehow the logistics company is still in business. I have no idea how.

A rant about presentations

My company’s internal conference is in a couple of weeks, so this seems like a good time to have a quick rant about some presentation failings I’ve seen over the last year or so.

If you want to, or are planning to present at a conference (or even just a usergroup), please, please, please pay attention to the following.

Don’t read your presentation

Please don’t read the bullets on your slides one by one. Please also don’t read a speech off your phone. If I wanted to have something read to me, I’d get an audio book.

A presentation should feel dynamic. It is, and should feel like, a live performance.

If you need reminders or cue cards, that’s fine, but put keywords on them, points that need to be discussed, not the entire speech

Watch your font size

This is for the slides but especially for the demos. Font size of 30 is probably the smallest you should be using on slides.

In demos, if I’m sitting in the back row and can’t read the code, there may be a problem. My eyes are not the best though, so that might be a failing on my part. If, however, I’m sitting in the second row and can’t read the code, there’s definitely a problem.

If the conference insists on, or offers time for a tech check, take the opportunity to check your fonts. A tech check isn’t just ‘does my laptop see the projector? Yes, done.’ Walk to the back of the room, go through all the slides, start your demo, walk back to the back of the room. Make sure that everything is clearly visible.

Minimalistic slides

Please don’t put an essay on your slide. Please don’t have fancy animation (unless you’re doing a presentation on animation). Don’t have things that flash, flicker, spin or dance.

It’s distracting, and it probably means your audience is watching your slides and not listening to you. You should be the star of the presentation, not your slides. They’re a support character.


I like the Visual Studio dark theme. It’s nice to code with, it’s absolutely terrible on a projector. Especially if the room is not dark. For projectors you want strong contrast. Dark font on light background usually works. Dark blue on black does not, two similar shades of blue doesn’t.

Check that your demos are visible, check that the code is readable from the back of the room.

Learn how to zoom in, whether with the windows built in tools or installed apps. Use the zoom any time that what you’re showing may not be clear.

Repeat the question

Especially if the session is being recorded. Your voice is being recorded, the audience isn’t. It is so frustrating to listen to a recorded session, hear a minute of silence followed by the presenter giving a single word answer.

Even if the session is not being recorded, acoustics often make it possible for the presenter to hear a question while part of the audience hasn’t.

It also gives you a chance to confirm that you heard the question correctly and gives you a few moments to think on an answer.

DBA Hell

On the first day of DBA hell, the server gave to me
A database with damaged system tables and no good backups (1)

On the second day of DBA hell, the server gave to me
Two databases with widespread corruption, no backups (1, 2)

On the third day of DBA hell, the server gave to me
Three suspect databases, no backups (1,2,3)

On the fourth day of DBA hell, the server gave me nothing, cause I didn’t have a job any longer…

How does one end up with a critical production database that has no backups? I could kinda understand if the backups were damaged, if the corruption went undetected for long enough that it was in the backups as well, but to have no backups at all? Of an important database?

The only excuse for having no backups is if the database can be trivially and completely recreated from another source with minimal impact to the users. This is not the normal scenario.

There’s an immense amount of information available on backup and restore strategies.

That’s just a quick list, there’s far more information available than that. Enough that there’s really no good excuse to not have backups when they’re needed.

As Steve Jones (blog|twitter) is fond of saying “Good backup, good resume. You only need one”

The Root of all Evil

Or “Shot gun query tuning

There have been a fair few forums questions in recent months asking for help in removing index scans, loop joins, sorts or other, presumed, slow query operators. There have been just as many asking how to replace a subquery with a join or a join with a subquery or similar aspects of a query usually for performance reasons.

The first question that I have to ask when looking at requests like that is “Why?”

Why is removing a particular query operator the goal? Why is changing a where clause predicate the goal? If it’s to make the query faster, has the query been examined and has it been confirmed that query operator or predicate really is the problem?

The title of this post refers to a comment I’ve seen again and again in blogs or articles about front-end development. “Premature optimisation is the root of all evils.” It’s true in the database field as well.


Developers vs DBAs

I was listening to a podcast on nHibernate this morning. Most of it was very good, but there was one section that absolutly made me see red.

The guest on the show was asked how the DBAs felt about the use of nHibernate for a new app on an existing (old) database. The reply essentially was ‘We haven’t told them and we’re not going to tell them.’ Apparently the plan was to only tell the DBAs about this app if there was a problem with it in production

Now there’s a whole lot of problems with that scenario and I can sum most of them up in one word – Trust.

If an application was mostly to be written in C#, but there was a critical portion that depended on (say) Biztalk no one would seriously suggest keeping the biztalk expert on staff in the dark and doing it completely themselves. So why do developers do that with databases?

I’ve often heard developers complain that the DBA doesn’t trust them. Guess what, pull this kind of stunt and they’ve got a good reason for not trusting those developers. Go behind someone’s back, hide stuff from them and they’re not going to trust you. No big surprise there.

The whole ‘them and us’ attitude that a lot of developers and DBAs have is, quite frankly, stupid and highly counter productive. The DBAs are the ones responsible for the database portion of custom apps after they become production, they should have some input during the development phase whether it be on the database design, the stored procedures or just ensuring that the app will scale to the required production load.

So, in conclusion…

Developers, speak to the DBAs, get them involved in projects early. You may be surprised how much value they can add to the project and how much smoother things can go when everyone’s working together instead of fighting each other. Of course, if your DBAs are the arrogant, overbearing type that give the profession a bad name, if may not be that easy.

DBAs, if you know that the developers avoid you, hide projects and prefer to go their own way, first take a look at your own behaviour. Ask to be involved in the early stages of development. Chat with the developers, see if there’s anything that you can help out with or any areas that they need assistance. You may be surprised how many problems can be ironed out before they become problems in the production environment. Of course, if your developers are they type that triggered this rant, it may not be that easy.


Bad Advice

It’s no secret that I’ve been fairly active on a couple of SQL forums for a couple of years. In that time I’ve seen all sorts of behaviour on forums, good and bad. By this point, most of that just doesn’t bother me any longer. There is one thing however that still gets me angry when I see it. Blatently bad advice.

Now, I’m not talking about first-attempt solutions that solve half of the query problem, I’m not talking about honest mistakes and I’m not talking about attempted answers to questions so vague they’re near-impossible to understand. I’m talking about advice that’s so bad it’s dangerous. I’m talking about things like this:

Q: My transaction log’s very large. What can I do to fix this?
A: Stop SQL, find the ldf file, delete it and then start SQL

Q: Dropping a clustered index on a large table takes a long time using drop index. Is there a faster way?
A: Run sp_configure “allow updates, 1 and then run delete from sysindexes where id = OBJECT_ID(‘MyTableName’) AND indid = 1


There’s two main problems with bad advice.

Firstly, the person asking possibly doesn’t understand enought to realise the advice is risky, and if they follow it without testing they could end up in a much worse situation than they were. Dependiong on the circumstances they may end up in trouble with their boss, they may even end up getting fired. Whether they realise the advice is bad before or after trying it out, it’ll erode their faith both in the forum and in the person who provided the information. That leads to the the second problem.

The second problem is damaged credibility and reputation. A good reputation is so hard to get in this industry and so easy to lose. Bad advice damages the forum’s reputation and the reputation of the person posting the advice. Brent Ozar puts it very well: “Being an expert means having credibility. It doesn’t matter how much you know if people don’t trust your answers.”

Then, of course, some other person on the forum has to come along and correct the bad advice and explain why it’s bad so that people who find the thread via google won’t think it’s useful

Bottom line, bad advice helps no one and hurts a number of people.


SQL Injection

This is a bit of a rant, so please ignore it you’re looking for technical info.

There’s been a fair bit of news on SQL injection in the last week or so. Mainly cause some people figured out a way to automate the exploit.

What scares me if the widespread lack of knowledge of SQL injection. I’m fairly active on a couple of the SQL forums and on monday this week there were 2 posts by people who have had their databases hacked via a SQL injection exploit.

If this was a new exploit, I wouldn’t be so disappointed, but it’s not. SQL injection’s been around for years. I first read about it in 2001 when I started in web development.

So, why 7 years later are people still being hit with it? Why does a quick google search turn up a number of web sites with entire queries in the url? Sites including some government organisations, a couple of police departments, online stores and the coast guard (No, I’m not posting links. Search for yourself if you’re curious)

Completely preventing SQL injection is not hard. If web pages call the database using properly parameterised calls to stored procedures then SQL injection cannot be done. Set minimum permissions in the database for the web user and it’s even more secure.

So, why is it that so many sites, new and old, are still vulnerable?

Edit: For some in-depth info on preventing SQL injection, see this blog post on Technet

On recoverability

Had a lovely situation this morning.

I can’t go into details, but essentially a database had no off-site backups, no tape backups and the only full backups were on disk. Then the SAN glitched and both the data files and backups were corrupted.

I’m probably preaching to the converted, but I don’t think there are many more important things on a production server than ensuring good backups.

However, having the backups is not enough. Can they be restored? If there’s a complete drive failure of all drives connected to a server, can the databases be recovered?

When the pawpaw hits the fan (as the local saying goes), that’s the only thing that matters.