I'm looking at the image now, and I'm not sure what to make of it, to be honest. It doesn't look like a lens-related problem... it looks like something related to the sensor, perhaps. In fact, it looks like the noise-reduction is on too high, and it's destroying the image resolution.
Do you have the same type of results with both of your lenses? That would definitively rule out a lens problem, I'd think. It looks like your lighting equipment is working just fine; that's certainly not an issue here.
It's also not an exposure related problem, as I initially guessed... your exposure is fine.
What I'd do is this:
Switch you camera settings to shoot RAW + JPG fine, and shoot a few test photos. They don't have to be in the studio (in fact, it would probably be more interesting for me if they weren't :)
and if you want to test the other possibilities,
Shoot at least one photo with each of your lenses.
Shoot at least one photo from a tripod, using the camera's timer or a remote release.
The important thing is to see the difference between your JPG and RAW files. You should be shooting RAW all the time, anyway. I know that they can be a little more tedious to work with, but JPG is throwing away 98% of your captured data per channel (256 level in JPG vs 16,384 in 14-bit RAW). In any case... if the resolution problems you're experiencing are related to the camera's settings, then you'll have problems with the JPG files but NOT the RAW files, since noise reduction, etc, is not applied to RAW.
If you do take a few test photos, I'd love to see them... especially the RAW. Just let me know when they're uploaded.
The RAW looks better, but the resolution still isn't quite where it could be. Unfortunately, I have some work to do this afternoon, so I'll have to return to this topic in a few hours or later this evening. Until then-
I just took another look at your image. I don't know why I didn't notice it at first, but I just realized that you're using continuous lighting instead of flash! That being the case, I think that the minor amount of blur becomes much more easily understandable as camera shake... especially with a non-IS lens.
If you had been using flash only, as I had assumed (I thought you meant that you were using small flash units instead of studio strobes), there would be virtually no possibility of camera shake. As is, 1/125th sec is enough for just a little camera shake, and that will create just enough blur to create the effect you're getting.
If you take a shot like this with the camera mounted on a steady tripod (and you use a remote trigger or timer), that should solve the problem and improve your image quality. Also, processing the RAW file with about +15 on the "luminance noise" setting in the Noise Reduction settings will help get rid of any pesky digital noise.
Thanks. But I am using a tripod and 2 strobes for taking shots now. Again I use strobes which are not professional. I adjust the output just by looking at the shots I take. How can I increase the resolution and quality more? Actually, is the image quality alright at all?
OK. The overhead light in the image (looks like a bare CFL bulb) is not a primary light source :) I am curious whether it's contributing to the exposure, though.
Your image quality in the RAW file is good; no real problems. Any time that I zoom in to the 100% level and start looking at detail, I start to notice problems... whether it's in my own work or someone else's... so you don't need to be too worried. I do notice that there's some minor movement in the face, but in the hands, there's none... they're as sharp as can be expected.
Let me show you an image that I shot with the same camera (Canon 60D) with a similar lens (17-50mm f/2.8) at similar ISOs, for comparison. Mine were hand-held... no tripod (no sharpening applied to either image).
As you can see, your image quality is at least as good, if not better, than mine. You have nice, fine detail in the fingers, and the stitching is visible in the clothing.
What worried me, initially, was the quality of the JPG. If you compare the detail in the hands, the JPG looks pixelated and blocky. This is just the result of your camera settings, though... it may be that your sharpening, contrast, noise reduction, etc, are set too high. However, if you're processing the RAW files for professional use and just using the JPGs for reference, then it doesn't matter... your RAW files look good.
I don't see anything that makes me think that you NEED a new lens, but all photographers benefit from buying the best optics that they can. For the studio, you might consider buying a large aperture prime lens, like an 85mm f/1.8 or even a 50mm f/1.4. They're both very sharp, and you'll get much better control of depth-of-field than you have with your current zooms.
As for improving your performance with what you have... the only thing that comes to mind is to make sure that when you're taking flash photos in the studio, keep the ambient light level as low as possible. Ideally, you'd want your exposure to be completely black if your flashes don't fire. You may already do that, but the overhead light raises the concern for me.
I am not primarily a studio photographer myself, though I do some studio work. I did take studio classes in college, of course, and do quite a bit of location work with studio lighting, but I don't consider myself to be a studio expert. So, I may not be able to answer all of your questions, but I'd be happy to do what I can.
Converting formats will never improve image quality. With most format conversions, you'll lose quality, but it doesn't depend on the software... it depends on the codec/format. There are lossless formats, but most are not (especially for video). I have never used the one you mention, so I can't say about that particular piece of software.