Honestly, you might as well skip washing your hands if you are going to touch the water faucet handle with your clean hands to turn off the water. The germs that were on your hands when you turned the water on, will be there when you turn it off and get right back on your hands. The same goes for the door handle.
And maybe you won't catch anything deadly, just a cold or two. But would you lick a total stranger on their hands, or anywhere else? Because when you touch a public door handle (or escalator, or shopping cart, or elevator buttons) then touch your face, you are getting all those strangers' germs right into your body. So go around and lick hands, its the same thing.
I might sound paranoid, but my handwashing procedures (using paper towels to turn the water off, open the door, etc) are what medical professionals use and its required by law where I work. And if you saw what some of the people do where I work, you'd be paranoid too. And we do take them out in public, so they are also touching all those public door handles. And you don't want to know what their hands have been in...
I never wash my hands!
Just kidding
am I?