You're coming from a very high end DSLR that has wickedly fast auto focus, focus during burst modes, etc. The X100 is NOTHING like that. Its a very deliberative camera to use. Its AF is very good but its not fast (even by compact camera standards) and until you learn its subtleties you can miss focus or have a lot of trouble focussing in low light. It also really helps to understand parallax in terms of understanding how to best AF with the optical viewfinder. Its a great camera but it seemed like a LOT of folks who went from a DSLR to the X100 when it first came out were highly dis-satisfied. So, if that all sounds OK to you, go for it.
But if not...
Consider one of the better new m43 cameras. The image quality isn't QUITE as good as the X100 or other APS sensor cameras, but its very good and only really falls off a bit in very low light (and its not BAD in low light). These cameras (Olympus EP3, Panasonic G3 for example) are MUCH MUCH MUCH smaller than your DSLR, but are very fast in operation, have excellent auto-focus, etc, and you can shoot it in a way you're probably familiar with. You can also get a few lenses to cover your favorite types of shooting and they'll be MUCH MUCH MUCH smaller than the equivalent DSLR lenses. You'll be able to walk around with a pretty complete m43 kit in a pretty small and very light bag.
You need to go play with some of these cameras somewhere - don't take our word for any of it. You have a lot of experience with some pretty high end gear and only you'll know what feels right to you...
-Ray