A ritual performed by Jedi younglings before they move on to be a Padawan, is to venture into the crystal caves where they have to "find" their crystal. This crystal is what they then use to create their lightsaber. The finding of the crystal is believed to be guided by the force. This then means that the crystals colour will depend on the jedi. It in essence chooses the Jedi. A Jedi's first saber is a reflection of the user, but as the user changes, it becomes more and more difficult to determine the character of the Jedi by looking at their sabers. So of course anyone can use another saber, but the original crystal is a reflection of who the Jedi was when he found it..