The daunting task of actual musical creation is a little much for a young one to handle. But if they've expressed an interest in the computer (and let's face it, if you use one at home, you probably have a time and a half keeping little hands away from the mouse), there's more than a few programs out there that let them experiment with music and sound in a playful environment. Heck, some of them are even fun for adults as well!
Here's one of my favorites:
This Japanese program has no English instructions, so you and your young one will be at about the same level of expertise going in. Click on one of the ten buttons at the top of the screen to enter an activity, and then use the mouse in a variety of ways to interact with the simple shapes and make music. Hit Enter to return to the main menu. The abstract forms and very basic interactions make this an oddly soothing toy that can actually be used to create some pleasing music.
Download it here