It depends on what you want to do. If you're just screwing around, there a few possibilities, some of them even free, like Audacity (free recording program). If you're actually planning on making
real music, then here's some advice:
Firstly, assuming you want to get into this semi-seriouisly, you'll need a recording interface. Don't use the crappy built-in soundcard on your computer (Realtek, for instance). Those things are ok for listening to music, but weren't designed for recording. So, you'll get cracks and pops and latency and a generally awful sound. An interface is something that usually plugs into your computer via USB or firewire and acts as a soundcard (for recording and playback).
You can spend thousands on an interface, depending on how many features you want, but if you're looking for a budget one, M-Audio, Presonus and a few other companies make interfaces that are in the $100-$200 range and are GREAT quality for home recording. I use a Presonus Inspire, myself that I paid $200 for - and, BTW, it came with Cubase LE, a lite version of Cubase. A lot of companies bundle their products with the lite versions of recording software as a perk so you buy their products. So, you pay $200 bucks or so and you get the interface and the software. Not a bad deal!
Secondly, again, assuming you're into this somewhat seriously - get a pair of monitors. Listening to your music and mixing it on computer speakers and other stuff is not a good idea. When I first got into recording, I made this mistake and I always wondered why my music sounded awful whenever I burned it to a CD and brought it somewhere else. Well, the reason is this: computer speakers aren't flat; they color the sound by adding bass/ treble, etc. They make exisiting music sound good.
Monitors on the other hand are completely flat. That's what you want when you mix, so you get the most accurate representation of what your music sounds like. That way, when you mix, it'll ALWAYS sound that way (or extremely close to that) whether you play it in the car, CD player, etc.
I bought my Samson monitors for about $200 as well, but I'm sure you can find them cheaper. People are always selling these recording interfaces and monitors, etc. on Craigslist, too. So, you might be able to save even more money there by check it out where you live.
I won't get into microphones and the whole guitar amp modeler vs. real tube amp debate yet.