I started hearing odd noises coming from the engine bay whenever I swept past 4000 rpm's, sort of like a howling/whistling noise.

I checked under the hood to find anything peculiar and the best I could come across was the alternator belt being cracked to the extent that its cut between a couple of the threads. Is this a really big problem, so much that I shouldn't drive at all or is it barable enough until I can get it repaired say tomorrow at Gearbox? I'm worried that the belt is going to rip off

It's normal to see some small cracks along the belt grooves. The service schedule doesn't call for belt replacement until 105K, although if sufficient cracking or glazing is observed they can certainly be changed before then.
If your belt looks shiny like it has been slipping by all means replace it now. You can tell if it's loose by pressing down on the belt midway between pullies, it should deflect just a small amount, maybe 1/8" or so.
I doubt this is your noise although I can't say for sure.

Alright thanks for help