visit:
www.maxim-ic.comread about multiplexing and charlieplexing ... Maxim's innovative charlieplexing method allows a designer to use fewer control lines in number or alphanumeric displays.
Maxim also has a LOT of chips dedicated to driving 7 segment displays, along with evaluation boards for those chips, so you can see it in action.
If you want to reinvent the wheel and build something rather than go prefab (experience tells me prefab will cost a lot less), you'll need to learn about microcontrollers, multiplexing, serial communication, just to desribe the tip of the iceberg

Alternately, you may still be able to find old fashioned (manual) 7 segment drivers from surplus shops, but you will still need some method of generating the data they require (usually BCD or Binary format), which means a parallel printer port at the very least, and then you're looking at a mountain of software to make it happen.
I don't understand how you come to 32 digits for a score board.
I see 3 for visitors, 3 for home, 4 for the time ... so that is 10 digits? or maybe 12 digits if you wanted hundredths of a second on the time
In summary, you either want to watch the auction sites for a good deal on a used pre-fab display (maybe it has a few burnt out leds you can replace)... or you want to visit
www.maxim-ic.com and read up on building your own.
The MAX6954 / 6955 looks like the chip you need, it contains everything to run up to 16x seven-segment numberic displays or 8x fourteen-segment alphanumberic displays, by everything, I mean, it even includes a font table and character generator - you feed it number or alphanumeric data, and it displays it, that sure beats the olden days.
http://www.maxim-ic.com/appnotes.cfm/appnote_number/1131