Access Control on listening ports

[requeth@gmail.com]

I have a few servers that have ports open and listening for data, but
I've found some security vulnerabilities with leaving them open. The
ports are needed to talk between several known servers. Does anyone
know a way to make it so only the servers that need to talk to each
other can access the open port, any other server/machine that tries to
hit the port gets server refused or something of that nature?

I could have it done with a lot of firewall rules on my routers, but I
really want this taken care of on the servers rather then through the
routers.

Ideas oh smart admins?