I'd suggest starting with a thin client and some kind of remote access solution.
It won't cost any less to set up, but administration will be easier.
Since I'm assuming that Linux is off the table for you, you're left with Windows Server. Start with Terminal Server. Set the RDP client to use Firefox or IE as a shelll. If you want a user to be able to launch some other application, install an extension or toolbar for that purpose. You can do whatever you want to restrict internet access, or not, but since it's controlled on the server, that's pretty straightforward. Lock everything else down with group policies.
Build your kiosk application as a web-based something-or-other.