SharePoint Library Modal Pop-up for specific Content Type

Does your Excel Content type have a template attached to it? If not then the default behaviour of the New button becomes an upload window.

Don't forget that you're dealing with a document library in this case which is not the same as a list, they are similar in many ways but annoyingly different in others.

For a bit more information:

I hate linking to videos but this shows the process to create a new CT and then attach a template to it: http://www.youtube.com/watch?v=Jh-zIP_YBBw. Jump to 58 seconds in to see how to add a template to an existing Content Type.

This link covers some of the surrounding information but not how to attach a specific template to a Content Type. Worth a read for background knowledge: http://office.microsoft.com/en-gb/sharepoint-foundation-help/set-a-file-template-for-a-document-or-form-library-HA010377912.aspx