The larger the file size the slower the upload speed. If your file is taking a while to upload, it is most likely because of the amount of Contacts within it.
As a general approximation, a file containing any amount of contacts up to 5,000 will upload within 5 minutes. Files containing over 10,000 contacts will upload within 20 minutes. But files containing anything above 100,000 contacts may take an hour.
Which50 can handle large files, but if your contacts are approaching 500,000 or so, it can be wise to split up the files and upload them separately.
The above guidelines may be inaccurate depending on how many fields are in your file, and if there is lots of data to import.