You don't mention what hardware you're using, or if its ethernet or Wifi.
If you running off of a SD card, then that is more than likely the issue, especially if it's not a class 10. Also if it's an older RPi (Model A/B) then you've only got a small CPU.
Also if it's an RPi, then you're limited to 10/100 ethernet.
As always - the more information you provide, the easier your resolution.