If you are not using Visual Studio to refresh your Nuget packages then you need to check them in to source control. In my case this is the version control in TFS. To solve the problem you just need to add a .tfignore file to the packages directory.
The easiest way to do this is to create a new .txt file in the packages directory and call it ".tfignore.". Notice it starts and ends with the period. The last period will be removed automatically and you will be left with a file called ".tfignore".
You can also use notepad.exe to create the file. The trick with this is to change the type to All File and just type the name ".tfignore".
Once you have a file created open it in notepad and add the following lines to it.
The ! tell the source control to NOT ignore the files with these extension.