This occurs when I have made mistaken commits and then tried to push in a latter commit when those large files no more exist;
in this case merely clean up the cache doesn't work and requires more labor.
I found a better approach at this post from StackOverflow
So the remedy is
git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch <file/dir>' HEAD
And you should be good to go.