find all non UTF-8 encoded files

gistfile1.sh

find . -type f | xargs -I {} bash -c "iconv -f utf-8 -t utf-16 {} &>/dev/null || echo {}" > utf8_fail

I copied this from http://blog.endpoint.com/2010/04/tip-find-all-non-utf8-files.html

Thanks a lot!

find . -type f -exec file --mime {} \; | grep -v '.git' | grep -v "charset=utf-8"

I only wanted to check files that were actually tracked in my git repo, so I used the following modified form of the original (only change is the very first command):

git ls-files . | xargs -I {} bash -c "iconv -f utf-8 -t utf-16 {} &>/dev/null || echo {}" > utf8_fail

To exclude multiple encodings

find . -type f -exec file --mime {} \;  | grep -v 'utf-8\|binary\|ascii'

This is great. Thanks!

you can also instruct find to skip the content of .git directories:
find -type d -iname .git -prune -o -type f -exec file --mime {} \; | grep "text/" | grep -v "utf-8\|us-ascii"
./xyz/q: text/html; charset=iso-8859-1
./cdr/index.html.iso8859: text/html; charset=iso-8859-1

with grep "text/" we can filter empty files which are displayed as
inode/x-empty; charset=binary

Using file command, I think you can find a way to do it in a simpler way, yet I don't know how.

Try this to understand what I mean:

file --mime-encoding *

Thanks a lot!