If you ever want to debug a kinesis stream, copy this bash one liner.

gistfile1.txt

On a mac, `brew install awscli gnu-sed` before.

streamname=staging;aws kinesis describe-stream --stream-name $streamname --output text | grep SHARDS | awk '{print $2}' | while read shard; do aws kinesis get-shard-iterator --stream-name $streamname --shard-id $shard --shard-iterator-type LATEST --output text | while read iterator; do while output=`aws kinesis get-records --shard-iterator $iterator --output text`; do iterator=`echo "$output" | head -n1`; echo "$output" | gsed 1d | grep RECORDS | while read record; do echo $record | awk '{print $2}' | base64 -D; done; done; done; done

This fail on aws-cli/1.10.38 at least for me, working onliner:

streamname=staging; aws kinesis describe-stream --stream-name $streamname --output text | grep SHARDS | awk '{print $2}' | while read shard; do aws kinesis get-shard-iterator --stream-name $streamname --shard-id $shard --shard-iterator-type LATEST --output text | while read iterator; do while output=`aws kinesis get-records --shard-iterator $iterator --output text`; do iterator=`echo "$output" | head -n1 | awk '{print $2}'`; echo "$output" | gsed 1d | grep RECORDS | while read record; do echo $record | awk '{print $3}' | base64 -D; done; done; done; done

Thanks for this. Very useful!

I made one small amendment, by swapping out gsed 1d for awk 'NR > 1'. Therefore, you only need to brew install awscli

I had to use base64 -id to get it to work

Thanks guys, I used a combination of all your answers to get mine (ubuntu 18.04 | aws-cli/1.14.44 Python/3.6.5 Linux/4.15.0-36-generic botocore/1.8.48):

1. use vre's one-liner
2. switched out gsed 1d for awk 'NR > 1'
3. switched out base64 -D for base64 -id
   Result:

streamname=my-dev-stream-name;aws kinesis describe-stream --stream-name $streamname --output text | grep SHARDS | awk '{print $2}' | while read shard; do aws kinesis get-shard-iterator --stream-name $streamname --shard-id $shard --shard-iterator-type LATEST --output text | while read iterator; do while output=`aws kinesis get-records --shard-iterator $iterator --output text`; do iterator=`echo "$output" | head -n1 | awk '{print $2}'`; echo "$output" | awk 'NR > 1' | grep RECORDS | while read record; do echo $record | awk '{print $3}' | base64 -id; done; done; done; done

We got a Unable to open 'd': No such file or directory error, on Mac, and it looks like the base64 command works if you just use base64 -d. Great script!