Avoiding TypeError: Incorrect padding with Python's base64 encoding

base64_padding.md

Avoiding padding errors with Python's base64 encoding

>>> import base64
>>> data = '{"u": "test"}'
>>> code = base64.b64encode(data)
>>> code
'eyJ1IjogInRlc3QifQ=='

Note the trailing == to make len a multiple of 4. This decodes properly

>>> len(code)
20
>>> base64.b64decode(code)
'{"u": "test"}'
>>> base64.b64decode(code) == data
True

without the == padding (this is how many things are encoded for e.g. access tokens)

>>> base64.b64decode(code[0:18]) == data
...
TypeError: Incorrect padding 

However, you can add back the padding

>>> base64.b64decode(code + "==") == data
True

Or add an arbitrary amount of padding (it will ignore extraneous padding)

>>> base64.b64decode(code + "========") == data
True

This last property of python's base64 decoding ensures that the following code adding 3 padding = will never succumb to the TypeError and will always produce the same result.

>>> base64.b64decode(code + "===") == data
True

It's clumsy but effective method to deal with strings from different implementations of base64 encoders

Or add an arbitrary amount of padding (it will ignore extraneous padding)

Unfortunately, b32decode doesn't do the same. 😞

Thank you so much!

Thanks :)

very helpful solution

Really helpful. Thank you. I did base64.b64encode and then again base64.b64decode and this solved the issue (python 3.5.2).

I think you shouldn't add an arbitrary amount of padding characters. This might work in this specify case. However, not every base64 implementation will honour this. For example the linux based tool base64 will return an error message when to many padding characters are found:

echo "SGVsbG8gd29ybGQK==" |base64 -d
Hello world
base64: invalid input

The underlying implementation makes the assumption that a chunk has a fixed size of 24 Bits worth of information. When the encoded information does not provides enough information to fill the chunk, padding becomes necessary. Therefore you should only add as much padding character as necessary to have a completed chunk.

That is genuinely a horrible but beautiful thing. It saves so much extra and padding code!

Very thank you!

Very helpful!

Didn't feel quite right to pad more that needed so I ended up doing this

code = code.ljust((int)(math.ceil(len(code) / 4)) * 4, '=')
data = base64.b64decode(code)

A similar way to add the necessary padding:

code_with_padding = f"{code}{'=' * (len(code) % 4)}"

You saved my bacon buddy. Thank you.

A similar way to add the necessary padding:

code_with_padding = f"{code}{'=' * (len(code) % 4)}"

That is not quite right. Modulo alone isn't going to give you the number of padding characters you need. This works:

code_with_padding = f"{code}{'=' * ((4 - len(code) % 4) % 4)}"

Thank you!

I don't understand this theme

I don't know what black magic is this, but it is working. Thanks

Works like a charm

Nike trick! Thanks a lot!

https://gist.github.com/perrygeo/ee7c65bb1541ff6ac770?permalink_comment_id=3478388#gistcomment-3478388

I appreciate the modular arithmetic one-liner.

Here's what my version looks like, it might be worse for performance...

code_with_padding += '=' * ((4 - len(code_with_padding) % 4) % 4)

This is really helpful information! Nice trick!