post-receive hook: error encoding fixes for Python 3.

Problem:
p.communicate expects bytes in and bytes out.
When out_data is not empty, the code formatting err_msg was producing an exception.

Solution:
Formatting everything as bytes literals.