libcompat: Fix vasprintf() to error out if vsnprintf() returns >= INT_MAX

libcompat: Fix vasprintf() to error out if vsnprintf() returns >= INT_MAX

If the initial vsnprintf() call inside vasprintf() returns the needed
amount of bytes >= INT_MAX, that means we could overflow either when
adding one for the allocation if sizeof(size_t) == sizeof(int), or when
passing the size of that buffer to the next vsnprintf() call where even
if sizeof(size_t) > sizeof(int), the function will still need to return
the number of written bytes. Set errno appropriately and return -1.

Warned-by: coverity