rnd_next(uchar* buf) for a storage engine sets up the current row of data. If there is data, it returns a 0. Otherwise it returns a HA_ERR_END_OF_FILE or other error.
Row data is written into the uchar* buf field. You might imagine you would return structured data for a row and the API would be like rnd_next(Fields**, field_count int). But that's not how the API works. You write the fields as bytes directly into uchar* buf.
The first X bytes are a NULL bitmap. One bit for each NULL-able column, padded out to a full byte. After that is each value for each field. For a table with two INT columns (they'd both be NULL-able), you'd write 9 bytes to uchar* buf. The first byte would be a NULL bitmap of all zeroes if both values in the row were not NULL. The next 4 bytes would be the first column's value. The last 4 bytes would be the second column's value.
Here's a diff for the BLACKHOLE engine, hardcoding a single row response: https://github.com/MariaDB/server/compare/11.4...eatonphil:mariadb:e0