1) Return from a custom action with the value ERROR_INSTALL_FAILURE
2) Use a type 19 Custom Action to generate error
3) Use process message to send error message eg.
hError=MsiCreateRecord(2);
MsiRecordSetString(hError, 0, NULL);
MsiRecordSetInteger(hError, 1, MyErrorCode);
MsiProcessMessage(hMsi, INSTALLMESSAGE_FATALEXIT, hError);
return ERROR_INSTALL_FAILURE;
MyErrorCode is an index into the error table. hMsi is the handle of the installer object.
4) Validate ealy and use the install conditions.
Note a possibly better way to record log messages is to use MsiProcessMessage to record a log message in the msi log eg.
MSIHANDLE hInfo;
hInfo = MsiCreateRecord(1);
MsiRecordSetString(hInfo, 0, "Something Interesting");
MsiProcessMessage(hMsi, INSTALLMESSAGE_INFO, hInfo);
MsiCloseHandle(hInfo);
Hope this helps