There may be a more direct route, but short of writing a custom dialog, here's one way to do it.
Upon receiving a Next result from the dialog simply check the length of the previously entered information. It either exceeds your max. length, put up an error message stating such being the case and go back to showing the user info. dialog.
The code would basically be structured as follows:
Dlg_SdRegisterUserEx:
szTitle = "Enter User Information";
szMsg = "Please enter your Name, Company Name, and Serial Number.";
nResult = SdRegisterUserEx(szTitle, szMsg, svName, svCompany, svSerial);
if (nResult = BACK) then
goto Dlg_SdWelcome; // OR WHATEVER THE PREVIOUS DIALOG IS
else
if( StrLengthChars (svName) > 40 || StrLengthChars (svCompany) > 40 )
MessageBox("Your name and/or company exceeds the maximum length of 40 characters", SEVERE);
goto Dlg_SdRegisterUserEx:
endif;
endif;
NOTE: I didn't compiling this to check the syntax, but it should be pretty close.