The thing with basic msi projects and dialogs is that windows installer is not able to act on feature (state/action) changes as long as this dialog is active.
Only when clicking next on the feature dialog you will be able to test the feature selections. This is how I do this within my project: when the user clicks next and I detect that he hasn't detected any features, I show a messagebox and go back to the feature dialog.
There is no other way I think