Quote - For example, if the algorithm for checking a serial number S is to compute some hash d of S and use d to decrypt some other block of data E to a known block M, generating a serial is going to be a rather difficult problem if you don't know d.  In particular, attempting to solve the equation  Edº M (mod n) for d where E, M, and n are known (and n is extremely large) is an extremely difficult problem.

You are thinking in mathematics, but you must think in software code.
Software code is very limited to some few functions: add, substract, multiply and shift.
Division can be used in only a small amount because it is slow.
Many mathematical functions have no inverse, software code always has an inverse.
If in the forward direction you shift right, in the backward direction you shift left.

Quote - This can be taken one step further, and once the has is verified to be correct, it is then used to decrypt a block of code needed for the installation to continue; this way, disassembling the program and finding the code that checks the serial number and changing it to bypass this check won't work because you will be missing an essential piece of code.

In almost all  cases no code is encrypted (it was common in DOS). All is reduced to checking to key, if the key is ok then the software goes on, if the key is invalid the program aborts.
If you bypass the verification, any key that you enter will give ok and the program will continue.
Very few software encrypts the code and in general encryption cannot be done.
The reason why you cannot encrypt the code in Windows is because Windows allocate dynamically the program in memory and Windows must relocate all the program references to memory to the address where the program reside in memory.
What Windows does when you start a program is:
1)- Load the code in memory.
2)- Relocate all memory referrences
3)- Tranfer control to the code so it begins to run.
If the code is encrypted Windows in the relocating process will change the value of parts of the encrypted code!!!
The program only starts after the relocation!. When the program is decrypting the code and reach a part that was changed by Windows it can give an decryption error or if it pass the code will have a wrong memory referrence that will make Windows crash when the program is executed.
You can see that is very difficult and tricky to make a software that can be encrypted and limited only to small programs.
The only thing that you can encrypt without problem is data.

Stupidity also evolves!