In Standard Windows 2003 installation and looking on the FileMaker Server folder, I found 3 files with wrong extensions. These are:
I changed the extension to this files to dll
After this and rebooting the server machine, the connection was OK.
Then, log into the FM Server Admin.
Once connected to ther FM Server, right click on the server name.
Under the Clients tab, make sure the connections is set at 250
Under the Databases tab, make sure the maximum number of files is 125 and reserve is 800 MB
Reboot the server one more time