In Standard Windows 2003 installation and looking on the FileMaker Server folder, I found 3 files with wrong extensions. These are:
server.pem
root.pem
FMStrs.dls
I changed the extension to this files to dll
server.dll
root.dll
FMStrs.dll
After this and rebooting the server machine, the connection was OK.
Then, log into the FM Server Admin.
Once connected to ther FM Server, right click on the server name.
Click Properties
Under the Clients tab, make sure the connections is set at 250
Under the Databases tab, make sure the maximum number of files is 125 and reserve is 800 MB
Reboot the server one more time