Unable to run simple example

[jeslago]

I have installed pysmac and try to run the simple rosenbrock problem. When I do so, i get the following error:

python rosenbrock_example.py
Traceback (most recent call last):
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 284, in remote_smac_function
    config_dict = smac.next_configuration()
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 193, in next_configuration
    self.__conn, addr = self.__sock.accept()
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 206, in accept
    sock, addr = self._sock.accept()
timeout: timed out
Traceback (most recent call last):
  File "rosenbrock_example.py", line 58, in <module>
    parameters)    # the parameter dictionary
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/optimizer.py", line 303, in minimize
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/utils/smac_output_readers.py", line 151, in read_trajectory_file
IOError: [Errno 2] No such file or directory: '/var/folders/9_/mztwlxpx5bv0_ljvn3614wb80000gp/T/tmpoknTRc/out/scenario/traj-run-0.txt'

I thought that it might be a problem with the working directory, so I have changed as:

 opt = opt = pysmac.SMAC_optimizer(
                             working_directory = '/Users/jesus/pysmac/')

But I get the same exactly error:

python rosenbrock_example.py
Traceback (most recent call last):
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 284, in remote_smac_function
    config_dict = smac.next_configuration()
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/remote_smac.py", line 193, in next_configuration
    self.__conn, addr = self.__sock.accept()
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 206, in accept
    sock, addr = self._sock.accept()
timeout: timed out
Traceback (most recent call last):
  File "rosenbrock_example.py", line 59, in <module>
    parameters)    # the parameter dictionary
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/optimizer.py", line 303, in minimize
  File "build/bdist.macosx-10.12-x86_64/egg/pysmac/utils/smac_output_readers.py", line 151, in read_trajectory_file
IOError: [Errno 2] No such file or directory: '/Users/jesus/pysmac/out/scenario/traj-run-0.txt'

[sfalkner]

Thanks for filing the issue, and sorry for the delayed response.
Unfortunately, I cannot reproduce the problem here on my end.
Could you rerun it and add the 'persistent_files=True' flag to the SMAC_optimizer? If the folder (you specified via working directory) is not empty after running the script, could you send me the files in there?
Best,
Stefan

[jeslago]

I have solved the problem, but to be honest, I don't know what caused it. When the problem started I moved the project to Ubuntu where pysmac worked. Today I tried it back in the OSX and it just worked.

Sorry I can't help more, but at least the problem is not there.

Thank you in any case.

[ecemsogancioglu]

Hello,

I am having the same problem in OSX, but it works fine in Ubuntu.

What I noticed is that scenario folder which is created by pysmac is empty so that I get an error that '.../out/scenario/traj-run-0.txt' does not exist.

I already added the 'persistent_files=True' flag to the SMAC_optimizer which does not solve the problem.

Do you have any idea why it might occur?

Thanks for your helps,
Regards
Ecem

[jeslago]

Again I am unable to run the optimizer. As the other user has said, the flag 'persistent_files=True' does not solve the issue.

I am attaching the files that appear in the working directory right before the program crashes. Any idea of what might be happening?

pysmac.zip

[sfalkner]

The persistent_files=True flag isn't supposed to solve the issue, it just enables me to inspect the temporary files that are created before 'Java-SMAC' is run. The files themselves look OK, although your function is never called.
Is the error still the same as in your original post?
Could you please rerun it with the debug=True flag for the SMAC_optimizer enabled, and post the output here.
The underlying problem is the socket implementation in MacOS, which seems to be not as well behaved as the Linux implementation. (Un)fortunately, I don't have access to any OSX machine, and can't really test that. Sorry...

[craymichael]

I received the same error (unsure what caused it) with and without persistent_files=True. I received much better error messages with debug=True, which ended up being errors with my forbidden clause syntax. Possibly useful debugging traceback/log output attached below.

Traceback

[DEBUG/MainProcess] Writing output into: ./smac/
[DEBUG/MainProcess] created semlock with handle 139719695450112
[DEBUG/MainProcess] created semlock with handle 139719693623296
[DEBUG/MainProcess] created semlock with handle 139719693619200
[DEBUG/MainProcess] created semlock with handle 139719693615104
[DEBUG/MainProcess] added worker
[INFO/NoDaemonPoolWorker-1] child process calling self.run()
[DEBUG/MainProcess] SMAC classpath: /usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-io-2.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jackson-databind-2.3.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/aeatk-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/numerics4j-1.3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jackson-annotations-2.3.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-math3-3.3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-collections-3.2.1-sources.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jackson-core-2.3.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/DomainInter.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jcip-annotations.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/logback-access-1.1.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/spi-0.2.4.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/exp4j-0.3.10.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/fastrf.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jmatharray.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/logback-classic-1.1.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/fastrf-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jcip-annotations-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/opencsv-2.3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/guava-14.0.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/Jama-1.0.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/smac.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-math-2.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/smac-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jcommander.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/logback-core-1.1.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/exp4j-0.4.3.BETA-3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-collections-3.2.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/aeatk.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/slf4j-api-1.7.5.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/conf:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/patches
[DEBUG/NoDaemonPoolWorker-1] picked port 35249
[DEBUG/NoDaemonPoolWorker-1] SMAC command: java -XX:ParallelGCThreads=4 -cp /usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-io-2.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jackson-databind-2.3.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/aeatk-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/numerics4j-1.3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jackson-annotations-2.3.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-math3-3.3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-collections-3.2.1-sources.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jackson-core-2.3.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/DomainInter.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jcip-annotations.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/logback-access-1.1.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/spi-0.2.4.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/exp4j-0.3.10.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/fastrf.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jmatharray.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/logback-classic-1.1.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/fastrf-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jcip-annotations-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/opencsv-2.3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/guava-14.0.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/Jama-1.0.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/smac.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-math-2.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/smac-src.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/jcommander.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/logback-core-1.1.2.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/exp4j-0.4.3.BETA-3.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/commons-collections-3.2.1.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/aeatk.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/lib/slf4j-api-1.7.5.jar:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/conf:/usr/lib/python2.7/site-packages/pysmac/smac/smac-v2.10.03-master-778/patches ca.ubc.cs.beta.smac.executors.SMACExecutor --scenario-file ./smac/scenario.dat --tae IPC --ipc-mechanism TCP --ipc-remote-port 35249 --seed 0 --skip-features True --rf-full-tree-bootstrap False --rf-ignore-conditionality False --numPCA 7 --console-log-level INFO --runcount-limit 100 --intensification-percentage 0.5 --rf-num-trees 10 --abort-on-first-run-crash False --validation False
[DEBUG/NoDaemonPoolWorker-1] Starting SMAC in ICP mode
[DEBUG/NoDaemonPoolWorker-1] Started SMAC subprocess
[DEBUG/NoDaemonPoolWorker-1] trying to retrieve the next configuration from SMAC
[INFO ] Logging to: /home/user/PycharmProjects/MNIST/./smac/out/scenario/log-run0.txt
[INFO ] Version of SMAC is v2.10.03-master-778 (3ee628ef9bf2), running on OpenJDK 64-Bit Server VM (1.8.0_131) and Linux 4.11.3-1-ARCH (amd64) 
[INFO ] Call String: smac --scenario-file ./smac/scenario.dat --tae IPC --ipc-mechanism TCP --ipc-remote-port 35249 --seed 0 --skip-features True --rf-full-tree-bootstrap False --rf-ignore-conditionality False --numPCA 7 --console-log-level INFO --runcount-limit 100 --intensification-percentage 0.5 --rf-num-trees 10 --abort-on-first-run-crash False --validation False 
Error occurred while running SMAC
>Error Message:The following forbidden line seems to be invalid: {28 % (max_pool_n ** 2) != 0 || 28 % (max_pool_n ** 2) != 0} exp4j says:[Too many operators]. Besides the operator and functions listed on http://www.objecthunter.net/exp4j/, the following are supported: [!=, ==, >, <, >=, <=, &&, ||]
>Encountered Exception:IllegalArgumentException
>Error Log Location: /home/user/PycharmProjects/MNIST/./smac/out/scenario/log-run0.txt
[ERROR] Message: The following forbidden line seems to be invalid: {28 % (max_pool_n ** 2) != 0 || 28 % (max_pool_n ** 2) != 0} exp4j says:[Too many operators]. Besides the operator and functions listed on http://www.objecthunter.net/exp4j/, the following are supported: [!=, ==, >, <, >=, <=, &&, ||]
[INFO ] Maybe try running in DEBUG mode if you are missing information
[ERROR] Exception:java.lang.IllegalArgumentException
[ERROR] StackTrace:java.lang.IllegalArgumentException: The following forbidden line seems to be invalid: {28 % (max_pool_n ** 2) != 0 || 28 % (max_pool_n ** 2) != 0} exp4j says:[Too many operators]. Besides the operator and functions listed on http://www.objecthunter.net/exp4j/, the following are supported: [!=, ==, >, <, >=, <=, &&, ||]
        at ca.ubc.cs.beta.aeatk.parameterconfigurationspace.ParameterConfiguration.isForbiddenParameterConfiguration(ParameterConfiguration.java:1248)
        at ca.ubc.cs.beta.aeatk.parameterconfigurationspace.ParameterConfigurationSpace.<init>(ParameterConfigurationSpace.java:498)
        at ca.ubc.cs.beta.aeatk.parameterconfigurationspace.ParameterConfigurationSpace.<init>(ParameterConfigurationSpace.java:386)
        at ca.ubc.cs.beta.aeatk.parameterconfigurationspace.ParameterConfigurationSpace.<init>(ParameterConfigurationSpace.java:373)
        at ca.ubc.cs.beta.aeatk.parameterconfigurationspace.ParamFileHelper.getParamFileParser(ParamFileHelper.java:43)
        at ca.ubc.cs.beta.aeatk.parameterconfigurationspace.ParamFileHelper.getParamFileParser(ParamFileHelper.java:31)
        at ca.ubc.cs.beta.aeatk.parameterconfigurationspace.ParamConfigurationSpaceOptions.getParamConfigurationSpace(ParamConfigurationSpaceOptions.java:184)
        at ca.ubc.cs.beta.aeatk.algorithmexecutionconfiguration.AlgorithmExecutionOptions.getAlgorithmExecutionConfig(AlgorithmExecutionOptions.java:159)
        at ca.ubc.cs.beta.aeatk.options.scenario.ScenarioOptions.getAlgorithmExecutionConfig(ScenarioOptions.java:129)
        at ca.ubc.cs.beta.aeatk.smac.SMACOptions.getAlgorithmExecutionConfig(SMACOptions.java:278)
        at ca.ubc.cs.beta.smac.executors.SMACExecutor.oldMain(SMACExecutor.java:133)
        at ca.ubc.cs.beta.smac.executors.SMACExecutor.main(SMACExecutor.java:104)

[INFO ] Exiting SMAC with failure. Log: /home/user/PycharmProjects/MNIST/./smac/out/scenario/log-run0.txt
[INFO ] For a list of available commands use:  --help
[INFO ] The Quickstart guide at available at: http://www.cs.ubc.ca/labs/beta/Projects/SMAC/ or alternatively (doc/quickstart.html) gives simple examples for getting up and running.
[INFO ] The FAQ (doc/faq.pdf) contains commonly asked questions regarding troubleshooting, and usage.
[INFO ] The Manual (doc/manual.pdf) contains detailed information on file format semantics.
[INFO ] If you are stuck, please ask a question in the SMAC forum: https://groups.google.com/forum/#!forum/smac-forum
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pysmac/remote_smac.py", line 284, in remote_smac_function
    config_dict = smac.next_configuration()
  File "/usr/lib/python2.7/site-packages/pysmac/remote_smac.py", line 193, in next_configuration
    self.__conn, addr = self.__sock.accept()
  File "/usr/lib/python2.7/socket.py", line 206, in accept
    sock, addr = self._sock.accept()
timeout: timed out
[DEBUG/NoDaemonPoolWorker-1] SMAC terminated with returncode 255
[DEBUG/MainProcess] closing pool
[DEBUG/MainProcess] joining pool
[DEBUG/MainProcess] worker handler exiting
[DEBUG/MainProcess] task handler got sentinel
[DEBUG/MainProcess] task handler sending sentinel to result handler
[DEBUG/MainProcess] task handler sending sentinel to workers
[DEBUG/MainProcess] result handler got sentinel
[DEBUG/MainProcess] task handler exiting
[DEBUG/NoDaemonPoolWorker-1] worker got sentinel -- exiting
[DEBUG/MainProcess] ensuring that outqueue is not full
[DEBUG/NoDaemonPoolWorker-1] worker exiting after 1 tasks
[INFO/NoDaemonPoolWorker-1] process shutting down
[DEBUG/MainProcess] result handler exiting: len(cache)=0, thread._state=0
[DEBUG/NoDaemonPoolWorker-1] running all "atexit" finalizers with priority >= 0
[DEBUG/NoDaemonPoolWorker-1] running the remaining "atexit" finalizers
[INFO/NoDaemonPoolWorker-1] process exiting with exitcode 0
Traceback (most recent call last):
  File "mnist.py", line 247, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "mnist.py", line 235, in main
    conditional_clauses=conditionals, forbidden_clauses=forbidden_confs)
  File "/usr/lib/python2.7/site-packages/pysmac/optimizer.py", line 303, in minimize
    run_incumbents.append(read_trajectory_file(fn)[-1])
  File "/usr/lib/python2.7/site-packages/pysmac/utils/smac_output_readers.py", line 151, in read_trajectory_file
    with open(fn,'r') as fh:
IOError: [Errno 2] No such file or directory: './smac/out/scenario/traj-run-0.txt'
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_terminate_pool, args=(<Queue.Queue instance at 0x7f12d97ae050>, <multiprocessing.queues.SimpleQueue object at 0x7f12eb805310>, <multiprocessing.queues.SimpleQueue object at 0x7f12d9bbfd90>, [<NoDaemonProcess(NoDaemonPoolWorker-1, stopped)>], <Thread(Thread-1, stopped daemon 139719350646528)>, <Thread(Thread-2, stopped daemon 139719392593664)>, <Thread(Thread-3, stopped daemon 139719434540800)>, {}), exitprority=15>
[SUBDEBUG/MainProcess] finalizer calling <bound method type._terminate_pool of <class 'pysmac.utils.multiprocessing_wrapper.MyPool'>> with args (<Queue.Queue instance at 0x7f12d97ae050>, <multiprocessing.queues.SimpleQueue object at 0x7f12eb805310>, <multiprocessing.queues.SimpleQueue object at 0x7f12d9bbfd90>, [<NoDaemonProcess(NoDaemonPoolWorker-1, stopped)>], <Thread(Thread-1, stopped daemon 139719350646528)>, <Thread(Thread-2, stopped daemon 139719392593664)>, <Thread(Thread-3, stopped daemon 139719434540800)>, {}) and kwargs {}
[DEBUG/MainProcess] finalizing pool
[DEBUG/MainProcess] helping task handler/workers to finish
[DEBUG/MainProcess] removing tasks from inqueue until task handler finished
[DEBUG/MainProcess] joining worker handler
[DEBUG/MainProcess] terminating workers
[DEBUG/MainProcess] joining task handler
[DEBUG/MainProcess] joining result handler
[DEBUG/MainProcess] joining pool workers
[DEBUG/MainProcess] running the remaining "atexit" finalizers
</details>