MatRPC is a small application written in C. It uses POSIX threads and Matlab Engine libraries. It can be launched as a daemon (background) process and left continually running so that it can listen to incoming Matlab commands. MatRPC acts as a simple message passer to Matlab engines, and handles the creation, destruction, and housekeeping of Matlab engines. Matlab engines essentially act as individual Matlab workspaces that operate in the background. Each Ensemble QPI session that uses MatRPC is assigned its own engine and is linked to it by its unique session ID. At the end of the QPI session, the MatRPC engine corresponding to that session is normally destroyed by the end_session form handler. MatRPC only handles returning of strings from function calls. If you want to return a number, which is often the case (e.g. stimulus_id), simply convert the number to a string in your Matlab function before returning it. Since the QPI is written in PHP, string variables are transparently converted to numbers in the PHP code. MatRPC has primarily been used on Linux and is stable on this platform. We have successfully compiled and run it on OS X, but do not know how stable it is on this platform. You will need Matlab on the computer where you wish to install it. Follow these steps to compile it. ----------------------------------------------------------------------------------- MATLAB ENVIRONMENT ----------------------------------------------------------------------------------- Before you run MatRPC, you will very likely want to configure your Matlab environment so that MatRPC can communicate with the Ensemble database. See "Matlab_README.txt" for instructions on setting up your Matlab environment ----------------------------------------------------------------------------------- INSTRUCTIONS FOR LINUX AND OS X: ----------------------------------------------------------------------------------- 1) Locate the engopts.sh script in your Matlab installation. This is likely in the bin directory of Matlab (e.g. /usr/local/matlab/bin/engopts.sh in Linux or /Applications/MATLAB/bin/engopts.sh in OS X). Open the included Makefile and change the path of engopts.sh for your installation of Matlab, if necessary. 2) Locate engine.h in your matlab installation, which is probably in the extern/include subdirectory of your matlab installation (e.g. /usr/local/matlab/extern/include/engine.h in Linux, or /Applications/MATLAB/extern/include/engine.h in OS X). Find the line in matrpc.c that points to this header file and change it to the appropriate path. 3) Type 'make matrpc' at the command line. 4) Move the matrpc binary to an appopriate location (e.g. /usr/local/bin). 5) It is highly recommended that you create a new user to run the matrpc process. We called this user "matrpcuser" and gave it only the permissions that it needed to successfully run matrpc on the server. Note that the user will need the usual matlab paths set up in ~/matlab/startup.m for matlab scripts. 6) You will need to set the LD_LIBRARY_PATH environment variable to the location of libeng.so before you can run matrpc. You can set this in the startup script for matrpcuser: $!/bin/bash #sample .bashrc file LD_LIBRARY_PATH=/usr/local/matlab/bin/glnxa64 export LD_LIBRARY_PATH OS X uses a different extension (.dylib instead of .so) and different environment variable (DYLD_LIBRARY_PATH), but otherwise this step is the same. The following should work in either your .bashrc or .profile: #!/bin/bash #sample .bashrc or .profile DYLD_LIBRARY_PATH=/Applications/MATLAB/bin/maci export DYLD_LIBRARY_PATH 7) Set up matrpcuser's environment. If you have not configured your Matlab environment for Ensemble (see Matlab_README.txt in the ensemble root directory) do that now. Make sure to add the "database" directory and MySQL Database connector to matrpcuser's Matlab path (in startup.m). You should also grant matrpcuser access to the "private" directory (explained in the installation README file for the web interfaces). It is a good idea to log in as matrpcuser and test any scripts that you will need to run with matrpc. ----------------------------------------------------------------------------------- STARTING MATRPC ----------------------------------------------------------------------------------- When you call MatRPC, you can specify three arguments: matrpc The port number specifies which TCP/IP port to listen on. This must be the same port as the one you specified in the QEI/QPI installation script. The verbose level can be between 0-3 and specifies the amount of information that matrpc sends to the standard output. This is useful for setting up MatRPC for logging usage. Verbose level 0 indicates no logging, while 3 is the highest level of logging. The number of listeners indicates the number of ports that MatRPC will listen on. Usually one port is sufficient. Assuming you have configured the QPI to communicate with MatRPC over port 60007, you issue the following command to run MatRPC as a daemon (background server process): > nohup matrpc 60007 0 1 & Alternatively, if you wish to log MatRPC processing, you may issue (assuming you have created a log file in /var/log/matrpc.log): >nohup matrpc 60007 1 1 >> /var/log/matrpc.log & The 'nohup' tells linux not to hang up the process when you exit your login shell. You will also want to delete or archive this log from time to time (perhaps only once every few months), or create an automated process to do so. ----------------------------------------------------------------------------------- SHELL SCRIPTS ----------------------------------------------------------------------------------- Several shell scripts are included with this distribution for automating matrpc: bashrc - This is an example .bashrc file that sets the path for LD_LIBRARY_PATH You may either replace your matrpcuser's .bashrc file with this one, or copy the relevant lines for setting LD_LIBRARY_PATH. Edit the contents to point to the location of libeng.so for your Matlab installation run_matrpc.sh - This shell script sources matrpcuser's .bashrc startup script to set the LD_LIBRARY_PATH, and launches matrpc. Note that this will launch matrpc as the currently logged in user. The primary function of this script is to be called by the matrpcd startup script. Copy this script to an appropriate location (e.g. /usr/local/bin) and edit the variables matrpc_exec (path to matrpc binary), matrpc_logfile1 (path to the matrpc logfile), and matrpc_port1 (the port number matrpc should use). matrpcd - This is a startup script that should go in /etc/init.d on your linux installation. Since startup scripts are launched as root, this script launches run_matrpc.sh as matrpcuser through "sudo". Make sure to set the correct username for matrpcuser ("runas" variable) and path to matrpc (run_matrpc_path). revive_matrpc.sh - This script can be called by matrpcuser periodically through a cron job. Copy this script to an appropriate location, such as /usr/local/bin. Set the matrpc_exec path, port number, and matrpc log path. You may then set up matrpcuser's crontab to call it every 1 or 2 minutes. ----------------------------------------------------------------------------------- STOPPING MATRPC ----------------------------------------------------------------------------------- To stop matrpc, find out its process ID first. This can be done by using the "ps" utility (e.g. >> ps aux | grep matrpc). Then issue a kill -15 signal. For example, if matrpc has a process ID of 3655, issue: kill -15 3655 If MatRPC doesn't stop within a minute or so, issue a kill -9 (kill immediately): kill -9 3655 After killing matrpc (especially if issuing kill -9), the sockets that it was bound to may not be immediately available. You may need to wait up to 5 minutes for them to be freed up before launching matrpc again. Matrpc will issue a message indicating that there was an error binding to the socket if this is the case. ----------------------------------------------------------------------------------- TESTING ----------------------------------------------------------------------------------- In order to test whether matrpc is working, a simple script is located at /admin/test_matrpc.php. You will be asked the experimenter account and password for this page if you haven't already entered them, as with all admin pages. The script will send a simple matlab command to matrpc: num2str(zeros(1,5)). The output of the script should look like the following: Sending command "num2str(zeros(1,5))" to matrpc Received "0 0 0 0 0" If you receive an error message when running this script, there may be a communication problem with matrpc. ----------------------------------------------------------------------------------- USING MATRPC IN YOUR EXPERIMENTS ----------------------------------------------------------------------------------- Stimulus Presentation Matlab stimulus selection scripts used with MatRPC normally return successive stimulus IDs as strings. In order to use MatRPC for stimulus selection, associate an appropriate form handler with the form you wish to use to present stimuli (e.g. form_stimulus_matlab_s.php). This form handler will send the Matlab function call to MatRPC. The Matlab function call is specified in the stimulus_matlab field in the experiment_x_form table. One usually enters the Matlab call by using phpMyAdmin. More information on configuring this can be found on the Ensemble wiki (http://atonal.ucdavis.edu/ensemble/wiki). Conditional Form Handling Conditional forms are forms that only display if certain conditions are met. The conditions can either be set by the condition editor (accessed in the Experiment Editor) or by a MatRPC call. The Matlab function call is specified in the condition_matlab field in the experiment_x_form table. Normally, one uses the form_conditional.php or form_conditional_s.php form handler for either of these methods. Note that if conditions are set using both methods (condition editor and through condition_matlab field), then both conditions will be checked. If either condition returns false, the form will not display. The conditional form handlers expect a single string ("TRUE" or "FALSE") to be returned from the Matlab function. A return value of "TRUE" will allow the form to display. See the Ensemble wiki (http://atonal.ucdavis.edu/ensemble/wiki) for more information on configuring conditional forms. ----------------------------------------------------------------------------------- TROUBLESHOOTING ----------------------------------------------------------------------------------- Here are some troubleshooting steps for MatRPC: 1) MatRPC is capable of passing several error messages to the QPI. Here is a list of possible error messages sent from MatRPC (shown in the QPI and in the MatRPC log), and possible solutions: Error code -1: Matlab script returned empty string Make sure that your Matlab function returns a single string (usually a stimulus ID number in string format). Note that MatRPC only supports sending or receiving strings. You may need to log in as matrpcuser and test the script. It is possible that your script returns stim IDs after multiple successful calls and then fails. You can test this easily by creating a script to call your stimulus selection function iteratively until all stimulus IDs have been exhausted. Error code -2: Matlab script returned non-string Usually, this error results from a Matlab function that returns a number rather than a number converted to a string. This error also appears if returning an empty vector []. Error code -3: 'outstr' not assigned. Matlab script probably resulted in error Your Matlab script probably errored out. The result from Matlab's "lasterror" should appear in the QPI and the MatRPC log to help you troubleshoot. You can debug by logging in as MatRPC and running the script Error code -4: matOut was NULL. There may have been a problem copying the result from the Matlab Engine This error should not normally occur. This means that MatRPC's matOut variable was NULL. This could have resulted from a memory problem. If your Matlab script returned an empty string, you should receive Error code -1, and if the Matlab script returns the empty vector ([]), you should receive Error code -2, since MatRPC sees this as not-a-string. Error code -5: No empty matrpc sessions available MatRPC ran out of sessions. Check the MAX_SESSIONS setting in the matrpc code (matrpc.c). If this is not high enough, set it to a higher number and recompile matrpc. If it is high enough, your Matlab engines may not be terminating properly. Make sure that SESSION_TIMEOUT_SECONDS is set properly in the matrpc.c code. Also, make sure that your Ensemble experiments are sending the terminate_matlab_session control message in the end_session form handler. Error code -6: Received socket message was too long The QPI attempted to send a message to MatRPC that exceeded the buffer length. See the 'bufLength' variable in matrpc.c. The buffer length should generally be long enough for most normal Matlab calls, and it is possible that you have a poorly constructed matrpc message string Check the stimulus_matlab field for the corresponding form in the experiment_x_form table. Error code -7: Missing '|' character separater for session id This error message should not normally occur since the message passing functions for MatRPC automatically format the messages. MatRPC normally expects a string that begins with a session ID, a pipe character (|) and the matlab function call. The function matrpc_message in functions.php formats the MatRPC call for you. 2) The connection between the QPI and MatRPC may have failed. Each connection attempt is made in "bursts." After detecting a connection failure, the QPI pauses before attempting to retry the connection. This continues until the connection attempts time out. The following error messages will appear in the event that MatRPC is either not running, or a firewall is preventing the QPI from communicating with it: Timeout occurred while attempting to connect to MatRPC socket. Socket Connection Error Stack (in order of most recent to least recent): Error code: 111, Error string: Connection refused Error code: 115, Error string: Operation now in progress Error code: 111, Error string: Connection refused Error code: 115, Error string: Operation now in progress Error code: 111, Error string: Connection refused Error code: 115, Error string: Operation now in progress These errors appear most commonly due to a firewall configuration issue, an incorrect IP address, or incorrect port number. You can troubleshoot by temporarly turning off the firewalls on the MatRPC server and the web server (if they are different computers). Also double-check the IP address and port number for MatRPC in /variables_config.php. Each error line above corresponds to a single connection attempt. You may change the following configuration constants in variables.php in order to increase the fault tolerance of your connetions: // Number of seconds before quitting socket connection attempts ENS_SOCKET_CONNECT_TIMEOUT //number of milliseconds to wait in between socket connection attempts ENS_SOCKET_WAIT_BETWEEN_REQUESTS_MS //number of milliseconds to wait after a socket connection failure ENS_SOCKET_WAIT_AFTER_FAIL_MS 3) If you didn't launch matrpc in its highest verbose level (3), stop matrpc and relaunch it with verbose level 3. Then check the matrpc log (e.g. /var/log/matrpc.log referenced above) after each communication attempt with matrpc. If MatRPC is receiving communications, there will be log entries indicating: sending command () to Matlab. This should be followed by: trying to return string: 2) Try the clientTest program. This is a simple program in the matrpc directory that sends commands to matrpc over a socket connection. To compile it, simply type "make clientTest". To run clientTest, type: ./clientTest e.g. ./clientTest 127.0.0.1 60000 where is the ip address where MatRPC is running. Hostnames will not work with clientTest, since it doesn't perform DNS lookup. If matrpc is running on the same computer from which you are running clientTest, an obvious choice is 127.0.0.1. At clientTest's cmd> prompt, type a simple matlab command that returns a string. Remember, matrpc, only returns strings! e.g., cmd> num2str(2) clientTest should respond with: send-->1121|num2str(2)<-- Sent Y bytes to matrpc Received Z bytes from matrpc received-->2<--- 1121 is an arbitrary session ID number that clientTest is sending to matrpc. Y and Z are the number of bytes sent and received from MatRPC. 3) If clientTest worked but test_matrpc.php did not work: The problem is likely in your Ensemble configuration. Check the variables_config.php file at /include/variables_config.php. Make sure that the $MATRPC_HOST and $MATRPC_PORT variables are set correctly. Try 'localhost' if matrpc is on the same server as your web server for ensemble. If 'localhost' or the hostname of the server do not work, try the IP address as a string (e.g. '129.1.40.1'). $MATRPC_PORT should be set to the same port number that you used when launching matrpc. 4) If test_matrpc.php worked, but your matlab stimulus selection script does not work: Log in as matrpcuser, launch matlab, and run the script. If it does not work correctly on the command line, there may be a problem with your matlab environment (see Matlab_README.txt) or there may simply be a bug in your script.