Standard Operating Procedures: Preparing and running an experiment with the Rule Game Server
Updated 2021-04-02 for Game Server ver. 2.006
Preparing the experiment control files
The control files
As discussed in the Rule Game server data guide, a number of files need to be prepared in the server's game data directory (/opt/tomcat/game-data) in order to control the players' experience during your experiment. At a minimum, this should include:
- The experiment plan directory (a subdirectory under /opt/tomcat/game-data/trial-lists), containing one or several trial list files. The server will pseudo-randomly assign players, in a balanced manner, to different trial lists, so each of the trial list files will be the top-level file defining the experiences of a particulalry (randomly selected) subgroup of the participants of your experiment.
- For each rule set mentioned in the trial list files, a rule set file must be created in the rules directory, /opt/tomcat/game-data/rules. If you are creating a group of rule set files to be used in just one experiment, or in some series of experiments, it would make sense to place them into a new, specially created subdirectory under the rules directory. This will make it easier to manage rule set files.
Depending on the design of your experiment, you may also need to create or modify other files:
- If your trial lists refer to pre-created initial boards, you need to create the initial board directory, which will contains the initial border files, and, possibly, an ordering control file.
- If your experiment intends to use custom shapes (as reflected in the trial list files or rule sets), you need to make sure you have the SVG files for all shapes at the appropriate locations. (See: Using custom shapes and colors in Rule Game Server 2.*)
- If your experiment intends to use custom colors (as reflected in the trial list files or rule sets), make sure you have those colors defined in the color map file
Using a GitHub repository
One should not try to edit the experiment control files directly in the server's game data directory. Instead, one should follow the procedures outlined in the Deployment SOP (in that document, you only need to look at the sections written on yellow background). In a nutshell:
- The "master copy" of the experiment control files (the entire content of the /opt/tomcat/game-data directory tree) is kept in a GitHub repository;
- You can "check out" these files to your working directory, either on your home PC or on sapir;
- You can edit the files in your working directory as desired, adding new subdirectories and files as needed;
- Once done, you can check in your work into the repository; that will result in the updated files being automatically deployed to the /opt/tomcat/game-data directory tree on sapir.
This procedure ensures that at any point in the future we will have a reliable to see, by examining the repository's history, what set of control files was deployed to the server during any particular experiment.
Checking your math
One of the most complicated parts of the syntax of rule set files is the field which specifies the bucket(s) into which a rule allows game pieces to be put. (See the >Bucket expression arithmetic Guide for details). To help you test your arithmetic, we have the Arithmetic test form, which you can use to test your expression, for various values of the built-in variables, before actually using it in a rule.
Validating your plan
Once you have prepared your trial list files, rule set files, and any other data you need, you should validate your experiment plan. Go to the Validate your plan link, and enter the name of your experiment plan. The validation tool will read the trial list files for your experiment, and various other files (rule sets etc) they refer to, and will check them for syntax errors, references to non-existing resources, etc. Please review the validator's report; if any deficiencies are reported, try to correct them, clear the server tables, and validate your plan again.
Testing your plan
One can do this with the GUI client or with the HTML play interface.
If you realize that your rules need to be corrected, and edit your trial list files or rule set files, you should clear the server tables before playing again (with another player ID). This will ensure that the server will remove the earlier version of the rule sets from its memory.
Keeping track of what was done
For any later analysis to be valid, it is important that complete information were kept about the experimental environment, which includes both the control files and the software version.
- Once you have developed and extensively tested your experiment plan, you open your experiment to real players (e.g. those coming from Amazon's Mechanical Turk). After this has been done, it would be a very good idea to never modify the experiment's trial list files and the files it refers to (rule set files etc). This will ensure that all participants have a comparable experience, and hat the collected data could be properly analyzed in the future. If you want to modify your rules, or the order in which they are presented to players, it is much better to create a new experiment plan.
- Make a record of what set of control files was deployed. If you make sure to always have the current set of files in the GitHub repository, it is, in principle, sufficient to know when the experiment was run, and you would be able to find the appropriately timed snapshot of the control files in the repository. However, you may also want to make a "snapshot" of the server's game data directory before the experiment starts, e.g. by making a zip file:
cd /opt/tomcat
zip -r ~/game-data-`date -Idate`.zip game-data
This will produce a file with a name such as game-data-2021-03-15.zip in your home directory.
- Make a record of what version of the GUI client is used, and what version of the Game Server it uses. The GUI client has a special URL to check what it's version is; for the production version, it is
http://sapir.psych.wisc.edu/rule-game/prod/?version=true, and for the development version, it is http://sapir.psych.wisc.edu/rule-game/dev/?version=true. The version reported generated by that URL may look like this:
Client Version [environment-commitHash]: dev-962254ba6e35966540d0f06a2470ec89beaa0474
Server URL: http://sapir.psych.wisc.edu:7150/w2020-dev
Server Version: 2.006
Make sure to have thus information recorded in your experiment log. (Note that the server version is not reported prior to 2.006).
Data processing
See the section on exporting data from the Game Srever's SQL tables, and the section on files saved by the Game Server in the Data Guide for guidance on what data are recorded during the experiment, and how they can be extracted from the server.
[Main page]
[Rule Game Server data guide]