The demand for easy ways to upload interactive cognitive studies online will have increased with the pandemic. The design for my recent study, a 14-day mindfulness course with an interactive cognitive experiment in my pre/post surveys, posed several issues. I thought I’d share how I did this with the hopes it could help anyone attempting something similar. This guide will detail how to build an interactive cognitive test, how to upload this online, and how integrate each method with Qualtrics in case you need to use it to manage participants in a longitudinal study. The emphasis will be on doing this for free or as low cost as possible. I’ll go through the methods using more standard approaches first, but the bulk of this article will be with a more DIY method, and I would recommend checking that out.
This guide is written for someone with no coding knowledge, as I was before I started this project. The majority of this article, then, is just links to other tutorials and resources I used to create my own experiment, some of which took a while to find - hopefully this will save you the time and provide easier entry point for coding. This page is a work in progress and will be updated as I learn more about these processes. If anything on this page is in error or could use some improvements or clarification, or you have anything to add that worked for you, please feel free to email me at m.lovell@sussex.ac.uk - I’m also happy to help out anyone who is stuck. I’d love for this to become a comprehensive account of the different options available - I’m sure the specific needs of each experiment will be slightly different.
I’m aware of two main options here: Python and JavaScript. I don’t know anything about MATLAB so I’m not going into that here, but it might be an option you’d want to look in to.
Python is incredibly easy to use, and there is a library of functions called PsychoPy that is set up to make running psychology experiments fairly easy. However, whilst PsychoPy advertise that you can put your studies online using their software, this is only true if you use the visual/graphical Builder view of their program. This severely limits the options of what you can do. For the experiment I was trying to run, it didn’t have the capabilities I needed, although I did manage it with their coding library. The reason for this is that when PsychoPy puts your experiments online, they are no longer in Python. Python can’t be interpreted by browsers - that’s what the language JavaScript is for. So, PsychoPy converts your python into JavaScript as best it can, partly using their in-house equivalent called PsychoJS, to allow it to run online. If you choose to use PsychoPy, note that it will upload your experiment to their hosting service Pavlovia, and you can run your study for the reasonable cost of 20p per participant (more on that later). The PsychoPy website has some great tutorials and you’ll need to use their reference manual if you want to code with their library. This tutorial is another fantastic place to start, and this website is great for python in general. The PsychoPy forum is also very good.
However, if you want the ability to create any experiment you like, and run it for free online, you’ll need to go with JavaScript. JavaScript is sort of like the programming language your browser uses to do the more interesting things with websites, it sits between the more classic HTML code websites are written in. The added benefit of learning this is that this functions as a more complete introduction to not just coding but web development as well. Whilst the main language you will need to actually learn here is JavaScript, you’ll also be touching on other languages: HTML, CSS, and maybe even Bash, and PHP - but luckily you won’t need to know much of these other languages beyond following simple tutorials and googling for some forum answers. Fortunately, there is a free library of JavaScript tools designed for allowing psychologists easy and effective methods of making experiments, called jsPsych. A great introduction to both JavaScript and jsPsych, can be found in the tutorials on the jsPsych website . This is where you’ll also find all the documentation you need, and some useful worked examples. w3schools.com is another great resource for tutorials and information on web development languages (like those listed above). You can get help with jsPsych on the official github forum, and the learn JavaScript subreddit . The go-to forum for asking any coding related questions is Stack Overflow.
The only thing some of the tutorials above are missing is that you need an IDE (an Integrated Development Environment - basically a word-processor for writing code into so you can run it). I use Sublime, which comes highly recommended. If you wanted to start coding in PsychoPy, you can use their program as your IDE - I am on a Mac and downloading the packages for use in my IDE causes issues - let me know if any of you manage to get it working.
Once your study is up and running on your own computer, we need to get it up online. The easiest option is using one of the following experiment hosting sites. I’ll also detail below how to integrate these with a Qualtrics survey - although it’s often less seamless than the DIY method at the bottom. However, note that jsPsych is perfectly capable of producing surveys - Qualtrics is just useful for managing participants over studies that require several surveys at different time points (amongst other features).
cognition.run is completely free and incredibly easy to use - just create a new task and upload your JavaScript to the source code section! It doesn’t accept HTML, so your CSS will need to be in a separate file. This seems too good to be true - the site is run by one person and I’m sure it can’t stay free forever, but it’s fantastic for the moment.
The cheapest of the paid experiment hosting sites is Pavlovia, run by the University of Nottingham staff that created PsychoPy off of a grant from the Welcome trust, meaning it’ll only cost you 20p per participant! They are a great organisation to be supporting and don’t seem to be a commercial enterprise, and an institutional licence is only £1500(hint hint). Here’s the information on how to integrate jsPsych with Pavlovia: pavlovia.org/js-psych - this is a bit involved for me to try out here but if anyone tries it out and runs across any issues and solutions do let me know!
All of this can be done through the online platform Gorilla. As the university aren’t subscribed to Gorilla, it will cost a fair bit of money to run participants. Note PhD students will qualify for the researcher package (‘tokens’ just means ‘participants’), but this will still push the budget for most PhD students, especially if you are already paying participants. The researcher package would be £150 for sign up, which includes 200 free participants, and amongst other options you can get 500 more participants for £350, so £500 for your first 700 participants.
If you do choose this option, note that Gorilla currently uses the previous version of JavaScript (ES5) and so if your code has relied on changes made since the current version was released (ES6), you might need to make some changes to your code to get it to work. I contacted them about this, and it should be fixed by the end of the year, apparently.
This is the tutorial to get jsPsych working in Gorilla. Don’t forget to add your HTML <script> section to the ‘head’ tab in the format suggested in the example and upload your plugins etc to the ‘resources’ section.
You might also want to check these experiment hosting sites and resources out:
https://psiturk.org/
https://languagelearninglab.gitbook.io/pushkin/
https://kennysmithed.github.io/oels2020/oels_wk11.html
https://www.spatialhearing.org/remotetesting/Resources/Web-basedPlatforms
If you’d like to integrate experiments hosted on the above sites into Qualtrics you can do so in a few ways I got from this post. Perhaps the easiest option is using an HTML iFrame. To do this, all you need is to create a question block in Qualtrics with a ‘text/graphic’ question in it. Then, click the ‘HTML view’ tab in the top right and add the following code: <iframe height=500 width=600 src=“URL LINK TO YOUR TASK”> (make sure to change the height and width to the same as your JavaScript code too). It’s fairly easy but can look a little janky and need some reconfiguring. Another option is to have the experiment open in a new tab with this code: <a href=“URL LINK TO YOUR TASK” target="_blank">Click Here</a> (i.e. the standard HTML link syntax this sentence uses), although I’ve only been able to get this to work with Gorilla - I’ve asked around for help with that and will get back with an update when I can.
Finally, you can set up two Qualtrics surveys and use a link to redirect participants from the first one to your hosted survey, and then from that hosted survey back to the second survey. Place the redirect link at the end of the first Qualtrics survey as suggested here or with the following JavaScript: Qualtrics.SurveyEngine.addOnReady(function() {setTimeout(function () {window.location.href = “URL LINK TO YOUR TASK“;},5000)}); (see here on how to add JavaScript to Qualtrics - I also deal with this below).
One important component of this redirect method is making sure you’ve kept track of who your participants are. We’ll do this by embedding their random participant ID number into the URL and keeping this both times we do a redirect. Firstly, set a random participant ID. Then embed this into the URL we send from the the first Qualtrics survey, e.g. https://github.com/jspsych/jsPsych/discussions/754.
Then, you’ll need to use jsPsych to grab that data from the URL so we can pass it back to Qualtrics. The website you’ve used to host your experiment then needs to link back to the second qualtrics survey (which really just functions as part 2 of the first one), but make sure you’ve appended the participant ID to the URL in a way Qualtrics will understand. As for how to redirect from your survey hosting site back to Qualtrics: in cognition.run, check out the source code for their ‘copy of redirect participant after finish’ example task, where they just put the following line: window.location.href = "URL LINK TO YOUR TASK” at the end of their on_finish() function. Gorilla’s redirect node can be used to do the same, and also see this page on Qualtrics integration. As the Pavlovia integration with jsPsych was a little involved, I haven’t been able to set that up just yet, although apparently the information here should be of use.
It’s also possible to just place your study into a Qualtrics survey using their JavaScript question editor. This tutorial shows you how to do that: https://kywch.github.io/jsPsych-in-Qualtrics/. In my case, where I have to keep track of participants over 14 days and contact them each morning with a new survey and audio clip, I felt this approach was necessary as I was reliant on Qualtrics to manage participants and store their sensitive data securely. Outside of using cognition.run, this is the only completely free method I am aware of.
The problem with this approach, however, is that your data won’t save to Qualtrics. Instead, we’ll have to tell the JavaScript to send the data to a folder stored in your personal server space offered by the university. Before you follow the tutorial, note that the sections on saving your data won’t work if you’re at Sussex - I detail below how to get them to work. Regarding Qualtrics and JavaScript:
1. saving your data with json and not csv is recommended
2. template literals will not work properly in Qualtrics
3. The JavaScript version used by Qualtrics is in fact relying on the JavaScript version run by your browser, and so it’s best to not use code from the latest version of JavaScript (ES6) if you can avoid it, to ensure maximum usability for users with older/not updated browsers.
Disclaimer: note that as I am new to programming, I make no particular claim to knowing what I’m talking about, or how safe and secure the following setup is. This method could pose a potential security risk as I am certainly not a web developer nor a programmer. However, this method is suggested on the official jsPsych website here: https://www.jspsych.org/overview/data/#storing-data-permanently-as-a-file, and the tutorial listed here: https://kywch.github.io/jsPsych-in-Qualtrics/save-php/, and so I would presume it is currently in use by many researchers worldwide. As long as you haven’t asked participants to enter identifying information into your cognitive task specifically, this data is anonymous and not sensitive - it can only be tied to specific participants using their random identifier stored in Qualtrics. I have tried to make some security improvements as listed below. So you don’t have to take my word for it, I’ve sourced where I got all the information from to do so.
Considering the current pandemic, I hope this guide can help others like me who suddenly find the need to run online experiments on a small budget. If anyone has any suggestions for improvements, feel free to email me at m.lovell@sussex.ac.uk and I will be happy to update the document. I also don’t intend to take credit for much of the code below, and I owe many thanks to Kyoung Whan Choe for his excellent tutorial on which the below is very heavily based - I only list my changes to his method here, and direct you to the original tutorial where possible. Without the original tutorial to build off, I wouldn’t have had the time to TRY to improve on it here - I would’ve spent the whole time trying to get the project off the ground!
Going back to the tutorials on kywch.github, the section on saving data to a web server with php is not entirely applicable to members of the university of Sussex. You can see the university’s guidance on data storage here. Before you can follow along with the kywch.github PHP tutorial, you will need to set up a personal web space with the university, and follow this guide for authorising your computer to access your personal (N) drive. As a Mac user, I noted that the STFP (essentially just software for transferring files) recommended by the university was not on the App Store, and I opted to go for the free and open-source program called ‘FileZilla’ which is widely used. Perhaps windows and Linux users would also want to use FileZilla, as that is what I’ll be referring to in this tutorial. Note when signing-in that the host is `sftp://unix.sussex.ac.uk` and the port is ’22’. Finally, you can open your command line terminal and type the command ssh [username]@unix.sussex.ac.uk and log in with your university password.
You can now follow the steps in the guide 'Saving Data with PHP’ which lead you to make the exp_data directory within public_html, although stop there as we’ll do things a little differently.
Firstly, the line suggesting $ echo "DirectoryIndex index.html" >> .htaccess (so $ touch index.html isn’t needed either) won’t work. We aren’t allowed to redefine these values, and this is the default behaviour on the university servers anyway. Secondly, the line $ chmod 772 hello-world won’t work on the university servers. Instead, the highest security we could offer is chmod 1703, you can find more about these permission codes here (note the first `1` refers to a ‘sticky bit’).
Instead of doing this, we can increase security somewhat by following the steps found here: http://www.mysql-apache-php.com/fileupload-security.htm and integrating them with the guide from the Kywch website above. That means we won’t need the ‘hello-world’ directory within exp_data - your exp_data directory only needs the save_data.php file within it, having any subdirectories within that folder could pose issues.
Two quick side notes relating to the tutorial: file_uploads are already switched on - your php.ini file is located in a hidden subfolder called ‘/etc’ (how to), and you can see we are running Apache by putting your username in the following bash code: curl -s -I https://users.sussex.ac.uk/[username]/|grep Server.
I would suggest making your save_data.php file in your IDE and moving it to the correct location (exp_data) with FileZilla, rather than using Vim. I’ve made some changes to the php file that should increase security somewhat - you can see that file here: https://github.com/Max-Lovell/online-experiments/blob/main/save_data.php. I’ll go through what these changes are and why they’ve been made, in order of appearance - skip this section if you’re not interested!
if ( $_SERVER['REQUEST_METHOD']=='GET’ these first few lines of code make it so that people can’t just see whatever error message the php file throws up when you try to access it through the standard URL. Going to something like https://users.sussex.ac.uk/[YOUR_USERNAME]/exp_data/save_data.php it will look like nothing’s there, although if you take it out you’ll see an error message that reveals something about the code that’s best left hidden.
The line header('Access-Control-Allow-Origin: *’) in save_data.php doesn’t seem to work. Comments in the php file do suggest changing it to header('Access-Control-Allow-Origin: https://ssd.az1.qualtrics.com') for tighter security (the '*' means 'allow any website'). If you go to your developer console when running the study in Qualtrics you would find that this throws an error. If you are using the Sussex Qualtrics account, you would need to change the allowed origins to https://universityofsussex.eu.qualtrics.com to make this error go away. However, this still doesn’t work - whatever we put in the allowed origins section, the files will still write to our folder, which isn’t good. Following the highest upvoted comment in the Stack Overflow question linked above, you might want to just opt to remove this code, as I assume our URL is well protected. However, we can follow the example at the bottom of this page and instead wrap our entire php code after <?php and before ?> with the following: if($_SERVER['HTTP_ORIGIN'] == 'https://universityofsussex.eu.qualtrics.com') { ...reset of code... } else { exit; }. You can test that this works by changing the allowed origin in the php file and trying out the Qualtrics program. If one wanted to implement CORS better than this, the answer on stack overflow is a good place to start, along with this and this, apparently.
The chunk of code that declares a $blacklist is straight from mysql-apache-php.com, and I’ve placed it up top to catch any executable files being uploaded immediately.
Apparently, the filter_input_array from the Kywch.github tutorial isn’t actually for stopping xss attacks, but I recon it’s better to call this function before we even touch our json file, so I’ve put it up here. It basically removes any characters and strings that might be dangerous before the computer even touches them.
The guide at mysql-apache-php.com checks if what is uploaded is an image file. This isn’t applicable to us, but we can check if a .json file has been uploaded with that json_decode() and last_error code adapted from the answers here.
The if (isset($_POST… stuff just checks if the variables we are getting from the jQuery.ajax() function we put into Qualtrics exist; if they do, they are assigned to a PHP variable - if any of them aren’t there, the program exits. You might notice, if you looked at the original tutorial, that there’s no mention of ‘data_dir’ anymore - it’s better to not send the location of where we store our files over the internet, so I’ve removed it and we’re going to be putting what was in data_dir straight into the bottom of the php file. This means that you will need to make a new (or edit this) php file for each new experiment with that new location - this should make more sense once we get to that section.
file_put_contents($data_dir.'/'.$file_name, $exp_data); has become file_put_contents('../../hello-world/hello_'.$file_name, $exp_data);. This function is the bit that grabs the data and puts it in the folder we want. Since we’re not getting that information from our Qualtrics survey, it’s been hard-coded in, hence the ‘hello-world’, which you’ll want to change with something more relevant to your own experiment (we’ll get to making this folder in a moment). The reason I’ve added ../../ is that, according to the tutorial at mysql-apache-php.com and a few others online, it’s a good idea to not have our data accessible by a public URL. Instead, we’ll make our uploads folder out of public_html (our ‘wwwroot’). If your current save_data.php file is in a subfolder of ‘public_html’ called ‘exp_data’, as the Kywch.github example suggests, then we’ll need to go up two folder levels. ../ means going up a folder level from wherever you currently are (i.e. where the save_data.php file we’re writing in is). So, to get go up two levels and get out of public_html we use ../../. The reason for hello_'.$file_name is that file_name is just going to contain the Qualtrics participant ID and nothing else - the less sent online the better. The full-stop is how you concatenate the $file_name variable onto the `hello_` string.
Now, we’re going to create the folder where our data will upload (in the example on kywch.github it’s called ‘hello-world’). We’re going to make this outside of the ‘wwwroot’ - i.e. outside our public_html folder, and so it won’t be accessible using a URL in a browser. We can move up a directory level in terminal with the command cd .. (Note the space). Do this until you are in the folder you began in after signing in with `ssh`, before moving into the public_html folder - you can use the bash command pwd to make sure you are in /its/home/[username], or ‘ls’ to make sure you can see the public_html folder listed. Here is where you want to create the directory where you want to save your data (i.e. ‘hello-world’). Assign it the permission chmod 1703.
We can also set our other permissions slightly differently. exp_data requires execute permissions only as your JavaScript links directly to that file, hence you can assign it the permission chmod 1701. Any permissions above 0 on the save_data.php file are allowed, and limiting this to read only would be the best idea. Hence, assign this file chmod 1704. Finally, change the permissions of public_html to 1701.
Next, following the example in the mysql-apache-php.com post above, in your script editor, create a file called ‘.htaccess’ with the following content:
Options -Indexes
Options -ExecCGI
AddHandler cgi-script .php .php3 .php4 .phtml .pl .py .jsp .asp .htm .shtml .sh .cgi
<Files ^(*.json)>
order deny,allow
deny from all
</Files>
Note, If you are dealing with csv files and not json, change <Files ^(*.json)> to <Files ^(*.csv)>. These are explained here and here. You can then put this .htaccess file into the uploads folder (e.g. hello-world) using FileZilla. Note that this file might be hidden - for example, if you are on a Mac, you will need to use .+⇧SHIFT+⌘CMD in finder to reveal hidden files.
Now, follow along with the rest of the tutorial at kywch.github regarding how to add JavaScript code in order to save your data. However, you can make the following changes to that technique when you are done. Firstly, we won’t be needing most of this:
var task_name = "hello-world";
var sbj_id = "${e://Field/workerId}";
var save_url = "https://users.rcc.uchicago.edu/~kywch/exp_data/save_data.php";
var data_dir = task_name;
var file_name = task_name + '_' + sbj_id;
All we need are the sbj_id and save_url variables. Let’s now make the following changes to the jQuery.ajax() function:
function save_data_json() {
jQuery.ajax({
method: 'POST',
dataType: 'json',
cache: false,
url: save_url,
data: {
file_name: sbj_id + '.json',
exp_data: jsPsych.data.get().json()
}
});
}
A few things have happened here. file_name has been changed to just have the subject id, and there’s no data_dir. The reason for this is that we’re limiting all of the information we’re sending over the internet - these things are all just hard-coded into our php file. The `type` function is depreciated and so we’ll change that to method:'POST’ instead. Finally, it’s recommended that we specify what data type we are sending through the jquery.ajax() function, and so I’ve added dataType: ‘json’ too.
Make sure you have followed all the other steps in the tutorial at kywch.github, then try running your experiment in Qualtrics and hopefully the data should save to your target upload directory (e.g. ‘hello-world’) - you will need to refresh FileZilla before you can see the file appear. Since we can’t just access the data through a public URL anymore, we need to download it in a slightly different way than is suggested in the article. As our data is now housed outside public_html, and as such is a ‘protected system folder’, we’ll need to navigate to the left-hand panel of FileZilla called ‘local site’, which lists the contents of our own computers, create a directory where we want to put our data (e.g. navigate to the ‘documents’ folder on your computer and create ‘exp_data’). Then, when we right-click -> download on the remote site on the right-hand side of the screen, the files will be downloaded to this folder, and we can find them on our own computers. If you have a large amount of data, you may want to create a zip folder as suggested in Kyoung’s example. More info here and here.
You can find my PHP and .htaccess file at my GitHub repo here.
This is sort of a note-to-self, but I can’t put much more time into this guide for now. If you’d like to help out improving this page, here’s some places I think would be good to start (with full credit given of course!)
https://security.stackexchange.com/questions/10825/is-this-jquery-ajax-call-vulnerable-to-xss
https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html
https://www.codeproject.com/Articles/1121259/How-to-make-secure-AJAX-call
https://stackoverflow.com/questions/37912937/how-to-send-secure-ajax-requests-with-php-and-jquery
https://owasp.org/www-project-top-ten/#tab=OWASP_Top_10_for_2013
https://www.phpcluster.com/5-steps-to-secure-ajax-php-call/
https://stackoverflow.com/questions/409496/prevent-direct-access-to-a-php-include-file
https://api.jquery.com/jquery.ajax/
https://www.jspsych.org/overview/data/#storing-data-permanently-as-a-file
Control for file size: http://www.mysql-apache-php.com/fileupload-security.htm
https://stackoverflow.com/questions/33999475/prevent-direct-url-access-to-php-file/33999539
https://thisinterestsme.com/prevent-direct-access-php-file/
https://www.liquidweb.com/kb/what-is-umask-and-how-to-use-it-effectively/
https://www.w3schools.com/js/js_json_php.asp
https://www.w3schools.com/php
https://webmasters.stackexchange.com/questions/13658/when-creating-a-website-what-permissions-and-directory-structure
https://serverfault.com/questions/357108/what-permissions-should-my-website-files-folders-have-on-a-linux-webserver
https://stackoverflow.com/questions/13421463/htaccess-access-control-allow-origin.
Whilst the guide at https://kywch.github.io/jsPsych-in-Qualtrics/save-dropbox/ uses dropbox, Sussex is subscribed to Box instead. As with the (outdated) guide for setting up a dropbox app above, to do this we would want to create an app in box. Sussex members can log in and create a new app here. This is a tutorial for setting up a box app which does have mostly the correct settings for our purposes. If you navigate to the bottom of your box account settings, you will see that the admin contact is sgw21@sussex.ac.uk - an address of someone who has currently left the university. Perhaps for this reason (and this issue may have been fixed by now) box apps aren’t being authorised at the moment - I have contacted the IT service desk about this and will update the article when I hear back from them. Useful resources for going down this route will be the Node.js SDK, the JavaScript SDK for box, the qualtrics JavaScript API class, and information on application scopes.
This is the option suggested by the jsPysch website here. The university may have the ability to give us a MySQL space on their server although IT services didn’t reply to my request to get one set up. Security can be tricky here too - you would need to guard against an SQL injection attack, whatever that is.