Removing triggers from Hitachi ETG-4000 fNIRS recordings

Posted on 2024-04-122024-04-12 by Eryk Walczak

Homer2 needs a particular format of a .nirs file that cannot have consecutive triggers (also called Marks in Hitachi files).
hitachi2nirs Matlab script also removes the markers but I wanted to recreate the whole process and be sure that I’m doing it correctly. Answering Yes to the question Do you want to remove the marker at the end of each stimulus? y/n will run the following code:

To remove the triggers/markings in R follow the steps below.

Start with loading the packages and files

This will produce a table showing a structure

## 'data.frame': 2500 obs. of 50 variables: ## $ Probe1 : int 1 2 3 4 5 6 7 8 9 10 ... ## $ CH1.703.6. : num 0.1865 0.0182 -0.4738 -0.1521 -0.3078 ... ## $ CH1.829.0. : num 0.412 0.547 0.534 0.314 0.106 ... ## $ CH2.703.9. : num 0.739 0.764 0.746 0.751 0.762 ... ## $ CH2.829.3. : num 1.01 1.01 1.03 1.03 1.03 ... ## $ CH3.703.9. : num 1.57 1.58 1.59 1.59 1.6 ... ## $ CH3.829.3. : num 1.64 1.65 1.65 1.66 1.67 ... ## $ CH4.703.9. : num 1.48 1.45 1.55 1.51 1.47 ... ## $ CH4.828.8. : num 1.63 1.64 1.66 1.66 1.68 ... ## $ CH5.703.6. : num -1.226 -1.743 -0.546 -0.556 -0.75 ... ## $ CH5.829.0. : num 0.00397 -0.23102 -1.11099 -0.64056 -1.01425 ... ## $ CH6.703.1. : num -0.247 -0.335 -0.371 -0.667 -1.064 ... ## $ CH6.828.8. : num 0.987 0.892 0.892 0.933 0.796 ... ## $ CH7.703.9. : num 1.03 1.3 1.11 1.02 1.44 ... ## $ CH7.829.3. : num 1.2 1.22 1.21 1.23 1.23 ... ## $ CH8.702.9. : num 2 2.01 2.03 2.04 2.04 ... ## $ CH8.829.0. : num 1.79 1.81 1.81 1.83 1.85 ... ## $ CH9.703.9. : num 2.07 2.02 2.12 2.12 2.01 ... ## $ CH9.828.8. : num 1.82 1.82 1.82 1.84 1.85 ... ## $ CH10.703.1. : num -0.492 -0.135 -0.598 -0.598 -0.328 ... ## $ CH10.828.8. : num 0.672 0.61 0.823 0.724 0.724 ... ## $ CH11.703.1. : num -1.042 -0.255 -1.773 -1.419 -0.449 ... ## $ CH11.828.8. : num 1.071 1.052 0.804 1.107 1.047 ... ## $ CH12.702.9. : num 0.684 0.771 0.704 0.512 0.905 ... ## $ CH12.829.0. : num 1.02 1.01 1.03 1.08 1.07 ... ## $ CH13.702.9. : num 2.03 2.03 2.05 2.05 2.05 ... ## $ CH13.829.0. : num 1.76 1.78 1.79 1.79 1.81 ... ## $ CH14.703.6. : num -1.719 -1.196 -0.359 -0.883 -1.99 ... ## $ CH14.829.0. : num -0.0832 0.0209 -0.1123 -0.2014 -0.2011 ... ## $ CH15.703.1. : num 1.97 1.89 1.82 1.98 2.09 ... ## $ CH15.828.8. : num 1.81 1.78 1.8 1.81 1.84 ... ## $ CH16.703.4. : num 0.0209 -0.4283 -0.0848 -0.278 0.4996 ... ## $ CH16.829.0. : num 1.36 1.26 1.38 1.23 1.27 ... ## $ CH17.702.9. : num 2.35 2.35 2.36 2.37 2.38 ... ## $ CH17.829.0. : num 2.08 2.09 2.11 2.12 2.13 ... ## $ CH18.703.6. : num 2.1 2.1 2.09 2.1 2.1 ... ## $ CH18.828.5. : num 2.14 2.14 2.14 2.15 2.15 ... ## $ CH19.703.6. : num -1.104 -1.134 -0.658 -0.886 -0.336 ... ## $ CH19.829.0. : num -0.1239 0.09369 0.05463 0.01617 -0.00427 ... ## $ CH20.703.4. : num 1.65 1.55 1.28 1.35 1.56 ... ## $ CH20.829.0. : num 1.77 1.75 1.8 1.81 1.8 ... ## $ CH21.703.4. : num 1.41 1.43 1.31 1.42 1.46 ... ## $ CH21.829.0. : num 1.76 1.77 1.76 1.77 1.79 ... ## $ CH22.703.6. : num 2.11 2.29 2.18 2.24 2.21 ... ## $ CH22.828.5. : num 2.17 2.17 2.18 2.17 2.2 ... ## $ Mark : int 0 0 0 0 0 0 0 0 0 0 ... ## $ Time : num 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 ... ## $ BodyMovement: int 0 0 0 0 0 0 0 0 0 0 ... ## $ RemovalMark : int 0 0 0 0 0 0 0 0 0 0 ... ## $ PreScan : int 1 1 1 1 1 1 1 1 1 1 ...

table of triggers

## .
## 1 2 9 10
## 73 10 4 2

and a plot showing raw data in one channel (quite noisy) with all the triggers.

This shows several triggers (all plotted using red colour). I will only keep trigger ‘2’ to mark the beginning of a block. The first step is cleaning the data by removing all but trigger ‘2’.

Which results in fewer events

It turned out that there were two ‘2’ triggers next to each other. That’s because ETG-4000 does not allow odd triggers next to each other, e.g. 212 is invalid, but 22111122 is valid. I wrote a function (soon incorporated into fnirsr package) that deals with this problem.

The result, only the first block events, is here

Geofacet Polski - wykresy w miejscu województw

Posted on 2024-03-252024-03-25 by Eryk Walczak

Niedawno odnalazłem ciekawy pakiet geofacet, który umożliwia rozmieszczenie wykresów zgodnie z ich pozycją na mapie. Główna funkcja facet_geo() zastępuje facet_wrap() z ggplot2. Polska mapa jeszcze nie jest dostępna w standardowym pakiecie geofacet, ale mam nadzieję, że już wkrótce tam się znajdzie, bo dodałem ją na GitHubie.

Stworzyłem siatkę z koordynatami poszczególnych województw. Wykresy z pakietem geofacet mogą wyglądać tak:

Rozmieszczenie województw nie jest idealne, ale pakiet geofacet umożliwia użycie własnych ustawień.

Dane pochodzą z Banku Danych Lokalnych (XLS - tablica przestawna)

Kod do stworzenia wykresów:

Downloading UK property prices from Zoopla in R

Posted on 2024-01-072024-01-07 by Eryk Walczak

Zoopla allows a limited access to its API providing the latest property prices and area indices. I created a package in R that allows querying this database. See the GitHub documentation or zooplaR’s page for the latest info.

You can easily get prices in the last couple of months or years for a particular postcode, outcode or area:

Given, the limit number of queries, it might be worth double-checking the results with the property widget offered by Zoopla (redirects to zoopla.co.uk).

It doesn’t have as many options as the API and obviously is not automatic but it’s worth using for a sanity check.

Property search

Enabling MATLAB in Jupyter notebooks on Linux

Posted on 2023-11-062023-11-16 by Eryk Walczak

Introduction

In my previous post I showed how to enable MATLAB in Jupyter notebooks on Windows. Now it’s time for GNU/Linux (Ubuntu).

My main issue with enabling new kernel was having initially installed two Anacondas and two Python versions (2.7 and 3.5). After a lot of frustration, I decided to remove both Anacondas and have a clear install of the latest Anaconda with Python 2.7 and 3.5. In this tutorial I assume that Jupyter and MATLAB are already installed on your system.

Using the right environment

Although the official MATLAB website states that Python-MATLAB engine works with Python 2.7, 3.4, 3.5 and 3.6, I struggled to install it using Python 3.5. If you try to install it with a 3.5 version, you will see the following error:

OSError: MATLAB Engine for Python supports Python version 2.7, 3.3 and 3.4, but your version of Python is 3.5

The error makes it obvious that you need an older version of Python. I decided to use 2.7. To do that, I created another environment with Python 2.7:

conda create -n py27 python=2.7 anaconda

The guidelines to managing Python environments are here.

The next step was checking what environments were available:

conda info --envs

And activating Python 2.7 (py27):

source activate py27

Install Python-MATLAB engine

To install the engine connecting both languages: go to your MATLAB folder, find the Python engine folder and install setup.py. This can be done in the following way:

Change your working directory to where your MATLAB lives:
cd "MATLABROOT/extern/engines/python"

If you don’t know where your MATLAB is installed, use:
locate matlab

Then install the engine (it will only work with MATLAB >=2014b):

sudo python setup.py install

And the latest remaining dependencies:

sudo pip install -U metakernel sudo pip install -U matlab_kernel sudo pip install -U pymatbridge

That should do the job. Now open new Jupyter notebook:

jupyter notebook

To check whether you can find MATLAB among the available engines (top right corner):

Now check whether you can actually run the notebook. Initially, when I tried using Python 3.5, I could see MATLAB among the options but the kernels would die each time I tried running the MATLAB code. Moving to Python 2.7, as described in this tutorial, solved the problem.

If all works fine then the following notebook should generate correctly:

Even though I’m getting the MetaKernelApp error, the notebook continues to work correctly:
[MetaKernelApp] ERROR | No such comm target registered: jupyter.widget.version

To leave the environment used to run the notebook, simply type:

source deactivate

Notes

Initially, I struggled a bit with making it all work so in the meantime I also tried installing Octave (a free equivalent of MATLAB). I’m not sure whether that installation helped me with running MATLAB within Jupyter.

While trying to install the engine I came across several errors. I guess that most of them were related to my OS configuration and all of them were solved by searching for the error message. One of the errors was:

Error:
[I 00:58:19.847 NotebookApp] KernelRestarter: restarting kernel (3/5)
/home/eub/anaconda3/bin/python: No module named matlab_kernel

This was due to installing the Python engine in the wrong environment (i.e. my default Python 3.5). It was solved by activating Python 2.7 and using it to install the Python-MATLAB engine.

I think that there is an alternative way to activate MATLAB in Jupyter, without Anaconda, would be to explicitly point the installer to the Python version that supports the Py-MATLAB engine.

In my case:
sudo ~/anaconda/pkgs/python-2.7.13-0/bin/python2.7 setup.py install

You might also want install the engine in a non-default location. In that case, MATLAB has a solution to that problem and suggested installing Python in the home directory.

There is another Jupyter kernel (imatlab) engine that supposedly works with Python 3.5 and MATLAB R2016b+ but I haven’t tested it myself. As long as my current configuration works, I’m not planning to go through the hell of installing dependencies again.

Using MATLAB in Jupyter notebooks on Windows

Posted on 2023-07-292023-11-07 by Eryk Walczak

After using R notebooks for a while I found it really unintuitive to use MATLAB in IDE. I read that it’s possible to use MATLAB with IPython but the instructions seemed a bit out of date. When I tried to follow them, I still could not run MATLAB with Jupyter (spin-off from IPython).

I wanted to conduct analyses of electroencephalographic (EEG) activity and the best plug-ins to do it (EEGLAB and ERPLAB) were written in MATLAB. I still wanted to use a programming notebook so I had to combine Jupyter and MATLAB.

I spent a bit of time setting it all up so I thought it might be worthwhile to share the process. Initially, I had three version of MATLAB (2011a, 2011b, and 2016b) and two versions of Python (2.7 and 3.3). This did not make my life easier of Windows 7.

Eventually, I only kept the installation of MATLAB 2016b to avoid problems with paths pointing to other versions. MATLAB’s Python engine works only with MATLAB 2014b or later so keeping the older versions could only cause problems.

Instructions

Install Anaconda (2.7)

Install MATLAB (>=2014b) - if you are a student then it’s very likely that your university bought a license. There is also a free MATLAB-like language called Octave, but I have not used with Jupyter. Apparently, it is possible to combine Octave with Jupyter. I’m going to focus exclusively on MATLAB in this post.

Install MATLAB’s Python engine - run as admin and follow the steps on the official site.

Once the engine was installed, I could move to installing metakernel, matlab_kernel, and pymatbridge. Go to Anaconda prompt (run as admin) and run
pip install metakernel

In the Anaconda prompt run pip install matlab_kernel - this will use the development version of the MATLAB kernel.

Run pip install pymatbridge to install a connector between Python and MATLAB.

… voilà!

MATLAB should now be available in the list of available languages.
Once you choose it, you can start using it in a Jupyter notebook:

Issues
Obviously, thing were not always this smooth. Initially, I ran into problems with installing MATLAB’s Python engine. The official website suggested running the following code:
cd "matlabroot\extern\engines\python" python setup.py install

Which I did but it resulted in an error:

Luckily, the error message was clear so I had to point Python to run the 64-bit version. I double-checked my versions with:
import platform platform.architecture()

Which returned 64-bit as expected:

Using a command with full path to Python solved the problem:

Summary
I hope this will be useful. I have been messing with other issues which were pretty specific to my system so I did not include them here. Hopefully, these instructions will be enough to make MATLAB work with Jupyter.

PS: I have also explained how to use MATLAB with Jupyter on Ubuntu.

How to add code coverage (codecov) to your R package?

Posted on 2023-06-252023-06-26 by Eryk Walczak

During the development of another R package I wasted a bit of time figuring out how to add code coverage to my package. I had the same problem last time so I decided to write up the procedure step-by-step.

Provided that you’ve already written an R package, the next step is to create tests. Luckily, devtools package makes setting up both testing and code coverage a breeze.

Let’s start with adding an infrastructure for tests with devtools:
library(devtools) use_testthat()

Then add a test file of your_function() to your tests folder:
use_test("your_function")

Then add the scaffolding for the code coverage (codecov)
use_coverage(pkg = ".", type = c("codecov"))

After running this code you will get a code that can be added to your README file to display a codecov badge. In my case it’s the following:
[![Coverage Status](https://img.shields.io/codecov/c/github/erzk/PostcodesioR/master.svg)](https://codecov.io/github/erzk/PostcodesioR?branch=master)

This will create a codecov.yml file that needs to be edited by adding:
comment: false language: R sudo: false cache: packages after_success: - Rscript -e 'covr::codecov()'

Now log in to codecov.io using the GitHub account. Give codecov access to the project where you want to cover the code. This should create a screen where you can see a token which needs to be copied:

Once this is completed, go back to R and run the following commands to use covr:

install.packages("covr") library(covr) codecov(token = "YOUR_TOKEN_GOES_HERE")

The last line will connect your package to codecov. If the whole process worked, you should be able to see a percentage of coverage in your badge, like this:

Click on it to see which functions are not fully covered/need more test:

I hope this will be useful and will save a lot of frustrations.

Interactive plot of fNIRS data

Posted on 2023-06-02 by Eryk Walczak

The easiest way to plot ETG-4000 data in R is by using plot_ETG4000() from fnirsr package. However, if you want to explore your data in more detail, then an interactive plot is more appropriate.

I used dygraphs package to create the chart below. In case of using many channels, the colours in the legend can get a bit mixed up like in my example. I haven’t figured out yet how to add a custom colour palette that could deal with multiple channels.

One way or another, this code snippet should be enough to start generating interactive charts. I haven’t added the interactive chart to the main plotting function (i.e. plot_ETG4000) but I might do it in future releases.

The code used to generate the chart is here:

PS: The dygraph generated correctly in the interactive window, when using R notebooks, and when knitting. When I Saved as Web Page from RStudio, I got a header error that I had to clean by removing a tag (<!DOCTYPE html>) from the generated html file.

fnirsr - Fixing bugs, Travis CI, and detrending

Posted on 2024-04-162024-04-20 by Eryk Walczak

I haven’t worked on fnirsr (my R package for analysing fNIRS data) for a while so I thought it’s time for some improvements. I read a great introduction to Travis CI and decided to make it work this time. After running R CMD check (and devtools::check()) several times to fix multiple bugs, I finally got to see that lovely green badge 🙂

The package still needs more testing, but so far it does its job. On top of that, I finally added a function that removes a linear trend from an fNIRS signal:

For more details and the latest updates see the project’s GitHub page.
CRAN, here I come!

Preview of a scientific notebook in fnirsr

Posted on 2024-02-19 by Eryk Walczak

Here’s a demonstration of using the notebook within the fnirsr package:

More information about R Notebooks can be found in the RStudio’s blog post.

Sending serial triggers from PsychoPy to ETG-4000 fNIRS

Posted on 2024-02-122024-02-12 by Eryk Walczak

In the process of designing my latest experiment in PsychoPy I realised that setting up the serial port connection is not the most obvious thing to do. I wrote this tutorial so that others (and future me) won’t have to waste time reinventing the wheel.

After creating an initial version of my experiment (looping over a wav file) I tried to figure out how to send a relevant trigger to my fNIRS. Luckily, the PsychoPy tutorial has a section about using a serial port, but after reading this I still wasn’t sure how to use the code with my script. After a quick brainstorm with the lab technician, we figured out that a simple script (see below) is indeed sending triggers to the fNIRS:

To better understand the arguments of the serial.Serial class please consult the pySerial documentation.
The argument (‘COM1’) is the name of the serial port used. The second argument is the Baud rate, it should be the same as the Baud rate used in the Parameter/External settings of the ETG-4000:

The last line of the script is sending the signal through the serial port. In this example, it is “A ” followed by Python’s string literal for a Carriage Return. That was one of the strings expected by ETG-4000, i.e. it was on the list previously set up in Parameter/Stim Measurement:

The easiest way for me to test whether I was sending correct signals was to use the Communication Test in the External tab (see the second screenshot). Once the test is started, you can run the Python script to test whether the serial signals are coming through.

If the triggers work as expected, the code sections for serial triggers can be embedded in the experiment. It can be done via GUI or code editor. That’s where to add code using GUI:

The next step is adding the triggers for the beginning, each stimulus/block, and the end of recording.

Here is an example of an experiment using serial port triggers to delimit blocks of stimuli and individual stimuli.

Due to the sampling rate, I had to add delay between the triggers delimiting the blocks, otherwise they would not be captured accurately. The triggers for block sections had to be send consecutively because the triggers cannot be interspersed (not sure if that’s because of my settings). For instance, AABBAA is fine, but ABABAB is not.

Eryk Walczak

My Blog about Data

Author: Eryk Walczak