Can I use .spike file?

[Neuro-Lab-Code]

I want to cluster my data and I have both .ncs and .spike. Can I use either of them as an input for kluastaviewas or KlustaKwik? Or do I need to change the format. I do not need to do the spikedetekting.

Thanks

[emilmelnikov]

You can edit the convert.sh script or just run ncs2dat.py with the following all your NCS files as input parameters and the output DAT file as the last parameter.
Our scripts don't work with .spike files, we did spike detection by ourselves.

[Neuro-Lab-Code]

Hi Emil,

Thanks for your reply. I am sorry but I am a newbie with both Python and
changing file format.
I have made a script. Now I am not sure how I should call the program. I
have 24 bestrode and each has 4 electrodes and since I am using Neuralynx
it gives me 512 channels.

Now what exactly I should give as input and output to the program? If say I
have files named by TT1E1.ncs, TT1E2.ncs,...TT1E4.ncs, TT2E1.ncs,...
TT24E4.ncs.

Sorry again if my question is very basic.

Thanks

On Thu, Mar 10, 2016 at 5:49 AM, Emil Melnikov notifications@github.com
wrote:

You can edit the convert.sh script or just run ncs2dat.py with the
following all your NCS files as input parameters and the output DAT file as
the last parameter.
Our scripts don't work with .spike files, we did spike detection by
ourselves.

—
Reply to this email directly or view it on GitHub
#1 (comment).

[Neuro-Lab-Code]

By script I meant ncs2dat.py

On Thu, Mar 10, 2016 at 11:37 AM, Mina Shahi mi.shahi@gmail.com wrote:

Hi Emil,

Thanks for your reply. I am sorry but I am a newbie with both Python and
changing file format.
I have made a script. Now I am not sure how I should call the program. I
have 24 bestrode and each has 4 electrodes and since I am using Neuralynx
it gives me 512 channels.

Now what exactly I should give as input and output to the program? If say
I have files named by TT1E1.ncs, TT1E2.ncs,...TT1E4.ncs, TT2E1.ncs,...
TT24E4.ncs.

Sorry again if my question is very basic.

Thanks

On Thu, Mar 10, 2016 at 5:49 AM, Emil Melnikov notifications@github.com
wrote:

You can edit the convert.sh script or just run ncs2dat.py with the
following all your NCS files as input parameters and the output DAT file as
the last parameter.
Our scripts don't work with .spike files, we did spike detection by
ourselves.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[emilmelnikov]

You can execute the script in the following way (substitute names of all your files instead of ellipsis):
python ncs2dat.py TT1E1.ncs TT1E2.ncs TT1E3.ncs TT1E4.ncs TT2E1.ncs ... TT24E4.ncs experiment.dat
By doing that, you will get one big file experiment.dat. Then you can use KlustaKwik with that file, but first you need to specify the topology of your channels. Please look at the examples provided in KlustaKwik repository, or check out our experiment.prb file as an example.

[Neuro-Lab-Code]

Thanks for your answer.
I understand that this way gives one .dat file for all of my Tetrodes where each has four electrodes.

I just want to double check:
Wouldn't it be right to have one .dat file for each Tetrode? and give that file to Klaustawik? Or I should put all the Tetrodes in one .dat file.

Another question is that if I have 24 Tetrodes and 4 electrodes for each of them then the number of my channels are 4 or 24*4=96?

Thanks

[emilmelnikov]

Yes, that's right. You should get one .dat file for all your electrodes.
Then, you need to specify how electrodes are grouped together.
This is done in .prb file.
For each tetrode, you specify the channels and all pairs of channels in that tetrode.
Channel numbers correspond to the order of .ncs files in input parameters, starting from 0.
Again, look at our experiment.prb. We've also used tetrodes.
You need to tell all this information to KlustaKwik because it has been designed for high-density multi-electrode arrays, where you have contiguous field of electrodes instead of isolated tetrodes.

In your case, you use shell filename expansion features and run the following command:

python ncs2dat.py TT[1-9]E[1-4].ncs TT1[0-9]E[1-4].ncs TT2[0-4]E[1-4].ncs

[emilmelnikov]

Actually, if you are using tetrodes, you can use prbgen.py script to automatically generate .prb file.
Just set CHANNEL_GROUPS_COUNT = 24 in this script.
Be sure to use the latest version, I've spotted one small bug there and fixed it.

[Neuro-Lab-Code]

Thanks Emil,

Just to check the program I am only using 2 Tetrodes for now, and I have
done the following
1- run: python ncs2dat.py TT1E1.ncs TT1E2.ncs TT1E3.ncs TT1E4.ncs
TT2E1.ncs TT2E2.ncs TT2E3.ncs TT1E4.ncs test.dat
2- run: python prbgen.py *(Here I set CHANNEL_GROUPS_COUNT = 2 and
CHANNELS_IN_GROUP = 4 )
3- set nchannels = 8 and nbits = 16 in params.prm
4- run: *klusta params.prm

And in the end I get the following error:
test.dat has S=1166226680 bytes, but there are C=8 channels. C should be a
divisor of S.

Could you please tell me where I am doing a mistake?

Thanks
Mina

[Neuro-Lab-Code]

Hi Emil,

I figured out what the problem was. For some reason one of the files
(TT1E2.ncs) didn't have a header file. And I think *prbgen.py *does not
check if a file has a header or not, it would just skip the first part of
the file which is supposed to be the header.

However, even when I do all the previous steps on Tetrodes that have header
files and I run klusta params.prm I would still get this error:
ValueError: the shapes of the appended object and the
/channel_groups/1/waveforms_raw EArray differ in non-enlargeable
dimension 1

Do you know where I might be doing mistakes?

Thanks,
Mina

On Tue, Mar 15, 2016 at 3:48 PM, Mina Shahi mi.shahi@gmail.com wrote:

Thanks Emil,

Just to check the program I am only using 2 Tetrodes for now, and I have
done the following
1- run: python ncs2dat.py TT1E1.ncs TT1E2.ncs TT1E3.ncs TT1E4.ncs
TT2E1.ncs TT2E2.ncs TT2E3.ncs TT1E4.ncs test.dat
2- run: python prbgen.py *(Here I set CHANNEL_GROUPS_COUNT = 2 and
CHANNELS_IN_GROUP = 4 )
3- set nchannels = 8 and nbits = 16 in params.prm
4- run: *klusta params.prm

And in the end I get the following error:
test.dat has S=1166226680 bytes, but there are C=8 channels. C should be a
divisor of S.

Could you please tell me where I am doing a mistake?

Thanks
Mina

[Neuro-Lab-Code]

Hi Emil,

I was wondering if someone can help me with the confusion I have regarding
how I need to provide my data to Klustakwik.
If my data is recorded by 24 Tetrodes which each has 4 electrods, then I
was told that I have to put all the recorded data, i.e., from all Tetrodes
in one single .dat file. And then I make a prb file which basically would
show a graph with 24 components and in each component I have 4 vertices
which are fully connected. And then I think the algorithms (spikedetekt
and masked EM), would take into account of data recorded from all Tetrodes.
But as much as I know, clustering the units recorded from one Tetrode
should not depend on the data recorded with other Tetrodes. So Why do we
take all the data from all the tetrodes at the same time? Wouldn't it be
more efficient to cluster each Tetrode separately?

The same question applies when one has double-shank or quad-shank probes.
Do we need to give the data from all shanks at the same time or can we
cluster them separately. In other words, do spikedetakting and clustering
the algorithems need the data from all shanks or can ithey work only on one
shank at a time?

Thanks
Mina

On Tue, Mar 15, 2016 at 6:51 PM, Mina Shahi mi.shahi@gmail.com wrote:

Hi Emil,

I figured out what the problem was. For some reason one of the files
(TT1E2.ncs) didn't have a header file. And I think *prbgen.py *does not
check if a file has a header or not, it would just skip the first part of
the file which is supposed to be the header.

However, even when I do all the previous steps on Tetrodes that have
header files and I run klusta params.prm I would still get this error:
ValueError: the shapes of the appended object and the
/channel_groups/1/waveforms_raw EArray differ in non-enlargeable
dimension 1

Do you know where I might be doing mistakes?

Thanks,
Mina

On Tue, Mar 15, 2016 at 3:48 PM, Mina Shahi mi.shahi@gmail.com wrote:

Thanks Emil,

Just to check the program I am only using 2 Tetrodes for now, and I have
done the following
1- run: python ncs2dat.py TT1E1.ncs TT1E2.ncs TT1E3.ncs TT1E4.ncs
TT2E1.ncs TT2E2.ncs TT2E3.ncs TT1E4.ncs test.dat
2- run: python prbgen.py *(Here I set CHANNEL_GROUPS_COUNT = 2 and
CHANNELS_IN_GROUP = 4 )
3- set nchannels = 8 and nbits = 16 in params.prm
4- run: *klusta params.prm

And in the end I get the following error:
test.dat has S=1166226680 bytes, but there are C=8 channels. C should be
a divisor of S.

Could you please tell me where I am doing a mistake?

Thanks
Mina

[emilmelnikov]

I figured out what the problem was. For some reason one of the files
(TT1E2.ncs) didn't have a header file. And I think *prbgen.py *does not
check if a file has a header or not, it would just skip the first part of
the file which is supposed to be the header.

prbgen.py does not read .dat file, it only generates the probe files suitable for tetrodes (or other electrode configurations similar to tetrodes) so that you won't need to write it manually.

However, even when I do all the previous steps on Tetrodes that have header
files and I run klusta params.prm I would still get this error:
ValueError: the shapes of the appended object and the
/channel_groups/1/waveforms_raw EArray differ in non-enlargeable
dimension 1

We assumed that every channel has the same structure (i.e. the same sampling and recording length). Are you using the same settings for all your channels?

I was wondering if someone can help me with the confusion I have regarding
how I need to provide my data to Klustakwik.
If my data is recorded by 24 Tetrodes which each has 4 electrods, then I
was told that I have to put all the recorded data, i.e., from all Tetrodes
in one single .dat file. And then I make a prb file which basically would
show a graph with 24 components and in each component I have 4 vertices
which are fully connected. And then I think the algorithms (spikedetekt
and masked EM), would take into account of data recorded from all Tetrodes.
But as much as I know, clustering the units recorded from one Tetrode
should not depend on the data recorded with other Tetrodes. So Why do we
take all the data from all the tetrodes at the same time? Wouldn't it be
more efficient to cluster each Tetrode separately?

KlusaKwik was specifically redesigned to support large multielectrode arrays. There, you have many shanks evenly spread out across some surface, so you have to take into account all of them, one way or another. This is why new masked EM clustering algorithm has been invented.

The same question applies when one has double-shank or quad-shank probes.
Do we need to give the data from all shanks at the same time or can we
cluster them separately. In other words, do spikedetakting and clustering
the algorithems need the data from all shanks or can ithey work only on one
shank at a time?

If distance between your shanks is large enough (about 250 microns), then I believe that only one shank at a time can be clustered.
If you have higher densities, then you'll have to take all shanks into account.
Increasing the density makes spike sorting more robust to spike overlaying and other types of noise, so you should get better results with MEAs if you have an equipment and an appropriate specimen.