An example with double channel string marker by abcsds · Pull Request #4 · xdf-modules/example-files

abcsds · 2025-01-25T19:59:38Z

Multiple channels with string markers seem to have been an issue in both liblxdf and xdf.jl. This file was uploaded by @tstenner in xdf-modules/libxdf#19
It presents a single data point at time [16.725987961266686] with markers ["Marker 0A" "Marker 0B"].
The header of the file is:

<?xml version=\"1.0\"?><info><name>SendMarker</name><type>Marker</type><channel_count>2</channel_count><nominal_srate>1000</nominal_srate><channel_format>string</channel_format><created_at>10</created_at></info>

The footer of the file is:

<?xml version=\"1.0\"?><info><first_timestamp>10</first_timestamp><last_timestamp>10.001</last_timestamp><sample_count>1</sample_count></info>

cbrnr · 2025-01-27T06:10:11Z

Same comment as in #5, can you add a description for the example file please? Either in a separate .md file, or (this would be my slight preference) add a section in README.md.

cbrnr · 2025-01-29T07:08:38Z

Also, I think multichannel_string_marker.xdf would be a slightly better name, can you rename it please? Or maybe twochannel_string_marker.xdf?

abcsds · 2025-01-31T10:23:45Z

Used twochannel_string_marker.xdf.

cbrnr · 2025-02-03T13:11:03Z

@abcsds we have updated the README, can you please add a new section with a description of this dataset? Thanks!

cbrnr

Thanks @abcsds, I only have a few comments.

README.md

cbrnr · 2025-02-10T07:13:01Z

README.md

+Timestamp: 16.725987961266686
+
+These time-stamps precede the first clock offset measurement, but for
+synchronization they will be handled with respect to the first (and
+only in this case) detected clock segment.


How did you get this timestamp? When loading the file, I get the following output for that stream:

{'clock_times': [6.1], 'clock_values': [-0.1], 'footer': {'info': defaultdict(<class 'list'>, {'first_timestamp': ['10'], 'last_timestamp': ['10.001'], 'sample_count': ['1']})}, 'info': defaultdict(<class 'list'>, {'channel_count': ['2'], 'channel_format': ['string'], 'created_at': ['10'], 'effective_srate': np.float64(999.9999999987779), 'name': ['SendMarker'], 'nominal_srate': ['1000'], 'segments': [(np.int64(0), np.int64(0))], 'stream_id': 3735928559, 'type': ['Marker']}), 'time_series': [['Marker 0A', 'Marker 0B']], 'time_stamps': array([16.9])}

The time stamp is 16.9, or no? This also means that the clock offset measurements precede the timestamp and not the other way round.

Hi Clemens,
This PR, was suggested by you in cbrnr/XDF.jl#14, I loaded it with https://github.com/cbrnr/XDF.jl

Here's the output for twochannel_string_marker.xdf for every stream using XDF.jl:

Pair{Int64, Any}(3735928559, Dict{String, Any}( "time" => [16.725987961266686], "data" => ["Marker 0A" "Marker 0A"], "nchannels" => 2, "name" => "SendMarker", "dtype" => String, "srate" => 1000.0f0, "clock" => [6.1], "offset" => [-0.1], "footer" => "<?xml version=\"1.0\"?><info><first_timestamp>10</first_timestamp><last_timestamp>10.001</last_timestamp><sample_count>1</sample_count></info>", "header" => "<?xml version=\"1.0\"?><info><name>SendMarker</name><type>Marker</type><channel_count>2</channel_count><nominal_srate>1000</nominal_srate><channel_format>string</channel_format><created_at>10</created_at></info>", "type" => "Marker") ) Pair{Int64, Any}(46202862, Dict{String, Any}( "time" => [16.725987961266686], "data" => [0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0], "nchannels" => 64, "name" => "SendDataC", "dtype" => Float64, "srate" => 1000.0f0, "clock" => [6.1], "offset" => [-0.1], "footer" => "<?xml version=\"1.0\"?><info><first_timestamp>10</first_timestamp><last_timestamp>10.001</last_timestamp><sample_count>1</sample_count></info>", "header" => "<?xml version=\"1.0\"?><info><name>SendDataC</name><type>EEG</type><channel_count>64</channel_count><nominal_srate>1000</nominal_srate><channel_format>double64</channel_format><created_at>10</created_at></info>", "type" => "EEG") )

I get the same output as you from pyxdf.

Testing the outputs between the two libraries on minimal.xdf they seem to match:

( [ { 'info': defaultdict(<class 'list'>, { 'name': ['SendDataC'], 'type': ['EEG'], 'channel_count': ['3'], 'nominal_srate': ['10'], 'channel_format': ['int16'], 'created_at': ['50942.723319709003'], 'desc': [None], 'uid': ['xdfwriter_11_int'], 'stream_id': 0, 'effective_srate': np.float64(10.000000000000025), 'segments': [(np.int64(0), np.int64(8))] }), 'footer': { 'info': defaultdict(<class 'list'>, { 'writer': ['LabRecorder xdfwriter'], 'first_timestamp': ['5.1'], 'last_timestamp': ['5.9'], 'sample_count': ['9'], 'clock_offsets': [ defaultdict(<class 'list'>, { 'offset': [ defaultdict(<class 'list'>, { 'time': ['50979.76'], 'value': ['-.01'] }), defaultdict(<class 'list'>, { 'time': ['50979.86'], 'value': ['-.02'] }) ] }) ] }) }, 'time_series': array([[192, 255, 238], [ 12, 22, 32], [ 13, 23, 33], [ 14, 24, 34], [ 15, 25, 35], [ 12, 22, 32], [ 13, 23, 33], [ 14, 24, 34], [ 15, 25, 35]], dtype=int16), 'time_stamps': array([5. , 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8]), 'clock_times': [6.1, 7.1], 'clock_values': [-0.1, -0.1] }, { 'info': defaultdict(<class 'list'>, { 'name': ['SendDataString'], 'type': ['StringMarker'], 'channel_count': ['1'], 'nominal_srate': ['10'], 'channel_format': ['string'], 'created_at': ['50942.723319709003'], 'desc': [None], 'uid': ['xdfwriter_11_str'], 'stream_id': 46202862, 'effective_srate': np.float64(10.000000000000016), 'segments': [(np.int64(0), np.int64(8))] }), 'footer': { 'info': defaultdict(<class 'list'>, { 'writer': ['LabRecorder xdfwriter'], 'first_timestamp': ['5.1'], 'last_timestamp': ['5.9'], 'sample_count': ['9'], 'clock_offsets': [ defaultdict(<class 'list'>, { 'offset': [ defaultdict(<class 'list'>, { 'time': ['50979.76'], 'value': ['-.01'] }), defaultdict(<class 'list'>, { 'time': ['50979.86'], 'value': ['-.02'] }) ] }) ] }) }, 'time_series': [ [ '<?xml version="1.0"?><info><writer>LabRecorder xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp> <sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value ></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>' ], ['Hello'], ['World'], ['from'], ['LSL'], ['Hello'], ['World'], ['from'], ['LSL'] ], 'time_stamps': array([5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 5.9]), 'clock_times': [], 'clock_values': [] } ], {'info': defaultdict(<class 'list'>, {'version': ['1.0']})} )

and Julia

Pair{Int64, Any}(0, Dict{String, Any}( "time" => [5.0, 5.1000000000000005, 5.200000001490117, 5.300000002980233, 5.4, 5.5, 5.600000001490116, 5.700000002980232, 5.800000004470348], "data" => Int16[192 255 238; 12 22 32; 13 23 33; 14 24 34; 15 25 35; 12 22 32; 13 23 33; 14 24 34; 15 25 35], "nchannels" => 3, "name" => "SendDataC", "dtype" => Int16, "srate" => 10.0f0, "clock" => [6.1, 7.1], "offset" => [-0.1, -0.1], "footer" => "<?xml version=\"1.0\"?><info><writer>LabRecorder xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp><sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>", "header" => "<?xml version=\"1.0\"?><info><name>SendDataC</name><type>EEG</type><channel_count>3</channel_count><nominal_srate>10</nominal_srate><channel_format>int16</channel_format><created_at>50942.723319709003</created_at><desc/><uid>xdfwriter_11_int</uid></info>", "type" => "EEG") ) Pair{Int64, Any}(46202862, Dict{String, Any}( "time" => [5.1, 5.2, 5.300000001490116, 5.400000002980232, 5.5, 5.6, 5.700000001490116, 5.800000002980232, 5.900000004470348], "data" => ["<?xml version=\"1.0\"?><info><writer>LabRecorder xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp><sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>"; "Hello"; "World"; "from"; "LSL"; "Hello"; "World"; "from"; "LSL";;], "nchannels" => 1, "name" => "SendDataString", "dtype" => String, "srate" => 10.0f0, "clock" => Float64[], "offset" => Float64[], "footer" => "<?xml version=\"1.0\"?><info><writer>LabRecorder xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp><sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>", "header" => "<?xml version=\"1.0\"?><info><name>SendDataString</name><type>StringMarker</type><channel_count>1</channel_count><nominal_srate>10</nominal_srate><channel_format>string</channel_format><created_at>50942.723319709003</created_at><desc/><uid>xdfwriter_11_str</uid></info>", "type" => "StringMarker") )

This doesn't look like a rounding error. Do you see where the difference is? I can't seem to find the source of it.

This is strange. I don't have the time to dive into this right now, but maybe XDF.jl does clock correction by default?

Hi Clemens, I can work on it on short periods, but continuously. If you check it out every once in a while and point me in the right direction is very helpful.

Both do clock correction by default (clock syncing):

pyxdf https://github.com/xdf-modules/pyxdf/blob/main/src/pyxdf/pyxdf.py#L77

xdf.jl https://github.com/cbrnr/XDF.jl/blob/d3bb58bd11e1699471e747974125e40e1143a5fc/src/XDF.jl#L34C45-L34C60

However, pyxdf also does de-jittering by default. https://github.com/xdf-modules/pyxdf/blob/main/src/pyxdf/pyxdf.py#L79. XDF.jl still has it in pr: cbrnr/XDF.jl#13, so we should consider turn it it on as default on that branch. I think I left it off by default.

Still, when loading pyxdf.load_xdf("twochannel_string_marker.xdf", dejitter_timestamps=False) I get the same array:

... 'name': ['SendMarker'], 'type': ['Marker'], 'channel_count': ['2'], 'nominal_srate': ['1000'], 'channel_format': ['string'], 'created_at': ['10'], 'stream_id': 3735928559, 'effective_srate': 0.0, 'segments': [(0,0)] ... 'footer': { 'info': defaultdict(<class 'list'>, { 'first_timestamp': ['10'], 'last_timestamp': ['10.001'], 'sample_count': ['1'] }) }, 'time_series': [['Marker 0A', 'Marker 0B']], 'time_stamps': array([16.9]), 'clock_times': [6.1], 'clock_values': [-0.1] ...

I think it's unlikely to be the de-jittering. It doesn't really do anything to the time stamp, (at least for this file at the floating point precision we use to compare the two libraries). I guess next I'll go debugging pyxdf, to see what it actually does. Unless you have another idea of what it could be.

Thanks!

README.md

cbrnr · 2025-02-19T08:47:21Z

Looks good! Now we should try to figure out the time stamp issue (i.e., why XDF.jl reports a different time stamp than pyxdf)!

An example with double channel string marker

aea47e7

abcsds mentioned this pull request Jan 25, 2025

Add tests for string markers cbrnr/XDF.jl#14

Open

Rename file

2645170

abcsds added 2 commits February 6, 2025 10:11

Merge branch 'xdf-modules:master' into master

8cf234e

Add description of twochannel_string_marker.xdf to README.md

653d762

cbrnr reviewed Feb 10, 2025

View reviewed changes

Add second string of twochannel_string_marker to README

d0d9107

cbrnr reviewed Feb 18, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

cbrnr reviewed Feb 18, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

mend

cd60faf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

An example with double channel string marker#4

An example with double channel string marker#4
abcsds wants to merge 6 commits intoxdf-modules:masterfrom
abcsds:master

abcsds commented Jan 25, 2025

Uh oh!

cbrnr commented Jan 27, 2025

Uh oh!

cbrnr commented Jan 29, 2025

Uh oh!

abcsds commented Jan 31, 2025

Uh oh!

cbrnr commented Feb 3, 2025

Uh oh!

cbrnr left a comment

Uh oh!

Uh oh!

cbrnr Feb 10, 2025

Uh oh!

abcsds Feb 18, 2025

Uh oh!

cbrnr Feb 18, 2025

Uh oh!

abcsds Feb 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbrnr commented Feb 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

abcsds commented Jan 25, 2025

Uh oh!

cbrnr commented Jan 27, 2025

Uh oh!

cbrnr commented Jan 29, 2025

Uh oh!

abcsds commented Jan 31, 2025

Uh oh!

cbrnr commented Feb 3, 2025

Uh oh!

cbrnr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cbrnr Feb 10, 2025

Choose a reason for hiding this comment

Uh oh!

abcsds Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

cbrnr Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

abcsds Feb 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbrnr commented Feb 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants