How to save(persist) array data at one time.

[cyrus-baker]

I have some array data like

x = np.linspace(0, 10, 100)
y = np.linspace(20, 30, 50)
z = np.random.random((100, 50))

And actually z is depending on x and y.

I can get these data at one time for some reason, without need to iterate over axis x or y.

And, saving data like the simplest way

context = qcodes.Measurement(experiment, name="demo")
x_param = Parameter("x")
y_param = Parameter("y")
z_param = Parameter("z")
context.register_parameter(x_param)
context.register_parameter(y_param)
context.register_parameter(z_param, setpoints=(x_param, y_param))
with context.run() as datasaver:
    for i, _x in enumerate(x):
        for j, _y in enumerate(y):
            datasaver.add_result(
                (x_param, _x),
                (y_param, _y),
                (z_param, z[i, j]),
            )

is intolerably slow.

My question is that it there any way to save these data at one time without iteration.

I found there was a discussion before #3330, but it seems not help to meet my needs.

[astafan8]

@littlebaker

I can get these data at one time for some reason

i guess that's just how your instruments work, right? :)

now moving to answering your question:

1. there's no need to iterate over the values of the arrays that you're getting, add_results supports taking arrays of values, so you can pass a grid made of x and y and the z values directly to add_results like this .add_results((x_param, x_gridd), (y_param, y_grid), (z_param, z)) where x_grid, y_grid = numpy.meshgrid(x, y) (i might be wrong with the exact use of meshgrid, please look in numpy docs for details of how to create a grid from vectors) and qcodes internally will store them as (x_grid[1], y_grid[1], z[1]), (x_grid[2], y_grid[2], z[2]), ... etc.
2. another thing that you probably should look into is storing those arrays "as is", using what's called an "array" "paramtype", here are the docs pages about it https://qcodes.github.io/Qcodes/examples/DataSet/Paramtypes%20explained.html https://qcodes.github.io/Qcodes/examples/DataSet/Dataset_Performance.html (and read this to have a full picture of how to store data https://qcodes.github.io/Qcodes/examples/DataSet/Performing-measurements-using-qcodes-parameters-and-dataset.html), and also about a ParameterWithSetpoints which can be used to implement such z parameters for which data is returned together with it's "setpoints" x and y on the instrument driver https://qcodes.github.io/Qcodes/examples/Parameters/Parameter-With-Setpoints-defined-on-a-different-instrument.html https://qcodes.github.io/Qcodes/examples/Parameters/Parameters.html https://qcodes.github.io/Qcodes/examples/Parameters/Simple-Example-of-ParameterWithSetpoints.html .

let us know if this helps :)

[cyrus-baker]

Oh, thank you. It's really helpful. Using meshgrid is the solution.
By the way I think it seems to be wasty of disk space while using meshgrid. If that should be taken into count, I may need to consider other ways to persist data.
I'll consider, thanks.

[astafan8]

By the way I think it seems to be wasty of disk space while using meshgrid. If that should be taken into count, I may need to consider other ways to persist data.

yes :) this is why I highly recommend to check documentation is linked in the point 2 of the answer ;) this will make storing the x/y/z arrays noticably faster (also you'll notice improvement when reading/loading the data)

[cyrus-baker]

I have read the docs and I found that Qcodes is saving 2d data with meshgrid by default, which means that if I have z with shape (5, 3), and then I have to simultaneously save x with shape (5, 3) and y (5, 3). However, if we know that the z is well structured(meshgrided), we can just save x with shape (5, ) and y with shape (3, ). It could save nearly 2/3 of space.
I wonder if this is designed with some concerns. 'cause we often save a large amount of data and make the database very large.

[cyrus-baker]

Now I can save those param separately and rebuild them manually, but I wonder if there is any elegant solution.