Using DATASTORE to Work with Large Collections of Data

The function datastore can be used to handle large collections of data. As an example, the modern multisensor core loggers for the automatic determination of physical and chemical properties of drill cores generates huge amounts of data within a very short time. An array with 80 variables, each with 50,000 measured values, is not unusual. In addition, the corresponding files contain mixtures of character strings for core section names, measurement dates and times, and comments from the laboratory technician who carried out the measurements, along with the actual numeric readings.

As an example we import the content of the file geoxrf.txt into the MATLAB workspace. The file looks like this in the editor:
Location  Core  Depth     K    Si
ChewBahir   2A   0.30   975  1576
ChewBahir   2A   0.60  2785  9706
ChewBahir   2A   0.90  5469  9572
ChewBahir   2A   0.12  9575  4854
ChewBahir   2A   0.15  9649  8003
As we see the first line contains the header of the data. The first and the second column contains the location name and the name of the drill core. The third column contains the depth in the core. The fourth and fifth column contain numerical values representing X-ray fluorescence counts for potassium and silicon. The columns are separated by one or more spaces. We use the function datastore to create the datastore data without actually importing the data.
clear
data = datastore('geoxrf.txt');
Typing
data
yields a list of properties of data. We can read all of the data from the datastore using
dataxrf = readall(data)
which yields
dataxrf =

  5×5 table
      Location      Core    Depth    K      Si  
   _____________   ______   _____   ____   ____
   {'ChewBahir'}   {'2A'}    0.3     975   1576
   {'ChewBahir'}   {'2A'}    0.6    2785   9706
   {'ChewBahir'}   {'2A'}    0.9    5469   9572
   {'ChewBahir'}   {'2A'}   0.12    9575   4854
   {'ChewBahir'}   {'2A'}   0.15    9649   8003
If the file is too large to be loaded into memory, we can import only part of it. First we define the variables Depth and K we want to load and then we use readall again by typing
data.SelectedVariableNames = {'Depth','K'}
dataxrf = readall(data)
which yields
dataxrf =

  5×2 table

    Depth     K
    _____   ______
     0.3      975
     0.6     2785
     0.9     5469
    0.12     9575
    0.15     9649
Now we can use the properties of tables and their associated functions to convert the data into simple double arrays, e.g. by typing
depth = dataxrf.Depth
potassium = dataxrf.K
which yields
depth =

    0.3000
    0.6000
    0.9000
    0.1200
    0.1500
and
potassium =

         975
        2785
        5469
        9575
        9649
Alternatively, we can use the function table2array to convert the table to a double precision array by typing
dataxrfarray = table2array(dataxrf)
which yields
dataxrfarray =

1.0e+03 *

0.0003  0.9750
0.0006  2.7850
0.0009  5.4690
0.0001  9.5750
0.0001  9.6490