Meta Data
With the large amounts of data collected over the last couple semesters, we needed a better way of organizing and analyzing the data. However, complications arise from the fact that within the large set of data that we have lots of different parameters and settings. Therefore, I have developed a meta data organization scheme that will help us analyze the data and compare experiments of the same ilk.
Format
The Meta Data file is an Excel spreadsheet that contains columns of various important parameters that are relevant to describing each experimental run. These data categories were chosen based on the various parameters that the Mathcad Data Processor requires and also a few extra parameters that detail the nature of the experiment. Each row of the Meta Data file correspond to a continuous (usually automated) experimental run. These experimental runs can include many iterations of various parameters and can span over 48 hours; however, each run is controlled by one Process Controller method file, so the output data is of the same format. Each experimental run is tagged by an ID-currently the IDs are simply numbers, but changes are permitted should the need to augment or alter the ID schemes arise.
New Data Processor
Since the Meta Data file eliminates the need to manually enter in many of the parameters needed atop the old Data Processor, a new Data Processor was written specifically to use the new Meta Data file. In addition to the new Meta Data file, new functions were written to extract data from the Meta Data file and saved in the Mathcad file containing all of the lengthy functions called by the Data Processor.
The new Data Processor uses the same functions used in the old one to call up information from the 'statelog' files to correctly extract and display the data stored in the 'datalog' files. A new feature is that two (or more if desired) experimental runs can be loaded up and have their data displayed. Moreover, a new routine was written to look up the row number corresponding to the data types that are iterated. For example, if we are comparing experiments in which the plant flow rate was iterated and we wished to compare the runs where the flow rate was 3.1 mL/s for two different flocculator lengths, it is important that we display the column where the plant flow rates are stored so we will know where to find the run corresponding to a flow rate of 3.1 mL/s for each of the two experiments.
Currently, the new Data Processor still utilizes the 3-D plots found in the original Data Processor, but we are looking at the possibility of using the new Data Processor as a tool to identify data that should be further analyzed and create another program that will do more detailed graphing schemes. The data modeling efforts may be where more detailed 2-D graphs will be displayed.
Limitations
For the time being, only experiments of the same format can be automatically analyzed using the Meta Data Excel file and the new Mathcad Data Processor. For example, data from an experiment where flow rate and influent turbidity were iterated can only be compared (without additional manual adjustments) with another data set from an experiment where flow rate and influent turbidity were also iterated. Therefore, the aforementioned experiements cannot be compared to an experiment in which alum dose concentration was iterated without some kind of manual adjustment of the Mathcad Data Processor. This is not a serious problem because most of the analysis is done between experiments that have the same control conditions and which are measuring similar properties.
Attachments
The Meta Data file and the new Data Processor are attached for reference. Look up the "ID" of the experiment you wish to look at and enter it into one of the two places in the Mathcad file and it will automatically parse the data. Necessary inputs are needed for parsing relevant data types as well as for plotting the correct data.