When I display the IOM GUI back to my local computer, the refresh rate is very slow. Can I run the IOM without the GUI?
Yes, you can run the IOM without the GUI. However, we urge you set up your experiment using the GUI; this will generate a parameter file called "inversion.xml" in the "run" directory. Once this file is created, you can initiate a run from a command window by typing "iom_script.pl inversion.xml" in the run directory. You can also make changes to the parameter file (such as the number of outer iterations), but we urge you to use the GUI to make any substantial changes to your experimental parameters.
There are so many directories of output files! Which ones do I use?
A directory is created for the "observation" outputs for each outer iteration, e.g., "outputs.1," "outputs.2," "outputs.3," etc. These contain outputs related to the data, including penalties. The names of the files within these directories are fairly self explanatory (more details can be found in Appendix A of the Users Manual). For example:
outputs.1 contains the outputs for outer iterate #1
outputs.2 contains the outputs for outer iterate #2, etc.
Furthermore:
iom_penalty_jF.n in outputs.1 contains J(prior) for outer iterate # 1
iom_penalty_jF.n in outputs.2 contains J(prior) for outer iterate # 2, etc.
Another directory, called "outputs" is also created - this is a scratch directory and should contain outputs from the last outer iteration.
Similarly, a directory is created for the trajectories for each outer iteration iteration, e.g., "sharedata.1," "sharedata.2," "sharedata.3," etc. along with a scratch directory "sharedata." These directories contain output files with dimensions associated with model (rather than the data). For example:
sharedata.1 contains the traj for outer iterate #1
sharedata.2 contains the traj for outer iterate #2 and etc.
Futhermore:
in sharedata.1, traj_direct_representer_1.nc is the representer for obs #1, for outer iterate #1
traj_direct_representer_2.nc is the representer for obs #2, for outer iterate #1, etc.
traj_background_0.nc is the background field, for outer iterate #1
traj_prior_1.nc is the prior field, for outer iterate #1
traj_optimal_direct_representer.nc is the best-estimate, for outer iterate #1
in sharedata.2, traj_direct_representer_1.nc is the rep for obs #1, for outer iterate #2
traj_direct_representer_2.nc is the rep foe obs #2, for outer iterate #2, etc.
traj_background_1.nc is the background field, for outer iterate #2
traj_prior_2.nc is the prior field, for outer iterate #2
traj_optimal_direct_representer.nc is the best-estimate, for outer iterate #2
(Note the exact files that appear in your directories depend upon the choices made on the "output" tab in the GUI.)
In the output directories, there are some files with the word "optimal" in their names. Usually, there is another file with nearly the same name, except without the word "optimal." For example, in the outputs directory:
iom_penalty_jHat_direct_representer.nc
iom_penalty_jHat_optimal_direct_representer.nc
and in the sharedata directory:
traj_adjoint_direct_representer_1.nc
traj_adjoint_optimal_direct_representer.nc
What do these mean?
The word "optimal" is used in two different senses here. In outputs, iom_penalty_jHat_direct_representer.nc is the reduced penalty, computed directly by h^T beta_hat, where the beta_hat are computed by inv(P*)h. iom_penalty_jHat_optimal_direct_representer.nc is also the reduced penalty, computed by h^T C_eps^-1 ( d - L[u_hat] ). Comparison of the two is often used a sanity check that the inversion has been set up correctly. For more details, see Equation (3.1.3) in Bennett, 2002.
However, in sharedata, the sense of optimal is different. Here, the file "traj_adjoint_direct_representer_1.nc" is the adjoint from the first representer. There is a similar file for each representer. However, the file traj_adjoint_optimal_direct_representer.nc is the adjoint computed during the final sweep (that is, forced by a comb of scaled representers). For more details, see the discussion in the lesson pages.
There are a number of output files in sharedata related to "convolved" direct representers. What do these mean?
Files with names like traj_convolved_direct_representer_1.nc are the convolved (deweighted) adjoint associated with the first representer. Likewise, traj_convolved_direct_representer_2.nc is the convolved (deweighted) adjoint associated with the second representer. The file traj_convolved_optimal_direct_representer.nc is the convolved (deweighted) adjoint from the final sweep; these are the model residuals.
When I use the IOM using the "direct representer method," I get output files for both the "direct" and "indirect" representer methods. What do the "indirect" output files mean?
Most likely, when working with GUI, under the "algorithm" tab, you chose "direct representer" for the minimization algorithm, and then also chose a solver (either conjugate-gradient or biconjugate gradient). The direct representer method doesn't use a solver in this sense, and so an inconsistency has been introduced. When selecting the "direct representer" method, do not choose a solver. [Alternatively, when you choose the "indirect representer" method, you must choose a solver!]. Note, the files in the outputs and sharedata directories are copied into the outputs.1, outputs.2, etc. and sharedata.1, sharedata.2, etc. directories, respectively. If you've made the mistake of choosing a solver for the "direct representer" method, you'll want to remove all of the files in all of the outputs directories (outputs, outputs.1, outputs.2, etc.) and sharedata directories (sharedata, sharedata.1, sharedata.2, etc.) before rerunning the IOM.
How do I change the error covariance for the model?
We assume that you are using the default covariances offered by the IOM (that is, bell-shaped in space, Markovian in time). The covariance information needed by the IOM consists of a number of length and time scales as well as the variance. If you are using the GUI, then the length and time scales can be changed within the "parameters" tab. You can also change the name of the file that contains the variance information; because the variance need not be constant, it can't be entered directly into the GUI. So, you can either change the name of the variance file (for example, traj_model_std_err.nc to traj_model_std_err_new.nc), or you can change the variance file itself.
If you are not using the GUI, the length scales can be changed in the inversion.xml file. For example, for the KdV example:
<var name="lambda" type="correlated">
<space_len_dC idx="1">2687.0</space_len_dC>
<time_len_dC>100.0</time_len_dC>
</var>
<var name="xi" type="correlated">
<space_len_dC idx="1">5375.0</space_len_dC>
Additionally, the name of the file containing the variance information can also be changed in the inversion.xml file:
<traj_std_err_filename>traj_model_std_err.nc</traj_std_err_filename>
(the name of the file is shown here in purple).
What is the easiest way to change a weak constraint to a strong constraint, and vice versa?
Edit the section of the inversion.xml file where the character of the potentially weak variables is specified. For example, from the KdV example:
<var name="lambda" type="correlated">
<space_len_dC idx="1">2687.0</space_len_dC>
<time_len_dC>100.0</time_len_dC>
</var>
<var name="xi" type="correlated">
<space_len_dC idx="1">5375.0</space_len_dC>
Simply change the "type" for the desired variables. The options are:
"correlated": means that the error is correlated. If IOM-provided convolution algorithms are used, requires length/time scales as well as a file containing information about the variance.
"strongConstr": means that the error is zero. No length/time scales or variance information is required.
"uncorrelated": means that the error is nonzero, but uncorrelated. No length/time scales are required (they are known to be zero by the IOM), but does require a file containing information about the variance. Note, this is can be a useful investigational tool as it uses the unconvolved adjoint to force the representers when the variances are set to 1.0.
How can I edit the netcdf files used by the IOM? I don't have any special software for this.
Chances are you've been using "ncdump" to view netcdf files. Simply redirect the screen output to a file, for example:
ncdump traj_model_std_err.nc > temp.cdl
(where cdl indicates "network Common Data form Language"), and then edit the ascii cdl file using an editor of your choice. Then, use the companion command "ncgen" to convert from the cdl file temp.cdl into a netcdf file temp.nc:
ncgen -o temp.nc temp.cdl
and then rename temp.nc as desired. For more information about ncdump and ncgen:
http://www.unidata.ucar.edu/software/netcdf/docs/ncdump-man-1.html
http://www.unidata.ucar.edu/software/netcdf/docs/ncgen-man-1.html
Yes, you can run the IOM without the GUI. However, we urge you set up your experiment using the GUI; this will generate a parameter file called "inversion.xml" in the "run" directory. Once this file is created, you can initiate a run from a command window by typing "iom_script.pl inversion.xml" in the run directory. You can also make changes to the parameter file (such as the number of outer iterations), but we urge you to use the GUI to make any substantial changes to your experimental parameters.
There are so many directories of output files! Which ones do I use?
A directory is created for the "observation" outputs for each outer iteration, e.g., "outputs.1," "outputs.2," "outputs.3," etc. These contain outputs related to the data, including penalties. The names of the files within these directories are fairly self explanatory (more details can be found in Appendix A of the Users Manual). For example:
outputs.1 contains the outputs for outer iterate #1
outputs.2 contains the outputs for outer iterate #2, etc.
Furthermore:
iom_penalty_jF.n in outputs.1 contains J(prior) for outer iterate # 1
iom_penalty_jF.n in outputs.2 contains J(prior) for outer iterate # 2, etc.
Another directory, called "outputs" is also created - this is a scratch directory and should contain outputs from the last outer iteration.
Similarly, a directory is created for the trajectories for each outer iteration iteration, e.g., "sharedata.1," "sharedata.2," "sharedata.3," etc. along with a scratch directory "sharedata." These directories contain output files with dimensions associated with model (rather than the data). For example:
sharedata.1 contains the traj for outer iterate #1
sharedata.2 contains the traj for outer iterate #2 and etc.
Futhermore:
in sharedata.1, traj_direct_representer_1.nc is the representer for obs #1, for outer iterate #1
traj_direct_representer_2.nc is the representer for obs #2, for outer iterate #1, etc.
traj_background_0.nc is the background field, for outer iterate #1
traj_prior_1.nc is the prior field, for outer iterate #1
traj_optimal_direct_representer.nc is the best-estimate, for outer iterate #1
in sharedata.2, traj_direct_representer_1.nc is the rep for obs #1, for outer iterate #2
traj_direct_representer_2.nc is the rep foe obs #2, for outer iterate #2, etc.
traj_background_1.nc is the background field, for outer iterate #2
traj_prior_2.nc is the prior field, for outer iterate #2
traj_optimal_direct_representer.nc is the best-estimate, for outer iterate #2
(Note the exact files that appear in your directories depend upon the choices made on the "output" tab in the GUI.)
In the output directories, there are some files with the word "optimal" in their names. Usually, there is another file with nearly the same name, except without the word "optimal." For example, in the outputs directory:
iom_penalty_jHat_direct_representer.nc
iom_penalty_jHat_optimal_direct_representer.nc
and in the sharedata directory:
traj_adjoint_direct_representer_1.nc
traj_adjoint_optimal_direct_representer.nc
What do these mean?
The word "optimal" is used in two different senses here. In outputs, iom_penalty_jHat_direct_representer.nc is the reduced penalty, computed directly by h^T beta_hat, where the beta_hat are computed by inv(P*)h. iom_penalty_jHat_optimal_direct_representer.nc is also the reduced penalty, computed by h^T C_eps^-1 ( d - L[u_hat] ). Comparison of the two is often used a sanity check that the inversion has been set up correctly. For more details, see Equation (3.1.3) in Bennett, 2002.
However, in sharedata, the sense of optimal is different. Here, the file "traj_adjoint_direct_representer_1.nc" is the adjoint from the first representer. There is a similar file for each representer. However, the file traj_adjoint_optimal_direct_representer.nc is the adjoint computed during the final sweep (that is, forced by a comb of scaled representers). For more details, see the discussion in the lesson pages.
There are a number of output files in sharedata related to "convolved" direct representers. What do these mean?
Files with names like traj_convolved_direct_representer_1.nc are the convolved (deweighted) adjoint associated with the first representer. Likewise, traj_convolved_direct_representer_2.nc is the convolved (deweighted) adjoint associated with the second representer. The file traj_convolved_optimal_direct_representer.nc is the convolved (deweighted) adjoint from the final sweep; these are the model residuals.
When I use the IOM using the "direct representer method," I get output files for both the "direct" and "indirect" representer methods. What do the "indirect" output files mean?
Most likely, when working with GUI, under the "algorithm" tab, you chose "direct representer" for the minimization algorithm, and then also chose a solver (either conjugate-gradient or biconjugate gradient). The direct representer method doesn't use a solver in this sense, and so an inconsistency has been introduced. When selecting the "direct representer" method, do not choose a solver. [Alternatively, when you choose the "indirect representer" method, you must choose a solver!]. Note, the files in the outputs and sharedata directories are copied into the outputs.1, outputs.2, etc. and sharedata.1, sharedata.2, etc. directories, respectively. If you've made the mistake of choosing a solver for the "direct representer" method, you'll want to remove all of the files in all of the outputs directories (outputs, outputs.1, outputs.2, etc.) and sharedata directories (sharedata, sharedata.1, sharedata.2, etc.) before rerunning the IOM.
How do I change the error covariance for the model?
We assume that you are using the default covariances offered by the IOM (that is, bell-shaped in space, Markovian in time). The covariance information needed by the IOM consists of a number of length and time scales as well as the variance. If you are using the GUI, then the length and time scales can be changed within the "parameters" tab. You can also change the name of the file that contains the variance information; because the variance need not be constant, it can't be entered directly into the GUI. So, you can either change the name of the variance file (for example, traj_model_std_err.nc to traj_model_std_err_new.nc), or you can change the variance file itself.
If you are not using the GUI, the length scales can be changed in the inversion.xml file. For example, for the KdV example:
<var name="lambda" type="correlated">
<space_len_dC idx="1">2687.0</space_len_dC>
<time_len_dC>100.0</time_len_dC>
</var>
<var name="xi" type="correlated">
<space_len_dC idx="1">5375.0</space_len_dC>
Additionally, the name of the file containing the variance information can also be changed in the inversion.xml file:
<traj_std_err_filename>traj_model_std_err.nc</traj_std_err_filename>
(the name of the file is shown here in purple).
What is the easiest way to change a weak constraint to a strong constraint, and vice versa?
Edit the section of the inversion.xml file where the character of the potentially weak variables is specified. For example, from the KdV example:
<var name="lambda" type="correlated">
<space_len_dC idx="1">2687.0</space_len_dC>
<time_len_dC>100.0</time_len_dC>
</var>
<var name="xi" type="correlated">
<space_len_dC idx="1">5375.0</space_len_dC>
Simply change the "type" for the desired variables. The options are:
"correlated": means that the error is correlated. If IOM-provided convolution algorithms are used, requires length/time scales as well as a file containing information about the variance.
"strongConstr": means that the error is zero. No length/time scales or variance information is required.
"uncorrelated": means that the error is nonzero, but uncorrelated. No length/time scales are required (they are known to be zero by the IOM), but does require a file containing information about the variance. Note, this is can be a useful investigational tool as it uses the unconvolved adjoint to force the representers when the variances are set to 1.0.
How can I edit the netcdf files used by the IOM? I don't have any special software for this.
Chances are you've been using "ncdump" to view netcdf files. Simply redirect the screen output to a file, for example:
ncdump traj_model_std_err.nc > temp.cdl
(where cdl indicates "network Common Data form Language"), and then edit the ascii cdl file using an editor of your choice. Then, use the companion command "ncgen" to convert from the cdl file temp.cdl into a netcdf file temp.nc:
ncgen -o temp.nc temp.cdl
and then rename temp.nc as desired. For more information about ncdump and ncgen:
http://www.unidata.ucar.edu/software/netcdf/docs/ncdump-man-1.html
http://www.unidata.ucar.edu/software/netcdf/docs/ncgen-man-1.html