This section will follow an example to illustrate the data loading process. Follow along each step to load Lonestar Pipeline data for the fictitious Widgets Inc. company.
Create a directory titled WID. The directory name should be the same as the uppercase, three letter package identifier name.
 | The identifier may also contain numbers (i.e., WI1 or WI2) but the first character of the identifier must be a letter. |
$ mkdir WID $ cd WID
The WID directory will contain the make_data file that sets up the columns and categories to store your data and the csv data file.
For this example we are going to add some Lonestar pipeline data for the Widgets Inc. Company. We need to create a WidgetsInc category to house all company specific data so that it is easy for users to find. Under that we will add a WID_Pipeline category to house all the pipeline data. Then we will create a symbol, WID.LONESTAR_1 to reside in the WID_Pipeline category. We will also need to create a column for this symbol called Storage under the pre-existing Volume category.
Our hierarchy should look like this:
Other
WidgetsInc
WID_Pipeline
WID.LONESTAR_1The make_data file MUST be called make_data. If the file is not named make_data then the loader scripts will not be able to execute it.
$ vi make_data
Below is a sample of the required contents of a make_data file. The make_data file is composed of BMIM code that sets up the structure of your data; creating the necessary columns and folders to store your data.
#The “database_narrow” command limits which database will be accessed.
database_narrow {database = ~/data.widgets/xmim.mim;}
#The “lock_files” command locks the database and is required when adding or changing data.
lock_files
#Commands that tell the load_updates.sh script to only log certain errors and ignore others.
suppress_duplicate_add
suppress_duplicate_delete
#BODY OF MAKE_DATA
#This section creates a category named “WidgetsInc” under the “Other” category. The category is #identified by assigning the “type” as a category.
relation_add
{
parent = TopRelation:Other;
name = WidgetsInc;
type = category;
description = ”WidgetsInc Data”;
}
#This section creates a category named “WID_Pipeline” under the “WidgetsInc” category (created in #the section above). Use “type” to identify the entry as a “category”.
relation_add
{
parent = TopRelation:Other:WidgetsInc;
name = WID_Pipeline;
type = category;
description = “WID Pipeline”;
}
#This section creates a symbol named “WID.LONESTAR_1” under the “WID_Pipeline” category (created #in the section above). The symbol is defined by assigning the “type” as “normal”.
relation_add
{
parent = TopRelation:Other:WidgetsInc:WID_Pipeline;
name = WID.LONESTAR_1;
type = normal;
description = “WID Lonestar Pipeline”;
}
#This section creates a new column under the pre-existing category “Volume”. Notice that the #command is “column_add” instead of “relation_add”.
column_add
{
parent = Volume;
name = Storage;
description = “Storage”;
type = normal;
}
#This section links the new symbol with the new column. This link makes the column available for #symbols.
relation_column_add
{
relation = TopRelation:Other:WidgetsInc:WID_Pipeline:WID.LONESTAR_1;
column = Storage;
type = base;
}
#The facts_read command tells the MIM what file actually holds the data points to be loaded.
facts_read {
column = Relation,Date,Storage;
file = wid_data.csv;
}
#The “unlock_files” command unlocks the database. This is a required field.
unlock_files
database_widenExplanation of make_data file starting from top to bottom:
database_narrow – tells the MIM to load the enclosed data into the specified database. It is very important that you tell it to narrow to the empty database you built. If you do not narrow to your database, the MIM will narrow to the ~/data database by default which can cause data integrity problems.
lock_files – Advises the database that the script is going to write. Permissions for writing are checked.
suppress – these are commands that tell the load_updates.sh script to only log certain errors and ignore others.
BODY – this is where you will insert all your commands to build your hierarchy and add symbols and columns. Information and examples on how to author the body of a make_data file is located in the section below. For more coverage of BMIM see BMIM Scripting Language.
facts_read – this is the line that tells the MIM what file actually holds the data points that are to be loaded. In the above example, the file that contains the actual data is the wid_data.csv file. The data file must also be present in the /home/lim/WID/ directory. The
column =line tells the MIM what each column of data in the .csv file represents.
For information on how to create a make_data file for futures type data, see Loading Futures Type Data (Monthly Curves) .
The data file can be a text file (.txt) or a .csv file. For this example the file is saved as a .csv file. Enter the data using the following format:
Relation,Date,Storage
The contents of the file will look similar to this:
WID,20040429,100.45 WID,20040429,100.49 Etc..
The first column is the symbol you created, the second is the date the data belongs to and the third is the actual data values to be loaded.
Once the make_data script file and the data file (wid_data.csv) are built and ready to go, the directory will resemble the following:
WID/make_data WID/wid_data.csv
Now you are ready to package your directory into a LIM update package. To use the Package Maker, enter the following from a command prompt:
$ run.packager source_dir upd_name
where:
source_dir – is the directory containing the make_data and data files
upd_name – is the name assigned to the resulting package
For this example, build the data package as follows:
$ run.packager WID upd_0_WID_20040429
This command will create the upd_0_WID_20040429 package out of your WID directory. For more information on running the Package Maker, see Using the Package Maker.
To load your custom data package into your database, you will need to copy it into the updates directory in your home directory typically located in /home/lim/updates/. Before doing this, you will need to verify that your normal update process is not currently running. To check for update processes, run the following command:
ps –ef | grep 4091
If there is a 4091 XMIM slave server process
currently running for your server, wait until it is finished before
continuing.
Now that there are no update processes running, copy your data package to the updates directory:
cp upd_0_WID_20040429 /home/lim/updates/
You have two choices at this point. You can wait for the package to process at the server’s normal updating interval, or you can manually run the updates process now. To run the updates process, go to your home directory and execute the following:
cron_updates.sh &
The & forces the cron_updates.sh script to run in the background so that you can continue to work in your current window. The cron_updates.sh script will first pickup any LIM packages sitting in queue in your FTP account, and then it will execute updates (load_updates.sh) for all packages that are picked up and your custom package. To learn more about load_updates.sh and other systems administration procedures, please refer to the MIM System Administration Guide .
Once the updates process has finished loading data into your server, you can check the file ~/conf/load_updates.hst to verify that there were no errors encountered with your custom package. The file will contain the last exit status of load_updates.sh. If all worked will, the status will be 0. Any other status indicates a warning or error. If the status indicates a warning or error then proceed with the following log checks to locate the problem.
A log file is created each time the MIM server runs the update process. The log file naming convention is log. appended to the package number and the date of the package. So, upd_0_20040429 would create the log file log.0_20040429. Use an editor such as vi to search the log for the words error or fatal to locate any problems while loading the data.
The following command will return the package name if there are any errors:
$ egrep -il error updates/logs/log_0.20040429
If there are errors located, the command will return the following as output:
updates/logs/log_0.20040429
It simply repeats the package name you entered. If there are no errors, then you will get nothing. Repeat the same for fatal:
$ egrep -il fatal updates/logs/log_0.20040429
If there were no errors, your data will be loaded onto your MIM Server in the manner in which you specified. If errors are encountered, the problems will be specified in the associated log file. For help with errors you do not understand, please contact LIM support at support@lim.com.