GRASS 5.0.x and PostgreSQL - First Steps



[$Id: index.html,v 1.8.4.2 2003/08/20 08:36:51 markus Exp $]
[Go here for the GRASS 5.1.x tutorial]
Recommended reading:
http://www.PostgreSQL.org/docs/index.html


GRASS 5.0.x/PostgreSQL interface modules
  1. g.select.pg - select a database to use in GRASS 5.0.x.
  2. g.table.pg - list tables in currently selected DB.
  3. g.columns.pg - list columns information in a table of selected DB.
  4. g.stats.pg - calculate means, min, max, frequencies of specified numeric column.
  5. d.site.pg - display and optionally copy/reclass sites selected by DB query.
  6. d.vect.pg - display and optionally extract lines/areas selected by DB query.
  7. d.rast.pg - display(optionally) and reclass raster maps through DB query.
  8. d.what.s.pg - query with mouse sites on graphics monitor. Option to dump the fields in comma-separated lists for other programs.
  9. d.what.v.pg - query with mouse lines/areas. May be used by UPDATE command in PostgreSQL input through SQL command file. Consider piping anything you grabbed on screen through various filters.
  10. d.what.r.pg - the same as above on raster maps.
  11. v.reclass.pg - reclass vector maps, rules based on any SELECT filter.
  12. v.in.shape.pg - DEPRECATED. Use v.in.shape + pg.in.dbf instead.
  13. v.in.arc.pg - DEPRECATED. Use v.in.arc + pg.in.dbf instead.
  14. pg.in.dbf - program that inputs DBF file and dumps all columns to PostgreSQL.
  15. v.to.pg - this program is used to export GRASS 5.0.x vectors (lines or polygons) to PostgreSQL native "line" or "polygon" formats.
  16. NVIZ2.2 - PostgreSQL interface. I have added new entries field in the "What's here" panel; PostgreSQL queries may be disabled in "Attributes" checkbox.
Please see these modules manuals for explanations and examples.

NB: Tcl-Tk modules may have input "where" clause restricted to one rule typed without "whitespace" while in terminal input the number of sub-clause is not limited by this.

Bugs and other possible caveats reports are welcome.

Alex Shevlakov,
sixote@yahoo.com

==================================================================
Short introduction to GRASS 5.0.x/PostgreSQL interface Markus Neteler and Alex Shevlakov
The following text shall introduce you to interface usage. This text is subject to change...

Let's start.
Enter GRASS 5.0.x

Of course you need to have a location defined and some data you want to import here.

------------------------------------------------------------------------
A) First see, if PostgreSQL is working - check for errors
------------------------------------------------------------------------
We begin with looking at the list of existing databases:
g.select.pg -l
Error: select PostgreSQL:connectDB() failed: Is the postmaster running and accepting TCP/IP(with -i) connections at '130.77.22.66' on port '5432'? See for the log file:
cat /var/log/PostgreSQL.log-> No data directory -- can't proceed.
/usr/lib/pgsql/bin/postmaster does not find the database system. Expected to find it in the PGDATA directory "/var/lib/pgsql/data", but unable to open file with pathname "/var/lib/pgsql/data/base/template1/pg_class".In this case we have to install further packages (here: SuSe Linux, names may be different in your installation):
- pg_datab.rpm needs to be installed
- ps_ifa.rpm, maybe pg_iface.rpm Restart the "postmaster" (the daemon listening for db-queries):
cd /sbin/init.d/rc2.d/
./S25PostgreSQL start

ps -ax |grep postmaster

Is postmaster there? You should see it in the process list. Check again, if it working now:

g.select.pg -l
Error: select PostgreSQL: User authentication failed
You need to setup the PostgreSQL user's list. Otherwise jump to letter B in the text.

In case of error, enter:
su
Set a password for user "PostgreSQL" (this is needed first time only!):
passwd PostgreSQL
[set the password]
exit Now login as user "PostgreSQL":
su - PostgreSQL

and add yourself as PostgreSQL-user:
createuser neteler

Enter user's PostgreSQL ID or RETURN to use unix user ID: 601 -> <return>
Is user "neteler" allowed to create databases (y/n) y
Is user "neteler" allowed to add users? (y/n) n
createuser: neteler was successfully added

exit
Now you are again back in GRASS 5.0.x environment: Again we try:

g.select.pg -l
The following databases are in the Unix catalogue:
template1
This indicates that you PostgreSQL environment is o.k.
Fine. Now we can proceed and start using the interface.

------------------------------------------------------------------------
B) Using GRASS 5.0.x/PostgreSQL: creating a database and importing table
------------------------------------------------------------------------

Say, you have a SHAPE-file set: humus.shp, humus.shx and humus.dbf.
The file *.shp will be imported into GRASS 5.0.x, the file *.dbf into PostgreSQL.

Now you have to use "createdb" do create database tables. First we create an empty database
createdb humus
This new table we select in GRASS 5.0.x:
g.select.pg database=humus To destroy a database use: "destroydb humus" (PostgreSQL 6.x) or "dropdb humus" (PostgreSQL 7.x).

In this first step we import the plain attribute table only without importing geographical features.
To import the Dbase-table into PostgreSQL enter:
pg.in.dbf in=humus.dbf
Executing create table humus (AREA float4,PERIMETER float4,G2_UEB09_
int8,G2_UEB09_I int8,STONR int4,BOTYP text,HORIZ text,BODART text,HUMUS
float4,SKELETT text)
You will be asked: Additionally dump to ASCII file (enter full Unix name or hit <return> for none):
Enter "ENTER" if you don't need an additional ASCII file in your local directory.

The table is imported into PostgreSQL now. Note: The vectors are not yetr imported!

------------------------------------------------------------------------
C) Getting simple table statistics
------------------------------------------------------------------------
Now we want to get some information: For teaching real life we start with an error...
g.stats.pg table=humus col=BOTYP
Error: connect PostgreSQL:ERROR: No such function 'min' with the specified attributes The reason is that BOTYP is a text field (yes, see above in the output of pg.in.dbf!). Of course we cannot calculate statistics from letters.
So we try another field:

g.stats.pg table=humus col=HUMUS
Min, Max, Mean
-------------------------------------
2.81, 1.72372

Well, quite nice.
It tells us about humus contents of soil patches in percent.
To proceed we remove this database:

destroydb humus # on PostgreSQL 6.x
dropdb humus # on PostgreSQL 7.x

------------------------------------------------------------------------
D) Importing a SHAPE file with DBASE table (*.dbf)
------------------------------------------------------------------------
Deprecated. Use v.in.shape + pg.in.dbf

------------------------------------------------------------------------
E) Getting table information
------------------------------------------------------------------------

Here we can use
g.column.pg -v table=humus

| columnname | type | length |
----------------------------------------------------
| area | float4 4 |
| perimeter | float4 4 |
| g2_ueb09_ | int8 8 |
| g2_ueb09_i | int8 8 |
| stonr | int4 4 |
| botyp | text var |
| horiz | text var |
| bodart | text var |
| humus | float4 4 |
| skelett | text var |

As you can see you get information about the field types within the PostgreSQL table.

------------------------------------------------------------------------
F) Selecting table entries and displaying on the map
------------------------------------------------------------------------

To select map features and display the the selected areas/lines in GRASS 5.0.x Monitor, you can use
d.vect.pg A SQL-statement is required to select the features of interest.
Create an ASCII file "sql.query" (using an text editor) with contents:
select stonr from humus where HUMUS > 1.2

[change "humus" to your table name and "HUMUS" to a column existing in your table]

Now query:
d.vect.pg -f -s sql.query map=humus color=red

Alternate method: Write query into command line:

d.vect.pg -f key=HUMUS tab=humus where='HUMUS>1.2' map=humus col=red

Executing
SELECT Distinct HUMUS from humus where HUMUS>1.2 and HUMUS is not null;
16 Rows

key is the column name, tab the table, and where the statement.
"-f" fills the selected areas, col defines their colors.

Now you see the selected areas in the GRASS 5.0.x Monitor.

------------------------------------------------------------------------
G) Query example
------------------------------------------------------------------------

There's a map of digitized forest stands called "terney_id" (say three hundred in the region); each plot has unique rec_id in database table info_terney.
Now we'd like to calculate correlations between hights and diameters of trees in plots with main species Pinus koraensis (type_id <21), with age more than 100 years, growing in southern slopes and lying along specific routes.
(Remember: If you forgot the column names of your table, use g.column.pg - see above)

First thing we highlight red all plots that satisfy these conditions. We write a sql-statement ASCII file "query1.sql":

select rec_id from info_terney where type_id <21 and age > 100 and expo ~ 's' Then we use the query command:

d.vect.pg -f -s query1.sql map=terney_id color=red
Then, we pick these plots in d.what.v.pg, following along our routes:

d.what.v.pg -f map=terney_id tab=info_terney col=rec_id color=green fillcolor=gray hv=h

Recommendation: Use the TclTkGRASS with PostgreSQL support. Then you can easily copy-paste results.
After this module is done, we Control-C the results from tcltkgrass output window and paste them to spreadsheet.
Now we can calculate correlations (and other stats).

------------------------------------------------------------------------
H) Import of ARC ungenerate files
------------------------------------------------------------------------
Deprecated. Use v.in.[shape|arc] + pg.in.dbf

------------------------------------------------------------------------
I) Reclass of vector map
------------------------------------------------------------------------

1. Reclass to vector map of quartiles from forest stands map (kuruma_id).

v.reclass.pg -s -d sql=reclass.sql input=kuruma_id output=kuruma_quart type=area

and reclass.sql is:

select rec_id, quartnum from info_kuruma

2. Reclass to vector map of forest types (kuruma_oak) from map of forest plots (kuruma_id) taking only oak (types 32-37).
v.reclass.pg -d kuruma_id key=rec_id col=type_id tab=info_kuruma where='type_id > 31 and type_id < 38' output=kuruma_oak type=area


------------------------------------------------------------------------
J) Problems importing DBF-tables into PostgreSQL?
------------------------------------------------------------------------
The most common problem ( as i run into it too often) while converting *.dbf files to PostgreSQL with pg.in.dbf is format dismatch - pg_atoi() ERROR - saying there's something in the field declared as int (or float) that does not seem like number, such as "***", "NO", "infrared spectrum", etc.

Alas, PostgreSQL is very restrictive (unlike we people who type in this stuff). My approach to this (as i still want to import the .shp from Arcview that my collegue sent me from ArcviewLand) - exit the program - it would already have done so-, and rerun the module saying "no" to dbf-to-PostgreSQL dump.okay, after we have imported the coverage , let's use pg.in.dbf and say "yes" to question "Do you want additionally DUMP to ASCII?".

Now, find the line(s) that spoils your breakfast (hopethere are not many), kill'em all and then - use psql and COPY TABLE from '/home/user/user.stuff'. That's it. Besides, dumping to ascii file and doing things about it before importing to PostgreSQL is a must when there are some weird encoding chars in text fields.

--alex
-------------------------

Please send tutorial improvements to
Markus Neteler <neteler@itc.it>