GRASS and PostgreSQL - First Steps

[$Id: index.html,v 1.1 2000/03/10 10:57:57 markus Exp $]

Recommended reading:
  http://www.postgresql.org/docs/tutorial/index.html


General Module Overview:

        By Alex Shevlakov

Modules in this pack, in general, provide the following functions:


Bugs and other possible caveats reports are welcome.

Alex Shevlakov,
sixote@yahoo.com

==================================================================

Short introduction to GRASS/PostgreSQL interface

by
  Markus Neteler and Alex Shevlakov


The following text shall introduce you to interface usage. This text is subject to change...

Let's start.

Enter GRASS 5. Of course you need to have a location defined and some data you want to import here.

------------------------------------------------------------------------
A) First see, if PostgreSQL is working - check for errors
------------------------------------------------------------------------

We begin with looking at the list of existing databases:

g.select.pg -l


Error: select Postgres:connectDB() failed: Is the postmaster running and
accepting TCP/IP(with -i) connections at '130.77.22.66' on port '5432'?

See for the log file:

cat /var/log/postgresql.log


It tells us:

-> No data directory -- can't proceed.
/usr/lib/pgsql/bin/postmaster does not find the database system.  Expected
to find it in the PGDATA directory "/var/lib/pgsql/data", but unable to open
file with pathname "/var/lib/pgsql/data/base/template1/pg_class".
In this case we have to install further packages (here: SuSe Linux, names may be different in your installation):
   - pg_datab.rpm needs to be installed
   - ps_ifa.rpm, maybe pg_iface.rpm

Restart the "postmaster" (the daemon listening for db-queries):

cd /sbin/init.d/rc2.d/
./S25postgres start

ps -ax |grep postmaster

Is postmaster there? You should see it in the process list.
Check again, if it working now:
 
g.select.pg -l


If you get the error:

Error: select Postgres: User authentication failed
you need to setup the PostgreSQL user's list. Otherwise jump to letter B in the text.
In case of error, enter:
su
Set a password for user "postgres" (this is needed first time only!):
passwd postgres
[set the password]
exit
Now login as user "postgres":
su - postgres


and add yourself as PostgreSQL-user:

createuser neteler

Enter user's postgres ID or RETURN to use unix user ID: 601 -> <return>
Is user "neteler" allowed to create databases (y/n) y
Is user "neteler" allowed to add users? (y/n) n
createuser: neteler was successfully added

exit

Now you are again back in GRASS environment: Again we try:
 
g.select.pg -l
   The following databases are in the Unix catalogue:
    template1
This indicates that you PostgreSQL environment is o.k.
Fine. Now we can proceed and start using the interface.

------------------------------------------------------------------------
B) Using GRASS/PostgreSQL: creating a database table
------------------------------------------------------------------------

Say, you have a SHAPE-file set: humus.shp, humus.shx and humus.dbf.
The file *.shp will be imported into GRASS, the file *.dbf into PostgreSQL.

Now you have to use "createdb" do create database tables.
First we create an empty table:

createdb humus
This new table we select in GRASS:
g.select.pg database=humus
To destroy a database use: "destroydb humus" (PostgreSQL 6.x) or "dropdb humus" (PostgreSQL 7.x).

In this first step we import the plain attribute table only without importing geographical features.
To import the Dbase-table into PostgreSQL enter:

pg.in.dbf in=humus.dbf
  Executing create table humus (AREA float4,PERIMETER float4,G2_UEB09_
  int8,G2_UEB09_I int8,STONR int4,BOTYP text,HORIZ text,BODART text,HUMUS
  float4,SKELETT text)
You will be asked: Additionally dump to ASCII file (enter full Unix name or hit  <return> for none):
Enter "ENTER" if you don't need an additional ASCII file in your local directory.

The table is imported into PostgreSQL now. Note: The vectors are not yetr imported!

------------------------------------------------------------------------
C) Getting simple table statistics
------------------------------------------------------------------------
Now we want to get some information:
For teaching real life we start with an error...

g.stats.pg table=humus col=BOTYP
 Error: connect Postgres:ERROR:  No such function 'min' with the specified attributes
The reason is that BOTYP is a text field (yes, see above in the output of pg.in.dbf!). Of course we cannot calculate statistics from letters.
So we try another field:
 
g.stats.pg table=humus col=HUMUS
      Min,        Max,       Mean
-------------------------------------
        0,       2.81,    1.72372
Aha, quite nice. It tells us about humus contents of soil patches in percent.
To proceed we remove this table:
destroydb humus # on PostgreSQL 6.x
dropdb humus # on PostgreSQL 7.x
------------------------------------------------------------------------
D) Importing a SHAPE file with DBASE table (*.dbf)
------------------------------------------------------------------------

Now it is becoming more amazing! We will import our SHAPE file:
Using "verbose=9" we get information about import progress. First step is to create an empty Postgres table:

createdb humus
v.in.shape.pg input=humus.shp verbose=9 pgdump=humus
NOTE: You don't have to import the .dbf table manually. It is done by v.in.shape.pg automatically (many thanks to Alex Shevlakov)!

See if the GRASS map there:

g.list vect
----------------------------------------------
vector files available in mapset geosum:
humus
----------------------------------------------
First we have to build the map topology:
v.support m=humus o=build
Start a GRASS monitor:
d.mon x0
...and display the map:
d.vect humus
Query it using PostgreSQL:
d.what.v.pg -f m=humus tab=humus col=HUMUS

-> Now we see that v.in.shape[.pg] only supports lines
but not areas yet. We hope for an upgrade...

Interim solution: ------------------------------------------------------------------------
E) Getting table information
------------------------------------------------------------------------

Here we can use

g.column.pg -v table=humus

  | columnname              | type      | length    | 
  ---------------------------------------------------- 
  | area                    | float4    |         4 | 
  | perimeter               | float4    |         4 | 
  | g2_ueb09_               | int8      |         8 | 
  | g2_ueb09_i              | int8      |         8 | 
  | stonr                   | int4      |         4 | 
  | botyp                   | text      |       var | 
  | horiz                   | text      |       var | 
  | bodart                  | text      |       var | 
  | humus                   | float4    |         4 | 
  | skelett                 | text      |       var | 

As you can see you get information about the field types within the PostgreSQL table.

------------------------------------------------------------------------
F) Selecting table entries and displaying on the map
------------------------------------------------------------------------

To select map features and display the the selected areas/lines in GRASS Monitor, you can use

d.vect.pg
A SQL-statement is required to select the features of interest.
Create an ASCII file "sql.query"  (using an text editor) with contents:
select stonr from humus where HUMUS > 1.2


[change "humus" to your table name and "HUMUS" to a column existing
 in your table]

Now query:

d.vect.pg -f  -s sql.query map=humus color=red
Alternate method: Write query into command line:
d.vect.pg -f key=HUMUS tab=humus where='HUMUS>1.2' map=humus col=red
   Executing
   SELECT Distinct HUMUS from humus  where  HUMUS>1.2 and HUMUS is not null;
   16 Rows

 key is the column name, tab the table, and where the statement.
 "-f" fills the selected areas, col defines their colors.

 Now you see the selected areas in the GRASS Monitor.

------------------------------------------------------------------------
G) Query example 1
------------------------------------------------------------------------

There's a map of digitized forest stands called "terney_id" (say three hundred in the region); each plot has unique rec_id in database table info_terney.
Now we'd like to calculate correlations between hights and diameters of trees in plots with main species Pinus koraensis (type_id <21), with age more than 100 years, growing in southern slopes and lying along specific routes. (Remember: If you forgot the column names of your table, use g.column.pg - see above)

First thing we highlight red all plots that satisfy these conditions. We write a sql-statement ASCII file "query1.sql":

select rec_id from info_terney where type_id <21 and age > 100 and expo ~ 's'
Then we use the query command:
d.vect.pg -f -s query1.sql map=terney_id color=red
Then, we pick these plots in d.what.v.pg, following along our routes:
d.what.v.pg -f map=terney_id tab=info_terney col=rec_id color=green fillcolor=gray hv=h
Recommendation: Use the TclTkGRASS with PostgreSQL support. Then you can easily copy-paste results.
After this module is done, we Control-C the results from tcltkgrass output window and paste them to spreadsheet.
Now we can calculate correlations (and other stats).

------------------------------------------------------------------------
H) Import of ARC ungenerate files
------------------------------------------------------------------------
Note: Currently import is limited to interactive mode.

In this example we have a landuse map in ungen format (one file containing lines, one file containing label points and one file containing label text = attributes).
The files will be found in your current directory (opposite to v.in.arc which expects them in $LOCATION/arc/ !).
Copy PAT.dbf  (or AAT.dbf) there, too, and rename to a reasonable name comparing to your other ungen files.
Note: PostgreSQL does not allow field names containing "-" (ilde-id is forbidden, ilde_id is accepted)!

Our files are:

il_nutz.pnt  il_nutz.pol  il_nutz.dbf
Now we start to import:
createdb il_nutz   (only if you do not have a database created)

v.in.arc.pg
GRASS vector file: landuse            (the new name of map in GRASS)
Coverage type: polygon
Neatline: no                                    (a neatline is a line surrounding the map)
Lines file ARC/INFO: il_nutz.pol
Labels file ARCINFO: il_nutz.pnt
Name of .dbf file to be imported: il_nutz.dbf
Admin/normal user dump mode: <return>
Additionally dump to ASCII file: <return>

Table il_nutz successfully copied into Postgres. Congratulations!
 

Now run v.support on the imported vector map.
 

------------------------------------------------------------------------
I) Reclass of vector map
------------------------------------------------------------------------
v.reclass.pg
[...]


 

------------------------------------------------------------------------
J) Problems importing DBF-tables into PostgreSQL?
------------------------------------------------------------------------
The most common problem ( as i run into it too often) while converting *.dbf files to Postgres with pg.in.dbf, v.in.arc.pg and v.in.shape.pg is format dismatch - pg_atoi() ERROR - saying there's something in the field declared as int (or float) that does not seem like number, such as "***", "NO", "infrared spectrum", etc.

Alas, postgres is very restrictive (unlike we people who type in this stuff). My approach to this (as i still want to import the .shp from Arcview that my collegue sent me from ArcviewLand) - exit the program - it would already have done so-, and rerun the module saying "no" to dbf-to-postgres dump.okay, after we have imported the coverage , let's use pg.in.dbf and say "yes" to question "Do you want additionally DUMP to ASCII?".

Now, find the line(s) that spoils your breakfast (hopethere are not many), kill'em all and then - use psql and COPY TABLE from '/home/user/user.stuff'. That's it. Besides, dumping to ascii file and doing things about it before importing to postgres is a must when there are some weird encoding chars in text fields.

Alex



Please send tutorial improvements to
Markus Neteler