doc/src/sgml/monitoring.sgml

   1 <!-- $PostgreSQL$ -->
   2
   3 <chapter id="monitoring">
   4  <title>Monitoring Database Activity</title>
   5
   6  <indexterm zone="monitoring">
   7   <primary>monitoring</primary>
   8   <secondary>database activity</secondary>
   9  </indexterm>
  10
  11  <indexterm zone="monitoring">
  12   <primary>database activity</primary>
  13   <secondary>monitoring</secondary>
  14  </indexterm>
  15
  16  <para>
  17   A database administrator frequently wonders, <quote>What is the system
  18   doing right now?</quote>
  19   This chapter discusses how to find that out.
  20  </para>
  21
  22   <para>
  23    Several tools are available for monitoring database activity and
  24    analyzing performance.  Most of this chapter is devoted to describing
  25    <productname>PostgreSQL</productname>'s statistics collector,
  26    but one should not neglect regular Unix monitoring programs such as
  27    <command>ps</>, <command>top</>, <command>iostat</>, and <command>vmstat</>.
  28    Also, once one has identified a
  29    poorly-performing query, further investigation might be needed using
  30    <productname>PostgreSQL</productname>'s <xref linkend="sql-explain"
  31    endterm="sql-explain-title"> command.
  32    <xref linkend="using-explain"> discusses <command>EXPLAIN</>
  33    and other methods for understanding the behavior of an individual
  34    query.
  35   </para>
  36
  37  <sect1 id="monitoring-ps">
  38   <title>Standard Unix Tools</Title>
  39
  40   <indexterm zone="monitoring-ps">
  41    <primary>ps</primary>
  42    <secondary>to monitor activity</secondary>
  43   </indexterm>
  44
  45   <para>
  46    On most platforms, <productname>PostgreSQL</productname> modifies its
  47    command title as reported by <command>ps</>, so that individual server
  48    processes can readily be identified.  A sample display is
  49
  50 <screen>
  51 $ ps auxww | grep ^postgres
  52 postgres   960  0.0  1.1  6104 1480 pts/1    SN   13:17   0:00 postgres -i
  53 postgres   963  0.0  1.1  7084 1472 pts/1    SN   13:17   0:00 postgres: writer process
  54 postgres   965  0.0  1.1  6152 1512 pts/1    SN   13:17   0:00 postgres: stats collector process
  55 postgres   998  0.0  2.3  6532 2992 pts/1    SN   13:18   0:00 postgres: tgl runbug 127.0.0.1 idle
  56 postgres  1003  0.0  2.4  6532 3128 pts/1    SN   13:19   0:00 postgres: tgl regression [local] SELECT waiting
  57 postgres  1016  0.1  2.4  6532 3080 pts/1    SN   13:19   0:00 postgres: tgl regression [local] idle in transaction
  58 </screen>
  59
  60    (The appropriate invocation of <command>ps</> varies across different
  61    platforms, as do the details of what is shown.  This example is from a
  62    recent Linux system.)  The first process listed here is the
  63    master server process.  The command arguments
  64    shown for it are the same ones given when it was launched.  The next two
  65    processes are background worker processes automatically launched by the
  66    master process.  (The <quote>stats collector</> process will not be present
  67    if you have set
  68    the system not to start the statistics collector.)  Each of the remaining
  69    processes is a server process handling one client connection.  Each such
  70    process sets its command line display in the form
  71
  72 <screen>
  73 postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <replaceable>activity</>
  74 </screen>
  75
  76   The user, database, and connection source host items remain the same for
  77   the life of the client connection, but the activity indicator changes.
  78   The activity can be <literal>idle</> (i.e., waiting for a client command),
  79   <literal>idle in transaction</> (waiting for client inside a <command>BEGIN</> block),
  80   or a command type name such as <literal>SELECT</>.  Also,
  81   <literal>waiting</> is attached if the server process is presently waiting
  82   on a lock held by another server process.  In the above example we can infer
  83   that process 1003 is waiting for process 1016 to complete its transaction and
  84   thereby release some lock or other.
  85   </para>
  86
  87   <para>
  88    If you have turned off <xref linkend="guc-update-process-title"> then the
  89    activity indicator is not updated; the process title is set only once
  90    when a new process is launched.  On some platforms this saves a useful
  91    amount of per-command overhead, on others it's insignificant.
  92   </para>
  93
  94   <tip>
  95   <para>
  96   <productname>Solaris</productname> requires special handling. You must
  97   use <command>/usr/ucb/ps</command>, rather than
  98   <command>/bin/ps</command>. You also must use two <option>w</option>
  99   flags, not just one. In addition, your original invocation of the
 100   <command>postgres</command> command must have a shorter
 101   <command>ps</command> status display than that provided by each
 102   server process.  If you fail to do all three things, the <command>ps</>
 103   output for each server process will be the original <command>postgres</>
 104   command line.
 105   </para>
 106   </tip>
 107  </sect1>
 108
 109  <sect1 id="monitoring-stats">
 110   <title>The Statistics Collector</Title>
 111
 112   <indexterm zone="monitoring-stats">
 113    <primary>statistics</primary>
 114   </indexterm>
 115
 116   <para>
 117    <productname>PostgreSQL</productname>'s <firstterm>statistics collector</>
 118    is a subsystem that supports collection and reporting of information about
 119    server activity.  Presently, the collector can count accesses to tables
 120    and indexes in both disk-block and individual-row terms.  It also tracks
 121    total numbers of rows in each table, and the last vacuum and analyze times
 122    for each table.  It can also count calls to user-defined functions and
 123    the total time spent in each one.
 124   </para>
 125
 126   <para>
 127    <productname>PostgreSQL</productname> also supports determining the exact
 128    command currently being executed by other server processes.  This is an
 129    independent facility that does not depend on the collector process.
 130   </para>
 131
 132  <sect2 id="monitoring-stats-setup">
 133   <title>Statistics Collection Configuration</Title>
 134
 135   <para>
 136    Since collection of statistics adds some overhead to query execution,
 137    the system can be configured to collect or not collect information.
 138    This is controlled by configuration parameters that are normally set in
 139    <filename>postgresql.conf</>.  (See <xref linkend="runtime-config"> for
 140    details about setting configuration parameters.)
 141   </para>
 142
 143   <para>
 144    The parameter <xref linkend="guc-track-counts"> controls whether
 145    statistics are collected about table and index accesses.
 146   </para>
 147
 148   <para>
 149    The parameter <xref linkend="guc-track-functions"> enables tracking of
 150    usage of user-defined functions.
 151   </para>
 152
 153   <para>
 154    The parameter <xref linkend="guc-track-activities"> enables monitoring
 155    of the current command being executed by any server process.
 156   </para>
 157
 158   <para>
 159    Normally these parameters are set in <filename>postgresql.conf</> so
 160    that they apply to all server processes, but it is possible to turn
 161    them on or off in individual sessions using the <xref
 162    linkend="sql-set" endterm="sql-set-title"> command. (To prevent
 163    ordinary users from hiding their activity from the administrator,
 164    only superusers are allowed to change these parameters with
 165    <command>SET</>.)
 166   </para>
 167
 168   <para>
 169    The statistics collector communicates with the backends needing
 170    information (including autovacuum) through temporary files.
 171    These files are stored in the <filename>pg_stat_tmp</filename> subdirectory.
 172    When the postmaster shuts down, a permanent copy of the statistics
 173    data is stored in the <filename>global</filename> subdirectory. For increased
 174    performance, the parameter <xref linkend="guc-stats-temp-directory"> can
 175    be pointed at a RAM based filesystem, decreasing physical I/O requirements.
 176   </para>
 177
 178  </sect2>
 179
 180  <sect2 id="monitoring-stats-views">
 181   <title>Viewing Collected Statistics</Title>
 182
 183   <para>
 184    Several predefined views, listed in <xref
 185    linkend="monitoring-stats-views-table">, are available to show the results
 186    of statistics collection.  Alternatively, one can
 187    build custom views using the underlying statistics functions.
 188   </para>
 189
 190   <para>
 191    When using the statistics to monitor current activity, it is important
 192    to realize that the information does not update instantaneously.
 193    Each individual server process transmits new statistical counts to
 194    the collector just before going idle; so a query or transaction still in
 195    progress does not affect the displayed totals.  Also, the collector itself
 196    emits a new report at most once per <varname>PGSTAT_STAT_INTERVAL</varname>
 197    milliseconds (500 unless altered while building the server).  So the
 198    displayed information lags behind actual activity.  However, current-query
 199    information collected by <varname>track_activities</varname> is
 200    always up-to-date.
 201   </para>
 202
 203   <para>
 204    Another important point is that when a server process is asked to display
 205    any of these statistics, it first fetches the most recent report emitted by
 206    the collector process and then continues to use this snapshot for all
 207    statistical views and functions until the end of its current transaction.
 208    So the statistics will appear not to change as long as you continue the
 209    current transaction.  Similarly, information about the current queries of
 210    all processes is collected when any such information is first requested
 211    within a transaction, and the same information will be displayed throughout
 212    the transaction.
 213    This is a feature, not a bug, because it allows you to perform several
 214    queries on the statistics and correlate the results without worrying that
 215    the numbers are changing underneath you.  But if you want to see new
 216    results with each query, be sure to do the queries outside any transaction
 217    block.  Alternatively, you can invoke
 218    <function>pg_stat_clear_snapshot</function>(), which will discard the
 219    current transaction's statistics snapshot (if any).  The next use of
 220    statistical information will cause a new snapshot to be fetched.
 221   </para>
 222
 223   <table id="monitoring-stats-views-table">
 224    <title>Standard Statistics Views</title>
 225
 226    <tgroup cols="2">
 227     <thead>
 228      <row>
 229       <entry>View Name</entry>
 230       <entry>Description</entry>
 231      </row>
 232     </thead>
 233
 234     <tbody>
 235      <row>
 236       <entry><structname>pg_stat_activity</></entry>
 237       <entry>One row per server process, showing database OID, database
 238       name, process <acronym>ID</>, user OID, user name, current query,
 239       query's waiting status, time at which the current transaction and
 240       current query began execution, time at which the process was
 241       started, and client's address and port number.  The columns that
 242       report data on the current query are available unless the parameter
 243       <varname>track_activities</varname> has been turned off.
 244       Furthermore, these columns are only visible if the user examining
 245       the view is a superuser or the same as the user owning the process
 246       being reported on.
 247      </entry>
 248      </row>
 249
 250      <row>
 251       <entry><structname>pg_stat_bgwriter</></entry>
 252       <entry>One row only, showing cluster-wide statistics from the
 253       background writer: number of scheduled checkpoints, requested
 254       checkpoints, buffers written by checkpoints and cleaning scans,
 255       and the number of times the background writer stopped a cleaning scan
 256       because it had written too many buffers.  Also includes
 257       statistics about the shared buffer pool, including buffers written
 258       by backends (that is, not by the background writer) and total buffers
 259       allocated.
 260      </entry>
 261      </row>
 262
 263      <row>
 264       <entry><structname>pg_stat_database</></entry>
 265       <entry>One row per database, showing database OID, database name,
 266       number of active server processes connected to that database,
 267       number of transactions committed and rolled back in that database,
 268       total disk blocks read, total buffer hits (i.e., block
 269       read requests avoided by finding the block already in buffer cache),
 270       number of rows returned, fetched, inserted, updated and deleted.
 271      </entry>
 272      </row>
 273
 274      <row>
 275       <entry><structname>pg_stat_all_tables</></entry>
 276       <entry>For each table in the current database (including TOAST tables),
 277       the table OID, schema and table name, number of sequential
 278       scans initiated, number of live rows fetched by sequential
 279       scans, number of index scans initiated (over all indexes
 280       belonging to the table), number of live rows fetched by index
 281       scans, numbers of row insertions, updates, and deletions,
 282       number of row updates that were HOT (i.e., no separate index update),
 283       numbers of live and dead rows,
 284       the last time the table was vacuumed manually,
 285       the last time it was vacuumed by the autovacuum daemon,
 286       the last time it was analyzed manually,
 287       and the last time it was analyzed by the autovacuum daemon.
 288       </entry>
 289      </row>
 290
 291      <row>
 292       <entry><structname>pg_stat_sys_tables</></entry>
 293       <entry>Same as <structname>pg_stat_all_tables</>, except that only
 294       system tables are shown.</entry>
 295      </row>
 296
 297      <row>
 298       <entry><structname>pg_stat_user_tables</></entry>
 299       <entry>Same as <structname>pg_stat_all_tables</>, except that only user
 300       tables are shown.</entry>
 301      </row>
 302
 303      <row>
 304       <entry><structname>pg_stat_all_indexes</></entry>
 305       <entry>For each index in the current database,
 306       the table and index OID, schema, table and index name,
 307       number of index scans initiated on that index, number of
 308       index entries returned by index scans, and number of live table rows
 309       fetched by simple index scans using that index.
 310       </entry>
 311      </row>
 312
 313      <row>
 314       <entry><structname>pg_stat_sys_indexes</></entry>
 315       <entry>Same as <structname>pg_stat_all_indexes</>, except that only
 316       indexes on system tables are shown.</entry>
 317      </row>
 318
 319      <row>
 320       <entry><structname>pg_stat_user_indexes</></entry>
 321       <entry>Same as <structname>pg_stat_all_indexes</>, except that only
 322       indexes on user tables are shown.</entry>
 323      </row>
 324
 325      <row>
 326       <entry><structname>pg_statio_all_tables</></entry>
 327       <entry>For each table in the current database (including TOAST tables),
 328       the table OID, schema and table name, number of disk
 329       blocks read from that table, number of buffer hits, numbers of
 330       disk blocks read and buffer hits in all indexes of that table,
 331       numbers of disk blocks read and buffer hits from that table's
 332       auxiliary TOAST table (if any), and numbers of disk blocks read
 333       and buffer hits for the TOAST table's index.
 334       </entry>
 335      </row>
 336
 337      <row>
 338       <entry><structname>pg_statio_sys_tables</></entry>
 339       <entry>Same as <structname>pg_statio_all_tables</>, except that only
 340       system tables are shown.</entry>
 341      </row>
 342
 343      <row>
 344       <entry><structname>pg_statio_user_tables</></entry>
 345       <entry>Same as <structname>pg_statio_all_tables</>, except that only
 346       user tables are shown.</entry>
 347      </row>
 348
 349      <row>
 350       <entry><structname>pg_statio_all_indexes</></entry>
 351       <entry>For each index in the current database,
 352       the table and index OID, schema, table and index name,
 353       numbers of disk blocks read and buffer hits in that index.
 354       </entry>
 355      </row>
 356
 357      <row>
 358       <entry><structname>pg_statio_sys_indexes</></entry>
 359       <entry>Same as <structname>pg_statio_all_indexes</>, except that only
 360       indexes on system tables are shown.</entry>
 361      </row>
 362
 363      <row>
 364       <entry><structname>pg_statio_user_indexes</></entry>
 365       <entry>Same as <structname>pg_statio_all_indexes</>, except that only
 366       indexes on user tables are shown.</entry>
 367      </row>
 368
 369      <row>
 370       <entry><structname>pg_statio_all_sequences</></entry>
 371       <entry>For each sequence object in the current database,
 372       the sequence OID, schema and sequence name,
 373       numbers of disk blocks read and buffer hits in that sequence.
 374       </entry>
 375      </row>
 376
 377      <row>
 378       <entry><structname>pg_statio_sys_sequences</></entry>
 379       <entry>Same as <structname>pg_statio_all_sequences</>, except that only
 380       system sequences are shown.  (Presently, no system sequences are defined,
 381       so this view is always empty.)</entry>
 382      </row>
 383
 384      <row>
 385       <entry><structname>pg_statio_user_sequences</></entry>
 386       <entry>Same as <structname>pg_statio_all_sequences</>, except that only
 387       user sequences are shown.</entry>
 388      </row>
 389
 390      <row>
 391       <entry><structname>pg_stat_user_functions</></entry>
 392       <entry>For all tracked functions, function OID, schema, name, number
 393       of calls, total time, and self time.  Self time is the
 394       amount of time spent in the function itself, total time includes the
 395       time spent in functions it called. Time values are in milliseconds.
 396      </entry>
 397      </row>
 398
 399     </tbody>
 400    </tgroup>
 401   </table>
 402
 403   <para>
 404    The per-index statistics are particularly useful to determine which
 405    indexes are being used and how effective they are.
 406   </para>
 407
 408   <para>
 409    Beginning in <productname>PostgreSQL</productname> 8.1, indexes can be
 410    used either directly or via <quote>bitmap scans</>.  In a bitmap scan
 411    the output of several indexes can be combined via AND or OR rules;
 412    so it is difficult to associate individual heap row fetches
 413    with specific indexes when a bitmap scan is used.  Therefore, a bitmap
 414    scan increments the
 415    <structname>pg_stat_all_indexes</>.<structfield>idx_tup_read</>
 416    count(s) for the index(es) it uses, and it increments the
 417    <structname>pg_stat_all_tables</>.<structfield>idx_tup_fetch</>
 418    count for the table, but it does not affect
 419    <structname>pg_stat_all_indexes</>.<structfield>idx_tup_fetch</>.
 420   </para>
 421
 422   <note>
 423    <para>
 424     Before <productname>PostgreSQL</productname> 8.1, the
 425     <structfield>idx_tup_read</> and <structfield>idx_tup_fetch</> counts
 426     were essentially always equal.  Now they can be different even without
 427     considering bitmap scans, because <structfield>idx_tup_read</> counts
 428     index entries retrieved from the index while <structfield>idx_tup_fetch</>
 429     counts live rows fetched from the table; the latter will be less if any
 430     dead or not-yet-committed rows are fetched using the index.
 431    </para>
 432   </note>
 433
 434   <para>
 435    The <structname>pg_statio_</> views are primarily useful to
 436    determine the effectiveness of the buffer cache.  When the number
 437    of actual disk reads is much smaller than the number of buffer
 438    hits, then the cache is satisfying most read requests without
 439    invoking a kernel call. However, these statistics do not give the
 440    entire story: due to the way in which <productname>PostgreSQL</>
 441    handles disk I/O, data that is not in the
 442    <productname>PostgreSQL</> buffer cache might still reside in the
 443    kernel's I/O cache, and might therefore still be fetched without
 444    requiring a physical read. Users interested in obtaining more
 445    detailed information on <productname>PostgreSQL</> I/O behavior are
 446    advised to use the <productname>PostgreSQL</> statistics collector
 447    in combination with operating system utilities that allow insight
 448    into the kernel's handling of I/O.
 449   </para>
 450
 451   <para>
 452    Other ways of looking at the statistics can be set up by writing
 453    queries that use the same underlying statistics access functions as
 454    these standard views do.  These functions are listed in <xref
 455    linkend="monitoring-stats-funcs-table">.  The per-database access
 456    functions take a database OID as argument to identify which
 457    database to report on.  The per-table and per-index functions take
 458    a table or index OID.  The functions for function-call statistics
 459    take a function OID.  (Note that only tables, indexes, and functions
 460    in the current database can be seen with these functions.)  The
 461    per-server-process access functions take a server process
 462    number, which ranges from one to the number of currently active
 463    server processes.
 464   </para>
 465
 466   <table id="monitoring-stats-funcs-table">
 467    <title>Statistics Access Functions</title>
 468
 469    <tgroup cols="3">
 470     <thead>
 471      <row>
 472       <entry>Function</entry>
 473       <entry>Return Type</entry>
 474       <entry>Description</entry>
 475      </row>
 476     </thead>
 477
 478     <tbody>
 479      <row>
 480       <entry><literal><function>pg_stat_get_db_numbackends</function>(<type>oid</type>)</literal></entry>
 481       <entry><type>integer</type></entry>
 482       <entry>
 483        Number of active server processes for database
 484       </entry>
 485      </row>
 486
 487      <row>
 488       <entry><literal><function>pg_stat_get_db_xact_commit</function>(<type>oid</type>)</literal></entry>
 489       <entry><type>bigint</type></entry>
 490       <entry>
 491        Transactions committed in database
 492       </entry>
 493      </row>
 494
 495      <row>
 496       <entry><literal><function>pg_stat_get_db_xact_rollback</function>(<type>oid</type>)</literal></entry>
 497       <entry><type>bigint</type></entry>
 498       <entry>
 499        Transactions rolled back in database
 500       </entry>
 501      </row>
 502
 503      <row>
 504       <entry><literal><function>pg_stat_get_db_blocks_fetched</function>(<type>oid</type>)</literal></entry>
 505       <entry><type>bigint</type></entry>
 506       <entry>
 507        Number of disk block fetch requests for database
 508       </entry>
 509      </row>
 510
 511      <row>
 512       <entry><literal><function>pg_stat_get_db_blocks_hit</function>(<type>oid</type>)</literal></entry>
 513       <entry><type>bigint</type></entry>
 514       <entry>
 515        Number of disk block fetch requests found in cache for database
 516       </entry>
 517      </row>
 518
 519      <row>
 520       <entry><literal><function>pg_stat_get_db_tuples_returned</function>(<type>oid</type>)</literal></entry>
 521       <entry><type>bigint</type></entry>
 522       <entry>
 523        Number of tuples returned for database
 524       </entry>
 525      </row>
 526
 527      <row>
 528       <entry><literal><function>pg_stat_get_db_tuples_fetched</function>(<type>oid</type>)</literal></entry>
 529       <entry><type>bigint</type></entry>
 530       <entry>
 531        Number of tuples fetched for database
 532       </entry>
 533      </row>
 534
 535      <row>
 536       <entry><literal><function>pg_stat_get_db_tuples_inserted</function>(<type>oid</type>)</literal></entry>
 537       <entry><type>bigint</type></entry>
 538       <entry>
 539        Number of tuples inserted in database
 540       </entry>
 541      </row>
 542
 543      <row>
 544       <entry><literal><function>pg_stat_get_db_tuples_updated</function>(<type>oid</type>)</literal></entry>
 545       <entry><type>bigint</type></entry>
 546       <entry>
 547        Number of tuples updated in database
 548       </entry>
 549      </row>
 550
 551      <row>
 552       <entry><literal><function>pg_stat_get_db_tuples_deleted</function>(<type>oid</type>)</literal></entry>
 553       <entry><type>bigint</type></entry>
 554       <entry>
 555        Number of tuples deleted in database
 556       </entry>
 557      </row>
 558
 559      <row>
 560       <entry><literal><function>pg_stat_get_numscans</function>(<type>oid</type>)</literal></entry>
 561       <entry><type>bigint</type></entry>
 562       <entry>
 563        Number of sequential scans done when argument is a table,
 564        or number of index scans done when argument is an index
 565       </entry>
 566      </row>
 567
 568      <row>
 569       <entry><literal><function>pg_stat_get_tuples_returned</function>(<type>oid</type>)</literal></entry>
 570       <entry><type>bigint</type></entry>
 571       <entry>
 572        Number of rows read by sequential scans when argument is a table,
 573        or number of index entries returned when argument is an index
 574       </entry>
 575      </row>
 576
 577      <row>
 578       <entry><literal><function>pg_stat_get_tuples_fetched</function>(<type>oid</type>)</literal></entry>
 579       <entry><type>bigint</type></entry>
 580       <entry>
 581        Number of table rows fetched by bitmap scans when argument is a table,
 582        or table rows fetched by simple index scans using the index
 583        when argument is an index
 584       </entry>
 585      </row>
 586
 587      <row>
 588       <entry><literal><function>pg_stat_get_tuples_inserted</function>(<type>oid</type>)</literal></entry>
 589       <entry><type>bigint</type></entry>
 590       <entry>
 591        Number of rows inserted into table
 592       </entry>
 593      </row>
 594
 595      <row>
 596       <entry><literal><function>pg_stat_get_tuples_updated</function>(<type>oid</type>)</literal></entry>
 597       <entry><type>bigint</type></entry>
 598       <entry>
 599        Number of rows updated in table (includes HOT updates)
 600       </entry>
 601      </row>
 602
 603      <row>
 604       <entry><literal><function>pg_stat_get_tuples_deleted</function>(<type>oid</type>)</literal></entry>
 605       <entry><type>bigint</type></entry>
 606       <entry>
 607        Number of rows deleted from table
 608       </entry>
 609      </row>
 610
 611      <row>
 612       <entry><literal><function>pg_stat_get_tuples_hot_updated</function>(<type>oid</type>)</literal></entry>
 613       <entry><type>bigint</type></entry>
 614       <entry>
 615        Number of rows HOT-updated in table
 616       </entry>
 617      </row>
 618
 619      <row>
 620       <entry><literal><function>pg_stat_get_live_tuples</function>(<type>oid</type>)</literal></entry>
 621       <entry><type>bigint</type></entry>
 622       <entry>
 623        Number of live rows in table
 624       </entry>
 625      </row>
 626
 627      <row>
 628       <entry><literal><function>pg_stat_get_dead_tuples</function>(<type>oid</type>)</literal></entry>
 629       <entry><type>bigint</type></entry>
 630       <entry>
 631        Number of dead rows in table
 632       </entry>
 633      </row>
 634
 635      <row>
 636       <entry><literal><function>pg_stat_get_blocks_fetched</function>(<type>oid</type>)</literal></entry>
 637       <entry><type>bigint</type></entry>
 638       <entry>
 639        Number of disk block fetch requests for table or index
 640       </entry>
 641      </row>
 642
 643      <row>
 644       <entry><literal><function>pg_stat_get_blocks_hit</function>(<type>oid</type>)</literal></entry>
 645       <entry><type>bigint</type></entry>
 646       <entry>
 647        Number of disk block requests found in cache for table or index
 648       </entry>
 649      </row>
 650
 651      <row>
 652       <entry><literal><function>pg_stat_get_last_vacuum_time</function>(<type>oid</type>)</literal></entry>
 653       <entry><type>timestamptz</type></entry>
 654       <entry>
 655        Time of the last vacuum initiated by the user on this table
 656       </entry>
 657      </row>
 658
 659      <row>
 660       <entry><literal><function>pg_stat_get_last_autovacuum_time</function>(<type>oid</type>)</literal></entry>
 661       <entry><type>timestamptz</type></entry>
 662       <entry>
 663        Time of the last vacuum initiated by the autovacuum daemon on this table
 664       </entry>
 665      </row>
 666
 667      <row>
 668       <entry><literal><function>pg_stat_get_last_analyze_time</function>(<type>oid</type>)</literal></entry>
 669       <entry><type>timestamptz</type></entry>
 670       <entry>
 671        Time of the last analyze initiated by the user on this table
 672       </entry>
 673      </row>
 674
 675      <row>
 676       <entry><literal><function>pg_stat_get_last_autoanalyze_time</function>(<type>oid</type>)</literal></entry>
 677       <entry><type>timestamptz</type></entry>
 678       <entry>
 679        Time of the last analyze initiated by the autovacuum daemon on this
 680        table
 681       </entry>
 682      </row>
 683
 684      <row>
 685        <!-- See also the entry for this in func.sgml -->
 686       <entry><literal><function>pg_backend_pid</function>()</literal></entry>
 687       <entry><type>integer</type></entry>
 688       <entry>
 689        Process ID of the server process attached to the current session
 690       </entry>
 691      </row>
 692
 693      <row>
 694       <entry><literal><function>pg_stat_get_activity</function>(<type>integer</type>)</literal></entry>
 695       <entry><type>setof record</type></entry>
 696       <entry>
 697        Returns a record of information about the backend with the specified pid, or
 698        one record for each active backend in the system if <symbol>NULL</symbol> is
 699        specified. The fields returned are the same as in the
 700        <structname>pg_stat_activity</structname> view
 701       </entry>
 702      </row>
 703
 704      <row>
 705       <entry><literal><function>pg_stat_get_function_calls</function>(<type>oid</type>)</literal></entry>
 706       <entry><type>bigint</type></entry>
 707       <entry>
 708        Number of times the function has been called.
 709       </entry>
 710      </row>
 711
 712      <row>
 713       <entry><literal><function>pg_stat_get_function_time</function>(<type>oid</type>)</literal></entry>
 714       <entry><type>bigint</type></entry>
 715       <entry>
 716        Total wall clock time spent in the function, in microseconds.  Includes
 717        the time spent in functions called by this one.
 718       </entry>
 719      </row>
 720
 721      <row>
 722       <entry><literal><function>pg_stat_get_function_self_time</function>(<type>oid</type>)</literal></entry>
 723       <entry><type>bigint</type></entry>
 724       <entry>
 725        Time spent in only this function. Time spent in called functions
 726        is excluded.
 727       </entry>
 728      </row>
 729
 730      <row>
 731       <entry><literal><function>pg_stat_get_backend_idset</function>()</literal></entry>
 732       <entry><type>setof integer</type></entry>
 733       <entry>
 734        Set of currently active server process numbers (from 1 to the
 735        number of active server processes).  See usage example in the text
 736       </entry>
 737      </row>
 738
 739      <row>
 740       <entry><literal><function>pg_stat_get_backend_pid</function>(<type>integer</type>)</literal></entry>
 741       <entry><type>integer</type></entry>
 742       <entry>
 743        Process ID of the given server process
 744       </entry>
 745      </row>
 746
 747      <row>
 748       <entry><literal><function>pg_stat_get_backend_dbid</function>(<type>integer</type>)</literal></entry>
 749       <entry><type>oid</type></entry>
 750       <entry>
 751        Database ID of the given server process
 752       </entry>
 753      </row>
 754
 755      <row>
 756       <entry><literal><function>pg_stat_get_backend_userid</function>(<type>integer</type>)</literal></entry>
 757       <entry><type>oid</type></entry>
 758       <entry>
 759        User ID of the given server process
 760       </entry>
 761      </row>
 762
 763      <row>
 764       <entry><literal><function>pg_stat_get_backend_activity</function>(<type>integer</type>)</literal></entry>
 765       <entry><type>text</type></entry>
 766       <entry>
 767        Active command of the given server process, but only if the
 768        current user is a superuser or the same user as that of
 769        the session being queried (and
 770        <varname>track_activities</varname> is on)
 771       </entry>
 772      </row>
 773
 774      <row>
 775       <entry><literal><function>pg_stat_get_backend_waiting</function>(<type>integer</type>)</literal></entry>
 776       <entry><type>boolean</type></entry>
 777       <entry>
 778        True if the given server process is waiting for a lock,
 779        but only if the current user is a superuser or the same user as that of
 780        the session being queried (and
 781        <varname>track_activities</varname> is on)
 782       </entry>
 783      </row>
 784
 785      <row>
 786       <entry><literal><function>pg_stat_get_backend_activity_start</function>(<type>integer</type>)</literal></entry>
 787       <entry><type>timestamp with time zone</type></entry>
 788       <entry>
 789        The time at which the given server process' currently
 790        executing query was started, but only if the
 791        current user is a superuser or the same user as that of
 792        the session being queried (and
 793        <varname>track_activities</varname> is on)
 794       </entry>
 795      </row>
 796
 797      <row>
 798       <entry><literal><function>pg_stat_get_backend_xact_start</function>(<type>integer</type>)</literal></entry>
 799       <entry><type>timestamp with time zone</type></entry>
 800       <entry>
 801        The time at which the given server process' currently
 802        executing transaction was started, but only if the
 803        current user is a superuser or the same user as that of
 804        the session being queried (and
 805        <varname>track_activities</varname> is on)
 806       </entry>
 807      </row>
 808
 809      <row>
 810       <entry><literal><function>pg_stat_get_backend_start</function>(<type>integer</type>)</literal></entry>
 811       <entry><type>timestamp with time zone</type></entry>
 812       <entry>
 813        The time at which the given server process was started, or
 814        null if the current user is not a superuser nor the same user
 815        as that of the session being queried
 816       </entry>
 817      </row>
 818
 819      <row>
 820       <entry><literal><function>pg_stat_get_backend_client_addr</function>(<type>integer</type>)</literal></entry>
 821       <entry><type>inet</type></entry>
 822       <entry>
 823        The IP address of the client connected to the given
 824        server process. Null if the connection is over a Unix domain
 825        socket. Also null if the current user is not a superuser nor
 826        the same user as that of the session being queried
 827       </entry>
 828      </row>
 829
 830      <row>
 831       <entry><literal><function>pg_stat_get_backend_client_port</function>(<type>integer</type>)</literal></entry>
 832       <entry><type>integer</type></entry>
 833       <entry>
 834        The IP port number of the client connected to the given
 835        server process.  -1 if the connection is over a Unix domain
 836        socket. Null if the current user is not a superuser nor the
 837        same user as that of the session being queried
 838       </entry>
 839      </row>
 840
 841      <row>
 842       <entry><literal><function>pg_stat_get_bgwriter_timed_checkpoints</function>()</literal></entry>
 843        <entry><type>bigint</type></entry>
 844        <entry>
 845         The number of times the background writer has started timed checkpoints
 846         (because the <varname>checkpoint_timeout</varname> time has expired)
 847        </entry>
 848      </row>
 849
 850      <row>
 851       <entry><literal><function>pg_stat_get_bgwriter_requested_checkpoints</function>()</literal></entry>
 852       <entry><type>bigint</type></entry>
 853       <entry>
 854        The number of times the background writer has started checkpoints based
 855        on requests from backends because the <varname>checkpoint_segments</varname>
 856        has been exceeded or because the <command>CHECKPOINT</command>
 857        command has been issued
 858       </entry>
 859      </row>
 860
 861      <row>
 862       <entry><literal><function>pg_stat_get_bgwriter_buf_written_checkpoints</function>()</literal></entry>
 863       <entry><type>bigint</type></entry>
 864       <entry>
 865        The number of buffers written by the background writer during checkpoints
 866       </entry>
 867      </row>
 868
 869      <row>
 870       <entry><literal><function>pg_stat_get_bgwriter_buf_written_clean</function>()</literal></entry>
 871       <entry><type>bigint</type></entry>
 872       <entry>
 873        The number of buffers written by the background writer for routine cleaning of
 874        dirty pages
 875       </entry>
 876      </row>
 877
 878      <row>
 879       <entry><literal><function>pg_stat_get_bgwriter_maxwritten_clean</function>()</literal></entry>
 880       <entry><type>bigint</type></entry>
 881       <entry>
 882        The number of times the background writer has stopped its cleaning scan because
 883        it has written more buffers than specified in the
 884        <varname>bgwriter_lru_maxpages</varname> parameter
 885       </entry>
 886      </row>
 887
 888      <row>
 889       <entry><literal><function>pg_stat_get_buf_written_backend</function>()</literal></entry>
 890       <entry><type>bigint</type></entry>
 891       <entry>
 892        The number of buffers written by backends because they needed
 893        to allocate a new buffer
 894       </entry>
 895      </row>
 896
 897      <row>
 898       <entry><literal><function>pg_stat_get_buf_alloc</function>()</literal></entry>
 899       <entry><type>bigint</type></entry>
 900       <entry>
 901        The total number of buffer allocations
 902       </entry>
 903      </row>
 904
 905      <row>
 906       <entry><literal><function>pg_stat_clear_snapshot</function>()</literal></entry>
 907       <entry><type>void</type></entry>
 908       <entry>
 909        Discard the current statistics snapshot
 910       </entry>
 911      </row>
 912
 913      <row>
 914       <entry><literal><function>pg_stat_reset</function>()</literal></entry>
 915       <entry><type>void</type></entry>
 916       <entry>
 917        Reset all statistics counters for the current database to zero
 918        (requires superuser privileges)
 919       </entry>
 920      </row>
 921     </tbody>
 922    </tgroup>
 923   </table>
 924
 925    <note>
 926     <para>
 927      <function>pg_stat_get_blocks_fetched</function> minus
 928      <function>pg_stat_get_blocks_hit</function> gives the number of kernel
 929      <function>read()</> calls issued for the table, index, or
 930      database; the number of actual physical reads is usually
 931      lower due to kernel-level buffering.  The <literal>*_blks_read</>
 932      statistics columns use this subtraction, i.e., fetched minus hit.
 933     </para>
 934    </note>
 935
 936   <para>
 937    All functions to access information about backends are indexed by backend id
 938    number, except <function>pg_stat_get_activity</function> which is indexed by PID.
 939    The function <function>pg_stat_get_backend_idset</function> provides
 940    a convenient way to generate one row for each active server process.  For
 941    example, to show the <acronym>PID</>s and current queries of all server processes:
 942
 943 <programlisting>
 944 SELECT pg_stat_get_backend_pid(s.backendid) AS procpid,
 945        pg_stat_get_backend_activity(s.backendid) AS current_query
 946     FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s;
 947 </programlisting>
 948   </para>
 949
 950  </sect2>
 951  </sect1>
 952
 953  <sect1 id="monitoring-locks">
 954   <title>Viewing Locks</title>
 955
 956   <indexterm zone="monitoring-locks">
 957    <primary>lock</primary>
 958    <secondary>monitoring</secondary>
 959   </indexterm>
 960
 961   <para>
 962    Another useful tool for monitoring database activity is the
 963    <structname>pg_locks</structname> system table.  It allows the
 964    database administrator to view information about the outstanding
 965    locks in the lock manager. For example, this capability can be used
 966    to:
 967
 968    <itemizedlist>
 969     <listitem>
 970      <para>
 971       View all the locks currently outstanding, all the locks on
 972       relations in a particular database, all the locks on a
 973       particular relation, or all the locks held by a particular
 974       <productname>PostgreSQL</productname> session.
 975      </para>
 976     </listitem>
 977
 978     <listitem>
 979      <para>
 980       Determine the relation in the current database with the most
 981       ungranted locks (which might be a source of contention among
 982       database clients).
 983      </para>
 984     </listitem>
 985
 986     <listitem>
 987      <para>
 988       Determine the effect of lock contention on overall database
 989       performance, as well as the extent to which contention varies
 990       with overall database traffic.
 991      </para>
 992     </listitem>
 993    </itemizedlist>
 994
 995    Details of the <structname>pg_locks</structname> view appear in
 996    <xref linkend="view-pg-locks">.
 997    For more information on locking and managing concurrency with
 998    <productname>PostgreSQL</productname>, refer to <xref linkend="mvcc">.
 999   </para>
1000  </sect1>
1001
1002  <sect1 id="dynamic-trace">
1003   <title>Dynamic Tracing</title>
1004
1005  <indexterm zone="dynamic-trace">
1006   <primary>DTrace</primary>
1007  </indexterm>
1008
1009   <para>
1010    <productname>PostgreSQL</productname> provides facilities to support
1011    dynamic tracing of the database server. This allows an external
1012    utility to be called at specific points in the code and thereby trace
1013    execution.
1014   </para>
1015
1016   <para>
1017    A number of probes or trace points are already inserted into the source
1018    code. These probes are intended to be used by database developers and
1019    administrators. By default the probes are not compiled into
1020    <productname>PostgreSQL</productname>; the user needs to explicitly tell
1021    the configure script to make the probes available.
1022   </para>
1023
1024   <para>
1025    Currently, only the
1026    <ulink url="http://opensolaris.org/os/community/dtrace/">DTrace</ulink>
1027    utility is supported, which is available
1028    on OpenSolaris, Solaris 10, and Mac OS X Leopard. It is expected that
1029    DTrace will be available in the future on FreeBSD and possibly other
1030    operating systems.  The
1031    <ulink url="http://sourceware.org/systemtap/">SystemTap</ulink> project
1032    for Linux also provides a DTrace equivalent.  Supporting other dynamic
1033    tracing utilities is theoretically possible by changing the definitions for
1034    the macros in <filename>src/include/utils/probes.h</>.
1035   </para>
1036
1037   <sect2 id="compiling-for-trace">
1038    <title>Compiling for Dynamic Tracing</title>
1039
1040   <para>
1041    By default, probes are not available, so you will need to
1042    explicitly tell the configure script to make the probes available
1043    in <productname>PostgreSQL</productname>. To include DTrace support
1044    specify <option>--enable-dtrace</> to configure.  See <xref
1045    linkend="install-procedure"> for further information.
1046   </para>
1047   </sect2>
1048
1049   <sect2 id="trace-points">
1050    <title>Built-in Probes</title>
1051
1052   <para>
1053    A number of standard probes are provided in the source code,
1054    as shown in <xref linkend="dtrace-probe-point-table">.
1055    More can certainly be added to enhance <productname>PostgreSQL</>'s
1056    observability.
1057   </para>
1058
1059  <table id="dtrace-probe-point-table">
1060   <title>Built-in DTrace Probes</title>
1061   <tgroup cols="3">
1062    <thead>
1063     <row>
1064      <entry>Name</entry>
1065      <entry>Parameters</entry>
1066      <entry>Description</entry>
1067     </row>
1068    </thead>
1069
1070    <tbody>
1071
1072     <row>
1073      <entry>transaction-start</entry>
1074      <entry>(LocalTransactionId)</entry>
1075      <entry>Probe that fires at the start of a new transaction.
1076       arg0 is the transaction id.</entry>
1077     </row>
1078     <row>
1079      <entry>transaction-commit</entry>
1080      <entry>(LocalTransactionId)</entry>
1081      <entry>Probe that fires when a transaction completes successfully.
1082       arg0 is the transaction id.</entry>
1083     </row>
1084     <row>
1085      <entry>transaction-abort</entry>
1086      <entry>(LocalTransactionId)</entry>
1087      <entry>Probe that fires when a transaction completes unsuccessfully.
1088       arg0 is the transaction id.</entry>
1089     </row>
1090     <row>
1091      <entry>query-start</entry>
1092      <entry>(const char *)</entry>
1093      <entry>Probe that fires when the processing of a query is started.
1094       arg0 is the query string.</entry>
1095     </row>
1096     <row>
1097      <entry>query-done</entry>
1098      <entry>(const char *)</entry>
1099      <entry>Probe that fires when the processing of a query is complete.
1100       arg0 is the query string.</entry>
1101     </row>
1102     <row>
1103      <entry>query-parse-start</entry>
1104      <entry>(const char *)</entry>
1105      <entry>Probe that fires when the parsing of a query is started.
1106       arg0 is the query string.</entry>
1107     </row>
1108     <row>
1109      <entry>query-parse-done</entry>
1110      <entry>(const char *)</entry>
1111      <entry>Probe that fires when the parsing of a query is complete.
1112       arg0 is the query string.</entry>
1113     </row>
1114     <row>
1115      <entry>query-rewrite-start</entry>
1116      <entry>(const char *)</entry>
1117      <entry>Probe that fires when the rewriting of a query is started.
1118       arg0 is the query string.</entry>
1119     </row>
1120     <row>
1121      <entry>query-rewrite-done</entry>
1122      <entry>(const char *)</entry>
1123      <entry>Probe that fires when the rewriting of a query is complete.
1124       arg0 is the query string.</entry>
1125     </row>
1126     <row>
1127      <entry>query-plan-start</entry>
1128      <entry>()</entry>
1129      <entry>Probe that fires when the planning of a query is started.</entry>
1130     </row>
1131     <row>
1132      <entry>query-plan-done</entry>
1133      <entry>()</entry>
1134      <entry>Probe that fires when the planning of a query is complete.</entry>
1135     </row>
1136     <row>
1137      <entry>query-execute-start</entry>
1138      <entry>()</entry>
1139      <entry>Probe that fires when the execution of a query is started.</entry>
1140     </row>
1141     <row>
1142      <entry>query-execute-done</entry>
1143      <entry>()</entry>
1144      <entry>Probe that fires when the execution of a query is complete.</entry>
1145     </row>
1146     <row>
1147      <entry>statement-status</entry>
1148      <entry>(const char *)</entry>
1149      <entry>Probe that fires anytime the server process updates its
1150       <structname>pg_stat_activity</>.<structfield>current_query</> status.
1151       arg0 is the new status string.</entry>
1152     </row>
1153     <row>
1154      <entry>checkpoint-start</entry>
1155      <entry>(int)</entry>
1156      <entry>Probe that fires when a checkpoint is started.
1157       arg0 holds the bitwise flags used to distinguish different checkpoint
1158       types, such as shutdown, immediate or force.</entry>
1159     </row>
1160     <row>
1161      <entry>checkpoint-done</entry>
1162      <entry>(int, int, int, int, int)</entry>
1163      <entry>Probe that fires when a checkpoint is complete.
1164       (The probes listed next fire in sequence during checkpoint processing.)
1165       arg0 is the number of buffers written. arg1 is the total number of
1166       buffers. arg2, arg3 and arg4 contain the number of xlog file(s) added,
1167       removed and recycled respectively.</entry>
1168     </row>
1169     <row>
1170      <entry>clog-checkpoint-start</entry>
1171      <entry>(bool)</entry>
1172      <entry>Probe that fires when the CLOG portion of a checkpoint is started.
1173       arg0 is true for normal checkpoint, false for shutdown
1174       checkpoint.</entry>
1175     </row>
1176     <row>
1177      <entry>clog-checkpoint-done</entry>
1178      <entry>(bool)</entry>
1179      <entry>Probe that fires when the CLOG portion of a checkpoint is
1180       complete. arg0 has the same meaning as for clog-checkpoint-start.</entry>
1181     </row>
1182     <row>
1183      <entry>subtrans-checkpoint-start</entry>
1184      <entry>(bool)</entry>
1185      <entry>Probe that fires when the SUBTRANS portion of a checkpoint is
1186       started.
1187       arg0 is true for normal checkpoint, false for shutdown
1188       checkpoint.</entry>
1189     </row>
1190     <row>
1191      <entry>subtrans-checkpoint-done</entry>
1192      <entry>(bool)</entry>
1193      <entry>Probe that fires when the SUBTRANS portion of a checkpoint is
1194       complete. arg0 has the same meaning as for
1195       subtrans-checkpoint-start.</entry>
1196     </row>
1197     <row>
1198      <entry>multixact-checkpoint-start</entry>
1199      <entry>(bool)</entry>
1200      <entry>Probe that fires when the MultiXact portion of a checkpoint is
1201       started.
1202       arg0 is true for normal checkpoint, false for shutdown
1203       checkpoint.</entry>
1204     </row>
1205     <row>
1206      <entry>multixact-checkpoint-done</entry>
1207      <entry>(bool)</entry>
1208      <entry>Probe that fires when the MultiXact portion of a checkpoint is
1209       complete. arg0 has the same meaning as for
1210       multixact-checkpoint-start.</entry>
1211     </row>
1212     <row>
1213      <entry>buffer-checkpoint-start</entry>
1214      <entry>(int)</entry>
1215      <entry>Probe that fires when the buffer-writing portion of a checkpoint
1216       is started.
1217       arg0 holds the bitwise flags used to distinguish different checkpoint
1218       types, such as shutdown, immediate or force.</entry>
1219     </row>
1220     <row>
1221      <entry>buffer-sync-start</entry>
1222      <entry>(int, int)</entry>
1223      <entry>Probe that fires when we begin to write dirty buffers during
1224       checkpoint (after identifying which buffers must be written).
1225       arg0 is the total number of buffers.
1226       arg1 is the number that are currently dirty and need to be written.</entry>
1227     </row>
1228     <row>
1229      <entry>buffer-sync-written</entry>
1230      <entry>(int)</entry>
1231      <entry>Probe that fires after each buffer is written during checkpoint.
1232       arg0 is the ID number of the buffer.</entry>
1233     </row>
1234     <row>
1235      <entry>buffer-sync-done</entry>
1236      <entry>(int, int, int)</entry>
1237      <entry>Probe that fires when all dirty buffers have been written.
1238       arg0 is the total number of buffers.
1239       arg1 is the number of buffers actually written by the checkpoint process.
1240       arg2 is the number that were expected to be written (arg1 of
1241       buffer-sync-start); any difference reflects other processes flushing
1242       buffers during the checkpoint.</entry>
1243     </row>
1244     <row>
1245      <entry>buffer-checkpoint-sync-start</entry>
1246      <entry>()</entry>
1247      <entry>Probe that fires after dirty buffers have been written to the
1248       kernel, and before starting to issue fsync requests.</entry>
1249     </row>
1250     <row>
1251      <entry>buffer-checkpoint-done</entry>
1252      <entry>()</entry>
1253      <entry>Probe that fires when syncing of buffers to disk is
1254       complete.</entry>
1255     </row>
1256     <row>
1257      <entry>twophase-checkpoint-start</entry>
1258      <entry>()</entry>
1259      <entry>Probe that fires when the two-phase portion of a checkpoint is
1260       started.</entry>
1261     </row>
1262     <row>
1263      <entry>twophase-checkpoint-done</entry>
1264      <entry>()</entry>
1265      <entry>Probe that fires when the two-phase portion of a checkpoint is
1266       complete.</entry>
1267     </row>
1268     <row>
1269      <entry>buffer-read-start</entry>
1270      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid, bool, bool)</entry>
1271      <entry>Probe that fires when a buffer read is started.
1272       arg0 and arg1 contain the fork and block numbers of the page (but
1273       arg1 will be -1 if this is a relation extension request).
1274       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1275       identifying the relation.
1276       arg5 is true for a local buffer, false for a shared buffer.
1277       arg6 is true for a relation extension request, false for normal
1278       read.</entry>
1279     </row>
1280     <row>
1281      <entry>buffer-read-done</entry>
1282      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid, bool, bool, bool)</entry>
1283      <entry>Probe that fires when a buffer read is complete.
1284       arg0 and arg1 contain the fork and block numbers of the page (if this
1285       is a relation extension request, arg1 now contains the block number
1286       of the newly added block).
1287       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1288       identifying the relation.
1289       arg5 is true for a local buffer, false for a shared buffer.
1290       arg6 is true for a relation extension request, false for normal
1291       read.
1292       arg7 is true if the buffer was found in the pool, false if not.</entry>
1293     </row>
1294     <row>
1295      <entry>buffer-flush-start</entry>
1296      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid)</entry>
1297      <entry>Probe that fires before issuing any write request for a shared
1298       buffer.
1299       arg0 and arg1 contain the fork and block numbers of the page.
1300       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1301       identifying the relation.</entry>
1302     </row>
1303     <row>
1304      <entry>buffer-flush-done</entry>
1305      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid)</entry>
1306      <entry>Probe that fires when a write request is complete.  (Note
1307       that this just reflects the time to pass the data to the kernel;
1308       it's typically not actually been written to disk yet.)
1309       The arguments are the same as for buffer-flush-start.</entry>
1310     </row>
1311     <row>
1312      <entry>buffer-write-dirty-start</entry>
1313      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid)</entry>
1314      <entry>Probe that fires when a server process begins to write a dirty
1315       buffer.  (If this happens often, it implies that
1316       <xref linkend="guc-shared-buffers"> is too
1317       small or the bgwriter control parameters need adjustment.)
1318       arg0 and arg1 contain the fork and block numbers of the page.
1319       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1320       identifying the relation.</entry>
1321     </row>
1322     <row>
1323      <entry>buffer-write-dirty-done</entry>
1324      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid)</entry>
1325      <entry>Probe that fires when a dirty-buffer write is complete.
1326       The arguments are the same as for buffer-write-dirty-start.</entry>
1327     </row>
1328     <row>
1329      <entry>wal-buffer-write-dirty-start</entry>
1330      <entry>()</entry>
1331      <entry>Probe that fires when when a server process begins to write a
1332       dirty WAL buffer because no more WAL buffer space is available.
1333       (If this happens often, it implies that
1334       <xref linkend="guc-wal-buffers"> is too small.)</entry>
1335     </row>
1336     <row>
1337      <entry>wal-buffer-write-dirty-done</entry>
1338      <entry>()</entry>
1339      <entry>Probe that fires when a dirty WAL buffer write is complete.</entry>
1340     </row>
1341     <row>
1342      <entry>xlog-insert</entry>
1343      <entry>(unsigned char, unsigned char)</entry>
1344      <entry>Probe that fires when a WAL record is inserted.
1345       arg0 is the resource manager (rmid) for the record.
1346       arg1 contains the info flags.</entry>
1347     </row>
1348     <row>
1349      <entry>xlog-switch</entry>
1350      <entry>()</entry>
1351      <entry>Probe that fires when a WAL segment switch is requested.</entry>
1352     </row>
1353     <row>
1354      <entry>smgr-md-read-start</entry>
1355      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid)</entry>
1356      <entry>Probe that fires when beginning to read a block from a relation.
1357       arg0 and arg1 contain the fork and block numbers of the page.
1358       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1359       identifying the relation.</entry>
1360     </row>
1361     <row>
1362      <entry>smgr-md-read-done</entry>
1363      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid, int, int)</entry>
1364      <entry>Probe that fires when a block read is complete.
1365       arg0 and arg1 contain the fork and block numbers of the page.
1366       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1367       identifying the relation.
1368       arg5 is the number of bytes actually read, while arg6 is the number
1369       requested (if these are different it indicates trouble).</entry>
1370     </row>
1371     <row>
1372      <entry>smgr-md-write-start</entry>
1373      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid)</entry>
1374      <entry>Probe that fires when beginning to write a block to a relation.
1375       arg0 and arg1 contain the fork and block numbers of the page.
1376       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1377       identifying the relation.</entry>
1378     </row>
1379     <row>
1380      <entry>smgr-md-write-done</entry>
1381      <entry>(ForkNumber, BlockNumber, Oid, Oid, Oid, int, int)</entry>
1382      <entry>Probe that fires when a block write is complete.
1383       arg0 and arg1 contain the fork and block numbers of the page.
1384       arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs
1385       identifying the relation.
1386       arg5 is the number of bytes actually written, while arg6 is the number
1387       requested (if these are different it indicates trouble).</entry>
1388     </row>
1389     <row>
1390      <entry>sort-start</entry>
1391      <entry>(int, bool, int, int, bool)</entry>
1392      <entry>Probe that fires when a sort operation is started.
1393       arg0 indicates heap, index or datum sort.
1394       arg1 is true for unique-value enforcement.
1395       arg2 is the number of key columns.
1396       arg3 is the number of kilobytes of work memory allowed.
1397       arg4 is true if random access to the sort result is required.</entry>
1398     </row>
1399     <row>
1400      <entry>sort-done</entry>
1401      <entry>(bool, long)</entry>
1402      <entry>Probe that fires when a sort is complete.
1403       arg0 is true for external sort, false for internal sort.
1404       arg1 is the number of disk blocks used for an external sort,
1405       or kilobytes of memory used for an internal sort.</entry>
1406     </row>
1407     <row>
1408      <entry>lwlock-acquire</entry>
1409      <entry>(LWLockId, LWLockMode)</entry>
1410      <entry>Probe that fires when an LWLock has been acquired.
1411       arg0 is the LWLock's ID.
1412       arg1 is the requested lock mode, either exclusive or shared.</entry>
1413     </row>
1414     <row>
1415      <entry>lwlock-release</entry>
1416      <entry>(LWLockId)</entry>
1417      <entry>Probe that fires when an LWLock has been released (but note
1418       that any released waiters have not yet been awakened).
1419       arg0 is the LWLock's ID.</entry>
1420     </row>
1421     <row>
1422      <entry>lwlock-wait-start</entry>
1423      <entry>(LWLockId, LWLockMode)</entry>
1424      <entry>Probe that fires when an LWLock was not immediately available and
1425       a server process has begun to wait for the lock to become available.
1426       arg0 is the LWLock's ID.
1427       arg1 is the requested lock mode, either exclusive or shared.</entry>
1428     </row>
1429     <row>
1430      <entry>lwlock-wait-done</entry>
1431      <entry>(LWLockId, LWLockMode)</entry>
1432      <entry>Probe that fires when a server process has been released from its
1433       wait for an LWLock (it does not actually have the lock yet).
1434       arg0 is the LWLock's ID.
1435       arg1 is the requested lock mode, either exclusive or shared.</entry>
1436     </row>
1437     <row>
1438      <entry>lwlock-condacquire</entry>
1439      <entry>(LWLockId, LWLockMode)</entry>
1440      <entry>Probe that fires when an LWLock was successfully acquired when the
1441       caller specified no waiting.
1442       arg0 is the LWLock's ID.
1443       arg1 is the requested lock mode, either exclusive or shared.</entry>
1444     </row>
1445     <row>
1446      <entry>lwlock-condacquire-fail</entry>
1447      <entry>(LWLockId, LWLockMode)</entry>
1448      <entry>Probe that fires when an LWLock was not successfully acquired when
1449       the caller specified no waiting.
1450       arg0 is the LWLock's ID.
1451       arg1 is the requested lock mode, either exclusive or shared.</entry>
1452     </row>
1453     <row>
1454      <entry>lock-wait-start</entry>
1455      <entry>(unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, LOCKMODE)</entry>
1456      <entry>Probe that fires when a request for a heavyweight lock (lmgr lock)
1457       has begun to wait because the lock is not available.
1458       arg0 through arg3 are the tag fields identifying the object being
1459       locked.  arg4 indicates the type of object being locked.
1460       arg5 indicates the lock type being requested.</entry>
1461     </row>
1462     <row>
1463      <entry>lock-wait-done</entry>
1464      <entry>(unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, LOCKMODE)</entry>
1465      <entry>Probe that fires when a request for a heavyweight lock (lmgr lock)
1466       has finished waiting (i.e., has acquired the lock).
1467       The arguments are the same as for lock-wait-start.</entry>
1468     </row>
1469     <row>
1470      <entry>deadlock-found</entry>
1471      <entry>()</entry>
1472      <entry>Probe that fires when a deadlock is found by the deadlock
1473       detector.</entry>
1474     </row>
1475
1476    </tbody>
1477    </tgroup>
1478   </table>
1479
1480  <table id="typedefs-table">
1481   <title>Defined Types Used in Probe Parameters</title>
1482   <tgroup cols="2">
1483    <thead>
1484     <row>
1485      <entry>Type</entry>
1486      <entry>Definition</entry>
1487     </row>
1488    </thead>
1489
1490    <tbody>
1491
1492     <row>
1493      <entry>LocalTransactionId</entry>
1494      <entry>unsigned int</entry>
1495     </row>
1496     <row>
1497      <entry>LWLockId</entry>
1498      <entry>int</entry>
1499     </row>
1500     <row>
1501      <entry>LWLockMode</entry>
1502      <entry>int</entry>
1503     </row>
1504     <row>
1505      <entry>LOCKMODE</entry>
1506      <entry>int</entry>
1507     </row>
1508     <row>
1509      <entry>BlockNumber</entry>
1510      <entry>unsigned int</entry>
1511     </row>
1512     <row>
1513      <entry>Oid</entry>
1514      <entry>unsigned int</entry>
1515     </row>
1516     <row>
1517      <entry>ForkNumber</entry>
1518      <entry>int</entry>
1519     </row>
1520     <row>
1521      <entry>bool</entry>
1522      <entry>char</entry>
1523     </row>
1524
1525    </tbody>
1526    </tgroup>
1527   </table>
1528
1529
1530   </sect2>
1531
1532   <sect2 id="using-trace-points">
1533    <title>Using Probes</title>
1534
1535   <para>
1536    The example below shows a DTrace script for analyzing transaction
1537    counts in the system, as an alternative to snapshotting
1538    <structname>pg_stat_database</> before and after a performance test:
1539 <programlisting>
1540 #!/usr/sbin/dtrace -qs
1541
1542 postgresql$1:::transaction-start
1543 {
1544       @start["Start"] = count();
1545       self->ts  = timestamp;
1546 }
1547
1548 postgresql$1:::transaction-abort
1549 {
1550       @abort["Abort"] = count();
1551 }
1552
1553 postgresql$1:::transaction-commit
1554 /self->ts/
1555 {
1556       @commit["Commit"] = count();
1557       @time["Total time (ns)"] = sum(timestamp - self->ts);
1558       self->ts=0;
1559 }
1560 </programlisting>
1561    When executed, the example D script gives output such as:
1562 <screen>
1563 # ./txn_count.d `pgrep -n postgres` or ./txn_count.d &lt;PID&gt;
1564 ^C
1565
1566 Start                                          71
1567 Commit                                         70
1568 Total time (ns)                        2312105013
1569 </screen>
1570   </para>
1571   <para>
1572    You should remember that DTrace scripts need to be carefully written and
1573    debugged, otherwise the trace information collected might
1574    be meaningless. In most cases where problems are found it is the
1575    instrumentation that is at fault, not the underlying system. When
1576    discussing information found using dynamic tracing, be sure to enclose
1577    the script used to allow that too to be checked and discussed.
1578   </para>
1579   <para>
1580    More example scripts can be found in the PgFoundry
1581    <ulink url="http://pgfoundry.org/projects/dtrace/">dtrace project</ulink>.
1582   </para>
1583   </sect2>
1584
1585   <sect2 id="defining-trace-points">
1586    <title>Defining New Probes</title>
1587
1588   <para>
1589    New probes can be defined within the code wherever the developer
1590    desires, though this will require a recompilation. Below are the steps
1591    for inserting new probes:
1592   </para>
1593
1594   <procedure>
1595    <step>
1596     <para>
1597      Decide on probe names and data to be made available through the probes
1598     </para>
1599    </step>
1600
1601    <step>
1602     <para>
1603      Add the probe definitions to <filename>src/backend/utils/probes.d</>
1604     </para>
1605    </step>
1606
1607    <step>
1608     <para>
1609      Include <filename>pg_trace.h</> if it is not already present in the
1610      module(s) containing the probe points, and insert
1611      <literal>TRACE_POSTGRESQL</> probe macros at the desired locations
1612      in the source code
1613     </para>
1614    </step>
1615
1616    <step>
1617     <para>
1618      Recompile and verify that the new probes are available
1619     </para>
1620    </step>
1621   </procedure>
1622
1623   <formalpara>
1624    <title>Example:</title>
1625    <para>
1626     Here is an example of how you would add a probe to trace all new
1627     transactions by transaction ID.
1628    </para>
1629   </formalpara>
1630
1631   <procedure>
1632    <step>
1633     <para>
1634      Decide that the probe will be named <literal>transaction-start</> and
1635      requires a parameter of type LocalTransactionId
1636     </para>
1637    </step>
1638
1639    <step>
1640     <para>
1641      Add the probe definition to <filename>src/backend/utils/probes.d</>:
1642 <programlisting>
1643       probe transaction__start(LocalTransactionId);
1644 </programlisting>
1645      Note the use of the double underline in the probe name. In a DTrace
1646      script using the probe, the double underline needs to be replaced with a
1647      hyphen, so <literal>transaction-start</> is the name to document for
1648      users.
1649     </para>
1650    </step>
1651
1652    <step>
1653     <para>
1654      At compile time, <literal>transaction__start</> is converted to a macro
1655      called <literal>TRACE_POSTGRESQL_TRANSACTION_START</> (notice the
1656      underscores are single here), which is available by including
1657      <filename>pg_trace.h</>.  Add the macro call to the appropriate location
1658      in the source code.  In this case, it looks like the following:
1659
1660 <programlisting>
1661     TRACE_POSTGRESQL_TRANSACTION_START(vxid.localTransactionId);
1662 </programlisting>
1663     </para>
1664    </step>
1665
1666    <step>
1667     <para>
1668      After recompiling and running the new binary, check that your newly added
1669      probe is available by executing the following DTrace command.  You
1670      should see similar output:
1671 <screen>
1672 # dtrace -ln transaction-start
1673    ID    PROVIDER          MODULE           FUNCTION NAME
1674 18705 postgresql49878     postgres     StartTransactionCommand transaction-start
1675 18755 postgresql49877     postgres     StartTransactionCommand transaction-start
1676 18805 postgresql49876     postgres     StartTransactionCommand transaction-start
1677 18855 postgresql49875     postgres     StartTransactionCommand transaction-start
1678 18986 postgresql49873     postgres     StartTransactionCommand transaction-start
1679 </screen>
1680     </para>
1681    </step>
1682   </procedure>
1683
1684   <para>
1685    There are a few things to be careful about when adding trace macros
1686    to the C code:
1687
1688    <itemizedlist>
1689     <listitem>
1690      <para>
1691       You should take care that the data types specified for a probe's
1692       parameters match the data types of the variables used in the macro.
1693       Otherwise, you will get compilation errors.
1694      </para>
1695     </listitem>
1696
1697
1698     <listitem>
1699      <para>
1700       On most platforms, if <productname>PostgreSQL</productname> is
1701       built with <option>--enable-dtrace</>, the arguments to a trace
1702       macro will be evaluated whenever control passes through the
1703       macro, <emphasis>even if no tracing is being done</>.  This is
1704       usually not worth worrying about if you are just reporting the
1705       values of a few local variables.  But beware of putting expensive
1706       function calls into the arguments.  If you need to do that,
1707       consider protecting the macro with a check to see if the trace
1708       is actually enabled:
1709
1710 <programlisting>
1711     if (TRACE_POSTGRESQL_TRANSACTION_START_ENABLED())
1712         TRACE_POSTGRESQL_TRANSACTION_START(some_function(...));
1713 </programlisting>
1714
1715       Each trace macro has a corresponding <literal>ENABLED</> macro.
1716      </para>
1717     </listitem>
1718    </itemizedlist>
1719
1720   </para>
1721
1722   </sect2>
1723
1724  </sect1>
1725
1726 </chapter>