Progress since the last update
Work on zheap — PostgreSQL's proposed storage engine for handling UPDATE more efficiently — has moved forward on several fronts. The main items completed since the previous report are logical decoding support, work on UNDO, patch reviews for UNDO, code merges with current PostgreSQL versions, and a long list of fixes and improvements.
Logical decoding and the UNDO dependency
Logical decoding turns the write-ahead log back into SQL that can be replayed elsewhere to produce identical results. It does not come for free: code must exist that decodes zheap records and renders them as readable output. That implementation currently looks solid, and no bugs are known in this area.
UNDO is the other half of the design. A standard heap table keeps every version of a row inside the same physical files; zheap instead moves old row versions out of the table and restores them on ROLLBACK, in the style of Oracle and other engines. The zheap code has now been made compatible with the community's proposed UNDO infrastructure, which is expected to reach core around version 15. That infrastructure is deliberately generic rather than zheap-specific, so other storage engines can build on it later — which is why aligning zheap with it is essential.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
[hs@node1 zheap_postgres]$ cd src/backend/access/zheap/ [hs@node1 zheap]$ ls -l *c -rw-rw-r--. 1 hs hs 14864 May 27 04:25 prunetpd.c -rw-rw-r--. 1 hs hs 27935 May 27 04:25 prunezheap.c -rw-rw-r--. 1 hs hs 11394 May 27 04:25 rewritezheap.c -rw-rw-r--. 1 hs hs 96748 May 27 04:25 tpd.c -rw-rw-r--. 1 hs hs 13997 May 27 04:25 tpdxlog.c -rw-rw-r--. 1 hs hs 285703 May 27 04:25 zheapam.c -rw-rw-r--. 1 hs hs 59175 May 27 04:25 zheapam_handler.c -rw-rw-r--. 1 hs hs 62970 May 27 04:25 zheapam_visibility.c -rw-rw-r--. 1 hs hs 61636 May 27 04:25 zheapamxlog.c -rw-rw-r--. 1 hs hs 16608 May 27 04:25 zheaptoast.c -rw-rw-r--. 1 hs hs 16218 May 27 04:25 zhio.c -rw-rw-r--. 1 hs hs 21039 May 27 04:25 zmultilocker.c -rw-rw-r--. 1 hs hs 16480 May 27 04:25 zpage.c -rw-rw-r--. 1 hs hs 43128 May 27 04:25 zscan.c -rw-rw-r--. 1 hs hs 27760 May 27 04:25 ztuple.c -rw-rw-r--. 1 hs hs 55849 May 27 04:25 zundo.c -rw-rw-r--. 1 hs hs 51613 May 27 04:25 zvacuumlazy.c [hs@node1 zheap]$ cat *c | wc -l 29696 |
Bug fixing and code merges
Quality management on a project of this size is substantial work in itself. The issues addressed include bugs, compiler warnings and similar defects; the details are not worth enumerating here. The zheap code has also been merged with current PostgreSQL versions to stay in step with ongoing development. For anyone waiting on a production-ready zheap, it is worth stressing that this remains a major undertaking. The code can already be tried and tested, but it is not there yet: more time and, above all, community feedback are needed before it can handle arbitrary workloads reliably and without bugs.
What comes next
The next major item is the discard worker. UNDO chunks currently do not disappear on their own — they keep accumulating — and they can only be purged manually at the moment. That manual cleanup works, but the goal is for it to happen automatically, which the discard worker is meant to provide.
PgSQL
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
test=# BEGIN BEGIN test=*# CREATE TABLE sample (x int) USING zheap; CREATE TABLE test=*# INSERT INTO sample SELECT * FROM generate_series(1, 1000000) AS x; INSERT 0 1000000 test=*# SELECT * FROM pg_stat_undo_chunks; logno | start | prev | size | discarded | type | type_header --------+------------------+------+----------+-----------+------+----------------------------------- 000001 | 000001000021AC3D | | 57 | f | xact | (xid=745, dboid=16384, applied=f) 000001 | 000001000021AC76 | | 44134732 | f | xact | (xid=748, dboid=16384, applied=f) (2 rows) test=*# COMMIT; COMMIT test=# SELECT * FROM pg_stat_undo_chunks; logno | start | prev | size | discarded | type | type_header --------+------------------+------+----------+-----------+------+----------------------------------- 000001 | 000001000021AC3D | | 57 | f | xact | (xid=745, dboid=16384, applied=f) 000001 | 000001000021AC76 | | 44134732 | f | xact | (xid=748, dboid=16384, applied=f) (2 rows) |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
test=# SELECT pg_advance_oldest_xid_having_undo(); pg_advance_oldest_xid_having_undo ----------------------------------- 750 (1 row) test=# SELECT * FROM pg_stat_undo_chunks; logno | start | prev | size | discarded | type | type_header -------+------------------+------+----------+-----------+------+----------------------------------- 000001 | 000001000021AC3D | | 57 | t | xact | (xid=745, dboid=16384, applied=f) 000001 | 000001000021AC76 | | 44134732 | t | xact | (xid=748, dboid=16384, applied=f) (2 rows) test=# SELECT pg_discard_undo_record_set_chunks(); pg_discard_undo_record_set_chunks ----------------------------------- (1 row) test=# SELECT * FROM pg_stat_undo_chunks; logno | start | prev | size | discarded | type | type_header ------+-------+------+------+-----------+------+------------- (0 rows) |
Community feedback is one of the current bottlenecks. Anyone interested in zheap is invited to contribute; load testing and design feedback are both welcome and appreciated. zheap matters for UPDATE-heavy workloads, and moving it forward matters too.
Getting started
A tarball containing the latest zheap code (as of May 27th, 2021) is available for download. Compile PostgreSQL normally, then create a database instance and start the server as usual. When creating a new table, add USING zheap — otherwise PostgreSQL creates a standard heap table instead.
|
1 2 3 4 |
./configure --prefix=/your_path/pg --enable-debug --with-cassert make install cd contrib make install |
Thanks go to Heroic Labs for the support that made this work possible; their commitment allowed resources to be allocated to the project, to the benefit of the whole community.
