diff options
| author | Marko Kreen | 2012-03-09 12:21:10 +0000 |
|---|---|---|
| committer | Marko Kreen | 2012-03-17 11:46:36 +0000 |
| commit | 2623621847a276f0a4a34cb5e9b01945960910f0 (patch) | |
| tree | 80b94068a9d5fd2030dbb369eb9ccfb3d6a78922 | |
| parent | 3390a10ead354138440dfe3ec7de44193f7bb39e (diff) | |
Add howtos and clean up docs
Include howtos written by Hannu Krosing.
Clean up rest of the docs for release.
| -rw-r--r-- | INSTALL | 24 | ||||
| -rw-r--r-- | Makefile | 2 | ||||
| -rw-r--r-- | NEWS | 372 | ||||
| -rw-r--r-- | README | 47 | ||||
| -rw-r--r-- | doc/Makefile | 47 | ||||
| -rw-r--r-- | doc/TODO.txt | 41 | ||||
| -rw-r--r-- | doc/faq.txt | 91 | ||||
| -rw-r--r-- | doc/howto/londiste3_cascaded_rep_howto.txt | 259 | ||||
| -rw-r--r-- | doc/howto/londiste3_merge_howto.txt | 217 | ||||
| -rw-r--r-- | doc/howto/londiste3_partitioning_howto.txt | 210 | ||||
| -rw-r--r-- | doc/howto/londiste3_simple_rep_howto.txt | 212 | ||||
| -rw-r--r-- | doc/howto/setup_walmgr_replication.txt | 398 | ||||
| -rw-r--r-- | doc/index.txt | 47 | ||||
| -rw-r--r-- | doc/skytools3.txt | 67 |
14 files changed, 1474 insertions, 560 deletions
@@ -1,10 +1,6 @@ = SkyTools - tools for PostgreSQL = -This is a package of tools in use in Skype for replication and -failover. Also it includes a generic queuing mechanism PgQ and -utility library for Python scripts. - -== Dependencies == +== Builing == Skytools modules use some other code to run, you need to install the following dependencies on the system where you want to build and run @@ -21,9 +17,7 @@ When building code from GIT: === Runtime dependencies === - python - psycopg2 or psycopg1 - rsync + python psycopg2 rsync == Building from source tarball == @@ -33,18 +27,22 @@ When building code from GIT: == Building from GIT == - ## fetch git tree +Fetch git tree: + $ git clone git://github.com/markokr/skytools.git - ## fetch libusual submodule +Fetch libusual submodule: + $ git submodule init $ git submodule update - ## generate ./configure script +Generate ./configure script: + $ ./autogen.sh - ## now build as usual (--with-asciidoc is required when building from GIT) - $ ./configure --prefix=... --with-asciidoc +Now build as usual: + + $ ./configure --prefix=... $ make == Building a debian package == @@ -66,7 +66,7 @@ deb: debuild -uc -us -b tgz: config.mak clean - $(MAKE) -C doc man html + $(MAKE) -C doc man rm -f source.list $(PYTHON) setup_skytools.py sdist -t source.cfg -m source.list @@ -1,365 +1,9 @@ -2009-xx-xx - SkyTools 3.0 - "" - -* make python modules parallel installable with skytools 2.x: - - new pkgloader module, (ala pygtk but universal) - - installed under `skytools-3.0` top dir - - python modules import skytools via pkgloader: - - import pkgloader - pkgloader.require('skytools', '3.0') - import skytools - -* newadm: rename to qadmin -* pgq - - drop failed event concept - - lazy fetch is now default - - relaxed event processing for python modules - .tag_done() is not needed - - get_{queue|consumer}_info() now report quick stats - - ability to request big batches (10000 rows, 5 min), which combine several actual batches. - - ability to stay behind on queue. - - pgq.Consumer new config vars: pgq_batch_collect_events, pgq_batch_collect_interval, pgq_keep_lag - - pgq triggers: magic fields to set event fields directly: - `_pgq_ev_type, _pgq_ev_data, _pgq_ev_extra[1-4]` -* python/skytools: - - config templates in docstr: `./script.py --ini` shows default conf - - show connection name with error message - - use config file name as default job name, `job_name` is thus optional now - - new querybuilder class - - .getdict() method for Config -* pgqd: - - specifying fixed list of databases in config works - - periodic detection works -* cascading - - fix watermark publishing - - several log message and level fixes - - Replace 'switchover'/'failover' with 'takeover' - - 'status' with dead nodes works - - consumers last error is stored in db, showed by `londiste status` -* londiste - - resync - - missing - - --all - - globbing - - compare/repair - - apply sql one-by-one if last batch got error - - support WHERE for COPY: londiste add --copy-condition=XX - -* Simple Python consumer for pgq_coop, where each subconsumer - is separate process. - -* psycopgwrapper: make .server_version always available - -* Sleeping while waiting notices from db. It seems good to speed up - change-provider, switchover etc commands by using LISTEN/NOTIFY - to wake up daemons. - -* londiste/pgq: truncate trigger / event - - -2009-03-13 - SkyTools 2.1.9 - "Six Pack of Luck" - - = WalMgr improvements = - - * walmgr.py: WAL files purge procedure now pays attention to recovery - restart points. (%r patch by Omar Kilani, Mark Kirkwood) - - * walmgr.py: archive_mode GUC is now set during setup. Followed by an - optional restart if master_restart_cmd is specified (Mark Kirkwood) - - * walmgr.py: PostgreSQL configuration files are backed up to - "config_backup" and restored to "slave_config_dir". (Mark Kirkwood) - - * walmgr.py: Backups taken from slave now generate backup_label and - history files. - - * walmgr.py Configuration files now default to PostgreSQL 8.3 - - = Fixes = - - * londiste copy: Add missing identifier quoting for - TRUNCATE and ADD CONSTRAINT. (Luc Van Hoeylandt) - - * Quote identifiers starting with numeric. (Andrew Dunstan) - - * Fix crash with pgq.sqltrigq/pgq.logutriga, which automatically - detect table structure. Problem was handling table invalidation - event during internal query. - (Götz Lange, André Malo) - - * Fix 'uninitialized variable' warning and potentially bad code - in pgq.sqltriga(). (Götz Lange, André Malo) - - * skytools._cquoting: Fix crash with Decimal type. - (May affects users who have compiled psycopg2 to use - Decimal() instead of float for NUMERIC) - - * skytools.DBStruct: The DEFAULT keyword was missing when - creating table columns with default value. Table creation - functionality is unused in 2.1.x thus no users should be affected. - - * skytools.magic_insert: be more flexible about whats dict. - In particular, now accept DictRow. - - * configure.ac: don't detect xmlto and asciidoc version - if --with-asciidoc is not used. Otherwise unnecessary error - message was printed. - - * Add few headers for Postgres 8.4 that are not included - automatically anymore. (Devrim Gündüz) - -2008-10-12 - SkyTools 2.1.8 - "Perestroika For Your Data" - - = Fixes = - - * deb: Make debian package accept skytools-8.3 too. - * add skylog.ini as installalble file (David Fetter) - * Londiste: - - Document the fkeys, triggers, restore-triggers commands. - - Run ANALYZE after table is copied over. - - Fix "provider add-seq --all" - * pgq.SerialConsumer (used by Londiste) - fix position check, - which got confused when sequence numbers did large jump. - * PgQ database functions: - - pgq.maint_rotate_step1() function removed tick #1 - even if consumer was registered on it. - - pgq triggers: import cache invalidation from HEAD. - Now pgq.sqltriga() and pgq.logutriga() should be preferable - to pgq.logtriga(). - - uninstall_pgq.sql: Correct syntax for DROP SCHEMA CASCADE. - * skytools.DBScript, used by all Python scripts: - - Don't try to stay running if MemoryError was thrown. - - Detect stale pidfile and ignore it. - - Stop hooking SIGTERM anymore. Python signal handling and - libpq polling did not interact well. - * scriptmgr: - - Don't crash on missing pidfile. - - Fix few typos on scriptmgr (Martin Pihlak) - * walmgr (Martin Pihlak): - - improved error messages on startup if no config file specified. - - walmgr.py stop now also stops syncdaemon - - pg_auth file is copied to slave as part of archiving. restored during boot. - - cleanup in remote_walmgr() to always pass config file to slave walmgr. - - master_backup() now reports exceptions. - - = Features = - - * londiste -s/-k now kill "copy" process with SIGTERM. - Previously the process was left running. On startup - "replay" will check if "copy" process had died before - finishing and launches one if needed. (Dimitri Fontaine) - * New skytools.signal_pidfile() function. - * Londiste: New lock_timeout parameter to limit time - locks are held on provider. Applies to operations: - provider add/remove, compare, repair. - * Londiste: on psycopg2 v2.0.6+ use new .copy_expert() API, - which does not crash on tables with large number of columns. - -2008-06-03 - SkyTools 2.1.7 - "Attack of the Werebugs" - - = Fixes = - - * Fix pgq trigger compilation with Postgres 8.2 (Marcin Stępnicki) - * Replace `make` calls with $(MAKE) in Makefiles (Pierre-Emmanuel André) - * londiste: Fix incompatibility with Python 2.3 (Dimitri Fontaine) - * walmgr: Fix typo in config symlinking code. (pychecker) - * bulk_loader: Fix typo in temp table check. (pychecker) - * Install upgrade .sql files. Otherwise skytools_upgrade.py could - be used only from source directory. - * pgq.Consumer: Fix bug in retry/failed event handling. (pychecker) - * pgq: Fix pgq.maint_retry_events() - it could create double events when - amount of event to be moved back into main queue was more than 10. - - = Features = - - * Quoting of table and column names in Londiste and dispatcher scripts. - - = Upgrade procedure = - - * Database code under pgq and londiste schemas need to be upgraded. - That can be done on running databases with following command: - - $ skytools_upgrade.py "connstr" - - Or by applying 'londiste.upgrade.sql' and 'pgq.upgrade.sql' by hand. - The changes were only in functions, no table sctructure changed. - -2008-04-05 - SkyTools 2.1.6 - "Quick Bugfix Release" - - Now we have upgrade script, see 'man skytools_upgrade' - for info how to upgrade database code. - - = Fixes = - - * Include upgrade sql scripts in .tgz - * Fix 'londiste provider seqs' - * Fix 'londiste provider fkeys' in no tables added. - * Fix "londiste copy" pidfile timing race - * Fix Solaris build - avoid grep -q / define HAVE_UNSETENV - * Fix "make debXX" when several Postgres version are installed. - * New-style AC_OUTPUT usage. - * Disable manpage creation by default, --with-asciidoc to enable. - They are still included in .tgz so users should have less problems now. - * Restore iter-on-values behaviour for rows from curs.fetch*. The attempt - to make them iter-on-keys seems now misguided, as iter-on-values is already - used in existing code, and iter-on-keys returns keys in random order. - * londiste subscriber add: Dont drop triggers on table if --expect-sync is used. - * londiste copy: drop triggers and fkeys in case "replay" or "subscriber add" was skipped - * walmgr restore: better detection if old postmaster is running (Charles Duffy) - * walmgr xrestore: detect the death of parent process - * walmgr restore: create pg_tblspc - its required for 8.3 (Zoltán Böszörményi) - * walmgr restore: copy old config files over if exist (Zoltán Böszörményi) - - = Features = - - * Table name globbing for Londiste commands (Erik Jones) - * New manpages for scripts (Asko Oja & me) - * Upgrade script with manpage: scripts/skytools_upgrade.py - * Add .version() function to pgq_ext & londiste schemas. - * pgqadm: allow parameters without queue_ prefix in 'config' command. - * skytools Python module: - - intern() keys in db_urldecode() to decrease memory usage - - udp-logger: more fields: hostaddr, service_name - - udp-logger: dont cache udp socket, seem to hang in some cases - - DBScript.get_database() allows explicit connect string - - DBScript allows disabling config file loading - - magic_insert on dicts allows missing columns, uses NULL - - new parsing functions: - - parse_pgarray(), parse_logtriga_sql(), parse_tabbed_table(), - - parse_statements() - this one is used to split SQL install - files to separate statements. - - exists_function() checks both 'public' and 'pg_catalog' - for unqualified functions names. - - skytools.quoting: add C implementation for quote_literal, quote_copy, - quote_bytea_raw, unescape, db_urlencode, db_urldecode. This gives - 20x speedup for db_urlencode and 50x for db_urldecode on some - real-life data. - - unquote_ident(), unquote_literal() - -2007-11-19 - SkyTools 2.1.5 - "Enterprise-Grade Duct Tape" - - = Big changes = - - * Lot of new docs [Dimitri Fontaine, Asko Oja, Marko Kreen] - * Support for fkey and trigger handling in Londiste. [Erik Jones] - * Rewrite pgq.insert_event() and log triggers in C, thus SkyTools does - not depend on PL/Python anymore. - - = Small changes = - - * pgq+txid: convert to new API appearing in 8.3 /contrib/txid/ - * Support psycopg2, preferring it to psycopg1. - * Improved bulk_loader, using temp tables exclusively. - * skytools.config: API change to allow usage without config file. - * skytools module: quote_ident(), quote_fqident() - * install .sql files under share/skytools in addition to contrib/ - * pgqadm: also vacuums londiste and pgq_ext tables, if they exist - * londiste: provider add/remove --all [Hans-Juergen Schoenig] - * backend modules support 8.3 - * pgq: switch pgq_lazy_fetch=NROWS for pgq.Consumer, which makes it use - cursor for event fetching, thus allowing larger batches - * txid: use bsearch() for larger snapshots - - = Fixes = - - * londiste fkeys: look also at dependers not only dependencies. - * pgq.consumer: make queue seeking in case of failover more strict. - * scriptmgr: dont die on user error. - * pgq: there was still fallout from reorg - 2 missing indexes. - * Due to historical reasons SerialConsumer (and thus Londiste) - accessed completed tick table directly, not via functions. - Make it use functions again. - * londiste: set client_encoding on subscriber same as on provider - * londiste: remove tbl should work also if table is already dropped [Dimitri Fontaine] - * couple walmgr fixed [Martin Pihlak] - - = Upgrade procedure for database code = - - * PgQ (used on Londiste provider side), table structure, plpgsql functions: - - $ psql dbname -f upgrade/final/v2.1.5.pgq_core.sql - - * PgQ new insert_event(), written in C: - - $ psql dbname -f sql/pgq/lowlevel/pgq_lowlevel.sql - - * PgQ new triggers (sqltriga, logtriga, logutriga), written in C: - - $ psql dbname -f sql/pgq/triggers/pgq_triggers.sql - - * Londiste (both provider and subscriber side) - - $ psql dbname -f upgrade/final/v2.1.5.londiste.sql - - * pgq_ext: - - $ psql dbname -f upgrade/final/v2.1.5.pgq_ext.sql - - -2007-04-16 - SkyTools 2.1.4 - "Sweets from last Christmas" - - = Fixes = - - * logtriga.c was supposed to survive mismatched column string, - but the logic were buggy. Thanks go to Dmitriy V'jukov for - good analysis. - * Couple of scripts were not converted to new API. Fix it. - * Quiet a warning in textbuf.c - * Make configure and Makefiles survive on BSD's where 'make' - is not GNU make. Thanks to Hans-Juergen Schoening. - - = Features = - - * Published WalMgr was an old version. Sync with internal code, - where Martin has done lot of enhancements. - * Small incompat change in PGQ: add fields to pgq.get_consumer_info() - return type. Example upgrade procedure: - - DROP TYPE pgq.ret_consumer_info cascade; - \i structure/types.sql - \i functions/pgq.get_consumer_info.sql - - It will show some errors but thats ok. Its annoying but needed - for the tick_id cleanup in SerialConsumer/Londiste. - -2007-04-10 - SkyTools 2.1.3 - "Brown Paper Bag" - - Still managed to sneak in a last-minute typo. - - * Fix copy-paste error in table_copy.py - * Remember to bump version in pgq.version() - -2007-04-09 - SkyTools 2.1.2 - "Help screen works" - - Most fallout from reorg is hopefully cleaned now. - - * Dumb bug in ticker wich made it almost non-working, - except it managed to complete the regression test... - * Move --skip-truncate switch from 'copy' to 'londiste add'. - * 'londiste install' also installs plpgsql+plpythonu. - * SQL installer logs full path to file it found. - * Change pgq.get_queue_info() FOR loop variable to record - instead text that was reported to fail, although it did work here. - * Remember where the SQL files were installed. - -2007-04-06 - SkyTools 2.1.1 - "Needs more thrust to fly" - - SkyTools got big reorg before release, but with the hoopla - with the 3 projects at once, it did not get much testing... - There are some untested areas still, but at least pgq/londiste - are in better shape now. - - * pgqadm: finish conversion... - * londiste.Syncer: - - Convert to new API - - Avoid ticking, parallel ticks are dangerous - - Bad arg in repair - * pgq: - - too aggressive check in register_consumer - - Algo desc for batch_event_sql - * Add some knobs to make regtests for londiste pass - more predictibly. - -2007-03-13 - SkyTools 2.1 - "Radioactive Candy" - - * Final public release. +2012-xx-xx - SkyTools 3.0 - "" + + * Cascaded queues + * Londiste: Parallel copy + * Londiste: EXECUTE + * Londiste: handlers + * qadmin + * pgqd @@ -7,7 +7,7 @@ standby servers. == Overview == -It contains following tools: +It contains following modules: === PgQ === @@ -41,10 +41,10 @@ Skytools. Documentation: -- PgQ ticker daemon (pgqd) usage: link:pgqd.html[] -- PgQ admin tool (qadm) usage: link:qadmin.html[] -- PgQ SQL API overview: link:pgq-sql.html[] -- PgQ SQL reference: http://skytools.projects.postgresql.org/skytools-3.0/pgq/ +- PgQ ticker daemon (pgqd) usage: link:doc/pgqd.html[] +- PgQ admin tool (qadm) usage: link:doc/qadmin.html[] +- PgQ SQL API overview: link:doc/pgq-sql.html[] +- PgQ SQL reference: link:pgq/[] === Londiste === @@ -59,11 +59,10 @@ Features: Documentation: -- Londiste script usage: doc/londiste.cmdline.txt +- Londiste script usage: doc/londiste3.txt (also available as `man 1 londiste`) -- Londiste configuration: doc/londiste.config.txt - (also available as `man 5 londiste`) -- Londiste reference: doc/londiste.ref.txt + +- Londiste HOWTOS: doc/howto/ === walmgr === @@ -75,51 +74,51 @@ so less than whole file is lost in case of loss of master database server. == Source tree contents == - doc/ +doc/:: Documentation in asciidoc format. Source for both html and man pages. - python/ +python/:: Python modules and primary executables - walmgr, londiste, qadmin, pgqadm. - python/pgq/ +python/pgq/:: Python framework for PgQ. - python/londiste/ +python/londiste/:: Londiste replication. - python/skytools/ +python/skytools/:: Low-level utilities for writing database scripts in Python. - sql/ +sql/:: Database modules. - sql/pgq/ +sql/pgq/:: Table definitions and functions for PgQ queueing. - sql/pgq_node/ +sql/pgq_node/:: Framework for cascaded consuming. - sql/pgq_coop/ +sql/pgq_coop/:: Functions for cooperative consuming. - sql/londiste/ +sql/londiste/:: Table definitions and functions for Londiste replication. - sql/ticker/ +sql/ticker/:: PgQ ticker written in C. - scripts/ +scripts/:: Python scripts with lesser priority. - lib/ +lib/:: libusual C libary, for pgqd. - debian/ +debian/:: Debian packaging. This is for creating private packages, official Debian packages uses it's own packagin code. - misc/ +misc/:: Random scripts used for building. == Upgrade from 2.1 == diff --git a/doc/Makefile b/doc/Makefile index 6c7aaa2c..d62a7355 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -7,19 +7,24 @@ EPYDOC = epydoc EPYARGS = --no-private --url="http://pgfoundry.org/projects/skytools/" \ --name="Skytools" --html --no-private -v -HTMLS = README.html INSTALL.html \ +TOPHTML = README.html INSTALL.html index.html +DOCHTML = \ TODO.html pgq-sql.html pgq-nodupes.html \ - $(SCRIPT_HTMLS) faq.html set.notes.html skytools3.html devnotes.html + faq.html set.notes.html skytools3.html devnotes.html pgqd.html \ + londiste3.html walmgr3.html qadmin.html scriptmgr.html \ + skytools_upgrade.html queue_mover.html queue_splitter.html \ + howto/londiste3_cascaded_rep_howto.html \ + howto/londiste3_merge_howto.html \ + howto/londiste3_partitioning_howto.html \ + howto/londiste3_simple_rep_howto.html \ + howto/setup_walmgr_replication.html -SCRIPT_TXTS = londiste3.txt walmgr3.txt qadmin.txt scriptmgr.txt skytools_upgrade.txt \ - queue_mover.txt queue_splitter.txt -SCRIPT_HTMLS = $(SCRIPT_TXTS:.txt=.html) MAN5 = MAN1_SFX = scriptmgr.1 skytools_upgrade.1 queue_mover.1 queue_splitter.1 MAN1 = qadmin.1 pgqd.1 walmgr3.1 londiste3.1 -FQHTML = $(addprefix html/doc/, $(HTMLS)) +FQHTML = $(addprefix html/doc/, $(DOCHTML)) $(addprefix html/, $(TOPHTML)) FQMAN1 = $(addprefix man/, $(MAN1)) FQMAN1_SFX = $(addprefix man/, $(MAN1_SFX)) FQMAN5 = $(addprefix man/, $(MAN5)) @@ -33,13 +38,13 @@ FIXMAN = python ../misc/fixman.py #AFLAGS = -a linkcss #AFLAGS = -a stylesheet=extra.css -all: $(FQMAN) $(FQHTML) +all: $(FQMAN) man: $(FQMAN) html: $(FQHTML) -install: $(FQMAN) $(FQHTML) +install: $(FQMAN) mkdir -p $(DESTDIR)/$(mandir)/man1 mkdir -p $(DESTDIR)/$(mandir)/man5 mkdir -p $(DESTDIR)/$(docdir) @@ -53,6 +58,8 @@ install: $(FQMAN) $(FQHTML) for m in $(FQMAN5); do \ install -m 644 $$m $(DESTDIR)/$(mandir)/man5 || exit 1; \ done + +htmlinstall: $(FQHTML) for h in $(FQHTML); do \ install -m 644 $$h $(DESTDIR)/$(docdir) || exit 1; \ done @@ -84,13 +91,13 @@ apiupload: apidoc -rsync -rtlz ../sql/londiste/docs/html/* $(web)/londiste/ clean: - rm -rf html *.xml + rm -rf html *.xml api distclean: clean rm -rf ../sql/pgq/docs/pgq realclean: distclean - rm -f *.[15] *.xml *.html + rm -rf man *.xml *.html ifneq ($(ASCIIDOC),no) ifneq ($(XMLTO),no) @@ -104,31 +111,37 @@ man/%.1: man/%.xml endif html/doc/%.html: %.txt $(COMMON) - @mkdir -p html/doc + @mkdir -p $(dir $@) LANG=C cat $< \ | sed -e '/^include/b' -e 's,\([A-Za-z.0-9]*\)[.]txt,link:\1.html[],g' \ | $(ASCIIDOC) $(AFLAGS) -a toc `$(GETATTRS) $<` -o - - \ | sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \ > $@ -html/doc/README.html: ../README - @mkdir -p html/doc +html/README.html: ../README + @mkdir -p $(dir $@) cat $< \ | sed -e 's,doc/\([!-~]*\)[.]txt,link:\1.html[],g' \ -e 's,http:[!-~]*,&[],g' \ - | $(ASCIIDOC) $(AFLAGS) -o - - \ + | $(ASCIIDOC) $(AFLAGS) -a toc -o - - \ + | sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \ + > $@ + +html/INSTALL.html: ../INSTALL + @mkdir -p $(dir $@) + $(ASCIIDOC) $(AFLAGS) -o - $< \ | sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \ > $@ -html/doc/INSTALL.html: ../INSTALL - @mkdir -p html +html/index.html: index.txt + @mkdir -p $(dir $@) $(ASCIIDOC) $(AFLAGS) -o - $< \ | sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \ > $@ endif web: $(FQHTMLS) - rsync -avz $(FQHTML) $(web)/doc/ + rsync -avz html/* $(web)/ diff --git a/doc/TODO.txt b/doc/TODO.txt index 8f64742c..df78a053 100644 --- a/doc/TODO.txt +++ b/doc/TODO.txt @@ -4,24 +4,24 @@ Gut feeling about priorities: High:: - Needer for wider deployment / 3.0-final. + Needed soon. Medium:: Good if done, but can be postponed. Low:: Interesting idea, but OK if not done. -== High Priority == +== Medium Priority == + +* tests: takeover testing + - wal behind + - wal ahead + - branch behind * londiste takeover: check if all tables exist and are in sync. Inform user. Should the takeover stop if problems? How can such state be checked on-the-fly? Perhaps `londiste missing` should show in-copy tables. -* docs: - - londiste manpage - - qadmin manpage - - pgqd manpage - * cascade takeover: wal failover queue sync. WAL-failover can be behind/ahead from regular replication with partial batch. Need to look up-batched events in wal-slave and full-batches on branches @@ -36,14 +36,10 @@ Low:: . Replay rest of batches fully . Promote to root -* tests: takeover testing - - wal behind - - wal ahead - - branch behind - -== Medium Priority == +* Load public connect string for local node: + - from .ini (db=, public_db=?) + - with query sql -* Load public connect string from .ini * Load provider connect string: - from ini (not good) - describe connstr + query to look up node names? @@ -55,17 +51,6 @@ Low:: - skytools.DBStruct - londiste handlers -* cascade watermark limit nodes. A way to avoid affecting root - with lagging downstream nodes. Need to tag some nodes to coordinate - watermark with each other and not send it upstream to root. - -* automatic sql upgrade mechanism - check version in db, apply .sql - if contains newer version. - -* integrate skytools.checker. It is generic 'check/repair' script - that can also automatically apply fixes. Should rebase londiste - check/repair on that. - * londiste add-table: automatic serial handling, --noserial switch? Currently, `--create-full` does not create sequence on target, even if source table was created with `serial` column. It does associate column @@ -78,8 +63,6 @@ Low:: * qadmin: merge cascade commands (medium) - may need api redesign to avoid duplicating pgq.cascade code? -* dbscript: configurable error timeout (currently it's hardwired to 20s) - * londiste replay: when buffering queries, check their size. Current buffering is by count - flushed if 200 events have been collected. That does not take account that some rows can be very large. @@ -91,10 +74,6 @@ Low:: generally slow (GP, bad network, etc). Perhaps we can use psycopg2 async mode for that. Or threading. -* developer docs for: - - DBScript, pgq.Consumer, pgq.CascadedConsumer? - - Update pgq/sql docs for current pgq. - == Low Priority == * dbscript: switch (-q) for silence for cron/init scripts. diff --git a/doc/faq.txt b/doc/faq.txt index ad9e1232..09fe2918 100644 --- a/doc/faq.txt +++ b/doc/faq.txt @@ -8,62 +8,10 @@ It is bunch of database management tools we use and various frameworks / modules they depend on. -Main components are: - -==== Python scripts ==== - -Main tools: - - walmgr:: walshipping manager - londiste:: replication on top of pgq - pgqadm:: pgq administration and maintenance - setadm:: cascaded pgq administration - -Special scripts: - - bulk_loader - cube_dispatcher - table_dispatcher - -Queue copy - - queue_mover:: copy queue contents from one db to another - queue_splitter:: copy queue contents to another db splitting it into several queues - -Operate bunch of scripts together - - scriptmgr - -==== Python modules ==== - - pgq - skytools - londiste - -==== SQL modules ==== - - londiste - pgq - pgq_node - pgq_ext - -=== Where is the code located? === - -Code layout: - - debian/ - doc/ - python/bin/ - python/londiste/ - Londiste python modules - python/modules/ - C extension modules for Python (string handling) - python/pgq/ - pgq and cascaded pgq python modules - python/skytools/ - python modules for generic database scripting - scripts/ - Special purpose python scripts (python) - sql/londiste/ - database code for londiste (plpgsql) - sql/pgq/ - PgQ database code (C + plpgsql) - sql/pgq_ext/ - PgQ event/batch tracking on remote database (plpgsql) - sql/pgq_node/ - cascaded pgq support (plpgsql) - sql/txid/ - Obsolete txid code for Postgres 8.2 and below +Main components are `pgq` SQL module which implements generic +queue in database, Python module for writing consumers for it, +Londiste replication on top of them and walmgr for setting up +WAL-based standby servers. == PgQ - The generic queue == @@ -101,6 +49,37 @@ Yes, PgQ was created by generalizing queueing parts from Slony-I. === Dump-restore === +Database which contains `pgq` schema can be dumped and restored with +`pg_dump`, but extra steps must be taken because PgQ tables +contains transaction id-s and snapsnot which are extracted from +Postgres code. PgQ depends on transaction id values going always +higher. Thus restoring database in new Postgres clusten will break +PgQ if that cluster has txids smaller than they were in old cluster. + +Postgres interally uses rolling 32-bit txids which on export are +combined with 'txid epoch', which simply is count how many times +the 32-bit txid has already cycled. Thus the way to make sure +new cluster has highed txids than old one is to set the epoch higher +in new cluster than it was in old cluster. + +To see epoch in old cluster in running database: + + SELECT (txid_current() >> 32) as epoch; + epoch + ----- + 1 + +To see epoch on shut down database: + + $ pg_resetxlog -n $DATADIR + ... + Latest checkpoint's NextXID: 0/3938 (epoch -> 0) + ... + +To set epoch in new cluster - it must be shut down first: + + $ pg_resetxlog -e $NEWEPOCH $DATADIR + == Londiste - The replication tool == === What type of replication it does? === diff --git a/doc/howto/londiste3_cascaded_rep_howto.txt b/doc/howto/londiste3_cascaded_rep_howto.txt new file mode 100644 index 00000000..baed95e6 --- /dev/null +++ b/doc/howto/londiste3_cascaded_rep_howto.txt @@ -0,0 +1,259 @@ + += How to Set up Cascaded Replication in londiste = + +== Basic cluster setup == + +Basic configuration for cascaded replication setup. + +The configuration file for ticker (pgqd) restricting ticker to only the 5 +replicated databases: + +---- +$ cat conf/pgqd.ini +[pgqd] +database_list = db1,db2,db3,db4,db5 +logfile = log/pgqd.log +pidfile = pid/pgqd.pid +---- + +The ini files for databases are are created similar to the one below, only +the "db1" part changes + +---- +$ cat conf/londiste_db1.ini +[londiste3] +job_name = londiste_db1 +db = dbname=db1 +queue_name = replika +logfile = log/%(job_name)s.log +pidfile = pid/%(job_name)s.pid + +pgq_autocommit = 1 +pgq_lazy_fetch = 0 +---- + +After creating the ini files we are ready to install londiste3 and initialize nodes + +---- +$ londiste3 -q conf/londiste_db1.ini create-root node1 dbname=db1 +$ londiste3 -q conf/londiste_db2.ini create-branch node2 dbname=db2 --provider=dbname=db1 +$ londiste3 -q conf/londiste_db3.ini create-branch node3 dbname=db3 --provider=dbname=db1 +$ londiste3 -q conf/londiste_db4.ini create-branch node4 dbname=db4 --provider=dbname=db2 +$ londiste3 -q conf/londiste_db5.ini create-branch node5 dbname=db5 --provider=dbname=db3 +---- + +Now that schemas are installed, we can start the ticker + +---- +$ pgqd -q -d conf/pgqd.ini +---- + +To see the topology of replication, you can run the status command against any node: + +---- +$ londiste3 -q conf/londiste_db4.ini status +Queue: replika Local node: node4 + +node1 (root) + | Tables: 0/0/0 + | Lag: 4s, Tick: 2, NOT UPTODATE + +--node3 (branch) + | Tables: 0/0/0 + | Lag: 12s, Tick: 1, NOT UPTODATE + +--node2 (branch) + | Tables: 0/0/0 + | Lag: 12s, Tick: 1, NOT UPTODATE + +--node4 (branch) + Tables: 0/0/0 + Lag: 11s, Tick: 1, NOT UPTODATE +---- + +And you need a londiste worker process on each node to actually carry out +the actions + +---- +$ londiste3 -q -d conf/londiste_db1.ini worker +$ londiste3 -q -d conf/londiste_db2.ini worker +$ londiste3 -q -d conf/londiste_db3.ini worker +$ londiste3 -q -d conf/londiste_db4.ini worker +$ londiste3 -q -d conf/londiste_db5.ini worker +---- + +== Adding tables and data == + +Now let's play with data. + +Create table on root node and fill couple of rows + +---- +$ psql -d db1 -c create table mytable (id serial primary key, data text) +$ psql -d db1 -c insert into mytable (data) values ('row1') +$ psql -d db1 -c insert into mytable (data) values ('row2') +$ psql -d db1 -c insert into mytable (data) values ('row3') +$ psql -d db1 -c insert into mytable (data) values ('row4') +---- + +Creat some load on table + +---- +$ ./loadgen.py -d conf/gen1.ini +---- + +Register table on root node + +---- +$ londiste3 -q conf/londiste_db1.ini add-table mytable +$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq +---- + +Register table on other node with creation + +---- +$ psql -d db2 -c create sequence mytable_id_seq +CREATE SEQUENCE +$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq +$ londiste3 -q conf/londiste_db2.ini add-table mytable --create-full +$ psql -d db3 -c create sequence mytable_id_seq +CREATE SEQUENCE +$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq +$ londiste3 -q conf/londiste_db3.ini add-table mytable --create-full +$ psql -d db4 -c create sequence mytable_id_seq +CREATE SEQUENCE +$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq +$ londiste3 -q conf/londiste_db4.ini add-table mytable --create-full +$ psql -d db5 -c create sequence mytable_id_seq +CREATE SEQUENCE +$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq +$ londiste3 -q conf/londiste_db5.ini add-table mytable --create-full +---- + + +== Change topology == + +The main advantage of skytools3 cascaded replication is how easy it is to +change the replication topology. + + +---- +$ londiste3 -q conf/londiste_db4.ini change-provider --provider=node2 +$ londiste3 -q conf/londiste_db4.ini status +Queue: replika Local node: node4 + +node1 (root) + | Tables: 1/0/0 + | Lag: 1s, Tick: 57 + +--node2 (branch) + | | Tables: 1/0/0 + | | Lag: 1s, Tick: 57 + | +--node4 (branch) + | Tables: 1/0/0 + | Lag: 1s, Tick: 57 + +--node3 (branch) + | Tables: 1/0/0 + | Lag: 1s, Tick: 57 + +--node5 (branch) + Tables: 1/0/0 + Lag: 7s, Tick: 53 + ERR: londiste_db5: duplicate key value violates unique constraint "mytable_pkey" +---- + +Now let's move it to node 3 + +---- +$ londiste3 -q conf/londiste_db4.ini change-provider --provider=node3 +$ londiste3 -q conf/londiste_db4.ini status +Queue: replika Local node: node4 + +node1 (root) + | Tables: 1/0/0 + | Lag: 0s, Tick: 59 + +--node2 (branch) + | Tables: 1/0/0 + | Lag: 3s, Tick: 59 + +--node3 (branch) + | Tables: 1/0/0 + | Lag: 3s, Tick: 59 + +--node4 (branch) + | Tables: 1/0/0 + | Lag: 3s, Tick: 58 + +--node5 (branch) + Tables: 1/0/0 + Lag: 12s, Tick: 53 +---- + + + +---- +$ londiste3 -q conf/londiste_db5.ini change-provider --provider=node2 +$ londiste3 -q conf/londiste_db1.ini status +Queue: replika Local node: node1 + +node1 (root) + | Tables: 1/0/0 + | Lag: 1s, Tick: 64 + +--node2 (branch) + | | Tables: 1/0/0 + | | Lag: 1s, Tick: 64 + | +--node5 (branch) + | Tables: 1/0/0 + | Lag: 1s, Tick: 64 + +--node3 (branch) + | Tables: 1/0/0 + | Lag: 1s, Tick: 64 + +--node4 (branch) + Tables: 1/0/0 + Lag: 1s, Tick: 64 +---- + +The topology change can also be accomplished from the "other" end, +using the `takeover` command: + +---- +$ londiste3 -q conf/londiste_db3.ini takeover node2 +$ londiste3 -q conf/londiste_db2.ini status +Queue: replika Local node: node2 + +node1 (root) + | Tables: 1/0/0 + | Lag: 0s, Tick: 66 + +--node3 (branch) + | Tables: 1/0/0 + | Lag: 0s, Tick: 66 + +--node4 (branch) + | Tables: 1/0/0 + | Lag: 3s, Tick: 66 + +--node2 (branch) + | Tables: 1/0/0 + | Lag: 0s, Tick: 66 + +--node5 (branch) + Tables: 1/0/0 + Lag: 3s, Tick: 65 +---- + +The takeover command is in fact the only way to change the root node + +---- +$ londiste3 -q conf/londiste_db2.ini takeover node1 +$ londiste3 -q conf/londiste_db2.ini status +Queue: replika Local node: node2 + +node2 (root) + | Tables: 1/0/0 + | Lag: 1s, Tick: 72 + +--node5 (branch) + | Tables: 1/0/0 + | Lag: 1s, Tick: 72 + +--node1 (branch) + | Tables: 1/0/0 + | Lag: 1s, Tick: 71 + +--node3 (branch) + | Tables: 1/0/0 + | Lag: 3s, Tick: 71 + +--node4 (branch) + Tables: 1/0/0 + Lag: 3s, Tick: 71 +---- + +That's it! + + diff --git a/doc/howto/londiste3_merge_howto.txt b/doc/howto/londiste3_merge_howto.txt new file mode 100644 index 00000000..11caa1e4 --- /dev/null +++ b/doc/howto/londiste3_merge_howto.txt @@ -0,0 +1,217 @@ + += How To Set Up "Merge" Replication of the Same Table from Multiple Partitions = + +== Introduction == + +In this howto we will set up a replication scheme , where data is collected +from multiple "partition" databases into a single table on "full" database. + +This situation is common when using PL/Proxy or other similar partitioning +solution for OLTP, but still wanting the data in one table for Data Warehousing. + +Here we will demonstrate the simplest possible setup with 2 partition databases +(`part1` and `part2`) replicating their data into one full database `full1`. + +=== Setting the nodes === + +==== Partition databases ==== + +On both partition databases you need to set up a londiste3 "root" node to +replicate from. We will use the the following .ini files for that + +for database part1 `conf/l3_part1_q_part1.ini`: + +---- +[londiste3] +job_name = l3_part1_q_part1 +db = dbname=part1 +queue_name = l3_part1_q +logfile = log/%(job_name)s.log +pidfile = pid/%(job_name)s.pid +---- + +and for part2 `conf/l3_part2_q_part2.ini`: + +---- +[londiste3] +job_name = l3_part2_q_part2 +db = dbname=part2 +queue_name = l3_part2_q +logfile = log/%(job_name)s.log +pidfile = pid/%(job_name)s.pid +---- + +These ini files are then used for setting up the nodes and adding tables to root node + +Set up the root nodes on part1 and part2 + +---- +$ londiste3 -v conf/l3_part1_q_part1.ini create-root part1_root dbname=part1 +$ londiste3 -v conf/l3_part2_q_part2.ini create-root part2_root dbname=part2 +---- + +==== Full database ==== + +On the full database, which will hold data from both partitions you need to +set up two londiste nodes, one for each of the partition nodes. +These will act as the receiving nodes to replicate to. + +These look very similar and differ only in queue name + +File `conf/l3_part1_q_full1.ini`: +---- +[londiste3] +job_name = l3_part1_q_full1 +db = dbname=full1 +queue_name = l3_part1_q +logfile = log/%(job_name)s.log +pidfile = pid/%(job_name)s.pid +---- + +File `conf/l3_part2_q_full1.ini`: +---- +[londiste3] +job_name = l3_part2_q_full1 +db = dbname=full1 +queue_name = l3_part2_q +logfile = log/%(job_name)s.log +pidfile = pid/%(job_name)s.pid +---- + +These are first used to set up the leaf nodes: + +---- +$ londiste3 -v conf/l3_part1_q_full1.ini create-leaf merge_part1_full1 dbname=full1 --provider=dbname=part1 +$ londiste3 -v conf/l3_part2_q_full1.ini create-leaf merge_part2_full1 dbname=full1 --provider=dbname=part2 +---- + +and later also for launching the replication worker daemons + +But before launching the workers you need to start the pgqd or the "ticker daemon" + +---- +$ pgqd -v -d conf/pgqd.ini +---- + +the `conf/pgqd.ini` file for the command above looks like this: + +---- +[pgqd] +database_list = part1,part2,full1 +logfile = log/pgqd.log +pidfile = pid/pgqd.pid +---- + +Now that the ticker is running, it's time to launch londiste3 workers which will +do tha actual replication: + +---- +$ londiste3 -v -d conf/l3_part1_q_full1.ini worker +$ londiste3 -v -d conf/l3_part2_q_full1.ini worker +---- + +=== Setting up the tables === + +in order to have something to replicate, we need some tables, so let's create +them on partition nodes: + +---- +$ psql -d "part1" -c "create table mydata (id int4 primary key, data text)" +$ psql -d "part2" -c "create table mydata (id int4 primary key, data text)" +---- + +and then add them to set of replicated tables on the root node + +---- +$ londiste3 -v conf/l3_part1_q_part1.ini add-table mydata +$ londiste3 -v conf/l3_part2_q_part2.ini add-table mydata +---- + +Now we need some data in these tables, as replicating empty tables is no fun + +---- +$ psql -d "part1" -c "insert into mydata values (1, 'part1')" +$ psql -d "part2" -c "insert into mydata values (2, 'part2')" +---- + +We can check that the tables are actually registered for replication in londiste: + +---- +$ psql -d "full1" -c "select * from londiste.table_info order by queue_name" + nr | queue_name | table_name | local | merge_state | custom_snapshot | dropped_ddl | table_attrs | dest_table +----+------------+---------------+-------+-------------+-----------------+-------------+-------------+------------ + 1 | l3_part1_q | public.mydata | f | | | | | + 2 | l3_part2_q | public.mydata | f | | | | | +(2 rows) +---- + +Now lets subscribe them on full database. As the table is not yet created on +full1, we specify `--create` so londiste creates the table on leaf node based on +structure that is on root. The switch `--merge-all` tells londiste to add the table to all +queues which have it on root side, not just the one from the .ini file + +---- +$ londiste3 -v conf/l3_part1_q_full1.ini add-table mydata --create --merge-all +---- + +And yes, there it is, subscribed from both queues: + +---- +$ psql -d "full1" -c "select * from londiste.table_info order by queue_name" + nr | queue_name | table_name | local | merge_state | custom_snapshot | dropped_ddl | table_attrs | dest_table +----+------------+---------------+-------+-------------+-----------------+-------------+-------------+------------ + 1 | l3_part1_q | public.mydata | t | | | | | + 2 | l3_part2_q | public.mydata | t | | | | | +(2 rows) +---- + +Now we can put more data to partition tables: + +---- +$ psql -d "part1" -c "insert into mydata values (4 + 1, 'part1')" +$ psql -d "part2" -c "insert into mydata values (4 + 2, 'part2')" +---- + +Wait a few seconds: + +---- +$ sleep 10 +---- + +And check that the data has indeed appeared on full database: + +---- +$ psql -d "full1" -c "select * from mydata order by id" + id | data +----+------- + 1 | part1 + 2 | part2 + 5 | part1 + 6 | part2 +(4 rows) +---- + +the rows fith ids 1 and 2 where replicated during initial copy, the ones with +5 and 6 were captured by triggers into event log on partition database and then +replicated to full1 using the standard replication process. + +=== checking subscription === + +Just to check if we really did achieve what we wanted, we see which tables +are present and fully sobscribed ('ok') + +---- +$ psql -d "full1" -c "select * from londiste.table_info order by queue_name" + nr | queue_name | table_name | local | merge_state | custom_snapshot | dropped_ddl | table_attrs | dest_table +----+------------+---------------+-------+-------------+-----------------+-------------+-------------+------------ + 1 | l3_part1_q | public.mydata | t | ok | | | | + 2 | l3_part2_q | public.mydata | t | ok | | | | +(2 rows) +---- + +ok, here we have the table public.mydata subscribed from 2 queues and it's +merge_state is 'ok', meaning the initial copy process has been successfull + + +That's it , we have successfully set up replication from two partition +databases to one single full database. diff --git a/doc/howto/londiste3_partitioning_howto.txt b/doc/howto/londiste3_partitioning_howto.txt new file mode 100644 index 00000000..ba83fabb --- /dev/null +++ b/doc/howto/londiste3_partitioning_howto.txt @@ -0,0 +1,210 @@ += Setting up Londiste3 replication to partitions = + +== Introduction == + +This sample shows how to use Londiste `part` handler module to split +one big table between two databases. + +The target databases will have `partconf` schema which is usually +used to drive PL/Proxy. Here it is used simply to provide +configuration to `part` handler. + +== Setting up the Root Database == + +=== Create database === + +Run the following SQL: +---- +CREATE DATABASE l3part_root; +---- + +=== Set up pgbench schema === + +In this HowTo we are using pgbench for setting up the schema, +populating it with sampledata and later running SQL loads to be replicated. + + +This command will create pgbanch tables and fill them with data: +---- +/usr/lib/postgresql/9.1/bin/pgbench -i -s 2 -F 80 l3part_root +---- + +=== Change primary key columns to text === + +Standard pgbench schema has integer primary key columns for its tables. +The standard partitioning handler is able to partition only text columns, +so we change the primary key column types to text + + +---- +alter table pgbench_accounts alter column aid type text; +alter table pgbench_branches alter column bid type text; +alter table pgbench_tellers alter column tid type text; +---- + +Now create the partition databases to replicate to. +Each of these will get roughly half of the individual data rows. + + +Create database for partition #0: +---- +createdb l3part_part0; +---- + +And create a partition configuration table in this database +---- + +CREATE SCHEMA partconf; +CREATE TABLE partconf.conf ( + part_nr integer, + max_part integer, + db_code bigint, + is_primary boolean, + max_slot integer, + cluster_name text +); +insert into partconf.conf(part_nr, max_part) values(0,1); + +---- + + +Create database for partition #1: +---- +CREATE DATABASE l3part_part1; +---- + + +---- + +CREATE SCHEMA partconf; +CREATE TABLE partconf.conf ( + part_nr integer, + max_part integer, + db_code bigint, + is_primary boolean, + max_slot integer, + cluster_name text +); +insert into partconf.conf(part_nr, max_part) values(1,1); +---- + +Next create configuration files file for root node and both partitions + +st3partsplit/st3_l3part_root.ini +---- +[londiste3] +job_name = st3_l3part_root +db = dbname=l3part_root +queue_name = replika +logfile = st3partsplit/log/st3_l3part_root.log +pidfile = st3partsplit/pid/st3_l3part_root.pid +---- + +st3partsplit/st3_l3part_part0.ini +---- +[londiste3] +job_name = st3_l3part_part0 +db = dbname=l3part_part0 +queue_name = replika +logfile = st3partsplit/log/st3_l3part_part0.log +pidfile = st3partsplit/pid/st3_l3part_part0.pid +---- + +st3partsplit/st3_l3part_part1.ini +---- +[londiste3] +job_name = st3_l3part_part1 +db = dbname=l3part_part1 +queue_name = replika +logfile = st3partsplit/log/st3_l3part_part1.log +pidfile = st3partsplit/pid/st3_l3part_part1.pid +---- + +Then create root node: + +---- +londiste3 st3partsplit/st3_l3part_root.ini create-root node1 dbname=l3part_root +---- + +And start the worker on root: +---- +londiste3 -d st3partsplit/st3_l3part_root.ini worker +---- + + + +And create leaf nodes and start the workers on partitions : + +---- +londiste3 st3partsplit/st3_l3part_part0.ini create-leaf node2_0 dbname=l3part_part0 --provider=dbname=l3part_root +londiste3 -d st3partsplit/st3_l3part_part0.ini worker +---- + +Second node: +---- +londiste3 st3partsplit/st3_l3part_part1.ini create-leaf node2_1 dbname=l3part_part1 --provider=dbname=l3part_root +londiste3 -d st3partsplit/st3_l3part_part1.ini worker +---- + + + +Create config file st3partsplit/pgqd.ini for `pgqd` ("the ticker") +---- +[pgqd] + +logfile = st3partsplit/log/pgqd.log +pidfile = st3partsplit/pid/pgqd.pid + +---- + + +Start the ticker process : +---- +pgqd -d st3partsplit/pgqd.ini +---- + + + +Now add the replicated tables to root and partitions. +Here we use `--create` switch to add them to partition, +which means Londiste takes schema from root node and +creates tables on target nodes automatically. + +The `--handler=part` tells londiste to use the `part` handler for replication, +the `--handler-arg=key=*id` specifyies which key field to partition on. + + + +Run command the following commands : +---- +londiste3 st3partsplit/st3_l3part_root.ini add-table pgbench_accounts --handler=part --handler-arg=key=aid +londiste3 st3partsplit/st3_l3part_part0.ini add-table pgbench_accounts --create --handler=part --handler-arg=key=aid +londiste3 st3partsplit/st3_l3part_part1.ini add-table pgbench_accounts --create --handler=part --handler-arg=key=aid + +londiste3 st3partsplit/st3_l3part_root.ini add-table pgbench_branches --handler=part --handler-arg=key=bid +londiste3 st3partsplit/st3_l3part_part0.ini add-table pgbench_branches --create --handler=part --handler-arg=key=bid +londiste3 st3partsplit/st3_l3part_part1.ini add-table pgbench_branches --create --handler=part --handler-arg=key=bid + +londiste3 st3partsplit/st3_l3part_root.ini add-table pgbench_tellers --handler=part --handler-arg=key=tid +londiste3 st3partsplit/st3_l3part_part0.ini add-table pgbench_tellers --create --handler=part --handler-arg=key=tid +londiste3 st3partsplit/st3_l3part_part1.ini add-table pgbench_tellers --create --handler=part --handler-arg=key=tid +---- + +The following command will run pgbench full speed with 5 parallel +database connections for 10 seconds. + +---- +/usr/lib/postgresql/9.1/bin/pgbench -T 10 -c 5 l3part_root +---- + +After this is done, you can check that the tables on both sides hanve the same data with + +---- +londiste3 st3partsplit/st3_l3part_part0.ini compare +londiste3 st3partsplit/st3_l3part_part0.ini compare +---- + +Except of course that they dont - each partition will only have roughly half +the data from the root. But the row counts and checksums of the partitions +should both add up to the numbers on the master. + diff --git a/doc/howto/londiste3_simple_rep_howto.txt b/doc/howto/londiste3_simple_rep_howto.txt new file mode 100644 index 00000000..c05a95b0 --- /dev/null +++ b/doc/howto/londiste3_simple_rep_howto.txt @@ -0,0 +1,212 @@ += Setting up simple Londiste3 replication = + +== Introduction == + +This sample does the following actions: + + * sets up te databases + - creates a database 'l3simple_db1', which will be master + - populates this with pgbench schema and data + - adds primary and foreign keys to make the db more realistic + - makes a copy of the database ('l3simple_db2') to be used as slave + * sets up replication from the master to slave database + - creates the root node on 'l3simple_db1' + - creates a leaf node on 'l3simple_db2' + - starts the ticker daemon + - adds all tables to replication set on both databases + - waits the replication to complete + +It also runs pgbench a to test that the replication actually happens and works properly. + + +== Set up schema for root database == + +=== Create database === + +Run the following SQL: +---- +CREATE DATABASE l3simple_db1; +---- + +=== Set up pgbench schema === + +In this HowTo we are using pgbench for setting up the schema, +populating it with sampledata and later running SQL loads to be replicated. + + +Run command : +---- +/usr/lib/postgresql/9.1/bin/pgbench -i -s 2 -F 80 l3simple_db1 +---- + +=== And add primary and foreign keys needed for replication === + +Standard pgbench schema lacks Primary Key on the history table. +As Londiste need primary keys we add one. We also add Foreign Keys between tables, +as these may help detect possible replication failures. + +create file /tmp/prepare_pgbenchdb_for_londiste.sql with the following ... +---- + +-- add primary key to history table +ALTER TABLE pgbench_history ADD COLUMN hid SERIAL PRIMARY KEY; + +-- add foreign keys +ALTER TABLE pgbench_tellers ADD CONSTRAINT pgbench_tellers_branches_fk FOREIGN KEY(bid) REFERENCES pgbench_branches; +ALTER TABLE pgbench_accounts ADD CONSTRAINT pgbench_accounts_branches_fk FOREIGN KEY(bid) REFERENCES pgbench_branches; +ALTER TABLE pgbench_history ADD CONSTRAINT pgbench_history_branches_fk FOREIGN KEY(bid) REFERENCES pgbench_branches; +ALTER TABLE pgbench_history ADD CONSTRAINT pgbench_history_tellers_fk FOREIGN KEY(tid) REFERENCES pgbench_tellers; +ALTER TABLE pgbench_history ADD CONSTRAINT pgbench_history_accounts_fk FOREIGN KEY(aid) REFERENCES pgbench_accounts; + +---- + +then load it into database: + +---- +psql l3simple_db1 -f /tmp/prepare_pgbenchdb_for_londiste.sql +---- + + +Create and populate target database: + +---- +psql -d postgres -c "CREATE DATABASE l3simple_db2;" +pg_dump -s l3simple_db1 | psql l3simple_db2 +---- + +Create configuration file st3simple/st3_l3simple_db1.ini + +---- +[londiste3] +job_name = st3_l3simple_db1 +db = dbname=l3simple_db1 +queue_name = replika +logfile = st3simple/log/st3_l3simple_db1.log +pidfile = st3simple/pid/st3_l3simple_db1.pid +---- + + +Create Londiste root node: + +---- +londiste3 st3simple/st3_l3simple_db1.ini create-root node1 dbname=l3simple_db1 +---- + +Run worker daemon for root node: + +---- +londiste3 -d st3simple/st3_l3simple_db1.ini worker +---- + +Create configuration file st3simple/st3_l3simple_db2.ini +for worker daemon on target node: + +---- +[londiste3] +job_name = st3_l3simple_db2 +db = dbname=l3simple_db2 +queue_name = replika +logfile = st3simple/log/st3_l3simple_db2.log +pidfile = st3simple/pid/st3_l3simple_db2.pid +---- + +Initialize node in target database: + +---- +londiste3 st3simple/st3_l3simple_db2.ini create-leaf node2 dbname=l3simple_db2 --provider=dbname=l3simple_db1 +---- + +Launch worker daemon for target database. + +---- +londiste3 -d st3simple/st3_l3simple_db2.ini worker +---- + +Create config file `st3simple/pgqd.ini` +for PgQ ticker daemon: + +---- +[pgqd] + +logfile = st3simple/log/pgqd.log +pidfile = st3simple/pid/pgqd.pid +---- + +Launch ticker daemon: + +---- +pgqd -d st3simple/pgqd.ini +---- + +To generate some data traffic on the master database while replicating, +run the following command in background: + +---- +/usr/lib/postgresql/9.1/bin/pgbench -T 120 -c 5 l3simple_db1 -f /tmp/throttled.pgbench +---- + +the /tmp/throttled.pgbench contains the standard pgbencg workload, except that +ther are random length waits between commands. + +Now add all the tables to replication, first on root node and then on the leaf + +Run command : +---- +londiste3 st3simple/st3_l3simple_db1.ini add-table --all +londiste3 st3simple/st3_l3simple_db2.ini add-table --all +---- + + +== Checking and testing == + +To test our newly set up replication + +The following command will run pgbench full speed with 5 parallel +database connections generating database traffic +for 10 seconds + +---- +/usr/lib/postgresql/9.1/bin/pgbench -T 10 -c 5 l3simple_db2 +---- + +After this is done, you can check that the tables on both sides have the same data with + +---- +londiste3 st3simple/st3_l3simple_db2.ini compare +---- + +Compare command will establish the same logical point in time on provider and +subscriber nodes and then count and checksum the rows on both sides. + +The result will look like this + +---- +2011-12-25 08:24:42,138 29189 INFO Locking public.pgbench_accounts +2011-12-25 08:24:42,147 29189 INFO Syncing public.pgbench_accounts +2011-12-25 08:24:45,233 29189 INFO Counting public.pgbench_accounts +2011-12-25 08:24:46,154 29189 INFO srcdb: 200000 rows, checksum=3864719477 +2011-12-25 08:24:46,953 29189 INFO dstdb: 200000 rows, checksum=3864719477 +2011-12-25 08:24:46,961 29189 INFO Locking public.pgbench_branches +2011-12-25 08:24:46,965 29189 INFO Syncing public.pgbench_branches +2011-12-25 08:24:50,528 29189 INFO Counting public.pgbench_branches +2011-12-25 08:24:50,549 29189 INFO srcdb: 2 rows, checksum=-82377983 +2011-12-25 08:24:50,556 29189 INFO dstdb: 2 rows, checksum=-82377983 +2011-12-25 08:24:50,568 29189 INFO Locking public.pgbench_history +2011-12-25 08:24:50,572 29189 INFO Syncing public.pgbench_history +2011-12-25 08:24:53,641 29189 INFO Counting public.pgbench_history +2011-12-25 08:24:53,660 29189 INFO srcdb: 1310 rows, checksum=-34927328558 +2011-12-25 08:24:53,670 29189 INFO dstdb: 1310 rows, checksum=-34927328558 +2011-12-25 08:24:53,675 29189 INFO Locking public.pgbench_tellers +2011-12-25 08:24:53,677 29189 INFO Syncing public.pgbench_tellers +2011-12-25 08:24:56,733 29189 INFO Counting public.pgbench_tellers +2011-12-25 08:24:56,737 29189 INFO srcdb: 20 rows, checksum=518235101 +2011-12-25 08:24:56,740 29189 INFO dstdb: 20 rows, checksum=518235101 +---- + +the "checksum" is computed by adding up hashtext() sums for all database rows. + + +== Done == + +The setup of simple 2 node cluster is done. + diff --git a/doc/howto/setup_walmgr_replication.txt b/doc/howto/setup_walmgr_replication.txt new file mode 100644 index 00000000..a358bfc7 --- /dev/null +++ b/doc/howto/setup_walmgr_replication.txt @@ -0,0 +1,398 @@ += Setting up backup and streaming replication with walmgr3 = + +Hannu Krosing + +== Introduction == + +This is a HowTo for setting up PostgreSQL for backup +and then continue to setting up both wal-based and streaming' +replication using waplmgr3 + +The samle commandlines and configuration files are produced by +running command + +----- +python WALManager.py +----- + +in directory python/testwrappers. This test/sample + + * sets up a postgresql master server using directory /tmp/test_master + * creates a sample database tyhere using pgbench + * creates walmgr3 master configuration file /tmp/test_master/wal-master.ini + * creates a directory and configuration file "on slave" - /tmp/test_slave/wal-slave.ini + * runs walmgr3 setup command, which modifies postgresql.conf file on master server for replication + * restarts master server for changes of "walmgr3 ... setup" to take effect + * runs walmgr3 backup command, which creates a backup of master database and wal files on slace server + * modifies master's postgresql.conf and pg_hba.conf file some more for enabling Streaming Replication + * runs walmgr3 restore command ON SLAVE which moves backup directory into the right place for slave + server and then starts the slave server + * verifies that replication is running by running a SQL UPDATE query on master and checking the results on slave. + +== Preparation == + +For things to work you need to have passwordless ssh access +from master to slave set up using public/private key pair + +This can be done using walmgr3 + +---- +1. Set up passwordless ssh-key on Master and write configuration file + + master$ walmgr3 --ssh-keygen --init-master --slave <slave_hostname> + +2. Set up passwordless ssh authentication from Master to Slave and write configuration file on Slave + + slave$ walmgr3 --init-slave --ssh-add-key=/tmp/id_rsa.pub +---- + +or directly from commandline + +---- +master$ test -f ~/.ssh/id_dsa.pub || ssh-keygen -t dsa +master$ cat ~/.ssh/id_dsa.pub | ssh slave cat \>\> .ssh/authorized_keys +---- + +== setting up an population master database === + +This part is for playing with walmgr3 without disturbing your existing +databases. + +You can skip this part, if you already have a database you want to replicate. + +=== create database server === + +Create database master server directory structure under /tmp/test_master +by running the command : + +---- +/usr/lib/postgresql/9.1/bin/initdb -D /tmp/test_master +---- + +This database wil run as the user who was logged in at the time of +creation, vs. the default user 'postgres' + +Change the port and socket directory (and set up some logging if you want +to see what's going on in the database) + +Edit /tmp/test_master/postgresql.conf and set up the following + +---- + unix_socket_directory = /tmp + port = 55401 + # optional, for logging + log_connections = on + log_disconnections = on +---- + +Now you are ready to start up this server + +---- +/usr/lib/postgresql/9.1/bin/pg_ctl -D /tmp/test_master -l /tmp/test_master/postgresql.log start +---- + +you can use "tail /tmp/test_master/postgresql.log" to check that server started up correctly, and + +---- +psql -h /tmp -p 55401 -l +---- + +to check that it accepts connections + +=== create database and generate some data for it === + +Create database to be used + +---- +createdb pgbdb; +---- + +and initialise a pgbench database structure and data in this database + +---- +/usr/lib/postgresql/9.1/bin/pgbench -i -s 1 -F 80 pgbdb -h /tmp -p 55401 +---- + +== Setting up the replication == + +OK. Now we have a database server to replicate, so lets configure walmgr3 + +=== setting up master === + +First we need a master configuration file /tmp/test_master/wal-master.ini : + +---- +[walmgr] +job_name = wal-master +logfile = /tmp/test_master/%(job_name)s.log +pidfile = /tmp/test_master/%(job_name)s.pid +use_skylog = 0 + +master_db = port=55401 host=/tmp dbname=template1 +master_data = /tmp/test_master +master_config = /tmp/test_master/postgresql.conf +master_bin = /usr/lib/postgresql/9.1/bin/ + +# set this only if you can afford database restarts during setup and stop. +# master_restart_cmd = pg_ctlcluster 9.1 main restart + +slave = 127.0.0.1 +slave_config = /tmp/test_slave/wal-slave.ini + +walmgr_data = /tmp/test_slave_walmanager/backup/ + +completed_wals = %(walmgr_data)s/logs.complete +partial_wals = %(walmgr_data)s/logs.partial +full_backup = %(walmgr_data)s/data.master +config_backup = %(walmgr_data)s/config.backup + +# syncdaemon update frequency +loop_delay = 10.0 +# use record based shipping available since 8.2 +use_xlog_functions = 0 + +# pass -z to rsync, useful on low bandwidth links +compression = 0 + +# keep symlinks for pg_xlog and pg_log +keep_symlinks = 1 + +# tell walmgr to set wal_level to hot_standby during setup +hot_standby = 1 + +# periodic sync +#command_interval = 600 +#periodic_command = /var/lib/postgresql/walshipping/periodic.sh +---- + +The things to takew notice hera ar that + + * walmgr_data is a directory _on_the_slave_host_ + * it is a bad idea to put slave_config in slave data directory (that + would be /tmp/test_slave/data/ as defined in wal-slave.ini below) + as then it gets overwritten when doing the restore. + + +=== setting up slave === + +You also need a walmgr3 conf file on slave, /tmp/test_slave/wal-slave.ini : + +---- +[walmgr] +job_name = wal-standby +logfile = /tmp/test_slave_walmanager/%(job_name)s.log +use_skylog = 0 + +slave_data = /tmp/test_slave/data +slave_bin = /usr/lib/postgresql/9.1/bin/ +slave_stop_cmd = /usr/lib/postgresql/9.1/bin//pg_ctl -D /tmp/test_slave/data stop +slave_start_cmd = /usr/lib/postgresql/9.1/bin//pg_ctl -D /tmp/test_slave/data -l /tmp/test_slave/data/postgresql.log start +#slave_config_dir = /tmp/test_slave +slave_config_dir = /tmp/test_slave/data + +walmgr_data = /tmp/test_slave_walmanager/backup/ +completed_wals = %(walmgr_data)s/logs.complete +partial_wals = %(walmgr_data)s/logs.partial +full_backup = %(walmgr_data)s/data.master +config_backup = %(walmgr_data)s/config.backup + +backup_datadir = no +keep_backups = 0 +# archive_command = + +# primary database connect string for hot standby -- enabling +# this will cause the slave to be started in hot standby mode. +primary_conninfo = host=127.0.0.1 port=55401 host=/tmp +--- + +=== Configuring postgreSQL for replication using walmgr3 === + +Running the command : +---- +walmgr3 /tmp/test_master/wal-master.ini setup +---- + +Modifies master postgresql ini file (/tmp/test_master/postgresql.conf) +with these values +---- +wal_level = 'hot_standby' +archive_mode = 'on' +archive_command = '/usr/local/bin/walmgr3 /tmp/test_master/wal-master.ini xarchive %p %f' +---- + +To enable streaming replication, you will need see that max_wal_senders +is above 0 in /tmp/test_master/postgresql.conf + +---- +setting: max_wal_senders = 3 +---- + +And yoy need to specifically enable the access to "replication" in /tmp/test_master/pg_hba.conf +(Replication has to be enabled by name, database wildcard * does not cover it) + +---- +local replication ubuntu trust +---- + +it is "local" here, as this test/sample does the replication iover local socket, +follow the commenst in pg_hba.conf for your case. + +And again, fo the changes to take effect, you need to restart the server + +---- +/usr/lib/postgresql/9.1/bin/pg_ctl -D /tmp/test_master restart +---- + +== Making the backup == + +Once the configuration files are in place, making a backup is as simple as running + +---- +walmgr3 /tmp/test_master/wal-master.ini backup +---- + +If all goes well, this is what gets output (or written to logsfiles, if so configured) + +---- +0 2012-01-27 16:58:31,464 30870 INFO Backup lock obtained. +2012-01-27 16:58:31,485 30750 INFO Execute SQL: select pg_start_backup('FullBackup'); [port=55401 host=/tmp dbname=template1] +2012-01-27 16:58:36,779 30750 INFO Checking tablespaces +2012-01-27 16:58:36,786 30750 INFO pg_log does not exist, skipping +2012-01-27 16:58:36,873 30750 INFO Backup conf files from /tmp/test_master +2012-01-27 16:58:38,599 31256 INFO First useful WAL file is: 000000010000000000000002 +2012-01-27 16:58:45,442 31633 INFO Backup lock released. +2012-01-27 16:58:45,461 30750 INFO Full backup successful +---- + +and there will be a copy of your masters data directory on slave host +under /tmp/test_slave_walmanager/backup/data.master, WAL files in +files in /tmp/test_slave_walmanager/backup/logs.complete/ and copies +of configuration files in /tmp/test_slave_walmanager/backup/config.backup/ + +=== Backup is done === + +If your aim is to just make copies, you can stop this howto here + +== Starting a Hot Standby replica == + +=== Setting Up the replica === + +If you want your replica to be usable for read-only queries, and not +just report "server is starting up" when you try to connect, then your +slave postgresql.conf neeeds to have "hot_standby = on" set: + +In our sample we also change the port, so we can run both servers on the +same host. + +Editi /tmp/test_slave_walmanager/backup/config.backup/postgresql.conf + +---- +hot_standby = on +port = 55402 +---- + +=== starting the replica === + +You restore backup to your the replica server and start it in one command + +---- +walmgr3 /tmp/test_slave/wal-slave.ini restore +---- + +Herew is what it outputs if everything runs ok + +---- +0 server starting +2012-01-27 16:58:45,709 31636 WARNING backup_datadir is disabled, deleting old data dir +2012-01-27 16:58:45,709 31636 INFO Move /tmp/test_slave_walmanager/backup//data.master to /tmp/test_slave/data +2012-01-27 16:58:45,766 31636 INFO Write /tmp/test_slave/data/recovery.conf +2012-01-27 16:58:45,768 31636 INFO Restoring configuration files +2012-01-27 16:58:45,771 31636 INFO Starting postmaster: /usr/lib/postgresql/9.1/bin//pg_ctl -D /tmp/test_slave/data -l /tmp/test_slave/data/postgresql.log start +---- + +Now you have a streaming replica running at port 55402 and master at port 55401 + +== Testing replication == + +You can test that replication is really working by opening connections +to both master and server and then watching changes done on master appear +instantaneously oin the slave: + +Check the initial replicated state on slave + +---- +ubuntu@ubuntu-VirtualBox:~/skytools-markokr/python/testwrappers$ psql -h /tmp -p 55402 pgbdb +psql (9.1.1) +Type "help" for help. + +pgbdb=# select * from pgbench_tellers; + tid | bid | tbalance | filler +-----+-----+----------+-------- + 1 | 1 | 0 | + 2 | 1 | 0 | + 3 | 1 | 0 | + 4 | 1 | 0 | + 5 | 1 | 0 | + 6 | 1 | 0 | + 7 | 1 | 0 | + 8 | 1 | 0 | + 9 | 1 | 0 | + 10 | 1 | 0 | +(10 rows) +---- + +Update the pgbench_tellers table on master + +---- +pgbdb=# \q +ubuntu@ubuntu-VirtualBox:~/skytools-markokr/python/testwrappers$ psql -h /tmp -p 55401 pgbdb +psql (9.1.1) +Type "help" for help. + +pgbdb=# update pgbench_tellers set filler = random(); +UPDATE 10 +pgbdb=# select * from pgbench_tellers; + tid | bid | tbalance | filler +-----+-----+----------+-------------------------------------------------------------------------------------- + 1 | 1 | 0 | 0.755602441262454 + 2 | 1 | 0 | 0.130802480038255 + 3 | 1 | 0 | 0.725358869414777 + 4 | 1 | 0 | 0.65205558296293 + 5 | 1 | 0 | 0.0436737341806293 + 6 | 1 | 0 | 0.797202748246491 + 7 | 1 | 0 | 0.909699931740761 + 8 | 1 | 0 | 0.981106289196759 + 9 | 1 | 0 | 0.656265312805772 + 10 | 1 | 0 | 0.759600875433534 +(10 rows) + +pgbdb=# \q +---- + +And check that it is the same on slave + +---- +ubuntu@ubuntu-VirtualBox:~/skytools-markokr/python/testwrappers$ psql -h /tmp -p 55402 pgbdb +psql (9.1.1) +Type "help" for help. + +pgbdb=# select * from pgbench_tellers; + tid | bid | tbalance | filler +-----+-----+----------+-------------------------------------------------------------------------------------- + 1 | 1 | 0 | 0.755602441262454 + 2 | 1 | 0 | 0.130802480038255 + 3 | 1 | 0 | 0.725358869414777 + 4 | 1 | 0 | 0.65205558296293 + 5 | 1 | 0 | 0.0436737341806293 + 6 | 1 | 0 | 0.797202748246491 + 7 | 1 | 0 | 0.909699931740761 + 8 | 1 | 0 | 0.981106289196759 + 9 | 1 | 0 | 0.656265312805772 + 10 | 1 | 0 | 0.759600875433534 +(10 rows) + +pgbdb=# +----- + +The replication is done now! + diff --git a/doc/index.txt b/doc/index.txt new file mode 100644 index 00000000..852bb783 --- /dev/null +++ b/doc/index.txt @@ -0,0 +1,47 @@ += Skytools 3 Documentation = + +== Overview == + +* link:README.html[Package overview] +* link:INSTALL.html[Installation help] +* link:doc/faq.html[FAQ] +* link:doc/skytools3.html[Skytools 3.0 Release Notes] + +== HOWTOs == + +* link:doc/howto/londiste3_simple_rep_howto.html[Setting up simple replication with 2 nodes] +* link:doc/howto/londiste3_cascaded_rep_howto.html[Setting up cascaded replication with 5 nodes] +* link:doc/howto/londiste3_merge_howto.html[Merging 2 partitions to one big database] +* link:doc/howto/londiste3_partitioning_howto.html[Replicating from one master to 2 partitions] +* link:doc/howto/setup_walmgr_replication.html[Setting up walmgr] + +== Manpages == + +* link:doc/londiste3.html[londiste3] - Londiste command line interface. +* link:doc/qadmin.html[qadmin] - psql-like console for managing queues +* link:doc/queue_mover.html[queue_mover] - copy queue to another database +* link:doc/queue_splitter.html[queue_splitter] - split queue into different queues +* link:doc/scriptmgr.html[scriptmgr] - bulk start/stopping of skytools scripts +* link:doc/skytools_upgrade.html[skytools_upgrade] - Update database modules +* link:doc/walmgr3.html[walmgr3] - tool for managing WAL-base replication + +== API docs == + +* Python link:api/[API] documentation +* SQL API documentation: + - link:pgq[]: SQL module for generic queue + - link:pgq_coop[]: SQL module for sharing workload + - link:pgq_ext[]: batch tracking in target database + - link:pgq_node[]: cascaded queueing + - link:londiste[]: Londiste state + +== Internal == + +* link:doc/devnotes.html[Notes for contributors] +* link:doc/set.notes.html[Technical notes for cascading] +* link:doc/TODO.html[TODO list] + +// == Old docs == +// * link:doc/pgq-nodupes.html[] +// * link:doc/pgq-sql.html[] + diff --git a/doc/skytools3.txt b/doc/skytools3.txt index 6b35037a..532ee692 100644 --- a/doc/skytools3.txt +++ b/doc/skytools3.txt @@ -22,7 +22,7 @@ New features in Skytools 3 * Cascading is implemented as generic layer on top of PgQ - *Cascaded PgQ*. - Its goal is to keep identical copy of queue contents in several nodes. - Not replication-specific - can be used for any queue. - - Advanced admin operations: switchover, failover, change-provider, pause/resume. + - Advanced admin operations: takeover, change-provider, pause/resume. - For terminology and technical details see here: set.notes.txt. * New Londiste features: @@ -51,7 +51,10 @@ New features in Skytools 3 were introduced in 2.1.x, but were not on by default. Now they are used by default. - - Londiste processes events via 'handlers'. (table partitioning) + - Londiste processes events via 'handlers'. Thus we can do table partitioning + in Londiste, instead of custom consumer, which means all Londiste features + are available in such situation - like proper initial COPY. + To see list of them: `londiste3 x.ini show-handlers`. - Target table can use different name (--dest-table) @@ -59,16 +62,11 @@ New features in Skytools 3 user-friendly, this is an experiment on interactive console with heavy emphasis on tab-completion. -* New multi-database ticker. It is possible to set up one process that +* New multi-database ticker: `pgqd`. It is possible to set up one process that maintains all PgQ databases in one PostgreSQL instance. It will auto-detect both databases and whether they have PgQ installed. This also makes core PgQ usable without need for Python. -* New cascaded dispatcher script. Previous 3 dispatcher scripts - (bulk_loader, cube_dispatcher, table_dispatcher) shared quite - a lot of logic for partitionaing, differing on few details. - So instead of porting all 3 to cascaded consuming, I merged them. - Minor improvements ------------------ @@ -95,55 +93,16 @@ Minor improvements * PgQ does not handle "failed events" anymore. -Open questions --------------- - -* New ticker - - Name for final executable: pgqd, pgq-ticker, or something else? - - Should it serialize retry and maint operations on different dbs? - - Should it serialize ticking? - -* Python modules - - Skytools 3 modules should be parallel installable with Skytools 2. - we decided to solve it via loader module - (like http://faq.pygtk.org/index.py?req=all#2.4[pygtk]). - The question is should we have Skytools-specific loader or more generic. - And what should the name be? - - import skytools_loader - skytools_loader.require('3.0') - - vs +* Skytools 3 modules are parallel installable with Skytools 2. + Solved via loader module (like http://faq.pygtk.org/index.py?req=all#2.4[pygtk]). + + import pkgloader + pkgloader.require('skytools', '3.0') + import skytools - import pkgloader - pkgloader.require('skytools', '3.0') - -* Londiste EXECUTE command - - Should the scripts have ability to inform Londiste of the tables - they operate on, so that nodes that do not have such tables - can ignore the script. - -* Is there good reason not to drop following modules: - - logtriga(), pgq.logtriga() - non-automatic triggers - - cube_dispatcher, table_dispatcher, bulk_loader - they are merged into queue_loader Further reading --------------- -* Skytools 3 todo list: TODO.txt -* QAdmin design and todo list: qadmin.txt -* Technical notes about cascading: set.notes.txt -* Notes for contributors: devnotes.txt - -* Python API docs: - - http://skytools.projects.postgresql.org/skytools-3.0/api/[skytools, pgq, londiste modules] - -* Database API docs: - - http://skytools.projects.postgresql.org/skytools-3.0/pgq/[PgQ] - - http://skytools.projects.postgresql.org/skytools-3.0/pgq_node/[Cascaded PgQ] - - http://skytools.projects.postgresql.org/skytools-3.0/pgq_coop/[Cooperative PgQ] - - http://skytools.projects.postgresql.org/skytools-3.0/londiste/[Londiste] - -* Londiste Demo: - - http://skytools.projects.postgresql.org/skytools-3.0/demo.html[] +* http://skytools.projects.postgresql.org/skytools-3.0/[Documentation] for skytools3. |