Various doc updates in preparation for alpha.

6 files changed, 266 insertions, 226 deletions

diff --git a/doc/Makefile b/doc/Makefile
index a2578c7e..a9c0c030 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -11,7 +11,7 @@ EPYARGS = --no-private --url="http://pgfoundry.org/projects/skytools/" \
 
 HTMLS = londiste.cmdline.html londiste.config.html README.html INSTALL.html \
 	londiste.ref.html TODO.html pgq-sql.html pgq-admin.html pgq-nodupes.html \
-	$(SCRIPT_HTMLS) faq.html set.notes.html newadm.html
+	$(SCRIPT_HTMLS) faq.html set.notes.html newadm.html skytools3.html
 
 SCRIPT_TXTS = walmgr.txt cube_dispatcher.txt table_dispatcher.txt \
 	      queue_mover.txt queue_splitter.txt bulk_loader.txt \
@@ -97,14 +97,18 @@ endif
 
 ifneq ($(ASCIIDOC),)
 %.xml: %.txt $(COMMON)
-	$(ASCIIDOC) -b docbook -d manpage `$(GETATTRS) $<` -o - $< \
+	$(ASCIIDOC) -b docbook `$(GETATTRS) $<` -o - $< \
 	| python fixman.py > $@
 
 %.1: %.xml
 	$(XMLTO) man $<
 
 %.html: %.txt $(COMMON)
-	$(ASCIIDOC) -a toc `$(GETATTRS) $<` $<
+	LANG=C cat $< \
+	| sed -e '/^include/b;s,\([A-Za-z.0-9]*\)[.]txt,link:\1.html[],g' \
+	      -e 's,http:[!-~]*,&[],g' \
+	| $(ASCIIDOC) -a toc `$(GETATTRS) $<` -o $@ -
+	#$(ASCIIDOC) -a toc `$(GETATTRS) $<` $<
 
 README.html: ../README
 	cat $< \
diff --git a/doc/TODO.txt b/doc/TODO.txt
index c76454c0..25b537ba 100644
--- a/doc/TODO.txt
+++ b/doc/TODO.txt
@@ -1,143 +1,126 @@
 
 = Skytools ToDo list =
 
-== Next major release - 3.0 ==
+== Required for 3.0-alpha ==
 
-=== Open Questions ===
+* copy_event check
+* pgq.CascadedConsumer event processing
+* londiste.ref.txt update with new commands?
+  or at least demo for new londiste functionality.
 
- * Ticker in C for all db's in one Postgres instance?
- * seq / watermark functions to ticker?
- * pgq.CascadedConsumer event processing
+== Required for 3.0-final ==
 
-=== Standalone changes ===
+* londiste check
+* londiste fkeys
+* londiste globbing
+* Refresh default .ini files, add comments.
+  Alternative: have defailt .ini in script docstring,
+  have DBScript switch to dump it.
 
- - copy_event check
+* dead node handling / failover.
+* pk vs table exists check - missing tables gets pk error
 
- - usr/share default .ini files up-to-date
- - drop_queue(force)
- - public vs. local connect string
- - node connstr change
- - unregister_node / drop_node
- - pk vs table exists check - missing tables gets pk error
+* new ticker:
+  - logging
+  - daemonizing
+  - error handling
+  - config file?
 
- * Make Londiste support table wildcards.
- * plpgsql trigger for TRUNCATE / support for installing such trigger
- * Convert dispatcher scripts to new cascading framework
-   - queue_mover - CascadedWorker (cascaded_worker.py?  node_worker.py)
-   - queue_splitter - CascadedConsumer
-   - bulk_loader - CascadedConsumer
-   - cube_dispatcher - CascadedConsumer
-   - table_dispatcher - CascadedConsumer
- * bulk_loader / cube_dispatcher / table_dispatcher could be merged
-   to one script with 3 backends.
- * 'Q' event type for londiste, (queue_splitter event), for event
-   that needs to be inserted into queue.  Also trigger flag to
-   create such event.
-   - better to be done as "redirect" - allow events for a table
-     to be redirected to another table or queue.
- * londiste check
- * londiste fkeys
+== Good to have changes ==
 
-=== Core changes ===
+=== sql/pgq ===
 
-ASAP:
+* drop_queue(force) - to drop consumers
+* pgq_node.is_root_event() rettype
 
- - move leaf event copy away from consumer/worker?
+=== python/skytools ===
 
- - logging from db is mess, needs full cleanup
- - docstring review
- - dead node handling
- - failover - may affect db code
+* DBScript: db error should show conn name
+* sleeps while waiting notices from db
+* exec_cmd better name
 
- - dbscript: document self.args
- - dbscript: easier help string setting
+=== python/pgq ===
 
-Soon:
+* public vs. local connect string
+* node connstr change
+* unregister_node / drop_node
+* make everything use next_batch_info()
+* pgq.fetch_batch_cursor(cursor_name, limit) returns setof event.
+  If returns less than limit, cursor is closed, all events done.
+  Otherwise the cursor is kept open and consumer can read from it.
 
- - pgq_node.is_root_event() rettype
- - exec_cmd better name
- - cleanup of hierarchical remove_event
+=== python/londiste ===
 
- - make everything use next_batch_info()
- - londiste sql: fq names?  glob names?
- - CascadeAdmin: job_name vs. consumer_name as worker_name
- - setadm rename-node
+* Make Londiste support table wildcards.
+* 'Q' event type for londiste, (queue_splitter event), for event
+  that needs to be inserted into queue.  Also trigger flag to
+  create such event.
+  - better to be done as "redirect" - allow events for a table
+    to be redirected to another table or queue.
+* --wait/--nowait switch for execute, to wait until script is applied
+  to everywhere.
+* reorg vs. copy: Check what happens if some table is in copy but reorg is attempted.
+* execute vs. copy: needs wait?
 
-No hurry:
 
- - psycopgwrapper full doc
- - londiste syncer: get provider db?
- - sleeps while waiting notices from db
- - dispatcher scripts: no need to check tables repeatedly
- - replace "raise Exception" for user errors with something
-   nicer, (UsageError) which also should avoid traceback
+=== sql/londiste ===
 
- - copy vs. reorg
- - execute vs. copy - needs wait?
- - --wait/--nowait switch for execute
+* plpgsql trigger for TRUNCATE / support for installing such trigger
 
-=== done ===
+== Doc changes ==
 
- - Move remaining sql functions away from ret types.
- - get rid of 'set' in londiste code?
- - londiste: add-seq refresh from provider
+* logging from db is mess, needs full cleanup
+* docstring review
+* dbscript: document self.args
+* dbscript: easier help string setting
+* Document DB-API and psycopg2 details a bit under psycopgwrapper.
+* replace "raise Exception" with "raise UsageError" , where we just
+  want to inform user and so traceback is not needed
 
-== old todo (not up-to-date) ==
-
- * pgqadm ticker delays, separate retry delay
- * pgq.drop_queue -> flag to drop consumers
-
- * cascaded replication, switchover, failover [marko]
-   - advanced admin commands
-     - failover
-     - node-status
-     - set-status
-     - rename-node
-     - on root switch/failover check if all tables are present
 
-== High-prority ==
+== old todo (not up-to-date) ==
 
-=== Smaller things ===
+* CascadeAdmin: job_name vs. consumer_name as worker_name
+* londiste syncer: get provider db?
+* dispatcher scripts: no need to check tables repeatedly
 
- * pgqadm: Utility commands:
-  - reg-copy que cons1 cons2
-  - reg-move que cons1 cons2
-  - queue-rename
-  - show-batch-events
-  - del-event
- * londiste: londiste status cmd
-  - check what modules are installed
-  - check if replay is running (lag?)
-  - check if copy is running (lag?)
-  - count of tables done, count of unsynced
-  - table in the middle of copy
+* cascaded replication, switchover, failover
+  - node-status
+  - set-status
+  - rename-node
+  - on root switch/failover check if all tables are present
 
 == Low-priority ==
 
 === Larger things ===
 
- * skylog: publish sample logdb schema, with some tools
- * londiste/pgqset: support creating slave from master by pg_dump / PITR.
+* skylog: publish sample logdb schema, with some tools
+* londiste: support creating slave from master by pg_dump / PITR.
+* pgq/cascade: rename node
 
 === Smaller things ===
 
- * own async conn-to-conn copy loop
- * londiste: make sure unprivileged provider connection works for ordinery replication,
-   but not for admin commands.  Need to remove SECURITY DEFINER from londiste
-   admin functions (both provider and subscriber, because londiste needs admin
-   rights on subscriber anyway).
- * pgqadm: separate priod for retry queue processing
- * skytools: switch for silence for cron scripts
- * DBScript: failure to write pidfile should be logged (cronscripts)
- * ideas from SlonyI:
+* londiste: londiste status cmd
+ - check what modules are installed
+ - check if replay is running (lag?)
+ - check if copy is running (lag?)
+ - count of tables done, count of unsynced
+ - table in the middle of copy
+
+* We need own async conn-to-conn copy loop in Python/PythonC.
+  Currently we simply pipe simply one copy_to() to another copy_from(),
+  but that likely halves the potential throughput.
+* pgqadm/new-ticker: separate priod for retry queue processing
+* skytools: switch (-q) for silence for cron/init scripts
+* DBScript: failure to write pidfile should be logged (cronscripts)
+* ideas from SlonyI:
   - when buffering queries, check their size
- * pgqadm: show count of events to be processed [--count switch].
-  - broken by retry events, rollbacked transactions and use of force_tick()
- * automatic "repair" - after reaching sync point, the "replay" must be killed/paused, then fixes can be applied
+* pgqadm: show count of events to be processed [--count switch].
+* automatic "repair" - after reaching sync point, the "replay" must be killed/paused, then fixes can be applied
 
 == Just ideas ==
 
- * skytools: config from database
- * skytools: config-less operation?
- * londiste: somehow automatic sync of table structure/functions/...?
+* skytools: config from database
+* skytools: config-less operation?
+* londiste: somehow automatic sync of table structure/functions/...?
 
diff --git a/doc/londiste.ref.txt b/doc/londiste.ref.txt
index f8d5b13a..5524f84e 100644
--- a/doc/londiste.ref.txt
+++ b/doc/londiste.ref.txt
@@ -6,8 +6,8 @@
 === PgQ daemon ===
 
 Londiste runs as a consumer on PgQ.  Thus `pgqadm.py ticker` must be running
-on provider database.  It is preferable to run it in same machine, because
-it needs low latency, but that is not a requirement.
+on provider database.  It is preferable to run ticker on same machine as database,
+because it needs low latency, but that is not a requirement.
 
 For monitoring you can use `pgqadm.py status` command.
 
@@ -17,9 +17,9 @@ Londiste internally uses table names always fully schema-qualified.
 If table name without schema is given on command line, it just
 puts "public." in front of it, without looking at search_path.
 
-=== PgQ events ===
+=== PgQ events used ===
 
-==== Table change event ====
+==== Table data change event in SQL format ====
 
 Those events will be inserted by triggers on tables.
 
@@ -33,18 +33,50 @@ Those events will be inserted by triggers on tables.
 Such partial SQL format is used for 2 reasons - to conserve space
 and to make possible to redirect events to another table.
 
-==== Registration change event ====
+==== Table data change event in urlencoded format ====
 
-Those events will be inserted by `provider add` and `provider remove`
-commands.  Then full registered tables list will be sent to the queue
-so subscribers can update their own registrations.
+Those events will be inserted by triggers on tables.
+
+ * ev_type = 'I' / 'U' / 'D' + ':' + list of pkey columns
+   Eg: I:lastname,firstname
+ * ev_data = urlencoded values of all columns of the row.
+   NULL is signified by omitting '=' after column name.
+ * ev_extra1 = table name with schema
+
+Urlencoded events take more space that SQL events, but are more
+easily parseable by other scripts.
+
+==== Table addition event ====
+
+This event will be inserted by 'londiste add-table' on root.
+
+ * ev_type = 'londiste.add-table'
+ * ev_data = table name
+
+All subscribers downstream will also registister this table 
+as being available on the queue.
+
+==== Table removal event ====
+
+This event will be inserted by 'londiste remove-table' on root.
+
+ * ev_type = 'londiste.remove-table'
+ * ev_data = table name
 
- * ev_type = 'T'
- * ev_data = comma-separated list of table names.
+All subscribers downstream will now unregistister this table 
+as being available on the queue.  If they happen to be subscriber
+to this table locally, the table is unsubscribed.
 
-Currently subscribers only remove tables that were removed from provider.
-In the future it's possible to make subscribers also automatically add
-tables that were added on provider.
+==== SQL script execution event ====
+
+This event is inserted by 'londiste execute' on root node.
+The insert happens in same TX as the actual command are executed.
+
+ * ev_type = 'EXECUTE'
+ * ev_data = script body
+ * ev_extra1 = script name
+
+Script name is used as unique ID - the event is registered as applied.
 
 == log file ==
 
@@ -52,13 +84,21 @@ Londiste normal log consist just of statistics log-lines, key-value
 pairs between `{}`.  Their meaning:
 
  * count: how many event was in batch.
- * ignored: how many of them was ignores - table not registered on subscriber or not yet in sync.
- * duration: how long the batch processing took.
+ * ignored: how many of them was ignored - table not registered on subscriber or not yet in sync.
+ * duration: how long the batch processing took, in seconds.
 
 Example:
 
   {count: 110, duration: 0.88}
 
+
+
+
+
+
+
+
+
 == Commands for managing provider database ==
 
 === provider install ===
diff --git a/doc/newadm.txt b/doc/newadm.txt
index 58942f40..67f4ba23 100644
--- a/doc/newadm.txt
+++ b/doc/newadm.txt
@@ -119,4 +119,15 @@ Information:
    - Implement command under newadm in non-DBScript way and make
      CascadeAdmin call newadm implementation.
 
+* Londiste: It seems to be good idea to make Londiste management also available in newadm.
+  - LONDISTE ADD|REMOVE TABLE <tbl>
+  - LONDISTE ADD|REMOVE SEQUENCE <seq>
+  - LONDISTE SHOW TABLES / SHOW LONDISTE TABLES / SHOW TABLES ?
+  - SHOW MISSING TABLES?
+
+* random other ideas:
+  - queue rename.  (on remote side too?)
+  - copy consumer pos
+  - rename consumer
+  - event del?
 
diff --git a/doc/set.notes.txt b/doc/set.notes.txt
index 57d2960c..8feb44c9 100644
--- a/doc/set.notes.txt
+++ b/doc/set.notes.txt
@@ -1,45 +1,43 @@
 
-= Cascaded replication =
-
-== Goals ==
-
-* Main goals:
-  - Nodes in set are in sync - that means they can change their provider to
-    any other node in set.
-  - Combining data from partitioned plproxy db's.
-
-* Extra goals:
-  - Queue-only nodes.  Cannot be initial providers to other nodes, because initial COPY cannot be done.
-  - Data-only nodes, without queue - leafs.  Cannot be providers to other nodes.
-
-* Extra-extra goals:
-  - The node that combines events from partitions can be queue-only.
+= Technical notes about cascaded queuing =
 
 == Termins ==
 
 set::
-  Group of nodes that synchronise set of tables using same data (same events, batch boundaries and tick_ids).
+  Group of nodes that distribute a single queue.  In addition to
+  copying events around, they also keep same batch boundaries
+  and tick_ids.
 
 node::
-  A database that participates in a set.
-  - contains tables which are updated by replaying events from queue.
-  - contains queue which has events from.
+  A database that participates in cascaded copy of a queue.
 
 provider::
-  Node that provides data to another.
+  Node that provides queue data to another.
 
 subscriber::
-  Node that receives data from another.
+  Node that receives queue data from another.
+
+== Goals ==
+
+* Main goals:
+  - Nodes share same queue structure - in addition to events, also
+    batches and their tick_ids are same.  That means they can change
+    their provider to any other node in set.
+  - (Londiste) Queue-only nodes.  Cannot be initial providers to other nodes, because initial COPY cannot be done.
+
+* Extra goals:
+  - Combining data from partitioned plproxy db's.
+  - (Londiste) Data-only nodes, without queue - leafs.  Cannot be providers to other nodes.
 
 == Node types ==
 
 root::
-  current master for set data.
+  Place where queue data is generated.
 
 branch::
-  * carries full contents of the queue.
-  * may subscribe to all/some/none of the tables.
-  * can be provider for initial copy only if subscribes to table
+  * Carries full contents of the queue.
+  * (Londiste) May subscribe to all/some/none of the tables.
+  * (Londiste) Can be provider for initial copy only if subscribes to table
 
 leaf::
   Data-only node.  Events are replayed, but no queue, thus cannot be provider to other nodes.
@@ -50,18 +48,18 @@ merge-leaf::
   [Does not exist as separate type, detected as 'leaf' that has 'combined_queue' set.]
   Exists in per-partition set.
   - Does not have it's own queue.
-  - Initial COPY is done with --skip-truncate,
+  - (Londiste) Initial COPY is done with --skip-truncate,
   - Event data is sent to combined queue.
   - tick_id for each batch is sent to combined queue.
 
-  - replica from part to combined-failover must lag behind combined set coming
-    from combined-root
+  - Queue reader from partition to combined-failover must lag behind
+    combined queue coming from combined-root
   
 combined-root::
   [Does not exist as separate type, detected as 'root' that has 'leaf's with 'combined_queue' set.]
-  - Master for combined set.  Received data from several per-partition set's.
-  - also is merge-leaf in every part-set.
-  - queue is filled directly from partition sets.
+  - Master for combined queue.  Received data from several per-partition queues.
+  - Also is merge-leaf in every per-partition queue.
+  - Queue is filled directly from partition queues.
 
 combined-failover::
   [Does not exist as separate type, detected as 'branch' that has 'leaf's with 'combined_queue' set.]
@@ -91,7 +89,7 @@ combined-failover::
 * Ticks+data can only be deleted if all nodes have already applied it.
   - Special consumer registration on all queues - ".global_watermark".
     This avoids PgQ from deleting old events.
-  - Nodes propagate upwards their lowest tick: (local_watermark)
+  - Nodes propagate upwards their lowest tick: local_watermark
   - Root sends it's local watermark as "pgq.global-watermark" event to the queue.
   - When branch/leaf gets new watermark event, it moves the ".global_watermark" registration.
 
@@ -138,65 +136,3 @@ On the loss of S3, it should be possible to direct S4 to receive data from S1/S2
 On the loss of S1, it should be possible to redirect S3 to S2
 and ABCD -> S2 -> S3 must stay in sync.
 
-== UI spec (incomplete) [obsolete] ==
-
-Here should be end-user interface specced.
-
- setmgr.py <set.ini> create-node TYPE NODE_NAME  NODE_LOCATION --provider=<provider_connstr>
- setmgr.py <set.ini> change-provider <node> --provider <node>
- setmgr.py <set.ini> set-dead <node>
- setmgr.py <set.ini> set-live <node>
- setmgr.py <set.ini> pause --node=<node> [--consumer=]
- setmgr.py <set.ini> resume --node=<node> [--consumer=]
-
-
-
- londiste.py <node.ini> add-table <tbl> ....
- londiste.py <node.ini> remove-table <tbl> ....
- londiste.py <node.ini> add-seq <seq> ....
- londiste.py <node.ini> remove-seq <seq> ....
- londiste.py <node.ini> tables
- londiste.py <node.ini> seqs
- londiste.py <node.ini> missing --seqs --tables
- londiste.py <node.ini> check
- londiste.py <node.ini> fkeys
- londiste.py <node.ini> triggers
- londiste.py <node.ini> tables
- londiste.py <node.ini> seqs
- londiste.py <node.ini> resync TBL
- londiste.py <node.ini> compare TBL
- londiste.py <node.ini> repair TBL
-
- ** actions must be resumable **
-
- setmgr.py <set.ini> init-master ( <node>  --connstr <connstr> | --infoset <infoset> )
-  - install pgq, pgq_set
-  - check if db is already in set
-  - add member to set
-  - create_node()
- setmgr.py <set.ini> init-node <node> --type <type> --connstr <connstr> --provider <node>
-  - install pgq, pgq_set
-  - check if db is alreeady in set
-  - add to node master member list
-  - fetch full list of member from master
-  - add full list to node
-  - subscribe to provider
-  - create_node
- setmgr.py <set.ini> change-provider <node> --provider <node>
-  - pause node, wait until reacts
-  - fetch pos from old provider
-  - subscribe to new provider
-  - change info in subscriber
-  - resume script
-  - unregister in old provider
- setmgr.py <set.ini> set-dead <node>
- setmgr.py <set.ini> set-live <node>
- setmgr.py <set.ini> pause <node>
- setmgr.py <set.ini> resume <node>
- setmgr.py <set.ini> master-failover <newmaster>
- setmgr.py <set.ini> master-switchover <newmaster>
- setmgr.py <set.ini> node-failover <newnode>
- setmgr.py <set.ini> node-switchover <newnode>
- setmgr.py <set.ini> candidates <node>
-
-
diff --git a/doc/skytools3.txt b/doc/skytools3.txt
index 56dd36f0..2094f3f7 100644
--- a/doc/skytools3.txt
+++ b/doc/skytools3.txt
@@ -13,8 +13,8 @@ Keep old design from Skytools 2
   - Can go down anytime, without affecting anything else
 * Relaxed attitude about tables
   - Tables can be added/removed any time.
-  - Inital sync happens table-by-table, no attempt is made to keep
-    consistent picture between tables when syncing.
+  - Inital data sync happens table-by-table, no attempt is made to keep
+    consistent picture between tables during initial copy.
 
 New features in Skytools 3
 --------------------------
@@ -22,25 +22,91 @@ New features in Skytools 3
 * Cascading is implemented as generic layer on top of PgQ - *Cascaded PgQ*.
   - Its goal is to keep identical copy of queue contents in several nodes.
   - Not replication-specific - can be used for any queue.
+  - Advanced admin operations: switchover, failover, change-provider, pause/resume.
 
 * New Londiste features:
   - Parallel copy - during inital sync several tables can be
     copied at the same time.   In 2.1.x the copy already happened in separate
     process, making it parallel was just a matter of tuning launching/syncing logic.
 
-  - EXECUTE command, to run random SQL script on all nodes.
+  - EXECUTE command, to run random SQL script on all nodes.  The script is executed
+    in single TX on root, and insterted as event into queue, in the same TX.
+    Bascially it emulates DDL AFTER TRIGGER that way.
+
+    Londiste does no locking and no coordination between nodes.  The assumption
+    is that the DDL commands itself do enough locking.
 
   - Automatic table or sequence creation by importing the structure
     from provider node.  Activeted with --create switch for add-table, add-seq.
 
-* Advanced admin operations:
-  - switchover
-  - failover
-  - change provider for node
-  - pause / resume node
-  - rename node
+    By default *everything* is copied, including Londiste own triggers.
+    The basic idea is that the triggers may be customized and that way
+    we avoid the need to keep track of trigger customizations.
+
+  - Ability to merge replication queues coming from partitioned database.
+
+  - Now it uses the intelligent log-triggers by default.  The triggers
+    were introduced in 
+
+* New interactive admin console.
+
+* New multi-database ticker.  It is possible to set up one process that
+  maintains all PgQ databases in one PostgreSQL instance.  It will
+  auto-detect both databases and whether they have PgQ installed.
+
+* New cascaded dispatcher script.
+
+Minor improvements
+------------------
+
+* sql/pgq: ticks also store last sequence pos with them.  This allowed
+  also to move most of the ticker functionality into database.  Ticker
+  daemon now just needs to call SQL function periodically, it does not
+  need to keep track of seq positions.
+
+* sql/pgq: Ability to enforce max number of events that one TX can insert.
+  In addition to simply keeping queue healthy, it also gives a way to
+  survive bad UPDATE/DELETE statements with buggy or missing WHERE clause.
+
+* sql/pgq: If Postgres has autovacuum turned on, internal vacuuming for
+  fast-changing tables is disabled.
+
+* python/pgq: pgq.Consumer does not register consumer automatically,
+  cmdline switches --register / --unregister need to be used for that.
+
+* londiste: sequences are now pushed (into queue), instead pulled,
+  from root node.  This reduces load on root and also allows
+  in-between nodes that do not have sequences.
+
+* psycopg1 is not supported anymore.
+
+Open questions
+--------------
+
+* New ticker
+  - Should it use libevent-based async or multi-threaded architecture?
+  - Name for final executable: pgqd, pgq-ticker, or something else?
+
+* Python modules
+  - Skytools 3 modules should be parallel installable with Skytools 2.
+    we decided to solve it via loader module (pygtk does).  The question
+    is 
+
+* PgQ Cascading
+  - There are some periodic operations that should be done on root node.
+    Should we let the ticker do them or do we require a Londiste daemon there?
+
+* Londiste EXECUTE command
+  - Should the scripts have some sort of 
+
+* Newadm:
+  - Good name for it.
+
 
-Example session
+Further reading
 ---------------
 
+* Skytools 3 todo list: TODO.txt
+* Newadm design and todo list: newadm.txt
+* Technical notes about cascading: set.notes.txt