postgres-xc-developers Mailing List for Postgres-XC (Page 136)

I am sending an updated patch. The previous version mistakenly treated 
some variations as a single-step statement when it should not have.

Mason

On 11/23/10 3:15 PM, Mason Sharp wrote:
> I am sending this out to allow for feedback.
>
> This patch adds support for INSERT SELECT (including "multi-step" 
> queries).
>
> This is done by utilizing COPY. We query the data nodes with the tuples
> being sent to the Coordinator. These are then converted into COPY lines
> and sent back down to the appropriate nodes. (Long term when we add
> data node to data node communication, these can be sent directly).
>
> We also optimize for the case when the SELECT is single-step, the
> destination table is partitioned, the input column value comes from
> the partition column of the source, and there are no limit or offset
> clauses. In this case, we just pass down the INSERT SELECT to the data
> nodes.
>
> There is one kluge here in that I added a static variable for the
> copy state. I did this to avoid having to move the CopyState definition
> to a header file (for usage in execMain.c), in order to make merging
> with vanilla PostgreSQL easier.
>


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

We should output an error message as well if the drop failed. In addition, since it cannot be wrapped in 2PC, we may have to consider a utility to cleanup partially dropped databases, but execute direct could do that, we just need to document it, and fix execute direct.


Sent from my IPhone

On Nov 26, 2010, at 5:53 AM, Michael Paquier <mic...@gm...> wrote:

> This seems to be linked with the pooler connections.
> Drop database does not work correctly if you don't empty pool before dropping it.
> 
> You should do something like:
> \c postgres
> clean connection to all for database dbt1;
> drop database dbt1;
> create database dbt1;
> 
> Have a look at the commit 54a648a3ffe7f8fc0273295ee17e91cebcb34948 in git repository for more details.
> 
> I admit that we need to call automatically pool cleaning when dropping a database.
> This is not done yet. We should commit a fix soon to support that.
> 
> Regards,
> 
> -- 
> Michael Paquier
> http://michaelpq.users.sourceforge.net
> 
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

This seems to be linked with the pooler connections.
Drop database does not work correctly if you don't empty pool before
dropping it.

You should do something like:
\c postgres
clean connection to all for database dbt1;
drop database dbt1;
create database dbt1;

Have a look at the commit 54a648a3ffe7f8fc0273295ee17e91cebcb34948 in git
repository for more details.

I admit that we need to call automatically pool cleaning when dropping a
database.
This is not done yet. We should commit a fix soon to support that.

Regards,

-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

Dears,

If I drop a database, it seems that content of database isn't really
deleted.

The steps as follows:
1.postgres=# drop database dbt1;
DROP DATABASE
2.postgres=# create database dbt1;
CREATE DATABASE
3.postgres=# \c dbt1
psql (8.4.3)
You are now connected to database "dbt1".
4.dbt1=# create table t(a int);
CREATE TABLE
5.dbt1=# insert into t values(1);
INSERT 0 1
6.dbt1=# \c postgres
psql (8.4.3)
You are now connected to database "postgres".
7.postgres=# drop database dbt1;
DROP DATABASE
8.postgres=# create database dbt1;
CREATE DATABASE
9.postgres=# \c dbt1
psql (8.4.3)
You are now connected to database "dbt1".
10.dbt1=# \d
        List of relations
 Schema | Name | Type  |  Owner
--------+------+-------+----------
 public | t    | table | postgres
(1 row)
11.dbt1=# select * from t;
 a
---
 1
(1 row)


Regards,
Benny

>
>
>
>>
>> I can also see in your results that method 1 is using in total 2
>> connections between the loaders and coordinators.
>> Method 2 is using 4 connections.
>> I think you did so but... Did you include for method 2 the results of
>> loader1-coordinator1 + loader1-coordinator2 for loader1 results?
>> Same for loader2?
>>
> As I know one loader can get one DBT1 result. As you told, a coordinator
> should get one DBT1 result beacuse you said I should use
> loader1-coordinator1 + loader1-coordinator2 as the result of method2. I don'
> t know how can I get DBT1 results according to coordinator.
>
>
If you use one DBT-1 folder for two instances to coordinators from a loader,
both instances try to write at the same time in the same mix.log, what could
result in a loss of data.
Why not separating the instances by launching each loaderX-coordinatorX
instance in a different DBT-1 folder?
In the case of method 2, create 2 DBT-1 folders on each loader server, for a
total of 4 folders.
You will have 4 mix.log file for the output of:
- loader1/coordinator1
- loader1/coordinator2
- loader2/coordinator1
- loader2/coordinator2

If you do that, you have just to combine mix.log files of
loader2/coordinator1 and loader2/coordinator2 to get the output of a loader
2 for instance.
It may be also interesting to get the output from Coordinator 1 and 2, by
combining for instance log files of loader1/coordinatorX and
loader2/coordinatorX.

-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

Hi, Michael

Thank you for your advice.


2010/11/25 Michael Paquier <mic...@gm...>

> I have a comment about some of the parameters you are using with DBT-1.
>
> #eu
> 1500
> #eu/min
> 1000
>
> The current algorithm of DBT-1 just recognizes multiple numbers when
> calculating ramp-up value.
> For example, if you set eu at 1500, you have to set eu/min at 500, 750 or
> 1500 to have a ramp up time of 1, 2 or 3 minutes.
> If you don't do that ramp up is set I set at 0, what might give you wrong
> results.
>
> Then, you seem still have a lot of idle CPU.
> You should increase eu by 15%~20% to saturate your cpu.
>
> I would recommend to set eu at 2000~2200 for a eu/min at 500~550.
>
I will test it again later.


>
> I can also see in your results that method 1 is using in total 2
> connections between the loaders and coordinators.
> Method 2 is using 4 connections.
> I think you did so but... Did you include for method 2 the results of
> loader1-coordinator1 + loader1-coordinator2 for loader1 results?
> Same for loader2?
>
As I know one loader can get one DBT1 result. As you told, a coordinator
should get one DBT1 result beacuse you said I should use
loader1-coordinator1 + loader1-coordinator2 as the result of method2. I don'
t know how can I get DBT1 results according to coordinator.

Thanks again.

Regards,
Benny

>
> --
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>
>

Hi,

I compared your result with ours and I found your environment consumes
more system CPU (in our case, with 90% workload, system CPU is around 18
to 19%) and also your result idle % (14% and 17%) is a bit higher than
your case.   In our case, user CPU is around 70%, system around 19% and
the idle around was 10% (we put a workload so that 90% of the server CPU
resource is used).  We can reduce the idle to a few % by charging much
more workload.

Idle time shows you've not put a transaction workload to fully use
coordinator/datanode CPUs.

Please take a look at loader's situation (number of transactions per
sec, etc).  I suppose there could be another thing to put DBT-1
transactions equally to all the coordinators.

Kind Regards;
-----------
Koichi Suzuki

(2010年11月25日 14:25), xiong wang wrote:
> Dears,
> I am really appreciated your response.
> 
> Here is the basic information about my test.
> My configure on DBT1 as follows:
> [appServer]
> #dbconnection-connection from 1 dbdriver to 1 backend, with 4 loaders, 
> each of the 5 coords receives 40 connections
> 10
> #transaction_queue_size
> 1500
> #transaction_array_size
> 1500
> 
> [dbdriver]
> #items
> 1000
> #customers
> 28800
> #eu
> 1500
> #eu/min
> 1000
> #mean think_time
> 0.1
> #run_duration in seconds
> 1200
> #access mode access with access_direct or access_appServer
> access_appServer
> #access clean of order table with access_clean if cleanup, by default 
> disactivated if let empty
> access_clean
> The results by sar as follows:
> Method1:
> 
> *coordinator/datanode 1:*
> 
> CPU %user %nice %system %iowait %steal %idle
> 
> Average: all 58.90 0.00 23.61 0.81 0.00 16.69**
> 
> *coordinator/datanode 2:*
> 
> CPU %user %nice %system %iowait %steal %idle
> 
> Average: all 64.19 0.00 28.67 0.55 0.00 6.58
> 
> Method2:
> 
> *coordinator/datanode 1:*
> 
> CPU %user %nice %system %iowait %steal %idle
> 
> Average: all 59.35 0.00 25.39 0.47 0.00 14.79
> 
> *coordinator/datanode 2:*
> 
> CPU %user %nice %system %iowait %steal %idle
> 
> Average: all 57.71 0.00 24.56 0.55 0.00 17.18
> 
> The average results by DBT1 as follows:
> 
> Method1:
> 
> *loader 1:*
> 
> 1356.6 bogotransactions per second**
> 
> *loader 2:*
> 
> 1843.7 bogotransactions per second
> 
> Method2:
> 
> *loader 1:*
> 
> 757.4 bogotransactions per second**
> 
> *loader 2:*
> 
> 779.3 bogotransactions per second
> 
> It's obvious that there is a big difference between Method1 and Method2. 
> I am courious about why.
> 
> Thanks again.
> 
> Regards,
> 
> Benny
> 
> 
> 2010/11/24 Koichi Suzuki <ko...@in... 
> <mailto:ko...@in...>>
> 
>     Hi, Xiong;
> 
>     Could you tell me CPU and I/O usage you can measure by sar? I'm
>     afraid load balance is not good in Method 2. How many backend did
>     you use in each coordinator? Did you have any warning that
>     connection overflew in data nodes? 
> 
>     Also, how long warm-up did you have?
> 
>     I'll let you know our configuration (sorry please let me have a bit).
> 
>     Regards;
>     ---
>     Koichi Suzuki
> 
> 
>     (2010年11月24日 14:40), xiong wang wrote:
> 
>         Hi Mason,
>         I tested it by 5 PCs.
>         The enviroment as follows:
>         2 PCs, one datanode and one coordinator together on each of them,
>         GTM is on another PC,
>         2 Loaders are on other 2 PCs.
>         Network 1G.
>         I tested Postgres-XC in two methods as follows:
>         Method 1.
>         loader -------- coordinator & datanode
>         \
>         GTM
>         /
>         loader -------- coordinator & datanode
>         Method 2.
>         loader-------- coordinator & datanode
>         \ / \
>         \ / \
>         /\ GTM
>         / \ /
>         / \ /
>         loader -------- coordinator & datanode
>         The DBT1 test results in these two methods are very different.
>         Method 1
>         is much better than Method 2. I don't know why.
>         If I test Postgres-XC in Method 1, the DBT1 performance is close
>         to what
>         the document declares. If I test it in Method 2, the result is much
>         worse than what the document writes. Could you tell me why the two
>         methods have so much effect on the DBT1 performace.
>         Thanks,
>         Regards,
>         Benny
>          >How much worse?
>          >How many physical servers are in each configuration? How is
>         each server
>          >configured in each, with how many data nodes? What kind of
>         network?
>          >Gigabit?
>          >Or was everything on one system? With virtual machines or
>         without and
>          >just using different ports?
>          >Are there errors in the log file (connection limits hit)?
>          >Regards,
>          >Mason
> 
> 
> 
>         ------------------------------------------------------------------------------
>         Increase Visibility of Your 3D Game App& Earn a Chance To Win $500!
>         Tap into the largest installed PC base& get more eyes on your
>         game by
>         optimizing for Intel(R) Graphics Technology. Get started today
>         with the
>         Intel(R) Software Partner Program. Five $500 cash prizes are up
>         for grabs.
>         http://p.sf.net/sfu/intelisp-dev2dev
> 
> 
> 
>         _______________________________________________
>         Postgres-xc-developers mailing list
>         Pos...@li...
>         <mailto:Pos...@li...>
>         https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> 
> 

I have a comment about some of the parameters you are using with DBT-1.
#eu
1500
#eu/min
1000

The current algorithm of DBT-1 just recognizes multiple numbers when
calculating ramp-up value.
For example, if you set eu at 1500, you have to set eu/min at 500, 750 or
1500 to have a ramp up time of 1, 2 or 3 minutes.
If you don't do that ramp up is set I set at 0, what might give you wrong
results.

Then, you seem still have a lot of idle CPU.
You should increase eu by 15%~20% to saturate your cpu.

I would recommend to set eu at 2000~2200 for a eu/min at 500~550.

I can also see in your results that method 1 is using in total 2 connections
between the loaders and coordinators.
Method 2 is using 4 connections.
I think you did so but... Did you include for method 2 the results of
loader1-coordinator1 + loader1-coordinator2 for loader1 results?
Same for loader2?

-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

Dears,

I am really appreciated your response.

Here is the basic information about my test.

 My configure on DBT1 as follows:

[appServer]
#dbconnection-connection from 1 dbdriver to 1 backend, with 4 loaders, each
of the 5 coords receives 40 connections
10
#transaction_queue_size
1500
#transaction_array_size
1500

[dbdriver]
#items
1000
#customers
28800
#eu
1500
#eu/min
1000
#mean think_time
0.1
#run_duration in seconds
1200
#access mode access with access_direct or access_appServer
access_appServer
#access clean of order table with access_clean if cleanup, by default
disactivated if let empty
access_clean

The results by sar as follows:

 Method1:

*coordinator/datanode 1:*

        CPU     %user     %nice   %system   %iowait    %steal     %idle

Average:   all     58.90      0.00     23.61      0.81      0.00     16.69**

*coordinator/datanode 2:*

        CPU     %user     %nice   %system   %iowait    %steal     %idle

Average:   all     64.19      0.00     28.67      0.55      0.00      6.58



Method2:

*coordinator/datanode 1:*

        CPU     %user     %nice   %system   %iowait    %steal     %idle

Average:   all     59.35      0.00     25.39      0.47      0.00     14.79

*coordinator/datanode 2:*

        CPU     %user     %nice   %system   %iowait    %steal     %idle

Average:   all     57.71      0.00     24.56      0.55      0.00     17.18





The average results by DBT1 as follows:



Method1:

*loader 1:*

1356.6 bogotransactions per second**

*loader 2:*

1843.7 bogotransactions per second



Method2:

*loader 1:*

757.4 bogotransactions per second**

*loader 2:*

779.3 bogotransactions per second





It's obvious that there is a big difference between Method1 and Method2. I
am courious about why.



Thanks again.





Regards,



Benny




2010/11/24 Koichi Suzuki <ko...@in...>

> Hi, Xiong;
>
> Could you tell me CPU and I/O usage you can measure by sar?   I'm afraid
> load balance is not good in Method 2.   How many backend did you use in each
> coordinator?   Did you have any warning that connection overflew in data
> nodes?

Also, how long warm-up did you have?
>
> I'll let you know our configuration (sorry please let me have a bit).
>
> Regards;
> ---
> Koichi Suzuki
>
>
> (2010年11月24日 14:40), xiong wang wrote:
>
>>  Hi Mason,
>> I tested it by 5 PCs.
>> The enviroment as follows:
>> 2 PCs,  one datanode and one coordinator together on each of them,
>> GTM is on another PC,
>> 2 Loaders are on other 2 PCs.
>> Network 1G.
>>  I tested Postgres-XC in two methods as follows:
>> Method 1.
>> loader -------- coordinator & datanode
>>                                                      \
>>                                                          GTM
>>                                                      /
>> loader -------- coordinator & datanode
>> Method 2.
>> loader-------- coordinator & datanode
>>           \      /                                    \
>>             \  /                                        \
>>              /\                                             GTM
>>            /   \                                         /
>>          /       \                                     /
>> loader -------- coordinator & datanode
>> The DBT1 test results in these two methods are very different. Method 1
>> is much better than Method 2. I don't know why.
>> If I test Postgres-XC in Method 1, the DBT1 performance is close to what
>> the document declares. If I test it in Method 2, the result is much
>> worse than what the document writes. Could you tell me why the two
>> methods have so much effect on the DBT1 performace.
>> Thanks,
>> Regards,
>> Benny
>>  >How much worse?
>>  >How many physical servers are in each configuration? How is each server
>>  >configured in each, with how many data nodes? What kind of network?
>>  >Gigabit?
>>  >Or was everything on one system? With virtual machines or without and
>>  >just using different ports?
>>  >Are there errors in the log file (connection limits hit)?
>>  >Regards,
>>  >Mason
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Increase Visibility of Your 3D Game App&  Earn a Chance To Win $500!
>> Tap into the largest installed PC base&  get more eyes on your game by
>> optimizing for Intel(R) Graphics Technology. Get started today with the
>> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
>> http://p.sf.net/sfu/intelisp-dev2dev
>>
>>
>>
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>

Hi,

I have additional questions:
What are the parameters you used for the measurement? How many emulated
users? What is the ramp-up time?
Note: ramp-up time can be decided with the parameter called emulated
users/min.
What is the average response time of DBT-1 transactions?
What is the think time, the time a DBT-1 backend waits between receiving a
response and sending its next request?

Based on those parameters, how much output did you get, in term of
transactions per second?
Instead of output, could you give us a rate of how much it became worse?

-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

Hi, Xiong;

Could you tell me CPU and I/O usage you can measure by sar?   I'm afraid 
load balance is not good in Method 2.   How many backend did you use in 
each coordinator?   Did you have any warning that connection overflew in 
data nodes?

Also, how long warm-up did you have?

I'll let you know our configuration (sorry please let me have a bit).

Regards;
---
Koichi Suzuki

(2010年11月24日 14:40), xiong wang wrote:
> Hi Mason,
> I tested it by 5 PCs.
> The enviroment as follows:
> 2 PCs,  one datanode and one coordinator together on each of them,
> GTM is on another PC,
> 2 Loaders are on other 2 PCs.
> Network 1G.
>   I tested Postgres-XC in two methods as follows:
> Method 1.
> loader -------- coordinator & datanode
>                                                       \
>                                                           GTM
>                                                       /
> loader -------- coordinator & datanode
> Method 2.
> loader-------- coordinator & datanode
>            \      /                                    \
>              \  /                                        \
>               /\                                             GTM
>             /   \                                         /
>           /       \                                     /
> loader -------- coordinator & datanode
> The DBT1 test results in these two methods are very different. Method 1
> is much better than Method 2. I don't know why.
> If I test Postgres-XC in Method 1, the DBT1 performance is close to what
> the document declares. If I test it in Method 2, the result is much
> worse than what the document writes. Could you tell me why the two
> methods have so much effect on the DBT1 performace.
> Thanks,
> Regards,
> Benny
>  >How much worse?
>  >How many physical servers are in each configuration? How is each server
>  >configured in each, with how many data nodes? What kind of network?
>  >Gigabit?
>  >Or was everything on one system? With virtual machines or without and
>  >just using different ports?
>  >Are there errors in the log file (connection limits hit)?
>  >Regards,
>  >Mason
>
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App&  Earn a Chance To Win $500!
> Tap into the largest installed PC base&  get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
>
>
>
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Hi Mason,

I tested it by 5 PCs.

The enviroment as follows:

2 PCs,  one datanode and one coordinator together on each of them,
GTM is on another PC,
2 Loaders are on other 2 PCs.
Network 1G.


 I tested Postgres-XC in two methods as follows:

Method 1.

loader -------- coordinator & datanode
                                                     \
                                                         GTM
                                                     /
loader -------- coordinator & datanode


Method 2.
loader-------- coordinator & datanode
          \      /                                    \
            \  /                                        \
             /\                                             GTM

           /   \                                         /
         /       \                                     /
loader -------- coordinator & datanode


The DBT1 test results in these two methods are very different. Method 1 is
much better than Method 2. I don't know why.

If I test Postgres-XC in Method 1, the DBT1 performance is close to what
the document declares. If I test it in Method 2, the result is much worse
than what the document writes. Could you tell me why the two methods have so
much effect on the DBT1 performace.

Thanks,

Regards,
Benny

>How much worse?

>How many physical servers are in each configuration? How is each server
>configured in each, with how many data nodes? What kind of network?
>Gigabit?

>Or was everything on one system? With virtual machines or without and
>just using different ports?

>Are there errors in the log file (connection limits hit)?

>Regards,

>Mason

I am sending this out to allow for feedback.

This patch adds support for INSERT SELECT (including "multi-step" queries).

This is done by utilizing COPY. We query the data nodes with the tuples
being sent to the Coordinator. These are then converted into COPY lines
and sent back down to the appropriate nodes. (Long term when we add
data node to data node communication, these can be sent directly).

We also optimize for the case when the SELECT is single-step, the
destination table is partitioned, the input column value comes from
the partition column of the source, and there are no limit or offset
clauses. In this case, we just pass down the INSERT SELECT to the data
nodes.

There is one kluge here in that I added a static variable for the
copy state. I did this to avoid having to move the CopyState definition
to a header file (for usage in execMain.c), in order to make merging
with vanilla PostgreSQL easier.

-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

In the document, we installed one coordinator and one data node into
one physical server to make best user of both coordinators and data
nodes, independent from the workload shared by coordinators and data
nodes.

It also make parameters simple to claim the sacalability against the
number of servers involved.   I don't thinks it's a good idea to
install loader and coordinator in the same server.   Loader's work is
simply application-oriented and should not be included as a part of
database performance.

As Mason mentioned, I'm very interested in the spec of servers, number
of servers used and configuration information what components (GTM,
coordinators and data nodes) you installed in what servers, as well as
network connections.

As you might have seen, we need gigabit network links between GTM,
Coordinators and Data Nodes.   I strongly recommend to use good L2
switch to reduce the network worklord too.

Kind Regards;
----------
Koichi Suzuki



2010/11/23 Mason Sharp <mas...@en...>:
> On 11/23/10 3:27 AM, xiong wang wrote:
>> Dears,
>>
>> I tested the postgres-xc DBT1 performance according to the published
>> document. But the result is worse than what the document declares. One
>> loader with one coordinator is much better. One loader with two
>> coordinators is much worse than the former. I don't know which way is
>> right. And I don't know the reason why the latter method is so worse
>> than the former.
>>
>>
> How much worse?
>
> How many physical servers are in each configuration? How is each server
> configured in each, with how many data nodes? What kind of network?
> Gigabit?
>
> Or was everything on one system? With virtual machines or without and
> just using different ports?
>
> Are there errors in the log file (connection limits hit)?
>
> Regards,
>
> Mason
>
>> You reply will be appreciated.
>>
>> Thanks.
>>
>> Best regards,
>>
>> Benny
>>
>> ------------------------------------------------------------------------------
>> Increase Visibility of Your 3D Game App&  Earn a Chance To Win $500!
>> Tap into the largest installed PC base&  get more eyes on your game by
>> optimizing for Intel(R) Graphics Technology. Get started today with the
>> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
>> http://p.sf.net/sfu/intelisp-dev2dev
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>
> --
> Mason Sharp
> EnterpriseDB Corporation
> The Enterprise Postgres Company
>
>
> This e-mail message (and any attachment) is intended for the use of
> the individual or entity to whom it is addressed. This message
> contains information from EnterpriseDB Corporation that may be
> privileged, confidential, or exempt from disclosure under applicable
> law. If you are not the intended recipient or authorized to receive
> this for the intended recipient, any use, dissemination, distribution,
> retention, archiving, or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify
> the sender immediately by reply e-mail and delete this message.
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

On 11/23/10 3:27 AM, xiong wang wrote:
> Dears,
>
> I tested the postgres-xc DBT1 performance according to the published
> document. But the result is worse than what the document declares. One
> loader with one coordinator is much better. One loader with two
> coordinators is much worse than the former. I don't know which way is
> right. And I don't know the reason why the latter method is so worse
> than the former.
>
>    
How much worse?

How many physical servers are in each configuration? How is each server 
configured in each, with how many data nodes? What kind of network? 
Gigabit?

Or was everything on one system? With virtual machines or without and 
just using different ports?

Are there errors in the log file (connection limits hit)?

Regards,

Mason

> You reply will be appreciated.
>
> Thanks.
>
> Best regards,
>
> Benny
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App&  Earn a Chance To Win $500!
> Tap into the largest installed PC base&  get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>    


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

Dears,

I tested the postgres-xc DBT1 performance according to the published
document. But the result is worse than what the document declares. One
loader with one coordinator is much better. One loader with two
coordinators is much worse than the former. I don't know which way is
right. And I don't know the reason why the latter method is so worse
than the former.


You reply will be appreciated.

Thanks.

Best regards,

Benny

On 11/21/10 9:13 PM, Michael Paquier wrote:
>
> I made a couple of times the same tests and it looked to work all the 
> time.
> The logs on each node also looked correct.
>
I just committed, after doing a bug fix. The problem appeared to be that 
the list was getting
reassigned when an item is deleted in the foreach loop that depended on it.
I changed that to be more careful.

You guys probably just got lucky in your environment, but without the 
change it would
always crash under MacOS.

I also made a small change to check if the connection to GTM is NULL 
before rolling back.
(The GTM crash was causing a coordinator crash.)

Thanks,

Mason

> Regards,
>
> -- 
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

Hi,

I made the same test as you and didn't notice any errors when dropping the
second database.

By Having a look at the logs of your nodes when you stopped them, it looks
that it entered in recovery mode.
So I assume you stopped it in "fast" mode with pg_ctl or with SIGQUIT.
In this case database shuts down without waiting for the client to
disconnect.

My cluster structure is 2 Coordinators and 2 datanodes, with 2 proxies.
After doing the same SQL, I stopped my nodes in the most violent way.
GTM and proxies were not shut down.

pgsql]$ ./bin/pg_ctl stop -m fast -D ~/pgsql/coord1/
waiting for server to shut down.... done
server stopped
pgsql]$ ./bin/pg_ctl stop -m fast -D ~/pgsql/coord2/
waiting for server to shut down.... done
server stopped
pgsql]$ ./bin/pg_ctl stop -m fast -D ~/pgsql/datanode2/
waiting for server to shut down.... done
server stopped
pgsql]$ ./bin/pg_ctl stop -m fast -D ~/pgsql/datanode1/
waiting for server to shut down.... done
server stopped

Then I restarted the nodes, and dropped a database. It worked.

I did also a second test.
I stopped my nodes, then the proxies (GTM not touched):
pgsql]$ ./bin/gtm_ctl stop -m fast -S gtm_proxy -D
/home/michael/pgsql/proxy2
waiting for server to shut down.... done
server stopped
pgsql]$ ./bin/gtm_ctl stop -m fast -S gtm_proxy -D
/home/michael/pgsql/proxy1
waiting for server to shut down... done
server stopped
I restart the whole cluster, then drop again a dabatase.
It worked once again.

I made a couple of times the same tests and it looked to work all the time.
The logs on each node also looked correct.

Regards,

-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

On 11/18/10 9:44 AM, Michael Paquier wrote:
> This is going to be a pretty long message, but...
> I made a couple of tests with this patch.
>
> Successively I created two databases, sequences on it, then dropped
> the databases.
> I created again and a couple of times databases (and so sequences) and
> didn't notice any error.
> What are the SQL you used for your tests? Is it reproducible?
>
Yes, it is still happening. My sequence of steps was a bit different:

$ inst/bin/createdb -p 5445 mds1
$ inst/bin/createdb -p 5445 mds2
$ inst/bin/psql -p 5445 mds1
psql (8.4.3)
Type "help" for help.

mds1=# create sequence foo;
CREATE SEQUENCE
mds1=# create sequence foo2;
CREATE SEQUENCE
mds1=# \c mds2
psql (8.4.3)
You are now connected to database "mds2".
mds2=# create sequence foo;
CREATE SEQUENCE
mds2=# create sequence foo2;
CREATE SEQUENCE
mds2=# select nextval('foo');
nextval
---------
1
(1 row)

mds2=# select nextval('foo');
nextval
---------
2
(1 row)

mds2=# select nextval('foo2');
nextval
---------
1
(1 row)

mds2=# select nextval('foo2');
nextval
---------
2
(1 row)

mds2=# \q

72:postgres-xc masonsharp$ stopnodes.sh
waiting for server to shut down.... done
server stopped
waiting for server to shut down.... done
server stopped
waiting for server to shut down.... done
server stopped
waiting for server to shut down.... done
server stopped

(this stops the coordinators and datanodes, but not GTM)

72:postgres-xc masonsharp$ startnodes.sh
LOG: database system was interrupted; last known up at 2010-11-18
11:46:52 JST
LOG: database system was not properly shut down; automatic recovery in
progress
LOG: redo starts at 0/4BCF34
LOG: record with zero length at 0/4BCFE4
LOG: redo done at 0/4BCFB8
LOG: last completed transaction was at log time 2010-11-18 11:48:52.44638+09
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
LOG: database system was interrupted; last known up at 2010-11-18
11:46:52 JST
LOG: database system was not properly shut down; automatic recovery in
progress
LOG: redo starts at 0/4BCF34
LOG: record with zero length at 0/4BCFE4
LOG: redo done at 0/4BCFB8
LOG: last completed transaction was at log time 2010-11-18 11:48:52.43121+09
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
14151: 0: LOG: database system was shut down at 2010-11-18 11:49:16 JST
14159: 0: LOG: autovacuum launcher started
14142: 0: LOG: database system is ready to accept connections
14160: 0: LOG: database system was shut down at 2010-11-18 11:49:17 JST
14143: 0: LOG: database system is ready to accept connections
14165: 0: LOG: autovacuum launcher started
72:postgres-xc masonsharp$ inst/bin/psql -p 5445 mds1
psql (8.4.3)
Type "help" for help.

mds1=# drop database mds2;
WARNING: can not connect to GTM: Connection refused
ERROR: Deletion of sequences on database mds2 not completed
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>

Regards,

Mason


> -- create databases
> template1=# create database foo1;
> CREATE DATABASE
> template1=# create database foo2;
> CREATE DATABASE
> template1=# \c foo1
> psql (8.4.5, サバ 8.4.3)
> デタベス "foo1" に接しました。.
> foo1=# create sequence foo;
> CREATE SEQUENCE
> foo1=# create sequence foo2;
> CREATE SEQUENCE
> foo1=# \c foo2
> psql (8.4.5, サバ 8.4.3)
> デタベス "foo2" に接しました。.
> foo2=# create sequence foo;
> CREATE SEQUENCE
> foo2=# create sequence foo2;
> CREATE SEQUENCE
> foo2=# select nextval('foo');
> nextval
> ---------
> 1
> (1 行)
>
> foo2=# select nextval('foo');
> nextval
> ---------
> 2
> (1 行)
>
> foo2=# \c template1
> psql (8.4.5, サバ 8.4.3)
> デタベス "template1" に接しました。.
> template1=# drop database foo1;
> DROP DATABASE
> template1=# clean connection to all for database foo2;
> CLEAN CONNECTION
> template1=# drop database foo2;
> DROP DATABASE
> template1=# create database foo2;
> CREATE DATABASE
> template1=# \c foo2
> psql (8.4.5, サバ 8.4.3)
> デタベス "foo2" に接しました。.
> foo2=# create sequence foo;
> CREATE SEQUENCE
> foo2=# create sequence foo2;
> CREATE SEQUENCE
> foo2=# select nextval('foo');
> nextval
> ---------
> 1
> (1 行)
>
> foo2=# select nextval('foo');
> nextval
> ---------
> 2
> (1 行)
>
> foo2=# select nextval('foo2');
> nextval
> ---------
> 1
> (1 行)
>
> foo2=# select nextval('foo2');
> nextval
> ---------
> 2
> (1 行)
>
> foo2=# \c template1
> psql (8.4.5, サバ 8.4.3)
> デタベス "template1" に接しました。.
> template1=# create database foo1;
> CREATE DATABASE
> template1=# \c foo1
> psql (8.4.5, サバ 8.4.3)
> デタベス "foo1" に接しました。.
> foo1=# create sequence foo;
> CREATE SEQUENCE
> foo1=# create sequence foo2;
> CREATE SEQUENCE
> foo1=# select nextval('foo');
> nextval
> ---------
> 1
> (1 行)
>
> foo1=# select nextval('foo');
> nextval
> ---------
> 2
> (1 行)
>
> foo1=# select nextval('foo2');
> nextval
> ---------
> 1
> (1 行)
> foo1=# \c template1
> psql (8.4.5, サバ 8.4.3)
> デタベス "template1" に接しました。.
> template1=# clean connection to all for database foo1;
> CLEAN CONNECTION
> template1=# clean connection to all for database foo2;
> CLEAN CONNECTION
> template1=# drop database foo1;
> DROP DATABASE
> template1=# drop database foo2;
> DROP DATABASE
> template1=# create database foo1;
> CREATE DATABASE
> template1=# create database foo2;
> CREATE DATABASE
> template1=# \c foo1
> psql (8.4.5, サバ 8.4.3)
> デタベス "foo1" に接しました。.
> foo1=# create sequence foo1;
> CREATE SEQUENCE
> foo1=# create sequence foo2;
> CREATE SEQUENCE
> foo1=# create sequence foo;
> CREATE SEQUENCE
> foo1=# select nextval('foo1');
> nextval
> ---------
> 1
> (1 行)
>
> foo1=# alter sequence foo2 increment 2;
> ALTER SEQUENCE
> foo1=# select nextval('foo2');
> nextval
> ---------
> 1
> (1 行)
>
> foo1=# select nextval('foo2');
> nextval
> ---------
> 3
> (1 行)
>
> foo1=# select nextval('foo2');
> nextval
> ---------
> 5
> (1 行)
>
> -- 
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

This is going to be a pretty long message, but...
I made a couple of tests with this patch.

Successively I created two databases, sequences on it, then dropped the
databases.
I created again and a couple of times databases (and so sequences) and
didn't notice any error.
What are the SQL you used for your tests? Is it reproducible?

-- create databases
template1=# create database foo1;
CREATE DATABASE
template1=# create database foo2;
CREATE DATABASE
template1=# \c foo1
psql (8.4.5, サバ 8.4.3)
デタベス "foo1" に接しました。.
foo1=# create sequence foo;
CREATE SEQUENCE
foo1=# create sequence foo2;
CREATE SEQUENCE
foo1=# \c foo2
psql (8.4.5, サバ 8.4.3)
デタベス "foo2" に接しました。.
foo2=# create sequence foo;
CREATE SEQUENCE
foo2=# create sequence foo2;
CREATE SEQUENCE
foo2=# select nextval('foo');
 nextval
---------
       1
(1 行)

foo2=# select nextval('foo');
 nextval
---------
       2
(1 行)

foo2=# \c template1
psql (8.4.5, サバ 8.4.3)
デタベス "template1" に接しました。.
template1=# drop database foo1;
DROP DATABASE
template1=# clean connection to all for database foo2;
CLEAN CONNECTION
template1=# drop database foo2;
DROP DATABASE
template1=# create database foo2;
CREATE DATABASE
template1=# \c foo2
psql (8.4.5, サバ 8.4.3)
デタベス "foo2" に接しました。.
foo2=# create sequence foo;
CREATE SEQUENCE
foo2=# create sequence foo2;
CREATE SEQUENCE
foo2=# select nextval('foo');
 nextval
---------
       1
(1 行)

foo2=# select nextval('foo');
 nextval
---------
       2
(1 行)

foo2=# select nextval('foo2');
 nextval
---------
       1
(1 行)

foo2=# select nextval('foo2');
 nextval
---------
       2
(1 行)

foo2=# \c template1
psql (8.4.5, サバ 8.4.3)
デタベス "template1" に接しました。.
template1=# create database foo1;
CREATE DATABASE
template1=# \c foo1
psql (8.4.5, サバ 8.4.3)
デタベス "foo1" に接しました。.
foo1=# create sequence foo;
CREATE SEQUENCE
foo1=# create sequence foo2;
CREATE SEQUENCE
foo1=# select nextval('foo');
 nextval
---------
       1
(1 行)

foo1=# select nextval('foo');
 nextval
---------
       2
(1 行)

foo1=# select nextval('foo2');
 nextval
---------
       1
(1 行)
foo1=# \c template1
psql (8.4.5, サバ 8.4.3)
デタベス "template1" に接しました。.
template1=# clean connection to all for database foo1;
CLEAN CONNECTION
template1=# clean connection to all for database foo2;
CLEAN CONNECTION
template1=# drop database foo1;
DROP DATABASE
template1=# drop database foo2;
DROP DATABASE
template1=# create database foo1;
CREATE DATABASE
template1=# create database foo2;
CREATE DATABASE
template1=# \c foo1
psql (8.4.5, サバ 8.4.3)
デタベス "foo1" に接しました。.
foo1=# create sequence foo1;
CREATE SEQUENCE
foo1=# create sequence foo2;
CREATE SEQUENCE
foo1=# create sequence foo;
CREATE SEQUENCE
foo1=# select nextval('foo1');
 nextval
---------
       1
(1 行)

foo1=# alter sequence foo2 increment 2;
ALTER SEQUENCE
foo1=# select nextval('foo2');
 nextval
---------
       1
(1 行)

foo1=# select nextval('foo2');
 nextval
---------
       3
(1 行)

foo1=# select nextval('foo2');
 nextval
---------
       5
(1 行)

-- 
Michael Paquier
http://michaelpq.users.sourceforge.net

I created a couple of databases with a couple of sequences, bounced the 
coordinators and data nodes (not GTM), then if I try and drop a 
database, it causes a crash of both the coordinator and GTM.

mds1=# drop database msd2;
ERROR:  database "msd2" does not exist
mds1=# drop database mds2;
WARNING:  can not connect to GTM: Connection refused
ERROR:  Deletion of sequences on database mds2 not completed
server closed the connection unexpectedly
     This probably means the server terminated abnormally
     before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!> \q

#0  0x907916fa in select$DARWIN_EXTSN ()
(gdb) bt
#0  0x907916fa in select$DARWIN_EXTSN ()
#1  0x00002dd1 in ServerLoop ()
#2  0x00002b7f in main ()

and

(gdb) bt
#0  0x003e54ed in gtmpqPutMsgStart ()
#1  0x003e89fd in abort_transaction ()
#2  0x00077272 in RollbackTranGTM (gxid=1142) at gtm.c:174
#3  0x000583f5 in AbortTransaction () at xact.c:2320
#4  0x00058e33 in AbortCurrentTransaction () at xact.c:2783
#5  0x00058f18 in AbortCurrentTransactionOnce () at xact.c:2911
#6  0x0029aa16 in PostgresMain (argc=4, argv=0x1002ff8, 
username=0x1002fc8 "masonsharp") at postgres.c:3622
#7  0x0025665c in BackendRun (port=0x7016f0) at postmaster.c:3607
#8  0x002559c3 in BackendStartup (port=0x7016f0) at postmaster.c:3216
#9  0x002523f5 in ServerLoop () at postmaster.c:1445
#10 0x00251a01 in PostmasterMain (argc=5, argv=0x7005a0) at 
postmaster.c:1098
#11 0x001cd3a5 in main (argc=5, argv=0x7005a0) at main.c:

We should probably check if the connection is NULL.

Thanks,

Mason

-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

When I tried to test this, I noticed some problem. Maybe it exists 
without the patch though:

mds1=# create sequence myseq2;
CREATE SEQUENCE
mds1=# select nextval('myseq2');
  nextval
---------
        0
(1 row)

mds1=# select nextval('myseq2');
  nextval
---------
        0
(1 row)

Mason

On 11/12/10 1:23 PM, Michael Paquier wrote:
> Hi Benny,
>
> I had a look at your patch,
> and it is really a nice feature you implemented.
>
> With this functionnality, it is possible to drop sequences on GTM in a 
> very smooth way when a database is dropped.
> I have a couple of comments about it though.
>
> 1) in gtm_seq.c, function seq_start_with_nsp
> You make a memory compare that will always return a true result 
> because you compare the same string.
>        return ((memcmp(nsp->gsk_key, nsp->gsk_key, nsp->gsk_keylen) == 0)
> && (seq->gsk_key[nsp->gsk_keylen] == '.'));
> I believe this is the right code:
>        return ((memcmp(nsp->gsk_key, seq->gsk_key, nsp->gsk_keylen) == 0)
> && (seq->gsk_key[nsp->gsk_keylen] == '.'));
>
> 2) in gtm_seq.c, function GTM_RmSeqWithNsp
> When you encounter a sequence that is used by another process, you 
> mark it for deletion and then you exit the process,
> letting perhaps a lot of sequences not dropped.
> I think priority should be given to deletion, as DROP DATABASE has to 
> be forced.
> Btw, a busy sequence cannot basically be found because that would mean 
> that an instance is still using the database,
> and in this case database drop is not possible.
>
> 3) in gtm_c.h, about the message type definition.
> I made a modification in message type so as not to use a unique character.
> Using an enumeration (typedef enum) makes it more consistent with 
> Postgres-xc code, and it makes code maintenance easier.
>
> 4) Also I notice a couple of typo problems, take care when you use 
> spaces and tabs!
>
> I corrected those points above in the patch attached.
> I also changed a couple of function names to make it clearer.
>
> You should check it and make a couple of additional tests also.
> Perhaps you have additional ideas about this implementation, don't 
> hesitate to share what you think.
>
> Regards,
>
> -- 
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>
>
> ------------------------------------------------------------------------------
> Centralized Desktop Delivery: Dell and VMware Reference Architecture
> Simplifying enterprise desktop deployment and management using
> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end
> client virtualization framework. Read more!
> http://p.sf.net/sfu/dell-eql-dev2dev
>
>
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>    


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

Hi Koichi,

Thanks for your kindly notifications. I will pay more attention on
what you mentioned during my later work.

Thanks again.

Regards,
                                          Benny

在 2010年11月16日 下午12:06，Koichi Suzuki <ko...@in...> 写道：
> Hi,
>
> I understand the idea.  I'd just like to be a bit careful what should be
> nice to include as a temporary or a partial solution and what we should
> provide as a general solution.   I understand this patch will be useful
> and people may not be confused.
>
> Cheers;
> ---
> Koichi
>
> (2010年11月16日 12:09), xiong wang wrote:
>> Dears,
>>
>> Thanks for your suggestions.
>>
>> I should clarify my idea.
>> 1)This patch is a temporary solution. It cann't really solve the
>> problem raised by updating on a partition column until the measure
>> that moving tuples from one node to another comes true.
>>     It just forbids users to update on a partition column. Therefore, I
>> didn't consider too much because it will be deleted later.
>> 2)The patch forbids update on partition column on coordinators. It has
>> nothing to do with data nodes.
>>
>> If I misunderstand you, please let me know.
>>
>> Regards,
>> Benny
>>
>>> Hi,
>>
>>> I'm curious about two issues on this patch.
>>
>>> 1) Does it handle TOAST table correctly?
>>> 2) Does it handle SET clause such as SET key = key+1 ?  Only a data node
>> can determine if such UPDATE statement make rows invalid to stay in the
>> data node.   Unfortunately, data node is not equipped with distribution
>> key.  To handle this correctly, data node has to handle TOAST table as
>>> mentioned in 1).
>>
>>> Regards;
>>> ---
>>> Koichi Suzuki
>>
>> (2010年11月15日 23:59), Mason Sharp wrote:
>>> On 11/15/10 5:13 PM, mei le wrote:
>>>> Dears,
>>>>
>>>> I am sorry. The bug#3107683.patch I commited before introduces a warning I didn't noticed. The enclosure is a new patch after I eliminated the warning.
>>>>
>>> Thanks. I just committed this and fixed a couple of other warnings that
>>> we had.
>>>
>>> Regards,
>>>
>>> Mason
>>>> Regards,
>>>>
>>>>                                               Benny
>>>>
>>>>
>>>>
>>>> --- 10年11月12日，周五, Mason Sharp<mas...@en...>   写道：
>>>>
>>>>
>>>> 发件人: Mason Sharp<mas...@en...>
>>>> 主题: Fwd: [ postgres-xc-Bugs-3107683 ] UPDATE on partition column
>>>> 收件人: "Postgres-XC Developers"<pos...@li...>, "mei le"<lem...@ya...>
>>>> 日期: 2010年11月12日,周五,上午10:34
>>>>
>>>>
>>>> Benny,
>>>>
>>>> I thought this might be one you might be interested in working on, too, if you have time.
>>>>
>>>> You could validate the update statement in pgxc_planner() if it is on a partitioned table.
>>>>
>>>> Thanks,
>>>>
>>>> Mason
>>>>
>>>> -------- Original Message --------
>>>>
>>>>
>>>>
>>>> Subject:
>>>> [ postgres-xc-Bugs-3107683 ] UPDATE on partition column
>>>>
>>>> Date:
>>>> Fri, 12 Nov 2010 02:29:56 +0000
>>>>
>>>> From:
>>>> SourceForge.net<no...@so...>
>>>>
>>>> To:
>>>> no...@so...
>>>>
>>>> Bugs item #3107683, was opened at 2010-11-11 21:29
>>>> Message generated for change (Tracker Item Submitted) made by mason_s
>>>> You can respond by visiting:
>>>> https://sourceforge.net/tracker/?func=detail&atid=1310232&aid=3107683&group_id=311227
>>>>
>>>> Please note that this message will contain a full copy of the comment thread,
>>>> including the initial issue submission, for this request,
>>>> not just the latest update.
>>>> Category: Database Server
>>>> Group: None
>>>> Status: Open
>>>> Resolution: None
>>>> Priority: 6
>>>> Private: No
>>>> Submitted By: mason_s (mason_s)
>>>> Assigned to: Nobody/Anonymous (nobody)
>>>> Summary: UPDATE on partition column
>>>>
>>>> Initial Comment:
>>>> Until we support moving tuples from one node to another when the partition column of a table is updated, we should at least block this from happening. At the moment you can update it, and bad things can start happening as a result.
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> You can respond by visiting:
>>>> https://sourceforge.net/tracker/?func=detail&atid=1310232&aid=3107683&group_id=311227
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture
>>>> Simplifying enterprise desktop deployment and management using
>>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end
>>>> client virtualization framework. Read more!
>>>> http://p.sf.net/sfu/dell-eql-dev2dev
>>>>
>>>>
>>>> _______________________________________________
>>>> Postgres-xc-developers mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>
>>>
>>>
>>> --
>>> Mason Sharp
>>> EnterpriseDB Corporation
>>> The Enterprise Postgres Company
>>>
>>>
>>> This e-mail message (and any attachment) is intended for the use of
>>> the individual or entity to whom it is addressed. This message
>>> contains information from EnterpriseDB Corporation that may be
>>> privileged, confidential, or exempt from disclosure under applicable
>>> law. If you are not the intended recipient or authorized to receive
>>> this for the intended recipient, any use, dissemination, distribution,
>>> retention, archiving, or copying of this communication is strictly
>>> prohibited. If you have received this e-mail in error, please notify
>>> the sender immediately by reply e-mail and delete this message.
>>>
>>
>> ------------------------------------------------------------------------------
>> Beautiful is writing same markup. Internet Explorer 9 supports
>> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2&  L3.
>> Spend less time writing and  rewriting code and more time creating great
>> experiences on the web. Be a part of the beta today
>> http://p.sf.net/sfu/msIE9-sfdev2dev
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Hi,

I understand the idea.  I'd just like to be a bit careful what should be
nice to include as a temporary or a partial solution and what we should
provide as a general solution.   I understand this patch will be useful
and people may not be confused.

Cheers;
---
Koichi

(2010年11月16日 12:09), xiong wang wrote:
> Dears,
> 
> Thanks for your suggestions.
> 
> I should clarify my idea.
> 1)This patch is a temporary solution. It cann't really solve the
> problem raised by updating on a partition column until the measure
> that moving tuples from one node to another comes true.
>     It just forbids users to update on a partition column. Therefore, I
> didn't consider too much because it will be deleted later.
> 2)The patch forbids update on partition column on coordinators. It has
> nothing to do with data nodes.
> 
> If I misunderstand you, please let me know.
> 
> Regards,
> Benny
> 
>> Hi,
> 
>> I'm curious about two issues on this patch.
> 
>> 1) Does it handle TOAST table correctly?
>> 2) Does it handle SET clause such as SET key = key+1 ?  Only a data node
> can determine if such UPDATE statement make rows invalid to stay in the
> data node.   Unfortunately, data node is not equipped with distribution
> key.  To handle this correctly, data node has to handle TOAST table as
>> mentioned in 1).
> 
>> Regards;
>> ---
>> Koichi Suzuki
> 
> (2010年11月15日 23:59), Mason Sharp wrote:
>> On 11/15/10 5:13 PM, mei le wrote:
>>> Dears,
>>>
>>> I am sorry. The bug#3107683.patch I commited before introduces a warning I didn't noticed. The enclosure is a new patch after I eliminated the warning.
>>>
>> Thanks. I just committed this and fixed a couple of other warnings that
>> we had.
>>
>> Regards,
>>
>> Mason
>>> Regards,
>>>
>>>                                               Benny
>>>
>>>
>>>
>>> --- 10年11月12日，周五, Mason Sharp<mas...@en...>   写道：
>>>
>>>
>>> 发件人: Mason Sharp<mas...@en...>
>>> 主题: Fwd: [ postgres-xc-Bugs-3107683 ] UPDATE on partition column
>>> 收件人: "Postgres-XC Developers"<pos...@li...>, "mei le"<lem...@ya...>
>>> 日期: 2010年11月12日,周五,上午10:34
>>>
>>>
>>> Benny,
>>>
>>> I thought this might be one you might be interested in working on, too, if you have time.
>>>
>>> You could validate the update statement in pgxc_planner() if it is on a partitioned table.
>>>
>>> Thanks,
>>>
>>> Mason
>>>
>>> -------- Original Message --------
>>>
>>>
>>>
>>> Subject:
>>> [ postgres-xc-Bugs-3107683 ] UPDATE on partition column
>>>
>>> Date:
>>> Fri, 12 Nov 2010 02:29:56 +0000
>>>
>>> From:
>>> SourceForge.net<no...@so...>
>>>
>>> To:
>>> no...@so...
>>>
>>> Bugs item #3107683, was opened at 2010-11-11 21:29
>>> Message generated for change (Tracker Item Submitted) made by mason_s
>>> You can respond by visiting:
>>> https://sourceforge.net/tracker/?func=detail&atid=1310232&aid=3107683&group_id=311227
>>>
>>> Please note that this message will contain a full copy of the comment thread,
>>> including the initial issue submission, for this request,
>>> not just the latest update.
>>> Category: Database Server
>>> Group: None
>>> Status: Open
>>> Resolution: None
>>> Priority: 6
>>> Private: No
>>> Submitted By: mason_s (mason_s)
>>> Assigned to: Nobody/Anonymous (nobody)
>>> Summary: UPDATE on partition column
>>>
>>> Initial Comment:
>>> Until we support moving tuples from one node to another when the partition column of a table is updated, we should at least block this from happening. At the moment you can update it, and bad things can start happening as a result.
>>>
>>> ----------------------------------------------------------------------
>>>
>>> You can respond by visiting:
>>> https://sourceforge.net/tracker/?func=detail&atid=1310232&aid=3107683&group_id=311227
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture
>>> Simplifying enterprise desktop deployment and management using
>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end
>>> client virtualization framework. Read more!
>>> http://p.sf.net/sfu/dell-eql-dev2dev
>>>
>>>
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>
>>
>> --
>> Mason Sharp
>> EnterpriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>> This e-mail message (and any attachment) is intended for the use of
>> the individual or entity to whom it is addressed. This message
>> contains information from EnterpriseDB Corporation that may be
>> privileged, confidential, or exempt from disclosure under applicable
>> law. If you are not the intended recipient or authorized to receive
>> this for the intended recipient, any use, dissemination, distribution,
>> retention, archiving, or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify
>> the sender immediately by reply e-mail and delete this message.
>>
> 
> ------------------------------------------------------------------------------
> Beautiful is writing same markup. Internet Explorer 9 supports
> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2&  L3.
> Spend less time writing and  rewriting code and more time creating great
> experiences on the web. Be a part of the beta today
> http://p.sf.net/sfu/msIE9-sfdev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

On 11/16/10 12:09 PM, xiong wang wrote:
> Dears,
>
> Thanks for your suggestions.
>
> I should clarify my idea.
> 1)This patch is a temporary solution. It cann't really solve the
> problem raised by updating on a partition column until the measure
> that moving tuples from one node to another comes true.
>    It just forbids users to update on a partition column. Therefore, I
> didn't consider too much because it will be deleted later.
> 2)The patch forbids update on partition column on coordinators. It has
> nothing to do with data nodes.
>
> If I misunderstand you, please let me know.
>   

Yes, I think this is helpful for now. Long term, handling moving tuples
will be more complicated.

Applications do have a work-around in that they can do a delete and
insert if they require updating the partitioning column value.

Thanks,

Mason

> Regards,
> Benny
>
>   
>> Hi,
>>     
>   
>> I'm curious about two issues on this patch.
>>     
>   
>> 1) Does it handle TOAST table correctly?
>> 2) Does it handle SET clause such as SET key = key+1 ?  Only a data node
>>     
> can determine if such UPDATE statement make rows invalid to stay in the
> data node.   Unfortunately, data node is not equipped with distribution
> key.  To handle this correctly, data node has to handle TOAST table as
>   
>> mentioned in 1).
>>     
>   
>> Regards;
>> ---
>> Koichi Suzuki
>>     
> (2010年11月15日 23:59), Mason Sharp wrote:
>   
>> On 11/15/10 5:13 PM, mei le wrote:
>>     
>>> Dears,
>>>
>>> I am sorry. The bug#3107683.patch I commited before introduces a warning I didn't noticed. The enclosure is a new patch after I eliminated the warning.
>>>
>>>       
>> Thanks. I just committed this and fixed a couple of other warnings that
>> we had.
>>
>> Regards,
>>
>> Mason
>>     
>>> Regards,
>>>
>>>                                              Benny
>>>
>>>
>>>
>>> --- 10年11月12日，周五, Mason Sharp<mas...@en...>  写道：
>>>
>>>
>>> 发件人: Mason Sharp<mas...@en...>
>>> 主题: Fwd: [ postgres-xc-Bugs-3107683 ] UPDATE on partition column
>>> 收件人: "Postgres-XC Developers"<pos...@li...>, "mei le"<lem...@ya...>
>>> 日期: 2010年11月12日,周五,上午10:34
>>>
>>>
>>> Benny,
>>>
>>> I thought this might be one you might be interested in working on, too, if you have time.
>>>
>>> You could validate the update statement in pgxc_planner() if it is on a partitioned table.
>>>
>>> Thanks,
>>>
>>> Mason
>>>
>>> -------- Original Message --------
>>>
>>>
>>>
>>> Subject:
>>> [ postgres-xc-Bugs-3107683 ] UPDATE on partition column
>>>
>>> Date:
>>> Fri, 12 Nov 2010 02:29:56 +0000
>>>
>>> From:
>>> SourceForge.net<no...@so...>
>>>
>>> To:
>>> no...@so...
>>>
>>> Bugs item #3107683, was opened at 2010-11-11 21:29
>>> Message generated for change (Tracker Item Submitted) made by mason_s
>>> You can respond by visiting:
>>> https://sourceforge.net/tracker/?func=detail&atid=1310232&aid=3107683&group_id=311227
>>>
>>> Please note that this message will contain a full copy of the comment thread,
>>> including the initial issue submission, for this request,
>>> not just the latest update.
>>> Category: Database Server
>>> Group: None
>>> Status: Open
>>> Resolution: None
>>> Priority: 6
>>> Private: No
>>> Submitted By: mason_s (mason_s)
>>> Assigned to: Nobody/Anonymous (nobody)
>>> Summary: UPDATE on partition column
>>>
>>> Initial Comment:
>>> Until we support moving tuples from one node to another when the partition column of a table is updated, we should at least block this from happening. At the moment you can update it, and bad things can start happening as a result.
>>>
>>> ----------------------------------------------------------------------
>>>
>>> You can respond by visiting:
>>> https://sourceforge.net/tracker/?func=detail&atid=1310232&aid=3107683&group_id=311227
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture
>>> Simplifying enterprise desktop deployment and management using
>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end
>>> client virtualization framework. Read more!
>>> http://p.sf.net/sfu/dell-eql-dev2dev
>>>
>>>
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>       
>>
>> --
>> Mason Sharp
>> EnterpriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>> This e-mail message (and any attachment) is intended for the use of
>> the individual or entity to whom it is addressed. This message
>> contains information from EnterpriseDB Corporation that may be
>> privileged, confidential, or exempt from disclosure under applicable
>> law. If you are not the intended recipient or authorized to receive
>> this for the intended recipient, any use, dissemination, distribution,
>> retention, archiving, or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify
>> the sender immediately by reply e-mail and delete this message.
>>
>>     
> ------------------------------------------------------------------------------
> Beautiful is writing same markup. Internet Explorer 9 supports
> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
> Spend less time writing and  rewriting code and more time creating great
> experiences on the web. Be a part of the beta today
> http://p.sf.net/sfu/msIE9-sfdev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>   


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.