BUG #15858: could not stat file - over 4GB

The following bug has been logged on the website:

Bug reference: 15858
Logged by: William Allen
Email address: williamedwinallen(at)live(dot)com
PostgreSQL version: 11.3
Operating system: Windows Server 2012 R2
Description:

Issue using copy from command for files over 4GB.

ERROR: could not stat file "E:\file.txt": Unknown error
SQL state: XX000

On Tue, Jun 18, 2019 at 10:02:53AM +0000, PG Bug reporting form wrote:
> Issue using copy from command for files over 4GB.
>
> ERROR: could not stat file "E:\file.txt": Unknown error
> SQL state: XX000

Windows is known for having limitations in its former implementations
of stat(), and the various _stat structures they use make actually
that much harder from a compatibility point of view:
https://www.postgresql.org/message-id/1803D792815FC24D871C00D17AE95905CF5099@g01jpexmbkw24

Nobody has actually dug enough into this set of issues to get a patch
out of the ground, which basically requires more tweaks that one may
think at first sight (look at pgwin32_safestat() in src/port/dirmod.c
for example).
--
Michael

On Wed, Jun 19, 2019 at 3:26 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>
> Windows is known for having limitations in its former implementations
> of stat(), and the various _stat structures they use make actually
> that much harder from a compatibility point of view:
> https://www.postgresql.org/message-id/1803D792815FC24D871C00D17AE95905CF5099@g01jpexmbkw24
>

Going through this discussion it is not clear to me if there was a
consensus about the shape of an acceptable patch. Would something like
the attached be suitable?

Regards,

Juan José Santamaría Flecha

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo(dot)santamaria(at)gmail(dot)com> writes:
> On Wed, Jun 19, 2019 at 3:26 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>> Windows is known for having limitations in its former implementations
>> of stat(), and the various _stat structures they use make actually
>> that much harder from a compatibility point of view:
>> https://www.postgresql.org/message-id/1803D792815FC24D871C00D17AE95905CF5099@g01jpexmbkw24

> Going through this discussion it is not clear to me if there was a
> consensus about the shape of an acceptable patch. Would something like
> the attached be suitable?

I think there's general agreement that the correct fix involves somehow
mapping stat() to _stat64() and mapping "struct stat" to "struct __stat64"
to go along with that. Beyond that, things get murky.

1. Can we assume that _stat64() and struct __stat64 exist on every Windows
version and build toolchain that we care about? Windows itself is
probably OK --- googling found a (non-authoritative) statement that these
were introduced in Windows 2K. But it's less clear whether they'll work
on builds with Cygwin, or Mingw, or Mingw-64, or how far back that support
goes. I found one statement that Mingw declares them only "#if
__MSVCRT_VERSION__ >= 0x0601".

2. Mapping stat() to _stat64() seems easy enough: we already declare
stat(a,b) as a macro on Windows, so just change it to something else.

3. What about the struct name? I proposed just "define stat __stat64",
but Robert thought that was too cute, and he's got a point --- in
particular, it's not clear to me how nicely it'd play to have both
function and object macros for the same name "stat". I see you are
proposing fixing this angle by suppressing the system definition of
struct stat and then defining it ourselves with the same contents as
struct __stat64. That might work. Ordinarily I'd be worried about
bit-rot in a struct that has to track a system definition, but Microsoft
are so religiously anal about never breaking ABI that it might be safe
to assume we don't have to worry about that.

I don't like the specific way you're proposing suppressing the system
definition of struct stat, though. "#define _CRT_NO_TIME_T" seems
like it's going to be a disaster, both because it likely has other
side-effects and because it probably doesn't do what you intend at all
on non-MSVC toolchains. We have precedents for dealing with similar
issues in, eg, plperl; and what those precedents would suggest is
doing something like

#define stat microsoft_native_stat
#include <sys/stat.h>
#undef stat

after which we could do

struct stat {
... same contents as __stat64
};

#define stat(a,b) _stat64(a,b)

Another issue here is that pgwin32_safestat() probably needs revisited
as to its scope and purpose. Its use of GetFileAttributesEx() can
presumably be dropped. I don't actually believe the header comment
claiming that stat() is not guaranteed to update the st_size field;
there's no indication of that in the Microsoft documentation. What
seems more likely is that that's a garbled version of the truth,
that you won't get a correct value of _st_size for files over 4GB.
But the test for ERROR_DELETE_PENDING might be worth keeping. So
that would lead us to

struct stat {
... same contents as __stat64
};

extern int pgwin32_safestat(const char *path, struct stat *buf);
#define stat(a,b) pgwin32_safestat(a,b)

and something like

int
pgwin32_safestat(const char *path, struct stat *buf)
{
int r;

/*
* Don't call stat(), that would just recurse back to here.
* We really want _stat64().
*/
r = _stat64(path, buf);

if (r < 0)
{
if (GetLastError() == ERROR_DELETE_PENDING)
{
/*
* File has been deleted, but is not gone from the filesystem yet.
* This can happen when some process with FILE_SHARE_DELETE has it
* open and it will be fully removed once that handle is closed.
* Meanwhile, we can't open it, so indicate that the file just
* doesn't exist.
*/
errno = ENOENT;
}
}
return r;
}

Not sure if we'd need an explicit cast to override passing struct
stat * to _stat64(). If so, a StaticAssert that sizeof(struct stat)
matches sizeof(struct __stat64) seems like a good idea.

I'd also be very strongly inclined to move pgwin32_safestat into its
own file in src/port and get rid of UNSAFE_STAT_OK. There wouldn't
be a good reason to opt out of using it once we got to this point.

regards, tom lane

I wrote:
> Another issue here is that pgwin32_safestat() probably needs revisited
> as to its scope and purpose. Its use of GetFileAttributesEx() can
> presumably be dropped. I don't actually believe the header comment
> claiming that stat() is not guaranteed to update the st_size field;
> there's no indication of that in the Microsoft documentation. What
> seems more likely is that that's a garbled version of the truth,
> that you won't get a correct value of _st_size for files over 4GB.

So after further digging around, it seems that this is wrong. The
existence of pgwin32_safestat() can be traced back to these threads:

https://www.postgresql.org/message-id/flat/528853D3C5ED2C4AA8990B504BA7FB850106DF10%40sol.transas.com
https://www.postgresql.org/message-id/flat/528853D3C5ED2C4AA8990B504BA7FB850106DF2F%40sol.transas.com

in which it's stated that

It seems I've found the cause and the workaround of the problem.
MSVC's stat() is implemented by using FindNextFile().
MSDN contains the following suspicious paragraph аbout FindNextFile():
"In rare cases, file attribute information on NTFS file systems
may not be current at the time you call this function. To obtain
the current NTFS file system file attributes, call
GetFileInformationByHandle."
Since we generally cannot open an examined file, we need another way.

I'm wondering though why we adopted the existing coding in the face of
that observation. Couldn't the rest of struct stat be equally out of
date?

In short it seems like maybe we should be doing something similar to the
patch that Sergey actually submitted in that discussion:

https://www.postgresql.org/message-id/528853D3C5ED2C4AA8990B504BA7FB850658BA5C%40sol.transas.com

which reimplements stat() from scratch on top of GetFileAttributesEx(),
and thus doesn't require any assumptions at all about what's available
from the toolchain's <sys/stat.h>.

regards, tom lane

On Wed, Jun 19, 2019 at 8:02 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> In short it seems like maybe we should be doing something similar to the
> patch that Sergey actually submitted in that discussion:
>
> https://www.postgresql.org/message-id/528853D3C5ED2C4AA8990B504BA7FB850658BA5C%40sol.transas.com
>

I will not have much time for this list in the next couple of weeks,
so I will send this patch in its current WIP state rather than
stalling without a reply.

Most of its functionality comes from Sergey's patch with some cosmetic
changes, and the addition of the 64 bits struct stat and fstat().

Regards,

Juan José Santamaría Flecha

On Tue, Jun 25, 2019 at 12:00:45PM +0200, Juan José Santamaría Flecha wrote:
> I will not have much time for this list in the next couple of weeks,
> so I will send this patch in its current WIP state rather than
> stalling without a reply.
>
> Most of its functionality comes from Sergey's patch with some cosmetic
> changes, and the addition of the 64 bits struct stat and fstat().

The former patch was rather impressive. Or scary. Or both. At which
extent have you tested it? I think that we really need to make sure
of a couple of things which satisfy our needs:
1) Are we able to fix the issues with stat() calls on files larger
than 2GB and report a correct size?
2) Are we able to detect properly that files pending for deletion are
discarded with ENOENT?
3) Are frontends able to use the new layer?

It seems to me that you don't need the configure changes.

Instead of stat_pg_fixed which is confusing because it only involves
Windows, I would rename the new file to stat.c or win32_stat.c. The
location in src/port/ is adapted. I would also move out of
win32_port.h the various inline declarations and keep only raw
declarations. That could be much cleaner.

The code desperately needs more comments to help understand its
logic. Don't we have in the tree an equivalent of cvt_ft2ut? What
does cvt_attr2uxmode do? It would be nice to avoid conversion
wrappers as much as possible, and find out system-related equivalents
if any, and actually if necessary.

+static unsigned short
+cvt_attr2uxmode(int attr, const _TCHAR * name)
This looks rather bug-prone...

I think that this stuff has not been tested and would break at
compilation. If src/tools/msvc/Mkvcbuild.pm is not changed, then the
new file won't get included in the compiled set.
--
Michael

On Wed, Jun 26, 2019 at 4:23 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>
> The former patch was rather impressive. Or scary. Or both. At which
> extent have you tested it? I think that we really need to make sure
> of a couple of things which satisfy our needs:

I wanted to make a quick test on the previous patch. So let me state
what have I tested and what I have not: it builds and pass tests in
Windows and Cygwin, but I have not setup a MinGW environment.

> 1) Are we able to fix the issues with stat() calls on files larger
> than 2GB and report a correct size?

I have successfuly tested a COPY with large files.

> 2) Are we able to detect properly that files pending for deletion are
> discarded with ENOENT?

Cannot reproduce reliably, but is using the same logic as pgwin32_safestat().

> 3) Are frontends able to use the new layer?

After removing UNSAFE_STAT_OK, is this still an issue?

> It seems to me that you don't need the configure changes.

The changes in configuration are meant for gcc compilations in Windows
(Cygwin and Mingw).

> Instead of stat_pg_fixed which is confusing because it only involves
> Windows, I would rename the new file to stat.c or win32_stat.c. The
> location in src/port/ is adapted. I would also move out of
> win32_port.h the various inline declarations and keep only raw
> declarations. That could be much cleaner.

Ok.

> The code desperately needs more comments to help understand its
> logic. Don't we have in the tree an equivalent of cvt_ft2ut? What
> does cvt_attr2uxmode do? It would be nice to avoid conversion
> wrappers as much as possible, and find out system-related equivalents
> if any, and actually if necessary.

I have only found something similar in ./src/port/gettimeofday.c, but
not sure if this patch should touch that code.

> +static unsigned short
> +cvt_attr2uxmode(int attr, const _TCHAR * name)
> This looks rather bug-prone...

I wanted to keep as much of the original code as possible, but if this
is found as a viable solution, what shape should it have?

> I think that this stuff has not been tested and would break at
> compilation. If src/tools/msvc/Mkvcbuild.pm is not changed, then the
> new file won't get included in the compiled set.

The previous patch was broken, taken from the wrong local branch
(sorry about that). The attached is still a WIP but it has to do the
things above-mentioned.

Regards,

Juan José Santamaría Flecha

On Fri, Jun 28, 2019 at 11:34:38PM +0200, Juan José Santamaría Flecha wrote:
> I wanted to make a quick test on the previous patch. So let me state
> what have I tested and what I have not: it builds and pass tests in
> Windows and Cygwin, but I have not setup a MinGW environment.

Thanks. Could you attach this patch to the next commit fest? We had
many complaints with the current limitations with large files (pg_dump
syncs its result files, so that breaks on Windows actually if the dump
is larger than 2GB..), and we are going to need to do something. I
find that stuff rather hard to backpatch, but let's see.
--
Michael

On Sat, Jun 29, 2019 at 4:30 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>
> Thanks. Could you attach this patch to the next commit fest? We had
> many complaints with the current limitations with large files (pg_dump
> syncs its result files, so that breaks on Windows actually if the dump
> is larger than 2GB..), and we are going to need to do something. I
> find that stuff rather hard to backpatch, but let's see.

Done. [1]

Regards,

Juan José Santamaría Flecha

[1] https://commitfest.postgresql.org/23/2189/

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo(dot)santamaria(at)gmail(dot)com> writes:
> On Wed, Jun 26, 2019 at 4:23 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>> It seems to me that you don't need the configure changes.

> The changes in configuration are meant for gcc compilations in Windows
> (Cygwin and Mingw).

Directly editing the configure script is Not Done ... or at least,
such changes wouldn't survive the next correctly-done configure
update. You have to edit configure.in (or one of the sub-files in
config/) and then regenerate configure using autoconf.

It seems likely that we *don't* need or want this for Cygwin;
that should be providing a reasonable stat() emulation already.
So probably you just want to add "AC_LIBOBJ(win32_stat)" to
the stanza beginning

# Win32 (really MinGW) support
if test "$PORTNAME" = "win32"; then
AC_CHECK_FUNCS(_configthreadlocale)
AC_REPLACE_FUNCS(gettimeofday)
AC_LIBOBJ(dirmod)

I'd also recommend that stat() fill all the fields in struct stat,
even if you don't have anything better to put there than zeroes.
Otherwise you're just opening things up for random misbehavior.

I'm not in a position to comment on the details of the conversion from
GetFileAttributesEx results to struct stat, but in general this
seems like a reasonable way to proceed.

regards, tom lane

Thanks for looking into this.

On Fri, Aug 23, 2019 at 11:49 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Directly editing the configure script is Not Done ... or at least,
> such changes wouldn't survive the next correctly-done configure
> update. You have to edit configure.in (or one of the sub-files in
> config/) and then regenerate configure using autoconf.
>
> It seems likely that we *don't* need or want this for Cygwin;
> that should be providing a reasonable stat() emulation already.
> So probably you just want to add "AC_LIBOBJ(win32_stat)" to
> the stanza beginning
>
> I'd also recommend that stat() fill all the fields in struct stat,
> even if you don't have anything better to put there than zeroes.
> Otherwise you're just opening things up for random misbehavior.
>

Fixed.

> I'm not in a position to comment on the details of the conversion from
> GetFileAttributesEx results to struct stat, but in general this
> seems like a reasonable way to proceed.
>

Actually, due to the behaviour of GetFileAttributesEx with symbolic
links I think that using GetFileInformationByHandle instead can give a
more resilient solution. Also, by using a handle we get a good test
for ERROR_DELETE_PENDING. This is the approach for the attached patch.

Regards,

Juan José Santamaría Flecha

Hi - is this likely to be applied to an upcoming release? / How does a novice apply a patch..?

Thanks

-----Original Message-----
From: Juan José Santamaría Flecha <juanjo(dot)santamaria(at)gmail(dot)com>
Sent: 04 September 2019 22:48
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>; williamedwinallen(at)live(dot)com; pgsql-bugs(at)lists(dot)postgresql(dot)org; Magnus Hagander <magnus(at)hagander(dot)net>; PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: BUG #15858: could not stat file - over 4GB

Thanks for looking into this.

On Fri, Aug 23, 2019 at 11:49 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Directly editing the configure script is Not Done ... or at least,
> such changes wouldn't survive the next correctly-done configure
> update. You have to edit configure.in (or one of the sub-files in
> config/) and then regenerate configure using autoconf.
>
> It seems likely that we *don't* need or want this for Cygwin; that
> should be providing a reasonable stat() emulation already.
> So probably you just want to add "AC_LIBOBJ(win32_stat)" to the stanza
> beginning
>
> I'd also recommend that stat() fill all the fields in struct stat,
> even if you don't have anything better to put there than zeroes.
> Otherwise you're just opening things up for random misbehavior.
>

Fixed.

> I'm not in a position to comment on the details of the conversion from
> GetFileAttributesEx results to struct stat, but in general this seems
> like a reasonable way to proceed.
>

Actually, due to the behaviour of GetFileAttributesEx with symbolic links I think that using GetFileInformationByHandle instead can give a more resilient solution. Also, by using a handle we get a good test for ERROR_DELETE_PENDING. This is the approach for the attached patch.

Regards,

Juan José Santamaría Flecha

On Mon, Oct 28, 2019 at 3:29 PM william allen <williamedwinallen(at)live(dot)com>
wrote:

> Hi - is this likely to be applied to an upcoming release? / How does a
> novice apply a patch..?
>
>
At this moment is missing review, so it is probably far from being
commitable. Any attention is appreciated and might help pushing it forward.
As a personal note, I have to check that is still applies before the
upcoming commitfest.

As for applying this patch you would need a Windows development
environment. I would recommend Visual Studio as a starting point [1]. You
also have a very visual guide in the wiki [2].

[1] https://www.postgresql.org/docs/current/install-windows.html
[2] https://wiki.postgresql.org/wiki/Working_With_VisualStudio

Regards,

Juan José Santamaría Flecha

The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: tested, passed
Spec compliant: not tested
Documentation: not tested

I ran into this problem when using psql.exe and copy command.

I have checked out 11.6-release tarball and applied the patch.
The patch does not apply cleanly, but can be easily modified to apply. See Note 1.
After applying the patch I built using "build psql" and ran the new psql.exe binary.

In order to test I have done the following:
Against a PostgreSQL 11 server run two commands:
"COPY public.table FROM 'C:/file'" and "\copy public.table FROM 'C:/file'"
The first one runs in the context of the server, and does not work. It aborts with an error saying "cannot stat file", as expected.
The seconds on runs in the context of the new binary and does work. It copies data as expected.

Note 1:
src/tools/msvc/Mkvcbuild.pm should be

- sprompt.c strerror.c tar.c thread.c getopt.c getopt_long.c dirent.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ sprompt.c tar.c thread.c getopt.c getopt_long.c dirent.c
+ win32env.c win32error.c win32security.c win32setlocale.c win32_stat.c);

The new status of this patch is: Waiting on Author

On Wed, Feb 5, 2020 at 12:47 PM Emil Iggland <emil(at)iggland(dot)com> wrote:

> The following review has been posted through the commitfest application:
> make installcheck-world: not tested
> Implements feature: tested, passed
> Spec compliant: not tested
> Documentation: not tested
>

The latest version of this patch could benefit from an update. Please find
attached a new version.

Most changes are cosmetic, but they have been more extensive than a simple
rebase so I am changing the status back to 'needs review'.

To summarize those changes:
- Rename 'win32_stat.c' file to 'win32stat.c', as a better match of
project files.
- Improve indentation and comments.
- Remove cruft about old Windows versions.

Regards,

Juan José Santamaría Flecha

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo(dot)santamaria(at)gmail(dot)com> writes:
> The latest version of this patch could benefit from an update. Please find
> attached a new version.

The cfbot thinks this doesn't compile on Windows [1]. Looks like perhaps
a missing-#include problem?

regards, tom lane

[1] https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.81541

On Sat, Feb 29, 2020 at 12:44 AM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

>
> The cfbot thinks this doesn't compile on Windows [1]. Looks like perhaps
> a missing-#include problem?

The define logic for _WIN32_WINNT includes testing of _MSC_VER, and is not
a proper choice for MSVC 2013 as the cfbot is showing.

Please find attached a new version addressing this issue.

Regards,

Juan José Santamaría Flecha

>
>

On Sat, Feb 29, 2020 at 9:40 AM Juan José Santamaría Flecha <
juanjo(dot)santamaria(at)gmail(dot)com> wrote:

> On Sat, Feb 29, 2020 at 12:44 AM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>>
>> The cfbot thinks this doesn't compile on Windows [1]. Looks like perhaps
>> a missing-#include problem?
>
>
> The define logic for _WIN32_WINNT includes testing of _MSC_VER, and is not
> a proper choice for MSVC 2013 as the cfbot is showing.
>

The cfbot is not happy yet. I will backtrack a bit on the cruft cleanup.

Please find attached a new version addressing this issue.

Regards,

Juan José Santamaría Flecha

>
>>
>

I assigned myself as a reviewer for this patch, as I hit this bug today and had to perform a workaround. I have never reviewed a patch before but will try to update within the next 5 days. I intend on performing "Implements Feature" reviewing.

On Mon, Oct 28, 2019 at 06:13:58PM +0100, Juan José Santamaría Flecha wrote:
> At this moment is missing review, so it is probably far from being
> commitable. Any attention is appreciated and might help pushing it forward.
> As a personal note, I have to check that is still applies before the
> upcoming commitfest.

Could you send a rebase of the patch? Thanks!
--
Michael

On Thu, Sep 17, 2020 at 9:46 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:

>
> Could you send a rebase of the patch? Thanks!
>

Thanks for the reminder. Please find attached a rebased version.

Regards,

Juan José Santamaría Flecha

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo(dot)santamaria(at)gmail(dot)com> writes:
> Thanks for the reminder. Please find attached a rebased version.

(This hasn't shown up on -hackers yet, maybe caught in moderation?)

I took a quick look through this. I'm not qualified to review the
actual Windows code in win32stat.c, but as far as the way you're
plugging it into the system goes, it looks good and seems to comport
with the discussion so far.

One thing I noticed, which is a pre-existing problem but maybe now
is a good time to consider it, is that we're mapping lstat() to be
exactly stat() on Windows. That made sense years ago when (we
believed that) Windows didn't have symlinks, but surely it no longer
makes sense.

Another more trivial point is that it'd be good to run the new code
through pgindent before committing.

regards, tom lane

Em qui., 17 de set. de 2020 às 14:37, Juan José Santamaría Flecha <
juanjo(dot)santamaria(at)gmail(dot)com> escreveu:

>
> On Thu, Sep 17, 2020 at 9:46 AM Michael Paquier <michael(at)paquier(dot)xyz>
> wrote:
>
>>
>> Could you send a rebase of the patch? Thanks!
>>
>
> Thanks for the reminder. Please find attached a rebased version.
>
Sorry, I'm missing something?
What's wrong with _stat64?

Pasta de C:\tmp

18/08/2020 16:51 6.427.512.517 macOS_Catalina.7z
1 arquivo(s) 6.427.512.517 bytes
0 pasta(s) 149.691.797.504 bytes disponíveis

C:\usr\src\tests\stat>crt_stat
File size : 6427512517
Drive : C:
Time modified : Tue Aug 18 16:51:47 2020

regards,
Ranier Vilela

Ranier Vilela <ranier(dot)vf(at)gmail(dot)com> writes:
> What's wrong with _stat64?

See upthread.

regards, tom lane

On Thu, Sep 17, 2020 at 6:04 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> =?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <
> juanjo(dot)santamaria(at)gmail(dot)com> writes:
> > Thanks for the reminder. Please find attached a rebased version.
>
> (This hasn't shown up on -hackers yet, maybe caught in moderation?)
>

Thanks for looking into it. Finally, it went through. I will be removing
bug-list from now on.

>
> I took a quick look through this. I'm not qualified to review the
> actual Windows code in win32stat.c, but as far as the way you're
> plugging it into the system goes, it looks good and seems to comport
> with the discussion so far.
>
> One thing I noticed, which is a pre-existing problem but maybe now
> is a good time to consider it, is that we're mapping lstat() to be
> exactly stat() on Windows. That made sense years ago when (we
> believed that) Windows didn't have symlinks, but surely it no longer
> makes sense.
>

I will have to take a better look at it, but from a quick look it, all
lstat() calls seem to test just if the file exists, and that can be done
with a cheap call to GetFileAttributes(). Would a limited (but fast)
lstat(), where only st_mode could be informed, be acceptable?

>
> Another more trivial point is that it'd be good to run the new code
> through pgindent before committing.
>

I do not have pgindent in the WIN32 machine, but I will try to for the next
version.

Regards,

Juan José Santamaría Flecha

On Thu, Sep 17, 2020 at 8:47 PM Juan José Santamaría Flecha <
juanjo(dot)santamaria(at)gmail(dot)com> wrote:

> On Thu, Sep 17, 2020 at 6:04 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>>
>> One thing I noticed, which is a pre-existing problem but maybe now
>> is a good time to consider it, is that we're mapping lstat() to be
>> exactly stat() on Windows. That made sense years ago when (we
>> believed that) Windows didn't have symlinks, but surely it no longer
>> makes sense.
>>
>
> I will have to take a better look at it, but from a quick look it, all
> lstat() calls seem to test just if the file exists, and that can be done
> with a cheap call to GetFileAttributes(). Would a limited (but fast)
> lstat(), where only st_mode could be informed, be acceptable?
>

After thinking more about this, that approach would be problematic for
DELETE_PENDING files. The proposed patch logic is meant to maintain current
behaviour, which is not broken for WIN32 symlinks AFAICT.

Regards,

Juan José Santamaría Flecha

The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: not tested
Documentation: not tested

I tested the patch at hand, and it performs as expected. Files larger than 4GB can be imported.

Steps:
0) create a csv-file that is sufficiently big (>4GB), and one that is small. Use these files to test.
1a) Attempt to import the small file using devel-version.
1b) EXPECTED: success, ACTUAL: success
2a) Attempt to import the big file using devel-version.
2b) EXPECTED: failure, ACTUAL: failure
3) Apply patch and build new version
4a) Attempt to import the small file using patched-version.
4b) EXPECTED: success, ACTUAL: success
4a) Attempt to import the big file using patched-version.
4b) EXPECTED: success, ACTUAL: success

The code looks sensible, it is easy to read and follow. The code uses appropriate win32 functions to perform the task.

Code calculates file size using the following method: buf->st_size = ((__int64) fiData.nFileSizeHigh) << 32 | (__int64)(fiData.nFileSizeLow);
The hard coded constant 32 is fine, nFileSizeHigh is defined as a DWORD in the Win32 API, which is a 32 bit unsigned integer. There is no need to a dynamic calculation.

There are minor "nit-picks" that I would change if it were my code, but do not change the functionality of the code.

1)
if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
{
errno = ENOENT;
return -1;
}

Here I would call _dosmaperr(GetLastError()) instead, just to take account of the possibility that some other error occurred. Following this change there are slight inconsistency in the order of "CloseHandle(hFile), errno = ENOENT; return -1" and "_dosmaperr(GetLastError()); CloseHandle(hFile); return -1". I would prefer consistent ordering, but that is not important.

The new status of this patch is: Ready for Committer

Emil Iggland <emil(at)iggland(dot)com> writes:
> I tested the patch at hand, and it performs as expected. Files larger than 4GB can be imported.

Thanks for testing!

I'd been expecting one of our Windows-savvy committers to pick this up,
but since nothing has been happening, I took it on myself to push it.
I'll probably regret that :-(

I made a few cosmetic changes, mostly reorganizing comments in a way
that made more sense to me.

regards, tom lane

On Fri, Oct 9, 2020 at 10:22 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Emil Iggland <emil(at)iggland(dot)com> writes:
> > I tested the patch at hand, and it performs as expected. Files larger
> than 4GB can be imported.
>
> Thanks for testing!
>

Thanks for testing! +1

>
> I'd been expecting one of our Windows-savvy committers to pick this up,
> but since nothing has been happening, I took it on myself to push it.
> I'll probably regret that :-(
>

Thanks for taking care of this. I see no problems in the build farm, but
please reach me if I missed something.

>
> I made a few cosmetic changes, mostly reorganizing comments in a way
> that made more sense to me.
>
> I was working on a new version, which was pgindent-friendlier and clearer
about reporting an error when 'errno' is not informed. Please find attached
a patch with those changes.

Regards,

Juan José Santamaría Flecha

On Sat, Oct 10, 2020 at 01:31:21PM +0200, Juan José Santamaría Flecha wrote:
> Thanks for taking care of this. I see no problems in the build farm, but
> please reach me if I missed something.

Thanks for continuing your work on this patch. I see no related
failures in the buildfarm.

- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
+ /* report when not ERROR_SUCCESS */
+ if (err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND)
+ errno = ENOENT;
+ else
+ _dosmaperr(err);
Why are you changing that? The original coding is fine, as
_dosmaperr() already maps ERROR_FILE_NOT_FOUND and
ERROR_PATH_NOT_FOUND to ENOENT.

- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
CloseHandle(hFile);
+ _dosmaperr(err);
These parts are indeed incorrect. CloseHandle() could overwrite
errno.
--
Michael

On Sat, Oct 10, 2020 at 2:24 PM Michael Paquier <michael(at)paquier(dot)xyz> wrote:

>
> - _dosmaperr(GetLastError());
> + DWORD err = GetLastError();
> +
> + /* report when not ERROR_SUCCESS */
> + if (err == ERROR_FILE_NOT_FOUND || err ==
> ERROR_PATH_NOT_FOUND)
> + errno = ENOENT;
> + else
> + _dosmaperr(err);
> Why are you changing that? The original coding is fine, as
> _dosmaperr() already maps ERROR_FILE_NOT_FOUND and
> ERROR_PATH_NOT_FOUND to ENOENT.
>

If the file does not exist there is no need to call _dosmaperr() and log
the error.

>
> - _dosmaperr(GetLastError());
> + DWORD err = GetLastError();
> +
> CloseHandle(hFile);
> + _dosmaperr(err);
> These parts are indeed incorrect. CloseHandle() could overwrite
> errno.
>

The meaningful error should come from the previous call, and an error from
CloseHandle() could mask it. Not sure it makes a difference anyhow.

Regards,

Juan José Santamaría Flecha

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo(dot)santamaria(at)gmail(dot)com> writes:
> If the file does not exist there is no need to call _dosmaperr() and log
> the error.

I concur with Michael that it's inappropriate to make an end run around
_dosmaperr() here. If you think that the DEBUG5 logging inside that
is inappropriate, you should propose removing it outright.

Pushed the rest of this.

(pgindent behaved differently around PFN_NTQUERYINFORMATIONFILE today
than it did yesterday. No idea why.)

> The meaningful error should come from the previous call, and an error from
> CloseHandle() could mask it. Not sure it makes a difference anyhow.

Would CloseHandle() really touch errno at all? But this way is
certainly safer, so done.

regards, tom lane

On Sat, Oct 10, 2020 at 7:43 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

>
> I concur with Michael that it's inappropriate to make an end run around
> _dosmaperr() here. If you think that the DEBUG5 logging inside that
> is inappropriate, you should propose removing it outright.
>
> Pushed the rest of this.
>

Great, thanks again to everyone who has taken some time to look into this.

Regards,

Juan José Santamaría Flecha

On Sat, Oct 10, 2020 at 09:00:27PM +0200, Juan José Santamaría Flecha wrote:
> Great, thanks again to everyone who has taken some time to look into this.

We are visibly not completely out of the woods yet, jacana is
reporting a compilation error:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2020-10-10%2018%3A00%3A28
Oct 10 14:04:40
c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.build/../pgsql/src/port/win32stat.c:
In function 'fileinfo_to_stat':
Oct 10 14:04:40
c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.build/../pgsql/src/port/win32stat.c:151:13:
error: 'BY_HANDLE_FILE_INFORMATION {aka struct
_BY_HANDLE_FILE_INFORMATION}' has no member named 'nFileSizeLowi'; did
you mean 'nFileSizeLow'?
Oct 10 14:04:40 fiData.nFileSizeLowi);
Oct 10 14:04:40 ^~~~~~~~~~~~~
Oct 10 14:04:40 nFileSizeLow

I don't have the time to check MinGW and HEAD now, so that's just a
heads-up.
--
Michael

Michael Paquier <michael(at)paquier(dot)xyz> writes:
> We are visibly not completely out of the woods yet, jacana is
> reporting a compilation error:

Nah, I fixed that hours ago (961e07b8c). jacana must not have run again
yet.

regards, tom lane

On Sat, Oct 10, 2020 at 08:34:48PM -0400, Tom Lane wrote:
> Nah, I fixed that hours ago (961e07b8c). jacana must not have run again
> yet.

Indeed, thanks. I have missed one sync here.

+ hFile = CreateFile(name,
+ GENERIC_READ,
+ (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+ &sa,
+ OPEN_EXISTING,
+ (FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS |
+ FILE_FLAG_OVERLAPPED),
+ NULL);
+ if (hFile == INVALID_HANDLE_VALUE)
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
_dosmaperr(GetLastError()) to get the correct errno? This code would
for example consider as non-existing a file even if we fail getting it
because of ERROR_SHARING_VIOLATION, which should map to EACCES. This
case can happen with virus scanners taking a non-share handle on files
being looked at in parallel of this code path.
--
Michael

Michael Paquier <michael(at)paquier(dot)xyz> writes:
> Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
> _dosmaperr(GetLastError()) to get the correct errno?

Fair question. Juan, was there some good reason not to look at
GetLastError() in this step?

regards, tom lane

On Mon, Oct 12, 2020 at 5:27 AM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Michael Paquier <michael(at)paquier(dot)xyz> writes:
> > Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
> > _dosmaperr(GetLastError()) to get the correct errno?
>
> Fair question. Juan, was there some good reason not to look at
> GetLastError() in this step?
>

Uhm, a good question indeed, forcing errno serves no purpose there.

Regards,

Juan José Santamaría Flecha

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo(dot)santamaria(at)gmail(dot)com> writes:
> On Mon, Oct 12, 2020 at 5:27 AM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Michael Paquier <michael(at)paquier(dot)xyz> writes:
>>> Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
>>> _dosmaperr(GetLastError()) to get the correct errno?

>> Fair question. Juan, was there some good reason not to look at
>> GetLastError() in this step?

> Uhm, a good question indeed, forcing errno serves no purpose there.

OK, changed.

regards, tom lane

On Mon, Oct 12, 2020 at 11:13:38AM -0400, Tom Lane wrote:
> Juan José Santamaría Flecha wrote:
>> Uhm, a good question indeed, forcing errno serves no purpose there.
>
> OK, changed.

Thanks!
--
Michael

Hi,

I noticed that this modification only commit into master branch,
there is still have a problem on 12.6 or 13.2 on Windows.

Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?

The commit:
https://github.com/postgres/postgres/commit/bed90759fcbcd72d4d06969eebab81e47326f9a2
https://github.com/postgres/postgres/commit/ed30b1a60dadf2b7cc58bce5009ad8676b8fe479

------
Best regards
Shenhao Wang

"wangsh(dot)fnst(at)fujitsu(dot)com" <wangsh(dot)fnst(at)fujitsu(dot)com> writes:
> Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?

https://www.postgresql.org/message-id/[email protected]

regards, tom lane

On Thu, Feb 25, 2021 at 06:07:06AM +0000, wangsh(dot)fnst(at)fujitsu(dot)com wrote:
> I noticed that this modification only commit into master branch,
> there is still have a problem on 12.6 or 13.2 on Windows.
>
> Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?

The change to be able to fix that stuff is invasive. So, while I
don't really object to a backpatch of this change in the future, I
think that it would be wiser to wait until we get more feedback with
the release of Postgres 14 before doing a backpatch to older
versions. So we are in a wait phase for the moment.

Thanks,
--
Michael

Thank you for sharing

Best regards
Shenhao Wang

-----Original Message-----
From: Michael Paquier <michael(at)paquier(dot)xyz>
Sent: Thursday, February 25, 2021 2:22 PM
To: Wang, Shenhao/王 申豪 <wangsh(dot)fnst(at)fujitsu(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>; Juan José Santamaría Flecha <juanjo(dot)santamaria(at)gmail(dot)com>; Emil Iggland <emil(at)iggland(dot)com>; PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: BUG #15858: could not stat file - over 4GB

On Thu, Feb 25, 2021 at 06:07:06AM +0000, wangsh(dot)fnst(at)fujitsu(dot)com wrote:
> I noticed that this modification only commit into master branch,
> there is still have a problem on 12.6 or 13.2 on Windows.
>
> Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?

The change to be able to fix that stuff is invasive. So, while I
don't really object to a backpatch of this change in the future, I
think that it would be wiser to wait until we get more feedback with
the release of Postgres 14 before doing a backpatch to older
versions. So we are in a wait phase for the moment.

Thanks,
--
Michael