php-java-bridge-users Mailing List for PHP/Java Bridge (Page 37)

I am using the JavaBridge template war for version 5.5.4.1 on Jetty. I
am using PHP on a separate IIS server. Running test.php on IIS with
the Java.inc from the Jetty server has this behavior Jetty. The
request begins, exactly 15 seconds pass, and then the request
completes showing all the Java System properties. So, it works, but
only after exactly 15000 ms.

On Tomcat the same arrangement completes the request instantly. I have
tried various settings in the 'define' section of Java.inc. I have
tried to properly set promiscuous mode in the web.xml file.

The only other clue is that if I test with a telnet PUT the console
output of Jetty says:

2010-02-14 06:36:43.029:INFO:/JavaBridge:handleHttpConnection init failed

Any idea what I should do to diagnose the problem with Jetty?

Hi!

> After bridge becomes "rock solid" under *heavy* loads, we could propose such
> ambitious code refactoring.

Do you mean that the bridge isn't "rock solid" under heavy load? Which problems does it have?


Peter




      

> PHP Java Bridge fills up the /tmp/-directory of our server with millions of
> pipe-files 

No, it doesn't, if used properly. This has been discussed in an old
thread. Please use a version >= 5.4.4 or switch off named pipes.


Regards,
Jost Bökemeier

Hello,

PHP Java Bridge fills up the /tmp/-directory of our server with millions of
pipe-files and also doesn't clean up as suggested when stopped and started.

Example:

-rw-------  1 www-data www-data    0 10. Feb 17:42 .php_java_bridgevXFmAs
prw-rw-rw-  1 www-data www-data    0 10. Feb 17:42 .php_java_bridgevXFmAs.i
prw-rw-rw-  1 www-data www-data    0 10. Feb 17:42 .php_java_bridgevXFmAs.o

Since the filesystem doesn't handle millions of files in just one directory
very well, I'm asking how to fix this behaviour.

Thanks in advance!

Best regards,
Thomas Lutz

Hi Andre,

sorry for the misunderstanding. I meant that the current code base
should be cleaned up by me, before you or anyone else can take over. The
refactoring will not take more than 4 hours.

>From my point of view the PHP/Java Bridge is complete. Maybe some parts
could be optimized, I don't know.
 

Okay, here's my plan:

I will do the refactoring next week and commit the changes into the
branch Release-6-0-0, which will become the head trunk. I will keep the
makefile (for SEL), but add an Ant script to build the JavaBridge.war.

I think you already have full CVS and project admin access. You should
also be able to add new developers to the project, if you want to.


Regards,
Jost Bökemeier

Hello,
I guess the suggestion of code refactoring is too much to accomplish into the
allowed time frame and likely number of colaborators.
I took into account that permission by Serpro for its employees to work at
project during business hours goes until december.
As this will be a voluntary participation effort, I guess that up to 10 skilled
people could join. Maybe much less...
So, I suggest tasks, at first, that clean a bit, secure  and stabilize present
code base.
After bridge becomes "rock solid" under *heavy* loads, we could propose such
ambitious code refactoring.
Suggestions?
Regards.
Andre Felipe Machado

+1 for maven and GIT.

Maven eases usage of the bridge in other projects,
Git is just superior to svn... ;)

Hosting on Github would ease integration
of changes made by others.



On Tue, Feb 9, 2010 at 8:10 PM,
<php...@li...> wrote:
> On Tuesday 09 February 2010, php...@li... wrote:
>> * Remove the autoconf scripts and support for RedHat's GNU Java/GCJ.
>> Developers should be able to use a Java IDE like Netbeans or Eclipse to
>> contribute to the project.
>
> If I may add my humble opinion at this exact point,
>
> you should really consider using either Ant or Maven or even Ivy (or a
> combination of Java-based build tools) for the build system of the php-java
> bridge project (the war project and anything else, if build seperately).
>
> I've seen the current build script, I'm not sure that it's possible without
> major re-construction, but at least a try would be very much welcome! :)
>
>
> --
> Thanos Kyritsis <djart at linux.gr>
>
> - What's your ONE purpose in life ?
> - To explode, of course! ;-)
>
> ------------------------------------------------------------------------------
> The Planet: dedicated and managed hosting, cloud storage, colocation
> Stay online with enterprise data centers and the best network in the business
> Choose flexible plans and management services without long-term contracts
> Personal 24x7 support from experience hosting pros just a phone call away.
> http://p.sf.net/sfu/theplanet-com
> _______________________________________________
> php-java-bridge-users mailing list
> php...@li...
> https://lists.sourceforge.net/lists/listinfo/php-java-bridge-users
>



-- 
Dominik Dorn
http://dominikdorn.com

On Tuesday 09 February 2010, php...@li... wrote:
> * Remove the autoconf scripts and support for RedHat's GNU Java/GCJ.
> Developers should be able to use a Java IDE like Netbeans or Eclipse to
> contribute to the project.

If I may add my humble opinion at this exact point, 

you should really consider using either Ant or Maven or even Ivy (or a 
combination of Java-based build tools) for the build system of the php-java 
bridge project (the war project and anything else, if build seperately).

I've seen the current build script, I'm not sure that it's possible without 
major re-construction, but at least a try would be very much welcome! :)


-- 
Thanos Kyritsis <djart at linux.gr>

- What's your ONE purpose in life ?
- To explode, of course! ;-)

Hi Andre,

this sounds very interesting!

At the moment the code contains a lot of old cruft which nobody
understands, but which is kept for backward compatibility.

For example I would like to get rid of the following "features"

* java_require() and dynamic class loading

* The PhpCGIServlet and the CGIServlet. The FastCGI servlet should be
rewritten.

* .NET and mono support is obsolete

* the custom log4j support should go

* everything from the bridge/http/ package should be removed. These
context runners are difficult to understand. I would use chunked
connections everywhere.

* Remove the autoconf scripts and support for RedHat's GNU Java/GCJ.
Developers should be able to use a Java IDE like Netbeans or Eclipse to
contribute to the project.


What about creating a fork, something like "PHP/Java Bridge light"? The
old PHP/Java Bridge code could still be maintained by me. And bug fixes
could be taken from the new code base.

Do you need a new SVN or GIT repository or is a branch sufficient?


Regards,
Jost Bökemeier

Hi Jost,

Thanks for the very reasonable response. I won't respond to it in detail 
here as you've requested, but the way you've explained it has made it 
much clearer for me.

I still don't understand why my code is working given what you have 
said, though it is much revised from the first email I sent. Maybe it 
will break and I will experience what you are telling me for myself, 
which won't be a bad thing. But I've taken enough of your time already, 
so I'll shut up.

Thanks for helping out.


Mark...

Mark,

you don't seem to understand the issue.

1. A GET request must be answered by the HTTP 1.1 server by either
sending the Content-Length header or by sending the Transfer-Encoding:
chunked header.

2. To send large responses efficiently, Transfer-Encoding: chunked is
used.

3. The network buffer used by the PHP streams implementation reads data
eagerly. If you fread($socket, 1024) and the network buffer already
contains 24 bytes, PHP will try to read 1000 bytes nevertheless.

4. If you open a HTTP 1.0 connection to a HTTP server which cannot
respond with a Content-Length header, the behaviour is implementation
specific. It may send an error, send a raw byte stream or it may start
playing nethack.

5. PHP running within Apache behaves exactly the same as PHP running
within a JEE server or servlet engine. The only difference is that Java
prints a stack trace when the hanging PHP instance is killed, while
Apache silently kills it with SIGCHILD.


You don't have many choices to implement your "stream download". Either
you use HTTP/1.1 chunked connections, or you will have to deal with an
implementation-specific behaviour of both, the HTTP server (#4) *and*
PHP (#3).

Since the PHP streams implementation cannot handle raw byte streams
which don't have an explicit length (either Content-Length or the length
from each "chunked" header), you cannot use PHP fread --- your "fgets()"
is bogus because you do not and cannot distinguish between the \r\n from
the data and the \r\n used to split the packets, so your download is
garbage anyway --- to receive the data, unless the remote server shuts
down the connection at the end of the transfer, so that PHP's fread (or
fgets) doesn't try to fill its network buffer over the end of the data. 

If you want to discuss this issue further, please use the PHP mailing
list instead. 


Regards,
Jost Bökemeier

Hello,
I am beeing buried under other works last months, but finally got authorization
from my employer to announce this: 
http://www.serpro.gov.br is wanting to help php-java-bridge (mainly at Debian
systems) to improve its security, scalability, stability for optimal suitability
at government size production environments.
It is part of the SERPRO effort to help [0] Debian Project.

We can leverage the offer to get help improve the SE Linux rules, for example. 

What are other skills in need that we could ask for more help, towards their
objectives (that are also good for the project)?
Serpro can donate some of its employees man hours during business hours to the
project effort (mine included from now onwards) with tracking.

Regards.
Andre Felipe Machado


[0] http://www.techforce.com.br/news/linux_blog/debian_serpro_preliminary_report_1

> Yes, "mostly", indeed. The code you have posted doesn't work on any
> operating system or HTTP server.

I am very aware that you are much more experienced in these matters than 
me, but your statement does not explain why the code successfully 
downloaded about 2.6GB of data consisting of 25-30 different files in 
testing yesterday.

Experiences like this imply that it does work. Though I accept that it 
might not be the most efficient way of doing it and I'm happy to accept 
the advice of people who have more knowledge than me and can help me to 
see better solutions.

I did see this when I was reading back to look again at Peter's code:

 > If the data you receive is encoded in HTTP/1.1 chunks, your fgets() is
 > incorrect, too. The HTTP/1.1 chunk data format is:

Received data isn't chunk encoded. Maybe I've not been clear on this 
point by using the words 'chunk download' in a comment. It was my figure 
of speech, rather than a declaration of intent to use chunk encoding. 
Perhaps 'buffer download' would've been more appropriate.

 > Peter
> has already posted code which works

To my inexperienced eye, this looks like it will receive chunk encoded 
data, but the server I'm using is not passing data using this method. 
Though I do appreciate Peter's response - I don't think I acknowledged 
it at the time (Cheers Peter!).

To me, it seems that you believe I am receiving chunk encoded data but 
I'm not. I can see that my code would be completely wrong if I was 
trying to do that and your comments about the code would be completely 
valid. Your frustration that I've not understood this point is clear 
from the last two emails.

However, advice from StackOverflow confirms that my code should work 
fine, that fread() should be used to make it more binary-safe, that CURL 
would be even better, and chunk encoding is not needed. So I have other 
approaches to try.

Thanks again for taking the time out to reply. I don't think this needs 
to be discussed further either. And many thanks for providing libraries 
without which I would not have been able to proceed as far with this 
project as I have.

Regards,


Mark...

On Mon, 2010-02-08 at 00:34 +0700,
php...@li... wrote:

> I found an approach that worked...mostly.

Yes, "mostly", indeed. The code you have posted doesn't work on any
operating system or HTTP server. 


>  But it caused PHP to crash 

It certainly doesn't cause a crash. It causes PHP to hang for the
reasons outlined in the previous post.


> under JavaBridge and Tomcat. An experienced Java programmer saw the logs 
> and said it was due to a class error. 

Please re-read the error log. The hanging PHP instance has been killed
during the tomcat shutdown sequence...

Anyway, I don't think we need to discuss this issue any further, Peter
has already posted code which works (except that I would read the
terminating \r\n, too, but that's a matter of style). Just nuke your
code and use his code instead. 

Please re-read the HTTP/1.1 spec for details why your code is completely
wrong and causes PHP to hang sometimes.


Regards,
Jost Bökemeier

> In any case, this isn't due to bugs in PHP or the PHP/Java Bridge, but
> due to your misunderstanding of how HTTP chunked encoding works and what
> fgets() and feof() do.

I comfortably admit I am new to downloading very large files in PHP, and 
I'm trying to learn.

I found an approach that worked...mostly. But it caused PHP to crash 
under JavaBridge and Tomcat. An experienced Java programmer saw the logs 
and said it was due to a class error. I contacted this mailing list and 
explained what was happening and was told that it was down to a feature 
not yet supported in JavaBridge. Now I'm being told it's not down to that.

I don't have any understanding of HTTP chunked encoding works. I'm not 
using this header in my code and your references to it were the first 
that I had seen. I'll read up on this and see how I can use it given 
what you say. And looking at the implementation of chunked encoding will 
give me more information on how fgets() and feof() work generally, and 
in the context of chunked encoding.

Again, many thanks for helping out. It is appreciated.


Mark...

On Sun, 2010-02-07 at 15:19 +0700,
php...@li... wrote:
> I'm just trying to download the response in a way 

How else do you want to download the files efficiently, if not via
Transfer-Encoding: Chunked?


> The downloads only fail 'sometimes', so I don't understand how some of
> the explanations 

Given your code, the possibility that PHP reads "over the edge" depends
on the file size. If the file is large it is more likely that this will
happen.

In any case, this isn't due to bugs in PHP or the PHP/Java Bridge, but
due to your misunderstanding of how HTTP chunked encoding works and what
fgets() and feof() do.


Regards,
Jost Bökemeier

> If the data you receive is encoded in HTTP/1.1 chunks

I don't believe it is, but I don't know a lot about these things, so I 
might be wrong.

There's nothing special happening at the server side. It's just a GET 
request and response. I'm just trying to download the response in a way 
that doesn't cause me memory problems with files that can be 100MBs in size.

At present, the code below is producing an adequate result, ie: it's not 
failed in testing yet. The downloads only fail 'sometimes', so I don't 
understand how some of the explanations offer here fit to what's 
happening. The retry improves the chance of the downloading completing. 
If a file does not complete after the retries, the client handles this, 
and the downloads are retried again later.

Thanks thanks for everyone's advice though. Some of it is over my head 
as a PHP newbie, but I'm understanding and learning other parts of what 
is being said.

Cheers,


Mark...


           $download_attempt = 1;
           do {
             $fs = fsockopen($host, 80, $errno, $errstr, 30);

             if (!$fs) {
               $this->writeDebugInfo("FAILED ", $errstr . '(' . $errno . 
')');
             } else {
               $out = "GET $file HTTP/1.1\r\n";
               $out .= "Host: $host\r\n";
               $out .= "Connection: Close\r\n\r\n";
               fwrite($fs, $out);

               $fm = fopen ($temp_file_name, "w");
               stream_set_timeout($fs, 30);

               while(!feof($fs) && ($debug = fgets($fs)) != "\r\n" ); // 
ignore headers

               while(!feof($fs)) {
                 $contents = fgets($fs, 4096); // Chunk download
                 fwrite($fm, $contents);
                 $info = stream_get_meta_data($fs);
                 if ($info['timed_out']) {
                   break;
                 }
               }
               fclose($fm);
               fclose($fs);

               if ($info['timed_out']) {
                 // Delete temp file if fails
                 unlink($temp_file_name);
                 $this->writeDebugInfo("FAILED on attempt " . 
$download_attempt . " - Connection timed out: ", $temp_file_name);
                 $download_attempt++;
                 if ($download_attempt < 5) {
                   $this->writeDebugInfo("RETRYING: ", $temp_file_name);
                 }
               } else {
                 // Move temp file if succeeds
                 $media_file_name = str_replace('temp/', 'media/', 
$temp_file_name);
                 rename($temp_file_name, $media_file_name);
                 $this->writeDebugInfo("SUCCESS: ", $media_file_name);
               }
             }
           } while ($download_attempt < 5 && $info['timed_out']);

Hi Peter,

> The apache server sends garbage if you send http 1.0 but expect http
> 1.1.

are we talking about this apache feature which can stream bytes to the
client?

This feature is quite useful on C level, you can simply read() the bytes
as apache generates them. But it is difficult to handle in PHP or Bea
WebLogic for example, because their network implementations use a buffer
which reads data eagerly. 

If the PHP network buffer contains 4 bytes and fread() requests 20
bytes, PHP tries to read 16 bytes from the network, instead of simply
returning the 4 bytes available so far. I don't know why PHP and other
JEE servers implement their network buffers this way. It might be more
efficient for normal HTTP connections, but it is bad for data packets
which don't have an explicit length; in PHP you must call fread($sock,
1), if you don't want to run into a dead lock. Or you must unblock the
stream and call stream_select() for each packet. Very low tech...


Regards,
Jost Bökemeier

Hello!

The apache server sends garbage if you send http 1.0 but expect http 1.1.

The following code will work:

$sock = fsockopen(...);
fwrite($sock, "GET /foo.php HTTP/1.1\r\nHost: localhost\r\n\r\n");
while(fgets($sock)!="\r\n");
while($len=hexdec(fgets($sock))) {
  for($str=fread($sock, $len); $len>strlen($str); $str.=fread($sock, $len-strlen($str)));
echo $str;
}

Peter



      

On Sat, 2010-02-06 at 23:36 +0700,
php...@li... wrote:
> > Does the remote server respond with HTTP/1.1 by any chance?
> 
> It does. 

If the data you receive is encoded in HTTP/1.1 chunks, your fgets() is
incorrect, too. The HTTP/1.1 chunk data format is:
length\r\n
data\r\n

The last packet is:
000\r\n\r\n


I don't think that there's a bug in PHP or the PHP/Java Bridge: Java.inc
uses chunked connections too.  Please take a look at the class
java_ChunkedSocketChannel from Protocol.inc for an example how to
receive HTTP/1.1 chunked data.


I think the bug is in the line:

 contents = fgets($fs, 4096); // Chunk download

The packet may be longer than 4096 bytes. You must extract the data
length from eack packet header.


Regards,
Jost Bökemeier

> Does the remote server respond with HTTP/1.1 by any chance?

It does.

> If so, the feof() test will not work, the last packet is zero, but the
> connection isn't closed.

It's failing before EOF.

> So the 20KB you receive may be the remainder of the last connection.

This might still be something to do with it, but on the last test with 
CURL and a 8k buffer:

175kb file downloaded okay
66MB file hung after 44kb downloaded

It often works fine. So I'm considering using fsockopen and using 
'set_time_limit' on each iteration and catching the exception when it 
hangs to retry the file a number of times. This will be a first for me.

I appreciate your advice with this one. And the question I've been 
avoiding asking is now looming in my mind - any idea when 5.5.6 might be 
released? ;-)

Thanks again,


Mark...

Does the remote server respond with HTTP/1.1 by any chance?

If so, the feof() test will not work, the last packet is zero, but the
connection isn't closed.

So the 20KB you receive may be the remainder of the last connection.


Regards,
Jost Bökemeier

> May be not. -- I just saw that your code doesn't use php://input, but
> opens a HTTP/1.0 URL connection to some resource.
>
> What resource is it? A local or a remote resource?

It is actually a remote resource. The code is installed on any PC, and 
downloads data from a specified and fixed Amazon Web Service S3 bucket.

So I don't think this is covered by the scenario you described.

But many thanks for looking at this as I understand it's not really in 
the scope of JavaBridge.

Cheers,


Mark...

In PHP/Java Bridge version 5.5.5 you can use the PHP function
"virtual()" to include a local resource. I have added the following to
our FAQ:

How do I include a local (*.asp, *.jsp, ...) resource?
Use the virtual() function. If PHP is running within a JEE server or
servlet engine, the virtual() function is an alias for 

        java_context()->getHttpServletRequest()->getRequestDispatcher()->include()

Warning: Do not open a "loop back" url connection (e.g. via
fopen("http://localhost.../foo.asp")) to include the local resource.
This might exceed the HTTP server's pool size and create a deadlock!.


Regards,
Jost Bökemeier

> So it should be added in the next release, and I'll just need to add the
> 'Transfer-Encoding: Chunked' header.

May be not. -- I just saw that your code doesn't use php://input, but
opens a HTTP/1.0 URL connection to some resource.

What resource is it? A local or a remote resource?

If it is a resource served by the originating HTTP server or servlet
engine, we've found the bug.


If you want to include some local resource, use
requestDispather.include() or the Apache API instead. A "loop-back"
URL connection does not work, because the Apache process- and servlet
thread pool is limited. If it runs out of resources, further requests
might create a dead lock, they will block all further requests.
Imagine a thread pool of 2, with two php scripts creating 2 additional
requests to the same resource. These scripts will block the HTTP
server or servlet engine forever.



Regards,
Jost Boekemeier