Database query

[rofl90]

Hi guys I have a list of sites that have been generated by a script, eg www.someweb.com/a-a-a-d102289.html

 

how could I crawl each of them, take some information from a specific div for instance ATTRACTION_REVIEW

 

and make a query to insert the contents of the div into a database

[rofl90]

Is there a way to ONLY get that div without looking through it?

[DarkWater]

You could use regular expressions...

[rofl90]

Could you giv eme a code example to get it from a site?

[DarkWater]

Give me an exact example of a div that you'd want extracted.

[rofl90]

<div id="ATTRACTION_REVIEW" class="listing">
<div class="details">
<div class="information textual">
<div class="description">
<b>Traveler Description:</b> <div class="toggle">
<span class="onHide">Attractions include favorites such as the Log Flume, Dodgems, Adventure Golf, Pan for Gold or brave the Horror Hotel. Located next to the sandy...
<span class="show">more &#xbb;</span> </span>
<span class="onShow">Attractions include favorites such as the Log Flume, Dodgems, Adventure Golf, Pan for Gold or brave the Horror Hotel. Located next to the sandy beaches, marina and working harbour at Littlehampton in West Sussex. It has all you need for a great family day out. New for 2008 - Indoor Cannon Ball Play Zone & Caterpillar Roller Coaster. Book online for family and group discounts.
<span class="hide">&#xab; less</span> </span>

</div>
</div>
<div class="description addDesc">
Familiar with Harbour Park? Write your own description and <a href="/Travel-g504226-d1005323/Littlehampton:United-Kingdom:Harbour.Park.html" rel="nofollow">share what you know</a> with other travelers.<br/>
</div>
</div><!--/ information.textual-->
<div class="information textual">
<div class="type"><b>Attraction type:</b> Amusement/theme park</div>
</div><!--/ information.textual-->
<div class="information bulleted">

<ul class="arrows">
<li><a target="_blank" href="http://www.harbourpark.com/">http://www.harbourpark.com/</a></li>
<li><a href="mailto:[email protected]">[email protected]</a></li>
</ul>
</div><!--/ information.bulleted-->
<div class="information textual adr">
<div>
<b>Address:</b> <span class="street-address">Seafront</span>
<span class="locality"> Littlehampton BN17 5LL</span>
<br/><span class="country-name">England</span> </div>

<div><b>Tel:</b> <span class="tel">01903 721200</span></div>
<div><b>Fax:</b> <span class="fax">01903 716663</span></div>
</div><!--/ information.textual-->
</div><!--/ details-->
</div><!--/ ATTRACTION_REVIEW.listing-->

[DarkWater]

The WHOLE contents?

[rofl90]

Yeh

[DarkWater]

You're luck that comment is there (the one that says that ATTRACTION_REVIEW.listing is over), otherwise you'd be screwed.  Here:

 

$file = file_get_contents("http://something.com/something.html");

$match = array();

preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match);

 

Then, $match[1] will have the content in that DIV.

[rofl90]

Thanks - this is my current code -

<?php
ob_start();
$CONF = array();
$CONF["DATABASE"] 				= 'x';
$CONF["USERNAME"] 				= 'x';
$CONF["PASSWORD"] 				= 'x';
$CONF["HOST"] 					= 'x';
mysql_connect($CONF["HOST"], $CONF["USERNAME"], $CONF["PASSWORD"]);
mysql_select_db($CONF["DATABASE"]);
$first = 100000;
while($first < 100001) {
$fileName = "http://www.x.com/x-g--d" . $first . ".html";
$file = file_get_contents($fileName);
$match = array();
if(preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match)) {
	if(mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')")) {
		echo "Success\n\n";
		$first++;
	}
	else {
		echo "Failure\n\n";
	}
}
}
ob_end_flush();
?>

 

It just continuously loads.

[DarkWater]

$first must not be getting incremented because this is failing:

 

mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')")

 

Instead of using it in the if statement, do this:

 

$result = mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')") OR die(mysql_error()); //important

if ($result) { }

[rofl90]

Unfortunately still a continual load.

 

Heres code, minus the mysql stuff:

mysql_connect($CONF["HOST"], $CONF["USERNAME"], $CONF["PASSWORD"]);
mysql_select_db($CONF["DATABASE"]);
$first = 100000;
while($first < 100001) {
$fileName = "http://www.x.com/x-g--d" . $first . ".html";
$file = file_get_contents($fileName);
$match = array();
if(preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match)) {
	$result = mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')") or die(mysql_error()); //important
	if($result) {
		echo "Success\n\n";
		$first++;
	}
	else {
		echo "Failure\n\n";
	}
}
}
ob_end_flush();
?>

[DarkWater]

Are you sure it's retrieving the file?  Also, add an else to the preg_match part that breaks that loop and echos an error.

[rofl90]

Still continually loading:

 

code:

 

$first = 100000;
while($first < 100001) {
$fileName = "http://www.x.com/x-g--d" . $first . ".html";
$file = file_get_contents($fileName);
$match = array();
if(preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match)) {
	$result = mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')") or die(mysql_error()); //important
	if($result) {
		echo "Success\n\n";
		$first++;
	}
	else {
		echo "Failure\n\n";
	}
}
else {
	echo "Error: preg_match part did not work";
}
}
ob_end_flush();
?>

[DarkWater]

Make it:

 

echo "Error: preg_match failed.";

break;

 

Instead of what you have.

[rofl90]

get rid of the braces?

[rofl90]

with the braces it gave me: Error: preg_match part did not work

[DarkWater]

Yeah, you need the braces, lol.

 

Try changing the regex to:

 

~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW\.listing-->~i

[rofl90]

same error

[rofl90]

does the . in ATTRACTION_REVIEW\.listing need to be escaped?

[DarkWater]

That's why I put that \, lol.  Try removing the ? after the + sign in the ( ).

[rofl90]

still giving off the error

[DarkWater]

Hmm...Make sure that $file is being filled correctly.

[rofl90]

Just tested it, it definitely is, it MUST be a problem with the regex, is it because it has a new line maybe? (\n)?

[DarkWater]

Yes, forgot that . doesn't cover new lines.  Try adding \n to the regex inside of the ( ).

 

([.\n]+)

 

Make it that.