Loud Baking

Ok, ok - I’ve been slacking on this blog again, but I will keep that for another post where I will announce some major changes I have been thinking of lately. For today, I’d like to introduce the new DBO Source Driver: HtmlSource - which is completely functional but still lacking some of the features I have planned for it.

So what’s an HTML DBO driver you ask?

Simply put, it’s a way to treat any HTML page like a database and be able to retrieve (scrape) certain parts using an SQL-like command:

SELECT href, title FROM a WHERE class="submit"

I first stumbled on the idea when I discovered the HtmlSql class by Jonas John but apart from it being a third party class, I disliked a couple other things like having to use ‘$class’ instead of ‘class’ in the WHERE clause, only using eval() and regular expressions, etc.

But why a DBO driver?

My main goal was to be able to spend the time after it is released on adding more services instead of upgrading the already available ones. With that in mind, I identified here below some of the reasons, feel free to (dis)agree in the comments:

• Unlike someone would believe in 2007, 90% of the sites/services scraped do not offer any webservices to facilitate the task.

• Scraping 100+ different sites/pages quickly becomes a hassle to maintain; mainly because all of those services will not tell you when the HTML code has changed for you to update your regular expressions accordingly.

• PHP’s native DOM objects don’t cut it. At least not for mass scraping. Maybe not as bad as custom regular expressions but still, a daunting maintenance.

The code

So without further ado and sorry for the lack of complete instructions yet, but here is the code I wanted to share with you today and hear suggestions/criticism if any.

In your /app/models/datasources/dbo directory, create the dbo_html.php file and in the /app/ directory, the curl.php one.

Example

Not to leave you completely unaware of how to use it, here is a tiny example that should help you figuring it all out - and maybe help with it?

Create a new model HtmlModel - this acts here as a dynamic model:

class HtmlModel extends AppModel
{
   var $useDbConfig = 'html';
   /**
    * Not used for the moment, but usually, the returned HTML model should
    * always have an 'html' element/table
    *
    * @var string
    * @access public
    */
   var $useTable = 'html';
   /**
    * Overwrites the Model::setSource method that breaks with a MissingTable error when
    * the fetched url doesn't return any HTML
    *
    * @param string
    * @todo add a filter to send more useful errors when no default 'html'
    */
   function setSource($tableName){}
   /**
    * Sets a different datasource (table/element)
    *
    * @param string
    * @return void
    * @access public
    */
   function switchTable($table)
   {
      parent::__construct(false, $table);
   }
}

In your controller you can now easily query the HTML page like any other Model by doing the following:

class ExampleController extends AppController
{
   var $uses = array('AnyOtherModel'); //can also be null or not defined
   function getLinks($url)
   {
      $this->setDataSource($url);
      loadModel('HtmlModel');
      //use it like this:
      $this->HtmlModel->setSource('a'); //to assign a virtual table to read from
      $this->HtmlModel->findAll(array('class' => 'submit'), 'href');
      //or
      $this->HtmlModel->query('SELECT href FROM a WHERE class="test"'));


  //continue by defining your view or more processing here.....


}
   function setDataSource($url)
   {
      $config = array(
         ‘driver’ => ‘html’,
         ‘host’ => $url,
         ‘username’ => null,
         ‘password’ => null,
         ‘database’ => ”,
         ‘dom’ => true,
         );
      $this->ds[’html’] = ConnectionManager::create(’html’, $config);
   }
}

This entry was posted on Thursday, December 6th, 2007 at 5:45 pm and is filed under Scraping & Parsing, CakePHP. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.