VT’s Tech Blog

I’ve always wanted to know how the various opensource search engines performed when running head to head with each other. I’ve personally not had the time nor patience to setup such a benchmark. Luckily Vik Singh has done just that and blogged the results of the benchmark.

He’s benchmarked the latest versions of Lucene, sqlite, Xapian, zettair and sphinx while indexing twitter messages (968,937 tweets to be exact) and Medical data sets. I was not too surprised when Lucene came up as a winner from these benchmarks

Read the Blog post by Vik to get the details of the benchmarks.

How often have you tested your application on your local machine and had it run properly, but when you deploy it and the database starts getting filled up, you start noticing performance problems?
Here’s something you can use to test out your application with databases which are not empty – generatedata.com allows you to define your table structure and the type of data to fill it with. You can export the generated data as SQL (MySQL and Oracle are supported currently)  insert statements or csv which you can import into your databases. The online version allows you to export 200 rows of generated data. If you want larger datasets to be generated, you can download the datagenerator script and run it on your own server.

The best part of the script is instead of generating totally random string or numeric data, you some some semi-realistic data which can be used for testing.

Here’s a sample form I created on the site (click on the image to enlarge):

The SQL generated from this form:

CREATE TABLE myTable (
  id mediumint(8) unsigned NOT NULL auto_increment,
  id MEDIUMINT default NULL,
  name varchar(255) default NULL,
  city varchar(50) default NULL,
  country varchar(50) default NULL,
  dateofjoining varchar(50) default NULL,
  PRIMARY KEY (`id`)
) TYPE=MyISAM AUTO_INCREMENT=1;
 
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('1','Inez','Muncie','Grenada','2009-04-19 12:55:49');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('2','Reese','Washington','Suriname','2009-10-17 01:28:15');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('3','Hope','Edina','Guyana','2008-02-22 10:17:55');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('4','Cameron','Sister Bay','Malaysia','2008-02-21 01:38:18');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('5','Zephr','Lebanon','Syrian Arab Republic','2008-12-09 21:40:04');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('6','Byron','Indio','Afghanistan','2009-03-15 08:50:17');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('7','Carol','Bay St. Louis','British Indian Ocean Territory','2009-04-16 04:57:43');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('8','Latifah','Geneva','Albania','2009-08-25 00:14:26');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('9','Nina','Fall River','Solomon Islands','2009-01-03 21:45:51');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('10','Jorden','Ardmore','Latvia','2009-01-20 19:31:36');

Get more information at: generatedata.com

An interesting article I came across today by Johannes Schlüter, who works in the one of the MySQL teams in Sun. He’s done some interesting work writing a MySQL Storage Engine using PHP. His article shows how to do some basic using this.

MySQL 5.1 has a plugin interface to easily add storage engines. PHP can easily embedded into other applications. So why not combine these two things? – Writing a MySQL Storage Engine which reads data by calling a PHP script.

This article is interesting because of the uses which come up when you can create custom Storage Engines using PHP. An interesting usecase for this is in creating test cases, where you can feed the data for a query from PHP and trap the activities on the database.

Head out and read this article and take a look at the code at his blog: MySQL Storage Engine based on PHP

You can also take a look at the Launchpad page for this project.

The following output and code is from the Lauchpad page which shows the PHP code and the queries in MySQL:

< ?php
class Test extends ArrayIterator implements MySQLStorage_Writable, MySQLStorage_Updatable, MySQLStorage_Deletable {
    public function write($data) {
        $this[] = $data;
    }
 
    public function update($data) {
        $this[$this->key()] = $data;
    }
 
    public function delete() {
        unset($this[$this->key()]);
    }
}
 
function create_table($table, $data) {
    return true;
}
 
function open_table($table) {
    return new Test(array(
        array('id' => 1, 'a' => 'foo'),
        array('id' => 2, 'a' => 'bar')));
    }
}
?>

Sample Queries run on this engine:

mysql> SELECT * FROM php_write;
+------+------+
| id   | val  |
+------+------+
| 1    |  foo |
| 2    | bar  |
+------+------+
2 rows in set (0.00 sec)
mysql> CREATE TABLE php_write (id int, val CHAR(3)) ENGINE=PHP;
Query OK, 0 rows affected (0.00 sec)
mysql> UPDATE php_write SET val = 'baz' WHERE id = 1;
Query OK, 1 row affected (0.02 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> DELETE FROM php_write WHERE id = 2;
Query OK, 1 row affected (0.02 sec)
mysql> INSERT INTO php_write VALUES(3, 'bar');
Query OK, 1 row affected (0.00 sec)
mysql> SELECT * FROM php_write;
+------+------+
| id   | val  |
+------+------+
| 1    | baz  |
| 3    | bar  |
+------+------+
2 rows in set (0.01 sec)

Scrawlr is short for SQL Injector and Crawler, a tool developed by the HP Web Security Research Group in coordination with the Microsoft Security Response Center in response to the widespread SQL injection attacks on the web.

“Scrawlr will crawl a website while simultaneously analyzing the parameters of each individual web page for SQL Injection vulnerabilities. Scrawlr is lightning fast and uses our intelligent engine technology to dynamically craft SQL Injection attacks on the fly. It can even provide proof positive results by displaying the type of backend database in use and a list of available table names. There is no denying you have SQL Injection when I can show you table names!”

Key Features of Scrawlr include:

• Identify Verbose SQL Injection vulnerabilities in URL parameters
• Can be configured to use a Proxy to access the web site
• Will identify the type of SQL server in use
• Will extract table names (verbose only) to guarantee no false positives

Scrawlr which is a free tool has a few limitations which are it’s crawl only upto 1500 pages, doesn’t support Blind SQL injection and will not test for Post parameters for SQL injection. Overall even with these limitations, it’s still a useful tool to check your sites to see if you’re safe from SQL injections.

Links:
Download Scrawlr
Scrawlr Forum

via: communities.hp.com

Here’s a problem we had come across today. Whenever we tried inserting data into a certain table, MySQL kept throwing up the following error:

mySQL error: 1062
Duplicate entry '0' for key 1

After scratching our heads over this one, we tried to alter the table to set the autoincrement field to the next number and even that failed to fix the problem. Then hunting around the web, we found the cause of this problem. It seems that Mysql throws this error because the field type of the auto increment field is not large enough to hold the next value, so it tries to wraps the count back to ‘0′ where the is already a record with that value.

We changed the field from int to unsigned bigint which fixed the problem for us. So if you face the same problem, check if your autoincrement field has maxed the datatype for that field.

How do you sort a varchar field which has the dates stored in it? If you try the usual order by on the varchar fields you’ll have trouble with the sort order, since MySQL will order this field like a varchar and you will surely have problems with dates being sorted this way. Take the following example of a table which has a field called feed_date which defined as varchar. Now ordering the results by feed_date will not give the correct order like in the following example:

mysql> select * from datetest order by feed_date;
+----+-------------------+------------+
| id | feed_name         | feed_date  |
+----+-------------------+------------+
|  1 | VTs Tech Blog     | 1-23-2008  |
|  2 | vinuthomas.com    | 12-23-2006 |
|  3 | sitesandsounds.in | 3-12-2008  |
+----+-------------------+------------+
3 rows in set (0.00 sec)

Hmmm… 2006 coming in between 2008 doesn’t seem right does it? The only way to get MySQL to sort this result properly will be if we can get MySQL to understand this data as a date field. To to this, we can use the mysql function STR_TO_DATE. This function allows us to convert a string to date and allow us to specify in which format the date is present.

So to convert the current date format in the feed_date column which is in the mm-dd-yyyy format, we’ll have to use this syntax: STR_TO_DATE(feed_date, ‘%m-%d-%Y’). Now we can use this converted data to sort on:

mysql> SELECT id, feed_name, feed_date ,
 STR_TO_DATE( feed_date, '%m-%d-%Y' ) AS date_for_sort
 FROM `datetest` ORDER BY date_for_sort;
+----+-------------------+-------------+--------------+
| id   | feed_name        | feed_date  | date_for_sort|
+----+--------------------+------------+--------------+
|  2   | vinuthomas.com   | 12-23-2006 | 2006-12-23   |
|  1   | VTs Tech Blog    | 1-23-2008  | 2008-01-23   |
|  3   | sitesandsounds.in| 3-12-2008  | 2008-03-12   |
+----+--------------------+------------+--------------+
3 rows in set (0.00 sec)

Disclaimer: I know that doing this is going a round about way to getting this done, this is just a tip to help out when we have a problem like this and can’t really change the database field declaration and you have tons of data in the field which doesn’t confirm to the MySQL date format in a varchar field.

Link: MySQL Manual on STR_TO_DATE

Stumbled across this interesting article at Leo Büttiker’s blog which tells why pagination on a web app is a database killer:

On the view of your database the worst thing you can do in your web app is paging. Paging is horrible in the view of performance. To explain let me take a little example:

SELECT SQL_CALC_FOUND_ROWS gb.*,
u.username,
u.uid,
u.geschlecht,
u.mitfoto,
[... some more fields...]
FROM member_gold_guestbook gb
LEFT JOIN users u ON u.uid=gb.uid_from
[... some more left joins...]
WHERE gb.uid_to=’22152′
AND visible=’1′
LIMIT 0,10;

That’s not that bad at all, but when you go to page 300 your database server will hat you for this. The database server has not only to calculate the 10 items you want to show but also all 3000 previous items.

Sure you may argue nobody will ever go to page 300. Somebody will not, but “googlebot” and his evil brothers will. And the bad thing is that you can, as long as you need paging, nothing do against it. There are just a few tricks that may reduce your server load a bit.

Read the whole article at: @leo’s :: Why your database says paging sucks!

Last week I’d finally decided to move off from Windows on my work laptop. After the default installation off the 64 bit CD, here were some of the problems I found:

• The default Wireless drivers didn’t seem to work at all.
• The 64 bit version of the OS seemed to lack some of the softwares like Flash player and w32codecs.
• The external monitor/projecter didn’t work off the default installation.
• My development stack of LAMP had to be installed manually.
• Very basic multimedia support.

I reinstalled the laptop with the 32-bit OS instead, to get a hold of the missing software on the 64 bit version. The wireless was still down. A quick google search landed me at the following blog, which had a great step-by-step instruction on how to get ndiswrapper and the windows version of the Broadcom wireless driver to work on the laptop.
Instruction for installing the Broadcom drivers through ndiswrapper: http://vale.homelinux.net/wordpress/?p=144

To get the external display working, I followed the instructions over at Ubuntu’s community help site: https://help.ubuntu.com/community/BinaryDriverHowto/ATI.
The default open source drivers didn’t have support for TV-out. The instructions I followed were quite simple:

Install linux-restricted-modules and restricted-manager provied in the restricted repositories:
sudo apt-get update
sudo apt-get install linux-restricted-modules-generic restricted-manager
Open the restricted drivers manager included in 7.04 “System -> Administration -> Restricted Drivers Manager” and select “ATI accelerated graphics driver”. This will hopefully enable fglrx in a painless way. If not, follow the instructions for Edgy.

Apart from the instructions above you’ll also have to install fglrx-control. At the shell prompt type in:

sudo apt-get install fglrx-control

After this you can access the ATI control panel by issuing the following command at the prompt.

sudo fireglcontrol

These took care of my basic necessities. Had my network and basic software installed. Being a LAMP developer, I had to get LAMP installed on the laptop. Here’s how to get it done:

apt-get install apache2
apt-get install php5 libapache2-mod-php5
apt-get install php5-cli php5-dev
apt-get install php-pear

The first two gets apache2, and the php modules installed. The next two get the php command line, php dev libraries and pear installed on the system. Next let’s get MySQL up and running:

apt-get install mysql-server mysql-client

This installs MySQL server and the client setup. To setup the mysql libraries for PHP5, run the following:

apt-get install php5-mysql

Linux still lacks an IDE for LAMP development, so while I search the net for a suitable candidate, I guess gedit or Quanta will work fine for coding. One of the candidates in the open source area for this task is PDT, I’m yet to evaluate it.

To get all the other multimedia goodies installed on my system, I used BUMPS.

If your company has some nagging legacy web applications which refuse to work on anything other than IE, you can also install IE on Ubuntu using ies4linux. Here’s a link which shows you how do go about this:

http://www.psychocats.net/ubuntu/ies4linux

Be warned that having IE on your system legally requires that you have a valid license for Windows.

If you’re pulling out your hair wondering why the hell will some one install IE on Linux, it’s just that some nasty web-application developers make cross-browser compatible apps that work on any browser as long as it is IE!

A large number of wireless broadband services like sprint wireless and verizon wireless have sprouted with increasing trend of wireless. Also, with the advent of wireless internet hotspots, wireless internet coverage has expanded.

Amazon’s S3 service seems to be getting into everything ‘online’ these days. Here’s an announcement by Mark Atwood of a S3 storage engine for MySQL 5.

It allows one to view and manipulate Amazon’s S3 storage service as
tables and items by MySQL. You can keep your blobs or large varchars
or truely huge datasets in S3, and then join the tables against your
local ones.

Announcement: MySQL Lists: internals: Storage engine for Amazon S3
Mark’s project page: http://fallenpegasus.com/code/

Here’s a quick tip in MySQL. Let’s say you have a table called browsercount like the one below:

Let’s say you have a web-analytic application which has to add a browser into the table if it doesn’t exist, but update the count for the browser if the entry already exists. How do you go about that?

The usual way would be to first query the table to see if the entry exists, then fire an insert statement or an update statement based on the results.

Here’s how to perform the same action in one SQL statement:

INSERT INTO browsercount (browser,count) VALUES ('Firefox',1) ON DUPLICATE KEY UPDATE count=count+1;

This will work if the id field in the table above is declared as UNIQUE or is a Primary Key.

The query basically tells mysql to insert the data into the database, but if a duplicate key is found, increment the count field by 1. It’s much faster than firing up 2 sql queries from your PHP script