Here’s something in the making, a PHP extension which caches MySQL queries irrespective of the extension being used. It’s a plugin for mysqlnd, the native MySQL driver for PHP called  mysqlnd_qc. This currently requires PHP 5.3.3-dev which is still in development.

The extension allows caching buffered queries through mysql_query and mysqli_query. It allows cache-invalidation though TTL or a user defined callback. This extension is still in the protoype stage right now and will evolve as the project progresses.

The Query Cache is implemented as a PHP extension. It is written in C and operates “under the hood” of PHP. During the start up of the PHP interpreter it gets registered as a mysqlnd plugin to replaces selected mysqlnd methods.

At PHP run time it proxies queries send from mysqlnd/PHP to the MySQL server. If a query string starts with the SQL hint (/*qc=on*/) to enable caching of it and the query is not cached (Cache miss), the query cache plugin will record the raw wire protocol data send from MySQL to PHP to answer the query. The query cache records the wire protocol data in its cache medium and replays it, if still valid, on a cache hit.

Note that the query cache does not hold decoded result sets consisting of zvals (C struct representing a PHP variable). It stores the raw wire data of the MySQL client server protocol. In case of a cache hits, mysqlnd still needs to decode the cached raw wire data into PHP variables before passing the result to the user space. This approach has one major advantage: simplicity.

You can get more information and installation instructions over at the MySQLND Query Cache Project page at Mysql Forge.

Check out this slide which gives you more information about this extension:

I’ve always wanted to know how the various opensource search engines performed when running head to head with each other. I’ve personally not had the time nor patience to setup such a benchmark. Luckily Vik Singh has done just that and blogged the results of the benchmark.

He’s benchmarked the latest versions of Lucene, sqlite, Xapian, zettair and sphinx while indexing twitter messages (968,937 tweets to be exact) and Medical data sets. I was not too surprised when Lucene came up as a winner from these benchmarks

Read the Blog post by Vik to get the details of the benchmarks.

How often have you tested your application on your local machine and had it run properly, but when you deploy it and the database starts getting filled up, you start noticing performance problems?
Here’s something you can use to test out your application with databases which are not empty – generatedata.com allows you to define your table structure and the type of data to fill it with. You can export the generated data as SQL (MySQL and Oracle are supported currently)  insert statements or csv which you can import into your databases. The online version allows you to export 200 rows of generated data. If you want larger datasets to be generated, you can download the datagenerator script and run it on your own server.

The best part of the script is instead of generating totally random string or numeric data, you some some semi-realistic data which can be used for testing.

Here’s a sample form I created on the site (click on the image to enlarge):

The SQL generated from this form:

CREATE TABLE myTable (
  id mediumint(8) unsigned NOT NULL auto_increment,
  id MEDIUMINT default NULL,
  name varchar(255) default NULL,
  city varchar(50) default NULL,
  country varchar(50) default NULL,
  dateofjoining varchar(50) default NULL,
  PRIMARY KEY (`id`)
) TYPE=MyISAM AUTO_INCREMENT=1;
 
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('1','Inez','Muncie','Grenada','2009-04-19 12:55:49');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('2','Reese','Washington','Suriname','2009-10-17 01:28:15');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('3','Hope','Edina','Guyana','2008-02-22 10:17:55');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('4','Cameron','Sister Bay','Malaysia','2008-02-21 01:38:18');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('5','Zephr','Lebanon','Syrian Arab Republic','2008-12-09 21:40:04');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('6','Byron','Indio','Afghanistan','2009-03-15 08:50:17');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('7','Carol','Bay St. Louis','British Indian Ocean Territory','2009-04-16 04:57:43');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('8','Latifah','Geneva','Albania','2009-08-25 00:14:26');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('9','Nina','Fall River','Solomon Islands','2009-01-03 21:45:51');
INSERT INTO myTable (id,name,city,country,dateofjoining) VALUES ('10','Jorden','Ardmore','Latvia','2009-01-20 19:31:36');

Get more information at: generatedata.com

An interesting article I came across today by Johannes Schlüter, who works in the one of the MySQL teams in Sun. He’s done some interesting work writing a MySQL Storage Engine using PHP. His article shows how to do some basic using this.

MySQL 5.1 has a plugin interface to easily add storage engines. PHP can easily embedded into other applications. So why not combine these two things? – Writing a MySQL Storage Engine which reads data by calling a PHP script.

This article is interesting because of the uses which come up when you can create custom Storage Engines using PHP. An interesting usecase for this is in creating test cases, where you can feed the data for a query from PHP and trap the activities on the database.

Head out and read this article and take a look at the code at his blog: MySQL Storage Engine based on PHP

You can also take a look at the Launchpad page for this project.

The following output and code is from the Lauchpad page which shows the PHP code and the queries in MySQL:

< ?php
class Test extends ArrayIterator implements MySQLStorage_Writable, MySQLStorage_Updatable, MySQLStorage_Deletable {
    public function write($data) {
        $this[] = $data;
    }
 
    public function update($data) {
        $this[$this->key()] = $data;
    }
 
    public function delete() {
        unset($this[$this->key()]);
    }
}
 
function create_table($table, $data) {
    return true;
}
 
function open_table($table) {
    return new Test(array(
        array('id' => 1, 'a' => 'foo'),
        array('id' => 2, 'a' => 'bar')));
    }
}
?>

Sample Queries run on this engine:

mysql> SELECT * FROM php_write;
+------+------+
| id   | val  |
+------+------+
| 1    |  foo |
| 2    | bar  |
+------+------+
2 rows in set (0.00 sec)
mysql> CREATE TABLE php_write (id int, val CHAR(3)) ENGINE=PHP;
Query OK, 0 rows affected (0.00 sec)
mysql> UPDATE php_write SET val = 'baz' WHERE id = 1;
Query OK, 1 row affected (0.02 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> DELETE FROM php_write WHERE id = 2;
Query OK, 1 row affected (0.02 sec)
mysql> INSERT INTO php_write VALUES(3, 'bar');
Query OK, 1 row affected (0.00 sec)
mysql> SELECT * FROM php_write;
+------+------+
| id   | val  |
+------+------+
| 1    | baz  |
| 3    | bar  |
+------+------+
2 rows in set (0.01 sec)

“Scrawlr will crawl a website while simultaneously analyzing the parameters of each individual web page for SQL Injection vulnerabilities. Scrawlr is lightning fast and uses our intelligent engine technology to dynamically craft SQL Injection attacks on the fly. It can even provide proof positive results by displaying the type of backend database in use and a list of available table names. There is no denying you have SQL Injection when I can show you table names!”

Key Features of Scrawlr include:

• Identify Verbose SQL Injection vulnerabilities in URL parameters
• Can be configured to use a Proxy to access the web site
• Will identify the type of SQL server in use
• Will extract table names (verbose only) to guarantee no false positives

Scrawlr which is a free tool has a few limitations which are it’s crawl only upto 1500 pages, doesn’t support Blind SQL injection and will not test for Post parameters for SQL injection. Overall even with these limitations, it’s still a useful tool to check your sites to see if you’re safe from SQL injections.

Links:
Download Scrawlr
Scrawlr Forum

via: communities.hp.com

mySQL error: 1062
Duplicate entry '0' for key 1

After scratching our heads over this one, we tried to alter the table to set the autoincrement field to the next number and even that failed to fix the problem. Then hunting around the web, we found the cause of this problem. It seems that Mysql throws this error because the field type of the auto increment field is not large enough to hold the next value, so it tries to wraps the count back to ’0′ where the is already a record with that value.

We changed the field from int to unsigned bigint which fixed the problem for us. So if you face the same problem, check if your autoincrement field has maxed the datatype for that field.

How do you sort a varchar field which has the dates stored in it? If you try the usual order by on the varchar fields you’ll have trouble with the sort order, since MySQL will order this field like a varchar and you will surely have problems with dates being sorted this way. Take the following example of a table which has a field called feed_date which defined as varchar. Now ordering the results by feed_date will not give the correct order like in the following example:

mysql> select * from datetest order by feed_date;
+----+-------------------+------------+
| id | feed_name         | feed_date  |
+----+-------------------+------------+
|  1 | VTs Tech Blog     | 1-23-2008  |
|  2 | vinuthomas.com    | 12-23-2006 |
|  3 | sitesandsounds.in | 3-12-2008  |
+----+-------------------+------------+
3 rows in set (0.00 sec)

Hmmm… 2006 coming in between 2008 doesn’t seem right does it? The only way to get MySQL to sort this result properly will be if we can get MySQL to understand this data as a date field. To to this, we can use the mysql function STR_TO_DATE. This function allows us to convert a string to date and allow us to specify in which format the date is present.

So to convert the current date format in the feed_date column which is in the mm-dd-yyyy format, we’ll have to use this syntax: STR_TO_DATE(feed_date, ‘%m-%d-%Y’). Now we can use this converted data to sort on:

mysql> SELECT id, feed_name, feed_date ,
 STR_TO_DATE( feed_date, '%m-%d-%Y' ) AS date_for_sort
 FROM `datetest` ORDER BY date_for_sort;
+----+-------------------+-------------+--------------+
| id   | feed_name        | feed_date  | date_for_sort|
+----+--------------------+------------+--------------+
|  2   | vinuthomas.com   | 12-23-2006 | 2006-12-23   |
|  1   | VTs Tech Blog    | 1-23-2008  | 2008-01-23   |
|  3   | sitesandsounds.in| 3-12-2008  | 2008-03-12   |
+----+--------------------+------------+--------------+
3 rows in set (0.00 sec)

Disclaimer: I know that doing this is going a round about way to getting this done, this is just a tip to help out when we have a problem like this and can’t really change the database field declaration and you have tons of data in the field which doesn’t confirm to the MySQL date format in a varchar field.

Link: MySQL Manual on STR_TO_DATE

On the view of your database the worst thing you can do in your web app is paging. Paging is horrible in the view of performance. To explain let me take a little example:

SELECT SQL_CALC_FOUND_ROWS gb.*,
u.username,
u.uid,
u.geschlecht,
u.mitfoto,
[... some more fields...]
FROM member_gold_guestbook gb
LEFT JOIN users u ON u.uid=gb.uid_from
[... some more left joins...]
WHERE gb.uid_to=’22152′
AND visible=’1′
LIMIT 0,10;

That’s not that bad at all, but when you go to page 300 your database server will hat you for this. The database server has not only to calculate the 10 items you want to show but also all 3000 previous items.

Sure you may argue nobody will ever go to page 300. Somebody will not, but “googlebot” and his evil brothers will. And the bad thing is that you can, as long as you need paging, nothing do against it. There are just a few tricks that may reduce your server load a bit.

Read the whole article at: @leo’s :: Why your database says paging sucks!

• The default Wireless drivers didn’t seem to work at all.
• The 64 bit version of the OS seemed to lack some of the softwares like Flash player and w32codecs.
• The external monitor/projecter didn’t work off the default installation.
• My development stack of LAMP had to be installed manually.
• Very basic multimedia support.

I reinstalled the laptop with the 32-bit OS instead, to get a hold of the missing software on the 64 bit version. The wireless was still down. A quick google search landed me at the following blog, which had a great step-by-step instruction on how to get ndiswrapper and the windows version of the Broadcom wireless driver to work on the laptop.
Instruction for installing the Broadcom drivers through ndiswrapper: http://vale.homelinux.net/wordpress/?p=144

To get the external display working, I followed the instructions over at Ubuntu’s community help site: https://help.ubuntu.com/community/BinaryDriverHowto/ATI.
The default open source drivers didn’t have support for TV-out. The instructions I followed were quite simple:

Install linux-restricted-modules and restricted-manager provied in the restricted repositories:
sudo apt-get update
sudo apt-get install linux-restricted-modules-generic restricted-manager
Open the restricted drivers manager included in 7.04 “System -> Administration -> Restricted Drivers Manager” and select “ATI accelerated graphics driver”. This will hopefully enable fglrx in a painless way. If not, follow the instructions for Edgy.

Apart from the instructions above you’ll also have to install fglrx-control. At the shell prompt type in:

sudo apt-get install fglrx-control

After this you can access the ATI control panel by issuing the following command at the prompt.

sudo fireglcontrol

These took care of my basic necessities. Had my network and basic software installed. Being a LAMP developer, I had to get LAMP installed on the laptop. Here’s how to get it done:

apt-get install apache2
apt-get install php5 libapache2-mod-php5
apt-get install php5-cli php5-dev
apt-get install php-pear

The first two gets apache2, and the php modules installed. The next two get the php command line, php dev libraries and pear installed on the system. Next let’s get MySQL up and running:

apt-get install mysql-server mysql-client

This installs MySQL server and the client setup. To setup the mysql libraries for PHP5, run the following:

apt-get install php5-mysql

Linux still lacks an IDE for LAMP development, so while I search the net for a suitable candidate, I guess gedit or Quanta will work fine for coding. One of the candidates in the open source area for this task is PDT, I’m yet to evaluate it.

To get all the other multimedia goodies installed on my system, I used BUMPS.

If your company has some nagging legacy web applications which refuse to work on anything other than IE, you can also install IE on Ubuntu using ies4linux. Here’s a link which shows you how do go about this:

http://www.psychocats.net/ubuntu/ies4linux

Be warned that having IE on your system legally requires that you have a valid license for Windows.

If you’re pulling out your hair wondering why the hell will some one install IE on Linux, it’s just that some nasty web-application developers make cross-browser compatible apps that work on any browser as long as it is IE!

A large number of wireless broadband services like sprint wireless and verizon wireless have sprouted with increasing trend of wireless. Also, with the advent of wireless internet hotspots, wireless internet coverage has expanded.

It allows one to view and manipulate Amazon’s S3 storage service as
tables and items by MySQL. You can keep your blobs or large varchars
or truely huge datasets in S3, and then join the tables against your
local ones.

Announcement: MySQL Lists: internals: Storage engine for Amazon S3
Mark’s project page: http://fallenpegasus.com/code/

Let’s say you have a web-analytic application which has to add a browser into the table if it doesn’t exist, but update the count for the browser if the entry already exists. How do you go about that?

The usual way would be to first query the table to see if the entry exists, then fire an insert statement or an update statement based on the results.

Here’s how to perform the same action in one SQL statement:

INSERT INTO browsercount (browser,count) VALUES ('Firefox',1) ON DUPLICATE KEY UPDATE count=count+1;

This will work if the id field in the table above is declared as UNIQUE or is a Primary Key.

The query basically tells mysql to insert the data into the database, but if a duplicate key is found, increment the count field by 1. It’s much faster than firing up 2 sql queries from your PHP script

I see if frequently with web applications and it is often indication of trouble. Not only it means you may run out of MySQL connections quicker than you expected but it also frequently indicates serious problems in the application. If you do not use persistent connections and you have connection in Sleep stage for 600 seconds what could it be ? It may mean some of your pages take that long to generate (or might be the code simply gets into the tight loop and page never gets generated) it also could mean some of external Web Services are slow or not available and you’re not dealing with timeouts properly. Or may be you have several connections to MySQL server and right now running query which takes that long ? In any case it is something frequently worth looking at.

Link: MySQL Performance Blog » Debugging sleeping connections with MySQL

Link: MySQL Activity Report

For Installation Instructions, head over to: http://www.linuxforums.org/forum/servers/72890-how-mysql-activity-report.html

Today I was working with application which uses Innodb and foreign keys and got into locking problems possibly due to foreign keys, so I did a little investigation on that matter.

Peter Zaitsev goes on to outline an example of when this could happen and warns developers to look at locking overheads while implementing foreign keys in Innodb

Link: MySQL Performance Blog » Innodb locking and Foreign Keys

Use the following MySQL statements to convert your database and tables:

ALTER DATABASE mydatabase
CHARACTER SET utf8
DEFAULT CHARACTER SET utf8
COLLATE utf8_general_ci
DEFAULT COLLATE utf8_general_ci ;

ALTER TABLE mytable
DEFAULT CHARACTER SET utf8
COLLATE utf8_general_ci ;