Scenario 1:
Manager: we need to add twitter-like functionality
Programmer: We need to integrate micro-blogging in our product?
Manager: It’ll make our product better.
Programmer: You want me to make twitter from scratch?
Manager: Yes, just save texts to a database and output it. Stop being lazy.

This scenario goes beyond the simple feature creep phenomena in that it is a direct result of managerial inexperience or ignorance (as this issue could be avoided with a properly trained inexperienced manager.)

I had become very aware of this when I was involved in a startup with a friend who wasn’t a very good programmer, but was a better visual designer/business person than I. At first I thought it was just growing pains associated with working with someone new, but some people NEVER learn. My perspective only solidified when I became a part of the corporate culture.

I’ve learned that the best way to deal with a situation like this (if being civil doesn’t work) is to be complicit and get them into the role of architecting their idea. They will either: give up right away, as they don’t want to do the work themselves; give up eventually, as they will realize the idea is unrealistic; or eventually fail, which will kill their credibility and their ego.

Programming should be left up to people who are experienced programmers… not “tech-savvy” teach yourself C++ programming in 21 days MBAs.

select distinct method

1 000 000 rows. Duplicates unlikely. Datalen 12 bytes
mysql> SELECT DISTINCT DATA FROM test;
959058 rows IN SET (1 min 37.36 sec)
 
1 000 000 rows. Duplicates unlikely. Datalen 255 bytes
mysql> SHOW processlist;
+----+------+----------------+--------+---------+------+------------------------------+--------------------------------+
| Id | User | Host           | db     | Command | Time | State                        | Info                           |
+----+------+----------------+--------+---------+------+------------------------------+--------------------------------+
|  9 | root | localhost:2728 | testdb | Query   |    0 | NULL                         | SHOW processlist               |
| 17 | root | localhost:3592 | testdb | Query   | 1243 | Copying TO tmp TABLE ON disk | SELECT DISTINCT DATA FROM test |
+----+------+----------------+--------+---------+------+------------------------------+--------------------------------+
 
1 000 000 rows. Duplicates unlikely. Datalen 12 bytes
mysql> INSERT INTO tmp(DATA) SELECT DISTINCT DATA FROM test;
Query OK, 959044 rows affected (3 min 4.89 sec)
Records: 959044  Duplicates: 0  Warnings: 0
 
1 000 000 rows. Duplicates unlikely. Datalen 255 bytes
mysql> INSERT INTO tmp(DATA) SELECT DISTINCT DATA FROM test;
mysql> SHOW processlist;
+----+------+----------------+--------+---------+------+------------------------------+------------------------------------------------------+
| Id | User | Host           | db     | Command | Time | State                        | Info                                                 |
+----+------+----------------+--------+---------+------+------------------------------+------------------------------------------------------+
|  8 | root | localhost:2725 | testdb | Query   | 7591 | Copying TO tmp TABLE ON disk | INSERT INTO tmp(DATA) SELECT DISTINCT DATA FROM test |
|  9 | root | localhost:2728 | testdb | Query   |    0 | NULL                         | SHOW processlist                                     |
+----+------+----------------+--------+---------+------+------------------------------+------------------------------------------------------+

equal data different key method

10 000 rows. Duplicates unlikely. Datalen 12 bytes. 
DELETE FROM `test` USING `test`, `test` AS vtable WHERE (NOT `test`.id = `vtable`.id) AND (`test`.DATA=`vtable`.DATA);
>5 mins

equal data different key improved method

10 000 rows. Duplicates unlikely. Datalen 12 bytes. 
mysql>delete FROM test USING test, test AS vtable WHERE (test.id >; vtable.id) AND (test.DATA=vtable.DATA);
>5 mins

add unique index method

1 000 000 rows. Duplicates unlikely. Datalen 12 bytes
mysql> ALTER IGNORE TABLE `test` ADD UNIQUE INDEX(DATA);
Query OK, 1000000 rows affected (1 min 51.14 sec)
Records: 1000000  Duplicates: 4  Warnings: 0
 
1 000 000 rows. Duplicates unlikely. Datalen 255 bytes
mysql> ALTER IGNORE TABLE `test` ADD UNIQUE INDEX(DATA);
Query OK, 1000000 rows affected (6 hours 37 min 49.35 sec)
Records: 1000000  Duplicates: 0  Warnings: 0

group by data method

1 000 000 rows. Duplicates unlikely. Datalen 12 bytes
mysql> CREATE TABLE temp_table AS SELECT * FROM test GROUP BY DATA;
Query OK, 959175 rows affected (10 min 50.14 sec)
Records: 959175  Duplicates: 0  Warnings: 0
 
1 000 000 rows. Duplicates unlikely. Datalen 255 bytes
mysql> SHOW processlist;
+----+------+----------------+--------+---------+------+------------------------------+-------------------------------------------------------------+
| Id | User | Host           | db     | Command | Time | State                        | Info                                                        |
+----+------+----------------+--------+---------+------+------------------------------+-------------------------------------------------------------+
|  9 | root | localhost:2728 | testdb | Query   | 5013 | Copying TO tmp TABLE ON disk | CREATE TABLE temp_table AS SELECT * FROM test GROUP BY DATA |
| 19 | root | localhost:2185 | testdb | Query   |    0 | NULL                         | SHOW processlist                                            |
+----+------+----------------+--------+---------+------+------------------------------+-------------------------------------------------------------+
2 rows IN SET (0.09 sec)

None of the above solutions are suitable for a number of reasons. Adding a unique index creates the problem of long insertion times for subsequent INSERT queries. The growth seems to be exponential. There isn’t a way to see how long the query will take to complete. Should the machine fail while the week> long query is executing then the table is likely to be corrupted, without being partially completed.

A better solution is to create a python script (or use any other language for that matter) that makes use of the MySQL API to delete duplicates. The design is as follows:

-N in memory hash tables that store the indices to a distinct column in a file
 
-N stream formatted (data + \0) files that store distinct columns
 
-N queues which store the hashes and the rows that need to be checked for their respective hash table
 
-Uniformly distributed hashing function to be used on the duplicate column

Multiple files help reduce (or eliminate) the overall seek-time on the hard drive. The result of the hashing function on a column specifies a hash table and an index into the hash table. i.e.

hash_table = fn(data)%num_hash_tables, index = fn(data)%size_of_hash_table

File offsets to a distinct column are stored in the hash table.

As each row is hashed it is placed into a queue for later processing. When a finite number (10 000~) rows has been distributed among the queues, each queue is processed in sequence to reduce hard drive seek time.

Python-like pseudo code with brevity in mind

row_cnt = 0
for row in database:
	ntable = hash(data)%num_hash_tables
	index = hash(data)%size_of_hash_table
	queues[ntable].push([index, data, primary_key])
	row_cnt++
	if row_cnt == 10000:
		for each entry in each queue:
			fileoffset  = hash_tables[num][index]
			if data == read_data(num + “_file”, fileoffset):
				add_new_entry_to_file_and_hash_table()
			else:
				delete_row(primary_key)
	row_cnt=0

Depending on the complexity of the hash function, size of the table, and size of the column, this method could be slower. For any significantly large non-indexed table it is better than any of the above solutions. It is also easy to determine how long the script will take to complete and it is also easy to make the script fault tolerant (could be safely interrupted).

For large databases – the number of rows exceeds the number of offsets that can be stored in memory – one would use a variation of the algorithm. Store N distinct rows in the hash tables, and keep track of the primary key when N values have been stored. Iterate through remaining rows deleting any duplicates. When finished iterating, empty hash tables and start storing new hashes/deleting duplicates beginning from the previous primary key that was stored.

more python-like pseudo code with brevity in mind

start_value = 0
new_start_value = 0
while True:
	row_cnt = 0
	for row in database starting at primary key start_value:
		ntable = hash(data)%num_hash_tables
		index = hash(data)%size_of_hash_table
		queues[ntable].push([index, data, primary_key])
		row_cnt++
		if row_cnt == 10000:
			for each entry in each queue:
				fileoffset  = hash_tables[num][index]
				if data != read_data(num + “_file”, fileoffset):
					add_new_entry_to_file_and_hash_table_if_not_full()
					if hash_table_full() and start_value == new_start_value:
						new_start_value = primary_key
				else:
					delete_row(primary_key)
		row_cnt=0
	empty_hash_tables()
	if start_value == new_start_value:
		break
	else:
		start_value = new_start_value

To put the performance increase into perspective, for 1 000 000 rows, datalen 255 with duplicates unlikely, it handled 10 000 rows in an average time of 4 seconds. All duplicates were removed in roughly 7 minutes, whereas all other solutions took hours. The speed increase gets better as the dataset gets larger.

Problem 1: I couldn’t stream H.264 files.
Fine print: I COULD play a local H.264 file

I tried working around the problem of not being able to stream H264 files by downloading portions of the video – luckily the bleeding edge version of JavaFX (1.2) allowed the output of httprequests to be redirected to file – and playing those portions in sequence.

Problem 2: javafx.io.http.HttpRequest locks file while downloading… which makes using a media file impossible until the download is complete.
Fine Print: I imagine that HttpRequest wasn’t meant to download media files…  but it would be incredibly useful to download and play a media file.

I then tried to build a library in native java that would release the file lock and allow JavaFX to load the media. This worked for the first bit that had been downloaded (and flushed to a file), but then I had to update file. I deleted the media object that made use of the local file and was hoping that it would release the file lock it put on the file while reading it.

Problem 3: It didn’t release the file lock… and there IS NOW WAY to force the release of system resources held by JavaFX.

There is no fine print for this… this is completely inexcusable.  It makes it impossible to update any resource (I am assuming this problem exists beyond video files I tried to play.) Whoever designed the language and omitted an interface – or a means – to delete a language native object, had a very narrow minded view on how the language should be used. I could’ve wrote each bit to a new file and then load the new file, but I would have to keep track of FLV offsets and correctly build a new FLV header for each new file and even then it will end up looking like a series of clips. At this point I had given up and moved onto a new idea that caught my interest. These are some more problems I’ve encountered that relate to JavaFX.

Problem 4: JavaFX had a problem handling  fileoffsets meta tag in the FLV header for some videos for me. To play videos I had to filter them out.  (Yes, it meant correctly rebuilding an FLV stream… yes… I did it =\)

Problem 5: The JavaFX forums were inactive. Not one person had replied to any thread I had started.

Problem 6: No one wants to wait 20 seconds for the JVM to fire-up just to play a quick game or music video.

Problem 7: Video support doesn’t come out of the box. So if you want your users to be able to watch videos… they will not only have to download Java and JavaFX (both are bundled), but also a third party codec pack.

In retrospect, I put a lot of work in hacking away at the problem. I don’t think I failed… I got it to work… it’s just that the effort  I – and a user – would have to go through makes using JavaFX impractical. At least it was fun learning about FLVs [2].

[1] http://www.javafx.com/docs/articles/media/format.jsp
[2] http://www.adobe.com/devnet/flv/pdf/video_file_format_spec_v10.pdf

I decided to submit the application to the Facebook public application directory (stupid me… I know) – which would make it visible to any user that is specifically looking for a super awesome application that lets them view the public albums of any user on Facebook. The application was submitted for approval and I waited. One day passed, then two – I thought that the approval process would only take one business day, since it just involves a person to add and try an application out. A couple days after submitting the application I noticed that I could no longer view Mark Zuckerberg’s photos. Not that I was stalking Mark Zuckerberg, but I like to run through things I make – daily – to make sure they aren’t broken. Viewing Mark Zuckerberg’s photos had become routine from a testing standpoint. Testing my app on other users showed that my application _still_ worked.

Doing a regex search in my apache logs showed that my application had been added (and removed) by _many_ ips in the Facebook subnet over 250 times. The regex count returned 255 (WHAT A ROUND NUMBER :D ). Each IP interacted with my application in some way, which suggests that it wasn’t a bot that kept on adding and removing my application. I got really excited because I thought that many employees of facebook saw my app, and in Mark Zuckerberg’s embarrassment, he changed the privacy setting of his albums (thought: MARK ZUCKERBERG HIMSELF SAW MY APP, WOW :D).

(photo from one of Mark Zuckerberg’s albums)

Today I tried my app, it barely returns any results on any search. All the searches I tried previously no longer work. Facebook hurriedly fixed a problem that didn’t exist… and it broke my app. What I mean by a problem that doesn’t exist is that I was taking advantage of a design decision Facebook made. Albums had a privacy setting that allowed everyone to view them. I could view what albums a user had using FQL, now I can’t (it still works for some people… but everyday I’m getting less and less results… so I assume Facebook is fixing each account sequentially or something.) The world readable attribute still exists and it’s still the default choice when creating a new album.

What bothers me most about this experience is not the hours I put in developing the app. Not that I shared this application publicly and won’t be able to use it privately. What bothers me most about this is that Facebook delayed the approval of my application for the public directory listing until it didn’t work. It’s their platform. They can change whatever functionality they want… send no notification to you…and if they break your app there’s nothing you can do about it.

Facebook did what they had to do to accomplish their goals (avoid embarassment, fix an initial design error, w/e.) I now view them as an evil company, but that’s just my opinion. Luckily, I cached (for optimization purposes) the albums IDs of Mark Zuckerberg’s albums :D.

ref:
[1] My application: http://apps.facebook.com/publicphotos/
[2] http://apps.facebook.com/publicphotos/album.php?a=17182073944
[3] http://apps.facebook.com/publicphotos/album.php?a=17181871868
[4] http://apps.facebook.com/publicphotos/album.php?a=17181903264
[5] http://apps.facebook.com/publicphotos/album.php?a=17181916415

The circumstances started with me needing a table to store MD5 hashes. The table definition I had come up with at the time was

mysql> DESCRIBE mytable;
+-------+------------+------+-----+------------------+-------+
| FIELD | Type       | NULL | KEY | DEFAULT          | Extra |
+-------+------------+------+-----+------------------+-------+
| HASH  | CHAR  (16) | NO   | PRI |                  |       |
+-------+------------+------+-----+------------------+-------+
1 row IN SET (0.03 sec)

The design error is that I had defined the `HASH` as char(16). A properly escaped MD5 digest would be stored correctly in the table and I was able to fetch any digest – via MySQL command line applications – as it was stored in the table (Data after \0’s would get returned.) There wasn’t any problem until I chose to make optimizations to the project and make use of functionality provided by a MySQL API.

When selecting any char(x) column from a table the API would read up to a \0 (NULL, int(0).) So I would get varying column lengths depending on the digest in the current row. It’s reasonable for the APIs to assume that because I have defined a column as a sequence of chars then a null terminated string would be stored in it … especially since there is another data type that is specifically used for binary data.

This realization came AFTER I had already populated the table with a lot of data. The table had grown to a fairly large size.

mysql> SELECT COUNT(*) FROM `mytable`;
 
+----------+
| COUNT(*) |
+----------+
| 38744395 |
+----------+
1 row IN SET (0.33 sec)

I was looking to fix my mistake. The ideal solution would be to change the column type. Research done via Google and looking through a book (High Performance MySQL) lead me to believe that

ALTER TABLE `mytable` MODIFY HASH BINARY(16) NOT NULL DEFAULT '’;

would be the best way to go about changing a column type. I was wrong. The query ran for days. This is the result of a “SHOW PROCESSLIST” a few days into the query.

mysql> SHOW processlist;
+------+------+----------------+----------+---------+--------+-------------------+----------------------------------------------------------------------+
| Id   | User | Host           | db       | Command | Time   | State             | Info                                                                 |
+------+------+----------------+----------+---------+--------+-------------------+----------------------------------------------------------------------+
|   82 | root | localhost:4442 | NULL     | Sleep   |     52 |                   | NULL                                                                 |
|  912 | root | localhost:3677 | mydb     | Query   | 275084 | copy TO tmp TABLE | ALTER TABLE `mytable` MODIFY HASH BINARY(16) NOT NULL DEFAULT ''     |
| 2672 | root | localhost:4527 | ******** | Query   |      0 | NULL              | SHOW processlist                                                     |
+------+------+----------------+----------+---------+--------+-------------------+----------------------------------------------------------------------+

A day later the query stopped executing because `mytable` was corrupted. It didn’t occur to me that my experimenting may have corrupted the table – so I didn’t bother running a consistency check. The MySQL client should’ve probably verified the tables before executing any query that is expected to run for more than a day. Regardless, I would have thought that the column modification would be near instantaneous. The column data wouldn’t need to be changed or converted (I used a single byte character set). All that would be required is for the particular column to be recognized as a different type. An ALTER TABLE query shouldn’t be this slow.

In contrast, a full table scan – which involves reading all the data from the table off of the hard drive – is completed within 8 minutes.

mysql> SELECT * FROM `mytable` WHERE `HASH` LIKE "%dsdfsdfsdfsd%" LIMIT 1;
Empty SET (7 min 1.45 sec)

So even if it needed to modify all the rows, and assuming that reading from disk takes as much time as writing to disk, then it should finish within 20 minutes. (The columns would have the same width so no data re-arrangement would need to take place.)

Without digging through the source code I can only provide speculation based on behavior and on information provided by third party sources. An excerpt from High Performance MySQL reads:

MySQL performs most alterations by making an empty table with the desired new
structure, inserting all the data from the old table into the new one, and deleting the
old table.
....
mysql> ALTER TABLE sakila.film
-> MODIFY COLUMN rental_duration TINYINT(3) NOT NULL DEFAULT 5;
 
Profiling that statement with SHOW STATUS shows that it does 1,000 handler reads and
1,000 inserts.

My understanding is that it creates a temporary table, fetches one row at a time from the original table and inserts one row at a time into the temporary table. The problem with this method is that it incurs a lot of overhead performing queries one row at a time. This overhead could be avoided by handling a lot of rows in a lot fewer queries. All this occurred to me while the ALTER TABLE had been executing, but I didn’t kill it because I wasn’t completely confident that this was the problem or that I had a better solution.

When the ALTER TABLE failed I tried an alternate solution that involves bulk SELECTs and INSERTs. I did the following steps:

1.	Create an SQL dump of mytable using mysqldump command
2.	Opened the SQL dump in an editor that supports large files (I used UEStudio.)
3.	I did a string-replace of char(16) to binary(16) (The reason I did a string replace is because UEStudio – when opening large files – would write whatever you typed directly to disk.. so for very large files a single keystroke would lag for about a minute.)
4.	Import the edited SQL file via  “mysql –uuser –p mydb < edited.sql”

This entire process was completed within an hour. Hopefully this post will help someone save a few days of their life.

RE: http://oreilly.com/catalog/9780596003067/
RE: http://dev.mysql.com/doc/refman/5.1/en/alter-table.html

[1]

SELECT * FROM `TABLE` ORDER BY RAND() LIMIT 1;

The problem with this solution is that it assigns a random value to each row and attempts to sort the entire table by the random value. I list here a variety of solutions I have found on the web.

[2]

$range_result = mysql_query( "SELECT MAX(`id`) AS max_id , MIN(`id`) AS min_id FROM `table` ");
$range_row = mysql_fetch_object( $range_result ); 
$random = mt_rand( $range_row->min_id , $range_row->max_id );
$result = mysql_query( " SELECT * FROM `table` WHERE `id` >= $random LIMIT 0,1 ");

Solution [2] scans the entire table up to $random. Worst case scenario – this requires a full table scan.

mysql> EXPLAIN SELECT * FROM `to_visit` WHERE `ID` >= 250;
 
+----+-------------+----------+------+---------------+------+---------+------+----------+-------------+
| id | select_type | TABLE    | type | possible_keys | KEY  | key_len | ref  | rows     | Extra       |
+----+-------------+----------+------+---------------+------+---------+------+----------+-------------+
|  1 | SIMPLE      | to_visit | ALL  | PRIMARY       | NULL | NULL    | NULL | 20345551 | USING WHERE |
+----+-------------+----------+------+---------------+------+---------+------+----------+-------------+
 
 
1 row IN SET (0.00 sec)

[3]

$offset_result = mysql_query( " SELECT FLOOR(RAND() * COUNT(*)) AS `offset` FROM `table` ");
$offset_row = mysql_fetch_object( $offset_result ); 
$offset = $offset_row->offset;
$result = mysql_query( " SELECT * FROM `table` LIMIT $offset, 1 " );

Solution [3] scans the entire database up to $offset. Worst case scenario – a full table scan is required.

mysql> EXPLAIN SELECT * FROM `to_visit` LIMIT 20, 1;
 
+----+-------------+----------+------+---------------+------+---------+------+----------+-------+
| id | select_type | TABLE    | type | possible_keys | KEY  | key_len | ref  | rows     | Extra |
+----+-------------+----------+------+---------------+------+---------+------+----------+-------+
|  1 | SIMPLE      | to_visit | ALL  | NULL          | NULL | NULL    | NULL | 20345551 |       |
+----+-------------+----------+------+---------------+------+---------+------+----------+-------+
 
 
1 row IN SET (0.00 sec)

[4]

SELECT * FROM `table` WHERE id >= (SELECT FLOOR( MAX(id) * RAND()) FROM `table` ) ORDER BY id LIMIT 1;

Solution [4] is equivalent to solution [2]. I’m not sure why the “ORDER BY id” was required at all. Worst case scenario – a full table scan is required.

mysql> EXPLAIN SELECT * FROM `to_visit` WHERE ID >= (SELECT FLOOR(MAX(`ID`) * RAND()) FROM `to_visit`) LIMIT 1;
 
+----+----------------------+----------+-------+---------------+---------+---------+------+----------+-------------+
| id | select_type          | TABLE    | type  | possible_keys | KEY     | key_len | ref  | rows     | Extra       |
+----+----------------------+----------+-------+---------------+---------+---------+------+----------+-------------+
|  1 | PRIMARY              | to_visit | ALL   | NULL          | NULL    | NULL    | NULL | 20345551 | USING WHERE |
|  2 | UNCACHEABLE SUBQUERY | to_visit | INDEX | NULL          | PRIMARY | 4       | NULL | 20345551 | USING INDEX |
+----+----------------------+----------+-------+---------------+---------+---------+------+----------+-------------+
 
 
2 rows IN SET (0.00 sec)

[5]

@rand_id := FLOOR(1 + (RAND() * (@max_id - 1)));
SELECT `column` FROM `table` WHERE `id` >= @rand_id ORDER BY `id` ASC LIMIT 1

Solution [5] is equivalent to solution [4]. However, if `column` _is_ a primary key then only the index table will be searched. Worst case scenario (`column` is not a primary key) a full table scan may be done. Best case scenario is that `column` is indexed and it would be significantly faster.

The main problem in all these solutions is the assumption that MySQL will do some magical handling for any reference to a column that has a primary key. MySQL will not use an indexed column if you are fetching anything other than indexed columns (not exactly true – see http://dev.mysql.com/doc/refman/5.0/en/multiple-column-indexes.html .)

My solution makes use of a table index. The following code is in Python (which means it should be very easy to read :D .)

cursor.execute("SELECT MAX(`ID`), MIN(`ID`) FROM `to_visit`")
result = cursor.fetchall()
maxid = result[0][0]
minid = result[0][1]
id = random.randint(minid, maxid)
cursor.execute("SELECT * FROM `to_visit` as t1 JOIN (SELECT ID FROM `to_visit` WHERE `ID`>="+str(id)+" limit 1) as t2 where t1.id = t2.id")
result = cursor.fetchall()

Selecting MAX(`ID`) and MIN(`ID`) should be constant time (it is for MyISAM storage engine.) The second query uses a random ID generated in python. Since the inner query fetches the `ID` – which is a primary key – only a scan of the index table is required. The outer join is performed in constant time because you are referencing a specific row by using one specific primary key.

mysql> EXPLAIN SELECT * FROM `to_visit` AS t1 JOIN (SELECT ID FROM `to_visit` WHERE `ID`>=1 LIMIT 1) AS t2 ON t1.id = t2.id;
 
+----+-------------+------------+--------+---------------+---------+---------+-------+----------+--------------------------+
| id | select_type | TABLE      | type   | possible_keys | KEY     | key_len | ref   | rows     | Extra                    |
+----+-------------+------------+--------+---------------+---------+---------+-------+----------+--------------------------+
|  1 | PRIMARY     | <derived2> | system | NULL          | NULL    | NULL    | NULL  |        1 |                          |
|  1 | PRIMARY     | t1         | const  | PRIMARY       | PRIMARY | 4       | const |        1 |                          |
|  2 | DERIVED     | to_visit   | INDEX  | PRIMARY       | PRIMARY | 4       | NULL  | 21254018 | USING WHERE; USING INDEX |
+----+-------------+------------+--------+---------------+---------+---------+-------+----------+--------------------------+
 
 
3 rows IN SET (0.00 sec)

I used this small snippet of code to benchmark my solution.

def fetch_random_row_test():
	global db
	start = time.time()
	cursor = db.cursor()
	cursor.execute("SELECT MAX(`ID`), MIN(`ID`) FROM `to_visit`")
	result = cursor.fetchall()
	maxid = result[0][0]
	minid = result[0][1]
	for x in range(0,1000):
		id = random.randint(minid, maxid)
		cursor.execute("SELECT * FROM `to_visit` as t1 JOIN (SELECT ID FROM `to_visit` WHERE `ID`>="+str(id)+" limit 1) as t2 where t1.id = t2.id")
		result = cursor.fetchall()
 
	print time.time() – start

Even though the code doesn’t use a performance timer it should still give fairly accurate and practical results. On my home computer it completed in 21 seconds (that is it fetched 1000 random rows in 21 seconds.) There are other issues that need to be addressed when selecting random rows from a MySQL table (such as fragmentation – which would make the “randomness” not so random :P .) I will maybe discuss fragmentation in a future blog post.

RE: http://dev.mysql.com/doc/refman/5.0/en/using-explain.html
RE: http://dev.mysql.com/doc/refman/5.0/en/multiple-column-indexes.html

“In response to a complaint we received under the US Digital Millennium Copyright Act, we have removed 1 result(s) from this page. If you wish, you may read the DMCA complaint that caused the removal(s) at ChillingEffects.org.”

I don’t live in the United States – I live in Canada. I contacted Google via their help desk as to why my search results were being filtered according to the US Digital Millennium Copyright Act law when I’m not even in the US. The next day – after no reply from Google – I searched for ‘UEStudio’ once again and the warning was gone. I don’t know if the missing result was displayed this time or if they just omitted the error message for that particular query. Searching for ‘torrent’ still yields the warning with 2 results being filtered. Google obviously didn’t fix the problem.

I can only speculate about what kind of design Google has in its data centers where a fix can only be pushed down to one query. Thousands of servers assemble the final result to display. Complex algorithms prioritize Canadian based website (on google.ca) while still taking into account every other website in the entire world. No doubt Google knows what they’re doing.

PROTIP: on the machine that assembles and/or pumps out the final result

/*The server is probably tied to the hostname, so you’d only have to check if there is a DMCA warning*/

If(HOSTNAME == “google.ca”) omit_US_DMCA_WARNING();

This solution – although inane – would be a more acceptable outcome than the one query fix that may or may not have given me the missing result.

Another solution would be to find out where the computer for a particular IP is approximately located. Geo IP address lookup is a very simple technology that only requires an indexed lookup (QUICK) of an IP to the approximate area the IP address belongs to. Google has made use of this in previous scenarios *recalls not being able to watch the official video of Bleeding Love on Youtube due to location restrictions*.

To Google’s credibility, the server I connect to while accessing google.ca is located in Mountain View California. So even though I am outside of the US, the server isn’t. Google should just build another data center in Canada.

Ideally, everyone would be able to share and access all information. Hopefully, all limitations imposed on us are ones signed by people we put in office. Sadly, my search results are being filtered and I get a fucking US DMCA warning message. Google is still better than Yahoo.

Update: Searching for ‘torrent’ doesn’t show the error anymore. Searching for ‘torrent download’ does.

Update 2:

Google has finally replied – after 9 days – to my initial query about DMCA in Canada. This is their reply:

I agree that it is unfortunate that DMCA complaints affect our users worldwide.

Google.com is the index from which all other domain specific indexes are drawn, which is why items that are removed from Google.com will also be removed from any sub-index. We will keep you informed anytime we remove content due to a DMCA complaint by inserting a notice at the bottom of the search results page that links to the original complaint. While this is not an ideal solution, following the link will allow you to view the URLs you would have otherwise seen in the search results.

If the Chilling Effects notice is no longer showing up for the same search this usually means one of two things:

1. That the site that had its page blocked filed a counter notification with us that resulted in the page being re-included into our search results.

or

2. The ranking of the blocked page altered after our last crawl.

Hopefully this helps explain why you’re seeing the notice in some searches and not others; however, I know that this can be a bit confusing so don’t hesitate to ask if you would like further clarification.

Hope that helps!

The main point being that all other indices around the world are derived from the main index at Google.com and therefore are at the mercy of whatever laws that index is bound to. I am suspicious about the possible reasons they give for the error not showing up anymore. For the first reason, it seems unlikely that two counter notifications would be processed right after me complaining about them. The second reason also seems unlikely because any site that does have a very low pagerank would be less prone to DMCA complaints. Nevertheless, the Google employee that had replied to my query had joined the helpdesk the day of answering my question and only answered my question. I imagine that it probably went up the support chain – which is why it took so long to answer – and they eventually did get back to me with an answer. I still like Google better than Yahoo or Live.