It is a known fact that ext3 is not the most efficient file system out there and for example file removals can be painfully slow and cause a lot of random I/O. However, as it turns out, it can sometimes have a much more severe impact on the MySQL performance that it would seem. When or why?
When you run DROP TABLE there are several things that need to happen – write lock on a table so it cannot be used by any other thread, the data file(s) removal by the storage engine and of course in the end MySQL has to destroy the definition file (.frm). That's not all that happens, there is one other thing:
-
VOID(pthread_mutex_lock(&LOCK_open));
-
error= mysql_rm_table_part2(thd, tables, if_exists, drop_temporary, 0, 0);
-
pthread_mutex_unlock(&LOCK_open);
The entire code which removes a table is surrounded by LOCK_open mutex. The mutex is used in various places inside MySQL, but primarily when tables are being opened or closed. It means that when LOCK_open is locked, no query can be executed because they are stopped from accessing any table.
Here is when the slow file removal on the ext3 file system starts to be a pain. Deleting a 10GB file can take a few seconds and if that is a MySQL table, the mutex remains locked for all that time stalling all queries:
-
+-----+------+-----------+------+---------+------+----------------+------------------------------------------------+
-
| Id | User | Host | db | Command | Time | State | Info |
-
+-----+------+-----------+------+---------+------+----------------+------------------------------------------------+
-
| 1 | root | localhost | test | Query | 7 | NULL | drop table large_table |
-
| 329 | root | localhost | test | Query | 7 | Opening tables | select sql_no_cache * from other_table limit 1 |
-
+-----+------+-----------+------+---------+------+----------------+------------------------------------------------+
I tried some alternative approaches to trick MySQL into removing smaller files with DROP TABLE to minimize the effect, such as:
- TRUNCATE TABLE large_table; ALTER TABLE large_table ENGINE=…; DROP TABLE large_table;
- TRUNCATE TABLE large_table; OPTIMIZE TABLE large_table; DROP TABLE large_table;
Unfortunately as it turned out each of the administrative commands like ALTER TABLE or OPTIMIZE TABLE one way or another uses LOCK_open mutex when the old table files are deleted:
-
| 3 | root | localhost | test | Query | 7 | rename result table | ALTER TABLE large_table ENGINE=MyISAM |
-
| 679 | root | localhost | test | Query | 6 | Opening tables | select * from other_table limit 1 |
The only alternative seems to be changing the file system. To XFS for example, which handles file removals much more efficiently:
EXT3
-
mysql> drop table large_table;
-
Query OK, 0 rows affected (7.44 sec)
XFS
-
mysql> drop table large_table;
-
Query OK, 0 rows affected (0.29 sec)
A better solution through MySQL internals could be to simulate the table drop by renaming the corresponding data file or files and physically removing them outside of the mutex lock. However it may not be that simple, because the actual removal is performed by the storage engine, so it's not something MySQL code can control.
This is certainly not a common situation, but may become a problem to anyone when it's the least expected (e.g. dropping old unused tables).
Entry posted by Maciej Dobrzanski | One comment
Add to: 
| 
| 
| 
| 