Optimising databases

optimising_databases.md

SQL Databases

Underlying structure of these databases are rows and columns. They are called relational databases. This term doesn't address Rails relations - associations, but relations between data. Data is linked by keys. Main databases used across the world: postgresql, mysql, sqllite All SQL databases require creating migrations in rails.

PostgreSQL vs MySQL vs SQLite

PostreSQL is one of the best maintained databases. It is flexible and powerfull, but not so easy to use, because it requires running server and have its own syntax. This database implements full SQL standard and supports reliable transactional operations. It is programmable and extensible. In read-heavy apps it's performance might be worse than MySQL. In simple project postgresql maintenance and setup might be an overkill.

MySQL as well as PostgreSQL requires running server. It doesn't implement all features of SQL standard. It's secure and easy to setup and operate on but sometimes it's less reliable because of inner SQL implementation. This open source project isn't developed much anymore. It doesn't handle concurrency well.

SQLite is file-based so no server is required. This database is really fast and responsive because it depends on single file in app directory. Communication with file is bi-directional. The worst part is that is has no user management, it isn't flexible - no way to tinker around and it supports only one write operation. It's best for tests and rapid development / prototyping.

NoSQL databases

It is less used branch of database types. They are not relational in definition. he main difference between SQL and noSQL is that noSQL depends on JSON-like data structure with key/value pairs. These databases doesn't require Rails migrations - all fields can be specified within ActiveModel and changed by a reload. Example of noSQL databases: MongoDB, Redis. These databases are mostly less reliable. We don't use them at Netguru anymore.

Indexing in SQL

Proper indexing in database might speed it up exponentially, but it's not easy. Use indexing with caution!

1. Do not index every column
2. Index only frequently used columns
3. Avoid indexing heavily manipulated columns
4. Find most used queries and support it by proper indexing
5. Indexes have to be rebuilt from time to time (they get fractured)
6. Use concatenanted indexes wisely, think how queries work. No sense to place first column in index if it has unique values.
7. If concatenated indexes will cover range queries - don't place ranged column as first.
8. Consider using partial indexes.

Optimising Rails database queries

1. Watch out for n+1 queries. Remedy: includes, eager_load. With eager_load there might be duplicated data since it uses left_outer_join.
2. Map & pluck. Pluck returns only an array of selected fields and it's way faster than fetching all data and filtering it. We can't use instance methods on data fetched with pluck.
3. If we want to update columns on many records we might use update_columns or update_all - it minimizes number of queries and time taken but omits the callbacks.
4. While using sum on model make sure to pass column name as symbol, not a block. Product.sum(:value)
5. When fetching unique data watch out for uniq statement placement. It may be SQL or Array operation.
6. If we are sure about the data we will evefr need to fetch we might strip it down by select
7. Advanced technique is to to use data caching
8. We can use postgres views to aggregate complex queries and present them as a logical table in postgresql database. Rails doesn't support it out of the box, but we can use gem scenic. To store data physically we can use materialized views - the performance is better, but data is duplicated and needs to be refreshed periodically.