diff --git a/src/docs/contributor/database.diviner b/src/docs/contributor/database.diviner index 6572e42fc1..9a83a067b2 100644 --- a/src/docs/contributor/database.diviner +++ b/src/docs/contributor/database.diviner @@ -1,213 +1,213 @@ @title Database Schema @group developer This document describes key components of the database schema and should answer questions like how to store new types of data. Database System =============== Phorge uses MySQL or another MySQL-compatible database (like MariaDB or Amazon RDS). Phorge uses the InnoDB table engine. The only exception is the `search_documentfield` table which uses MyISAM because MySQL doesn't support fulltext search in InnoDB (recent versions do, but we haven't added support yet). We are unlikely to ever support other incompatible databases like PostgreSQL or SQLite. PHP Drivers =========== -Phorge supports [[ http://www.php.net/book.mysql | MySQL ]] and -[[ http://www.php.net/book.mysqli | MySQLi ]] PHP extensions. +Phorge supports [[ https://www.php.net/book.mysql | MySQL ]] and +[[ https://www.php.net/book.mysqli | MySQLi ]] PHP extensions. Databases ========= Each Phorge application has its own database. The names are prefixed by `phabricator_` (this is configurable with `storage.default-namespace`). Phorge uses a separate database for each application. To understand why, see @{article:Why does Phorge need so many databases?}. Connections =========== Phorge specifies if it will use any opened connection just for reading or also for writing. This allows opening write connections to a primary and read connections to a replica in primary/replica setups (which are not actually supported yet). Tables ====== Most table names are prefixed by their application names. For example, Differential revisions are stored in database `phorge_differential` and table `differential_revision`. This generally makes queries easier to recognize and understand. The exception is a few tables which share the same schema over different databases such as `edge`. We use lower-case table names with words separated by underscores. Column Names ============ Phorge uses `camelCase` names for columns. The main advantage is that they directly map to properties in PHP classes. Don't use MySQL reserved words (such as `order`) for column names. Data Types ========== Phorge defines a set of abstract data types (like `uint32`, `epoch`, and `phid`) which map to MySQL column types. The mapping depends on the MySQL version. Phorge uses `utf8mb4` character sets where available (MySQL 5.5 or newer), and `binary` character sets in most other cases. The primary motivation is to allow 4-byte unicode characters to be stored (the `utf8` character set, which is more widely available, does not support them). On newer MySQL, we use `utf8mb4` to take advantage of improved collation rules. Phorge stores dates with an `epoch` abstract data type, which maps to `int unsigned`. Although this makes dates less readable when browsing the database, it makes date and time manipulation more consistent and straightforward in the application. We don't use the `enum` data type because each change to the list of possible values requires altering the table (which is slow with big tables). We use numbers (or short strings in some cases) mapped to PHP constants instead. JSON and Other Serialized Data ============================== Some data don't require structured access -- we don't need to filter or order by them. We store these data as text fields in JSON format. This approach has several advantages: - If we decide to add another unstructured field then we don't need to alter the table (which is slow for big tables in MySQL). - Table structure is not cluttered by fields which could be unused most of the time. An example of such usage can be found in column `differential_diffproperty.data`. Primary Keys ============ Most tables have an auto-increment column named `id`. Adding an ID column is appropriate for most tables (even tables that have another natural unique key), as it improves consistency and makes it easier to perform generic operations on objects. For example, @{class:LiskMigrationIterator} allows you to very easily apply a migration to a table using a constant amount of memory provided the table has an `id` column. Indexes ====== Create all indexes necessary for fast query execution in most cases. Don't create indexes which are not used. You can analyze queries @{article:Using DarkConsole}. Older MySQL versions are not able to use indexes for tuple search: `(a, b) IN ((%s, %d), (%s, %d))`. Use `AND` and `OR` instead: `((a = %s AND b = %d) OR (a = %s AND b = %d))`. Foreign Keys ============ We don't use foreign keys because they're complicated and we haven't experienced significant issues with data inconsistency that foreign keys could help prevent. Empirically, we have witnessed first hand as `ON DELETE CASCADE` relationships accidentally destroy huge amounts of data. We may pursue foreign keys eventually, but there isn't a strong case for them at the present time. PHIDs ===== Each globally referenceable object in Phorge has an associated PHID ("Phorge ID") which serves as a global identifier, similar to a GUID. We use PHIDs for referencing data in different databases. We use both auto-incrementing IDs and global PHIDs because each is useful in different contexts. Auto-incrementing IDs are meaningfully ordered and allow us to construct short, human-readable object names (like `D2258`) and URIs. Global PHIDs allow us to represent relationships between different types of objects in a homogeneous way. For example, infrastructure like "subscribers" can be implemented easily with PHID relationships: different types of objects (users, projects, mailing lists) are permitted to subscribe to different types of objects (revisions, tasks, etc). Without PHIDs, we would need to add a "type" column to avoid ID collision; using PHIDs makes implementing features like this simpler. For more information, see @{article:Handles Technical Documentation} Transactions ============ Transactional code should be written using transactions. Example of such code is inserting multiple records where one doesn't make sense without the other, or selecting data later used for update. See chapter in @{class:LiskDAO}. Advanced Features ================= We don't use MySQL advanced features such as triggers, stored procedures or events because we like expressing the application logic in PHP more than in SQL. Some of these features (especially triggers) can also cause a great deal of confusion, and are generally more difficult to debug, profile, version control, update, and understand than application code. Schema Denormalization ====================== Phorge uses schema denormalization sparingly. Avoid denormalization unless there is a compelling reason (usually, performance) to denormalize. Schema Changes and Migrations ============================= To create a new schema change or migration: **Create a database patch**. Database patches go in `resources/sql/autopatches/`. To change a schema, use a `.sql` file and write in SQL. To perform a migration, use a `.php` file and write in PHP. Name your file `YYYYMMDD.patchname.ext`. For example, `20141225.christmas.sql`. **Keep patches small**. Most schema change statements are not transactional. If a patch contains several SQL statements and fails partway through, it normally can not be rolled back. When a user tries to apply the patch again later, the first statement (which, for example, adds a column) may fail (because the column already exists). This can be avoided by keeping patches small (generally, one statement per patch). **Use namespace and character set variables**. When defining a `.sql` patch, you should use these variables instead of hard-coding namespaces or character set names: | Variable | Meaning | Notes | |---|---|---| | `{$NAMESPACE}` | Storage Namespace | Defaults to `phabricator` | | `{$CHARSET}` | Default Charset | Mostly used to specify table charset | | `{$COLLATE_TEXT}` | Text Collation | For most text (case-sensitive) | | `{$COLLATE_SORT}` | Sort Collation | For sortable text (case-insensitive) | | `{$CHARSET_FULLTEXT}` | Fulltext Charset | Specify explicitly for fulltext | | `{$COLLATE_FULLTEXT}` | Fulltext Collate | Specify explicitly for fulltext | **Test your patch**. Run `bin/storage upgrade` to test your patch. See Also ======== - @{class:LiskDAO} diff --git a/src/docs/user/userguide/arcanist.diviner b/src/docs/user/userguide/arcanist.diviner index 74d3246bbd..fab1de5e37 100644 --- a/src/docs/user/userguide/arcanist.diviner +++ b/src/docs/user/userguide/arcanist.diviner @@ -1,172 +1,172 @@ @title Arcanist User Guide @group userguide Guide to Arcanist, a command-line interface to Phorge. Arcanist provides command-line access to many Phorge tools (like Differential, Files, and Paste), integrates with static analysis ("lint") and unit tests, and manages common workflows like getting changes into Differential for review. A detailed command reference is available by running `arc help`. This document provides an overview of common workflows and installation. Arcanist has technical, contributor-focused documentation here: <https://we.phorge.it/book/arcanist/> = Quick Start = A quick start guide is available at @{article:Arcanist Quick Start}. It provides a much more compact summary of how to get `arc` set up and running for a new project. You may want to start there, and return here if you need more information. = Overview = Arcanist is a wrapper script that sits on top of other tools (e.g., Differential, linters, unit test frameworks, git, Mercurial, and SVN) and provides a simple command-line API to manage code review and some related revision control operations. For a detailed list of all available commands, run: $ arc help For detailed information about a specific command, run: $ arc help <command> Arcanist allows you to do things like: - get detailed help about available commands with `arc help` - send your code to Differential for review with `arc diff` (for detailed instructions, see @{article:Arcanist User Guide: arc diff}) - show pending revision information with `arc list` - find likely reviewers for a change with `arc cover` - apply changes in a revision to the working copy with `arc patch` - download a patch from Differential with `arc export` - update Git commit messages after review with `arc amend` - commit SVN changes with `arc commit` - push Git and Mercurial changes with `arc land` - view enhanced information about Git branches with `arc branch` Once you've configured lint and unit test integration, you can also: - check your code for syntax and style errors with `arc lint` (see @{article:Arcanist User Guide: Lint}) - run unit tests that cover your changes with `arc unit` Arcanist integrates with other tools: - upload and download files with `arc upload` and `arc download` - create and view pastes with `arc paste` Arcanist has some advanced features as well, you can: - execute Conduit method calls with `arc call-conduit` - create or update libphutil libraries with `arc liberate` - activate tab completion with `arc shell-complete` - ...or extend Arcanist and add new commands. Except where otherwise noted, these workflows are generally agnostic to the underlying version control system and will work properly in git, Mercurial, or SVN repositories. = Installing Arcanist = Arcanist is meant to be installed on your local machine or development server -- whatever machine you're editing code on. It runs on: - Linux; - Other operating systems which are pretty similar to Linux, or which Linux is pretty similar to; - FreeBSD, a fine operating system held in great esteem by many; - Mac OS X (see @{article:Arcanist User Guide: Mac OS X}); and - Windows (see @{article:Arcanist User Guide: Windows}). Arcanist is written in PHP, so you need to install the PHP CLI first if you don't already have it. Arcanist should run on PHP 5.2 and newer. If you don't -have PHP installed, you can download it from <http://www.php.net/>. +have PHP installed, you can download it from <https://www.php.net/>. To install Arcanist, pick an install directory and clone the code from GitHub: some_install_path/ $ git clone https://github.com/phorgeit/arcanist.git Now add `some_install_path/arcanist/bin/` to your PATH environment variable. When you type "arc", you should see something like this: Usage Exception: No command provided. Try 'arc help'. If you get that far, you've done things correctly. If you get an error or have trouble getting this far, see these detailed guides: - On Windows: @{article:Arcanist User Guide: Windows} - On Mac OS X: @{article:Arcanist User Guide: Mac OS X} You can later upgrade Arcanist to the latest version with `arc upgrade`: $ arc upgrade == Installing Arcanist for a Team == Arcanist changes quickly, so it can be something of a headache to get it installed and keep people up to date. Here are some approaches you might be able to use: - Facebook does most development on development servers, which have a standard environment and NFS mounts. Arcanist lives on an NFS mount, and the default `.bashrc` adds them to the PATH. Updating the mount source updates everyone's versions, and new employees have a working `arc` when they first log in. - Another common approach is to write an install script as an action into existing build scripts, so users can run `make install-arc` or `ant install-arc` or similar. == Installing Tab Completion == If you use `bash`, you can set up tab completion by running this command: $ arc shell-complete This will install shell completion into your current shell. After installing, you may need to start a new shell (or open a new terminal window) to pick up the updated configuration. == Configuration == Some Arcanist commands can be configured. This configuration is read from three sources, in order: # A project can specify configuration in an `.arcconfig` file. This file is JSON, and can be updated using `arc set-config --local` or by editing it manually. # User configuration is read from `~/.arcconfig`. This file is JSON, and can be updated using `arc set-config`. # Host configuration is read from `/etc/arcconfig` (on Windows, the path is `C:\ProgramData\Phorge\Arcanist\config`). Arcanist uses the first definition it encounters as the runtime setting. Existing settings can be printed with `arc get-config`. Use `arc help set-config` and `arc help get-config` for more information about reading and writing configuration. == Next Steps == Continue by: - setting up a new project for use with `arc`, with @{article:Arcanist User Guide: Configuring a New Project}; or - learning how to use `arc` to send changes for review with @{article:Arcanist User Guide: arc diff}. Advanced topics are also available. These are detailed guides to configuring technical features of `arc` that refine its behavior. You do not need to read them to get it working. - @{article:Arcanist User Guide: Commit Ranges} - @{article:Arcanist User Guide: Lint} - @{article:Arcanist User Guide: Customizing Existing Linters} - @{article:Arcanist User Guide: Customizing Lint, Unit Tests and Workflows} - @{article:Arcanist User Guide: Code Coverage}