docs: Update email configuration documentation.

Fixes #2958.

docs/prod-email.md

@@ -0,0 +1,99 @@
+# Outgoing email
+
+This page documents everything you need to know about setting up
+outgoing email in a Zulip production environment.  It's pretty simple
+if you already have an outgoing SMTP provider; just start reading from
+[the configuration section](#configuration).
+
+### Free outgoing SMTP
+
+If you don't have an existing outgoing SMTP provider, don't worry!
+There are several SMTP providers with free tiers, such as
+[Mailgun](https://documentation.mailgun.com/quickstart-sending.html#send-via-smtp)
+or
+[Amazon SES](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp.html)
+(free for sending email from EC2), and dozens of products have free
+tiers as well.  Search the web for "Transactional email" and you'll
+find plenty of options to choose from.  Once you've signed up, you'll
+want to find your "SMTP credentials" (which can be different from the
+credentials for the custom APIs for many email service providers
+have).
+
+Using a transactional email service that is designed to send email
+from servers is much easier than setting up outgoing email with an
+inbox product like Gmail.  If you for whatever reason attempt to use a
+Gmail account to send outgoing email, you will need to read this
+Google support answer and configure that account as
+["less secure"](https://support.google.com/accounts/answer/6010255);
+Gmail doesn't allow servers to send outgoing email by default.  Note
+also that the rate limits for Gmail are also quite low (e.g. 100 /
+day), so it's easy to get rate-limited.
+
+### Logging outgoing email to a file for prototyping
+
+If for prototyping, you don't want to bother setting up an email
+provider, you can add to `/etc/zulip/settings.py` the following:
+
+```
+EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
+EMAIL_FILE_PATH = '/var/log/zulip/emails
+```
+
+Outgoing emails that Zulip would have sent will just be written to
+files in `/var/log/zulip/emails/`.  This is enough to get you through
+initial user registration without an SMTP provider.
+
+Remember to delete this configuration and restart the server if you
+later setup a real SMTP provider!
+
+### Configuration
+
+To configure outgoing SMTP, you will need to complete the following steps:
+
+1. Fill out the outgoing email sending configuration block in
+`/etc/zulip/settings.py`, including `EMAIL_HOST`, `EMAIL_HOST_USER`,
+`DEFAULT_FROM_EMAIL`, and `NOREPLY_EMAIL_ADDRESS`.  You may also need
+to set `EMAIL_PORT` if your provider doesn't use the standard
+SMTP submission port (587).
+
+2. Put the SMTP password for `EMAIL_HOST_USER` in
+`/etc/zulip/zulip-secrets.conf` as `email_password = yourPassword`.
+
+#### Testing and troubleshooting
+
+You can quickly test your outgoing email configuration using:
+
+```
+su zulip
+/home/zulip/deployments/current/manage.py send_test_email username@example.com
+```
+
+If it doesn't throw an error, it probably worked; you can confirm by
+checking your email.
+
+It's important to test, because outgoing email often doesn't work the
+first time.  Common causes of failures are:
+
+* Your hosting provider blocking outgoing SMTP traffic in its
+default firewall rules.  Check whether `EMAIL_PORT` is blocked in your
+hosting provider's firewall.
+* Forgetting to put the password in `/etc/zulip/zulip-secrets.conf`.
+* Typos in transcribing the username or password.
+
+Once you have it working from the management command, remember to
+restart your Zulip server using
+`/home/zulip/deployments/current/restart-server` so that the running
+server is using the latest configuration.
+
+#### Advanced troubleshooting
+
+Zulip's email sending configuration is based on the standard Django
+[SMTP backend](https://docs.djangoproject.com/en/1.10/topics/email/#smtp-backend)
+configuration.  The one thing we've changed from the defaults is
+reading `EMAIL_HOST_PASSWORD` from the `email_password` entry in the
+Zulip secrets file, so that secrets don't live in the
+`/etc/zulip/settings.py` file.
+
+So if you're having trouble getting your email provider working, you
+may want to search for documentation related to using your email
+provider with Django.

docs/prod-install.md

@@ -81,10 +81,11 @@ These settings include:
 - `EMAIL_*`, `DEFAULT_FROM_EMAIL`, and `NOREPLY_EMAIL_ADDRESS`:
   credentials for an outgoing SMTP server so Zulip can send emails
   when needed (don't forget to set `email_password` in the
-  `zulip-secrets.conf` file!).  We highly recommend testing your
+  `zulip-secrets.conf` file!).  We highly recommend reading our
-  configuration using `su zulip` and then
+  [production email docs](prod-email.html) and following the test
-  `/home/zulip/deployments/current/manage.py send_test_email` to
+  procedure discussed there to make sure you've setup outgoing email
-  confirm your outgoing email configuration is working correctly.
+  correctly, since outgoing email is the most common configuration
+  problem.
 
 - `AUTHENTICATION_BACKENDS`: a list of enabled authentication
   mechanisms.  You'll need to enable at least one authentication
@@ -116,14 +117,10 @@ in your Zulip installation.
 
 ## Step 5: Create a Zulip organization and login
 
-* If you haven't already, verify that your server can send email using
+* If you haven't already, verify that your
-
+  [outgoing email configuration works](prod-email.html#testing-and-troubleshooting).
-  ```
+  The organization creation process will fail if outgoing email is not
-  su zulip
+  configured properly.
-  /home/zulip/deployments/current/manage.py send_test_email user@example.com
-  ```
-
-  You'll need working outgoing email to complete the setup process.
 
 * Run the organization (realm) creation [management
 command](prod-maintain-secure-upgrade.html#management-commands) :

docs/prod-requirements.md

@@ -36,9 +36,9 @@ and an authentication mechanism.  Or, you can check out the
 [email-mirror-code]: https://github.com/zulip/zulip/blob/master/zerver/management/commands/email_mirror.py
 
 * Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so
-  that Zulip can properly manage inline image previews.  If you want
+  that Zulip can properly manage inline image previews.  You'll also
-  to be able to send email from Zulip, you'll also need outgoing SMTP
+  need outgoing SMTP access to your SMTP server (the standard port for
-  access to your mail server (using port 587).
+  this is 587) so that Zulip can send email.
 
 #### Operating System
 
@@ -62,10 +62,12 @@ need to update the domains A record to point to your production server.
 
 #### Outgoing email
 
-* Email credentials Zulip can use to send outgoing emails to users
+* Outgoing email (SMTP) credentials that Zulip can use to send
-  (e.g. email address confirmation emails during the signup process,
+  outgoing emails to users (e.g. email address confirmation emails
-  missed message notifications, password reminders if you're not using
+  during the signup process, missed message notifications, password
-  SSO, etc.).
+  reset, etc.).  If you don't have an existing outgoing SMTP solution,
+  read about
+  [free outgoing SMTP options and options for prototyping](prod-email.html#free-outgoing-smtp).
 
 Once you have met these requirements, see [full instructions for installing
 Zulip in production](prod-install.html).

zproject/prod_settings_template.py

@@ -70,8 +70,9 @@ DEFAULT_FROM_EMAIL = "Zulip <zulip@example.com>"
 # Messages sent to this address should not be delivered anywhere.
 NOREPLY_EMAIL_ADDRESS = "noreply@example.com"
 
-### AUTHENTICATION SETTINGS
 
+### AUTHENTICATION SETTINGS
+#
 # Enable at least one of the following authentication backends.
 # See http://zulip.readthedocs.io/en/latest/prod-authentication-methods.html
 # for documentation on our authentication backends.