Commit Graph

649 Commits

Author SHA1 Message Date
Mateusz Mandera b66dc9de50 saml: Support IdP-initiated SSO. 2020-05-25 16:09:30 -07:00
Alex Vandiver 031260573f docs: Link to section on migrating local -> S3 storage.
This section at the top was clearly written before the documentation
at the bottom existed, and hasn't been updated to point to the
now-existent docs below.

Add the link, rather than directing to #production-help.
2020-05-19 14:08:44 -07:00
Steve Howell 23c2198da3 docs: Use string_id in header. 2020-05-17 21:25:53 -07:00
Steve Howell 35df5f35d8 docs: Keep manual-restore steps sequential. 2020-05-17 21:25:53 -07:00
Tim Abbott 1c1c47b94a docs: Delete discussion of legacy push notifications signup.
This hasn't been used in months; it was only there in case of problems
with the rollout of the automated workflow, so it makes sense to
delete it now.
2020-05-05 15:59:35 -07:00
Vishnu Ks 9a8d0ca9fe docs: Recommend contact page over email wherever possible.
With a few tweaks from tabbott to preserve the enthusiasm for feedback
and de-emphasize twitter as a channel (we give better support
elsewhere).
2020-05-05 15:57:54 -07:00
Tim Abbott 9566905b81 docs: Clarify nginx reverse proxy documentation.
Fixes #14740.
2020-04-26 10:51:39 -07:00
Tim Abbott 1219a2e854 docs: Deprecate support for Xenial and Stretch.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.

Our reasoning for deprecating support for these releases is as follows:

* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
  EOL on the server in about a year.

* Debian Stretch will each EOL in 2020 (the precise date is unclear in
  Debian's documentation, but based on past precedent it's in the next
  few months, perhaps July 2020).
  https://wiki.debian.org/DebianReleases#Production_Releases

* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
  Python, which will reach EOL in September 2020 (and we're already
  seeing various third-party dependencies that we use drop support for
  them).

* While there is LTS support for these older releases, it's not clear it's
  going to be worth the added engineering effort for us to maintain EOL
  releases of the base OSes that we support.

* We (now) have clear upgrade instructions for moving to Debian Buster
  and Ubuntu 18.04.
2020-04-16 15:36:18 -07:00
Tim Abbott 724fcc74e8 help: Improve documentation on importing exported organizations.
This should help avoid confusion where some users used the Zulip
backup tool, not the import tool, to try to import their backups.
2020-03-30 13:25:03 -07:00
thoemie 7a48bec189 docs: Document EMAIL_USE_SSL for smtp providers on port 465.
I found the solution by simply trying out EMAIL_USE_SSL and it
worked. I had problems with sending emails (did not work at all, there
wasn't even a connection going on - I checked with tcpdump. Then I
found this: To use port 465, you need to call
smtplib.SMTP_SSL(). Currently, it looks like Django only uses
smtplib.SMTP() (source: https://code.djangoproject.com/ticket/9575).

Fixes #14350.
2020-03-27 00:25:49 -07:00
Anders Kaseorg 7ff9b22500 docs: Convert many http URLs to https.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:35:32 -07:00
Mateusz Mandera f9db77c400 docs: Updated links to python-social-auth docs.
The URL seems to have changed.
2020-03-20 08:54:37 -04:00
Stefan Weil c220b971ae
docs: Fix some typos in documentation (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-17 05:57:10 -07:00
Rike-Benjamin Schuppner bb49a60bd5
docs: Fix type in authentication methods. 2020-03-15 17:55:08 -07:00
Tim Abbott 6c74af4c06 docs: Document our stable release branches.
The experiment with the published 2.1.x tpye branches has been going
well, so we should document the feature.
2020-02-29 11:50:44 -08:00
Tim Abbott b775becc68 docs: Change our security contact to security@.
This has for a while been our only active Google Groups mailing list,
and given that folks will guess security@ as our security contact, we
might as well just standardize on that.

Also tweak some ambiguous text; it wouldn't be appropriate for us to
issue a CVE for e.g. an operational issue only affecting us.
2020-02-26 16:35:29 -08:00
Mateusz Mandera 7814f52d45 docs: Replace links to Django 1.11 docs with 2.2 links. 2020-02-19 11:51:18 -08:00
Tim Abbott 4a36ed6cb2 docs: Advertise support for GitLab authentication. 2020-02-11 14:13:39 -08:00
xpac1985 65fe1a9eed
docs: Add info about zulip-announce RSS feed to install docs.
The mailing list can also be subscribed to via RSS/Atom feeds, I just wanted to make that information easier accessible.
2020-01-31 17:24:43 -08:00
Tim Abbott d356622594 docs: Add link from LDAP docs to invitation docs.
This addresses confusion we had with some organizations where they
were surprised that with only LDAP enabled, the "invite more users"
feature was available.

Fixes #11685.
2020-01-25 23:41:19 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Mateusz Mandera e477cae800 docs: Fix missing apostrophe in EMAIL_HOST_USER value. 2020-01-03 16:52:31 -08:00
Mateusz Mandera dc59850d15 docs: Fix incorrect path to get-django-setting script. 2020-01-03 16:52:31 -08:00
Mateusz Mandera d88494deae docs: Add some troubleshooting notes for ldap. 2020-01-03 16:52:30 -08:00
Mateusz Mandera bfb963b9aa docs: Include suggested USERNAME_ATTR in example AD ldap configs. 2020-01-03 16:46:07 -08:00
Tim Abbott e38c58e7c7 docs: Rewrite LDAP discussion of AUTH_LDAP_REVERSE_EMAIL_SEARCH.
This moves the mandatory configuration for options A/B/C into a single
bulleted list for each option, rather than split across two steps; I
think the result is significantly more readable.

It also fixes a bug where we suggested setting
AUTH_LDAP_REVERSE_EMAIL_SEARCH = AUTH_LDAP_USER_SEARCH in some cases,
whereas in fact it will never work because the parameters are
`%(email)s`, not `%(user)s`.

Also, now that one needs to set AUTH_LDAP_REVERSE_EMAIL_SEARCH, it
seems worth adding values for that to the Active Directory
instructions.  Thanks to @alfonsrv for the suggestion.
2019-12-13 13:55:52 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Tim Abbott 35959d43c4 docs: Clean up troubleshooting guide.
This article is definitely still below our polish goals, but this is
also definitely an improvement.
2019-12-12 22:19:12 -08:00
Mateusz Mandera 8bd2a130a9 docs: Fix some typos. 2019-12-12 17:19:10 -08:00
Tim Abbott 171c6f119d docs: Clean up upgrade/modify documentation. 2019-12-12 17:02:07 -08:00
Tim Abbott 305adc4f64 docs: Clean up requirements page. 2019-12-12 16:31:02 -08:00
Tim Abbott 080864ca44 docs: Minor edits to export and management command docs. 2019-12-12 16:06:40 -08:00
Tim Abbott ea60670c9f docs: Clean up some editing issues in export docs. 2019-12-12 15:56:23 -08:00
Vishnu KS 6c97a36355 install: Support remote database services like RDS.
Documentation and variable names edited by tabbott.
2019-12-12 12:59:45 -08:00
Tim Abbott 23e3ae1211 docs: Reorganize some SAML setup instructions.
This reads and line-wraps better.
2019-12-11 16:32:43 -08:00
Mateusz Mandera ed513f1f59 docs: Explain SAML ACS url with SOCIAL_AUTH_SUBDOMAIN. 2019-12-11 16:29:46 -08:00
Tim Abbott f5eece9b4f docs: Fix broken social authentication links. 2019-12-10 17:34:27 -08:00
Mateusz Mandera b3085f924d docs: List the backends in section explaining SOCIAL_AUTH_SUBDOMAIN.
Tweaked by tabbott to remove the reference to `python-social-auth`;
the key detail is whether a callback URL is involved.
2019-12-10 17:16:20 -08:00
Tim Abbott c6fe6cf0a4 docs: Fix recommendations to copy secrets during data import.
The previous documentation was essentially wrong, in that it
recommended copying certain settings that would cause significant
problems post-import if they were indeed copied.
2019-12-09 17:57:37 -08:00
Tim Abbott 6ca56f81f2 docs: Document RabbitMQ configuration issue when importing. 2019-12-09 17:22:04 -08:00
Tim Abbott 79604c7817 docs: Update import docs to mention RAM requirements. 2019-12-09 17:22:04 -08:00
Tim Abbott b036fa897e docs: Move unattended-upgrades docs to troubleshooting guide.
This also rewrites the text to better explain what's happening.  It's
likely further polish would be valuable, but that's true for the whole
"Troubleshooting" page.

This block of text was misplaced when we split the long
maintain-secure-update; article; we want it to be easy to find by
folks who are looking into error emails Zulip is sending.
2019-12-02 11:37:16 -08:00
David Rosa 62d555115a docs: Improve formatting of important notes.
Improves formatting using reST and Sphinx admonition directives
so that important notes stand out when viewed in ReadTheDocs.
2019-12-02 10:37:49 -08:00
Tim Abbott c9a3e4e872 docs: Reorganize and refocus section on management commands. 2019-11-22 10:48:37 -08:00
Tim Abbott 5a7b5f1337 docs: Rework text for scalability and monitoring sections.
This text is very old and hadn't been edited in a long time, in large
part because it was buried within old docs.  This change cleans it up
to give accurate and better-organized information.
2019-11-22 10:22:07 -08:00
David Rosa 87a2831b83 docs: Split maintain-secure-upgrade into dedicated docs.
* Moves "Management commands" to a top-level section.
* Moves "Scalability" as a subsection at the bottom of "Requirements".
* Moves "Monitoring" as a subsections at the bottom of "Troubleshooting".
* Replaces "API and your Zulip URL" with a link to REST API docs.  This
  documentation text has been irrelevant for some time.
* Removes maintain-secure-upgrade from the TOC but the file remains to
  avoid breaking old links from release blog posts and emails.
2019-11-22 10:21:20 -08:00
Tim Abbott 6d17fea076 docs: Document a few management commands around email. 2019-11-18 15:45:21 -08:00
rht 517534777d export-and-import: Update repo url of Zulip Archive. 2019-11-15 11:39:52 -08:00
David Rosa b041948132 docs: Reorganize auth and migrations subsystems.
- Moves "Authentication in the development environment" from subsystems
to "development/authentication.md".
- Moves "Renumbering migrations" to a section within "Schema migrations".
2019-11-07 09:42:36 -08:00
David Rosa af4d7b4b52 docs: Merge "Upgrading" with modifying-zulip.
Merges the "Upgrades" section from production/maintain-secure-upgrade.md
with production/modifying-zulip.md.

Contains significant textual changes by tabbott to read more clearly.
2019-11-05 17:43:00 -08:00
Mateusz Mandera a62d084247 social_backends: Rename display_logo to display_icon. 2019-11-03 15:54:05 -08:00
Anders Kaseorg 94c8fffdf3 docs: Fix invalid link.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-30 18:41:40 -07:00
David Rosa dc5dbcbee3 docs: Merge "Backups" with export-and-import.
- Merges the "Backups" section from production/maintain-secure-upgrade.md
  with existing "Backups" section in production/export-and-import.md.
- Cleans up and makes content more clear/explicit.
- Adds short missing section on how to use wal-e configuration.
- Removes a lot of previously duplicate text explaining the difference between
  the tools.
- Various textual tweaks by tabbott.

Fixes #13184 and resolves #293.
2019-10-30 17:22:02 -07:00
Tim Abbott c7c6f01236 docs: Advertise SAML authentication as an official feature.
Fixes #13275.
2019-10-29 16:55:22 -07:00
Mateusz Mandera 4561652513 ldap docs: LDAP backend now works for initial realm creation.
Fixes #9576.

Initial realm creation now works fine with the LDAP backend, so the
part of the docs about needing to create the first realm with the
email backend is unnecessary and just complicates the setup process,
so it seems best to just remove it.
2019-10-29 16:21:58 -07:00
Tim Abbott 7a66dfa133 auth: Tweak docs now that SAML supports multiple IdPs. 2019-10-28 15:22:29 -07:00
Tim Abbott 03af3d2eb6 docs: Fix old-style markdown link in authentication-methods.
This was introduced recently due to a rebase of an older branch.
2019-10-23 13:04:30 -07:00
Mateusz Mandera fea4d0b2be ldap: Do a proper search for email in email_belongs_to_ldap.
This fixes a collection of bugs surrounding LDAP configurations A and
C (i.e. LDAP_APPEND_DOMAIN=None) with EmailAuthBackend also enabled.

The core problem was that our desired security model in that setting
of requiring LDAP authentication for accounts managed by LDAP was not
implementable without a way to

Now admins can configure an LDAPSearch query that will find if there
are users in LDAP that have the email address and
email_belongs_to_ldap() will take advantage of that - no longer
returning True in response to all requests and thus blocking email
backend authentication.

In the documentation, we describe this as mandatory configuration for
users (and likely will make it so soon in the code) because the
failure modes for this not being configured are confusing.

But making that change is pending work to improve the relevant error
messages.

Fixes #11715.
2019-10-22 15:53:39 -07:00
Mateusz Mandera 4dc3ed36c3 auth: Add initial SAML authentication support.
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.

This PR is a collaboration with Tim Abbott.

Fixes #716.
2019-10-10 15:44:34 -07:00
David Rosa bdbc384de5 docs: Reduce the number of apparently broken links on github.
- Updated 260+ links from ".html" to ".md" to reduce the number of issues
reported about hyperlinks not working when viewing docs on Github.
- Removed temporary workaround that suppressed all warnings reported
by sphinx build for every link ending in ".html".

Details:
The recent upgrade to recommonmark==0.5.0 supports auto-converting
".md" links to ".html" so that the resulting HTML output is correct.

Notice that links pointing to a heading i.e. "../filename.html#heading",
were not updated because recommonmark does not auto-convert them.
These links do not generate build warnings and do not cause any issues.
However, there are about ~100 such links that might still get misreported
as broken links.  This will be a follow-up issue.

Background:
docs: pip upgrade recommonmark and CommonMark #13013
docs: Allow .md links between doc pages #11719

Fixes #11087.
2019-10-07 12:08:27 -07:00
Shikhar Varshney fe5d975b33 settings: Add support for overriding APNS_TOPIC and ZULIP_IOS_APP_ID.
Documentation added by tabbott.
2019-10-05 22:53:59 -07:00
Tim Abbott 7e0ea61b00 upload: Support S3-compatible S3 hosting providers.
Previously, we were hardcoding the domain s3.amazonaws.com.  Given
that we already have an interface for configuring the host in
/etc/zulip/boto.cfg (which in turn, automatically configures boto), we
just need to actually use the value configured in boto for what S3
hostname to use.

We don't have tests for this new use case, in part because they're
likely annoying to write with `moto` and there hasn't been a huge
amount of demand for it.  Since this doesn't regress existing S3
backend support, it seems worth merging.
2019-09-24 17:17:21 -07:00
Tim Abbott 3c0f54e242 docs: Update documented string_id for zulipinternal realm.
See 9b2b24c617 for the update of the
string_id.
2019-09-23 11:51:10 -07:00
Tim Abbott 0815a9bd53 docs: Add a link to changelog from section on deploying master. 2019-09-18 11:18:12 -07:00
Mateusz Mandera a1fa0b011e docs: Fix typo in production/email.md. 2019-09-10 16:10:12 -07:00
Thomas Ashish Cherian ea17673885 docs: Include su in email docs restart-server command.
It's likely the user was root as this point in the instructions.
2019-09-10 16:00:34 -07:00
Tim Abbott 94251a9341 docs: Add a full article on modifying Zulip.
We have lots of documentation for Zulip developers; but previously
didn't have a nice top-level page for Zulip server administrators to
learn how to manage patches to Zulip.
2019-09-06 13:52:56 -07:00
David Rosa 4afdfa6055 docs: Place warning at the top of every page in /latest/production.
- Added warning block to dev docs using _templates/layout.html
- Removed copy-pasted warning from install.md and requirements.md
- Removed unreleased tag in docs/conf.py that's no longer used

Useful ref: https://www.sphinx-doc.org/en/master/templating.html

Fixes #13056.
2019-08-28 15:31:47 -07:00
Greg Price c6fb8c048a docs: Add troubleshooting entry for TLS <1.2 servers.
This corresponds to a user report at zulip/zulip-mobile#3596 .
2019-08-27 16:25:37 -07:00
Anders Kaseorg 51b2af9e5c docs: Shorten stackoverflow link.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:09:30 -07:00
Greg Price 3c07c054b6 docs: Add troubleshooting entry for TLS issue on Android 7.0.
Learned this thanks to a user report and some debugging:
  https://chat.zulip.org/#narrow/stream/48-mobile/near/782174
2019-08-26 13:03:00 -07:00
Tim Abbott e6340c6e54 docs: Make production unreleased documentation notices more visible.
Previously, Google might take you to /latest rather than /stable, and
you might get information about the next release, not the current one.

Fixes #13056.
2019-08-23 16:33:46 -07:00
Tim Abbott 6041c277a9 docs: Fix single-service deployment docs.
Apparently, the ordering matters.

This command now matches what we use for installing single-purpose
installations; I should have checked this directly.
2019-08-21 21:17:30 -07:00
Tim Abbott 92d8c5dec6 docs: Recommend zulip::base when deploying subsets of Zulip.
One needs to explicitly request zulip::base, it appears.  Ideally,
we'd just have every ruleset depend on zulip::base, but I seem to
recall Puppet didn't like your including the same module from multiple
places.  Worth testing as a follow-up investigation.

This was missed because we use the zulip_ops rules internally, which
include zulip::base via zulip_ops::base.
2019-08-21 10:34:34 -07:00
Tim Abbott ee63f68140 docs: Add links from SSL docs to our desktop app SSL guide.
This should help people setting up new Zulip servers discover how to
setup SSL properly for the desktop apps.
2019-08-19 14:21:56 -07:00
Hemanth V. Alluri dac068df31 production: Finish adding production support for Zulip on Debian Buster.
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.

We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.

Fixes #9828.
2019-08-17 12:22:32 -07:00
Mateusz Mandera 9b2b24c617 production: Rename the 'zulip' system bot realm to 'zulipinternal'. 2019-07-31 16:01:36 -07:00
Tim Abbott 934d7c2f0f docs: Fix development version warning.
This was sadly broken by the comment Greg added to document how it
works :(.
2019-07-26 17:29:21 -07:00
Tej Pochiraju cb10a3231c docs: Discuss outgoing email senders in email.md.
Outgoing email documentation should mention that the SMTP server needs
to allow emails originating from ZULIP_ADMINISTRATOR as well as the
noreply email addresses; previously, one would typically find this out
using the test tool, which was not ideal.

Substantially tweaked by tabbott for organization and content.
2019-07-23 11:25:57 -07:00
Harshit Bansal bf14a0af4d auth: Migrate google auth to python-social-auth.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend.  These are:

* GoogleMobileOauth2Backend, the ancient code path for mobile
  authentication last used by the EOL original Zulip Android app.

* The `finish_google_oauth2` code path in zerver/views/auth.py, which
  was the webapp (and modern mobile app) Google authentication code
  path.

This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
2019-07-21 20:51:34 -07:00
Rishi Gupta 6b350f092d docs: Make the intro to Production Installation less scary. 2019-07-20 14:39:15 -07:00
Tim Abbott bb746ab93b docs: Discourage direct mobile push notifications more strongly.
We had an organization with engineers for whom English was not their
native language think they needed to go through this laborious
process, which was definitely counterproductive for them.
2019-07-18 13:11:21 -07:00
Mateusz Mandera 7f556d1678 docs: Fix typo in production/email-gateway.md. 2019-07-13 11:31:01 -07:00
Tim Abbott 1d3312eede docs: Improve email gateway documentation based on feedback. 2019-07-11 14:01:29 -07:00
Mateusz Mandera 6f778ffe53 docs: Fix typos in email-gateway.md. 2019-07-10 17:58:38 -07:00
Mateusz Mandera 3a4f8ed737 docs: Move prose email mirror instructions out of settings template.
This should make it easier to understand the documentation.
2019-07-10 13:33:25 -07:00
Tim Abbott 361618f525 docs: Clarify documentation on Amazon RDS.
It's not actually a reasonable experience to use Zulip without working
full-text search, so we should more strongly direct folks to not do
it.

Fixes #12724.
2019-07-08 20:03:20 -07:00
Tim Abbott e14a40675b docs: Document changing EXTERNAL_HOST after backup restoration.
Fixes #12704.
2019-07-03 21:08:58 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
Anders Kaseorg 4e1060b29e docs: Remove references to Ubuntu 14.04 Trusty as a supported platform.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 15:09:13 -07:00
Anders Kaseorg 33c941407b puppet: Remove legacy unauthenticated local uploads backend.
This was only used in Ubuntu 14.04 Trusty.

Removing this also finally lets us simplify our security model
discussion of uploaded files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:31:46 -07:00
Tim Abbott 88b77af54f puppet: Add support for changing the nginx port directly.
This provides a clean process for changing Zulip's nginx port.
2019-06-17 12:24:22 -07:00
Greg Price b6ce366d61 docs: Add a comment explaining the fancy conditional-include syntax.
We only use it in this one place, so a comment right here seems the
most discoverable place to put it.  If we started using it more...
probably the section in docs/documentation/overview.md about the
dev/sysadmin docs system should split off into a new file, and this
info would become a subsection there.
2019-06-15 09:35:04 -07:00
Tim Abbott 6236dfab9a docs: Clarify section on local configuration changes.
We still should add a section documenting all the options.
2019-06-03 20:11:07 -07:00
Puneeth Chaganti 717d1e504d ldap: Turn off the AUTH_LDAP_ALWAYS_UPDATE_USER setting.
The `AUTH_LDAP_ALWAYS_UPDATE_USER` is `True` by default, and this would sync the
attributes defined in the `AUTH_LDAP_USER_ATTR_MAP` to the user profile. But,
the default code in `django-auth-ldap` would work correctly only for `full_name`
field. This commit disables the setting by default, in favour of using the
`sync_ldap_user_data` script as a cron job.
2019-06-02 11:24:19 -07:00
vinitS101 9ddc27f318 docs: Fixed a small typo in the LDAP documentation. 2019-06-01 16:54:32 -07:00
Puneeth Chaganti a1590c613e url preview: Enable server level setting for url embed previews.
This significantly simplifies the process for enabling this feature in
new organizations.
2019-05-31 15:37:03 -07:00
Tim Abbott 19666aacbf docs: Update link to FCM. 2019-05-29 15:52:11 -07:00
Rishi Gupta 529d424bed docs: Add numbering to import section of export-and-import.md.
The numbers are in parens to avoid the markdown processor interpreting them
as numbers -- I couldn't get the automatic numbering to work out with the
paragraph breaks and so on, and this is probably good enough for now.
2019-05-06 22:16:17 -07:00
Kouhei Sutou a046a0957b docs: Fix upgrade process for Ubuntu 16.04 -> 18.04.
We need to disable "CREATE EXTENSION pgroonga" in zulip-puppet-apply
by creating /usr/share/postgresql/10/pgroonga_setup.sql.applied.
Because PostgreSQL 10 isn't running in this case. If PostgreSQL 10
isn't running, we can't run "CREATE EXTENSION pgroonga".

We can't use pg_upgrade with PGroonga. PGroonga's install SQL
https://github.com/pgroonga/pgroonga/blob/master/data/pgroonga.sql has
conditions to support multiple PostgreSQL versions. So it's not safe
to use pg_upgrade. pg_upgrade copies metadata for PostgreSQL 9.5 to
PostgreSQL 10. We need to use pg_dump and pg_restore to upgrade
PGroonga correctly for PostgreSQL 10.
2019-05-03 17:35:32 -07:00
Kouhei Sutou 407ed45ec2 docs: Use systemctl instead of service.
Because memcached package on Ubuntu 18.04 supports systemd.
2019-05-03 17:33:29 -07:00
Tim Abbott 9ece4c9f51 docs: Add documentation for the delete and reimport process.
This is a common thing folks ask how to do.
2019-04-28 21:49:13 -07:00
Rishi Gupta 14428f7996 docs: Clarify you need to upgrade from git for Zulip Cloud exports. 2019-04-26 15:59:24 -07:00
Tim Abbott bf52ddc33a docs: Make sure that postgresql is running in OS upgrade instructions.
At least in the Trusty->Xenial upgrade flow, it can end up not running.
2019-04-17 12:58:42 -07:00
Tim Abbott e566cad1da docs: Add OS upgrade step to remove upstart.
It's not required on Ubuntu Xenial (having been replaced by systemd)
and causes problems when installing/upgrading other packages; this
change matches a similar block of code in our installer.
2019-04-17 12:58:42 -07:00
Tim Abbott 18146ed947 script: Add static asset pipeline option to support OS upgrades.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
2019-04-17 12:58:42 -07:00
Tim Abbott 6828cea76a docs: Clarify some key parts of OS upgrade documentation.
Thanks to Vishnu Ks for testing to figure these out.
2019-04-17 12:58:42 -07:00
Shubham Padia 31cedbc9b8 backup: Make `--output` an optional argument instead of a positional one.
Fixes #12150.
Previously the `output` argument was positional, while the docs described
the arguments as optional in the form of `--output=...`.
2019-04-17 12:14:11 -07:00
Shubham Padia 32cf192b9b backup: Add option `--skip-uploads` to skip uploads backup. 2019-04-17 12:14:11 -07:00
Shubham Padia 984d3f2f0e backup: Add option `--skip-db` to skip database backup.
This may be relevant for organizations that use a corporate remote
postgres server and just want to backup their configuration.
2019-04-17 12:14:11 -07:00
Tim Abbott 5d2984fb46 docs: Document upgrading the version of Ubuntu.
One longstanding gap in our production documentation is how to
properly do an upgrade to the operating system on which Zulip is
installed.

This adds that documentation.

Ideally, we'd get a few folks to test this procedure over the next few
days to make sure it's bulletproof.

Fixes #1705.
Fixes #10796.
2019-04-14 21:59:40 -07:00
Mohit Maroliya e16d721921 docs: Fix a few typos in documentation. 2019-04-10 10:44:35 -07:00
Tim Abbott abc7a00d35 Revert "docs: Update .html links to .md."
This doesn't work without the CommonMark upgrade.

This reverts commit c87893feea.
2019-04-05 17:58:54 -07:00
Samuel Searles-Bryant c87893feea docs: Update .html links to .md.
Sphinx/ReadTheDocs supports automatically translating links written as
to `.md` files to point to the corresponding `.html` files, so this
migration does not change the resulting HTML output in ReadTheDocs.
But it does fix apparent broken links on GitHub.

This doesn't prevent people from reading the documentation on GitHub
(so doesn't mitigate the fact that some rtd-specific syntax does not
render properly on GH), but it will prevent us from getting erroneous
issues reported about the hyperlinks not working.

Fixes: #11087.
2019-04-05 17:16:25 -07:00
ruchit2801 33b064aa29 documentation: Add missing link to upgrade-zulip-from-git. 2019-03-22 09:22:18 -07:00
Tim Abbott 0d04724da6 docs: Clarify push notification service privacy model.
This just makes what was already said there a lot more explicit.
2019-03-21 16:49:10 -07:00
Tim Abbott 0a17a2acaa docs: Recommend using an SSD for the Zulip database.
This is sorta common knowledge for folks who have managed databases,
but not everyone has.
2019-03-10 13:51:52 -07:00
Tim Abbott 0e2dc873ba docs: Explain options for preventing changes during export.
This makes it a bit clearer that one doesn't need to deactivate a
realm just to export data.
2019-03-04 11:38:02 -08:00
Tim Abbott 066ca0353d docs: Fix export/import manage.py instructions typos.
Fixes #11755.
2019-03-02 13:04:22 -08:00
Tim Abbott 216d2ec1bf production: Add optional support for submitting usage statistics.
See documentation for details.
2019-02-26 17:35:10 -08:00
Anders Kaseorg ce01a4c5a3 backup: Add Zulip, OS, and PostgreSQL version to the tarball.
==> zulip-backup/os-version <==
    Ubuntu trusty

    ==> zulip-backup/postgres-version <==
    90324

    ==> zulip-backup/zulip-version <==
    2.0.0-rc1+git
    2.0.0-rc1-50-gbf169b8-dirty

Documentation added by tabbott.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-14 15:11:57 -08:00
Vishnu Ks 949d098e99 management: Add tooling for transfering local uploads to s3.
This solves a common migration problem for folks who cut corners when
first setting up Zulip.

Fixes #11294.
2019-02-14 14:31:46 -08:00
Anders Kaseorg e0a51948d9 script: Add ready-to-run tooling for doing backups.
Based on an initial version by Tim Abbott (#11204).

Fixes #552.
2019-02-11 17:30:37 -08:00
Tim Abbott e3bc788660 docs: Announce deprecation of Ubuntu Trusty. 2019-02-11 16:10:12 -08:00
xiaomingplus 5120d97633 docs: Remove confusing comment from boto example.
The boto config file format should have comments on a separate line.

here is the docs:

http://boto.cloudhackers.com/en/latest/boto_config_tut.html#details
2019-02-11 15:01:18 -08:00
Tim Abbott a46640b325 docs: Document restricting access to an LDAP group. 2019-02-11 11:40:57 -08:00
Rishi Gupta 7adc7ed490 docs: Update mobile-push-notifications to discourage legacy signup. 2019-02-06 11:53:39 -08:00
Tim Abbott e91f22c416 docs: Reorganize and simplifiy LDAP data synchronization docs. 2019-02-05 10:37:04 -08:00
Harshit Bansal 2abe325147 docs: Add documentation for the new custom profile data sync feature.
Fixes: #10976.
2019-02-05 10:25:50 -08:00
Tim Abbott 001ec76e1f export: Document /etc/zulip when discussing server migrations. 2019-02-02 23:22:22 -08:00
Tim Abbott df436a55b4 scripts: Recommend apt update after enabling universe.
One needs to manually do an apt update after add-apt-repository, or it
won't actually work.
2019-01-26 12:29:05 -08:00
Tim Abbott 7824ce32a0 docs: Advertise HipChat/Slack import tools in installer docs.
This should help avoid confusion about when to use the import tools.
2019-01-16 15:15:01 -08:00
Tim Abbott f9b60b4803 docs: Update some notes about Tornado scalability. 2019-01-16 11:26:57 -08:00
Anders Kaseorg 25d6107a4f docs: Avoid hardcoded /tmp paths in miscellaneous documentation.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:16:12 -08:00
Harshit Bansal 71761bc2da ldap: Add a setting to automatically deactivate non_matching users.
Fixes: #11151.
2019-01-13 19:04:09 -08:00
Tim Abbott 40884dbb1d docs: Document active directory values for AUTH_LDAP_USER_SEARCH.
Fixes #10019.
2019-01-08 16:28:51 -08:00
Tim Abbott 534bf8f806 docs: Make more clear how installing from Git works.
The previous discussion made it sound like this did a development
environment installation.
2019-01-07 19:16:59 -08:00
Aastha a65cb91cc1 docs: Add nginx reload documentation.
This adds reference for reloading nginx when the certificates are
replaced so that the server works with the new certificates instead of
the old ones.

Fixes: #4849.
2019-01-04 14:30:26 -08:00
Tim Abbott 5d550145bd docs: Document need for zulip group for some nagios checks. 2019-01-02 10:22:48 -08:00
Tim Abbott af1c7a9573 docs: Improve documentation for adding a new authentication backend.
Now that we have the Azure Active Directory integration cleaned up and
merged, we can nicely link to it as an example.
2018-12-18 16:49:01 -08:00
seresheim 49dbd85a89 auth: Add support for Azure Active Directory authentication.
This takes advantage of all of our work on making the
python-social-auth integration reusable for other authentication
backends.
2018-12-18 16:39:03 -08:00
Tim Abbott d815e5a299 docs: Document how to sync additional fields in LDAP.
This probably isn't something folks will use a lot, but it is a
question we get.

Fixes #9710.
2018-12-13 16:24:15 -08:00
Tim Abbott 626e191201 ldap: Add support for automatic user deactivation/reactivation.
As part of this, extend our documentation on synchronizing data from
Active Directory.
2018-12-13 16:24:15 -08:00
Tim Abbott 5dd646f33f ldap: Add support for syncing avatar images from LDAP.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.

This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.

Fixes #286.
2018-12-13 13:39:22 -08:00
Tim Abbott e92e982342 docs: Add a troubleshooting note about cookies on root domain. 2018-12-12 12:08:25 -08:00
Joshua Pan 441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott a4a25c3e14 docs: Expand security model documentation to cover guests.
This is a pretty minor update, just clarifying the discussion of what
types of users exist.
2018-12-05 17:25:12 -08:00
Anders Kaseorg 9ba860b4f2 Avoid double redirects to /login and then to /login/.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-04 16:12:58 -08:00
Tim Abbott 250a2b5164 docs: Fix documentation details for boto configuration.
This fixes a couple key gaps in our documentation of the #9376
workaround.
2018-12-04 13:28:13 -08:00
Tim Abbott e8742f1dd5 docs: Expand documentation for running management commands. 2018-12-03 09:16:05 -08:00
Tim Abbott 1303f8df20 docs: Fix missing quotes in `su zulip -c` documentation.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
2018-11-30 12:12:27 -08:00
Tim Abbott 72db5d5a48 docs: Document how to setup system postfix email with Zulip. 2018-11-30 12:06:25 -08:00
Tim Abbott df5475996f docs: Add prod auth docs link to dev auth docs.
This should be helpful for folks working on things like a Facebook
OAuth2 integration.
2018-11-29 20:45:02 -08:00
Tim Abbott 0932a27647 docs: Move generic reverse proxy notes further down. 2018-11-28 15:53:54 -08:00
Igor Posledov 7aed0a2a66 docs: Add nginx reverse proxy basic config example. 2018-11-28 15:52:06 -08:00
Bruce 604364eb65 docs: Document how to use Zulip behind an haproxy reverse proxy.
With significant rearrangement by tabbott to have more common text
between different proxy implementations.
2018-11-28 13:56:26 -08:00
Tim Abbott f1f60bc9bb docs: Document how to use AWS SIGv4 with boto.
This is required in some AWS regions.

The right long-term fix is to move to boto3 which doesn't have this
problem.

Allows us to downgrade the priority of #9376.
2018-11-28 12:25:45 -08:00
Tim Abbott 51950ad352 docs: Fix accidental repeat bullet #1 in S3 backend documentation.
Due to missing indentation, the numbering was resetting to 1 rather
than continuing to 6.
2018-11-28 12:19:37 -08:00
Tim Abbott 5f3c8be6d1 docs: Fix some broken links in security model doc.
Apparently, we haven't been running test-documentation in production
of late.
2018-11-28 09:49:39 -08:00
Tim Abbott f1c65975ad docs: Clarify preparatory process for data import.
You need a Zulip server running the a matching version, and no longer
need to do an upgrade from master before using established import tools.
2018-11-14 17:02:33 -08:00
Tim Abbott a2371a96b7 docs: Clarify push registration for running manage.py correctly.
We've had several users get errors running this because they ran it as
a bash script; fix this my making the command super explicit.
2018-11-14 13:07:04 -08:00
Tim Abbott 8ab26df9f0 docs: Link to setup-certbot multiple hostname support. 2018-11-13 12:55:24 -08:00
Rohitt Vashishtha 95ba947f13 setup-cerbot: Allow issuing certificates for multiple domains.
This commit allows specifying Subject Alternative Names to issue certs
for multiple domains using certbot. The first name passed to certbot-auto
becomes the common name for the certificate; common name and the other
names are then added to the SAN field. All of these arguments are now
positional. Also read the following for the certbot syntax reference:

https://community.letsencrypt.org/t/how-to-specify-subject-name-on-san/

Fixes #10674.
2018-11-13 12:47:31 -08:00
Tim Abbott 2f36211fde docs: Document need to enable Universe repository on Ubuntu. 2018-10-31 11:53:24 -07:00
Rishi Gupta 39c56b3ba9 docs: Update security-model.md.
Tweaked by tabbott to fix a few typos and purge use of "realm".
2018-10-19 15:40:36 -07:00
Rishi Gupta f14ed03707 docs: Update import instructions in export-and-import. 2018-10-18 16:22:31 -07:00
Tim Abbott 50be5ceae3 docs: Document our now-official production Docker image. 2018-10-16 15:27:03 -07:00
Tim Abbott 3039f11dd4 docs: Document installing Zulip from GitHub.
This has been unofficially supported for a long time, and is handy for
common use cases, so worth documenting.
2018-10-16 15:19:51 -07:00
Tim Abbott 05b29ea47c docs: Remove production overview page.
This page didn't really add any value over the "Installation" page
after the refactorings we've done in recent time.
2018-10-16 15:10:28 -07:00
Greg Price cbb6692882 prod docs: Say exactly how to test the SSL cert chain.
Users continue to run into this and get confused, certain that there's
no certificate problem.
2018-10-03 16:03:05 -07:00
Rishi Gupta 0d1907fb71 user docs: Add test-mobile-notifications. 2018-09-28 12:09:33 -07:00
Tim Abbott 5064d9dd8a import: Remove documentation of --import-into-nonempty.
This is now the default behavior.
2018-09-27 14:14:51 -07:00
Tim Abbott 41b18c7067 docs: Clean up LDAP docs a bit more.
In particular, this improves:
* The explanation of how data is mapped into Zulip
* The explanation of what is printed out by `manage.py query_ldap`
* Makes sure users create their first account with EmailAuthBackend.
2018-09-27 13:38:14 -07:00
Tim Abbott a943e55863 docs: Move LDAP documentation out of plug-and-play section.
Also update the links to this section.
2018-09-27 13:38:14 -07:00
Greg Price a612049ac4 docs: Expand on LDAP config; try to clarify "username" concept.
The term "username" confusingly refers both to the Django concept of
"username" (meaning "the name the user types into the login form") and
a concept the admin presumably already has in their existing
environment; which may or may not be the same thing, and in fact this
is where we document the admin's choice of whether and how they should
correspond.  The Django concept in particular isn't obvious, and is
counterintuitive when it means something like an email address.

Explicitly explain the Django "username" concept, under the name of
"Zulip username" to take responsibility for our choice of how it's
exposed in the settings interface.  Then use an explicit qualifier,
like "LDAP username", whenever referring to some other notion of
username.  And make a pass over this whole side of the instructions,
in particular for consistent handling of these concepts.
2018-09-27 13:20:10 -07:00
Greg Price 6f23d2f691 docs: Make a pass over LDAP config/setup docs.
Expand on a few things that tend to confuse people (especially the
`%(user)s` thing); move the `LDAPSearchUnion` example out to docs;
adjust the instructions to fit a bit better in their new docs/ home.
2018-09-27 13:20:10 -07:00
Greg Price e7737c6342 docs: Move prose LDAP instructions out of settings template.
This makes it easier to iterate on these, and to expand supplemental
information (like troubleshooting, or unusual configurations) without
further straining the already-dauntingly-long settings.py.

It also makes it easier to consult the instructions while editing the
secrets file, or testing things, etc. -- most admins will find it more
natural to keep a browser open somewhere than a second terminal.
2018-09-27 13:20:10 -07:00
Greg Price 9dbb5561b1 docs: Use `sudo -s` to get a root shell without changing directory.
The `-i` is for "initial", creating a login shell -- which means
cd'ing to root's homedir.  That defeats the line after this one.
2018-09-25 13:15:06 -07:00
Rishi Gupta a04d1e3d98 docs: Update intros to production install pages.
We should cut to the chase where we can.
2018-08-21 11:32:16 -07:00
Tim Abbott c6bbb9a8d4 docs: Improve cross-linking between export/import and backups.
Now that we have nice documentation for our export/import tools, we've
been seeing a lot of users trying to use that as their primary backup
process.  Let's correct this.
2018-08-07 14:50:38 -07:00
Tim Abbott 3cbdb1efc7 docs: Document that minify-js can OOM kill during an upgrade. 2018-08-03 15:13:40 -07:00
Tim Abbott f336f58b81 docs: Expand documentation of running Zulip behind a reverse proxy. 2018-08-02 08:38:56 -07:00
Tim Abbott 35cb7528f9 models: Add new UserMessage flag active_mobile_push_notification.
This flag is used to track which user/message pairs correspond to an
active mobile push notification, that should potentially be cleared
when the user reads the message.

This flag should never appear on a message that is also marked as
read; eventually we may want a cron job to check for that condition.

We include a partial index on UserMessage for this flag.
2018-08-01 16:51:56 -07:00
Shubham Padia bf6dc4472b models: Add is_private flag to UserMessage and add index for it.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.

This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.

For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.

History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org.  A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
2018-07-30 15:43:55 -07:00
Tim Abbott b53463d6a4 docs: Remove use of multi-line 'su zulip' in docs.
It turns out this confused people into trying to run both pieces in
one line.
2018-07-30 13:38:55 -07:00
Rishi Gupta 5e7fd3a3cf docs: Move server import and export from /help to readthedocs. 2018-07-27 10:37:50 -07:00
Aditya Bansal 9b485f3ef4 auth: Fix bug with subdomains and GitHub auth causing apparent logouts.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.

If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in.  The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!

We fix this by moving python-social-auth to a separate subdomain.

Fixes: #9847.
2018-07-10 16:49:00 +05:30
Tim Abbott ddc2ef2ce8 docs: Point Docker users to the Docker upgrade docs.
The normal instructions don't make sense for the Docker image.
2018-07-01 06:58:44 -07:00
Jack Zhang 781e995865 docs: Fix typos in mobile-push-notifications. 2018-06-30 13:48:01 -07:00
Vishnu Ks 0212113569 docs: Document ADD_TOKENS_TO_NOREPLY_ADDRESS in email.md.
Rewritten and moved by tabbott.
2018-06-23 12:03:30 -07:00
Eeshan Garg 0a43e5e257 Replace all user-facing references to "invite-only" with "private".
Fixes #9611.
2018-06-12 13:37:45 -04:00
Tim Abbott 427c43723c docs: Document disabling unattended-upgrades.
I don't think this is exactly the right place to document this, but
I'm not sure there's a better one without some restructuring this page
in general (which would probably have value).

Fixes #8769.
2018-06-05 08:45:42 -07:00
Tim Abbott 40635beb23 docs: Document production support for Debian Stretch and Ubuntu Bionic.
Fixes #9197.
2018-06-05 08:09:29 -07:00
Tim Abbott b7af4c2162 docs: Update multiple organization docs to clarify root domain story.
Previously, we made it seem like it was a good idea to use a subdomain
and a root domain at the same time.
2018-05-31 12:18:41 -07:00
Sampriti Panda 018a33ff7c send_test_email: Send email from NOREPLY address as well.
Tweaked by tabbott to also set different subject lines, to make it
easier to see that one got 2 emails.
2018-05-27 22:18:44 -07:00
Abhigyan Khaund fda786952e documentation: Migrate all CZO stream links to new stream URL style
This includes changes like changing CZO stream URLs from new.20members
to 95-new-members.
2018-05-24 15:15:34 -07:00
Tim Abbott d6fb47385b docs: Clean up the instructions for installing from Git.
This is now much more common, and deserves clearer discussion.
2018-05-23 14:34:28 -07:00
Tim Abbott 9aee4c8c8e docs: Add production documentation on deployment options.
This describes the basics of installing Zulip in more complex
production environments using multiple services.

Fixes #1235.
2018-05-22 11:31:48 -07:00
Tim Abbott f8fcbbb672 docs: Update security model docs for new stream history feature. 2018-05-20 14:38:51 -07:00
Tim Abbott 0364ba8986 docs: Recommend Mailgun over Sendgrid for email setup. 2018-05-17 07:28:08 -07:00
Tim Abbott faa2cd9597 docs: Clarify setup documentation for S3 backend. 2018-05-17 07:20:07 -07:00
Tim Abbott 0ac8de2b47 docs: Document new configuration option for production nginx. 2018-05-17 07:14:18 -07:00
Tim Abbott e63541e616 docs: Extract documentation for file upload backends.
This moves the documentation for this feature out of
prod_settings_template.py, so that we can edit it more easily.

We also add a bucket policy, which is part of what one would want to
use this in production.

This addresses much, but not all, of #9361.
2018-05-16 15:00:54 -07:00
Tim Abbott 828104e020 docs: Document the REALM_HOSTS feature. 2018-05-16 07:02:13 -07:00
Tim Abbott cfd22c6832 zilencer: Clean up logic around mobile push notifications signup.
This fixes exceptions when sending PMs in development (where we were
trying to connect to the localhost push bouncer, which we weren't
authorized for, but even if we were, it wouldn't work, since there's
no APNS/GCM certs).

At the same time, we also set and order of operations that ensures one
has the opportunity to adjust the server URL before submitting
anything to us.
2018-05-05 16:42:00 -07:00
Tim Abbott 43098a6f7c zilencer: Add automated signup system for push notifications.
Based on an initial version by Rishi Gupta.

Fixes #7325.
2018-05-03 21:27:49 -07:00
Rishi Gupta cf735042b7 docs: Make minor fixes to mobile-push-notifications.html. 2018-04-23 12:37:29 -07:00
Tim Abbott f90b765824 docs: Document what's involved in doing your own push notifications.
We've had a few folks ask about this recently, and it seemed like it'd
be good to cover this a bit more explicitly.
2018-04-21 21:42:24 -07:00
Tim Abbott e53c0fe273 docs: Further clarify privacy details for push notifications. 2018-04-19 14:20:57 -07:00
Greg Price 21045d8cf0 prod docs: Call out more the need for a chained cert bundle.
This is kind of easy to gloss over, especially with the framing
as a "format"; surely if things work at all, the file format
must have been right, right?  It's really a bit more substantive
than that; say so and also add a bit more description.
2018-04-16 11:34:23 -07:00
Axel Tietje 8f984be457 docs: Fix typo in production docs. 2018-04-12 09:19:26 -07:00
Greg Price 6396b3aef7 docs/production: Make an editing pass through the SMTP doc.
In addition to many small edits for formatting and clarity, a few more
significant changes:

* In the main instructions, refer specifically to restarting the
  server and to testing that the config works.

* Add SendGrid to the recommended list, as it seems like people
  give it a somewhat stronger reputation these days than Mailgun.

* Discuss EMAIL_USE_TLS and EMAIL_PORT along with host, user, and
  password in the "free services" section.  Though those bullets feel
  kind of duplicative to me already.
2018-04-05 21:24:48 -07:00
Greg Price b9f1f9c0ae docs/production: Reorganize SMTP docs a bit.
Let's get right to the point of how to configure SMTP once you know
what you want.  That section is pretty short anyway; and we can have
a first step direct the reader to our suggestions if they don't know
what service they want to use.

Also adjust the hierarchy of the headings: group the various
alternative email services under one heading, and group
troubleshooting together under an independent heading.

Also correct what we say about EMAIL_PORT: the Django default is
apparently 25, so if the provider *does* use the usual port 587
then we'll need the port to be set.
2018-04-05 21:24:48 -07:00
YJDave 93ee0aace7 stream settings: Allow realm admins to remove others from any stream.
This will allow realm admins to remove others from private stream to
which the realm administrator is not subscribed; this is important for
managing those streams, because previously nobody could remove users
from private streams that didn't have any realm administrators
subscribed.
2018-03-13 14:59:09 -07:00
YJDave 2031118545 stream settings: Allow realm admins to access all private stream subs.
This will allow realm admins to access subscribers of unsubscribed
private stream.  This is a preparatory commit for letting realm admins
remove those users.
2018-03-13 14:59:09 -07:00
YJDave 37f9d5c193 stream settings: Allow realm admins to update any stream name & description.
This will allow realm admins to update the names and descriptions of
private streams even if they are not subscribed, which fixes the buggy
behavior that previously nobody could(!).
2018-03-13 14:59:09 -07:00
Greg Price e022b9e7ea install docs: Cut SSL cert from requirements summary.
The summary already has a qualifier that basically says it shouldn't
matter for most people -- making it simultaneously the most
complicated bullet there, and among the least likely to matter.

And in fact, this requirement shouldn't matter for *anyone* when first
experimenting with Zulip.  If certbot won't work in a given admin's
environment, and the available ways to get a cert aren't convenient,
they can always let the installer generate a self-signed cert to get
going, and circle back to the issue later.

So, make that option clear in the main requirements text, and then
just cut the whole bullet from the summary.
2018-03-06 19:43:02 -08:00
Greg Price d99ec6426d install docs: Move requirements tl;dr to requirements doc.
This further reduces the wall of text on the install instructions.
Simultaneously it lightens up the pressure on this summary to be quite
so terse; expand a couple of items into multiple bullets (yet with
fewer words!) for greater readability.
2018-03-06 19:43:02 -08:00
Greg Price c1d5364077 install docs: Consolidate follow-up steps.
Now down to just 4 steps!

This version tries to prioritize: just two items that we really want
all admins to look at even if they aren't already mentally committed
to running a big production service and reading all the docs.
Namely, the two required in order to really try out Zulip effectively
with one's colleagues.
2018-03-06 19:43:02 -08:00
Greg Price 526e301177 install docs: Tighten "create organization" step.
The screenshots weren't doing much good without being embedded in the
text... and in fact, looking at them for I think the first time,
they're badly out of date with the app.  So cut them.

We might add screenshots later, but on the other hand if we do a good
job with the forms themselves, they should be superfluous.
2018-03-06 19:43:02 -08:00
Greg Price 4edc121b49 install docs: Move background details to after main flow too.
This further shortens the wall of text inside the instructions.

Note that thanks to embedded reST, we now have the power to provide
custom anchors at section headings!  Which in particular means this
link won't break if we later tweak the wording of this heading.
2018-03-06 19:43:02 -08:00
Greg Price 53f80336ee install docs: Consolidate troubleshooting at bottom.
This helps shorten the wall of text between the start of the
instructions and the end.  Conversely, now that there are fewer
followup steps, the troubleshooting section at the end isn't so
far away to point.
2018-03-06 19:43:02 -08:00
Greg Price 0505036965 install docs: Reduce prose in requirements bullets. 2018-03-06 19:43:02 -08:00
Greg Price f22712e8e8 install: Go straight through by default.
This flips the experimental `--express` option to be the default.

We retain the old behavior, where the script exits before
`initialize-database`, as an option `--no-init-db`; it might be useful
in e.g. a migration scenario (from a Zulip install elsewhere, or
another chat system) where the admin wants to set up the database
separately.

The install instructions are adjusted to match, getting shorter by two
steps and a bunch of words.  I think this opens up opportunities to
refactor the text to simplify things further, too, but leaving that
for another commit.

Also tweak the "production" test suite to match.
2018-03-06 19:43:02 -08:00
Tim Abbott c732313c36 docs: Rename customize.html to settings.html.
This is a better description of what this is for.
2018-02-19 09:55:49 -08:00
Aditya Bansal efe8545303 local-uploads: Start running authentication checks on file requests.
From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.

NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.

Fixes: #320 and #291 partially.
2018-02-16 05:06:37 +05:30
Tim Abbott ddf83f65ce docs: Update dead references to puppet/zulip_internal.
This was renamed in the codebase to puppet/zulip_ops months ago.
2018-02-12 16:00:56 -08:00
Greg Price edfa3eabfd docs: Use `orphan` to directly exclude pages from toctree.
This is what the Sphinx docs recommend when you actually don't want
the page to be included in navigation:
  http://www.sphinx-doc.org/en/stable/markup/toctree.html
And now that we have `eval_rst`, we're able to take advantage of it!

One difference between doing this and the old way of making "hidden"
toctree entries is that with the latter, the "previous" and "next"
links at the bottom of each page would thread through the hidden
entries; which gets kind of confusing when they don't appear in the nav.
2018-01-24 14:37:17 -08:00
Greg Price 5a2b001c18 docs: Fix a bit of confusion over past vs. present.
The careful testing with dozens of realms was a discrete thing
that happened, not a general state that the tool was in;
certainly not the last state it was known to be in, as the rest
of the sentence explains how it hasn't been so carefully tested
since then.  So "as of" doesn't fit.
2018-01-24 14:37:17 -08:00
Greg Price da626c176f docs: ssl-certificates: 2017 is no longer the present.
Tweak this wording so it stays true and time-appropriate indefinitely.
2018-01-24 14:37:17 -08:00
Greg Price 6aa7a1b8d2 install docs: Document streamlined self-signed-cert flow.
This is easy now, so make it known to admins who are looking for a
fast path for a test install.

Also totally cut the painfully complicated steps for generating a
self-signed cert by hand.  Anyone who actually wants that can find a
hundred explanations on the Web, or can look at our script if they
want to specifically mirror how we do it (which is mercifully much
simpler than this.)
2018-01-23 18:08:52 -08:00
Greg Price c27f36a82a install: Rename generate-self-signed-cert.
This script generates one certificate.
2018-01-23 18:08:52 -08:00
Greg Price bd9340b4db docs: Warn on development version of install instructions. 2018-01-23 10:57:50 -08:00
Greg Price 2bf5d0074c docs: Update ETA for Let's Encrypt wildcard support.
It's now January 2018, so we can delete this caveat, right?
Not quite yet -- the original post we link to now has an
update saying 2018-02-27.  Let's make it less specific,
in case the date changes again.
2018-01-23 10:57:50 -08:00
Greg Price cdd35816fe docs: Link back to install step in SSL doc. 2018-01-23 10:57:50 -08:00
Greg Price 8fd98133d8 docs: Tweak discussion of SSL certs in prod requirements. 2018-01-23 10:57:50 -08:00
Greg Price 0cf0c51cad docs: Revise description of main install step.
Bulleted information instead of prose, huzzah!  Also I think we need
to explain the options a bit right here, or at least link to where
they're documented somewhere.  (If the list gets much longer, we'll
want to shift toward the latter.)

Also reorganize existing information a bit, and clean up a couple
of nits.
2018-01-23 10:57:50 -08:00
Greg Price bcc1456f55 docs: Tighten sub-headings in SSL certs doc.
What I really want is to give these sections nice stable slugs
to put on the anchors and use as the URL fragment, independent of
any wording tweaks on the text headings.  But I don't think we
have that feature with Markdown and our current docs infrastructure.

At least for Certbot, the brevity helps make this heading clearer
than the previous one.
2018-01-23 10:57:50 -08:00
Greg Price 93b957eaae docs: Revise explanation of manual SSL cert install.
Change a bit of prose to bullets.  Also lead with the "If you ..."
that helps readers skim past this section.
2018-01-23 10:57:50 -08:00
Tim Abbott 21d8ff14b4 docs: Update documentation to recommend certbot. 2018-01-23 10:57:50 -08:00
Greg Price e68f426286 docs/production: Revise doc on authentication methods.
Hopefully this version makes it somewhat clearer how the different
methods relate to each other, how to choose between them, what
`ZulipRemoteUserBackend` is for, and how the latter works.
2018-01-18 17:45:03 -08:00
XavierCooney 5409a98039 docs: Fix typo in multiple-organizations.md. 2017-12-31 09:11:30 -05:00
Greg Price c2cd76b69e docs: Note need to log out and in again on push notifs setup.
This often surprises people, so mention it up front.

(Also it'd probably be good to add some code to make this step
unnecessary.)
2017-11-29 13:32:02 -08:00
David Rosa Tamsen 84384196f2 docs: Exclude files from TOC warnings and navigation sidebar.
Sphinx was displaying "WARNING: document isn't included in any toctree"
for files we just don't want in the TOC. We can hide them from the index,
but the rtd theme defaults to display hidden index entries in the nav bar.

This commit excludes these files from such warnings, and patches layout.html
so that hidden index entries stay hidden from the navigation sidebar.

This commit also moves password-strength.md under docs/production and
adds it as a hidden entry in production/index.rst.

Fixes #7417.
2017-11-22 10:59:14 -08:00
Greg Price 0b543bfc0e docs: Rewrite certbot discussion, to enjoy the new automation.
This doesn't touch the main path through the install docs; that will
see a broader rewrite soon as we make outbound email optional for a
nascent server, make the hostname and admin email into mandatory
installer flags, and then radically simplify the instructions by
removing mandatory editing of `settings.py` and folding most of what's
left into the installer.
2017-11-20 15:52:07 -08:00
Balaji2198 1f43252903 docs: Update TOC in production to include SSL and email.
This adds a few missing entries to the TOC, which hadn't made sense
back when Zulip's ReadTheDocs didn't have the new collapsing feature.

Tweaked by tabbott to also give the SSL certificates doc an
appropriate title for its new role.
2017-11-20 10:21:02 -08:00
Greg Price 011114397b docs: Fix some Markdown links missing a close-paren.
These were just introduced today.
2017-11-16 11:43:19 -08:00
David Rosa Tamsen 7072fa5b37 docs: Reorganize developer docs to improve navigation.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.

This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.

Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).

Fixes #5265.
2017-11-16 09:45:08 -08:00