Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.
cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
Outgoing email documentation should mention that the SMTP server needs
to allow emails originating from ZULIP_ADMINISTRATOR as well as the
noreply email addresses; previously, one would typically find this out
using the test tool, which was not ideal.
Substantially tweaked by tabbott for organization and content.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend. These are:
* GoogleMobileOauth2Backend, the ancient code path for mobile
authentication last used by the EOL original Zulip Android app.
* The `finish_google_oauth2` code path in zerver/views/auth.py, which
was the webapp (and modern mobile app) Google authentication code
path.
This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
We had an organization with engineers for whom English was not their
native language think they needed to go through this laborious
process, which was definitely counterproductive for them.
Many Linux distros like Debian Buster, Fedora 29, etc. have a policy
where any services installed are disabled by default. So we should let
the developer know that they should explicitly enable and start the
Docker daemon.
Also, we should be verbose on how to do vagrant up using Docker as a
provider specifically (and similar for Virtualbox).
This was rewritten by tabbott to use markdown steps, avoid
distractions about things you don't need to change, and remove the
screenshots, which I felt were more clutter than value.
We had several patches to spectrum, but the only essential one
(0ea770fc18) had already been fixed upstream,
and another was just handling jQuery deprecation warnings for not yet removed features.
See #12749 for details.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It's not actually a reasonable experience to use Zulip without working
full-text search, so we should more strongly direct folks to not do
it.
Fixes#12724.
Moving bootstrap-typeahead from bundles/commons.js to bundles/app.js
and csrf.js from bundles/app.js to bundles/commons.js makes
bundles/commons.js equivalent to the "common" bundle, so we can
replace the latter with the former.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser. Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.
This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.
This reverts commit b546391f0b (#1148).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This was only used in Ubuntu 14.04 Trusty.
Removing this also finally lets us simplify our security model
discussion of uploaded files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We only use it in this one place, so a comment right here seems the
most discoverable place to put it. If we started using it more...
probably the section in docs/documentation/overview.md about the
dev/sysadmin docs system should split off into a new file, and this
info would become a subsection there.
This provides a better entrypoint for developers to learn about
internationalization in Zulip without cluttering the article for
translators.
I also took the opportunity to add a proper for-developers
introduction, including a link to the very nice EdX guide on the
topic.
The `AUTH_LDAP_ALWAYS_UPDATE_USER` is `True` by default, and this would sync the
attributes defined in the `AUTH_LDAP_USER_ATTR_MAP` to the user profile. But,
the default code in `django-auth-ldap` would work correctly only for `full_name`
field. This commit disables the setting by default, in favour of using the
`sync_ldap_user_data` script as a cron job.
Move docs/tutorials/documenting-api-endpoint.md to
docs/documentation/api.md.
This makes it easier to find when browsing the complete set of
materials on writing Zulip documentation.
This also means the default dev environment is now based on Ubuntu
18.04 (bionic), part of our overall effort to migrate off Ubuntu
Trusty.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This guide was originally written as part of the process of adding
i18n support to Zulip, so it had way too much focus on development
tooling details not relevant to translators.
There's probably a separate follow-up project we should do to move the
developers-only content to a separate article, since it's kinda lost
here for that audience.
Thanks to David Wood for asking a question that inspired me to do
this.
We now use a Proxy to wrap zjquery elements, so
that we can detect callers trying to invoke methods
(or access attributes) that do not exist. We try
to give useful error messages in those cases.
The main impact here is that we force lots of tests
to explicitly stub `length`.
Also, we can't do equality checks on zjquery
objects any more due to the proxy object, but the
easy workaround is to compare selectors. (This
is generally an unnecessary technique, anyway.)
The proxy wrapper is fairly straightforward, and
we just have a few special cases for things like
"inspect" that happen when you try to print out
objects.
The numbers are in parens to avoid the markdown processor interpreting them
as numbers -- I couldn't get the automatic numbering to work out with the
paragraph breaks and so on, and this is probably good enough for now.
This commit renames the `create_stream_permission` field in the
templates to `create_stream_policy`, matching the field used in the
database model. This matches what `invite_to_stream_policy` does and
will be clearer when the `waiting_period_threshold` is split into its
own field.
We need to disable "CREATE EXTENSION pgroonga" in zulip-puppet-apply
by creating /usr/share/postgresql/10/pgroonga_setup.sql.applied.
Because PostgreSQL 10 isn't running in this case. If PostgreSQL 10
isn't running, we can't run "CREATE EXTENSION pgroonga".
We can't use pg_upgrade with PGroonga. PGroonga's install SQL
https://github.com/pgroonga/pgroonga/blob/master/data/pgroonga.sql has
conditions to support multiple PostgreSQL versions. So it's not safe
to use pg_upgrade. pg_upgrade copies metadata for PostgreSQL 9.5 to
PostgreSQL 10. We need to use pg_dump and pg_restore to upgrade
PGroonga correctly for PostgreSQL 10.
"file:/srv/zulip/docs/_build/html/git/troubleshooting.html":266.5-266.12: error: Element "commit" not allowed as child of element "em" in this context. (Suppressing further errors from this subtree.)
"file:/srv/zulip/docs/_build/html/git/troubleshooting.html":266.13-266.17: error: End tag "em" violates nesting rules.
"file:/srv/zulip/docs/_build/html/overview/architecture-overview.html":211.69-211.136: error: Start tag "a" seen but an element of the same type was already open.
"file:/srv/zulip/docs/_build/html/overview/architecture-overview.html":211.171-211.174: error: Stray end tag "a".
"file:/srv/zulip/docs/_build/html/translating/hindi.html":252.36-252.48: info warning: Text run is not in Unicode Normalization Form C.
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":469.34-469.41: error: Element "method" not allowed as child of element "p" in this context. (Suppressing further errors from this subtree.)
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":471.122-471.125: error: End tag "p" seen, but there were open elements.
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":469.34-469.41: error: Unclosed element "method".
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It's not required on Ubuntu Xenial (having been replaced by systemd)
and causes problems when installing/upgrading other packages; this
change matches a similar block of code in our installer.
The comment explains this in more detail, but basically one previously
needed the `--from-git` option to `upgrade-zulip-stage-2` if one had
last installed/upgraded from Git, and not that option otherwise, which
would have forced us to make the OS upgrade documentation much more
complicated than it needed to be.
One longstanding gap in our production documentation is how to
properly do an upgrade to the operating system on which Zulip is
installed.
This adds that documentation.
Ideally, we'd get a few folks to test this procedure over the next few
days to make sure it's bulletproof.
Fixes#1705.
Fixes#10796.
There are two 'etc' words in the same line. This removes one of them.
Note from Tim: The old text was arguably correct, but removing it is definitely clearer.
Add references to TypeScript in documentation where appropriate, such
as in example bash commands and discussions of the file structure.
Add a new section to the Reading List with TypeScript resources.
Also update `.editorconfig` to support ".ts" files.
Fix part of #12000.
Two variables were declared and assigned the respective values of the
default settings for the system. If the keyword is used in the
~/.zulip-vagrant-config file, the value is assigned to the variable.
There is no straightforward way to customize the virtual machine's
number of cpus or memory, this commit addresses that fact.
This adds numerous manual testing methods and checks to the
`manual-testing.md` to verify the functioning of various hotkeys,
click handlers and appearance of the message box.
Sphinx/ReadTheDocs supports automatically translating links written as
to `.md` files to point to the corresponding `.html` files, so this
migration does not change the resulting HTML output in ReadTheDocs.
But it does fix apparent broken links on GitHub.
This doesn't prevent people from reading the documentation on GitHub
(so doesn't mitigate the fact that some rtd-specific syntax does not
render properly on GH), but it will prevent us from getting erroneous
issues reported about the hyperlinks not working.
Fixes: #11087.
The docs specify passing hostname with the --hostname flag, which
doesn't match the actual usage in scripts/setup/setup-certbot. This
change fixes the docs to match the actual usage.
This renames references to user avatars, bot avatars, or organization
icons to profile pictures. The string in the UI are updated,
in addition to the help files, comments, and documentation. Actual
variable/function names, changelog entries, routes, and s3 buckets are
left as-is in order to avoid introducing bugs.
Fixes#11824.
Add the recommendation in docs for not using GitHub Desktop client for
zulip code as it has a bug of keeping the symlinks disabled and ignoring
any configuration done to re-enable it resulting into failure to identify
the received symlinks leading to failing test cases in the zulip development
environment.
This documentation had a tendency to bitrot, and in any case now that
we have tooling for doing Fedora (etc.) from provision, it's likely
the case that adding other Linux/UNIX distros we care about to
provision would not be difficult and is a better path than maintaining
this manually-curated duplicate of `tools/provision`.
Further, even if this documentation was maintained, one would still
end up wanting to run `provision` after rebasing a branch, so it was
never particularly practical for extended development.
The history of this was that there was a period where half of Vagrant
releases were broken (for everyone, e.g. downloading a base container
didn't work). It seems Vagrant has cleaned up their act at this point.
We really just want 2.0.x, not a specific version, and the direct
links we had are now quite old.
Thanks to Jonathon Hinchley for reporting this.
Fixes#11836.
Visually, #zoom_help_text acts like
.organization-settings-parent div:first-of-type when the Zoom option
is selected, but isn't treated as such.
No visual change with the #google_hangouts_domain change; just there to make
the code more readable/defensible.
help.github.com seems to have a bug where HEAD on a redirected page
returns 404. This causes tools/test-documentation to fail. Fix it by
skipping the redirects.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This optimizes test-backend by skipping webhook
tests when run in default mode.
Tweaked by tabbott to extend the documentation and update the CI
commands.
Eventually, we'll just make this the default, but for now, we should
fix the typos and use our standing approach for documenting running
management commands.
This adds reference for reloading nginx when the certificates are
replaced so that the server works with the new certificates instead of
the old ones.
Fixes: #4849.
This endpoint serves requests which might originate from an image
preview link which had an http url and the message holding the image
link was rendered before we introduced thumbnailing. In that case
we would have used a camo proxy to proxy http content over https and
avoid mix content warnings.
In near future, we plan to drop use of camo and just rely on thumbor
to serve such images. This endpoint helps maintain backward
compatibility for links which were already rendered.
This setting splits away part of responsibility from THUMBOR_URL.
Now on, this setting will be responsible for controlling whether
we thumbnail images or not by asking bugdown to render image links
to hit our /thumbnail endpoint. This is irrespective of what
THUMBOR_URL is set to though ideally THUMBOR_URL should be set
to point to a running thumbor instance.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.
To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
Apparently, when we renamed these files to no longer have a .txt
extension, we accidentally removed them from the set of strings for
translation, because `manage.py makemessages` by default only
processes .txt and .html files under the templates/ directory.
Fix this by adding a .txt extension.
Bootstrap's typeahead is the main part of the project that we've
forked, and moving it to its own module should help unlock our ability
to upgrade bootstrap itself.
To support this, we add a pass_targets option to the main linter
library, because with current mypy, it's generally counterproductive
to pass the list of files in (can produce spurious errors; isn't
faster).
The testing section is more appropriate, since it's fundamentally part
of our CI system.
While we're at it, fix the fact that we were linking to GitHub, not
ReadTheDocs, in the run-mypy output.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.
This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.
Fixes#286.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.
Fixes#7423.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.
Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
notifications settings are requested.
Fixes#8051.
A bunch of details were out of date about the current state of Zulip
and thus what projects make sense; while this isn't likely to be our
final 2019 ideas list, this should help avoid confusion about what's
happening.
A key part of this is the new helper, get_user_by_delivery_email. Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.
Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
This is required in some AWS regions.
The right long-term fix is to move to boto3 which doesn't have this
problem.
Allows us to downgrade the priority of #9376.
This commit allows specifying Subject Alternative Names to issue certs
for multiple domains using certbot. The first name passed to certbot-auto
becomes the common name for the certificate; common name and the other
names are then added to the SAN field. All of these arguments are now
positional. Also read the following for the certbot syntax reference:
https://community.letsencrypt.org/t/how-to-specify-subject-name-on-san/Fixes#10674.
Older versions of virtual box were giving installation error in new
MacOS Mojave. While originally we considered changing the docs to
point to the current version 5.2.20, it seems better to just not pin a
version.
Significantly tweaked by tabbott.
This is a preparator refactor for supporting hosting different Tornado
processes on different servers; to look up which Tornado server we
should be sending the event to, we'll need the realm object.
On OSX, the user id and group id don't match. So while the previous
code was always wrong, it produced incorrect output there. We can fix
this by replacing `whoami` with `id -g` for finding the current user's
group ID.
Tweaked by tabbott to move most of the content into the remote
development documentation, both for brevity in the main docs as well
as clarity.
Fixes#10694.
We drop support for usage of `icon-vector` as base class when
including icons from font awesome icons package.
Now on, only icons as specified in font awesome v4.7.0 can be used
in the code base.
Surprisingly hard to find a good base example of a user doc. This one should
at least give the basic pointer to sidebar_index.md and where the new file
goes.
In particular, this improves:
* The explanation of how data is mapped into Zulip
* The explanation of what is printed out by `manage.py query_ldap`
* Makes sure users create their first account with EmailAuthBackend.
The term "username" confusingly refers both to the Django concept of
"username" (meaning "the name the user types into the login form") and
a concept the admin presumably already has in their existing
environment; which may or may not be the same thing, and in fact this
is where we document the admin's choice of whether and how they should
correspond. The Django concept in particular isn't obvious, and is
counterintuitive when it means something like an email address.
Explicitly explain the Django "username" concept, under the name of
"Zulip username" to take responsibility for our choice of how it's
exposed in the settings interface. Then use an explicit qualifier,
like "LDAP username", whenever referring to some other notion of
username. And make a pass over this whole side of the instructions,
in particular for consistent handling of these concepts.
Expand on a few things that tend to confuse people (especially the
`%(user)s` thing); move the `LDAPSearchUnion` example out to docs;
adjust the instructions to fit a bit better in their new docs/ home.
This makes it easier to iterate on these, and to expand supplemental
information (like troubleshooting, or unusual configurations) without
further straining the already-dauntingly-long settings.py.
It also makes it easier to consult the instructions while editing the
secrets file, or testing things, etc. -- most admins will find it more
natural to keep a browser open somewhere than a second terminal.
Fixes part of #10297.
Use FAKE_LDAP_NUM_USERS which specifies the number of LDAP users
instead of FAKE_LDAP_EXTRA_USERS which specified the number of
extra users.
There are several situations in which we want to create a Customer and
stripe.Customer object before we really have a billing relationship with a
customer. The main one is giving non-profit or educational discounts.
Due to copyright issues with potentially displaying Apple emojisets on
non-apple devices, as well as iamcal dropping support for the emojione
emojiset (see https://github.com/iamcal/emoji-data/pull/142), we are
dropping (perhaps temporarily) support for allowing users to switch
emojisets in Zulip.
This commit just hides the feature from the user but leaves most of
the infrastructure in place so that in the future if we decide to
re-enable the support we will not need to redo the infrastructure work
(some JS-side code is deleted, mostly because we'll want to re-add the
feature using the do_settings_change infrastructure anyway).
The most likely emoji set to add is the legacy "blobs" Google emoji
set, since it seems popular with some users.
Tweaked by tabbott to remove some additional JS code and update the
changelog.
Now that we have nice documentation for our export/import tools, we've
been seeing a lot of users trying to use that as their primary backup
process. Let's correct this.
This flag is used to track which user/message pairs correspond to an
active mobile push notification, that should potentially be cleared
when the user reads the message.
This flag should never appear on a message that is also marked as
read; eventually we may want a cron job to check for that condition.
We include a partial index on UserMessage for this flag.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.
Fixes part of #10042.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.
This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.
For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.
History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org. A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
As part of our effort to change the data model away from each user
having a single API key, we're eliminating the couple requests that
were made from Django to Tornado (as part of a /register or home
request) where we used the user's API key grabbed from the database
for authentication.
Instead, we use the (already existing) internal_notify_view
authentication mechanism, which uses the SHARED_SECRET setting for
security, for these requests, and just fetch the user object using
get_user_profile_by_id directly.
Tweaked by Yago to include the new /api/v1/events/internal endpoint in
the exempt_patterns list in test_helpers, since it's an endpoint we call
through Tornado. Also added a couple missing return type annotations.
It appears Luke's Dropbox folder/shared link that used to host some
tsearch_extras binaries was removed. It wasn't very high-value
regardless, because most of the platforms involved are deprecated, the
ones that don't generally have a PPA, and building from source is
pretty easy. So, we just remove these options from the documentation.
While we're at it, make clear we only support direct installation on
Ubuntu LTS.
Fixes#9863.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.
If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in. The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!
We fix this by moving python-social-auth to a separate subdomain.
Fixes: #9847.
As detailed in the documentation changes, this simplifies the
development workflow for doing UI work on the /stats pages.
The cost is a ~10% increase the time it takes to run `populate_db`,
which doesn't happen very often (and for most purposes manifests as a
1% increase in the time it takes to rebuild the database from scratch).
This saves about 400ms when running clean-unused-caches, basically by
calling its sub-rountines by import (rather than
`subprocess.check_call()`). The performance optimization seems well worth it.
Fixes#9766.
I don't think this is exactly the right place to document this, but
I'm not sure there's a better one without some restructuring this page
in general (which would probably have value).
Fixes#8769.
I only renamed references that I thought were absolutely necessary
and only if the resulting sentence structure wasn't awkward.
If the renaming resulted in awkward structure, I replaced the term
"webhook" with "integration" (but only in some very obvious cases).
Fixes#9500.
David Rosa
Anders Kaseorg
Hemanth V. Alluri
Mateusz Mandera
Wyatt Hoodes
Tim Abbott
Tej Pochiraju
Harshit Bansal
Rishi Gupta
Mohit Gupta
Amal Rajan
Tyler B. Thrailkill
Vinit Singh
YashRE42
Vishnu Ks
Thomas Ip
Aman Agrawal
Puneeth Chaganti
Greg Price
Kanishk Kakar
Rafid Aslam
Steve Howell
David Wood
Mayank Madan
Kouhei Sutou
Shubham Padia
ruchit2801
Marco Burstein
Mohit Maroliya
Vaibhav
Sahil Batra
Samuel Searles-Bryant
Zach Wener-Fligner
Ben Muschol
AsociTon
Dominik Grybos
Priyanshu Singh
Paurakh Sharma Humagain
Anders Kaseorg
Rohitt Vashishtha
Raymond Akornor
Abhigyan Khaund
xiaomingplus
synicalsyntax
Adarsh Patel
rht
Aastha
Aditya Bansal
seresheim
Sumanth V Rao
Marco Burstein
Realmagnum
Joshua Pan
Igor Posledov
Bruce
Cynthia Lin
Tim Abbott
Pranjal Vyas
Scott Lee
Kevin Lin
Jack Zhang
Roman Godov
Lyla Fischer
Shubham Dhama
Eeshan Garg
Yashashvi Dave
Sampriti Panda
Yago González
Abhigyan Khaund
Umair Khan