Commit Graph

34255 Commits

Author SHA1 Message Date
Steve Howell 613b8ade20 topic list: Add is_active_topic to template.
The only place we ever set active-sub-filter is
right after we build the template, so there is
no reason to have it be a separate step.

(I made a similar fix to pm_list recently, and
this helps set the stage for doing vdom-like
stuff.)
2020-01-22 14:28:50 -08:00
Tim Abbott e459a4b7a7 topic_list: Rewrite unzoomed rendering to be readable.
The previous logic was a bit byzantine, making a lot of inferences
based on which conditionals had already been processed that made it
hard to read.  This simple function approach promises to be more
readable.
2020-01-22 14:28:50 -08:00
Tim Abbott 37b563b82f topic list: Don't include muted topics in "more topics" count.
This is for consistency with how we show unreads in muted topics at
the stream level, avoiding distracting users with the appearance of
unread messages in muted topics that they've made clear they are not
interested in.

Arguably, we should show a faded count if there are unreads on muted
topics (but none on unmuted topics), but that seems somewhat complex
to maintain, and we'd benefit from user feedback to make an effective
decision on whether it'd be an improvement.

Fixes #13676.
2020-01-22 14:28:50 -08:00
Tim Abbott 6f4f6fde79 topic list: Skip muted topics when not zoomed.
I think this probably matches users' expected behavior that muted
streams shouldn't get in their way unless the user is actively looking
for them.  If a user has a lot of muted topics with active traffic
(e.g. because topics corresponding to channels in a mirrored Slack
instance), they would previously find their 5 slots cluttered with
those muted topics even if there were unmuted topics with unread
messages.

Fixes #13677.
2020-01-22 14:28:50 -08:00
Rodriq 8d466f6a25 docs: Update API docs usage example.
This docs on writing API docs usage example hadn't been updated to use
generate_code_example(curl) after we introduced that feature.
2020-01-22 12:00:10 -08:00
Mateusz Mandera 8dd95bd057 tests: Replace httpretty with responses.
responses is an module analogous to httpretty for mocking external
URLs, with a very similar interface (potentially cleaner in that it
makes use of context managers).

The most important (in the moment) problem with httpretty is that it
breaks the ability to use redis in parts of code where httpretty is
enabled.  From more research, the module in general has tendency to
have various troublesome bugs with breaking URLs that it shouldn't be
affecting, caused by it working at the socket interface layer.  While
those issues could be fixed, responses seems to be less buggy (based
on both third-party reports like ckan/ckan#4755 and our own experience
in removing workarounds for bugs in httpretty) and is more actively
maintained.
2020-01-22 11:56:15 -08:00
Tim Abbott 445a11753b docs: Update links to Django documentation.
Django 1.9 is no longer on the web.
2020-01-16 16:00:03 -08:00
Tim Abbott c113d74daf docs: Rewrite the guide on using the development environment.
This correct various inaccuracies and adds a bulleted list structure
for better clarity.

I think there's a lot more that could be done here in the form of
linking to other pages, discussing restarting `run-dev.py`, etc.
2020-01-16 15:57:08 -08:00
Chris Bobbe b3901c830b docs/development/using: Add link to ./authentication.
Added a link from docs/development/using ("Using the Development
Environment") to ./authentication ("Authentication in the development
environment") to help people working on the authentication systems
or anyone who needs an API key.
2020-01-16 15:41:12 -08:00
Chris Bobbe 6c3fcc252e docs/development/using: Small wording change.
An attempt to make the point about Django/Tornado server processes
auto-restarting less wordy.
2020-01-16 15:41:12 -08:00
Chris Bobbe 60f3c06d91 docs/development/using: Break into three headings: Server/Web/Mobile.
Separate using.html into Server/Web/Mobile sections so that readers
will find what they're looking for more quickly. Server is at the top
because it contains information relevant to web and mobile developers,
e.g., that the `run-dev.py` console output will provide useful errors.

Fixes #13655.
2020-01-16 15:41:10 -08:00
Jonathan Cobb c7433c83ff integrations: Add errbit integration.
Fixes #13685.
2020-01-16 15:33:51 -08:00
Tim Abbott 46f2aec674 register: Fix whitespace.
This would have been included in my previous set of commits fixing
/register, but was delayed because of
the templating bug fixed in d0492b103b.
2020-01-16 15:28:47 -08:00
Mateusz Mandera d37e6ef921 email_mirror: Use plaintext if html body empty with prefer-html option.
If an email is sent with the .prefer-html option, but it has no html
body, it's better to fall back to plaintext content instead of treating
it as a user error.
2020-01-16 15:25:27 -08:00
Mateusz Mandera 0c9c218e91 email_mirror: Add prefer-html and prefer-text address options.
Closes #13484.

These options tell zulip whether to prefer the plaintext or html version
of the email message. prefer-text is the default behavior, so including
the option doesn't change anything as of now, but we're adding it to
prepare to potentially change the default behavior in the future.
2020-01-16 15:25:19 -08:00
Mateusz Mandera 170e0ac2dd email_mirror: More abstract option system.
As we add more address options, which will have different behavior than
simply setting option_name=True, we need to migrate this subsystem to
something that better supports more complex logic and will allow
encapsulating it, instead of needing to be put all over the
decode_email_address function.
2020-01-16 15:16:04 -08:00
Tim Abbott e2681372f3 js: Clean up poorly named const id variables.
We have conventions for naming message_id, stream_id, etc. values that
way for readability; these entries are violations of those conventions.
2020-01-16 13:27:27 -08:00
Anders Kaseorg f13f744874 settings_profile_fields: Clean up field_id type confusion.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg 45bee2f512 js: Clean up stream_id type confusion.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg 1a07f7b158 js: Clean up user_id type confusion.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg e6178f2abd settings_account: Return IntDict from initialize_custom_user_type_fields.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg 6ec5a1f306 user_groups: Convert user_group_by_id_dict from Dict to IntDict.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg 38c7fc0038 typeahead_helper: Convert rendered fields from Dict to IntDict.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg 031afa6014 bots: Convert services from object to IntDict.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg bc626e2470 bots: Convert bots from object to IntDict.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Anders Kaseorg 4341b7b252 user_groups: Convert members from Dict to Set.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 13:23:47 -08:00
Tim Abbott eb8b3539ad test_classes: Remove DEFAULT_REALM variable.
This essentially unused legacy variable was causing Zulip to query the
database at import time, which is generally not something we aim to
do.

Combined with the issue fixed in the previous commit, this variable
resulted in test-backend providing an unhelpful crash when provision
hadn't updated the unit testing database.
2020-01-16 13:13:46 -08:00
Tim Abbott 8ff5d8ca89 test_classes: Clean up API_KEYS cache.
Since the intent of our testing code was clearly to clear this cache
for every test, there's no reason for it to be a module-level global.

This allows us to remove an unnecessary import from test_runner.py,
which in combination with DEFAULT_REALM's definition was causing us to
run models code before running migrations inside test-backend.

(That bug, in turn, caused test-backend's check for whether migrations
needs to be run to happen sadly after trying to access a Realm,
trigger a test-backend crash if the Realm model had changed since the
last provision).
2020-01-16 13:07:26 -08:00
Aditya Bansal d0492b103b template_parser: Fix Jinja2 tag matching with strip whitespace syntax.
In this commit, we basically match any kinda of jinja2 start tag,
    no matter its special kind (eg. jinja2_whitespace_stripped_start)
    to any kinda jinja2 end tag (eg. jinja2_whitespace_stripped_end)

    Idea is special operators like `-` do not change the meaning of
    inline tag and thus matching shouldn't depend upon this.
2020-01-16 12:53:34 -08:00
Tim Abbott c4ba44824d Update changelog for Zulip Server 2.1.2 release. 2020-01-16 12:39:00 -08:00
Anders Kaseorg 319e2231b8 thumbnail: Tighten fix for CVE-2019-19775 open redirect.
Due to a known but unfixed bug in the Python standard library’s
urllib.parse module (CVE-2015-2104), a crafted URL could bypass the
validation in the previous patch and still achieve an open redirect.

https://bugs.python.org/issue23505

Switch to using django.utils.http.is_safe_url, which already contains
a workaround for this bug.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-16 12:36:24 -08:00
Tim Abbott ef1f6b1c33 filter: Allow marking is:mentioned messages as read.
We may revisit this in the future, but similar to is:private, the
current Zulip user experience makes users expect that in the
is:mentioned view, they should really be able to mark messages as
read.

Further, the practice use case for not marking them as read is very
low, since it's rare for someone to have so many mentions that
revisiting the mentions view isn't sufficient to see everything that
needs their attention.
2020-01-16 11:10:36 -08:00
Tim Abbott 409e320d9d filter: Add streams:public to sorted_term_types.
This is for consistency with in:, has:, and similar values where
there's a fixed set of RHS entries.
2020-01-16 11:05:07 -08:00
Rohitt Vashishtha d4ef50634c filter: Remove is_exactly().
Previously, is_exactly() had already been repalced with can_bucket_by().
This commit removes is_exactly() and replaces its usage in our tests
with can_bucket_by().
2020-01-16 10:54:59 -08:00
Rohitt Vashishtha 8a1299d076 filter: Add 'in:*' to sorted_term_types.
This simplifies our handling of in-home and in-all cases in
can_mark_messages_read().
2020-01-16 10:54:59 -08:00
Steve Howell 316eda071d refactor: Clean up can_mark_messages_read.
We now explicitly enumerate various cases, which
should make it easier to change this code.
2020-01-16 10:54:59 -08:00
Rohitt Vashishtha bb579f8823 unreads: Remove is_reading_mode().
This was a part of an experiment we ran on chat.zulip.org in Jul 2018
and surrounding code that used it never got merged to master.

See: https://chat.zulip.org/#narrow/stream/2-general/topic/un-narrow.20view/near/609506
and c407ba5175.
2020-01-16 10:54:59 -08:00
Steve Howell 741da61407 node: Flatten pm_list testing code a bit.
The `messages` part of the data is going away in
the next commits.  This pre-factoring just makes
the next diff a bit more clean.
2020-01-16 10:32:36 -08:00
Anders Kaseorg 8c733a3f68 create-db.sql: Start by dropping the zulip database if needed.
At some point the PostgreSQL Docker image started creating the zulip
database for us, which caused our CREATE DATABASE to fail.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Anders Kaseorg 298d45b46a create-db.sql: Handle exception if zulip user already exists.
Fixes #13530.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 18:04:34 -08:00
Steve Howell 3cc6ff5480 templates: Cache translations.
For Manage Streams, when we render the subscriptions
template, a significant amount of time is taken
by the "t" helper.

Obviously for the first call, we expect "t" to be
somewhat expensive, but subsuquent calls should be
fast, but i18next seems to have some overhead.
Also, we can save a tiny bit of overhead (marking it
as a safe string) that comes from our helper.

As an aside, are we sure it's ok to mark translations
as safe strings?

To test before and after, use blueslip.timings before
and after this commit.  When I tested with about 300
streams, the difference is pretty striking:

    without cache: 100ms
    with cache: 20ms

This is particularly interesting, since the subscriptions
templates have long strings for things like the SVG-based
checkmarks, but they're not really the bottleneck.

Unfortunately, this doesn't seem to be a huge win
elsewhere.  In some places we don't call "t", but of
course those might change in the future and benefit from
the cache.  And in other places we have smart widgets
that avoid rendering all N objects at one (e.g. buddy
list and list_render).

So this might be too big a hammer to speed up one
screen (albeit a really slow one).  It's possible
that we should simply move the i18n.t step **outside**
of certain templates to avoid doing them in a loop.
2020-01-15 18:01:59 -08:00
Tim Abbott 5b30138a31 casper tests: Add an additional waitWhileVisible check.
This fix at least one race and help prevent more race issues in the
casper tests for the organization administration pages.
2020-01-15 17:51:46 -08:00
Anders Kaseorg a82032a182 generate_secrets: Enable Redis authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg 3360df7ad1 generate_secrets: Enable memcached authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg cdda983e90 settings: Support optional memcached authentication.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Tim Abbott d816a12db9 casper tests: Fix buggy submit_notifications_stream_settings.
This new function Casper testing function improperly used
`casper.then` in a nested fashion rather than in series, which doesn't
work how one expects.  This likely caused the test flakes we've
started seeing with this code path since adding
submit_notifications_stream_settings (though it's hard to prove).
2020-01-15 14:11:29 -08:00
Tim Abbott 3a9568b4d4 migrations: Fix zulipinternal migration corner case.
It's theoretically possible to have configured a Zulip server where
the system bots live in the same realm as normal users (and may have
in fact been the default in early Zulip releases?  Unclear.).  We
should handle these without the migration intended to clean up naming
for the system bot realm crashing.

Fixes #13660.
2020-01-15 13:59:31 -08:00
Steve Howell aea369f878 Refine user-related typeahead results for large realms.
We now incorporate people.get_message_people() in our
logic for compose/PM typeaheads.  This not only gives
users better results in some cases, but it will also
improve performance for large realms in some cases.
2020-01-15 12:22:23 -08:00
Steve Howell 8f35700da8 refactor: Extract get_message_people.
We'll use this in two places coming up, so it's
worth extracting, plus I wanted to add the
fairly lengthy comment here.  (Tim, feel free
to edit down the comment as you see fit).
2020-01-15 12:22:23 -08:00
Steve Howell c47fc36201 refactor: Extract filter_persons.
This extraction will make sense in the next commit.
2020-01-15 12:22:23 -08:00