This essentially unused legacy variable was causing Zulip to query the
database at import time, which is generally not something we aim to
do.
Combined with the issue fixed in the previous commit, this variable
resulted in test-backend providing an unhelpful crash when provision
hadn't updated the unit testing database.
Since the intent of our testing code was clearly to clear this cache
for every test, there's no reason for it to be a module-level global.
This allows us to remove an unnecessary import from test_runner.py,
which in combination with DEFAULT_REALM's definition was causing us to
run models code before running migrations inside test-backend.
(That bug, in turn, caused test-backend's check for whether migrations
needs to be run to happen sadly after trying to access a Realm,
trigger a test-backend crash if the Realm model had changed since the
last provision).
In this commit, we basically match any kinda of jinja2 start tag,
no matter its special kind (eg. jinja2_whitespace_stripped_start)
to any kinda jinja2 end tag (eg. jinja2_whitespace_stripped_end)
Idea is special operators like `-` do not change the meaning of
inline tag and thus matching shouldn't depend upon this.
Due to a known but unfixed bug in the Python standard library’s
urllib.parse module (CVE-2015-2104), a crafted URL could bypass the
validation in the previous patch and still achieve an open redirect.
https://bugs.python.org/issue23505
Switch to using django.utils.http.is_safe_url, which already contains
a workaround for this bug.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We may revisit this in the future, but similar to is:private, the
current Zulip user experience makes users expect that in the
is:mentioned view, they should really be able to mark messages as
read.
Further, the practice use case for not marking them as read is very
low, since it's rare for someone to have so many mentions that
revisiting the mentions view isn't sufficient to see everything that
needs their attention.
Previously, is_exactly() had already been repalced with can_bucket_by().
This commit removes is_exactly() and replaces its usage in our tests
with can_bucket_by().
At some point the PostgreSQL Docker image started creating the zulip
database for us, which caused our CREATE DATABASE to fail.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
For Manage Streams, when we render the subscriptions
template, a significant amount of time is taken
by the "t" helper.
Obviously for the first call, we expect "t" to be
somewhat expensive, but subsuquent calls should be
fast, but i18next seems to have some overhead.
Also, we can save a tiny bit of overhead (marking it
as a safe string) that comes from our helper.
As an aside, are we sure it's ok to mark translations
as safe strings?
To test before and after, use blueslip.timings before
and after this commit. When I tested with about 300
streams, the difference is pretty striking:
without cache: 100ms
with cache: 20ms
This is particularly interesting, since the subscriptions
templates have long strings for things like the SVG-based
checkmarks, but they're not really the bottleneck.
Unfortunately, this doesn't seem to be a huge win
elsewhere. In some places we don't call "t", but of
course those might change in the future and benefit from
the cache. And in other places we have smart widgets
that avoid rendering all N objects at one (e.g. buddy
list and list_render).
So this might be too big a hammer to speed up one
screen (albeit a really slow one). It's possible
that we should simply move the i18n.t step **outside**
of certain templates to avoid doing them in a loop.
This new function Casper testing function improperly used
`casper.then` in a nested fashion rather than in series, which doesn't
work how one expects. This likely caused the test flakes we've
started seeing with this code path since adding
submit_notifications_stream_settings (though it's hard to prove).
It's theoretically possible to have configured a Zulip server where
the system bots live in the same realm as normal users (and may have
in fact been the default in early Zulip releases? Unclear.). We
should handle these without the migration intended to clean up naming
for the system bot realm crashing.
Fixes#13660.
We now incorporate people.get_message_people() in our
logic for compose/PM typeaheads. This not only gives
users better results in some cases, but it will also
improve performance for large realms in some cases.
We'll use this in two places coming up, so it's
worth extracting, plus I wanted to add the
fairly lengthy comment here. (Tim, feel free
to edit down the comment as you see fit).
A recent commit removed test coverage for the
actual filtering/sorting of mention typeaheads
when you did a non-silent method. This commit
now tests that important step again.
Note that we also had (and still have) tests
that make sure the is_silent flag is set
correctly by get_candidates.
We don't have a true full-stack test, but those
can be quite tricky to set up and maintain.
This is relatively unobtrusive, and we don't send
anything to the server.
But any user can now enter blueslip.timings in the
console to see a map of how long things take in
milliseconds. We only record one timing per
event label (i.e. the most recent).
It's pretty easy to test this by just clicking
around. For 300 users/streams most things are
fast except for:
- initialize_everything
- manage streams (render_subscriptions)
Both do lots of nontrivial work, although
"manage streams" is a bit surprising, since
we're only measuring how long to build the
HTML from the templates (whereas the real
time is probably browser rendering costs).
This change sets us up to optimize how we
filter users in the admin user settings.
See #13554 for more context on the user
facing issues.
This fix is basically three related things:
- Add filterer options to list_render.
- Add helper method to people.js.
- Use filterer in settings_users.js.
The filter "callback" was only a "callback" in the
most general sense of the word.
It's just a filter predicate that returns a bool.
This is to prepare for another filtering option,
where the caller can filter the whole list
themselves. I haven't figured out what I will name
the new option yet, but I know I want to make the
two options have specific names.
We are already providing callbacks everywhere, so
it would be nice to eliminate some dead code.
This also speeds things up ever so slightly (no
longer type-checking the option every time through
the loop).
We also split out exports.filter to make unit testing
easier. The function seems kinda silly now, being so
small, but I hope to add another filtering option soon.
It's a bit confusing when you read this code to know
where the original list was created. I'm not a huge
fan of the cache scheme here, but it does seem to
work for live updates.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Currently, if we change stream we see the immediate saving of stream, but
it is more convenient to have "Save" and "Discard" buttons as we use
everywhere else in the organization setting subsystem.
This is a preliminary commit for further commits where we will be using the
newly created function `save_discard_widget_status_handler` in click
handler for changing the notification stream.
This refactors `discard_property_element_changes` and
`check_property_changed` function to move conditional statements of
properties that need to be handled separately. It's a preliminary commit in
the series of using save/discard widget for notification stream setting.
As the part of making notification stream settings to change using
"save/discard" widget instead of immediate saving, we need to access the
stream id which is being selected at the moment.
(This is another preliminary commit in the direction of having
"save/discard" widget show up rather than saving immediately.)
The code for selecting and processing the stream for both types of
notifications is almost the same, so de-duplicated.
For "New stream notifications" and "New user notifications" it is more
intuitive to just use the new system for showing success/saving status
feedback.
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.
At this point the tool is just a little too noisy
to be helpful. An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS. I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.
I just don't think it's worth maintaining the tool
at this point.
Fixes#13416
We used to search only one level in depth through the MIME structure,
and thus would miss attachments that were nested deeper (which can
happen with some email clients). We can take advantage of message.walk()
to iterate through each MIME part.
return in that loop was a bug, which would lead to the To: header not
being set even though data['recipient'] = str(message['To']) is being
run next, thus requiring the header. We can remove the return
statement and now the loop will overwrite all the potentially
troublesome headers.
Anders Kaseorg
Tim Abbott
Aditya Bansal
Rohitt Vashishtha
Steve Howell
Pragati Agrawal
Mateusz Mandera