Commit Graph

39466 Commits

Author SHA1 Message Date
Alex Vandiver 0d5760d59f install-aws-server: Force non-interactive dist-upgrade.
Installing an updated linux kernel package, as can happen during the
`apt dist-upgrade` done by the installer, can cause grub to pop up a
prompt to update its configuration file.  In an unattended headless
configuration, this will stop the installation.

Explicitly configure apt to be non-interactive, and prefer the newest
configuration, during the install.
2020-10-15 14:39:20 -07:00
Anders Kaseorg 449b16be59 dependencies: Use patched source-map with a performance fix.
https://github.com/mozilla/source-map/issues/370

This makes Webpack about 9% faster.  Before:

$ multitime -n9 -s0 node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
===> multitime results
1: node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
            Mean        Std.Dev.    Min         Median      Max
real        18.243      0.107       18.090      18.236      18.443
user        27.913      0.188       27.714      27.843      28.251
sys         2.028       0.043       1.933       2.039       2.074

After:

$ multitime -n9 -s0 node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
===> multitime results
1: node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
            Mean        Std.Dev.    Min         Median      Max
real        16.686      0.085       16.542      16.684      16.885
user        25.965      0.167       25.559      26.022      26.163
sys         1.965       0.064       1.807       1.998       2.010

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-10-15 14:38:41 -07:00
Tim Abbott bf66e9c4ab actions: Add transaction.atomic to bulk_add_subs_to_db_with_logging.
This will ensure that we always fully execute the database part of
modifying subscription objects.  In particular, this should prevent
invariant failures like #16347 where Subscription objects were created
without corresponding RealmAuditLog entries.

Fixes #16347.
2020-10-14 11:06:00 -07:00
Steve Howell 5728149e94 performance: Streamline query to add subscribers.
We don't need the select_related('user_profile')
optimization any more, because we just keep
track of user info in our own data structures.

In this codepath we are never actually modifying
users; we just occasionally need their ids or
emails.

This can be a pretty substantive improvement if
you are adding a bunch of users to a stream
who each have a bunch of their own subscriptions.

We could also limit the number of full rows in this
query by adding an extra hop to the DB just to
get colors (using values_list), and then only get
full sub info for the streams that we're adding, rather
than getting every single subscription, in full, for each user.

Apart from finding what colors the user has already
used, the only other reason we need all the columns
in Subscription here is to handle streams that
need to be reactivated.  Otherwise we could do
only("id", "active", "recipient_id", "user_profile_id")
or similar.  Fortunately, Subscription isn't
an overly wide table; it's mostly bool fields.

But by far the biggest thing to avoid is bringing
in all the extra user_profile data.

We have pretty good coverage on query counts here,
so I think this fix is pretty low risk.
2020-10-14 11:03:07 -07:00
Steve Howell 116a441bc5 refactor: Introduce SubInfo class.
This class removes a lot of the annoying tuples
we were passing around.

Also, by including the user everywhere, which
is easily available to us when we make instances
of SubInfo, it sets the stage to remove
select_related('user_profile').
2020-10-14 10:53:10 -07:00
Steve Howell febef45e38 minor: Add comments to do_get_streams. 2020-10-14 10:53:10 -07:00
Steve Howell a9356508ca events: Stop sending occupy/vacate events.
We used to send occupy/vacate events when
either the first person entered a stream
or the last person exited.

It appears that our two main apps have never
looked at these events.  Instead, it's
generally the case that clients handle
events related to stream creation/deactivation
and subscribe/unsubscribe.

Note that we removed the apply_events code
related to these events.  This doesn't affect
the webapp, because the webapp doesn't care
about the "streams" field in do_events_register.

There is a theoretical situation where a
third party client could be the victim of
a race where the "streams" data includes
a stream where the last subscriber has left.
I suspect in most of those situations it
will be harmless, or possibly even helpful
to the extent that they'll learn about
streams that are in a "quasi" state where
they're activated but not occupied.

We could try to patch apply_event to
detect when subscriptions get added
or removed. Or we could just make the
"streams" piece of do_events_register
not care about occupy/vacate semantics.
I favor the latter, since it might
actually be what users what, and it will
also simplify the code and improve
performance.
2020-10-14 10:53:10 -07:00
Steve Howell 1bcb8d8ee8 performance: Avoid computing page_params.streams in webapp.
The query to get "occupied" streams has been expensive
in the past.  I'm not sure how much any recent attempts
to optimize that query have mitigated the issue, but
since we clearly aren't sending this data, there is no
reason to compute it.
2020-10-14 10:53:10 -07:00
Steve Howell 79803f01f4 minor: Format some code in events.py. 2020-10-14 10:53:10 -07:00
Steve Howell 193ca397f9 tests: Include deactivated users for subscribe test. 2020-10-14 10:53:10 -07:00
Anders Kaseorg 2a4a775e2b requirements: Install PycURL for Thumbor.
Fixes “thumbor:WARNING pycurl usage is advised. It could not be loaded
properly. Verify install...”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-14 10:52:32 -07:00
Alex Vandiver f61ac4a28d puppet: Move frontend monitoring into its own file.
This allows it to be pulled in for deploys like czo, which don't use
the full `zulip_ops::app_frontend`, but we wish to monitor.
2020-10-13 17:37:32 -07:00
Aman Agrawal fbf7cb82a7 web_public_guest: Rename to web_public_visitor for clarity.
Using web_public_guest for anonymous users is confusing since
'guest' is actually a logged-in user compared to
web_public_guest which is not logged-in and has only
read access to messages. So, we rename it to
web_public_visitor.
2020-10-13 16:59:52 -07:00
Tim Abbott 7c2c82b190 nginx: Update nginx configuration for fhir/hl7 organization.
We should eventually add templating for the set of hosts here, but
it's worth merging this change to remove the deleted hostname and
replace it with the current one.
2020-10-13 16:50:26 -07:00
Anders Kaseorg 723d285e46 nginx: Redirect {www.,}zulipchat.com, www.zulip.com to zulip.com.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 16:49:23 -07:00
Steve Howell e7a8c7ac48 test: Improve tests for bulk-adding subscribers.
This is a more thorough test of adding multiple
streams for multiple users, including streams
that users have already subscribed to.

The extra queries here are due to the fact
that we call `principal_to_user_profile` in
a loop in the view.  So that's an example
of O(N) overhead.  We may be able to bulk-fetch
these users eventually.
2020-10-13 18:54:55 -04:00
Steve Howell c29ba75135 refactor: Extract send_messages_for_new_subscribers.
This is a pure extraction, except that I remove a
redundant check that `len(principals) > 0`.  Whenever
that value is false, then `new_subscriptions` will
only have one possible entry, which is the current
user, and we skip that in the loop.
2020-10-13 18:54:55 -04:00
Steve Howell 3b338ec32e performance: Optimize filter_stream_authorization.
We no longer do O(N) queries to get existing streams.

This is a somewhat contrived use case--generally, we
are not trying to re-subscribe a user to several
streams.  Still, we want to avoid this.

This commit also makes `test_bulk_subscribe_many`
do more work, and the change to the test helped
me discover this bug.
2020-10-13 18:54:55 -04:00
Anders Kaseorg 6564540d15 docs: Fix some spelling errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Anders Kaseorg dd48dbd912 docs: Add spaces to “check out”, “log in”, “set up”, “sign up” as verbs.
“Checkout”, “login”, “setup”, and “signup” are nouns, not verbs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Steve Howell 598601e8fc stream events: Prevent spurious events.
If a user asks to be subscribed to a stream
that they are already subscribed to, then
that stream won't be in new_stream_user_ids,
and we won't need to send an event for it.

This change makes that happen more automatically.
2020-10-13 11:28:17 -07:00
Steve Howell 18771099e4 performance: Introduce new_stream_user_ids.
Let
    U = number of users to subscribe
    S = number of streams to subscribe

We were technically doing N^3 amount of work
when we sent certain events, or to be more
precise, U * S * S amount of work.  For each
stream, we were looping through a list of tuples
of size U * S to find the users for the stream.

In practice either U or S is usually 1, so the
performance gains here are probably negligible,
especially since the constant factors here
were just slinging around Python data.

But the code is actually more readable now, so
it's a double win.
2020-10-13 11:28:17 -07:00
Steve Howell ebb605319b refactor: Rename stream_map to recipient_id_to_stream.
I want to make a new dict called stream_id_to_stream,
and stream_map would be confusing.
2020-10-13 11:28:17 -07:00
Steve Howell b502957184 refactor: Extract new_recipient_ids local.
We rename needs_new_sub (which sounds like
a boolean!) to new_recipient_ids, and we
calculate it explicitly within the loop, so
that we don't need to worry as much about
subsequent passes through the loop mutating it.

This allows us to also remove recipient_ids,
which in turn lets us remove recipients_map,
albeit with a small tweak for stream_map.

I also introduce the my_subs local, which
I use to more directly populate used_colors,
as well as using it as the loop var.
2020-10-13 11:28:17 -07:00
Steve Howell 766892d8aa import: Reuse get_last_message_id() helper. 2020-10-13 11:28:17 -07:00
Steve Howell 188cc9bb3b minor: Fix user/stream in test_subscriptions. 2020-10-13 11:28:17 -07:00
Steve Howell 9df9934ed6 refactor: Pass realm to bulk_add_subscriptions.
I think it's important that the callers understand
that bulk_add_subscriptions assumes all streams
are being created within a single realm, so I make
it an explicit parameter.

This may be overkill--I would also be happy if we
just included the assertions from this commit.
2020-10-13 11:28:17 -07:00
Steve Howell efc931a671 minor: Extract realm local. 2020-10-13 11:28:17 -07:00
Steve Howell b2d0a2efb9 refactor: Extract send_subscription_add_events.
This function now does all the work that we used
to do with notify_subscriptions_added happening
inside a loop.

There's a small fine-tuning here, where we only
get recent traffic on streams that we're actually
sending events for.
2020-10-13 11:28:17 -07:00
Steve Howell 223ce83a0a refactor: Clean up call to notify_subscriptions_added.
We now just pass in all_subscribers_by_stream, rather
than a callback.

We also move sub_tuples_by_user closer to the
loop where we call notify_subscriptions_added.
2020-10-13 11:28:17 -07:00
Alex Vandiver c8df9a150e puppet: Drop all log2zulip configuration.
Disabled on webservers in 047817b6b0, it has since lingered in
configuration, as well as running (to no effect) every minute on the
loadbalancer.

Remove the vestiges of its configuration.
2020-10-13 11:00:50 -07:00
Alex Vandiver b431b1b021 puppet: Remove misleading motd.
This banner shows on lb1, advertising itself as lb0.  There is no
compelling reason for a custom motd, especially one which needs to
be reconfigured for each host.
2020-10-13 11:00:36 -07:00
ryanreh99 111ba564ae popover: Show normal popover for the `stream_subscription_info` users.
This shows the normal popover instead of extended profile.
We use the standard event handler attached to the body element in
`popovers.js` instead of attaching a new one.
2020-10-12 17:42:52 -07:00
ryanreh99 60883b52ee node tests: Refactor and fix stub logic in `stream_edit.js`.
Since the denmark stream stub has subscribed = true,
we include the current user id in it's subscribers set.
We remove user `me` from sweden stream stub for the same
reason.
2020-10-12 17:41:02 -07:00
Vishnu KS b4eafaa022 settings: Typecast FREE_TRIAL_DAYS into integer.
This was missed out in d32362e53e
2020-10-12 17:15:28 -07:00
Steve Howell 811426b345 Extract send_stream_creation_events_for_private_streams.
We can probably avoid passing in users here.
2020-10-12 16:40:37 -07:00
Steve Howell 1cfaef0d1a refactor: Simplify pick_color logic.
This removes the need to jankily mutate
the active flag in the caller, and we don't
need to mutate our subs_by_user either.
2020-10-12 16:40:37 -07:00
Steve Howell 13569ff97a refactor: Eliminate new_subs.
We now just process new subs for a user immediately
within the loop.
2020-10-12 16:40:37 -07:00
Steve Howell 8c70fbde78 refactor: Use subs_to_add in return value.
The subs_to_add is directly related to a var
called new_subs, which I hope to eliminate
soon.
2020-10-12 16:40:37 -07:00
Steve Howell 1afca3d430 minor: Extract local for stream. 2020-10-12 16:40:37 -07:00
Steve Howell 84aa1389d8 Extract bulk_add_subs_to_db_with_logging.
This is a trivial code extraction.
2020-10-12 16:40:37 -07:00
Steve Howell 3ff9ce78ea refactor: Extract send_peer_add_events. 2020-10-12 16:40:37 -07:00
Areeb Beigh (Euphinx) 5d6f3eef29
docs: Fix typos in events-system.md.
Fixes 2 typos in events-system.md:
it's handler -> its handler
Event Queue Server server -> Event Queue Server
2020-10-12 12:10:50 -07:00
shanukun 537051e4f1 api docs: Fix JavaScript example for /streams api endpoint.
This commit fixes the JavaScript tab in 'Usage example'
for 'Get all streams' that currently shows python code.
2020-10-11 23:12:33 -07:00
Alex Vandiver f3ba227614 create_user: Strip whitespace from initial password file.
Fixes #12144.
2020-10-11 16:29:00 -07:00
Cody Piersall 5dab6e9d31 emoji-upload: Fix transparency issues on GIF emoji upload.
This preserves the alpha layer on GIF images that need to be resized
before being uploaded.  Two important changes occur here:

1. The new frame is a *copy* of the original image, which preserves the
   GIF info.
2. The disposal method of the original GIF is preserved.  This
   essentially determines what state each frame of the GIF starts from
   when it is drawn; see PIL's docs:
   https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#saving
   for more info.

This resolves some but not all of the test cases in #16370.
2020-10-11 16:23:07 -07:00
Anders Kaseorg b7a94be152 python: Catch BaseException when we need to clean something up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:16:16 -07:00
Anders Kaseorg 7f69c1d3d5 python: Catch specific exceptions from requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg 17ac17286c python: Catch specific exceptions from subprocess.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg aabef3d9be python: Catch specific exceptions from orjson.
Followup to #16120.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00