Commit Graph

4769 Commits

Author SHA1 Message Date
Steve Howell e3ad9baf1d presence: Add process_presence_event.
This lets us conditionally remove the email
field from a presence event if the client
has registered with the slim_presence flag.
2020-02-04 12:30:36 -08:00
Steve Howell 9847d4d9a3 refactor: Use user_id in get_status_dict_by_user.
This avoids a needless user lookup in apply_event.
2020-02-04 12:30:36 -08:00
Steve Howell a672a00677 presence: Add user_id to presence event.
In a later commit, we will eliminate email for
clients who have set slim_presence as their
preference.
2020-02-04 12:30:36 -08:00
Steve Howell bf9144ff69 presence: Add slim_presence flag.
This flag affects page_params and the
payload you get back from POSTs to this
url:

    users/me/presence

The flag does not yet affect the
presence events that get sent to a
client.
2020-02-04 12:30:34 -08:00
Vishnu Ks 5dfd4ea38d export: Remove unused parameter from _get_exported_s3_record. 2020-02-03 14:09:05 -08:00
Vishnu Ks 5a59bf329e import: Skip setting user_profile_id metadata only if unavailable. 2020-02-03 14:09:05 -08:00
Vishnu Ks 2ea53a347a import: Support importing realm icon and logo.
Fixes #11216
2020-02-03 14:09:05 -08:00
Vishnu Ks af3a37b58b upload: Refactor out realm_avatar_and_logo_path function. 2020-02-03 14:09:05 -08:00
Tim Abbott df110e8ff9 test_fixtures: Note populate_db depends on server_initialization.py.
This should ensure that folks rebasing past this commit from an older
database model get their database rebuilt in the way that will
match the test_subs.py query count of 40.
2020-02-03 10:38:04 -08:00
Ryan Rehman 3dc7d60ffe muting: Record DateTime when a Topic is muted.
This includes the necessary migration to add
the date_muted field to the MutedTopic class
and populates it with a hard coded value.
2020-02-02 20:49:53 -08:00
Mateusz Mandera bf89cf2b4b rate_limiter: Use ABC for defining the abstract class RateLimitedObject. 2020-02-02 19:15:13 -08:00
Mateusz Mandera cb71a6571e rate_limiter: Rename 'all' domain to 'api_by_user'. 2020-02-02 19:15:13 -08:00
Mateusz Mandera 06198af5b9 auth: Handle rate limiting in OurAuthenticationForm and user_settings.
These parts of the code should catch the RateLimited exception and
generate their own, apprioprate user-facing error message.
2020-02-02 19:15:13 -08:00
Mateusz Mandera 335b804510 exceptions: RateLimited shouldn't inherit from PermissionDenied.
We will want to raise RateLimited in authenticate() in rate limiting
code - Django's authenticate() mechanism catches PermissionDenied, which
we don't want for RateLimited. We want RateLimited to propagate to our
code that called the authenticate() function.
2020-02-02 19:15:00 -08:00
Mateusz Mandera a6a2d70320 rate_limiter: Handle multiple types of rate limiting in middleware.
As more types of rate limiting of requests are added, one request may
end up having various limits applied to it - and the middleware needs to
be able to handle that. We implement that through a set_response_headers
function, which sets the X-RateLimit-* headers in a sensible way based
on all the limits that were applied to the request.
2020-02-02 19:15:00 -08:00
Mateusz Mandera 4cc5d2464c rate_limiter: Expand support for different domains. 2020-02-02 19:15:00 -08:00
Tim Abbott 51706bdc3a stream: Deduplicate lists of stream/subscriptions fields.
While the result of this change doesn't completely do what we need, it
does remove a huge amount of duplicated lists of fields.  With a bit
more similar work, we should be able to eliminate a broad category of
potential bugs involving Stream and Subscription objects being
represented inconsistently in the API.

Work towards #13787.
2020-02-02 18:34:45 -08:00
Tim Abbott 238bc386cb actions: Deduplicate parts of get_web_public_subs.
This has the side of effect of making new fields we add to Stream be
automatically included, which will help maintain this code as we
upgrade it.

This commit adds is_web_public, history_public_to_subscribers, and
email_notifications fields to the dictionary.
2020-02-02 17:42:12 -08:00
Tim Abbott eac07698dd users: Add nocoverage tag for settings.SYSTEM_BOT_REALM conditional.
This is code for safety that should never happen and is likely
annoying to setup an automated test to verify.
2020-01-31 14:51:12 -08:00
Tim Abbott 5825a155cc users: Use format_user_row in events system as well.
This completes the deduplication of our logic for turning users into
dictionaries in the Zulip API.
2020-01-31 14:47:16 -08:00
akashaviator 20b8b29d11 users: Rewrite get_cross_realm_dicts to call format_user_row.
This modifies get_cross_realm_dicts in zerver.lib.users to call
format_user_row.  This is done to remove current and prevent future
inconsistencies between in the dictionary formats for get_raw_user_data
and get_cross_realm_dicts.

Implementation substantially rewritten by tabbott.

Fixes #13638.
2020-01-31 14:28:46 -08:00
akashaviator 7d06293ac0 refactor: Cleanup actions.py and events.py in zerver/lib.
This moves get_cross_realm_dicts (from zerver.lib.actions),
get_raw_user_data and get_custom_profile_field_values (from
zerver.lib.events) to zerver.lib.users.
2020-01-31 13:53:47 -08:00
Vishnu KS bd460af099 emails: Remove unecessary call to message_content_allowed_in_missedmessage_emails. 2020-01-31 12:29:58 -08:00
Vishnu KS 47e442e4a4 emails: Show proper message when email content is not shown. 2020-01-31 12:29:58 -08:00
akashaviator bd58e3397f events: Extract user_data function from get_raw_user_data.
This extracts the user_data inner function from get_raw_user_data as a
reusable function.  We intend to reuse it for cross-realm user dicts.
A few changes were made while extracting it:

* Renaming the UserProfile argument to acting_user, so we can do loops
  over a local user_profile variable.
* Moved it to zerver.lib.users, as that's a more appropriate home for
  this function formatting data on users.

* Simplified the calling convention for passing custom profile fields
  to reflect the fact that this function processes a single user (and
  is expected to be called in a loop).
2020-01-30 13:32:35 -08:00
Mateusz Mandera 8bd3752d13 email_mirror: Handle encoded attachment filenames. 2020-01-30 13:03:47 -08:00
Mateusz Mandera 49b76318c6 email_mirror: Extract handle_header_content function. 2020-01-30 13:03:47 -08:00
Tim Abbott dd969b5339 install: Remove references to "Zulip Voyager".
"Zulip Voyager" was a name invented during the Hack Week to open
source Zulip for what a single-system Zulip server might be called, as
a Star Trek pun on the code it was based on, "Zulip Enterprise".

At the time, we just needed a name quickly, but it was never a good
name, just a placeholder.  This removes that placeholder name from
much of the codebase.  A bit more work will be required to transition
the `zulip::voyager` Puppet class, as that has some migration work
involved.
2020-01-30 12:40:41 -08:00
Mateusz Mandera d68cf21952 server_initialization: Add server_initialized function. 2020-01-30 12:21:31 -08:00
Mateusz Mandera 682dea1b34 test_classes: Fix bug where UserProfile could be passed to client_post.
It would cause JSON overflow error while producing URL coverage report.
2020-01-30 12:13:54 -08:00
Mateusz Mandera 68abddb534 server_initialization: Rename some variables.
This makes the code of create_internal_realm identical to the
corresponding block in initialize_voyager_db.py.
2020-01-29 17:26:45 -08:00
Mateusz Mandera 39b012a276 server_initialization: Set internal bots owners to themselves. 2020-01-29 17:26:45 -08:00
Mateusz Mandera 9c20611a65 server_initialization: Remove unnecessary type annotation. 2020-01-29 17:26:45 -08:00
Mateusz Mandera d24936cbe3 server_initialization: Use tos_version argument in create_users. 2020-01-29 17:26:45 -08:00
Mateusz Mandera 261da5999d populate_db: Extract default client creation to server_initialization. 2020-01-29 17:26:45 -08:00
Mateusz Mandera a25f00a69c populate_db: Extract some functions to server_initialization.py. 2020-01-29 17:26:45 -08:00
Mateusz Mandera 9dcf677bf9 email_mirror: Parse encoded From headers with show_sender=True. 2020-01-29 12:27:35 -08:00
Tim Abbott 05108760f6 narrow: Add support for passing oldest/newest for anchor.
A wart that has long been present inin Zulip's get_messages API is how
to request "the latest messages" in the API.  Previously, the
recommendation was basically to pass anchor=10000000000000000 (for an
appropriately huge number). An accident of the server's implementation
meant that specific number of 0s was actually important to avoid a
buggy (or at least wasteful) value of found_newest=False if the query
had specified num_after=0 (since we didn't check).

This was the cause of the mobile issue
https://github.com/zulip/zulip-mobile/issues/3654.

The solution is to allow passing a special value of anchor='newest',
basically a special string-type value that the server can interpret as
meaning the user precisely just wants the most recent messages.  We
also add an analogous anchor='oldest' or similar to avoid folks
needing to write a somewhat ugly anchor=0 for fetching the very first
messages.

We may want to also replace the use_first_unread_anchor argument to be
a "first_unread" value for the anchor parameter.

While it's not always ideal to make a value have a variable type like
this, in this case it seems like a really clean way to express the
idea of what the user is asking for in the API.
2020-01-29 12:14:06 -08:00
Tim Abbott c0712431df openapi: Add hacky support for oneOf parameter types.
This is required for the upcoming type behavior of the "anchor"
parameter.

This change is the minimal work required to have our OpenAPI code not
fail when checking a union-type value of this form.  We'll likely want
to, in the future, do something nicer, but it'd require more extensive
infrastructure for parsing of OpenAPI data that it's worth with our
current approach (we may want to switch to using a library).
2020-01-29 11:24:58 -08:00
Tim Abbott 91f1825474 test_helpers: Fix POSTRequestMock typing.
The proximal issue here is that in upcoming commits, we're going to
change the type of the `anchor` field in `get_messages_backend` to
support passing either an integer or a string.

Many of our tests using POSTRequestMock currently define a query
object that uses integer values for the integer fields we're going to
pass into it, e.g. {'num_after': 0}.  That is the correct type for
that field in the Zulip API, before HTTP encoding turns it into a
string.  However, because POSTRequestMock didn't use HTTP encoding at
all (which will convert the 0 into a '0'), it ended up passing an
integer to a function that can't possible receive one as an argument.

Ideally, we'd just get rid of POSTRequestMock, since it's a hack, and
just do real HTTP requests instead.

But since it's used in a lot of places making doing so somewhat
impractical, we can get past this issue by just making POSTRequestMock
convert integers to strings.
2020-01-29 11:24:58 -08:00
Tim Abbott 8f50062e49 soft_deactivation: Fix incorrect logging function.
Using logging.info() rather than logger.info() meant that our
zulip.soft_deactivation logger configuration (which, in particular,
included not logging to the console) was not active on this log line,
resulting in the `manage.py soft_deactivate_users` cron job sending
emails every time it ran.

Fixes #13750.
2020-01-28 17:17:43 -08:00
Rohitt Vashishtha 630c564fc7 bugdown: Rewrite List Preprocessor logic to properly parse fences.
Previously, we didn't track opening and closing fences separately,
with led to bugs like not parsing a list that was immediately after
a quoted fence; we treated each ``` as a new fence.

This commit rewrites the function to maintain a stack of currently
open fences. If any of the parent fences is a code fence, we do not
insert a new line before a list.

We also add some test cases specifically to test this behavior with
complexly nested lists.

Fixes #13745.
2020-01-27 17:14:27 -08:00
Mateusz Mandera 92c16996fc redis_utils: Require key_format argument in get_dict_from_redis. 2020-01-26 21:40:15 -08:00
Mateusz Mandera ad460e6ccb redis_utils: Validate requested key length in helper functions. 2020-01-26 21:40:15 -08:00
Mateusz Mandera 8d987ba5ae auth: Use tokens, with data stored in redis, for log_into_subdomain.
The desktop otp flow (to be added in next commits) will want to generate
one-time tokens for the app that will allow it to obtain an
authenticated session. log_into_subdomain will be the endpoint to pass
the one-time token to. Currently it uses signed data as its input
"tokens", which is not compatible with the otp flow, which requires
simpler (and fixed-length) token. Thus the correct scheme to use is to
store the authenticated data in redis and return a token tied to the
data, which should be passed to the log_into_subdomain endpoint.

In this commit, we replace the "pass signed data around" scheme with the
redis scheme, because there's no point having both.
2020-01-26 21:32:44 -08:00
Abhishek-Balaji 434e8d3104 home: Extract compute_show_invites_and_add_streams.
This extracts a function for computing show_invites and
show_add_streams, for better readability and testability.

This commit was substantially cleaned up by tabbott.
2020-01-25 23:41:08 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Mateusz Mandera af2c4a9735 redis: Extract put_dict_in_redis and get_dict_from_redis helpers. 2020-01-23 16:24:07 -08:00
Jonathan Cobb c7433c83ff integrations: Add errbit integration.
Fixes #13685.
2020-01-16 15:33:51 -08:00
Mateusz Mandera d37e6ef921 email_mirror: Use plaintext if html body empty with prefer-html option.
If an email is sent with the .prefer-html option, but it has no html
body, it's better to fall back to plaintext content instead of treating
it as a user error.
2020-01-16 15:25:27 -08:00