Commit Graph

5843 Commits

Author SHA1 Message Date
Alex Vandiver e53be6d043 email: Set an envelope-from which may be different from the From: field.
The envelope-from is used by the MTA if the destination address is not
deliverable.  Route all such mail to the noreply address.
2021-02-24 17:32:28 -08:00
Mateusz Mandera 51d7f24d20 actions: Remove realm argument to internal_send_stream_message.
The argument is redundant.
2021-02-23 15:26:47 -08:00
Mateusz Mandera 09fc79f911 actions: Remove realm argument to internal_send_private_message.
The argument is redundant.
2021-02-23 15:26:47 -08:00
Suyash Vardhan Mathur dd8964a31f api docs: Fix id and type fields of events and display them.
Currently, the ID and Type fields didn't have a description,
and weren't being displayed. Added a schema component to add
descriptions, and display on the api page. Fixes part of #15967.
2021-02-23 15:22:53 -08:00
sahil839 d71afc5a26 actions: Include ROLE_MODERATOR in realm_user_count_by_role.
This commmit includes ROLE_MODERATOR in realm_user_count_by_role.

We also update test_change_role in test_audit_log.py to include
changes for moderator role as well.
2021-02-23 15:01:14 -08:00
sahil839 6b5cf231a1 users: Add new user 'shiva' as realm moderator.
Note that at this point, it's not possible to create moderator users;
this just will make it easier to write tests for logic involving them
as we develop the feature.
2021-02-23 15:00:49 -08:00
sahil839 81ae29d461 stream: Allow new bot to send message if its owner is full member.
We currently not allow new bots to send message in stream with post
policy as 'STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS', but we should
allow them to send messages if their owner is a full member.

This will make it consistent with behavior in stream with post
policy as 'STREAM_POST_POLICY_ADMINS_ONLY' where we allow non admin
bots with owner as admin to send messages.
2021-02-18 18:38:52 -08:00
sahil839 3df87d0901 stream: Fix error handling in access_stream_for_send_message.
According to tests we should not allow bot without owners to
post in streams with STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS.
But the code does not handle this and the related test passes
and raises error for case of bots without owner because the bot
is itself a new member.

This commit fixes this by adding a condition to check if there
is no bot owner and then raise error if there is no owner.
2021-02-18 18:38:52 -08:00
Abhijeet Prasad Bodas fc0488fdb1 actions: Rename notify_topic_moved_streams function.
This is a minor refactor which renames the
notify_topic_moved_streams function to
send_message_moved_breadcrumbs.

This is done because this function will be also used
for other things in the future, when moving streams
or when using the /digress command, for example.
2021-02-16 17:28:59 -08:00
Suyash Vardhan Mathur 96bfeeb9e6 api docs: Expand checking for deprecated fields.
Added assertion to check that if a deprecated flag is in a field's
schema, then it should have deprecated mentioned in description
as well, and moved these checks to a separate function.
Fixes part of #15967.
2021-02-16 15:34:52 -08:00
Sumanth V Rao 540cca595c hotspots: Fix typos in function name and code comment. 2021-02-15 18:33:21 -08:00
Anders Kaseorg b728727d9d timeout: Remove unnecessary varargs support.
Mypy can check it this way.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-15 17:05:28 -08:00
Anders Kaseorg 77b7914cd7 test_helpers: Strengthen some decorator types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-15 17:05:28 -08:00
Anders Kaseorg 6eb1705068 cache: Strengthen ignore_unhashable_lru_cache decorator type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-15 17:05:28 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 5028c081cb python: Merge concatenated string literals that Black would uglify.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Mateusz Mandera 90636d5e81 events: Fix bug in get_recent_conversations_recipient_id.
user_profile.id was confused for user_profile.recipient_id. These bugs
are particularly sneaky as they can go undetected by tests due to ids of
objects accidentally coinciding. We add a mitigation for this class of
mistakes by shifting the Recipient.id sequence in test db.

This was introduced in dda3ff41e1.
On the rare occasion where user_profile.id would coincide with
recipient_id passed to the function, we would return the wrong value.
That is, instead of correctly returning recipient_id, we would return
sender.recipient_id - recipient id of the sender of the message, thus
possibly returning user_profile.recipient_id (if user_profile is the
sender) - exactly the situation the function wanted to avoid
with the `if recipient_id == my_recipient_id:` if. Ultimately resulting
in incorrect/malformed data in
state['raw_recent_private_conversations'].
2021-02-09 17:45:34 -08:00
Vishnu KS 3f4f16f4f1 digest: Remove comments from get_hot_topics.
The code is self explanatory.
2021-02-09 10:35:47 -08:00
Vishnu KS e9587900e6 digest: Use heapq.nlargest instead of sorted.
nlargest is the natural fit for selecting n biggest items
from an unsorted list. It's more readable as well as more
efficent (even though we don't care much about the efficeny
in this particular case).
2021-02-09 10:35:47 -08:00
Vishnu KS 738d759e6f digest: Create MAX_HOT_TOPICS_TO_BE_INCLUDED_IN_DIGEST constant. 2021-02-09 10:35:47 -08:00
Vishnu KS c0bd05b52d digest: Check whether length of hot topics is 4.
The length of hot topics would not exceed 4.
2021-02-09 10:35:47 -08:00
Vishnu KS 5c026d67e3 digest: Sort topics in descending order in get_hot_topics.
We want topics with high diversity and large lengths.
So they should be sorted with reverse=True.

This bug seems to be introduced in 936171d258
2021-02-09 10:35:47 -08:00
Suyash Vardhan Mathur c9c40d4fd2 api docs: Cleaned up CSS for parameter classes.
Deduplicated CSS classes of data types of response and
request parameters in API Documentation to use a single
class.
2021-02-09 10:31:36 -08:00
Suyash Vardhan Mathur 9d74c7001d api docs: Fix non-rendering response parameter data types.
The current logic doesn't display data types when the additionalProperties
variables are not object, but are array of strings, etc. Changed the if
condition to allow rendering in such cases.
2021-02-09 10:29:25 -08:00
Alex Vandiver d0f0c2f2ed digest: Fix the structure that we enqueue across when digesting.
This rename was missed in bfa0bdf3d6.
Without this fix, digest messages fail to send.
2021-02-08 17:28:59 -08:00
Steve Howell d0ba3cadcf minor: Clean up code formatting for do_create_user.
This makes the code easier to visually scan.
2021-02-08 09:07:04 -05:00
Anders Kaseorg d13a039b54 actions: Sort available_notification_sounds.
os.listdir uses an arbitrary filesystem-dependent order.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-07 06:33:55 -05:00
m-e-l-u-h-a-n 0e6343c071 users: Clarify readability issues related to access_user_by_id.
zerver/lib/users.py has a function named access_user_by_id, which is
used in /users views to fetch a user by it's id. Along with fetching
the user this function also does important validations regarding
checking of required permissions for fetching the target user.

In an attempt to solve the above problem this commit introduces
following changes:
1. Make all the parameters except user_profile, target_user_id
   to be keyword only.
2. Use for_admin parameter instead of read_only.
3. Adds a documentary note to the function describing the reason for
   changes along with recommended way to call this function in future.
4. Changes in views and tests to call this function in this changed
   format.

Changes were tested using ./tools/test-backend.

Fixes #17111.
2021-02-05 17:31:45 -08:00
Suyash Vardhan Mathur 26a81ab3aa api docs: Display data type of responses in API Documentation.
Previously, the data type of responses wasn't displayed in the API
Documentation, even though that OpenAPI data is carefully validated
against the implementation. Here we add a recursive function to
render the data types visibly in API Documentation.
Fixes part of #15967.
2021-02-05 10:41:42 -08:00
Anders Kaseorg ae0afa2390 markdown: Explode config dict.
Commit 434094e599 (#11321) changed this
from an Extension to a subclass of Markdown, so it no longer has any
reason to use a config dict structured like that of an Extension.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-05 10:52:31 -05:00
Aman Agrawal b26727ed16 invite-new-users: Specify that the limit spans for the whole day. 2021-01-29 09:51:11 -08:00
Ganesh Pawar a42f7a67e1 populate_db: Add images in test data.
This isn't quite the right model, because we're not actually going
through the upload code path, but it does at least provide some inline
image previews in the data.

Fixes part of #14991.
2021-01-27 17:52:28 -08:00
Anders Kaseorg 4ca66e7278 timezone: Correct common_timezones dictionary.
The changes are as follows:

• Fix one day offset in all western zones.
• Correct CST from -64800 to -21600 and CDT from -68400 to -18000.
• Disambiguate PST in favor of -28000 over +28000.
• Add GMT, UTC, WET, previously excluded for being at offset 0.
• Add ACDT, AEDT, AKST, MET, MSK, NST, NZDT, PKT, which the previous
  code did not find.
• Remove numbered abbreviations -12, …, +14, which are unnecessary.
• Remove MSD and PKST, which are no longer used.

Hardcode the dict and verify it with a test, so that future
discrepancies won’t go silently unnoticed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-27 15:23:15 -08:00
Anders Kaseorg a7bd1f8049 requirements: Upgrade Python requirements. 2021-01-26 13:27:50 -08:00
Anders Kaseorg c0ad595855 email_notifications: Fix HTML injection bug.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-26 13:27:50 -08:00
Anders Kaseorg c36a66cc1b redis_utils: Convert percent formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-26 13:27:22 -08:00
Mateusz Mandera 1432067959 dependencies: Upgrade to Django 3.1.
https://docs.djangoproject.com/en/3.1/releases/3.1/

- django.contrib.postgres.fields.JSONField is deprecated and should be
  replaced with models.JSONField
-  The internals of the implementation in the postgresql backend have
   changed a bit in
   f48f671223
   and thus we need to make an ugly tweak in test_runner.
- app_directories.Loader.get_dirs() now returns a list of PosixPath so
  we need to make a small tweak in TwoFactorLoader for that (PosixPath
  is not iterable)

Fixes #16010.
2021-01-26 10:20:00 -08:00
akshatdalton 5f8a10124e url preview: Update Zulip User-Agent.
This commit updates the Zulip User-Agent to
'Mozilla/5.0 (compatible; ZulipURLPreview/{version}; +{external_host})'
as the older User-Agent was rendering Markdown YouTube titles as
'YouTube - YouTube'.

Fixes #16970.
2021-01-25 14:24:48 -08:00
Alex Vandiver 3381fad258 registration: Stop enqueueing to the signups queue.
c2526844e9 removed the `signups` queue
worker, and the command-line tool that enqueues to it -- but not the
automated process that enqueues during signups itself.

Remove the signup, since it is no longer in use.
2021-01-24 09:42:55 -08:00
Suyash Vardhan Mathur f4cf5166bb api docs: Display data type of parameters in API documentation.
Previously, the data type of parameters wasn't displayed in the API
Documentation, even though that OpenAPI data is carefully validated
against the implementation.  Here we add a recursive function to
render the data types visibly in the API documentation.

This only covers the request parameters; we'll want to do something
similar for response parameters in a follow-up PR.

Fixes part of #15967.
2021-01-21 15:56:07 -08:00
Steve Howell 1498b2ef69 apply_event: Fix broken deepcopy attempt for subs.
When we were getting an apply_event call for
a subscription/add event, we were trying not to
mutate the event itself, but this clumsy code
was still mutating the actual event:

    # Avoid letting 'subscribers' entries end up in the list
    for i, sub in enumerate(event['subscriptions']):
        event['subscriptions'][i] = \
            copy.deepcopy(event['subscriptions'][i])
        del event['subscriptions'][i]['subscribers']

This is only a theoretical bug.

The only person who receives a subscription/add
event is the current user.

And it wouldn't have affected the current user,
since the apply_event was correctly updating the
state, and we wouldn't actually deliver the event
to the client (because the whole point of apply_event
is to prevent us from having to piggyback the
super-recent events on to our payload or put
them into the event queue and possibly race).

The new code just cleanly makes a copy of each
sub, if necessary, as we add them to state["subscriptions"].

And I updated the event schemas to reflect that
subscribers is always present in subscription/add
event.

Long term we should probably avoid sending subscribers
on this event when the clients don't set something
like include_subscribers.  That's a fairly complicated
fix that involves passing in flags to ClientDescriptor.
Alternatively, we could just say that our policy is
that we never send subscribers there, but we instead
use peer_add events.  See issue #17089 for more
details.
2021-01-21 15:04:07 -08:00
Steve Howell c6acde9c63 apply_event: Use stream_ids, not names, for add/remove.
It's always cleaner to work in id space.  It probably
would have required a perfect storm to have broken
the existing code, but using ids is obviously more
robust in theory, and just as simple.
2021-01-21 15:04:07 -08:00
Steve Howell 0519f2d2b9 minor: Move include_subscribers guards in apply_event.
This sets us up for a cleaner diff in an
upcoming commit.
2021-01-21 15:04:07 -08:00
Steve Howell 3fa595ef85 minor: Clean up args for apply_event.
We now require keywords, so that there is no
pitfall for mixing up boolean parameters.
Positional parameters are basically evil
when you have a bunch of bools.

I also make user_profile the first argument.

Finally, the code is more diff-friendly.
2021-01-21 15:04:07 -08:00
Steve Howell e42baf9e13 minor: Clean up args for apply_events.
I eliminate the defaults, since the existing code
was already specificying values for most things.

I move all the booleans to the bottom for both
parameters and arguments.

I require explicit keywords for everything but
user_profile (which is now first).

And, finally, I format the code in a more
diff-friendly manner.
2021-01-21 15:04:07 -08:00
Steve Howell f2586d2f9b refactor: Introduce SubscriptionInfo dataclass.
We use this as the return type for
gather_subscriptions_helper and
get_web_public_subs, instead of tuples.
2021-01-21 15:04:07 -08:00
Steve Howell 768117f0ff refactor: Unify include_subscribers logic. 2021-01-21 15:04:07 -08:00
Steve Howell e735ce3f01 refactor: Move subscribers logic up to caller.
The gather_subscriptions_helper function now updates
subscribers instead of delegating.
2021-01-21 15:04:07 -08:00
Steve Howell d9740045a5 refactor: Eliminate checks in build_stream_dict_for_sub.
We eliminate some redundant checks.

We also consistently provide a `subscribers` field
in our stream data with `[]`, even if our users
can't access subscribers.  We therefore bump
the API version and tweak the docs.  (See further
down for a detailed justification of the change.)

Even though it is sometimes fine to have redundant code
that is defensive in nature, some upcoming changes are gonna
move subscriber-related logic out of build_stream_dict_for_sub
for certain codepaths as part of our effort to streamline
the payload for subscribers within page_params.

So we can't rely on the code that I removed here
inside of build_stream_dict_for_sub.

Anyway, it makes more sense to do these checks explicitly
in the validate function.

The code in build_stream_dict_for_sub was almost effectively
a noop, since the validation function was already preventing
us from getting subscriber info.  The only difference it
made was sometimes converting `[]` to `None`, and then
subsequently omitting the subscribers field.

Neither ZT nor the webapp make any distinction between
`[]` or <missing key> for the `subscribers` data in
`page_params`.

The webapp has had this code for a long time (and now
equivalent code elsewhere in this PR):

    if (!Object.prototype.hasOwnProperty.call(sub, "subscribers")) {
        sub.subscribers = new LazySet([]);
    }

The webapp calculates access based on booleans, anyway:

    sub.can_access_subscribers =
        page_params.is_admin || sub.subscribed ||
        (!page_params.is_guest && !sub.invite_only);

And ZT would choke if `subscribers` were missing, except that
it never gets to the relevant code due to other checks:

    def get_other_subscribers_in_stream(<snip>):
        assert stream_id is not None or stream_name is not None

        if stream_id:
            assert self.is_user_subscribed_to_stream(stream_id)

            return [sub
                    for sub in self.stream_dict[stream_id]['subscribers']
                    if sub != self.user_id]
        else:
            return [sub
                    for _, stream in self.stream_dict.items()
                    for sub in stream['subscribers']
                    if stream['name'] == stream_name
                    if sub != self.user_id]

You could make a semantic argument that we should prefer
<missing key> to `[]` when subscribers aren't even available, but
we have precedent from the way that `bulk_get_subscriber_user_ids`
has traditionally populated its result:

    result: Dict[int, List[int]] =
        {stream["id"]: [] for stream in stream_dicts}

If we changed `stream_dicts` to `target_stream_dicts` we
would faciliate a move toward `None`, but it would just cause
headaches for other server code as well as the frontends
(which, to reiterate, already prefer the empty array
for convenience).
2021-01-21 15:04:07 -08:00