Commit Graph

121 Commits

Author SHA1 Message Date
Tim Abbott 8f7a86f5ec events: Move remove_message_id_from_unread_mgs to message.py.
It belongs there, next to the related code for adding an event.
2019-08-02 17:26:13 -07:00
Roman Godov a50824e031 models: Rename Subscription.in_home_view field to is_muted.
This renames Subscription.in_home_view field to is_muted, for greater
clarity as to what it does just from seeing the setting name, without
having to look it up.

Also disabled an obsolete test_migrations test.

Fixes #10042.
2019-05-12 22:08:10 -07:00
Tim Abbott 4c3c669b41 mobile: Add support for recent_private_conversations API.
This adds experimental support in /register for sending key
statistical data on the last 1000 private messages that the user is a
participant in.  Because it's experimental, we require developers to
request it explicitly in production (we don't use these data yet in
the webapp, and it likely carries some perf cost).

We expect this to be extremely helpful in initializing the mobile app
user experience for showing recent private message conversations.

See the code comments, but this has been heavily optimized to be very
efficient and do all the filtering work at the database layer so that
we minimize network transit with the database.

Fixes #11944.
2019-04-04 16:47:43 -07:00
Bennet Sunder 7c5f316cb8 alert_words: Performance improvements in looking for alert_words.
This commit leverages the ahocorasick algorithm to build a set of user_ids
that have their alert_words present in the message. It runs in linear time
of the order of length of the input message as opposed to number of
alert_words. This is after building a ahocorasick Automaton which runs
in O(number of alert_words in entire realm) which is usually cached.
2019-03-01 15:36:39 -08:00
Greg Price b0a84cd7ab message: Add an O(1)-query variant of bulk_access_messages.
We'll use this in the push-notifications code, in a context where
there should definitely already be UserMessage rows if everything's
gone normally... but explicitly checking at the top seems like the
right pattern from a secure-coding perspective.
2019-02-26 16:41:54 -08:00
Anders Kaseorg f0ecb93515 zerver core: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:41:24 -08:00
Anders Kaseorg 4e0a79d146 test_messages: Don’t mock an unused import.
Since 8a1794caa3, we don't do any
caching in that system.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:29:53 -08:00
Vishnu Ks 8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Steve Howell 226a2826b5 Remove "subject" from zerver/lib/message.py.
We also lock this down file in the linter, so
that you can no longer introduce "subject"
to `message.py`.
2018-11-08 14:45:16 +00:00
Steve Howell 592dd290b2 Avoid "subject" in MessageDict. 2018-11-08 14:33:44 +00:00
Steve Howell 06f8d0af7e refactor: Extract do_render_markdown().
For the import-data codepath, we will call
the extracted function directly in a
subsequent commit.

The do_render_markdown() function has more
required parameters, which allows for more
explicit code and also allows us to flatten
out some logic related to alert words.  (We
just pass in empty sets/dicts as needed).
2018-11-07 10:20:14 -08:00
Steve Howell 35e9e5928f render: Upstream calculation of translate_emoticons. 2018-11-07 10:20:14 -08:00
Steve Howell 32232377f7 Rename bugdown.subject_links -> topic_links. 2018-11-07 10:03:53 -08:00
Tim Abbott 887d20795f message flags: Add where_starred helper and use it.
The previous query ended up doing a scan of all a user's UserMessage
rows, not just the ones tracked in the `starred` index.
2018-08-21 15:28:04 -07:00
Joshua Pan 83d8d662dc events: Query starred message ids in fetch_initial_state_data. 2018-08-21 13:01:41 -07:00
Rhea Parekh 26300110ca import: Fix rendered_content in imported messages.
After the messages have been imported, set the rendered_content of the
messages instead of leaving its value to be 'None'.

This is important to ensure that:
(1) Performance for users is good after completing the import.
(2) The database's full-text indexes have all of the imported messages
(which only happens properly when Message rows have their
rendered_content field edited).

Fixes #9168.
2018-08-09 15:12:53 -07:00
Tim Abbott f10e006135 message: Bump MAX_UNREAD_MESSAGES to 50000.
This doesn't seem to have a huge performance downside (less than 1s
extra time for loading / on chat.zulip.org), and it means the
possibility of users having so many unreads that we get weird/buggy
behavior is much more unlikely to exist.

We'll still want a better experience for users who somehow go over
this limit, but it can be pretty firmly "you need to go mark some
things as read".
2018-08-01 12:02:54 -07:00
Shubham Dhama c5d8fdf68c message: Add function to check message access in bulk.
This effectively just calls access_message in a loop.
2018-07-28 15:12:55 -07:00
Shubham Dhama 605a6ed3d8 message: Extract function for message access by a user. 2018-07-28 14:58:36 -07:00
Shubham Dhama 9cf67a5e71 models: Add function to get UserMessage from message id. 2018-07-28 14:58:36 -07:00
Tim Abbott bf9012a559 message: Do future-proofing for public streams with private history.
This should have no effect for now, but it'll make things a bit
simpler in case we make future changes to support public streams
without history public to subscribers (and other organization
members).
2018-05-28 12:25:57 -07:00
Shubham Dhama b410aeb897 message: Limit message access for guest users.
Tweaked by tabbott to interact correctly with the
history_public_to_subscribers feature.
2018-05-28 12:20:32 -07:00
jkiely 104fdd8bf9 mypy: enable strict optional for lib/message.
Add assert in order to pass checks under strict conditions.
2018-05-17 11:12:08 -07:00
Steve Howell 4332fd64f7 Add submessages to message payloads. 2018-05-16 15:13:33 -07:00
Aditya Bansal a68376e2ba zerver/lib: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Tim Abbott 53e47e6991 messages: Modify access_message for is_history_public_to_subscribers.
This completes the Message side of #2745.
2018-04-04 16:18:47 -07:00
Nikhil Kumar Mishra 3b4ff4f75c markdown: Clean render_markdown.
Remove Message is None and not None case as the
render_message_backend passes a fake Message object.
2018-03-19 10:36:26 -07:00
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
Vishnu Ks a11b742984 messages: Calculate value of first visible message ID using cron job.
[greg: Fixed buggy time conversion in estimate_recent_messages.]
2018-01-24 17:15:08 -08:00
Vishnu Ks 036dc53d20 messages: Rename last_visible_message_id to first_visible_message_id. 2018-01-22 19:53:44 -08:00
Vishnu Ks e6d3f8895f messages: Create function to calculate first visible message id. 2018-01-04 08:52:40 -05:00
Vishnu Ks 277d463d23 messages: Add option to limit users message history. 2018-01-02 13:45:01 -05:00
rht 3f4bf2d22f zerver/lib: Use python 3 syntax for typing.
Extracted from a larger commit by tabbott because these changes will
not create significant merge conflicts.
2017-11-21 20:56:40 -08:00
Steve Howell f54146b6d8 mypy: Add RawUnreadMessagesResult type. 2017-11-10 10:50:57 -08:00
Steve Howell c799ec0ca6 minor: Inline get_unread_message_ids_per_recipient().
This method was only used in one test.
2017-11-10 10:50:57 -08:00
Steve Howell 8eaeba8615 refactor: Remove dict_with_str_keys().
This function is no longer needed in python3, as it was decoding
keys that already came for a JSON-decoded string.
2017-11-09 10:43:20 -08:00
Steve Howell 63024b3a6f Use encode() in zerver/lib/message.py. 2017-11-09 10:38:58 -08:00
rht 5ee40bf718 Remove usage of six.moves.binary_type. 2017-11-09 10:00:00 -08:00
Harshit Bansal 65838bb825 email_gateway: Disable code block processor for email gateway.
Generally emails are not written with markdown in mind and hence
sometimes render in strange ways. This commit fixes a particular
issue that was causing whitespace before paragraphs to be treated
as code block due to which email content was being rendered in a
box that scrolls in right direction a lot.

Fixes: #7045.
2017-11-09 09:56:35 -08:00
Steve Howell ae0b27a7ed Extract messages_for_ids. 2017-11-07 17:48:27 -08:00
Steve Howell 88e1e284bb Restructure send-message code for gravatars.
This refactoring doesn't change behavior, but it sets us up
to more easily handle a register setting for `client_gravatar`,
which will allow clients to tell us they're going to compute
their own gravatar URLs.

The `client_gravatar` flag already exists in our code, but it
is only used for Django views (users/messages) but not for
Zulip events.

The main change is to move the call to `set_sender_avatar` into
`finalize_payload`, which adds the boolean `client_gravatar`
parameter to that function.  And then we update various callers
to supply that flag.

One small performance benefit of this change is that we now
lazily compute the client message payloads in
`event_queue.process_message_event` now, so this will improve
performance if all interested clients have the same value of
`apply_markdown`.  But the change here is really preparing us
for the additional boolean parameter, which will cause us to
have four variations of the payload.
2017-11-07 10:36:02 -08:00
rht e311842a1b zerver/lib: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
neiljp (Neil Pilgrim) fed757452c mypy: Clarify type of lookup_dict in aggregate_message_dict. 2017-11-04 19:47:44 -07:00
rht fef7d6ba09 zerver/lib: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-03 15:34:37 -07:00
Umair Khan 636046aec9 user-groups: Add basic backend for UserGroup model.
This adds the data model and bugdown support for the new UserGroup
mention feature.

Before it'll be fully operational, we'll still need:
* A backend API for making these.
* A UI for interacting with that API.
* Typeahead on the frontend.
* CSS to make them look pretty and see who's in them.
2017-10-31 15:16:14 -07:00
Steve Howell b3192d17ab refactor: Extract get_stream_subscriptions_for_user(). 2017-10-29 18:36:35 -07:00
Steve Howell 8ac26dfb9b refactor: Introduce bugdown.MentionData class.
We now have a MentionData class that encapsulates
the users who are possibly mentioned in a message.

Not that the rendering code may not keep all the mentions,
since things like backticks will suppress the mention.

We populate this now in do_send_messages, so that we can use
the info earlier in the message-sending process.  This info
now gets passed down the call stack as an optional parameter.

Note that bugdown.convert() still populates the data when its
callers decline to pass in a MentionData object.

This is mostly a preparatory commit, as we don't take advantage
of the data yet in do_send_messages.
2017-10-26 22:16:47 -07:00
Tim Abbott 8e2cdedf9a lint: Fix lines in Python codebase longer than 120 characters. 2017-10-26 17:47:30 -07:00
Steve Howell 635675fe48 Reduce queries needed for sending messages.
In do_send_messages, we only produce one dictionary for
the event queues, instead of different flavors for text
vs. html.  This prevents two unnecessary queries to the
database.

It also means we only put one dictionary on the "message"
event queue instead of two, albeit a wider one that has
some values that won't be sent to the actual clients.

This wider dictionary from MessageDict.wide_dict is also
used for the `feedback_messages` queue and service bot
queues.  Since the extra fields are possibly useful down
the road, and they'll just be ignored for now, we don't
bother to remove them.  Also, those queue processors won't
have access to `content_type`, which they shouldn't need.

Fixes #6947
2017-10-26 16:35:28 -07:00
Steve Howell 9b6a4d0b16 refactor: Extract MessageDict.finalize_payload(). 2017-10-26 16:35:28 -07:00