Previously, Zulip's server logs would not show which user or client
was involved in login or user registration actions, which made
debugging more annoying than it needed to be.
This is mostly pure code extraction.
It also removes some dead code in update_muted_topic, where
were updating muted_topics spuriously before calling
do_update_muted_topic.
Unlike creating a stream, there's really no reason one would want to
call the function to create a realm while uncertain whether that realm
already existed.
For filters like has:link, where the web app doesn't necessarily
want to guess whether incoming messages meet the criteria of the
filter, the server is asked to query rows that match the query.
Usually these queries are search queries, which have fields for
content_matches and subject_matches. Our logic was handling those
correctly.
Non-search queries were throwing an exception related to tuple
unpacking. Now we recognize when those fields are absent and
do the proper thing.
There are probably situations where the web app should stop hitting
this endpoint and just use its own filters. We are making the most
defensive fix first.
Fixes#6118
This change is mostly based on a similar commit from hackerkid
in a feature branch. It borrows both code and ideas. Some of
it's my own stuff, as I was working on a newer branch.
We now call get_user_including_cross_realm_email() inside of
user_profiles_from_unvalidated_emails(), instead of using
get_user_profile_by_email.
This requires a few of our callers to pass down sender into us.
One consequence of this change is that we change the symptoms
for trying to send to emails outside of your realm. In some
cases, we simply raise an error that an email is invalid to us
instead of getting into the deeper validate_recipient_user_profiles
check.
This class simplifies the calling sequence to methods like
check_message and _internal_prep_message, and it's also more
type safe.
Checking for message types is encapsulated with calls to is_stream()
and is_private(). There are also shortcut constructors when you
know that the type of the address (stream vs. private), which is often.
In this we basically seed a single message for the user which will
be soft deactivated by sending a stream message / group PM to
ensure that is has at least one UserMessage row, since in real
world every human user will always have at least one User Message
row.
Before this change, server searches for both
`is:mentioned` and `is:alerted` would return all messages
where the user is specifically mentioned (but not
at-all mentions).
Now we follow the JS semantics:
is:mentioned -- all mentions, including wildcards
is:alerted -- has an alert word
Here is one relevant JS snippet:
} else if (operand === 'mentioned') {
return message.mentioned;
} else if (operand === 'alerted') {
return message.alerted;
And here you see that `mentioned` is OR'ed over both mention flags:
message.mentioned = convert_flag('mentioned') || convert_flag('wildcard_mentioned');
The `alerted` flag on the JS side is a simple mapping:
message.alerted = convert_flag('has_alert_word');
Fixes#5020
Given typeahed and the fact that this only worked if the person had a
full name that didn't contain whitespace, this side effect of the
original @shortname mentionfeature that we removed was experienced by
users as a bug.
Fixes#6142.
We apparently were using the default of num_before=1, not
num_before=0, which meant that if the very last randomly generated
message was one by cordelia mentioning lunch,
test_get_messages_with_search would fail because there were actually 3
matches.
This adds the authors to the Zulip repository on GitHub from
/authors/ along with re-styling the page to fit the same
aesthetic as /for/open-source/ and other product-pages.
This fixes the significant duplication of code between the
authenticate_log_and_execute_json code path and the `validate_api_key`
code path.
These's till a bit of duplication, in the form of `process_client` and
`request._email` interactions, but it is very minor at this point.
The new endpoints are:
/json/mark_stream_as_read: takes stream name
/json/mark_topic_as_read: takes stream name, topic name
The /json/flags endpoint no longer allows streams or topics
to be passed in as parameters.
This function optimizes marking streams and topics as read,
by using UserMessage.where_unread(), which uses a partial
index on the "read" flag.
This also simplifies the code path for ordinary message
flag updates.
In order to keep 100% line coverage, I simplified the
logging in update_message_flags, so now all requests
will show the "actually" format.
This is an interim step toward creating dedicated endpoints
for marking streams/topics as reads, so we do error checking
with asserts for flag/operation, so we don't introduce a
temporary translation string.
This is the first part of a larger migration to convert Zulip's
reactions storage to something based on the codepoint, not the emoji
name that the user typed in, so that we don't need to worry about
changes in the names we're using breaking the emoji storage.
We recently changed the populate_db data set to include more variable
message content, which happened to include the possibility of the word
"lunch" appearing in the test messages. This caused occasional
failures of the search tests that looked for messages containing
"lunch" starting at the beginning of time, not the beginning of the
test.
This commits adds new helper functions which are:
* get_users_for_soft_deactivation(): This function can be used to
fetch a list of human users which pass the criteria of minimum
inactivity period (in days) passed as a parameter to the function.
* do_soft_activate_users(): Given a list of users this function
reactivates them and help them catch up with the missing message
rows for them in the UserMessage table.
This function will help us in creating undisturbed experience for
returning soft deactivated users.
Tweaked by tabbott to fix minor performance and clarity issues.
We were exiting this function in certain cases before updating
mentions. This bug was always there, but it was flaky in terms
of database setup whether the tests would fail, so now the
relevant test sends three consecutive messages.
We also avoid putting duplicate message ids in mentions.
This should significantly improve the user experience for new users
signing up with GitHub/Google auth. It comes complete with tests for
the various cases. Further work may be needed for LDAP to not prompt
for a password, however.
Fixes#886.
This allows us to go to Registration form directly. This behaviour is
similar to what we follow in GitHub oAuth. Before this, in registration
flow if an account was not found, user was asked if they wanted to go to
registration flow. This confirmation behavior is followed for login
oauth path.
We apparently were not correctly clearing the user_profile's email
address from caches when changing email addresses, which meant that
trying to look up the old email in the user_profile caches would still
work.
Fixes#6035.
The "all" option for 'message/flags' was dangerous, as it could
apply to any of our flags. The only flag it made sense for, the
"read" flag, now has a dedicated endpoint.
This change simplifies how we mark all messages as read. It also
speeds up the backend by taking advantage of our partial index
for unread messages. We also use a new statsd indicator.
This completes the major endpoint migrations to eliminate legacy API
endpoints from Zulip.
There's a few other things that will happen naturally, so I believe
this fixes#611.
interface_type select menu will be used to choose the interface
for outgoing webhooks. It will be displayed only when the selected
bot type is OUTGOING WEBHOOK type. The default value is GENERIC
interface type (1).
We are adding a new list of unread message ids grouped by
conversation to the queue registration result. This will allow
clients to show accurate unread badges without needing to load an
unbound number of historic messages.
Jason started this commit, and then Steve Howell finished it.
We only identify conversations using stream_id/user_id info;
we may need a subsequent version that includes things like
stream names and user emails/names for API clients that don't
have data structures to map ids -> attributes.
In anticipation of have all unread message ids available to the
web app in page_params (via a separate effort), we are simplifying
the /topics endpoint to no longer return unread counts.
Instead we have a list of tiny dictionaries with these fields:
name - name of the topic
max_id - max message id for the topic (aka most recent)
The items in the list are order by most-recent-topic-first.
This route is called only in `js/compose.js`, to handle autosubscribe.
That code doesn't check this "exists" field, because there's no need
-- the same information is already carried in whether the result was
success or failure. So just eliminate it.
This makes the logic here a little simpler. It also eliminates
another usage of the `data` parameter to `json_error`. I have half a
mind to eliminate that parameter, in favor of making `JsonableError`
subclasses whenever there's structured data to include, in particular
to get the benefits of typing. There are a couple of places where
that change isn't locally a clear win, but this is not one of them.
This error isn't saying that any kind of authentication or
authorization failed -- it's just a validation error like
any other validation error in the values the user is asking to
set. The thought of authentication comes into it only because
the setting happens to be *about* authentication.
Fix the error to look like the other validation errors around it,
rather than give a 403 HTTP status code and a "reason" field that
mimics the "reason" fields in `api_fetch_api_key`.
Process the unicode emojis in twitter link previews and render them
properly. Before this we were not processing the unicode emojis in
twitter link previews and hence on the systems which don't have
fonts for displaying them they were rendered as blank boxes.
Fixes: #5427.
Add test to check if the embedded bot service being used is in the
registry or not.
Add test to check if the bot being added to the registry has a valid
bot corresponding to it.
Move 'get_bot_handler' to 'zerver/lib/bot_lib.py' as it is an independent
function, not related to the 'EmbeddedBotWorker' class that it was
previously a part of.
This provides the main infrastructure for fixing #5598. From here,
it's a matter of on the one hand upgrading exception handlers -- the
many except-blocks in the codebase that look for JsonableError -- to
look beyond the string `msg` and pass on the machine-readable full
error information to their various downstream recipients, and on the
other hand adjusting places where we raise errors to take advantage
of this mechanism to give the errors structured details.
In an ideal future, I think all exception handlers that look (or
should look) for a JsonableError would use its contents in structured
form, never mentioning `msg`; but the majority of error sites might
continue to just instantiate JsonableError with a string message. The
latter is the simplest thing to do, and probably most error types will
never have code looking for them specifically.
Because the new API refactors the `to_json_error_msg` method which was
designed for subclasses to override, update the 4 subclasses that did
so to take full advantage of the new API instead.
The whole thing is an error, so "message" is a more apt word for the
error message specifically. We abbreviate that as `msg` in the actual
HTTP responses and in the signatures of `json_error` and friends, so
do the same here.
We do not need to test the exception message being logged in every
test case where an exception is raised by a webhook function.
Testing it once should be enough; this makes the tests less
verbose.
Exception logging within api_key_only_webhook_view fails when
ValueError is raised if the request.body passed to ujson.loads
isn't valid JSON. In this case, we now just convert the payload
to a string and log that. This allows us to inspect JSON payloads
that aren't being decoded properly.
Also adds Confirmation.type, and cleans up the rest of Confirmation to look
more like the model definitions in zerver.
In the migration, all existing confirmations adopt the type
USER_REGISTRATION, to be conservative. In a few commits, different
confirmation types will have different validity periods, and
USER_REGISTRATION will have the shortest default.
ScheduledJob was written for much more generality than it ended up being
used for. Currently it is used by send_future_email, and nothing
else. Tailoring the model to emails in particular will make it easier to do
things like selectively clear emails when people unsubscribe from particular
email types, or seamlessly handle using the same email on multiple realms.
This new setting controls whether or not users are allowed to see the
edit history in a Zulip organization. It controls access through 2
key mechanisms:
* For long-ago edited messages, get_messages removes the edit history
content from messages it sends to clients.
* For newly edited messages, clients are responsible for checking the
setting and not saving the edit history data. Since the webapp was
the only client displaying it before this change, this just required
some changes in message_events.js.
Significantly modified by tabbott to fix some logic bugs and add a
test.
I pushed a bunch of commits that attempted to introduce
the concept of `client_message_id` into our server, as
part of cleaning up our codepaths related to messages you
sent (both for the locally echoed case and for the host
case).
When we deployed this, we had some strange failures involving
double-echoed messages and issues advancing the pointer that appeared
related to #5779. We didn't get to the bottom of exactly why the PR
caused havoc, but I decided there was a cleaner approach, anyway.
We are deprecating local_id/local_message_id on the Python server.
Instead of the server knowing about the client's implementation of
local id, with the message id = 9999.01 scheme, we just send the
server an opaque id to send back to us.
This commit changes the name from local_id -> client_message_id,
but it doesn't change the actual values passed yet.
The goal for client_key in future commits will be to:
* Have it for all messages, not just locally rendered messages
* Not have it overlap with server-side message ids.
The history behind local_id having numbers like 9999.01 is that
they are actually interim message ids and the numerical value is
used for rendering the message list when we do client-side rendering.
Prior to this commit, 7 megabytes of images (through 253 individual requests)
were heavily slowing down the initial load. With this commit, we load only the
logos (60 or so images).
Documentation and images for the individual integration sub-pages is requested
separately using the /integrations/doc/ endpoint, which returns HTML.
Use bool_change if the user_display setting property_type is bool, so that no additional code needs to be added to test_events for new boolean user display settings.
The SubscriptionAPITest class variables `realm` and `test_realm` stores
the same information and are redundant. I have eliminated all occurances
of self.realm and replaced with self.test_realm.
We were using relative URLs for realm emojis in missed message emails.
Since the email server is not same as the Zulip server and doesn't
have the realm emoji files, they were rendered as broken images. This
commit fixes them to use absolute URL.
Fixes: #5692.
The MitUser table was removed in df525ad.
confirm_mituser.html could have been accessed through the last few lines of
confirmation/views.py:
templates.insert(0, 'confirmation/confirm_%s.html'
% (obj._meta.model_name,))
The commit message on df525ad suggests there was another way
confirm_mituser.html could have been called, but I don't currently see
evidence for it in the code.
The Realm.DoesNotExist exception will never be raised if we pass a
string_id as in that case zerver.models.get_realm function would be
used for fetching. realm.zerver.models.get_realm uses filter query
so the exception will not be raised even if there are no realm which
matches the query.
Tests added by tabbott.
This system hasn't been in active use for several years, and had some
problems with it's design. So it makes sense to just remove it to declutter
the codebase.
Fixes#5655.
The change password form http://localhost:9991/#settings/your-account
don't have data-min-length and data-min-quality attributes. The
account_settings.handlebar which has the change password form is
rendered client side. So we have to pass the value of min length
and quality in page params to set the data-min-length and
data-min-quality attributes.
This new library is intended to make it easy for management commands
to access a realm or a user in a realm without having to duplicate any
of the annoying parsing/extraction code.
This fixes a bug where we would previously not validate the format of
APNS tokens before writing them to the database, which could lead to
exceptions in the push notifications system if a buggy mobile app
submitted invalid format tokens.
The refer_friend endpoint is about to be deprecated, so we
test with invite_users instead, and I tried to make the
dummy value returned by the mocked gettext function a little
more clear.
Instead of removing an emoji from the database, just mark them as
deactivated so that they can't be used further but can be rendered
properly in reactions and messages.
Fixes: #4750.
No change in behavior.
Also makes the first step towards converting all uses of
settings.ZULIP_ADMINISTRATOR and settings.NOREPLY_EMAIL_ADDRESS to
FromAddress.*.
Once everything is converted, it will be easier to ensure that future
development doesn't break backwards compatibility with the old style of
settings emails.
This will allow for customized senders for emails, e.g. 'Zulip Digest' for
digest emails and 'Zulip Missed Messages' for missed message emails.
Also:
* Converts the sender name to always be "Zulip", if the from_email used to
be settings.NOREPLY_EMAIL_ADDRESS or settings.ZULIP_ADMINISTRATOR.
* Changes the default value of settings.NOREPLY_EMAIL_ADDRESS in the
prod_setting_template to no longer have a display name. The only use of
that display name was in the email pathway.
Previously, we were checking for "Zulip" entries in User-Agent strings
at the very end of the list, not the beginning. A Zulip User-Agent
should have priority, since it includes the User-Agent strings for a
lot of other browsers, and also it will always be the most specific
value.
Unicode emojis when rendered should display canonical short name.
Similarly, the alt text should be of the format `:<short_name>:`.
For both of these we currently display the actual unicode symbol.
As some systems don't have the fonts necessary for displaying them
properly, they are rendered as empty square blocks. This commit also
ensures that the markup generated for emoji generated by canonical
name and by an unicode emoji is same.
Fixes: #5555.
Once we implement org_type-specific features, it'll be easy to change a
corporate realm to a community realm, but hard to go the other way. The main
difference (the main thing that makes migrating from a community realm to a
corporate realm hard) is that you'd have to make everyone sign another terms
of service.
Currently we only pass headers in the first client_get call but
sometimes the effective request which reaches the view is through
a later call to the client_get in this function. Due to which
headers are not passed.
This isn't very slick, but it should get the main points down,
and it's past time we got something like this up. Definitely
needs in the future another pass at the text, and also some images
(screenshots, etc.) and styling.
We used shortnames for mentioning users before we had autocomplete
feature. Since we now have autocomplete typeahead, this syntax is
no more useful and just causes problems. This commit removes the
shortname mention syntax.
Fixes: #4189.
Previously, the only required field in RegistrationForm was the full
name (and possibly ToS, depending on settings). This meant that if
LDAP was configured, realm creation would break, because the form
would be valid the first time one landed on it, before the user even
filled it out!
The correct fix is to make the extra fields required in
RegistrationForm in the event that we're doing realm creation.
It's possible that a cleaner fix would be to use a subclass.
With a test from Umair Waheed Khan.
Fixes#5387.
A deactivated realm emoji should neither be accepted further as a
reaction nor its further occurences in a message be rendered as an
emoji. However, all the old occurences should continue to render
normally.
Server settings should just be added to the context in build_email, so that
the individual email pathways (and later, the email testing framework)
doesn't have to worry about it.
Realm.notifications_stream is not a boolean, Text or integer field, and
thus doesn't fit into the do_set_realm_property framework. Added function
to update it in actions.py. Altered the view, realm.py, to accept
stream-id. Also, notifications stream can be disabled by sending a
negative id.
Rationale: For the more off-to-the-side edit history view, changes
are easier to digest by highlighting deleted content in red followed
immediately by added and changed content in green.
TODO: Toggle for showing the edited messages without highlighting;
deleted content would not be shown in this view.
Add 'Type of bot' option for bots by adding dropdown option in
settings->"Your bots". For now, this allows creating incoming webhook
bots in addition to default bots.
This will enable users to add a bot as an incoming webhook
(in addition to add full-featured bots).
With various minor tweaks and cleanups by tabbott.
Fixes#2186.
This page describes software the user will get from upstream for
their own devices, independent of what's on the server they're
using. So it should live in a place maintained together with
that other software, rather than be distributed and versioned
with the server.
The use of ZILENCER_ENABLED to tell the difference is rather a hack
but is currently how we do this in the small handful of similar
spots; see #5245.
Fixes#5234.
When the last user on a private stream is removed, the stream is no
longer possible to administer, and thus should be marked as
deactivated, so that default streams entries are removed and it no
longer appears in the UI as a non-administerable broken stream.
Umair Khan
Tim Abbott
Aditya Bansal
Vishnu Ks
Steve Howell
Steve Howell
Vaida Plankyte
Brock Whittaker
Jack Zhang
neiljp (Neil Pilgrim)
Rishi Gupta
Anna Liao
vaibhav
Jason Michalski
Abhijeet Kaur
Harshit Bansal
digi0ps
Greg Price
Elliott Jin
James Rowan
Eeshan Garg
Sarah
Durga Akhil Mundroy
Cory Lynch
Franziska von der Goltz
Pweaver (Paul Weaver)
Yago González
sinwar
Sampriti Panda
kunall17
Aditya Ramesh