Finishes the refactoring started in c1bbd8d. The goal of the refactoring is
to change the argument to get_realm from a Realm.domain to a
Realm.string_id. The steps were
* Add a new function, get_realm_by_string_id.
* Change all calls to get_realm to use get_realm_by_string_id instead.
* Remove get_realm.
* (This commit) Rename get_realm_by_string_id to get_realm.
Part of a larger migration to remove the Realm.domain field entirely.
The new GitHub dispatcher integration was apparently totally broken,
because we hadn't tagged the new dispatcher endpoint as exempt from
CSRF checking. I'm not sure why the test suite didn't catch this.
- Add websocket client to create connection with SockJS websocket server.
It contains callback method to launch after connection setup.
- Add '--websocket' parameter to 'check_send_receive_time' script to
check websocket connection.
- Add testing websocket connection to production installation checking.
- Add cronjob to launch websocket connection nagios test.
This makes it possible for Zulip Nagios monitoring to check for
problems impacting the websockets sending code path, which is what all
web users use.
Update integration to use the latest Google API client.
Move Google Account authorization code to a separate file.
Move relevant files from 'bots/' to 'api/integrations/google/'.
Add documentation for integration.
Removes the dependence on postmonkey, which is a wrapper around
MailChimp API v1.3. MailChimp recommends using their v3.0 API directly,
rather than through a wrapper library.
Recent changes to the API removed Client.register(), and
this change restores the correct API call, although the
codepath this affects is probably ready for eventual
deprecation.
While this may not have created a clear user-facing bug, it seems less
confusing for do_invite_users to only create PreregistrationUser
objects for users who actually received an email invitation.
We only write domain to the session variable in one place,
accounts_home_with_domain, where we check that the domain is valid, that the
domain corresponds to an open realm, and that we are in the non-subdomains
case.
Previously, we were confusingly checking only a subset of the conditions
on reading back the domain in create_preregistration_user, and not checking
any of them when reading back the domain in get_realm_from_request.
Add test about avatar image was uploaded properly or not in
`tests.BotTest.test_add_bot_with_user_avatar` and
`tests.BotTest.test_patch_bot_avatar`.
Add `get_test_image_file()` and `avatar_disk_path()` in
`zerver.lib.test_helpers` and deduplicate some codes.
Fixes#1276.
- Markdow engine is not safe threading as global variable. In some cases
it generates previous markdown templates on concurent requests. It's
enough to generate markdown extensions per process to save time. Also
this solution is thread safe.
Previously, we included a special subscribe button in new stream
notifications, but that had 2 problems:
(1) The subscribe button would render badly if the stream was renamed.
(2) There wasn't an easy way to look at the stream when deciding
whether to subscribe.
This fixes the second problem, but not really the first.
This adds support for only allowing normal users with account age
equal or greater than a "waiting period" threshold to create streams;
this is useful for open organizations that want new members to
understand the community before creating streams.
If create_stream_by_admins_only setting is set to True, only admin users
were able to create streams. Now normal users with account age greater
or equal than waiting period threshold can also create streams.
Account age is defined as number of days passed since the user had
created his account.
Fixes: #2308.
Tweaked by tabbott to clean up the actual can_create_streams logic and
the tests.
First step in cleaning up populate_db.create_streams and
bulk_create.bulk_create_streams. Part of a series of commits to remove
Realm.domain from populate_db.
There is a change in Django 1.10 due to which whenever the password
of the user is changed the session hash changes. This change affects
us because we cache user profile objects and these cached objects need
to be refreshed. However, the signal sent by Django in which objects are
refreshed fails to refresh the cache for Tornado because it uses a
different cache prefix.
Note: Backend tests are not affected because they don't rely on Tornado.
Expose `is_mentioned` in `message` dict which contains
boolean value about our account is mentioned in the message
content or not.
This is already available via `flags`, but it seems worth making this
data point more explicit, given its importance in writing bots.
Fixes#2667.
Adding more additional information about user profile to
`zerver.views.pointer.get_profile_backend`, like `user_id`,
`full_name`, `email`, `is_bot`, `is_admin`, and `short_name` of the
user.
Zulip doesn't previously make use of the standard Django is_staff flag
(in that the Django admin site is disabled), but since conceptually
the /activity page would be part of the Django admin site if we were
using it (i.e. for server-level administrators), it makes sense to key
off of that rather than the previous, fragile, check for the realm
domain name.
Adding a reaction is now a PUT request to
/messages/<message_id>/emoji_reactions/<emoji_name>
Similarly, removing a reaction is now a DELETE request to
/messages/<message_id>/emoji_reactions/<emoji_name>
This commit changes the url and updates the views and tests.
This commit also adds a test for invalid emoji when removing reaction.
This includes making the default stream description setting into a
dict. That is an API change; we'll discuss it in the changelog but it
seems small enough to be OK.
With some small tweaks by tabbott to remove unnecessary backwards
compatibility code for the settings.
Fixes#2427.
This change adds support for displaying inline open graph previews for
links posted into Zulip.
It is designed to interact correctly with message editing.
This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.
By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.
Eventually, we may want to make this manageable via a (set of?)
per-realm settings. E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
This automates including of markdown files under
`templates/zerver/help/` to be tested by `test_public_urls` (in
zerver/tests/test_signup.py).
Fixes#2465.
This can be useful in scenarios where the network doesn't support
websockets. We don't include it in prod_settings_template.py since
it's a very rare setting to need.
Fixes#1528.
A few functions had arguments removed without having their type
annotations updated accordingly. As a result mypy version 0.4.6
thinks that the first type in the annotation is supposed to be
the type of `self`, a new mypy feature which we are not intending
to use here.
This commit adds support for removing reactions via DELETE requests to
the /reactions endpoint with parameters emoji_name and message_id.
The reaction is deleted from the database and a reaction event is sent
out with 'op' set to 'remove'.
Tests are added to check:
1. Removing a reaction that does not exist fails
2. When removing a reaction, the event payload and users are correct
The markdown files under templates/zerver/help/ are technically not
templates in the standard sense, and thus should not be being
checked with this code path.
(We probably do want to add a test to make sure they all render fine,
but that can be its own project.)
Modify backend test of create_streams_if_needed so that the newly
created streams have descriptions.
Modify casperjs test of filling out stream_creation_form so that
the newly created stream has a description.
Fixes: #2428.
- Add base tornado test case class.
- Add test for websocket connection.
- Add test for websocket authentication.
- Add test for sending private message with websocket.
- Add test for sending stream message with websocket.
Fixes#2230
This commit adds the following:
1. A reaction model that consists of a user, a message and an emoji that
are unique together (a user cannot react to a particular message more
than once with the same emoji)
2. A reaction event that looks like:
{
'type': 'reaction',
'op': 'add',
'message_id': 3,
'emoji_name': 'doge',
'user': {
'user_id': 1,
'email': 'hamlet@zulip.com',
'full_name': 'King Hamlet'
}
}
3. A new API endpoint, /reactions, that accepts POST requests to add a
reaction to a message
4. A migration to add the new model to the database
5. Tests that check that
(a) Invalid requests cannot be made
(b) The reaction event body contains all the info
(c) The reaction event is sent to the appropriate users
(d) Reacting more than once fails
It is still missing important features like removing emoji and
fetching them alongside messages.
Refactor list_to_streams and create_streams_if_needed to take a list
of dictionaries, instead of a list of stream names. This is
preparation for being able to pass additional arguments into the
stream creation process.
An important note: This removes a set of validation code from the
start of add_subscriptions_backend; doing so is correct because
list_to_streams has that same validation code already.
[with some tweaks by tabbott for clarity]
Previously, the key prefix was based on the process id due to which
the JS tests couldn't properly flush user profiles from the cache as
our application spans over multiple processes. This problem becomes
apparent when in json_change_settings view after changing the user_profile
the tornado views continue to get the cached user profile corresponding
to their process id.
Clean up the instances of self.assertIn("string", result.content.decode("utf-8")),
and replace them with self.assert_in_response("string").
Fixes: #2313
We are prone to case-sensitivity bugs, so I added AARON and ZOE.
Also, for good measure, I insert them in non-alphabetical order
to try to drive out bugs from non-consistent sorting of user ids.
We now instrument URL coverage whenever you run the back end tests,
and if you run the full suite and fail to test all endpoints, we
exit with a non-zero exit code and report failures to you.
If you are running just a subset of the test suite, you'll still
be able to see var/url_coverage.txt, which has some useful info.
With some tweaks to the output from tabbott.
Fixes#1441.
Previously, we rejected the HEAD requests that the trello integration
uses to check if the server accepts the integration.
Add decorator for returning 200 status code if request is HEAD.
Fixes: #2311.
Django reverts all the changes after running a test but the
client cache retained the deleted value, this caused the
subsequent tests to fail due to invalid foreign key constraints.
This commit fixes the issue by prefixing the cache name in
client cache with KEY_PREFIX which is bounced after every test.
Updates the HTML docs to match changes to the Desk.com website,
including all new screenshots for the custom action workflow.
Tests four types of messages that could be sent as notifications from
Desk.com. Desk.com allows an account administrator to send any text
in a custom action, so there isn't a standard format.
Custom actions send URL-encoded POST data, the test fixtures contain
URL-encoded text like what could be sent by a custom action configured
as described in the Zulip Integrations documentation to post a new
message to a stream. (See also #2169, errors in this documentation.)
New zerver.tests.webhooks.test_deskdotcom.DeskDotComHookTests:
* Static text: minimal plain text string
* Case updated: activity alert with link to a Desk.com case and message
* Unicode text Italian: activity alert with message in Italian
* Unicode text Japanese: activity alert with message in Japanese
Each posts a new message in the deskdotcom stream.
Tested on Ubuntu 14.04. I created the fixtures with Emacs, I would
appreciate if someone can check that the Italian and Japanese messages
look ok. I used the same text for a live test and it displayed correctly.
Fixes#2031
Add is_truncated param for cases when we know that there were more commits
but we dont know actual number and want just inform about existing of the rest.
bulk_create.bulk_create_users is only called by initialize_voyager_db.py and
populate_db.py, both of which only have realms with single domains.
Part of a larger project to remove the Realm.domain field.
No change in behavior. The for_you block had already been removed in
portico.html long ago in a6889080ce. The
contents of the block are still present in the non-portico 404.html
and 5xx.html error pages.
Currently refactoring the Realm and RealmAlias portions of models.py to no
longer use Realm.domain. Move various functions related and not-related to
the two model objects to more conventional places in the file.
No change to behavior.
Previously, if a new message arrived between when a user is subscribed
to the default streams and when the user's initial messages are
queried, we would try to create two UserMessage rows for the same
Message, resulting in an IntegrityError crash. We fix this and add a
test for that race condition.
This migration ensures that all historically uploaded files from
before we started tracking files in the Zulip database via the
Attachment model have Attachment objects.
This has been tested by tabbott against a production server to ensure
that it results in all old uploaded files having corresponding
attachment objects.
Merging this change is a key prerequisite for making our adding
attachment access controls in an enforcing fashion.
In Django 1.10, the get_token function returns a salted version of
csrf token which changes whenever get_token is called. This gives
us wrong result when we compare the state after returning from
Google authentication servers. The solution is to unsalt the token
and use that token to find the HMAC so that we get the same value
as long as t he token is same.
Needed in case the user was allowed to join the realm when they got the
confirmation email, but is no longer allowed to do so. Check was previously
applied to invited users (those with a prereg_user.referred_by), and is now
applied regardless of how they get to accounts_register.
Does change/fix behavior in various corner cases when the domain passed in
to HomepageForm and subdomain passed in to HomepageForm correspond to
different realms.
If the user comes in to HomepageForm with a set subdomain, use that to
determine the signup realm instead of the email address.
In the non-REALMS_HAVE_SUBDOMAINS case, still allow using the email address
if no subdomain is passed.
Ensure domain and subdomain correspond to the same realm when being passed
to forms.HomepageForm. Previously this was not the case when e.g. we got
here via the /register/<domain> endpoint.
This also effectively disables the register/<domain> endpoint when
REALMS_HAVE_SUBDOMAINS, or rather, foo.server.org/register/bar.com will try
to register you for the realm with string_id foo rather than realm with
domain bar.com.
`django.contrib.auth.get_user` function is updated in Django 1.10, due to
which everytime we update the password of the user the password hash changes.
This causes authentication failure. Previously, our code worked correctly
because we use our own session middleware and the `get_user` code had a
conditional statement which allowed our code to bypass the authentication
code.
Previously, the way that render_messages was calling bugdown meant
that the preview feature didn't have access to realm data like the
list of users or streams, resulting in previews for those elements
being wrong.
Now render_message_backend uses zerver.lib.render_markdown to render
messages correctly.
[Commit message tweaked and test added by tabbott]
No change to behavior. non_mit_mailing_list never returned False, so it was
never possible to reach the line "Otherwise, the user is an MIT mailing
list, and .."
send_event() expects a list of user ids (ints) except for the special case
of messages. This commit:
1. Fixes this in the call to send_event() in do_send_typing_notification()
2. Renames the variables in do_send_typing_notification() to better reflect
their content (for example, recipient_ids instead of recipients).
3. Renames the id field in the dicts sent in the typing event body (sender,
recipients) to user_id.
4. Adds assertions to the tests to verify that the tornado event user ids
are the same as the recipients in the event body.
5. Adds assertions to the tests to verify that the tornado event user
ids and the recipient user ids (in the event body) are the same as the
expected user ids (obtained from the emails using
get_user_profile_by_email)
6. Changes all assertTrues to assertEquals in the tests
This fixes#2151.
This makes it possible to configure only certain authentication
methods to be enabled on a per-realm basis.
Note that the authentication_methods_dict function (which checks what
backends are supported on the realm) requires an in function import
due to a circular dependency.
In fe1ba6f3eb, we change our auth
decorators etc. to use request.POST/request.GET rather than the (now
removed request.REQUEST). This broke sending messages via our
websockets codepath, because we failed to update the artificial
requests we generate to use request._post as well.
We recently made it so that a cross-realm bot can only send
messages to one realm at a time. (It can send to a realm
outside of its offical realm, but only one of them.) This
test adds coverage for that.
Disallow Realm.string_id's like "streams", "about", and several hundred
others. Also restrict string_id's to be at least 3 characters long, and only
use characters in [a-z0-9-].
Does not restrict realms created by the create_realm.py management command.
Previously realm_name and realm_subdomain defalted to None, which when
posted to /accounts/register, are submitted as u'None'. "None" is an invalid
Realm.subdomain, since subdomains can't have capital letters.
Before it was in UserSignUpTest, now it is in RealmCreationTest. The diff
makes it look like test_user_default_language is the target of the move,
but it isn't.
If a stream is public, we now send notifications to all realm users
if the name or description of the stream changes. For private
streams, the behavior remains the same.
We do this by introducing a method called
can_access_stream_user_ids().
(showell helped with this fix)
Fixes#2195
We use the queries_captured() context manager in our tests
to capture queries that happen during certain actions. Usually
we use the list of queries to validate that we're not doing
too many database calls.
For most tests the database calls are fairly deterministic,
but SAVEPOINT-related things seem to be more random, and the
number of savepoints is usually not relevant to what the test
is trying to prevent, which is more serious problems like
O(N) database fetches.
This (1) removes the check on whether the domain of the email matches
the Realm.domain of an existing realm and (2) avoids setting `realm =
get_realm(domain)` in the realm creation flow, which would cause the
wrong code path to be followed in the event that the domain in a
user's email address happens to match a deactivated realm.
Removes the uniqueness constraint on RealmAlias.domain, and adds a function
can_add_alias that checks for uniqueness conditional on
settings.REALMS_HAVE_SUBDOMAINS.
Previously, we set restrict_to_domain and invite_required differently
depending on whether we were setting up a community or a corporate
realm. Setting restrict_to_domain requires validation on the domain of the
user's email, which is messy in the web realm creation flow, since we
validate the user's email before knowing whether the user intends to set up
a corporate or community realm. The simplest solution is to have the realm
creation flow impose as few restrictions as possible (community defaults),
and then worry about restrict_to_domain etc. after the user is already in.
We set the test suite to explictly use the old defaults, since several of
the tests depend on the old defaults.
This commit adds a database migration.
This test seems intended to verify registration in the case of a
unique completely open domain; but because of the mit.edu realm, it
instead tested that a logic bug in the non-subdomains case was
present.
We now send dictionaries for cross-realm bots. This led to the
following changes:
* Create get_cross_realm_dicts() in actions.py.
* Rename the page_params field to cross_realm_bots.
* Fix some back end tests.
* Add cross_realm_dict to people.js.
* Call people.add for cross-realm bots (if they are not already part of the realm).
* Remove hack to add in feedback@zulip.com on the client side.
* Add people.is_cross_realm_email() and use it in compose.js.
* Remove util.string_in_list_case_insensitive().
In preparation for a change to do_create_realm where we will use the
database default for restricted_to_domain rather than computing it within
do_create_realm, and due to which do_create_realm will no longer know
whether we are creating an open realm or not.
Step 0 of a two step process:
1. Replace all occurances of get_realm(domain) with
get_realm_by_string_id(string_id)
2. Rename get_realm_by_string_id to get_realm.
Adds a database migration, adds a new string_id argument to the management
realm creation command, and adds a short name field to the web realm
creation form when REALMS_HAVE_SUBDOMAINS is False.
Does a database migration to rename Realm.subdomain to
Realm.string_id, and makes Realm.subdomain a property. Eventually,
Realm.string_id will replace Realm.domain as the handle by which we
retrieve Realm objects.
We now simply exclude all cross-realm bots from the set of emails
under consideration, and then if the remaining emails are all in
the same realm, we're good.
This fix changes two behaviors:
* You can no longer send a PM to an ordinary user in another realm
by piggy-backing a cross-realm bot on to the message. (This was
basically a bug, but it would never manifest under current
configurations.)
* You will be able to send PMs to multiple cross-realm bots at once.
(This was an arbitrary restriction. We don't really care about this
scenario much yet, and it fell out of the new implementation.)
We can currently send a PM to a user in another realm, as long
as we copy a cross-realm bot from the same realm. This loophole
doesn't yet affect us in practice--all cross-realm bots are
generally configured for the "admin" realm like the old zulip.com--
but we should lock it down in a subsequent commit.
Having each condition in a separate test was confusing to read,
especially since the tests were doing inconsistent setup, sometimes
calling user2 the user from 2.example.com realm and other times
calling user2 the cross-bot realm, etc.
Note that we still need the equivalent function in our
user-facing API, so there is not much code removal yet.
(Also, we will probably always keep this in our API,
as bot authors will usually just want a simple endpoint
here, whereas our client code gets page_params and events.)
Previously, we used to create one Google OAuth callback url entry
per subdomain. This commit allows us to authenticate subdomain users
against a single Google OAuth callback url entry.
- To avoid redefining migrate manage command is added new application
configuration class which emit post_migration signal. This signal
require models module inside application and defined AppConfig
Instance as signal sender. Documentation here:
https://docs.djangoproject.com/en/1.8/ref/signals/#post-migrate.
- Add AppConf subclass to __init__ zerver app file to make apllication
load it by default.
Fixes#1084.
For some reason, we use 'load' function but it doesn't exist in the JWT
library code. This commit updates the code to use the correct interface
of the JWT library.
The signature verification is done by the decode function.
This is a first step towards implementing a message retention policy
feature.
- Add Realm model message_retention_days field to setup
messages expired period for realm.
- Add migration.
- Add tool to get expired messages for each Realm.
- Add tests to cover tool for getting expired messages.
Now that we have updated python-markdown, we remove the deprecated
safe_mode. We used safe_mode to escape raw html, so now instead we
pass in an EscapeHtml markdown extension to the markdown engine.
See https://pythonhosted.org/Markdown/release-2.6.html for details on
the deprecation.
Fixes: #2037 (also addresses the remaining piece of #2043).
Adds additional tests in bugdown-data.json to check that we escape raw
html properly with script tags.
Many of these tests are based on the bleach "advanced" test suite.
Passes the allowed domains for a realm to the frontend, via
page_params.domains. Groundwork for allowing users to add and
remove domains via the admin setting page, rather than via the
realm_alias.py management command.
This is a preliminary step towards eliminating the realm.domain field
in favor of realm.subdomain. Includes a database migration to create
these for existing realms.
This adds a medium (500px) size avatar thumbnail, that can be
referenced as `{name}-medium.png`. It is intended to be used on the
user's own settings page, though we may come up with other use cases
for high-resolution avatars in the future.
This will automatically generate and upload the medium avatar images
when a new avatar original is uploaded, and contains a migration
(contributed by Kirill Kanakhin) to ensure all pre-existing avatar
images have a medium avatar.
Note that this implementation does not provide an endpoint for
fetching the medium-size avatar for another user.
[substantially modified by tabbott]
When we added data on never_subscribed streams to what
populate_subscribers is called on, we failed to add the corresponding
data on subscribers to email_dict, the mapping of user IDs to emails
for the subscribers.
Because in the Zephyr world, stream names can be a secret, and also
Zephyr mirroring tends to involve many thousands of streams, we
shouldn't send this data.
This is some of the code we'd need if we wanted to have Zulip generate
avatars for things. Since it is so little useful code, and it's not
clear we will need this feature ever, we can remove this code to make
the codebase less confusing. It'd be easy to dig this out of history
if we ever want it.
Fixes#2101.
- Add tests for SEND_MISSED_MESSAGE_EMAILS_AS_USER is False (the
default!).
- Reorganized test case code by removing repeated parts of code,
improving code style and moving common parts to separate class
methods.
Fixes#1697.
Steve Howell
Bojidar Marinov
Jackson
Vamshi Balanaga
Tomasz Kolek
Rishi Gupta
Umair Khan
Tim Abbott
anirudhjain75
K.Kanakhin
Prabod Rathnayaka
Juan Verhook
Harshit Bansal
paxapy
Igor Tokarev
Robert Hönig
Rafid Aslam
JefftheBest1
Sampriti Panda
Tommy Ip
vaibhav
Kracekumar R
reyha
reallyly
Jason Le
Zac Pullar-Strecker
Arpith Siromoney
Tomonori Murakami
Joy Chen
root
Bickio
Mikebarson
sidhant bhavnani
bulat22101
Noel Tautges
AZtheAsian
Reid Barton
nikolay
Sidhant Bhavnani
Alex Huang
sylvan1
Calvin Lee
Anders Kaseorg
Vladislav Manchev
hackerkid
Pweaver (Paul Weaver)
Tim Abbott
Tomasz Kolek
Feorlen
sinwar
Mohsen Ibrahim
Meena Rajan
umkay
Kevin Chen
Rishi Gupta
Brock Whittaker