Commit Graph

3390 Commits

Author SHA1 Message Date
Arpith Siromoney fbe8171813 Typing indicators backend: Pass list of user ids to send_event
send_event() expects a list of user ids (ints) except for the special case
of messages. This commit:
1. Fixes this in the call to send_event() in do_send_typing_notification()
2. Renames the variables in do_send_typing_notification() to better reflect
their content (for example, recipient_ids instead of recipients).
3. Renames the id field in the dicts sent in the typing event body (sender,
recipients) to user_id.
4. Adds assertions to the tests to verify that the tornado event user ids
are the same as the recipients in the event body.
5. Adds assertions to the tests to verify that the tornado event user
ids and the recipient user ids (in the event body) are the same as the
expected user ids (obtained from the emails using
get_user_profile_by_email)
6. Changes all assertTrues to assertEquals in the tests

This fixes #2151.
2016-11-07 22:28:09 -08:00
Meena Rajan 30acb23314 bugdown: Add support for strikethrough in markdown processor.
[Tweaked to move tests to bugdown_data.json, add additional tests, and
add frontend processor support by tabbott]
2016-11-07 22:26:38 -08:00
Rishi Gupta 6e9e2cc05f Change datetime.now() to timezone.now() in various tests.
Good practice to use timezone-aware datetime objects unless there is a
reason to do otherwise.
2016-11-07 20:13:53 -08:00
Rishi Gupta 9e6e1a1e69 Remove several instances of datetime.datetime.utcnow().
Change to the timezone-aware django.utils.timezone.now() where django is
available.
2016-11-07 20:13:53 -08:00
Rishi Gupta 9e5ec2fd29 notifications.send_future_email: Use timezone aware times.
Previous behavior raises an AmbiguousTimeError during daylight savings hour!
2016-11-07 20:13:53 -08:00
Rishi Gupta a416a9aeae test_runner.py: Fix "Import unexpectedly succeeded" messages in tests.
Previously we were getting this message whether or not the import would have
succeeded. The regression was introduced in ea050d5f.
2016-11-07 20:13:17 -08:00
umkay ab2fd4ef3a TestAdminSetBackends: Supply dev_auth_enabled() with realm argument.
This fixes the fact that these tests were not correctly running
against the actual realm.
2016-11-07 17:07:05 -08:00
Kevin Chen 6107c877e8 bugdown: Add option to support "file:///" as hyperlink.
This contains contributions from Tim Abbott and Igor Tokarev.

Fixes #380.
2016-11-06 22:30:05 -08:00
Igor Tokarev d0f37b4c88 text_upload: Check only size in avatar test.
This fixes an issue that made it unpleasant to upgrade the version of
PIL used in Zulip.

Fixes: #1414
2016-11-06 20:49:11 -08:00
Tim Abbott 11518d9fb3 Remove test_ldap_auth_email_auth_disabled_failure for now.
This test was having order-dependent failures when run after
`test_signup`.
2016-11-06 18:34:22 -08:00
Tim Abbott bb17e575c8 auth: Add SocialAuthMixinTest. 2016-11-06 18:07:52 -08:00
Tim Abbott c856e56afc test_auth_backend: Add missing LDAP tests in FetchAPIKeyTest. 2016-11-06 18:07:52 -08:00
Tim Abbott c94fe5bdef test_ldap: Fix spelling of tearDown method. 2016-11-06 16:33:20 -08:00
umkay f4c621ffe3 admin: Enable admins to toggle supported auth methods via UI.
Add a table to the administration page that will allow realm admins to
activate and deactivate the supported authentication methods for that
realm.
2016-11-06 16:29:35 -08:00
umkay 21c024fc29 auth: Make supported authentication backends a bitfield on realm.
This makes it possible to configure only certain authentication
methods to be enabled on a per-realm basis.

Note that the authentication_methods_dict function (which checks what
backends are supported on the realm) requires an in function import
due to a circular dependency.
2016-11-06 16:16:24 -08:00
Tim Abbott b41c15fa05 auth: Reject authentication if auth backends are disabled. 2016-11-06 16:16:22 -08:00
Tim Abbott f8bb55f9c1 auth: Refactor auth backend enabled checking code. 2016-11-06 16:16:12 -08:00
Tim Abbott 3a3cee411d auth: Remove old password_auth_enabled hack.
This was used by an old configuration for zulip.com, which is no
longer in production use.
2016-11-06 14:50:15 -08:00
Tim Abbott be3e9e3a05 Fix regression breaking websocket message sending.
In fe1ba6f3eb, we change our auth
decorators etc. to use request.POST/request.GET rather than the (now
removed request.REQUEST).  This broke sending messages via our
websockets codepath, because we failed to update the artificial
requests we generate to use request._post as well.
2016-11-05 11:33:04 -07:00
Steve Howell 75dd822d8e tests: Improve coverage of cross-realm bot limitations.
We recently made it so that a cross-realm bot can only send
messages to one realm at a time.  (It can send to a realm
outside of its offical realm, but only one of them.)  This
test adds coverage for that.
2016-11-05 10:39:32 -07:00
Rishi Gupta 6544c756d6 forms.py: Reorder imports. 2016-11-05 10:37:15 -07:00
Rishi Gupta 5e6d4da8a8 realm creation flow: Prevent disposable email signups.
Check user's email address against a list of 2000 disposable email
domains. Does not affect the create_realm management command.
2016-11-05 10:37:15 -07:00
Rishi Gupta 2033381d24 realm creation flow: Restrict subdomains one can create via the web flow.
Disallow Realm.string_id's like "streams", "about", and several hundred
others. Also restrict string_id's to be at least 3 characters long, and only
use characters in [a-z0-9-].

Does not restrict realms created by the create_realm.py management command.
2016-11-05 10:37:15 -07:00
Rishi Gupta e3646f28bd test_helpers.py: Fix default args for submit_reg_form_for_user.
Previously realm_name and realm_subdomain defalted to None, which when
posted to /accounts/register, are submitted as u'None'. "None" is an invalid
Realm.subdomain, since subdomains can't have capital letters.
2016-11-04 22:01:53 -07:00
Rishi Gupta 950831810b test_signup.py: Refactor test_create_realm_with_subdomain.
Refactor test_create_realm_with_subdomain to match test_create_realm more
closely, so that it's easier to see their similarities and differences.
2016-11-04 22:01:53 -07:00
Rishi Gupta 4db10a05de test_signup.py: Move test_create_realm_with_subdomain to correct place.
Before it was in UserSignUpTest, now it is in RealmCreationTest. The diff
makes it look like test_user_default_language is the target of the move,
but it isn't.
2016-11-04 22:01:53 -07:00
Tim Abbott 51fcf1fb7a tests: Fix spelling of test_transifex.py. 2016-11-04 21:49:39 -07:00
Mohsen Ibrahim 19b01d74fa bug fix: Send stream notifications to all users for public streams.
If a stream is public, we now send notifications to all realm users
if the name or description of the stream changes.  For private
streams, the behavior remains the same.

We do this by introducing a method called
can_access_stream_user_ids().

(showell helped with this fix)

Fixes #2195
2016-11-04 21:41:41 -07:00
Steve Howell c405b67138 tests: Ignore SAVEPOINT queries in queries_captured().
We use the queries_captured() context manager in our tests
to capture queries that happen during certain actions.  Usually
we use the list of queries to validate that we're not doing
too many database calls.

For most tests the database calls are fairly deterministic,
but SAVEPOINT-related things seem to be more random, and the
number of savepoints is usually not relevant to what the test
is trying to prevent, which is more serious problems like
O(N) database fetches.
2016-11-04 21:30:36 -07:00
Steve Howell 2088e8420f User user_id, not email, in peer_add events. 2016-11-04 11:37:14 -07:00
Steve Howell c3f2740481 Remove email_dict from gather_subscriptions_helper(). 2016-11-04 11:37:14 -07:00
Steve Howell 38f2789152 Remove page_params.email_dict. 2016-11-04 11:37:14 -07:00
Umair Khan fe1ba6f3eb Django 1.10: Deprecate request.REQUEST. 2016-11-04 10:20:24 -07:00
Umair Khan 682aa1f298 Django 1.10: Use add_argument for options in BaseCommand. 2016-11-04 10:20:23 -07:00
Umair Khan 4c07e57b43 Django 1.10: Use lru_cache instead of memoize. 2016-11-04 10:20:23 -07:00
Umair Khan d81446805c Django 1.10: Use `caches` object to access cache. 2016-11-04 10:06:00 -07:00
Rishi Gupta 4a74301a62 models.py: Replace resolve_subdomain_to_realm with get_realm_by_string_id.
No change in functionality.
2016-11-03 13:59:11 -07:00
Rishi Gupta 655940acaf create_realm.py: Add basic checks to name and subdomain.
Don't allow the empty string for organization name, and don't pass
do_create_realm a subdomain that's already taken.
2016-11-03 13:59:11 -07:00
Tim Abbott dd008ae559 realm creation flow: Allow signups from all emails.
This (1) removes the check on whether the domain of the email matches
the Realm.domain of an existing realm and (2) avoids setting `realm =
get_realm(domain)` in the realm creation flow, which would cause the
wrong code path to be followed in the event that the domain in a
user's email address happens to match a deactivated realm.
2016-11-03 13:58:52 -07:00
Rishi Gupta 1944ac844a RealmAlias: Drop uniqueness constraint when REALMS_HAVE_SUBDOMAINS.
Removes the uniqueness constraint on RealmAlias.domain, and adds a function
can_add_alias that checks for uniqueness conditional on
settings.REALMS_HAVE_SUBDOMAINS.
2016-11-03 13:53:01 -07:00
Rishi Gupta e4334f4735 do_create_realm: Remove domain argument from do_create_realm.
The domain argument was being used in do_create_realm to create a
RealmAlias. Callers are now responsible for creating a RealmAlias
when desired.
2016-11-03 13:53:01 -07:00
Rishi Gupta 4b60e06991 realm creation flow: No longer create RealmAlias's during sign-up flow.
There are a number of situations in which we will want to automatically
create one, but we should be a bit thoughtful about when we do so.
2016-11-03 13:53:01 -07:00
Rishi Gupta c0f1b5d9f6 Change Realm defaults to Community defaults.
Previously, we set restrict_to_domain and invite_required differently
depending on whether we were setting up a community or a corporate
realm. Setting restrict_to_domain requires validation on the domain of the
user's email, which is messy in the web realm creation flow, since we
validate the user's email before knowing whether the user intends to set up
a corporate or community realm. The simplest solution is to have the realm
creation flow impose as few restrictions as possible (community defaults),
and then worry about restrict_to_domain etc. after the user is already in.

We set the test suite to explictly use the old defaults, since several of
the tests depend on the old defaults.

This commit adds a database migration.
2016-11-03 13:53:01 -07:00
Tim Abbott 08ce391611 register: Fix using subdomains logic when subdomains is disabled. 2016-11-03 13:51:50 -07:00
Tim Abbott fa6aafd0a5 test_signups: Fix test_completely_open_domain.
This test seems intended to verify registration in the case of a
unique completely open domain; but because of the mit.edu realm, it
instead tested that a logic bug in the non-subdomains case was
present.
2016-11-03 13:44:51 -07:00
Tim Abbott 9f86638ce5 test_signups: Cleanup open realm registration tests.
This removes some confusing comments and unnecessary use of subdomain
logic.
2016-11-03 13:44:43 -07:00
Tim Abbott 9b6c199538 tests: Fix need for Internet to run backend test suite.
A few places where we weren't using mock.patch to mock out Hesiod
(DNS) queries have crept in.
2016-11-03 13:41:54 -07:00
Tim Abbott 90f57c92d4 compute_mit_user_fullname: Fix result for Python 3.
DNS replies are bytes, and usernames are strings.
2016-11-03 13:40:43 -07:00
K.Kanakhin c89a2ffe2f Add raised exception output to development tornado running.
Replaces the default tornado ioloop tracback exception handler in
debug mode to add output for exceptions to the logging.

Fixes #2150.
2016-11-03 11:16:12 -07:00
Tomasz Kolek 9026f56713 Improve English grammar in bitbucket integration modify comment message.
Add `a` between verb and comment word i.e. updated a [comment] instead of
updated [comment].
2016-11-03 10:18:46 -07:00
Umair Khan 4502872e8e subdomain: Test user registration through Google OAuth. 2016-11-03 10:09:24 -07:00
Steve Howell 9b04ec7472 Send dicts for cross realm users to JS.
We now send dictionaries for cross-realm bots.  This led to the
following changes:

* Create get_cross_realm_dicts() in actions.py.
* Rename the page_params field to cross_realm_bots.
* Fix some back end tests.
* Add cross_realm_dict to people.js.
* Call people.add for cross-realm bots (if they are not already part of the realm).
* Remove hack to add in feedback@zulip.com on the client side.
* Add people.is_cross_realm_email() and use it in compose.js.
* Remove util.string_in_list_case_insensitive().
2016-11-03 09:51:23 -07:00
Steve Howell af6512e782 Add feedback bot to the test database. 2016-11-03 09:42:27 -07:00
Steve Howell 6324fe9ddf Rename get_cross_realm_users to get_cross_realm_emails. 2016-11-03 09:42:27 -07:00
Tim Abbott bfff9e1968 do_send_messages: Fix sending messages to feedback-bot.
This feature was broken in the message_to_dict refactoring.
2016-11-02 23:42:05 -07:00
Rishi Gupta 70c29d8f90 do_create_realm: Remove "(open realm)" indication in sign-up message.
In preparation for a change to do_create_realm where we will use the
database default for restricted_to_domain rather than computing it within
do_create_realm, and due to which do_create_realm will no longer know
whether we are creating an open realm or not.
2016-11-02 22:46:34 -07:00
Rishi Gupta db0e509422 do_create_realm: Replace domain argument with string_id.
Turns string_id into a required argument, and domain into an optional
argument.
2016-11-02 22:46:34 -07:00
Rishi Gupta c1bbd8d70d models.py: Add get_realm_by_string_id to assist with Realm.domain migration.
Step 0 of a two step process:

1. Replace all occurances of get_realm(domain) with
   get_realm_by_string_id(string_id)

2. Rename get_realm_by_string_id to get_realm.
2016-11-02 22:46:34 -07:00
Rishi Gupta 9ef8536cc6 models.Realm: Require Realm.string_id to be non-NULL.
Adds a database migration, adds a new string_id argument to the management
realm creation command, and adds a short name field to the web realm
creation form when REALMS_HAVE_SUBDOMAINS is False.
2016-11-02 22:46:34 -07:00
Rishi Gupta 64bcd71d6e models.Realm: Rename subdomain to string_id.
Does a database migration to rename Realm.subdomain to
Realm.string_id, and makes Realm.subdomain a property.  Eventually,
Realm.string_id will replace Realm.domain as the handle by which we
retrieve Realm objects.
2016-11-02 22:46:34 -07:00
Tomasz Kolek 4dd8cd9879 Replace "Commit" to commits sha in commits comment events in git integrations. 2016-11-02 22:19:51 -07:00
Tomasz Kolek ce5bb22ace Normalize comment commit event in github integration. 2016-11-02 22:19:51 -07:00
Tomasz Kolek 6ad4cce35c Add object number/id to mention about Issue, PR, etc. to all git integrations. 2016-11-02 22:19:51 -07:00
Tomasz Kolek 762b6b5817 Add on betwen comment action and noun in comments messages from git integrations.
Improve English grammar in comment messages in git integrations
by adding 'on' between comment action and noun (PR, Issue, Commit).
2016-11-02 22:19:51 -07:00
Tomasz Kolek cbd05eb0b2 Normalize bitbucket commit, PR and issues comment event's subject and content. 2016-11-02 22:19:51 -07:00
Tomasz Kolek ef6a1328ce Normalize gitlab snippet or commit comment event's subject and content. 2016-11-02 22:19:51 -07:00
Tomasz Kolek 4fc1488edc Normalize gitlab issue or MR comment event's subject and content. 2016-11-02 22:19:51 -07:00
Steve Howell 6659664e59 Simplify/fix cross-realm validation in recipient_for_emails().
We now simply exclude all cross-realm bots from the set of emails
under consideration, and then if the remaining emails are all in
the same realm, we're good.

This fix changes two behaviors:
* You can no longer send a PM to an ordinary user in another realm
  by piggy-backing a cross-realm bot on to the message.  (This was
  basically a bug, but it would never manifest under current
  configurations.)
* You will be able to send PMs to multiple cross-realm bots at once.
  (This was an arbitrary restriction.  We don't really care about this
  scenario much yet, and it fell out of the new implementation.)
2016-11-02 22:10:49 -07:00
Steve Howell 497b8e8bc4 Make tests explicit about cross-realm loophole.
We can currently send a PM to a user in another realm, as long
as we copy a cross-realm bot from the same realm.  This loophole
doesn't yet affect us in practice--all cross-realm bots are
generally configured for the "admin" realm like the old zulip.com--
but we should lock it down in a subsequent commit.
2016-11-02 22:10:49 -07:00
Steve Howell 7d50598454 Test that non-zulip.com users can't PM random zulip.com users. 2016-11-02 22:10:49 -07:00
Steve Howell 45f9a60bc0 Add test coverage for talking to two cross-realm bots. 2016-11-02 22:10:49 -07:00
Steve Howell 067090d93e Test that we can send PMs to non-zulip cross-realm bots. 2016-11-02 22:10:49 -07:00
Steve Howell 7724dba08b Explicitly use the "feedback" bot in test_realm_scenarios.
(I'm about to create a more abstract cross-realm bot, but I think
concretely referring to feedback bot is helpful here.)
2016-11-02 22:10:49 -07:00
Steve Howell 517f7f672f Clean comments/whitespace in test_realm_scenarios(). 2016-11-02 22:10:49 -07:00
Steve Howell 94f53543e6 Test that users can PM themselves in test_realm_scenarios.
(This is probably tested elsewhere too, but it's helpful in the
context of testing cross-realm stuff to assert basic rules like
this.)
2016-11-02 22:10:49 -07:00
Steve Howell 7a127148db Add assert_message_received() to test_realm_scenarios. 2016-11-02 22:10:49 -07:00
Steve Howell b9bf74d54e Reorganize sections of test_realm_scenarios().
This change groups the positive scenarios together
and likewise for the negative scenarios.
2016-11-02 22:10:49 -07:00
Steve Howell 49c64bd7cb Consolidate cross-realm message tests.
Having each condition in a separate test was confusing to read,
especially since the tests were doing inconsistent setup, sometimes
calling user2 the user from 2.example.com realm and other times
calling user2 the cross-bot realm, etc.
2016-11-02 22:10:49 -07:00
umkay 0e0a11bb5e Refactor GitHub authentication backend tests. 2016-11-02 21:06:17 -07:00
Umair Khan 0eff88ec2d Test registration of mirror dummy user.
Tweaked by tabbott to do the from_confirmation=1 call.
2016-11-02 18:17:29 -07:00
Umair Khan dd0c4ca0bc Test registration using POPULATE_PROFILE_VIA_LDAP. 2016-11-02 17:50:40 -07:00
Umair Khan f300c12535 Accept from_confirmation kwarg in submit_reg_form_for_user. 2016-11-02 17:50:40 -07:00
Umair Khan 99f4174894 Test special registation link for completely open realms. 2016-11-02 17:50:39 -07:00
Umair Khan 8ff2cd3fd7 Only allow proper url for registration.
This commit fixes a bug which occurs when a user tries to bypass the
correct registration url for the completely open domains.
2016-11-02 17:49:30 -07:00
Tim Abbott bf1a7a7f2d test_bugdown: Fix misplaced type annotation. 2016-11-01 20:40:22 -07:00
paxapy 1d1c41551b bugdown: Auto-linkify #StreamName. 2016-11-01 19:59:21 -07:00
paxapy d965ff16fd bugdown: refactor AutoLink to use VerbosePattern.
This makes it possible to reuse the verbose pattern logic for other
regexs as well.
2016-11-01 19:50:59 -07:00
Steve Howell 403490a95b annotations: Fix to_non_negative_int(). 2016-10-28 13:48:44 -07:00
Steve Howell 7111bb1744 Deprecate /json/streams.
Note that we still need the equivalent function in our
user-facing API, so there is not much code removal yet.
(Also, we will probably always keep this in our API,
as bot authors will usually just want a simple endpoint
here, whereas our client code gets page_params and events.)
2016-10-28 13:44:46 -07:00
Steve Howell 768a04d2fa Send user_ids for update events.
Here we handle a few flavors of realm_bot/update
and realm_user/update.  (Different events send different
attributes.)
2016-10-28 13:40:14 -07:00
Steve Howell 3fd7ab950a Send user_id to the client when removing bots/users.
This touches these events:

    realm_bot/remove
    realm_user/remove
2016-10-28 13:40:14 -07:00
Steve Howell 7883cecf28 Send user_id to the client in more places.
This commit touches:

    realm_bot/add
    realm_user/add
    page_params.bot_list
    page_params.people_list
2016-10-28 13:40:14 -07:00
Tim Abbott 1a88ca4da1 queue_processors: Fix feedback bot sender.
This fixes an issue where emails sent by feedback-bot had a forged
sender and thus sometimes had delivery issues.
2016-10-28 11:04:08 -07:00
Umair Khan 4cf7040100 subdomains: Use redirect_and_log_into_subdomain for registration.
This fixes a bug where the user who registered a new organization
wasn't correctly logged into that organization.
2016-10-28 11:04:08 -07:00
Umair Khan 2dabfc562c subdomains: Add tests for single domain OAuth2. 2016-10-28 11:04:08 -07:00
Umair Khan 26646abe8c Authenticate subdomains using single Google OAuth entry.
Previously, we used to create one Google OAuth callback url entry
per subdomain. This commit allows us to authenticate subdomain users
against a single Google OAuth callback url entry.
2016-10-28 11:04:08 -07:00
Umair Khan d139c9cb71 Add tests for maybe_send_to_registration function. 2016-10-28 09:27:56 -07:00
Umair Khan 8064ed1c0b Add test for ZulipRemoteUserBackend. 2016-10-28 09:27:56 -07:00
Umair Khan 839cab3434 Add tests for GitHubAuthBackend. 2016-10-28 09:27:56 -07:00
Umair Khan b6046bc77e Add tests for password_auth_enabled function. 2016-10-28 09:27:56 -07:00
Umair Khan d9e1975d2c Add ZulipAuthMixin tests. 2016-10-28 09:27:56 -07:00
Umair Khan 6dbe425675 Add Google OAuth2 backend tests. 2016-10-28 09:27:56 -07:00
Umair Khan 088d8eebad Add ZulipLDAPUserPopulator test. 2016-10-28 09:27:56 -07:00
Umair Khan 50422e775b Add LDAP tests. 2016-10-28 09:27:55 -07:00
K.Kanakhin 78b1b80987 caching: Add configuration class for post-migration cache flushing.
- To avoid redefining migrate manage command is added new application
  configuration class which emit post_migration signal. This signal
  require models module inside application and defined AppConfig
  Instance as signal sender.  Documentation here:
  https://docs.djangoproject.com/en/1.8/ref/signals/#post-migrate.
- Add AppConf subclass to __init__ zerver app file to make apllication
  load it by default.

Fixes #1084.
2016-10-27 23:26:34 -07:00
Tim Abbott 9b7a3f040c Remove now-unused /json/get_events endpoint. 2016-10-27 21:34:58 -07:00
Tim Abbott 86e933a4a1 process_queue: Suppress USING_RABBITMQ warnings in test suite. 2016-10-27 12:36:06 -07:00
Umair Khan 486774b28d Add tests for JWT based login. 2016-10-26 14:40:56 -07:00
Umair Khan 52acdd28bd Fix JWT code in the auth backend.
For some reason, we use 'load' function but it doesn't exist in the JWT
library code. This commit updates the code to use the correct interface
of the JWT library.

The signature verification is done by the decode function.
2016-10-26 14:40:56 -07:00
Arpith Siromoney c5510599ef Typing backend: use dict instead of user profile for sender in event.
Fixes #2110.
2016-10-26 13:12:18 -07:00
K.Kanakhin 39e0886361 retention-policy: Add tool to determine expired messages.
This is a first step towards implementing a message retention policy
feature.

- Add Realm model message_retention_days field to setup
  messages expired period for realm.
- Add migration.
- Add tool to get expired messages for each Realm.
- Add tests to cover tool for getting expired messages.
2016-10-25 15:38:08 -07:00
umkay 1d077d1a4f bugdown: Remove safe_mode dependencies.
Now that we have updated python-markdown, we remove the deprecated
safe_mode. We used safe_mode to escape raw html, so now instead we
pass in an EscapeHtml markdown extension to the markdown engine.

See https://pythonhosted.org/Markdown/release-2.6.html for details on
the deprecation.

Fixes: #2037 (also addresses the remaining piece of #2043).
2016-10-25 15:22:18 -07:00
umkay 5fe6e7c07c bugdown: Add tests for escaping raw html.
Adds additional tests in bugdown-data.json to check that we escape raw
html properly with script tags.

Many of these tests are based on the bleach "advanced" test suite.
2016-10-25 15:21:46 -07:00
Tim Abbott 7b9d1a7824 auth: Simplify REALMS_HAVE_SUBDOMAINS redirect code.
This logic was just a duplicate on realm.uri, except that in the case
that subdomains are disabled, it preserves the host in the request.
2016-10-25 13:55:17 -07:00
hackerkid b5816bf99e Don't use zulipdev.com when subdomain support is disabled. 2016-10-25 13:52:13 -07:00
Arpith Siromoney 01665d9c3c Extract get_recipient_user_profiles.
This creates a common function for extracting recipients, which both
do_send_messages and do_send_typing_notifications can use.
2016-10-25 13:37:01 -07:00
Tomasz Kolek 4790316b57 Add user setting option to always send push notifications.
Add option in user's settings for getting mobile push notifications
even if a Zulip browser is online.  Default is False.

Fixes: #1596.
2016-10-25 10:52:29 -07:00
Rishi Gupta b6da450b96 admin.js: Use RealmAlias instead of Realm to show the restricted domains.
Passes the allowed domains for a realm to the frontend, via
page_params.domains. Groundwork for allowing users to add and
remove domains via the admin setting page, rather than via the
realm_alias.py management command.
2016-10-25 10:07:20 -07:00
Rishi Gupta 537ee75761 Create RealmAlias entries for Realm.domain.
This is a preliminary step towards eliminating the realm.domain field
in favor of realm.subdomain.  Includes a database migration to create
these for existing realms.
2016-10-25 10:06:39 -07:00
Brock Whittaker fca61b2031 Add medium size avatars for use on the user's own settings page.
This adds a medium (500px) size avatar thumbnail, that can be
referenced as `{name}-medium.png`.  It is intended to be used on the
user's own settings page, though we may come up with other use cases
for high-resolution avatars in the future.

This will automatically generate and upload the medium avatar images
when a new avatar original is uploaded, and contains a migration
(contributed by Kirill Kanakhin) to ensure all pre-existing avatar
images have a medium avatar.

Note that this implementation does not provide an endpoint for
fetching the medium-size avatar for another user.

[substantially modified by tabbott]
2016-10-25 09:42:14 -07:00
Umair Khan 9b42b7fad3 Add tests for TestZulipRemoteUserBackend. 2016-10-25 08:13:35 -07:00
Umair Khan 9f37b9b386 Add tests for TestDevAuthBackend. 2016-10-25 17:57:33 +05:00
Tim Abbott e714ddbfc8 gather_subscriptions_helper: Fix mypy error. 2016-10-22 22:37:05 -07:00
Tim Abbott c59cdbb92d gather_subscriptions: Fix missing subscriber data in email_dict.
When we added data on never_subscribed streams to what
populate_subscribers is called on, we failed to add the corresponding
data on subscribers to email_dict, the mapping of user IDs to emails
for the subscribers.
2016-10-22 21:37:05 -07:00
Tim Abbott 14018353fc Fix never_subscribed streams being sent for Zephyr mirroring.
Because in the Zephyr world, stream names can be a secret, and also
Zephyr mirroring tends to involve many thousands of streams, we
shouldn't send this data.
2016-10-22 21:36:11 -07:00
Tim Abbott 893679baff Add migration to remove system avatar source.
Fixes the last commit having broken master.
2016-10-22 20:07:21 -07:00
Tim Abbott 5bea2f5e20 Remove unused AVATAR_FROM_SYSTEM code.
This is some of the code we'd need if we wanted to have Zulip generate
avatars for things.  Since it is so little useful code, and it's not
clear we will need this feature ever, we can remove this code to make
the codebase less confusing.  It'd be easy to dig this out of history
if we ever want it.

Fixes #2101.
2016-10-22 19:48:50 -07:00
Steve Howell 4d6af33bad Use make_stream() in test_subs.py. 2016-10-22 18:09:34 -07:00
Steve Howell 2a739318b9 Use make_stream() in test_signup.py. 2016-10-22 18:09:34 -07:00
Steve Howell 8410159af2 Use make_stream() in test_narrow.py. 2016-10-22 18:09:34 -07:00
Steve Howell b1ebb09c96 Use make_stream() in test_messages.py. 2016-10-22 18:09:34 -07:00
Steve Howell 0861421912 Use make_stream() in test_events.py. 2016-10-22 18:09:34 -07:00
Steve Howell 79b6f3d39d Create ZulipTestCase.make_stream(). 2016-10-22 18:09:34 -07:00
K.Kanakhin 16c3da6db5 tests: Add tests for SEND_MISSED_MESSAGE_EMAILS_AS_USER=False.
- Add tests for SEND_MISSED_MESSAGE_EMAILS_AS_USER is False (the
  default!).
- Reorganized test case code by removing repeated parts of code,
  improving code style and moving common parts to separate class
  methods.

Fixes #1697.
2016-10-21 08:47:24 -07:00
Umair Khan 8891731d7d is_template_database_current: Open file in write mode. 2016-10-21 08:35:30 -07:00
Umair Khan b894135fae Check postgres-init-dev-db in is_template_database_current. 2016-10-21 08:35:30 -07:00
Umair Khan 176825d335 Refactor is_template_database_current.
Now all the files whose hash is to be checked are passed as a list
argument.
2016-10-21 08:35:30 -07:00
Umair Khan bf4e0800e3 is_template_database_current: Create hashes in var/test_db_status. 2016-10-21 08:35:30 -07:00
Umair Khan 8ade9e2408 Add tests to check subdomains with auth decorators.
Fixes: #1870
2016-10-21 12:21:47 +05:00
Tomasz Kolek 6e1674aca5 Normalize github issue or PR comment event's subject and content. 2016-10-20 23:15:51 -07:00
Tomasz Kolek 56dd1a9b90 Normalize gitlab issue event's subject and content. 2016-10-20 23:15:30 -07:00
Tomasz Kolek a8da46db92 Normalize bitbucket issue event's subject and content. 2016-10-20 23:15:30 -07:00
Tomasz Kolek 171a4861cc Normalize github issue event's subject and content. 2016-10-20 23:15:30 -07:00
Tomasz Kolek 89d53a0640 Change name of PR subject template to be applicable for issues. 2016-10-20 23:15:30 -07:00
Tomasz Kolek 289f400c8c Add git integrations method to build message for issues events. 2016-10-20 23:15:30 -07:00
Arpith Siromoney ee97ba04fe Add an API endpoint to send typing notification events.
POST to /typing creates a typing event
Required parameters are 'op' ('start' or 'stop') and 'to' (recipient
emails). If there are multiple recipients, the 'to' parameter
should be a JSON string of the list of recipient emails.
The event created looks like:
{
  'type': 'typing',
  'op': 'start',
  'sender': 'hamlet@zulip.com',
  'recipients': [{
    'id': 1,
    'email': 'othello@zulip.com'
  }]
}
2016-10-20 23:05:18 -07:00
Tim Abbott f9f8b18e2f lint: Ban use of json_success({}) instead of json_success(). 2016-10-20 22:34:04 -07:00
Amy Liu 3ee777a11a Add UI for deactivating your own Zulip account.
Fixes #1009.
2016-10-20 22:29:30 -07:00
Tim Abbott 3d183ffe23 text_fixtures: Fix case where source_hash_file doesn't exist.
We unfortunately just broke provisioning because we didn't correctly
handle the case where the source_hash_file didn't exist.
2016-10-20 22:21:45 -07:00
Steve Howell 409a84698c Handle peer_remove events correctly for subscriptions.
We now send peer_remove events to folks who have never subscribed
to the streams (except for private streams and zephyr).

We also use logic that is more similar to how
bulk_add_subscriptions() works.
2016-10-20 22:05:19 -07:00
Steve Howell 68ae4da626 Extract query_all_subs_by_stream().
This code was in bulk_add_subscriptions, but we want the same logic
to eventually happen in bulk_remove_subscriptions.
2016-10-20 22:05:19 -07:00
Steve Howell d13b0554b3 Remove emails_by_stream in bulk_add_subscription().
There are so many collections in this function, it's hard to keep
them all straight.  This one is fairly redundant, so it was easy
to remove.
2016-10-20 22:05:19 -07:00
Steve Howell d706bfa69f Move some code from notify_subscriptions_removed().
There are two reasons for this change.  First, we want to be
consistent with notify_subscriptions_added(), which doesn't
handle "peer" events.  Second, we want to fix this code in a
subsequent commit not to do one user at a time, which is
inefficient.
2016-10-20 22:05:19 -07:00
Steve Howell dda4c61f2a Remove do_remove_subscription(). 2016-10-20 22:05:19 -07:00
Steve Howell cba8be6bbb Use bulk_remove_subscriptions() in remove_users_from_stream.
We are deprecating do_remove_subscription.
2016-10-20 22:05:19 -07:00
Steve Howell 4ae117ed14 Use bulk_remove_subscriptions() in do_deactivate_stream().
We are deprecating do_remove_subscription().
2016-10-20 22:05:19 -07:00
Steve Howell 77c75df284 Remove do_remove_subscription() calls in test_events. 2016-10-20 22:05:19 -07:00
Steve Howell 44bbece5a6 Remove do_remove_subscription() calls in tests.py. 2016-10-20 22:05:19 -07:00
Steve Howell b2d06fad42 Extract get_peer_user_ids_for_stream_change(). 2016-10-20 22:05:19 -07:00
Steve Howell 5eee77fe81 Remove do_add_subcription(). 2016-10-20 22:05:19 -07:00
Steve Howell 98e4452e07 Replace do_add_subscription() in add_users_to_streams. 2016-10-20 22:05:19 -07:00
Steve Howell bea8ba620b Replace do_add_subscription() in tests/tests.py. 2016-10-20 22:05:19 -07:00
Steve Howell 9408ba7ca8 Replace do_add_subscription() in test_subs. 2016-10-20 22:05:19 -07:00
Steve Howell 45790264c2 Replace do_add_subscription() in test_signup. 2016-10-20 22:05:19 -07:00
Steve Howell f4fc8455de Replace do_add_subscription() in test_narrow. 2016-10-20 22:05:19 -07:00
Steve Howell d18e286131 Replace do_add_subscription() in test_messages. 2016-10-20 22:05:19 -07:00
Steve Howell 78b73a0c91 Replace do_add_subscription() in test_helpers. 2016-10-20 22:05:19 -07:00
Umair Khan 45166cb39c is_template_database_current: Check populate_db and postgres-init-test-db.
Compare the hash of 'zilencer/management/commands/populate_db' with
'var/populate_db_hash' and 'tools/setup/postgres-init-test-db' with
'var/postgres_init_test_db_hash' and if any comparison fails rebuild
the database.

With fixes from tabbott.
2016-10-20 22:04:47 -07:00
hackerkid bca1953aa4 Add support for !gravatar syntax in echo.js. 2016-10-20 14:40:45 -07:00
hackerkid 95b5ac1d5d Add support for !avatar syntax in echo.js. 2016-10-20 14:40:33 -07:00
Tomasz Kolek 67adf570e7 Remove @-mentions from jira integration.
Because of a lot of annoying notifcations we decided to remove
all @-mentions in jira. We replaced it by just login (without @).
2016-10-20 11:55:12 -07:00
Tim Abbott 31b245eaea bugdown: Add tests for avatar/gravatar syntax. 2016-10-20 11:18:41 -07:00
K.Kanakhin 7bc0c7f0d5 bugdown: Fix patterns ordering in Markdown extension.
- Put 'modal_link' and 'link' inline patterns after avatar inline pattern
  to correct applying them to parsed text.

Fixes:  #1654
2016-10-20 11:18:41 -07:00
Tim Abbott 50356cc325 event_queue: Eliminate buggy need_timeout heartbeat code.
As best I can tell, this option was completely confused in two ways:
* The name is confusing; it actually controls whether we _clear_ the
  timeout associated with the current handler
* It's not clear why one would not want it to be unconditionally true.

From reading the history, I'm pretty sure I had just misread the code
when I created this.

And this caused a real bug; a later refactoring caused us to basically
never cancel the timeouts, which in turn resulted in 90% of all events
traffic being hearbeats with a much lower frequency (~5s) than the
intended 45s.  Removing this code fixes that nasty bug.
2016-10-19 14:32:49 -07:00
Tim Abbott a6e779a56c events: Add event types to logging.
This should make it a lot easier to analyze our events traffic and
debug potential issues with it.
2016-10-19 14:32:16 -07:00
Rishi Gupta 83f902953a newsletter signup: Update merge_vars.
merge_vars is the user meta data we store in mailchimp. This commit changes
what we store from realm.domain to realm.id, since realm.domain is being
deprecated, and changes the OPTIN_TIME to a nicer format.
2016-10-18 19:29:12 -07:00
Brock Whittaker 77233c7e3b bugdown: Distinguish between YouTube Videos and Image Previews.
This distinguishes between YouTube Videos and Image Previews by adding
a particular “youtube-video” class to the preview along with changing
the title to the video ID rather than the link. This serves to allow
the lightbox to ID when a lightbox preview should be treated like a
YouTube video rather than an image preview.

This also modifies the tests in bug down to expect a youtube-video class
along with the title to just be the video ID on YouTube rather than the
entire URL link.
2016-10-18 14:45:47 -07:00
Tim Abbott 56d0cc69e9 socket: Remove unused djsession_engine. 2016-10-17 23:19:28 -07:00
Mohsen Ibrahim bfe5787197 Fix Django.utils.importlib deprecation warnings.
This changes `from django.utils.importlib import import_module` to
`from importlib import import_module`, as `django.utils.importlib` will
be removed in django version 1.9.
2016-10-17 23:19:28 -07:00
Mohsen Ibrahim e9899e4dc9 bugdown: Fix some Python-Markdown deprecation warnings.
Python-Markdown changed how one is supposed to declare extensions
being used.
2016-10-17 23:13:41 -07:00
Tim Abbott 63a49be09e event_queue: Import tornado.ioloop correctly.
We actually use tornado.ioloop, not the root tornado module, and while
the current code does work, it causes mypy errors with the incremental
cache.
2016-10-17 17:57:46 -07:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Tim Abbott 3b0b65c2d0 management: Fix recent argument parsing regressions.
Fixes a regression introduced in
fbdf539bf2.
2016-10-16 20:36:33 -07:00
Laura Hampton fbdf539bf2 Replace optparse with argparse in the several management commands.
Specifically:
* fill_memcached_caches.py
* initialize_voyager_db.py
* logout_all_users.py
2016-10-16 17:08:35 -07:00
Tim Abbott 2bea0c46fe HomeTest: Add a basic test for desktop_home. 2016-10-16 11:39:08 -07:00
Tim Abbott 3ee9e2c45f tests: Add some basic i18n public URL tests. 2016-10-16 11:26:01 -07:00
Tim Abbott 122164fffb DocPageTest: add test for /about/ page. 2016-10-16 11:19:27 -07:00
Tim Abbott e37b6488b8 Bugdown: Remove checked-in CodeHilite extension.
The changes that required us to fork this extension had been merged
into upstream CodeHilite, so we can remove it and switch to using the
version that comes with python-markdown.
2016-10-16 00:40:08 -07:00
umkay c028815385 bugdown: Update python-markdown to version 2.6.7.
This updates Bugdown to reflect the changes in the updated
markdown. In particular, we now pass a default config object in the
__init__ for the Bugdown extension, update the make_md_engine function
to take kwargs as opposed to a config list, and have UListProcessor
inherit from ulist as opposed to olist (which no longer works).

We update the (forked from upstream) fenced_code extension's
makeExtension to take args and kwargs, and update
FencedBlockPreprocessor __init__ method with updated Codehilite
arguments.

We update the (forked from upstream) Codehilite extension to
mirror the logic with the latest upstream Codehilite:
	Add parse_hl_lines function
	update makeExtension to take args and kwarfs instead of config
	list
	Add regex for highlight lines
	use linenums instead of linenos
	use get_formatter_by_name instead of HtmlFormatter
	user get_lexer_by_name instead of TextLexer
	add hl_lines and use_pygments arguments to the codehlite
	constructor
2016-10-15 23:38:32 -07:00
Tomasz Kolek 92bfadcca4 Normalize github pull requests event message.
Add detailed info (description, source and target branch, assignee) to message.
Change subject to 'repo_name / PR #id title'.
Modify some test fixtures for better coverage.
2016-10-14 16:54:13 -07:00
Tomasz Kolek ce9ac30fac Normalize bitbucket pull requests event message.
Add detailed info (description, source and target branch, assignee) to message.
Change subject to 'repo_name / PR #id title'.
Modify some test fixtures for better coverage.
2016-10-14 16:54:13 -07:00
Tomasz Kolek 421c51cc14 Normalize gitlab pull requests event message.
Add detailed info (description, source and target branch, assignee) to message.
Change subject to 'repo_name / MR #id title'.
Modify some test fixtures for better coverage.

Fixes: #1883.
2016-10-14 16:54:13 -07:00
Tomasz Kolek 2941c1f517 Move converting commits data to text to separate function.
Create get_commits_content function for converting
commits list to text.
2016-10-14 16:54:13 -07:00
Tomasz Kolek d2cedd3667 Rename consts in lib/webhooks/git.py to make it more general.
Rename:
   PUSH_COMMITS_LIMIT to COMMITS_LIMIT
   PUSH_COMMIT_ROW_TEMPLATE to COMMIT_ROW_TEMPLATE
   PUSH_COMMITS_MORE_THAN_LIMIT_TEMPLATE to COMMITS_MORE_THAN_LIMIT_TEMPLATE
2016-10-14 16:54:13 -07:00
Tomasz Kolek ee70cc5141 Add git integration method to build message for pull requests events. 2016-10-14 16:54:13 -07:00
Tomasz Kolek 0f01150449 Add bitbucket tests for push, push a lot of commits and force push events. 2016-10-14 16:54:13 -07:00
Tomasz Kolek 165cdb79d9 Rename bitbucket2 fixtures directory to bitbucket.
Rename all bitbucket fixtrues from bitbucket2.* to bitbucket_v2.*.
2016-10-14 16:54:13 -07:00
umkay 278574a6ec Fix cursor() arguments for connection classes.
In order to use the latest version of psycopg2 with the latest version
of sqlalchemy we must integrate the changes made in the 2.4.6 release of
psycopg2. See
c86ca7687f
and the release notes for psycopg2 2.4.6.
2016-10-14 13:35:06 -07:00
Rishi Gupta 82b814a1cd analytics: Simplify frequency and measurement interval options.
Change the CountStat object to take an is_gauge variable instead of a
smallest_interval variable. Previously, (smallest_interval, frequency)
could be any of (hour, hour), (hour, day), (hour, gauge), (day, hour),
(day, day), or (day, gauge).
The current change is equivalent to excluding (hour, day) and (day, hour)
from the list above.

This change, along with other recent changes, allows us to simplify how we
handle time intervals. This commit also removes the TimeInterval object.
2016-10-14 10:18:37 -07:00
Tim Abbott 8f145e03cf views: split events_register.py from main views file. 2016-10-11 21:27:06 -07:00