Commit Graph

1587 Commits

Author SHA1 Message Date
Sampriti Panda 91780180f6 integrations: Add webhook code, API endpoint, and tests for AppFollow 2016-12-15 11:13:27 -08:00
Umair Khan 770a899239 Django 1.10: Use single cache prefix for casper tests.
There is a change in Django 1.10 due to which whenever the password
of the user is changed the session hash changes. This change affects
us because we cache user profile objects and these cached objects need
to be refreshed. However, the signal sent by Django in which objects are
refreshed fails to refresh the cache for Tornado because it uses a
different cache prefix.

Note: Backend tests are not affected because they don't rely on Tornado.
2016-12-14 22:40:33 -08:00
Tim Abbott 0b33be50f3 lint: fix some whitespace issues in new reactions code. 2016-12-14 20:37:13 -08:00
Kracekumar R 61d2297c17 Add reactions in the /json/messages endpoint. 2016-12-14 19:21:04 -08:00
Harshit Bansal dc4e56abae Show offending stream name incase of a non existent stream.
Fixes: #2171.
2016-12-14 12:17:20 -08:00
Rishi Gupta 92a72f7039 mypy: Use typing.Text in actions.py. 2016-12-14 11:40:34 -08:00
Bojidar Marinov d28f1ddfb4 zerver: Handle `update_global_notifications` in `apply_events`
Fixes #2358
2016-12-13 20:20:26 -08:00
reallyly 94b28e0afb pep8: Fix E128 violations.
With some line-wrapping tweaks by tabbott.
2016-12-13 10:50:14 -08:00
Igor Tokarev fae59502ab URL preview: Improve test coverage. 2016-12-13 10:43:02 -08:00
Joy Chen bfb6ac5fdb streams: Add default stream description tests and functions.
This includes making the default stream description setting into a
dict.  That is an API change; we'll discuss it in the changelog but it
seems small enough to be OK.

With some small tweaks by tabbott to remove unnecessary backwards
compatibility code for the settings.

Fixes #2427.
2016-12-07 21:12:44 -08:00
Joy Chen dd95a5e03f streams: Add default stream description population data. 2016-12-07 21:02:17 -08:00
Igor Tokarev c93f1d4eda Add oembed/Open Graph/Meta tags data retrieval from inline links.
This change adds support for displaying inline open graph previews for
links posted into Zulip.

It is designed to interact correctly with message editing.

This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.

By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.

Eventually, we may want to make this manageable via a (set of?)
per-realm settings.  E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
2016-12-07 17:40:18 -08:00
Tim Abbott b68fef8933 actions: Extract update_to_dict_cache. 2016-12-07 17:26:16 -08:00
Sampriti Panda 7c3aff92d9 integrations: Add webhook payloads, webhook code, API endpoint, and tests for heroku 2016-12-06 11:05:00 -08:00
Bickio 84765e48a9 integrations: Add webhook code, API endpoint, and tests for papertrail 2016-12-05 21:48:11 -08:00
Mikebarson 690d72d35f mypy: Convert more zerver/lib files to typing.Text.
This commit touches files in zerver/lib/bugdown/ and
zerver/lib/webhooks.
2016-12-05 21:17:24 -08:00
Vamshi Balanaga f89e732d9e integrations: Add webhook code, API endpoint, and tests for mention. 2016-12-05 16:46:32 -08:00
sidhant bhavnani 4634c3d656 Change call signature for users_subscribed_to_stream()
The function users_subscribed_to_stream() now takes realm now, instead of domain.
2016-12-05 15:46:19 -08:00
AZtheAsian 441743cb89 integrations: Add webhook code, API endpoint, and tests for stripe.
This integration still needs documentation.
2016-12-03 20:42:43 -08:00
Reid Barton ecfa397567 Fix several type annotations for mypy 0.4.6 compatibility.
A few functions had arguments removed without having their type
annotations updated accordingly. As a result mypy version 0.4.6
thinks that the first type in the annotation is supposed to be
the type of `self`, a new mypy feature which we are not intending
to use here.
2016-12-03 13:45:52 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
bulat22101 a6f91064a2 pep8: Fix E129 violations 2016-12-03 10:56:36 -08:00
bulat22101 adebc75740 pep8: Fix E502 violations 2016-12-03 10:56:36 -08:00
Arpith Siromoney 4491ea8d6b reactions: Add support for removing emoji reactions.
This commit adds support for removing reactions via DELETE requests to
the /reactions endpoint with parameters emoji_name and message_id.

The reaction is deleted from the database and a reaction event is sent
out with 'op' set to 'remove'.

Tests are added to check:
1. Removing a reaction that does not exist fails
2. When removing a reaction, the event payload and users are correct
2016-12-02 16:39:53 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
Alex Huang 007b693cc7 pep8: Fix E131. 2016-12-01 23:16:47 -08:00
Alex Huang c8ddea16c3 pep8: Fix E122. 2016-12-01 23:16:35 -08:00
AZtheAsian 1ba150fa85 pep8: Fix E203 violations 2016-12-01 20:37:57 -08:00
AZtheAsian c23f9e0df7 pep8: Fix E111 violations 2016-12-01 14:18:11 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Tim Abbott f8f017c221 test_runner: Fix template rendering test.
The previous logic would allow the same template to be added to both
the shallow_tested list and the normal list.
2016-12-01 10:12:38 -08:00
Tim Abbott 5e0d2c4e1b tests: Exclude markdown files from shallow template testing.
The markdown files under templates/zerver/help/ are technically not
templates in the standard sense, and thus should not be being
checked with this code path.

(We probably do want to add a test to make sure they all render fine,
but that can be its own project.)
2016-12-01 10:12:38 -08:00
Vamshi Balanaga 5f60258a85 pep8: Fix E211 violations. 2016-11-30 20:13:10 -08:00
Bickio 6b0df43463 pep8: Fix E125. 2016-11-30 20:03:29 -08:00
Bickio e009383460 pep8: Fix E231. 2016-11-30 19:59:25 -08:00
Tommy Ip e4091c6413 pep8: Fix E222 violations. 2016-11-30 21:49:02 +00:00
Tim Abbott e6ebffefa0 test_helpers: Cleanup output a bit. 2016-11-29 16:40:05 -08:00
Rafid Aslam 41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
sylvan1 443cf92640 mypy: Change Generator[None, None, None] to Iterator[None].
Fixes #1648.
2016-11-27 10:42:16 -08:00
Tim Abbott 3d1bcb05e1 tornado: Move event_queue.py to zerver/tornado/.
Fixes #729.
2016-11-26 22:29:28 -08:00
Tim Abbott 169d404579 tornado: Move zerver.lib.handlers into zerver.tornado.handlers.
This cleans up the confusingly duplicated file names, while also
moving more of the Tornado-specific code under zerver/tornado/.
2016-11-26 22:29:28 -08:00
Tim Abbott d75f49b119 tornado: Move descriptor dict management code to handlers.py. 2016-11-26 22:29:28 -08:00
Tim Abbott 282f74609c tornado: Move socket code to zerver/tornado/. 2016-11-26 22:29:27 -08:00
Tim Abbott 8739f4d9f1 tornado: Move tornado_ioloop_logging to new zerver/tornado tree. 2016-11-26 21:24:05 -08:00
Arpith Siromoney 001847ac5b Add backend support for emoji reactions.
This commit adds the following:

1. A reaction model that consists of a user, a message and an emoji that
are unique together (a user cannot react to a particular message more
than once with the same emoji)
2. A reaction event that looks like:
    {
        'type': 'reaction',
	'op': 'add',
	'message_id': 3,
	'emoji_name': 'doge',
	'user': {
	    'user_id': 1,
            'email': 'hamlet@zulip.com',
            'full_name': 'King Hamlet'
	}
    }
3. A new API endpoint, /reactions, that accepts POST requests to add a
reaction to a message
4. A migration to add the new model to the database
5. Tests that check that
   (a) Invalid requests cannot be made
   (b) The reaction event body contains all the info
   (c) The reaction event is sent to the appropriate users
   (d) Reacting more than once fails

It is still missing important features like removing emoji and
fetching them alongside messages.
2016-11-26 18:10:21 -08:00
Calvin Lee 4e5c4c5ffb streams: Add support for setting stream description during creation.
Add new box for stream description in the stream creation form,
modify backend so its contents end up as the stream description.

Fixes: #2283.
2016-11-26 16:55:35 -08:00
Calvin Lee b8d7f8008a streams: move the invite_only field into the stream_dict argument.
Refactor list_to_streams and create_streams_if_needed.  The
stream_dict now has a "invite_only" key, which is mapped to
a boolean.
2016-11-26 16:52:40 -08:00
Calvin Lee 8461cc411e streams: refactor stream creation code path.
Refactor list_to_streams and create_streams_if_needed to take a list
of dictionaries, instead of a list of stream names.  This is
preparation for being able to pass additional arguments into the
stream creation process.

An important note: This removes a set of validation code from the
start of add_subscriptions_backend; doing so is correct because
list_to_streams has that same validation code already.

[with some tweaks by tabbott for clarity]
2016-11-26 16:48:59 -08:00
Tomasz Kolek a79acf854f docs: Automate creation of Hubot documentation lozenges.
This removes a bunch of semi-duplicated code.
2016-11-26 15:27:54 -08:00
Rishi Gupta 4b183cd526 domain migration: Remove several instances of get_realm.
Remove the easy to remove instances of get_realm.
2016-11-26 15:19:56 -08:00
Umair Khan 0536aeba4d Django 1.10: Use same cache prefix for JS tests.
Previously, the key prefix was based on the process id due to which
the JS tests couldn't properly flush user profiles from the cache as
our application spans over multiple processes. This problem becomes
apparent when in json_change_settings view after changing the user_profile
the tornado views continue to get the cached user profile corresponding
to their process id.
2016-11-26 15:10:50 -08:00
Calvin Lee 7a3ef2b0eb tests: Add assert_in_success_response in ZulipTestCase.
Clean up the instances of self.assertIn("string", result.content.decode("utf-8")),
and replace them with self.assert_in_response("string").

Fixes: #2313
2016-11-26 14:52:44 -08:00
Steve Howell 6ed1dc9341 Create api docs from url coverage data from tests.
I also retire tools/analyze-url-coverage in this commit,
since the API docs cover most of the functionality.
2016-11-26 14:49:23 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Anders Kaseorg 573ec14955 Remove shebang line from non-scripts
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Steve Howell 92030204bb Look for untested urls more exhaustively. 2016-11-26 09:52:37 -08:00
Steve Howell 1527823b9f code cleanup: Remove some unused variables in test classes. 2016-11-26 09:52:37 -08:00
Rishi Gupta 17f71befb4 mypy: Change six.text_type to typing.Text for a few files.
Preparation for GCI.
2016-11-25 13:55:38 -08:00
Steve Howell 5f5e6b6d83 tests: Enforce 100% URL coverage.
We now instrument URL coverage whenever you run the back end tests,
and if you run the full suite and fail to test all endpoints, we
exit with a non-zero exit code and report failures to you.

If you are running just a subset of the test suite, you'll still
be able to see var/url_coverage.txt, which has some useful info.

With some tweaks to the output from tabbott.

Fixes #1441.
2016-11-18 18:10:29 -08:00
Vladislav Manchev d7e1e4a2c0 Add initial implementation of custom realm filters.
This PR was abandoned by Vladislav and then substantially modified by
Igor Tokarev and Tim Abbott to complete it and fix a number of bugs.

Fixes #544.
2016-11-17 17:11:25 -08:00
hackerkid cdb716aa63 bugdown: Add explanation for unicode-emoji JS regex. 2016-11-17 10:44:07 -08:00
Steve Howell 3a8282ff44 Add test coverage for /notify_tornado.
We now have test coverage for all of our endpoints.
2016-11-17 10:11:46 -08:00
Tomasz Kolek 074571dddc Fix English grammar in push/remove tag message. 2016-11-15 10:48:12 -08:00
Tomasz Kolek 1feedbe346 Add github webhook integration. 2016-11-15 10:48:12 -08:00
Umair Khan 2314adc4fc Django 1.10: Compare CSRF tokens after unsalting them.
For reference: https://docs.djangoproject.com/en/1.10/ref/csrf/#how-it-works
2016-11-15 09:34:08 -08:00
Umair Khan 29254142f6 Django 1.10: Run maybe_update_realm_filter when it's needed. 2016-11-15 09:34:08 -08:00
hackerkid 67120d7a9d Support backend unicode-emoji rendering without whitespace. 2016-11-14 21:02:18 -08:00
Tim Abbott 8a5ae2893e send_pm_if_empty_stream: Clean up unnecessary nesting.
This has no functional changes.
2016-11-14 20:40:05 -08:00
Tim Abbott b09c2369c1 send_pm_if_empty_stream: Use realm object to check cross-realm.
The previous check didn't work correctly in the event that the target
stream did not exist.
2016-11-14 20:36:46 -08:00
Tim Abbott af675998d5 Fix send_pm_if_empty_stream with nonexistent streams. 2016-11-14 20:32:29 -08:00
hackerkid 8c83b34ad5 Support backend emoji rendering without whitespace. 2016-11-11 16:18:57 -08:00
Tomasz Kolek 4659abae65 test_fixtures: handle a case when given database does not exist.
Sometimes i.e. when you force close (ctrl-c) during ./tools/test-backend
a next execution might raise an error becuase DATBASE_NAME does not exist.
2016-11-11 15:40:10 -08:00
Tomasz Kolek ed1d7f7997 Add is_truncated param to git integrations push commits event.
Add is_truncated param for cases when we know that there were more commits
but we dont know actual number and want just inform about existing of the rest.
2016-11-11 15:36:45 -08:00
Tomasz Kolek 6be27a1396 Add get_push_tag_event_message function to git integrations common (git.py). 2016-11-11 15:36:45 -08:00
Rishi Gupta 5681ee90fa models.py: Rename split_email_to_domain to email_to_domain.
In order to better match models.email_to_username.
2016-11-11 15:26:51 -08:00
Rishi Gupta c1713c9659 Prevent code from using email domain to determine realm when subdomains.
Also removes the intermediate step of going through Realm.domain in the
non-subdomains case. Part of a larger project to remove Realm.domain
entirely.
2016-11-11 15:26:51 -08:00
Rishi Gupta 411a7b08e0 bulk_create_users: No longer de-alias user email addresses.
bulk_create.bulk_create_users is only called by initialize_voyager_db.py and
populate_db.py, both of which only have realms with single domains.

Part of a larger project to remove the Realm.domain field.
2016-11-11 15:26:51 -08:00
Tim Abbott 3b145b2370 add_new_user_history: Fix ordering of created UserMessage rows.
This has no functional effect because user_message IDs aren't used,
but will make the database cleaner.
2016-11-10 20:39:34 -08:00
Tim Abbott 08ca209aed add_new_user_history: Fix race with new messages arriving during signup.
Previously, if a new message arrived between when a user is subscribed
to the default streams and when the user's initial messages are
queried, we would try to create two UserMessage rows for the same
Message, resulting in an IntegrityError crash.  We fix this and add a
test for that race condition.
2016-11-10 20:38:27 -08:00
Tim Abbott 80c2df616e signup: Extract add_new_user_history. 2016-11-10 20:37:03 -08:00
sinwar 4582a98c09 tests: Split out ZulipTestCase and WebhookTestCase to a separate file.
Fixes #1671.
2016-11-10 19:29:43 -08:00
Igor Tokarev 852bc6b491 bugdown: Re-enable support for italics in bugdown.
Fixes: #1103.
2016-11-10 19:14:40 -08:00
Umair Khan 010ab0637d Django 1.10: Upgrade integration urls. 2016-11-10 16:20:03 +05:00
Umair Khan cfded8b5af Django 1.10: Resolve QuerySet returned by model_to_dict.
Django 1.10 resolves ManyToManyField into a QuerySet in model_to_dict.
This commit further resolves the QuerySet to the primary keys.
2016-11-09 15:29:58 -08:00
paxapy 6d93b3b60c lint: Fix E703 pep8 violations. 2016-11-09 15:18:35 -08:00
Steve Howell b5ef679360 Add get_topic_history_for_stream(). 2016-11-09 13:56:27 -08:00
Steve Howell a03a7d4c95 peer_remove: Send user_id, not email, for unsubscribe events. 2016-11-08 15:36:29 -08:00
paxapy ff1e97603d context_processors: use a common context for emails.
Fixes #1611.
2016-11-08 11:37:16 -08:00
Tim Abbott 56552007bf create_user: Disable stream desktop notifications by default.
This feature intended to be disabled by default months ago, but due to
the override code deleted in this commit, was actually still on by default.
2016-11-08 07:48:49 -08:00
Arpith Siromoney fbe8171813 Typing indicators backend: Pass list of user ids to send_event
send_event() expects a list of user ids (ints) except for the special case
of messages. This commit:
1. Fixes this in the call to send_event() in do_send_typing_notification()
2. Renames the variables in do_send_typing_notification() to better reflect
their content (for example, recipient_ids instead of recipients).
3. Renames the id field in the dicts sent in the typing event body (sender,
recipients) to user_id.
4. Adds assertions to the tests to verify that the tornado event user ids
are the same as the recipients in the event body.
5. Adds assertions to the tests to verify that the tornado event user
ids and the recipient user ids (in the event body) are the same as the
expected user ids (obtained from the emails using
get_user_profile_by_email)
6. Changes all assertTrues to assertEquals in the tests

This fixes #2151.
2016-11-07 22:28:09 -08:00
Meena Rajan 30acb23314 bugdown: Add support for strikethrough in markdown processor.
[Tweaked to move tests to bugdown_data.json, add additional tests, and
add frontend processor support by tabbott]
2016-11-07 22:26:38 -08:00
Rishi Gupta 9e6e1a1e69 Remove several instances of datetime.datetime.utcnow().
Change to the timezone-aware django.utils.timezone.now() where django is
available.
2016-11-07 20:13:53 -08:00
Rishi Gupta 9e5ec2fd29 notifications.send_future_email: Use timezone aware times.
Previous behavior raises an AmbiguousTimeError during daylight savings hour!
2016-11-07 20:13:53 -08:00
Rishi Gupta a416a9aeae test_runner.py: Fix "Import unexpectedly succeeded" messages in tests.
Previously we were getting this message whether or not the import would have
succeeded. The regression was introduced in ea050d5f.
2016-11-07 20:13:17 -08:00
Kevin Chen 6107c877e8 bugdown: Add option to support "file:///" as hyperlink.
This contains contributions from Tim Abbott and Igor Tokarev.

Fixes #380.
2016-11-06 22:30:05 -08:00
umkay f4c621ffe3 admin: Enable admins to toggle supported auth methods via UI.
Add a table to the administration page that will allow realm admins to
activate and deactivate the supported authentication methods for that
realm.
2016-11-06 16:29:35 -08:00
Rishi Gupta 5e6d4da8a8 realm creation flow: Prevent disposable email signups.
Check user's email address against a list of 2000 disposable email
domains. Does not affect the create_realm management command.
2016-11-05 10:37:15 -07:00
Rishi Gupta 2033381d24 realm creation flow: Restrict subdomains one can create via the web flow.
Disallow Realm.string_id's like "streams", "about", and several hundred
others. Also restrict string_id's to be at least 3 characters long, and only
use characters in [a-z0-9-].

Does not restrict realms created by the create_realm.py management command.
2016-11-05 10:37:15 -07:00
Rishi Gupta e3646f28bd test_helpers.py: Fix default args for submit_reg_form_for_user.
Previously realm_name and realm_subdomain defalted to None, which when
posted to /accounts/register, are submitted as u'None'. "None" is an invalid
Realm.subdomain, since subdomains can't have capital letters.
2016-11-04 22:01:53 -07:00
Mohsen Ibrahim 19b01d74fa bug fix: Send stream notifications to all users for public streams.
If a stream is public, we now send notifications to all realm users
if the name or description of the stream changes.  For private
streams, the behavior remains the same.

We do this by introducing a method called
can_access_stream_user_ids().

(showell helped with this fix)

Fixes #2195
2016-11-04 21:41:41 -07:00
Steve Howell c405b67138 tests: Ignore SAVEPOINT queries in queries_captured().
We use the queries_captured() context manager in our tests
to capture queries that happen during certain actions.  Usually
we use the list of queries to validate that we're not doing
too many database calls.

For most tests the database calls are fairly deterministic,
but SAVEPOINT-related things seem to be more random, and the
number of savepoints is usually not relevant to what the test
is trying to prevent, which is more serious problems like
O(N) database fetches.
2016-11-04 21:30:36 -07:00
Steve Howell 2088e8420f User user_id, not email, in peer_add events. 2016-11-04 11:37:14 -07:00
Steve Howell c3f2740481 Remove email_dict from gather_subscriptions_helper(). 2016-11-04 11:37:14 -07:00
Steve Howell 38f2789152 Remove page_params.email_dict. 2016-11-04 11:37:14 -07:00
Umair Khan fe1ba6f3eb Django 1.10: Deprecate request.REQUEST. 2016-11-04 10:20:24 -07:00
Umair Khan d81446805c Django 1.10: Use `caches` object to access cache. 2016-11-04 10:06:00 -07:00
Rishi Gupta e4334f4735 do_create_realm: Remove domain argument from do_create_realm.
The domain argument was being used in do_create_realm to create a
RealmAlias. Callers are now responsible for creating a RealmAlias
when desired.
2016-11-03 13:53:01 -07:00
Rishi Gupta c0f1b5d9f6 Change Realm defaults to Community defaults.
Previously, we set restrict_to_domain and invite_required differently
depending on whether we were setting up a community or a corporate
realm. Setting restrict_to_domain requires validation on the domain of the
user's email, which is messy in the web realm creation flow, since we
validate the user's email before knowing whether the user intends to set up
a corporate or community realm. The simplest solution is to have the realm
creation flow impose as few restrictions as possible (community defaults),
and then worry about restrict_to_domain etc. after the user is already in.

We set the test suite to explictly use the old defaults, since several of
the tests depend on the old defaults.

This commit adds a database migration.
2016-11-03 13:53:01 -07:00
Tim Abbott 90f57c92d4 compute_mit_user_fullname: Fix result for Python 3.
DNS replies are bytes, and usernames are strings.
2016-11-03 13:40:43 -07:00
Steve Howell 9b04ec7472 Send dicts for cross realm users to JS.
We now send dictionaries for cross-realm bots.  This led to the
following changes:

* Create get_cross_realm_dicts() in actions.py.
* Rename the page_params field to cross_realm_bots.
* Fix some back end tests.
* Add cross_realm_dict to people.js.
* Call people.add for cross-realm bots (if they are not already part of the realm).
* Remove hack to add in feedback@zulip.com on the client side.
* Add people.is_cross_realm_email() and use it in compose.js.
* Remove util.string_in_list_case_insensitive().
2016-11-03 09:51:23 -07:00
Steve Howell 6324fe9ddf Rename get_cross_realm_users to get_cross_realm_emails. 2016-11-03 09:42:27 -07:00
Tim Abbott bfff9e1968 do_send_messages: Fix sending messages to feedback-bot.
This feature was broken in the message_to_dict refactoring.
2016-11-02 23:42:05 -07:00
Rishi Gupta 70c29d8f90 do_create_realm: Remove "(open realm)" indication in sign-up message.
In preparation for a change to do_create_realm where we will use the
database default for restricted_to_domain rather than computing it within
do_create_realm, and due to which do_create_realm will no longer know
whether we are creating an open realm or not.
2016-11-02 22:46:34 -07:00
Rishi Gupta db0e509422 do_create_realm: Replace domain argument with string_id.
Turns string_id into a required argument, and domain into an optional
argument.
2016-11-02 22:46:34 -07:00
Rishi Gupta 64bcd71d6e models.Realm: Rename subdomain to string_id.
Does a database migration to rename Realm.subdomain to
Realm.string_id, and makes Realm.subdomain a property.  Eventually,
Realm.string_id will replace Realm.domain as the handle by which we
retrieve Realm objects.
2016-11-02 22:46:34 -07:00
Tomasz Kolek 4dd8cd9879 Replace "Commit" to commits sha in commits comment events in git integrations. 2016-11-02 22:19:51 -07:00
Tomasz Kolek 6ad4cce35c Add object number/id to mention about Issue, PR, etc. to all git integrations. 2016-11-02 22:19:51 -07:00
Steve Howell 6659664e59 Simplify/fix cross-realm validation in recipient_for_emails().
We now simply exclude all cross-realm bots from the set of emails
under consideration, and then if the remaining emails are all in
the same realm, we're good.

This fix changes two behaviors:
* You can no longer send a PM to an ordinary user in another realm
  by piggy-backing a cross-realm bot on to the message.  (This was
  basically a bug, but it would never manifest under current
  configurations.)
* You will be able to send PMs to multiple cross-realm bots at once.
  (This was an arbitrary restriction.  We don't really care about this
  scenario much yet, and it fell out of the new implementation.)
2016-11-02 22:10:49 -07:00
Umair Khan f300c12535 Accept from_confirmation kwarg in submit_reg_form_for_user. 2016-11-02 17:50:40 -07:00
paxapy 1d1c41551b bugdown: Auto-linkify #StreamName. 2016-11-01 19:59:21 -07:00
paxapy d965ff16fd bugdown: refactor AutoLink to use VerbosePattern.
This makes it possible to reuse the verbose pattern logic for other
regexs as well.
2016-11-01 19:50:59 -07:00
Steve Howell 768a04d2fa Send user_ids for update events.
Here we handle a few flavors of realm_bot/update
and realm_user/update.  (Different events send different
attributes.)
2016-10-28 13:40:14 -07:00
Steve Howell 3fd7ab950a Send user_id to the client when removing bots/users.
This touches these events:

    realm_bot/remove
    realm_user/remove
2016-10-28 13:40:14 -07:00
Steve Howell 7883cecf28 Send user_id to the client in more places.
This commit touches:

    realm_bot/add
    realm_user/add
    page_params.bot_list
    page_params.people_list
2016-10-28 13:40:14 -07:00
Arpith Siromoney c5510599ef Typing backend: use dict instead of user profile for sender in event.
Fixes #2110.
2016-10-26 13:12:18 -07:00
K.Kanakhin 39e0886361 retention-policy: Add tool to determine expired messages.
This is a first step towards implementing a message retention policy
feature.

- Add Realm model message_retention_days field to setup
  messages expired period for realm.
- Add migration.
- Add tool to get expired messages for each Realm.
- Add tests to cover tool for getting expired messages.
2016-10-25 15:38:08 -07:00
umkay 1d077d1a4f bugdown: Remove safe_mode dependencies.
Now that we have updated python-markdown, we remove the deprecated
safe_mode. We used safe_mode to escape raw html, so now instead we
pass in an EscapeHtml markdown extension to the markdown engine.

See https://pythonhosted.org/Markdown/release-2.6.html for details on
the deprecation.

Fixes: #2037 (also addresses the remaining piece of #2043).
2016-10-25 15:22:18 -07:00
Arpith Siromoney 01665d9c3c Extract get_recipient_user_profiles.
This creates a common function for extracting recipients, which both
do_send_messages and do_send_typing_notifications can use.
2016-10-25 13:37:01 -07:00
Tomasz Kolek 4790316b57 Add user setting option to always send push notifications.
Add option in user's settings for getting mobile push notifications
even if a Zulip browser is online.  Default is False.

Fixes: #1596.
2016-10-25 10:52:29 -07:00
Rishi Gupta 537ee75761 Create RealmAlias entries for Realm.domain.
This is a preliminary step towards eliminating the realm.domain field
in favor of realm.subdomain.  Includes a database migration to create
these for existing realms.
2016-10-25 10:06:39 -07:00
Brock Whittaker fca61b2031 Add medium size avatars for use on the user's own settings page.
This adds a medium (500px) size avatar thumbnail, that can be
referenced as `{name}-medium.png`.  It is intended to be used on the
user's own settings page, though we may come up with other use cases
for high-resolution avatars in the future.

This will automatically generate and upload the medium avatar images
when a new avatar original is uploaded, and contains a migration
(contributed by Kirill Kanakhin) to ensure all pre-existing avatar
images have a medium avatar.

Note that this implementation does not provide an endpoint for
fetching the medium-size avatar for another user.

[substantially modified by tabbott]
2016-10-25 09:42:14 -07:00
Tim Abbott e714ddbfc8 gather_subscriptions_helper: Fix mypy error. 2016-10-22 22:37:05 -07:00
Tim Abbott c59cdbb92d gather_subscriptions: Fix missing subscriber data in email_dict.
When we added data on never_subscribed streams to what
populate_subscribers is called on, we failed to add the corresponding
data on subscribers to email_dict, the mapping of user IDs to emails
for the subscribers.
2016-10-22 21:37:05 -07:00
Tim Abbott 14018353fc Fix never_subscribed streams being sent for Zephyr mirroring.
Because in the Zephyr world, stream names can be a secret, and also
Zephyr mirroring tends to involve many thousands of streams, we
shouldn't send this data.
2016-10-22 21:36:11 -07:00
Tim Abbott 5bea2f5e20 Remove unused AVATAR_FROM_SYSTEM code.
This is some of the code we'd need if we wanted to have Zulip generate
avatars for things.  Since it is so little useful code, and it's not
clear we will need this feature ever, we can remove this code to make
the codebase less confusing.  It'd be easy to dig this out of history
if we ever want it.

Fixes #2101.
2016-10-22 19:48:50 -07:00
Steve Howell 79b6f3d39d Create ZulipTestCase.make_stream(). 2016-10-22 18:09:34 -07:00
Umair Khan 8891731d7d is_template_database_current: Open file in write mode. 2016-10-21 08:35:30 -07:00
Umair Khan b894135fae Check postgres-init-dev-db in is_template_database_current. 2016-10-21 08:35:30 -07:00
Umair Khan 176825d335 Refactor is_template_database_current.
Now all the files whose hash is to be checked are passed as a list
argument.
2016-10-21 08:35:30 -07:00
Umair Khan bf4e0800e3 is_template_database_current: Create hashes in var/test_db_status. 2016-10-21 08:35:30 -07:00
Tomasz Kolek 89d53a0640 Change name of PR subject template to be applicable for issues. 2016-10-20 23:15:30 -07:00
Tomasz Kolek 289f400c8c Add git integrations method to build message for issues events. 2016-10-20 23:15:30 -07:00
Arpith Siromoney ee97ba04fe Add an API endpoint to send typing notification events.
POST to /typing creates a typing event
Required parameters are 'op' ('start' or 'stop') and 'to' (recipient
emails). If there are multiple recipients, the 'to' parameter
should be a JSON string of the list of recipient emails.
The event created looks like:
{
  'type': 'typing',
  'op': 'start',
  'sender': 'hamlet@zulip.com',
  'recipients': [{
    'id': 1,
    'email': 'othello@zulip.com'
  }]
}
2016-10-20 23:05:18 -07:00
Tim Abbott 3d183ffe23 text_fixtures: Fix case where source_hash_file doesn't exist.
We unfortunately just broke provisioning because we didn't correctly
handle the case where the source_hash_file didn't exist.
2016-10-20 22:21:45 -07:00
Steve Howell 409a84698c Handle peer_remove events correctly for subscriptions.
We now send peer_remove events to folks who have never subscribed
to the streams (except for private streams and zephyr).

We also use logic that is more similar to how
bulk_add_subscriptions() works.
2016-10-20 22:05:19 -07:00
Steve Howell 68ae4da626 Extract query_all_subs_by_stream().
This code was in bulk_add_subscriptions, but we want the same logic
to eventually happen in bulk_remove_subscriptions.
2016-10-20 22:05:19 -07:00
Steve Howell d13b0554b3 Remove emails_by_stream in bulk_add_subscription().
There are so many collections in this function, it's hard to keep
them all straight.  This one is fairly redundant, so it was easy
to remove.
2016-10-20 22:05:19 -07:00
Steve Howell d706bfa69f Move some code from notify_subscriptions_removed().
There are two reasons for this change.  First, we want to be
consistent with notify_subscriptions_added(), which doesn't
handle "peer" events.  Second, we want to fix this code in a
subsequent commit not to do one user at a time, which is
inefficient.
2016-10-20 22:05:19 -07:00
Steve Howell dda4c61f2a Remove do_remove_subscription(). 2016-10-20 22:05:19 -07:00
Steve Howell 4ae117ed14 Use bulk_remove_subscriptions() in do_deactivate_stream().
We are deprecating do_remove_subscription().
2016-10-20 22:05:19 -07:00
Steve Howell 44bbece5a6 Remove do_remove_subscription() calls in tests.py. 2016-10-20 22:05:19 -07:00
Steve Howell b2d06fad42 Extract get_peer_user_ids_for_stream_change(). 2016-10-20 22:05:19 -07:00
Steve Howell 5eee77fe81 Remove do_add_subcription(). 2016-10-20 22:05:19 -07:00
Steve Howell 78b73a0c91 Replace do_add_subscription() in test_helpers. 2016-10-20 22:05:19 -07:00
Umair Khan 45166cb39c is_template_database_current: Check populate_db and postgres-init-test-db.
Compare the hash of 'zilencer/management/commands/populate_db' with
'var/populate_db_hash' and 'tools/setup/postgres-init-test-db' with
'var/postgres_init_test_db_hash' and if any comparison fails rebuild
the database.

With fixes from tabbott.
2016-10-20 22:04:47 -07:00
K.Kanakhin 7bc0c7f0d5 bugdown: Fix patterns ordering in Markdown extension.
- Put 'modal_link' and 'link' inline patterns after avatar inline pattern
  to correct applying them to parsed text.

Fixes:  #1654
2016-10-20 11:18:41 -07:00
Tim Abbott 50356cc325 event_queue: Eliminate buggy need_timeout heartbeat code.
As best I can tell, this option was completely confused in two ways:
* The name is confusing; it actually controls whether we _clear_ the
  timeout associated with the current handler
* It's not clear why one would not want it to be unconditionally true.

From reading the history, I'm pretty sure I had just misread the code
when I created this.

And this caused a real bug; a later refactoring caused us to basically
never cancel the timeouts, which in turn resulted in 90% of all events
traffic being hearbeats with a much lower frequency (~5s) than the
intended 45s.  Removing this code fixes that nasty bug.
2016-10-19 14:32:49 -07:00
Tim Abbott a6e779a56c events: Add event types to logging.
This should make it a lot easier to analyze our events traffic and
debug potential issues with it.
2016-10-19 14:32:16 -07:00
Rishi Gupta 83f902953a newsletter signup: Update merge_vars.
merge_vars is the user meta data we store in mailchimp. This commit changes
what we store from realm.domain to realm.id, since realm.domain is being
deprecated, and changes the OPTIN_TIME to a nicer format.
2016-10-18 19:29:12 -07:00
Brock Whittaker 77233c7e3b bugdown: Distinguish between YouTube Videos and Image Previews.
This distinguishes between YouTube Videos and Image Previews by adding
a particular “youtube-video” class to the preview along with changing
the title to the video ID rather than the link. This serves to allow
the lightbox to ID when a lightbox preview should be treated like a
YouTube video rather than an image preview.

This also modifies the tests in bug down to expect a youtube-video class
along with the title to just be the video ID on YouTube rather than the
entire URL link.
2016-10-18 14:45:47 -07:00
Tim Abbott 56d0cc69e9 socket: Remove unused djsession_engine. 2016-10-17 23:19:28 -07:00
Mohsen Ibrahim bfe5787197 Fix Django.utils.importlib deprecation warnings.
This changes `from django.utils.importlib import import_module` to
`from importlib import import_module`, as `django.utils.importlib` will
be removed in django version 1.9.
2016-10-17 23:19:28 -07:00
Mohsen Ibrahim e9899e4dc9 bugdown: Fix some Python-Markdown deprecation warnings.
Python-Markdown changed how one is supposed to declare extensions
being used.
2016-10-17 23:13:41 -07:00
Tim Abbott 63a49be09e event_queue: Import tornado.ioloop correctly.
We actually use tornado.ioloop, not the root tornado module, and while
the current code does work, it causes mypy errors with the incremental
cache.
2016-10-17 17:57:46 -07:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Tim Abbott e37b6488b8 Bugdown: Remove checked-in CodeHilite extension.
The changes that required us to fork this extension had been merged
into upstream CodeHilite, so we can remove it and switch to using the
version that comes with python-markdown.
2016-10-16 00:40:08 -07:00
umkay c028815385 bugdown: Update python-markdown to version 2.6.7.
This updates Bugdown to reflect the changes in the updated
markdown. In particular, we now pass a default config object in the
__init__ for the Bugdown extension, update the make_md_engine function
to take kwargs as opposed to a config list, and have UListProcessor
inherit from ulist as opposed to olist (which no longer works).

We update the (forked from upstream) fenced_code extension's
makeExtension to take args and kwargs, and update
FencedBlockPreprocessor __init__ method with updated Codehilite
arguments.

We update the (forked from upstream) Codehilite extension to
mirror the logic with the latest upstream Codehilite:
	Add parse_hl_lines function
	update makeExtension to take args and kwarfs instead of config
	list
	Add regex for highlight lines
	use linenums instead of linenos
	use get_formatter_by_name instead of HtmlFormatter
	user get_lexer_by_name instead of TextLexer
	add hl_lines and use_pygments arguments to the codehlite
	constructor
2016-10-15 23:38:32 -07:00
Tomasz Kolek 2941c1f517 Move converting commits data to text to separate function.
Create get_commits_content function for converting
commits list to text.
2016-10-14 16:54:13 -07:00
Tomasz Kolek d2cedd3667 Rename consts in lib/webhooks/git.py to make it more general.
Rename:
   PUSH_COMMITS_LIMIT to COMMITS_LIMIT
   PUSH_COMMIT_ROW_TEMPLATE to COMMIT_ROW_TEMPLATE
   PUSH_COMMITS_MORE_THAN_LIMIT_TEMPLATE to COMMITS_MORE_THAN_LIMIT_TEMPLATE
2016-10-14 16:54:13 -07:00
Tomasz Kolek ee70cc5141 Add git integration method to build message for pull requests events. 2016-10-14 16:54:13 -07:00
umkay 278574a6ec Fix cursor() arguments for connection classes.
In order to use the latest version of psycopg2 with the latest version
of sqlalchemy we must integrate the changes made in the 2.4.6 release of
psycopg2. See
c86ca7687f
and the release notes for psycopg2 2.4.6.
2016-10-14 13:35:06 -07:00
Rishi Gupta 82b814a1cd analytics: Simplify frequency and measurement interval options.
Change the CountStat object to take an is_gauge variable instead of a
smallest_interval variable. Previously, (smallest_interval, frequency)
could be any of (hour, hour), (hour, day), (hour, gauge), (day, hour),
(day, day), or (day, gauge).
The current change is equivalent to excluding (hour, day) and (day, hour)
from the list above.

This change, along with other recent changes, allows us to simplify how we
handle time intervals. This commit also removes the TimeInterval object.
2016-10-14 10:18:37 -07:00
Tim Abbott 0f1468e64f do_update_message_flags: Use access_message for check. 2016-10-11 17:20:15 -07:00
Tim Abbott f1a399a4e1 message: Create new access_message library function.
With reactions and other upcoming features, we'll be adding several
places where we need to check whether a particular user can access a
particular message.  It's best to just have a single helper function
for this purpose that we can use everywhere.
2016-10-11 17:17:19 -07:00
Eklavya Sharma 41587edd2e bugdown: Change parameter name.
Change the parameter name of some functions from 'md' to 'content',
since the name 'md' seems to be the reason why this parameter was
wrongly annotated.
2016-10-11 11:31:01 -07:00
Eklavya Sharma f35d0f1c75 bugdown: Fix type annotations.
In some functions the first parameter was wrongly annotated as
`markdown.Markdown`. Change that to `text_type`.
2016-10-11 11:31:01 -07:00
Eklavya Sharma ccce3ec0be zerver/lib/digest.py: Fix a type annotation. 2016-10-11 11:27:35 -07:00
Tomasz Kolek 312905007e Add EMPTY_SHA const to zerver/lib/webhooks/git.py. 2016-10-10 17:39:05 -07:00
Tomasz Kolek b69b5e55ce Add git integration method to build message for remove branch events.
Add get_remove_branch_event_message method to zerver/lib/webhooks/git.py
for normalizing remove branch message in all git integrations.
2016-10-10 17:39:05 -07:00
Tomasz Kolek 5f7c1519d1 Add git integration method to build message for force push events.
Add get_force_push_commits_event_message method to zerver/lib/webhooks/git.py
for normalizing force push message in all git integrations.
2016-10-10 17:39:05 -07:00
Tomasz Kolek 32934a02e9 Add template for subject that contains repo name and branch name. 2016-10-10 17:39:05 -07:00
Tomasz Kolek 8697a1e5a2 Add common git integration method to build message for push events.
This adds get_push_commits_event_message method to
zerver/lib/webhooks/git.py for normalizing push messages in all git
integrations.
2016-10-10 17:32:41 -07:00
sonali0901 5623fa9994 request docs: Fix typo in spelling of "parameter". 2016-10-09 11:13:49 -07:00
Rishi Gupta ea050d5fd8 test_runner: Fix error message for failed import.
Previously, we suggested running
`python -c import zerver.tests.test_mytest`
when importing test_mytest failed, which doesn't work.

This commit adds the missing quotes, making it
`python -c 'import zerver.tests.test_mytest'`
2016-10-08 11:51:53 -07:00
Rishi Gupta 777fcaa6a0 Add new organization type field to Realm objects.
Adds a new field org_type to Realm.  Defaults for restricted_to_domain
and invite_required are now controlled by org_type at time of realm
creation (see zerver.lib.actions.do_create_realm), rather than at the
database level.  Note that the backend defaults are all
org_type=corporate, since that matches the current assumptions in the
codebase, whereas the frontend default is org_type=community, since if
a user isn't sure they probably want community.

Since we will likely in the future enable/disable various
administrative features based on whether an organization is corporate
or community, we discuss those issues in the realm creation form.
Before we actually implement any such features, we'll want to make
sure users understand what type of organization they are a member of.

Choice of org_type (via radio button) has been added to the realm
creation flow and the realm creation management command, and the
open-realm option removed.

The database defaults have not been changed, which allows our testing code
to work unchanged.

[includes some HTML/CSS work by Brock Whittaker to make it look nice]
2016-10-05 17:01:46 -07:00
Tomasz Kolek dbeab6aa6f Optimize checks of test database state by moving into Python.
Previously, the generate-fixtures shell script by called into Django
multiple times in order to check whether the database was in a
reasonable state.  Since there's a lot of overhead to starting up
Django, this resulted in `test-backend` and `test-js-with-casper`
being quite slow to run a single small test (2.8s or so) even on my
very fast laptop.

We fix this is by moving the checks into a new Python library, so that
we can avoid paying the Django startup overhead 3 times unnecessarily.
The result saves about 1.2s (~40%) from the time required to run a
single backend test.

Fixes #1221.
2016-10-05 10:40:19 -07:00
umkay d260a22637 Add a new statistics/analytics framework.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
  aggregating user/stream level stats into realm level stats
* A management command for pulling the data

Note that counts.py was added to the linter exclude list due to errors
around %%s.
2016-10-04 17:18:54 -07:00
Steve Howell b2f84f0fa4 Move render_markdown() to lib/message.py.
This removes a bugdown circular dependency.
2016-10-04 11:34:53 -07:00
Steve Howell 7fb992dba3 Simplify and "fix" render_old_messages management command.
The command to render old messages now looks for all messages
not matching the bugdown version, and it no longer directly calls
into model code.  We should still be extremely cautious about
using this code.
2016-10-04 11:31:20 -07:00
Steve Howell 6b71f5bd5f Inline most calls to set_rendered_content().
This is part of breaking the circular dependency on
bugdown in models.py.  A subsequent commit will fully
kill off set_rendered_content().
2016-10-04 11:31:20 -07:00
Steve Howell 583a6bbadd Extract zerver/lib/message.py.
This pulls message-related code from models.py into a new
module called message.py, and it starts to break some bugdown
dependencies.  All the methods here are basically related to
serializing Message objects as dictionaries for caches and
events.

    extract_message_dict
    stringify_message_dict
    message_to_dict
    message_to_dict_json
    MessageDict.to_dict_uncached
    MessageDict.to_dict_uncached_helper
    MessageDict.build_dict_from_raw_db_row
    MessageDict.build_message_dict

This fix also removes a circular dependency related
to get_avatar_url.

Also, there was kind of a latent bug in Message.need_to_render_content
where it was depending on other calls to Message to import bugdown
and set it globally in the namespace.  We really need to just
eliminate the function, since it's so small and used by code that
may be doing very sketchy things, but for now I just fix it.  (The
bug would possibly be exposed by moving build_message_dict out to the
library.)
2016-10-04 11:31:20 -07:00
Steve Howell ac994fdd51 Move three functions from models.py to lib/cache.py.
I move these three functions to lib/cache.py:

    to_dict_cache_key_id
    to_dict_cache_key
    flush_message

This will prepare us for a more significant refactoring that
eventually breaks down some circular dependencies with
Message and bugdown.
2016-10-04 11:31:20 -07:00
Tim Abbott ac2007dd9b resize_avatar: Add a size option. 2016-10-02 21:29:04 -07:00
Tim Abbott 87b9017845 upload: Extract get_avatar_url and add to UploadBackend. 2016-10-02 21:20:43 -07:00
Tim Abbott 22fd7ba02a avatar: Move avatar hash computations to their own file. 2016-10-02 21:19:10 -07:00
Tim Abbott a4e5450ace subdomains: Add support for aliases of the root subdomain.
We default to counting "www" as such an alias.
2016-09-27 23:25:07 -07:00
hackerkid ea39fb2556 Add option for hosting each realm on its own subdomain.
This adds support for running a Zulip production server with each
realm on its own unique subdomain, e.g. https://realm_name.example.com.

This patch includes a ton of important features:
* Configuring the Zulip sesion middleware to issue cookier correctly
  for the subdomains case.
* Throwing an error if the user tries to visit an invalid subdomain.
* Runs a portion of the Casper tests with REALMS_HAVE_SUBDOMAINS
  enabled to test the subdomain signup process.
* Updating our integrations documentation to refer to the current subdomain.
* Enforces that users can only login to the subdomain of their realm
  (but does not restrict the API; that will be tightened in a future commit).

Note that toggling settings.REALMS_HAVE_SUBDOMAINS on a live server is
not supported without manual intervention (the main problem will be
adding "subdomain" values for all the existing realms).

[substantially modified by tabbott as part of merging]
2016-09-27 23:24:14 -07:00
Tim Abbott bbab3cdc30 test_helpers: Add HostRequestMock helper class. 2016-09-27 21:18:29 -07:00
Tim Abbott 8442e9249c Reflow annotation for generic_bulk_cached_fetch.
This is a test of mypy's new support for annotating functions that
take lots of arguments.
2016-09-27 20:36:56 -07:00
Brock Whittaker 98937ac539 bugdown: Set "title" attribute to actual image title when available.
This sets the “title” attribute on the image to the actual title of
the image specified by the user in their markdown, rather than just
the URL of the full link to it.
2016-09-27 17:34:05 -07:00
Umair Khan fd9dd51d16 Url encoded name of the file should be an ascii.
The url encoded name of the file should not be a unicode. This
results in an error when we later try to unquote it.

Fixes: #1803
2016-09-26 21:13:34 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
Supriya 4959e8d10a tests: Extract and use assert_max_length test helper.
This makes it much more explicit in which cases we're checking
equality or a maximum in our use of `assert_length`.

Fixes #1400.
2016-09-25 15:44:58 -07:00
Steve Howell dfa416cfc8 Remove last_reminder_tzaware().
We can now rely on UserProfile.last_reminder being time zone
aware, or even if it isn't, it's a self-correcting problem the
first time a reminder is sent.  (It's a non-problem to be off
by a few timezones if somebody still has an old value there, because
they will still be outside the 1-minute nag window even with the
timezone disparity.)
2016-09-24 16:56:34 -07:00
Rishi Gupta 03ce3e5fa4 test_signup.py: Refactor getting confirmation_urls to test_helpers.py
It is a bit of tricky / scary looking code that was repeated several times
in test_signup.
2016-09-23 15:44:28 -07:00
Steve Howell a04a095738 Speed up alert word detection during message sends.
We no longer use all the alert words for all the users in the
entire realm when we look for alert words in a newly sent/edited
message.  Now we limit the search to only all the alert words
for all the users who will get UserMessage records.  This will
hopefully make a big difference for big realms where most messages
are only sent to a small subset of users.
2016-09-23 15:24:55 -07:00
Steve Howell 40b18094ec alert words: Refactor alert word detection.
The bugdown parser no longer has a concept of which users need which
alert words, since it can't really do anything actionable with that info
from a rendering standpoint.

Instead, our calling code passes in a set of search words to the parser.
The parser returns the list of words it finds in the message.
Then the model method builds up the list of user ids that should be
flagged as having alert words in the message.

This refactoring is a little more involved than I'd like, but there are
still some circular dependency issues with rendering code, so I need to
pass in the rather complicated realm_alert_words data structure all the way
from the action through the model to the renderer.

This change shouldn't change the overall behavior of the system, except
that it does remove some duplicate regex checks that were occurring when
multiple users may have had the same alert word.
2016-09-23 15:21:17 -07:00
Steve Howell cb0d75b23b rendering: Consolidate code to render new messages.
We now use render_incoming_message() to render all incoming
new messages (sends/edits), so that they will get the same treatment.

This change also establishes do_send_messages() as the code
path to get new messages rendered.  It removes some
logic from check_message() that only happened on certain code paths
for sending messages, and which would only detect failures by
expensively rendering messages, so it wasn't much of a guard.

This change also helps to phase out maybe_render_content(), which
deepens the call stack without providing much clarity to the reader,
since it's behavior is so variable.

Finally, this sets up to fix a flaw in the way we compute which
users have alert words in their messages (in a subsequent commit).
2016-09-23 15:21:17 -07:00
Tomasz Kolek 9197c82d8f Move WebhookTestCase class from test_hooks.py to test_helpers.py. 2016-09-23 10:02:16 -07:00
Steve Howell f0eaee68e4 bug: Fix traceback in get_missed_message_token_from_address().
If you supplied an unrecognizable address to our email system,
or you had EMAIL_GATEWAY_PATTERN configured wrong,
the get_missed_message_token_from_address() used to crash
hard and cryptically with a traceback saying that you can't
call startswith() on a None object.

Now we throw a ZulipEmailForwardError exception.  This will
still lead to a traceback, but it should be easier to diagnose
the problem.
2016-09-22 13:41:26 -07:00
Steve Howell dbbc64dbfe bug: Fix code that mis-identifies missed message formats.
In our email mirror, we have a special format for missed
message emails that uses a 32-bit randomly generated token
that we put into redis that is then prefixed with "mm" for
a total of 34 characters.

We had a bug where we would mis-classify emails like
mmcfoo@example.com as being these system-generated emails
that were part of the redis setup.

It's actually a little unclear how the bug in the library
function would have manifested from the user's point of view,
but it was definitely buggy code, and it's possibly related in
a subtle way to an error report we got from a customer where
only one of their users, who happened to have a name like
mmcfoo, was having problems with the mirror.
2016-09-22 13:41:26 -07:00
Steve Howell 0b7cac04d4 email mirror: Extract is_mm_32_format(). 2016-09-22 13:41:26 -07:00
Tim Abbott 68c51912c9 extract_recipients: Add a type: ignore for problematic arguments.
We'll want to redo this function to be less crazy, but this is a quick
fix to be able to upgrade mypy.
2016-09-19 21:48:12 -07:00
Tim Abbott e488d4d6e8 log_event: Fix unicode handling of log writing. 2016-09-19 21:45:52 -07:00
Steve Howell a036a72db6 bugdown: Raise BugdownRenderingException.
We now raise an exception in bugdown.do_convert() if rendering
fails, to avoid silent failures, and then calling code can convert
the exception to a JsonableError.
2016-09-16 10:01:55 -07:00
Steve Howell de25f07961 tests: Add BugdownErrorTests. 2016-09-16 10:01:55 -07:00
Steve Howell 152cebf33d bugdown: Extract log_bugdown_error().
(It makes it easier to mock out the logging.)
2016-09-16 10:01:54 -07:00
Steve Howell 8776cc0e35 Add create_streams_if_needed(). 2016-09-15 10:18:10 -07:00
Tomasz Kolek acbfe7e624 Add Librato integration.
Fixes: #68.
2016-09-13 21:55:25 -07:00
umkay 7eb6924841 Wrap dict view object in list for python3 compatibility.
Deleting items from a dictionary being iterated through directly
doesn't work properly in Python 3.
2016-09-13 21:21:45 -07:00
Steve Howell edf7ea7f51 tests: Add make_client() test helper. 2016-09-13 14:32:59 -07:00
Tim Abbott d964e5595a Finish annotating test_helpers.py. 2016-09-12 08:29:34 -07:00
Steve Howell cdee3dfb53 Annotate test_helpers.instrument_url(). 2016-09-12 08:21:46 -07:00
Steve Howell ffec98d85c Annotate zerver/lib/export.py.
This adds the remaining annotations to lib/export.py.
2016-09-12 08:21:46 -07:00
Steve Howell 62dd86bcce mypy: Set Path to str instead of text_type in export.py.
Using text_type for Path just breaks a lot of calls to
the core Python libraries that still want strings.
2016-09-12 08:21:46 -07:00
Steve Howell 30129f609b Annotate client_post() and client_get() in test_helpers. 2016-09-11 17:15:35 -07:00
Tomasz Kolek 8b9e9a386a Add mypy annotations to various integrations-related functions.
Added to:
zerver/lib/integrations.py
zerver/tests/test_integrations.py
zerver/tests/test_management_commands.py
zerver/views/integrations.py
zerver/views/webhooks/circleci.py
2016-09-11 17:11:25 -07:00
Tim Abbott 4634aa638e Annotate zerver/lib/tornado_ioloop_logging.py. 2016-09-10 12:10:41 -07:00
Tim Abbott 799c78fbcf Annotate zerver/lib/sqlalchemy_utils.py. 2016-09-10 12:08:37 -07:00
Tim Abbott 2e6aad669c cache: Add a basic annotation for cache_with_key. 2016-09-10 11:57:08 -07:00
Tim Abbott e48cdb8d7e Fix python 3 compatibility in logging code.
Detected by mypy.
2016-09-09 15:36:29 -07:00
Tim Abbott b41fc75767 notifications: Remove unused field from template context.
While one often might want to put the user's name in an email
template, `name` here was the user's full name, not their first name,
and thus reads as quite formal.
2016-09-08 17:03:00 -07:00
Tim Abbott 0d324d38b3 logging: Fix duplicate detection for email errors.
Our implementation of duplication detection in the Zulip email error
reporting system was buggy in two important ways:

* It did not look at the traceback, and thus considered all errors as
  the same.

* It reset the 10-minute duplicate timer every time an error happened,
  thus concealing situations where the same error was occuring more
  often than 1/10 minutes.
2016-09-08 16:55:36 -07:00
Tim Abbott 4423222d92 email_mirror: Add successful processing logging. 2016-09-08 16:54:10 -07:00
Tomasz Kolek 848635005f Add Sentry integration.
Fixes: #618.
2016-09-07 19:39:45 -07:00
Umair Khan 6a12ad7c33 Don't trust env while accessing Tornado.
This fixes a problem where the requests to Tornado would attempt to
use a configured outgoing HTTP proxy, when really we want to connect
directly to localhost.

Fixes: #468.
2016-09-06 14:17:50 -07:00
Umair Khan a4466c93a3 Use connection pooling for Django -> Tornado reqs.
Fixes: #391.
2016-09-06 14:15:35 -07:00
Tomasz Kolek 0e192d6aff Add colors to streams in development server fixtures.
Fixes: #1454.
2016-08-26 22:38:14 -07:00
Rishi Gupta 084bc63cad Change Day 2 email to be sent 24 hours after the user signs up.
Old behavior is to do something tricky that relies on the server being on
Pacific Time and the users being in the US. The goal is to have this message
appear during business hours, since click through rates are higher during
business hours. Our server is now on the East Coast though and our users are
in every timezone, so until we do something smarter this seems like a better
heuristic. We're also trying to cleanse our codebase of non-timezone-aware
datetime.datetime objects.
2016-08-26 15:59:30 -07:00
Tomasz Kolek 69a9fece70 Add GitLab integration.
Fixes: #33.
2016-08-25 13:49:51 -07:00
Tim Abbott 5b68e0defe settings: Fix buggy MANDRILL_API_KEY logic. 2016-08-25 09:37:33 -07:00
Tim Abbott 797a7ef97b notifications: Don't spoof emails as from users by default.
The previous default configuration resulted in delivery problems if
the Zulip server was authorized in the SPF records for the domains of
all users on the Zulip server.
2016-08-24 14:10:52 -07:00
Tim Abbott 55d7947b76 email_mirror: Use ERROR_BOT for error reporting.
This has the nice side effect of getting rid of the now-unnecessary
ADMIN_DOMAIN from this codepath -- we really just want whichever
realm ERROR_BOT is in.
2016-08-22 21:28:01 -07:00
Steve Howell cd2e36d66f export: Add lib.utils.query_chunker(). 2016-08-22 19:27:32 -07:00
Steve Howell fd6ee7117f tests: Renamed AuthedTestCase to ZulipTestCase. 2016-08-22 19:07:44 -07:00
Tim Abbott 96a5388fcb mandrill: Fix hardcoded check for settings.VOYAGER.
Since this delayed sending feature is the only thing
settings.MANDRILL_API_KEY is used for, it seems reasonable for that to
be the gate as to whether we actually use Mandrill.
2016-08-22 16:25:28 -07:00
Taranjeet Singh 5e5ad9de0c bugdown/__init__.py: Fix line with length greater than 120.
This updates file zerver/lib/bugdown/__init__.py.
2016-08-19 11:56:43 -07:00
Rishi Gupta 64179b2fd2 confirmation: Use realm host in activation URLs.
This is preparation for merging the subdomains feature.
2016-08-18 20:12:36 -07:00
Brock Whittaker 6936d49202 Live update new avatars across users.
This sends an event when a new avatar is uploaded that refreshes the
avatar for all browser clients without the need to reload the browser.

Fixes: #1359.
2016-08-18 15:32:29 -07:00
Tim Abbott e7aac717e8 Fix markdown ordering bugs with some custom syntax.
!avatar, !modal_link, !gravatar, etc. were incorrectly being processed
before the escape character for code blocks.

While we're at it, we add tests for these special syntaxes.
2016-08-18 14:47:37 -07:00
Steve Howell 125dcad379 tools: Report OverflowErrors in url coverage report. 2016-08-18 13:28:58 -07:00
Tim Abbott 685d63b6cc notifications: Update digest email link URLs to use realm.uri.
This is preparation for fully supporting each realm having its own
subdomain in Zulip.
2016-08-17 22:58:35 -07:00
Tim Abbott 91ad923d80 notifications: Use realm.uri in unsubscribe links. 2016-08-17 22:44:38 -07:00
Tim Abbott 48ed7e2e31 notifications: Use realm_uri for click here link. 2016-08-17 22:19:53 -07:00
Tim Abbott 5f12136e7c notifications: Fix broken one click unsubscribe link. 2016-08-17 22:19:53 -07:00
Tim Abbott 6ef79ac66c notifications: Remove support@zulip.com comment in docstring. 2016-08-17 22:19:53 -07:00
Tim Abbott f0a65fe52a export_messages_single_user: Fix missing order_by.
This fixes a nasty bug where exporting messages sent by a single user
might only contain some of the messages in the event that the
unspecified sort order by the database didn't happen to be sorted by
message ID.
2016-08-17 22:18:58 -07:00
Steve Howell c12bd853f7 export: Add basic export tests. (fixes #1584) 2016-08-16 13:38:37 -07:00
Steve Howell ab053845aa export: Fetch messages with two queries. 2016-08-16 13:38:37 -07:00
Steve Howell a5f651b81a export: Add some user_id filtering to do_export_realm().
This commit only addresses tables that currently derive from
user_profile_config in get_realm_config:

   zerver_userpresence
   zerver_useractivity
   zerver_useractivityinterval
   zerver_subscription
   zerver_recipient
   zerver_stream
   zerver_huddle

It also introduces an entry in realm.json for a virtual
table called "zerver_userprofile_mirrordummy" for dummy users,
which include prior dummy users and users excluded from the call
to do_export_realm().

Note that this feature is not yet exposed in the management command.
2016-08-16 13:38:37 -07:00
Tim Abbott 4fbb8c3eee templates: Add new context variables to all templates.
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm

This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
2016-08-13 16:27:35 -07:00
Tim Abbott 4a46b879ee import_uploads_s3: Fix setting of content-type. 2016-08-13 11:26:53 -07:00
Tim Abbott 8edc5110cd export: Set re_map_foreign_keys verbose default to False.
Otherwise, this is super spammy when doing a large import.
2016-08-13 11:26:16 -07:00
Tim Abbott 8b170665e4 export: Add assertion that ./manage.py exists in current directory.
Otherwise we'll fail starting the UserMessage export, later on.
2016-08-13 10:13:11 -07:00
Tim Abbott 856ab48ec6 export: Fix stream export sanity check. 2016-08-13 10:02:34 -07:00
Steve Howell e932b03999 export: Clean up path.join/makedirs for avatars/uploads. 2016-08-13 09:31:49 -07:00
Steve Howell 12c25e7d5c export: Filter attachments by message id. 2016-08-13 09:31:49 -07:00
Steve Howell 0f493d5000 export: Return msg ids from export_partial_message_files(). 2016-08-13 09:31:49 -07:00
Tim Abbott 924d0475c1 cache_helpers: Disable filling message cache.
The message cache filling script actually used both database and
memcached queries as part of filling the cache (all used to compute
the needed display_recipient values).  We would ideally fix this by
using bulk operations to fill the display_recipient cache, but until
we do so, this cache filler is counterproductive.

I believe this disabling fixes an issue where memcached would get
overloaded and stop handling requests during a server restart on busy
servers.
2016-08-13 09:27:56 -07:00
Rishi Gupta 708b416ca1 Terms of Service: Fix corner cases around new users being created.
This fixes a few bugs in 7910a6e134,
related to automatically created user accounts.
2016-08-13 00:19:54 -07:00
Steve Howell 0c3e98fa91 export: Introduce attachment.json file.
Now attachment data gets written to its own json file.  We are
splitting this out so that will be easier for us to cross-check
attachments against messages without holding up writing a lot
of the other realm data.  (message cross-checking is coming soon)
2016-08-12 18:59:14 -07:00
Steve Howell ea0a7d87c8 export: Refactor how we fetch attachment data.
This commit doesn't change any behavior; it just moves fetching
attachments out of the Config scheme and into its own method.
This prepares us to start writing attachment data to its own
file and cross-checking against message ids (coming soon).
2016-08-12 18:59:14 -07:00
Steve Howell fba7a9ca21 export: Unify top-down export configuration.
We now just have a single configuration get_realm_config() that
handles most of the top-down realm export tables.  (It basically
does everything not related to messages or uploads/avatars.)

Unifying the configs allows us to be more strict in our
configuration about checking for anomalies.  In the future
we may need to loosen up some of those restrictions again,
but for now we are picky and paranoid.
2016-08-12 15:27:23 -07:00
Steve Howell 5a5353b846 export: Fetch stream data only for stream recipients.
Fetch stream data only for stream recipients, instead of
getting streams via realm_id.

(This change is kind of moot for now, since our stream recipients
include all possible stream recipients in the realm, but this
sets us up for when we start restricting users that we export
within the realm.)
2016-08-12 15:27:22 -07:00
Steve Howell 7a429d1e30 export: Add sanity_check_stream_data(). 2016-08-12 15:27:22 -07:00
Steve Howell ec86e475b4 export: Add Config.post_process_data 2016-08-12 15:12:01 -07:00
Steve Howell 0c2c331905 export: Flip how we fetch stream subscription data.
We now get stream subscriptions BEFORE stream recipients.
2016-08-12 15:12:01 -07:00
Steve Howell 70a916aae3 export: Flip how we fetch user subscription data.
We now get user subscriptions BEFORE user recipients.
2016-08-12 15:12:01 -07:00
Steve Howell 2a2ce6ada1 export: Remove hard-to-maintain code comment.
Subsequent changes are gonna make the top-down/bottom-up comment
no longer valid.
2016-08-12 15:12:01 -07:00
Tim Abbott ed01842f95 notifications: Fix missed message emails with multiple senders.
Previously, missed message emails with multiple senders would
incorrectly have a "," outside the quoted sender name part of the from
address string, resulting in confusing email output.
2016-08-12 12:28:06 -07:00
Steve Howell 6fdd42c08b export: Create convenient soft links. 2016-08-12 10:48:33 -07:00
Steve Howell 70b68ddcc3 export: Use a config for export_single_user(). 2016-08-12 10:37:41 -07:00
Steve Howell c69a5bdec3 export: Handle more tables via export_from_config().
This commit introduces the ability to do custom fetches
and to essentially use temp tables for intermediate results.

(The temp table stuff deals with recipients/subscriptions
having three different flavors--user, stream, and huddle.)
2016-08-12 10:37:35 -07:00
Steve Howell f471a1779e export: Handle simple exports with export_from_config().
This handles the simple tables that don't need custom fetches.
2016-08-12 09:54:57 -07:00
Steve Howell 682155778d export: Add export_with_config().
Subsequent commits will start to use this.
2016-08-12 09:54:57 -07:00
Steve Howell b0e6d20321 export: Write stats.txt for `./manage.py export <realm>`. 2016-08-12 09:06:10 -07:00
Rishi Gupta 7910a6e134 Terms of Service: Add ability to update TOS and have users re-sign.
Most directly useful for the migration to zulipchat.com.

Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.

Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
2016-08-11 23:37:48 -07:00
Tim Abbott 11cac86a58 settings: Add support for specifying a remote redis password. 2016-08-11 16:54:22 -07:00
Steve Howell df3aa39be3 export: Extract write_data_to_file(). 2016-08-11 15:51:22 -07:00
Steve Howell f29b32bbb2 export: Clarify message exporting code.
The function to create the message partial files has been
renamed to export_partial_message_files().  It now gets its own
list of user profile ids and recipient ids from the response,
so that we can de-clutter do_export_realm().
2016-08-11 15:51:22 -07:00
Steve Howell 5cd915694a export: Extract launch_user_message_subprocesses().
This is the last in a series of commits that makes it
so that do_export_realm() mostly delegates work out
to other functions.
2016-08-11 15:21:30 -07:00
Steve Howell b383f5ca5d export: Extract fetch_user_profile_cross_realm(). 2016-08-11 15:21:30 -07:00
Steve Howell fee2106c6f export: Extract fetch_huddle_objects().
This also removes the dead codepath for include_private=False.
2016-08-11 15:21:30 -07:00
Steve Howell a6235f6a60 export: Add comments to export_single_user().
(This is a bit of a prefactoring to hopefully create a nice
diff in a subsequent commit.)
2016-08-11 15:21:30 -07:00
Steve Howell 6e7fe76cf4 export: s/avatar_bucket/processing_avatars
The name avatar_bucket was confusing for a boolean, and
in some places it was used for non-S3 paths.

I considered the more concise 'is_avatar', but that
was still confusing when you are processing multiple
files, because you think it's a calculated property
on one file instead of an overall codepath switch.

I also considered splitting up some functions, but
there is a lot of common logic between handling
file uploads and avatars that's not trivial to extract
into helpers, especially on the S3 side.
2016-08-11 15:21:30 -07:00
Steve Howell 3dab366733 export: Clean up names of upload/avatar export functions.
I did some minor moving around of code that made us have
one fewer function without any additional conditional
logic. The names are more explicit about saying
"from_local" and "from_s3".  Also, there is less clutter
now in do_export_realm(), which is evolving into more of
a dispatcher and less of a worker.
2016-08-11 15:21:30 -07:00
Steve Howell d62a351107 export: Add sanity_check_output(). 2016-08-11 15:21:30 -07:00
Umair Khan af9f308f0e Send double columned language list in page_params. 2016-08-10 16:19:28 -07:00
Umair Khan 73e5289736 Add default_language_name in page_params. 2016-08-10 16:19:28 -07:00