Commit Graph

3350 Commits

Author SHA1 Message Date
Rhea Parekh 220ad6a386 slack importer: Map standard reactions.
As mentioned in
https://get.slack.help/hc/en-us/articles/202931348-Use-emoji-and-emoticons,
slack supports the standard emoji codes
(https://www.webpagefx.com/tools/emoji-cheat-sheet/)
and majority of them are already supported in Zulip.
2018-04-01 23:24:35 -07:00
Rhea Parekh 6f867fee40 import script: Support import of reactions. 2018-04-01 23:24:33 -07:00
Rhea Parekh 8a028142d8 slack importer: Remove id allocation function and its implementation.
Remove allocation ID function from slack import script. All the IDs
count will start from 0. Hence the ID List returned
by the allocation function is of no use, and we remove its implementation.
(example: get_total_messages_and_attachments function is of no use anymore,
hence we remove it)
2018-04-01 23:10:55 -07:00
Rhea Parekh d147bd25d0 import script: Change file path of the upload in the import script.
In importing avatars, we use the implementation where the 'avatar_path'
is seperately calculated using realm and user ID and then the content
of the path provided in the avatar's 'records.json' are copied to this
'avatar_path'.

Similary, here for the uploads, 's3_file_name' is seperately calculated
using the realm ID and uploaded file name and then the content of the
path provided in upload's 'records.json' are copied to this 's3_file_name'.
2018-04-01 23:04:14 -07:00
Rhea Parekh ff34d07fa0 import script: Add function to update model ids after allocation.
Add function 'update_model_ids' to remove repetitive code.
2018-04-01 22:29:23 -07:00
Rhea Parekh a2ecdeb28d import script: re_map minor foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 078453554e import script: re_map Message foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 93aabcb81c import script: re_map Subscription foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 9ef7870c5a import script: re_map Recipient foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 4537223ba7 import script: re_map UserProfile foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 1314e7d247 import script: re_map Stream foreign keys.
'recipient_field' is added as a bool variable in the function
'update_id_map' to update the recipient foreign keys.

Recipient Foreign Key is equal to the UserProfile ID, if the
type is 1, and the same is equal to Stream ID, if the type is 2.
Hence a check is added in the 'update_id_map' field for this.
2018-04-01 22:29:23 -07:00
Rhea Parekh 8624ba4132 import script: re_map Realm foreign keys.
All the objects with realm ID as the foreign keys need to
be remapped with updated with the allocated ID.
Also the ID of the realm object itself is updated with the allocated
ID.
2018-04-01 22:29:23 -07:00
Rhea Parekh 2b0ee472af import script: Refactor re_map_foreign_keys.
The 'id_field' bool variable is added to the function just to check
if the field is the ID of that object, and not the foreign key relation.
For foreign key field names, a "_id" has to be added after the field name,
however we don't need that for the ID field of the object.
2018-04-01 22:29:23 -07:00
Rhea Parekh cd0871bae4 Import script: Add id allocation functions. 2018-04-01 22:29:23 -07:00
Shubham Padia 10115491a3 real-time sync: Server sends recipent data with delete_message event.
Fixes #8853.
In certain cases, the browser is not able to look up the message.
Include the recipient data for the message in the delete_message event,
so look up of those attributes by the browser isn't required.
2018-04-01 22:19:08 -07:00
Shubham Dhama b7aae89029 org settings: Add realm level default time format setting. 2018-03-31 16:23:26 -07:00
Puneeth Chaganti 4ce8f2aaa2 upload: Rename upload_message_image to upload_message_file.
Tweaked by tabbott to also fix a Slack import comment.
2018-03-30 13:38:31 -07:00
Rohitt Vashishtha a3ed83f4e2 markdown: Update bugdown emoticon translation logic to match frontend.
This PR solves some of the parity issues in the emoticon translation
logic. I was unable to find a way of matching only one of the
lookaround groups, so we still have some inconsistency (see
testcase). The approach of having another check while converting just
for this seemed like an inefficient way, so I've left that last change
as it is.
2018-03-29 15:56:24 -07:00
Umair Khan 7885dd4408 profile: Send field types to client side. 2018-03-29 13:09:12 -07:00
Umair Khan f6fb88549f profile: Send operation in the event.
This allows us to show a useful message in the handler when the
event is received.
2018-03-29 13:09:11 -07:00
Tim Abbott 656f882a44 bots: Eliminate NEW_USER_BOT.
This bot was basically a duplicate of NOTIFICATION_BOT for some
specific corner cases, and didn't add much value.  It's better to just
eliminate it, which also removes some ugly corner cases around what
happens if the user account doesn't exist.
2018-03-29 12:01:21 -07:00
Tim Abbott 2bc51931a8 test_fixtures: Add settings files to things that require reprovision.
Since the test database is in part controlled by the Zulip settings
files for testing, these settings files should be included in the list
of files that require populate_db to be rerun.

This issue was found due to changes to internal bots.
2018-03-29 11:53:12 -07:00
neiljp (Neil Pilgrim) 704c33331c mypy: Add explicit Optional for default=None parameters in export.py. 2018-03-28 12:31:51 -07:00
neiljp (Neil Pilgrim) 090b47ed19 mypy: Add explicit Optional for default=None parameters in various files. 2018-03-28 12:31:51 -07:00
Balaji2198 e1eabe286a integrations: Add Gocd webhook integration.
Fixes #38.
2018-03-28 11:08:27 -07:00
Tim Abbott c94deff920 mypy: Remove some now-unnecessary type: ignores. 2018-03-28 10:39:05 -07:00
Rohitt Vashishtha 3c96b04763 emoji: Add translate_emoticons flag in bugdown testcases.
Also switches the default behaviour of the code to not translate the
emoticons. Earlier, the code was testing-aware, and used to translate
when there was no user profile data available(assuming that as a testing
environment).
2018-03-27 17:16:55 -07:00
Eeshan Garg 538746fc65 webhooks: Stop raising an exception if stream does not exist.
webhook-errors.log file is cluttered with Stream.DoesNotExist
errors, which hides the errors that we actually need to see. So,
since check_message already sends the bot_owner a PM if the webhook
bot tries to send a message to a non-existent stream, we can ignore
such exceptions.
2018-03-24 13:50:17 -07:00
Ricky 4b8bd0bc3b webhooks: Add Dialogflow integration.
With minor fixes by eeshangarg!

Fixes #6990
2018-03-24 15:03:35 -02:30
Steve Howell 4f05cefb32 Remove obsolete code for stream subscribe buttons.
In 18e43895ff we replaced
stream subscribe buttons with stream links.  The new feature
has been well tested and well received for over a year now,
so it's safe to remove the older feature at this point.

Older sites will have super old messages that still have the
rendered markup; this commit does not attempt to address those
situations.  Most likely, clicking on an old button in the old
message will either do nothing or look like a message reply.
2018-03-24 12:31:36 -04:00
Aayush Agrawal 78a4924ecc markdown: Apply both bold and italic for words like ***text***.
Adds rule 'strong_em' to markdown parser to recognize ***{TEXT}***
Fixes #8155.
2018-03-23 15:01:24 -07:00
Tarun Kumar 86eaa553a6 user-groups: Add backend enforcing for new modification settings.
Add function in user-groups.py for getting member ids
for a group.
Update view to enforce checks for modifying user-groups.
Only admins and user group members can modify user-groups.
2018-03-23 14:44:47 -07:00
Greg Price b42a7b1701 digest: Add a server setting, and disable by default.
This feature isn't really ready yet -- the relevance isn't good, so
the emails aren't a great experience.  More work needed; pending that,
just don't send them.

There's already a per-realm setting, which doesn't have a control in
the org settings UI but does suppress it in the per-user settings UI.
Piggyback on that to suppress that UI control when the feature is
disabled at the server level too.

Also cut a comment that hasn't really made sense since the logic was
changed months ago -- the comment originally explained why we sent
digests on Tuesday, Wednesday, and Thursday, and doesn't correspond to
why we dialled back to weekly on Tuesdays.
2018-03-23 14:12:01 -07:00
neiljp (Neil Pilgrim) 97836c4bac mypy: Remove Optional & Mutable from do_schedule_message.
Removes various mypy errors.
2018-03-23 11:32:00 -07:00
neiljp (Neil Pilgrim) 2079c5e4d7 mypy: Mark Subscription as Optional in lib/streams.py:access_stream*. 2018-03-23 11:32:00 -07:00
neiljp (Neil Pilgrim) d741e0ea01 mypy: Introduce query variable in lib/users.py for clarity.
This also avoids mypy showsing an error.
2018-03-23 11:32:00 -07:00
Tim Abbott 62c0d27d1e push_notifications: Handle errors with token reuse.
If an Android token has been used to connect a given device with
multiple Zulip servers, and then is expired, we would 500 in trying to
remove the Zulip-side registration for it, because the code assumed
there was only one such registration.  If a token is no longer valid,
it's invalid for all servers, so the correct fix is to just remove them all.
2018-03-23 11:02:55 -07:00
Sarah f5c2fb8438 org settings: Create backend api for allow_community_topic_editing.
Adds the code for updating the allow_community_topic_editing
setting.
2018-03-22 16:02:27 -07:00
Jack Weatherilt 8535625341 parse_user_agent: Assert user agent is not None.
This commit asserts that parse_user_agent never returns None. The
RegEx will match any string, so that `match` is never None. This
brings test coverage of lib/user_agent.py to 100%. Changes were also
made in test/test_decorators.py and views/compatibility.py to reflect
that parse_user_agent cannot return None.

Improves: #7089.
Fixes: #8779.
2018-03-22 14:29:29 -07:00
Greg Price fe0f1edddb settings: Fix double negative in LOGGING_NOT_DISABLED.
Saying "not disabled" just makes it more work to read than it needs to
be -- instead say ENABLED.
2018-03-21 18:03:05 -07:00
Greg Price 73559e5320 queue: Suppress error mail from brief rabbitmq downtimes.
Details in comment.  Together with a few previous commits, this should
completely eliminate sending error mail to admins when the RabbitMQ
server is simply restarted and comes back up normally.
2018-03-21 18:03:05 -07:00
Jack Weatherilt d857f26cd3 refactoring: Remove unused assignment on ensure_stream.
There were two instances of `ensure_stream` being called and assigned to
a variable with the variable not being used elsewhere. pyflakes picked
up on this (where it didn't in the previous version likely due to tuple
unpacking), so the the variable assignment has been replaced with a call
to `ensure_stream`.
2018-03-21 16:47:52 -07:00
Jack Weatherilt 3396cfc2ef refactoring: Replaced occurences of create_stream_if_needed.
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.

The changes weren't significant enough to add any tests or do any
additional manual testing.

The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.

Fixes: #2088.
2018-03-21 16:47:36 -07:00
YJDave 11c995b70f custom profile data: Send event to active user on update.
On update of custom profile fields, send an event to all
active users of realm.
2018-03-21 16:08:12 -07:00
Harshit Bansal 64372690f9 emoji: Fix the filtering condition in `check_emoji_admin()`.
This commit fixes an unreported bug which if hadn't been fixed would
have caused errors while deactivating realm emojis in some corner
cases.
2018-03-20 22:24:44 -07:00
Harshit Bansal a49655e0d4 emoji: Migrate realm emoji to be addressed by `id` rather than `name`.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.

Fixes: #6977.
2018-03-20 22:24:44 -07:00
Greg Price 3b3154527f queue: Don't blow up when a connection closes quickly. 2018-03-20 16:49:05 -07:00
Greg Price 9dcc436766 queue: Fix __init__ logic so heartbeat choice works fully.
Because the base class's __init__ calls `_connect`, when we set the
value after that call has already returned, our new value only takes
effect if the first connection fails and we have to reconnect.
Make it take effect from the beginning.
2018-03-20 16:49:05 -07:00
Greg Price 5edc26a0df queue: Cut disused, broken parameter to `_connect`.
This parameter isn't used anywhere.  A good thing, because if it were,
the code would immediately raise an exception -- `self._on_open_cbs`
hasn't been initialized yet when we first call `_connect`, from the
base class's `__init__`.

So, just cut it.  If we later need something like this, it's easy to
add a working version then.
2018-03-20 16:49:05 -07:00
Tim Abbott 51018e12f5 message_edit: Remove obsolete block for diff highlighting.
We disabled the original "colorized HTML edit-history" feature way
back in 2013 in c51056ff8e.

That original feature involved showing what had been edited inline in
message bodies, so one could easily see what had been changed.

That old feature has since been replaced with the "view edit history"
menu option, and we're unlikely to ever want the old feature back.
So, we can just remove its code.  There's a few supporting variables
that were created to help implement this; we can clean those up and
simplify the `update_message` code now that this feature is fully
removed.
2018-03-20 15:22:53 -07:00
Rhea Parekh d0355f52cb slack importer: Show when files are being downloaded. 2018-03-20 14:42:26 -07:00
Rhea Parekh e7291148e8 slack importer: Add '/me' in content for specific subtypes. 2018-03-20 14:42:26 -07:00
Nikhil Kumar Mishra f29a1918f3 hotspots: Add ALWAYS_SEND_ALL_HOTSPOTS in dev_settings.
Replace the local variable SEND_ALL from get_next_hotspots.
Add unit test for the same.
2018-03-19 10:39:43 -07:00
Nikhil Kumar Mishra 3b4ff4f75c markdown: Clean render_markdown.
Remove Message is None and not None case as the
render_message_backend passes a fake Message object.
2018-03-19 10:36:26 -07:00
Harshit Bansal 646700b144 emoji: Start using `get_active_emoji()` in `emoji_name_to_emoji_code()`. 2018-03-18 19:34:02 -07:00
Greg Price 22071a44a7 antispam: Add a sitewide ratelimit on invites by new realms.
This applies only on a server open for anyone to create a realm.
Moreover, if the server admins have granted any given realm a
max_invites greater than the default, that realm is exempt too.
2018-03-16 18:00:11 -07:00
Tim Abbott c147d2e140 notifications: Fix missed-message emails for presence-idle users.
This fixes an unpleasant regression in
f5edeb01ae, where we stopped correctly
filtering users who have an open browser session that's idle.  These
users are tagged as "UserPresence.IDLE" with an current timestamp in
the database, and should be treated as idle for presence purposes.

As a result, if you had an open Zulip browser session, you incorrectly
wouldn't get missed-message emails for PMs and mentions before this fix.
2018-03-16 16:30:23 -07:00
Tim Abbott 34e165c100 webhooks: Fix passing client string to authenticated webhook API views.
This fixes a regression in 93678e89cd
and a4979410f9, where the webhooks using
authenticated_rest_api_view were migrated to a new model that didn't
include setting a custom Client string for the webhook.

When restoring these webhooks' client strings, we also fix places
where the client string was not capitalized the same was as the
product's name.
2018-03-16 15:43:19 -07:00
Shubham Dhama 610f2cbacf notification email: Send followup_day2 email two days later.
This changes the followup_day2 emails delay from one day later to two days
later if it is getting delivered on any working days(i.e. Mon - Fri).
For Thursday it is compromised to next day as it would be too late to
postponed to Monday and for Friday it should be Monday.
At last actually, emails should send one hour before the above calculated so
that user can catch them when they are dealing with these kinds of stuff.
Fixes: #7078.
2018-03-16 13:35:57 -07:00
neiljp (Neil Pilgrim) 5726d26d50 mypy: Use centralized Validator in request.pyi & validator.py.
These changes are in one commit, since the previous typing of check_url
does not match the centralized strict definition (object/Any vs Text),
actually already used elsewhere in validator.py, and also had a different
API.

check_url is updated here to match the API of the other check_* functions,
ie. val is an object (not Text) & returns Optional[str]. It also now checks
the value is text explicitly at run-time, which was only type-checked
previously. Tests are updated accordingly.
2018-03-16 13:30:32 -07:00
YJDave 0281079a39 stream settings: Send all private streams subs to realm admins on load.
Tweaked by tabbott to simplify the conditionals in actions.py.

Fixes #8695.
2018-03-16 12:26:56 -07:00
YJDave 72a440a86d stream settings: Fix error in real time sync in subs add/remove event.
Currently, when other private stream subscriber add realm admin to
stream, new copy private stream is created in realm admin's streams.
Which resulted in error, cause there are two similar stream element
in stream settings.

If new subscriber is added to private stream, we first send them
stream `create` event, cause private stream are not visible until
user don't get subscribed at least once. But realm admins can now
always access private stream, so when realm admin is subscribed to
stream, realm admin get stream `create` event even if stream already
exist in on realm admin client side.

Fix this by extracting realm admins from stream `create` event on
`add` subscription operation and sending private stream `create`
event to all realm admins on stream creation operation.

Fixes #8695
2018-03-16 12:22:06 -07:00
Eeshan Garg 93678e89cd webhooks: Migrate 14 webhooks to use check_send_webhook_message.
These are the straightforward ones.

Note that there is a line in zerver.lib.test_classes.build_webhook_url
that lost test coverage. That's because most of our tests test using
stream messages so the webhook URLs being tested always have a query
parameter. So the line that accounts for there being no query
parameters never gets called, which is fine, but we should still
keep it.
2018-03-16 11:34:20 -07:00
Eeshan Garg af56df7723 webhooks: Enable custom topics and default PM notifications.
This commit adds a generic function called check_send_webhook_message
that does the following:
* If a stream is specified in the webhook URL, it sends a stream
  message, otherwise sends a PM to the owner of the bot.
* In the case of a stream message, if a custom topic is specified
  in the webhook URL, it uses that topic as the subject of the
  stream message.

Also, note that we need not test this anywhere except for the
helloworld webhook. Since helloworld is our default example for
webhooks, it is here to stay and it made sense that tests for a
generic function such as check_send_webhook_message be tested
with an actual generic webhook!

Fixes #8607.
2018-03-16 11:34:20 -07:00
Tim Abbott 707af5ab56 cache: Remove a now-unnecessary TODO.
We solved the problem the TODO raised by using a different type
annotation syntax, and I'm not sure whether that refactor would
actually improve the code.
2018-03-16 11:32:14 -07:00
neiljp (Neil Pilgrim) 966ca7015f mypy: Finalize migration of cache.py to python3 function annotation.
- Use forward declarations of some types from models.py to avoid cycles.
- Remove cache.py from linter rule exclude list to ensure it stays that way.
2018-03-16 11:29:12 -07:00
neiljp (Neil Pilgrim) 005cb6bd03 mypy: Improve [get_]cache_with_key typing & use py3 annotation. 2018-03-16 11:29:12 -07:00
neiljp (Neil Pilgrim) b79c3840da mypy: Introduce strict definition of Validator into types.py. 2018-03-16 11:16:38 -07:00
Tim Abbott 99bae7fa5d slack import: Improve algorithm and formatting for attachments.
The previous system would crash with some files (because for some
reason the comment count was 1 but there was no "initial comment") and
also the file comment and file name were sorta redundant.
2018-03-16 11:12:58 -07:00
Rhea Parekh 06071166c7 slack importer: Handle case where attachment download link is not from slack. 2018-03-16 11:12:58 -07:00
Rhea Parekh b7d6608ba6 slack importer: Clean 'create_converted_data_files' function.
The 'make_new_dir' bool value was used to create a new directory
every time True is passed. Now that avatars and uploads directory
are being created seperately, we don't need this anymore.
2018-03-16 11:12:58 -07:00
Rhea Parekh 4b66a2d0dc slack importer: Add function to fetch and save uploads. 2018-03-16 11:12:58 -07:00
Rhea Parekh e62945eb86 slack importer: Implement changes in script due to zerver_attachment. 2018-03-16 11:12:58 -07:00
Rhea Parekh 8e2d930644 slack importer: Implement changes in script due to user upload object. 2018-03-16 11:12:58 -07:00
Rhea Parekh b0851eb20b slack importer: Add helper functions to build attachment object. 2018-03-16 11:12:58 -07:00
Rhea Parekh 68af6e4b7a slack importer: Add helper functions to build user uploads object. 2018-03-16 11:12:58 -07:00
Rhea Parekh 6d85f8c9ef slack importer: Randomize stream colors. 2018-03-15 23:50:32 -07:00
Rhea Parekh 90a3ffc5c0 slack importer: Include only slack's purpose field in description. 2018-03-15 23:50:32 -07:00
Rhea Parekh 8a4f307c43 slack importer: Change topic for imported content. 2018-03-15 23:50:32 -07:00
Tim Abbott ec21997bfe slack import: Fix importing of bot users from Slack. 2018-03-15 18:35:40 -07:00
Rhea Parekh a5b0957e5d slack importer: Set domain name in 'do_convert_data'.
The domain name is being set in the helper function
'slack_workspace_to_realm', but it should be set in the main function
'do_convert_data', as we need it in other child functions of
'do_convert_data'.
2018-03-15 18:34:51 -07:00
Tim Abbott 14e8ac5675 migrate: Remove dead migration code.
This code was originally written when we were using the old South
system, and hasn't been used in a few years.  It probably doesn't
work, and thus only serves to clutter the codebase.
2018-03-15 17:56:32 -07:00
Vishnu Ks 951b88dd30 models: Make email_allowed_for_realm raise exception. 2018-03-15 14:35:24 -07:00
neiljp (Neil Pilgrim) 17937175ac mypy: Centralize ViewFuncT definition into new file zerver/lib/types.py.
Originally was going to centralize this in zerver/lib/request.pyi, but this
file is not visible at run-time, being only a stub. The matching request.py
file seemed inappropriate, as it doesn't actually use ViewFuncT.
2018-03-15 14:16:40 -07:00
neiljp (Neil Pilgrim) a7bfb09067 Mypy: Use models.py QuerySet annotation approach in stream_subscription.py.
Namely, annotate as best as possible, and add notes to indicate preference,
if QuerySet develops generic typing.

Note that the return values of functions with annotations changed in this
commit are used elsewhere as QuerySets, so the Sequence[T] approach used
for some functions in models.py is not applicable.
2018-03-15 12:54:43 -07:00
Tim Abbott 1a3ae8a376 stream settings: Fix inadvertently merged broken whitespace.
I'm not sure why the pre-commit linter didn't flag this for me; maybe
it did and I just missed it.
2018-03-13 15:12:45 -07:00
YJDave 2031118545 stream settings: Allow realm admins to access all private stream subs.
This will allow realm admins to access subscribers of unsubscribed
private stream.  This is a preparatory commit for letting realm admins
remove those users.
2018-03-13 14:59:09 -07:00
YJDave 37f9d5c193 stream settings: Allow realm admins to update any stream name & description.
This will allow realm admins to update the names and descriptions of
private streams even if they are not subscribed, which fixes the buggy
behavior that previously nobody could(!).
2018-03-13 14:59:09 -07:00
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
neiljp (Neil Pilgrim) 21033669a8 mypy: Migrate lib/bugdown/__init__.py to python3.5 annotations. 2018-03-12 11:23:30 -07:00
neiljp (Neil Pilgrim) bb1585b2c2 mypy: Migrate lib/debug.py to python3.5 annotations. 2018-03-12 11:23:30 -07:00
Vishnu Ks 41f8618c04 email: Use PyPi module for disposable email providers list. 2018-03-11 21:48:56 -07:00
Tim Abbott 25d9731a3c custom profiles: Fix totally broken events.py logic.
Apparently, my manual testing here was in error; the new version
actaully works for delivering custom profile data to the frontend.
2018-03-11 21:25:23 -07:00
Tim Abbott ef92fcbe2b topic history: Fix fetching topic history of public streams.
Apparently, we did essentially all the work to support showing full
topic history to newly subscribed users from a data flow perspective,
but didn't actually enable this feature by having the topic history
endpoint grant access to historical topics.  This fixes that gap.

I'm not altogether happy with how the code and tests read for this
feature; the code itself has more duplication than I'd like, and the
tests do too, but it works.
2018-03-11 20:59:20 -07:00
Ricky 9e52a9f879 webhooks/flock: Add flock integration. 2018-03-11 20:31:13 -07:00
Tim Abbott 02b8453367 custom profiles: Send custom profile data to frontend.
This will fetch the data of custom fields for all users.
2018-03-11 18:08:17 -07:00
Harshit Bansal e41067a833 emoji: Refactor bugdown to use only active realm emojis.
This commit refactors the bugdown to perform a lookup only on active
realm emojis. This was needed because once we migrate realm emojis
to be addressed by `id` rather than name, it will be costly to
perform a lookup on all the realm emojis.
2018-03-11 16:24:44 -07:00
neiljp (Neil Pilgrim) 8d188f50eb bugdown: Clarify walk_tree_with_family via ElementPair NamedTuple. 2018-03-11 08:30:56 +00:00
neiljp (Neil Pilgrim) d4cfab4823 mypy: Use cast in walk_tree_with_family to access inside opaque Element. 2018-03-10 10:04:14 -08:00
YJDave ee87c146ff settings: If message content disabled, flush all missed emails informations.
If user has disabled message content in missed email notifications,
we shouldn't send any informations about missed messages i.e. sender,
stream, message text, etc. to email servers.
We are already hiding this informations in email templates, but we
shoudln't expose any information about message content if user
has disabled this setting.
2018-03-09 21:16:02 -08:00
YJDave c94b21e9ac settings: Add setting to disable message content in missed message emails.
Fixes #6938.
2018-03-09 21:16:02 -08:00
YJDave 382ac24151 create stream: Fix real time sync bug in private stream creation.
If new private stream is created by realm admin without realm admin
subscribed to it, then it doesn't automatically add created stream to
realm admin's stream list. We have to reload the browser to get newly
created stream in stream list. Cause private stream creation event is
only sent to the subscribed users to private stream, so even if realm
admin is acting user, they don't get creation event.

We should send private stream creation event to realm admin users along
with subscribed user to stream, as realm admins can access unsubscribed
private streams.

Tweaked by tabbott to fix various typos and clean up the code.
2018-03-09 18:35:12 -08:00
Harshit Bansal 5aa8629a10 reactions: Migrate `emoji_code` to store `id` for realm emoji.
Till now, we had been storing realm emoji's name in emoji code field
in reactions' model. This commit migrates it to store realm emoji's id.
It is a part of effort to migrate realm emojis to be referenced by their
id and not by name.
2018-03-09 13:46:27 -08:00
Shubham Dhama 777b6de689 org settings: Add setting to prevent users from adding bots.
Fixes: #7908.
2018-03-09 13:21:55 -08:00
Robert Hönig 649e76e932 Display error when creating embedded bot with incorrect config data.
"incorrect" here means rejected by a bot's validate_config() method.
A common scenario for this is validating API keys before the bot is
created. If validate_config() fails, the bot will not be created.
2018-03-08 15:05:42 -08:00
Eeshan Garg 12396dd39b webhooks/gogs: Update text to conform to doc style guide. 2018-03-08 08:19:32 -08:00
Shubham Padia 32e38d36aa bots: Send add/delete event on bot ownership change.
Adds realm_bot delete event. On bot ownership change, add event is
sent to the bot_owner(if not admin) and delete event to the
previous bot owner(if not admin). For admin, update event is sent.
2018-03-08 07:54:19 -08:00
Tim Abbott c3964dff6e i18n: Fix a last few strings mentioning realms. 2018-03-07 17:15:29 -08:00
Tim Abbott e2a6541133 i18n: Fix use of 'realm administrator' in translated strings.
These are user-facing and thus should refer to being an "organization
administrator".
2018-03-07 17:15:29 -08:00
Tim Abbott 8fba40fdd5 emoji: Clean up strings for emoji errors.
The main goal here is to remove the use of the term "realm", but we
also make these strings more consistent and using better English.
2018-03-07 17:15:29 -08:00
Rhea Parekh 0d788823c2 slack importer: realm_id cleanup. 2018-03-07 14:07:24 -08:00
Rhea Parekh 7878cc53a3 slack importer: Cleanup build_subscription. 2018-03-07 14:07:24 -08:00
Rhea Parekh f947194e4c slack importer: Cleanup build_avatar. 2018-03-07 14:07:24 -08:00
Rhea Parekh 5efe05d5b6 slack importer: Cleanup build_zerver_usermessage. 2018-03-07 14:07:24 -08:00
Rhea Parekh 6c4eef16e9 slack importer: Sort messages in the order of their dates.
Sort the list of all messages in the order of their timestamp.
2018-03-06 14:07:09 -08:00
Rhea Parekh c2d4b49bf3 slack importer: Use precomputed value in 'channel_message_to_zerver_message'.
Use the List of all messages in this helper function
instead of using the messages channel-wise.
2018-03-06 14:07:09 -08:00
Rhea Parekh 3e4086d1b8 slack importer: Use precomputed value in 'get_total_messages_and_usermessages'.
Use the List of all messages in this helper function
instead of using the messages channel-wise.
2018-03-06 14:07:09 -08:00
Rhea Parekh 10c73ae577 slack importer: Add function to precompute all the messages.
The messages were first being read and passed to the helper
functions channel wise.
This function makes a list of all the messages in the all the channels
beforehand which would be used to pass in the helper functions.
2018-03-06 14:07:09 -08:00
Vishnu Ks 59b8f85c63 bugdown: Do only image preview if relative URL. 2018-03-06 13:50:02 -08:00
Tim Abbott 3d8e45f1cb cache: Avoid caching /help/ documentation page content.
This should make it a lot less annoying to edit these pages locally,
without regressing the test performance which motivated the cache.
2018-03-05 09:26:58 -08:00
Aditya Bansal 9eeb1c59f6 bugdown: Remove email from rendered content of messages with mentions.
This field has been unused by clients for some time, and isn't great
for our public archive feature plans (where we'll not want to be
including email addresses in messages).
2018-03-04 20:04:27 -08:00
Tim Abbott 9acf30b3d3 display_settings: Fix real-time sync of default language.
We apparently weren't sending to the frontend the data needed to
update the display of the current language.
2018-03-04 19:23:58 -08:00
Yago González b475e74c47 i18n: Fix string interpolation. 2018-03-04 15:44:12 -08:00
Marco Burstein bdb86f1b5e emoji: Add support for translating emoticons.
Add `translate_emoticons` to `prop_types` and `expected_keys`.
Furthermore, create a emoji-translating Markdown inline pattern.

Also use a JavaScript version of `translate_emoticons` and then use
this function during Markdown previews and as a preprocessor. This
is only needed for previews, because usually emoticon translation
happens on the backend after sending.

Add tests for emoticon translation, a settings UI, and a /help/ page
as well.

Tweaked by tabbott to fix various test failurse as well as how this
handles whitespace, requiring emoticons to not have adjacent
characters.

Fixes #1768.
2018-03-04 15:37:24 -08:00
Tim Abbott 038579b840 streams: Fix counts using only one data point.
We had the aggregation query for StreamCount done slightly wrong.
2018-03-04 14:06:29 -08:00
Robert Hönig e69af3011e Add basic test for FeedbackBot.
This sets up a new test class with a simple
test, mostly for increasing coverage. The class
should in the future be extended to properly
verify the handle_feedback() logic.
2018-03-04 13:31:33 -08:00
Archana BS c7650c0d9d streams: send data for recent streams traffic to frontend.
Significantly edited by Tim Abbott and Vishnu Ks.
2018-03-04 13:24:53 -08:00
Robert Hönig 4c7dd971d0 embedded bots: Dispatch all config data on update event.
Previously, when a user updated the config data of an
embedded bot, only the updated fields were dispatched
back to the client. Dispatching all config data fields
makes the client updating code less brittle.
2018-03-04 13:11:23 -08:00
Shubham Padia 13664f1289 uploads: Convert CMYK to RGB when saving avatar/realm icon as png.
Fixes #8546.
PNG does not support CMYK mode. CMYK file is converted to RGB and
then saved as PNG.
2018-03-02 12:57:22 -08:00
Eeshan Garg ce56ee80e7 api_docs: Document the `POST /api/v1/user_uploads` endpoint.
Fixes #1353.
2018-03-02 09:44:17 -08:00
Umair Khan 5086ba8568 push_notification: Update comment for unregister. 2018-03-01 22:14:32 -08:00
Umair Khan 8a8b0dd87d code-search: Add comments to help grepping. 2018-03-01 22:14:32 -08:00
Rhea Parekh 6a07897d3a slack importer: Fetch and save the avatars.
This gets the avatar of size 512 px and saves it in the
user's avatar directory with both the extensions
'.png' and '.original'.
2018-03-01 16:49:37 -08:00
Rhea Parekh 0f183981e6 Import script: Make sure medium avatars exist during import.
During a slack import, we don't have medium-size avatars already
available in the export data set (and possibly also with a normal
import/export?).  The medium size avatar can be created by the
'ensure_medium_avatar_image' function, which checks if the medium
image exists, and if it doesn't, it creates the image.

This commit was substantially edited by tabbott to get rid of an
undefined variable bug, avoid initializing the upload backend classes
in a loop, and add some TODO notes on things that could be improved
later.
2018-03-01 16:48:06 -08:00
Rhea Parekh 7076a49e04 slack importer: Pass avatar data through the import script. 2018-03-01 16:41:49 -08:00
Rhea Parekh 30b9d35d5e slack importer: Add helper functions to get user avatars.
Here, we create the slack avatar url using the user data and
build the avatar object. Added Tests for the same.
2018-03-01 16:38:55 -08:00
Rhea Parekh 95df8452be slack importer: Remove function 'get_user_avatar_source'.
slack avatar urls have the format:
'https://ca.slack-edge.com/<team_id>-<user_id>-<avatar_hash>-<size>'
For any url of this form, if the user hasn't uploaded an image,
Slack uses default gravatar, but we don't have a way of knowing if Slack
has used the uploaded image or the custom gravatar
eg: https://ca.slack-edge.com/T5YFFM2QY-U6006P1CN-gd41c3c33cbe-512.
Hence, avatar_source should be mapped to 'U'.
2018-03-01 16:38:55 -08:00
Greg Price 6d35a697af antispam: Don't let new accounts in open realms immediately send invites.
We haven't had a problem with this yet, but this should help prevent it.
2018-03-01 14:56:19 -08:00
rht 71ff8c370e django-2.0: Don't assign directly to Many-to-Many field.
The old pattern of setting the value and then using .save() here has
been deprecated.  set() also saves the record.
2018-03-01 08:49:35 -08:00
Robert Hönig 48b2e4eb66 backend: Add bot config data patching. 2018-03-01 08:25:43 -08:00
Robert Hönig 312cabb9a6 models.py: Move two long functions with circular imports to actions.py.
models.py should only contain thin wrapper functions. Furthermore,
this move allows us to remove the circular imports. The two moved
functions are interdependent and are thus moved in one commit.
2018-03-01 08:25:43 -08:00
Alena Volkova 822dfc6a34 integrations: Add webhook code, API endpoint, and tests for Front. 2018-02-28 16:18:19 -05:00
Vishnu Ks 120ac21515 bugdown: Create a constant for inline preview limit. 2018-02-28 12:53:36 -08:00
Aastha Gupta d124597f4b org-settings: Add setting to turn off welcome emails.
This adds an organization-level setting to provide an option to turn
off the welcome emails.

Fixes #8000.
2018-02-28 12:39:01 -08:00
Robert Hönig 19bbc0e6f8 backend tests: Use correct bot owner in create_test_bot().
The correct owner is the one specified in the user_profile
parameter, which is different from self.user_profile.
2018-02-27 12:20:08 -05:00
Rhea Parekh 3bb14a867b slack importer: Change 'invite_only' mapping in streams.
'invite_only' should always be true for the slack's
standard export plan as the private channels are not
supported in it.
2018-02-25 09:22:01 -08:00
Rhea Parekh a2f6f4ba1c slack importer: Handle case where messages have no users. 2018-02-25 09:20:55 -08:00
Rhea Parekh 15a6f62fe7 slack importer: Refactor defaultstream handling.
The check for the channel ('general' and 'random') must be added before
'build_defaultstream' function is called and then the id is incremented.
Otherwise, the id appended at the end of second defaultstream object, which would be
greater than the total number of defaultstream objects would crash at
'defaultstream_id_list[defaultstream_id]' which is a paramater of 'build_defaultstream'.
Added tests to prevent the same.
2018-02-25 09:20:55 -08:00
Eeshan Garg 0d39b05cf6 webhooks/beanstalk: Update text and remove screenshots.
This commit:

* Removes the unnecessary screenshot.
* Reorders the instructions and combines them in to 4 steps.
* Improves the contents of the webhook-url-with-bot-email-indented.md
  macro and makes it more consistent with create-bot-construct-url.md.
* Sets the recommended stream name to "commits", since that's what
  the webhook function for Beanstalk expects in
  zerver/webhooks/beanstalk/view.py. This allows us to use the
  create-stream.md macro.
2018-02-24 06:21:02 -05:00
Aditya Bansal f50e325075 RealmAuditLog: Fill subscription events with event_last_message_id=None.
Creating the very first organization administrator user and
subscribing them to streams before any messages were sent resulted in
RealmAuditLog entries being created with a `event_last_message_id` of
None, because that's the maximum ID in the empty set.

We correct this by fixing the incorrectly created RealmAuditLog
entries, both for new servers and also fixing old broken entries on
existing servers.

This fixes an issue where if a user setup a Zulip server with just the
organization administrator, and then forgot about it (so that the
initial user became soft-deactivated), trying to sign in 3 weeks later
would throw an exception.

This fixes the issue reported here:

https://chat.zulip.org/#narrow/stream/9-issues/subject/500.20error.20on.20login/near/511981
2018-02-22 09:45:22 -08:00
Rhea Parekh aff9099c3b slack importer: Get domain name from settings.EXTERNAL_HOST. 2018-02-21 08:58:27 -08:00
Rhea Parekh f6d267e773 slack importer: Change 'total_users' to 'total_channels'.
The total number of stream objects are allocated to
total_users. They should be allocated to the total_channels.
This passed the tests as the total number of users in the test
where greater than the total number of channels.
2018-02-21 08:06:10 -08:00
Eeshan Garg 96036f07a3 api_code_example: Use json instead of ujson; specify separators.
The Markdown extension that lives inside
zerver/lib/bugdown/api_code_example.py previously used ujson.
ujson's `dumps` function doesn't accept a `separators` argument,
which means we have no control over how the JSON is pretty-printed.
This resulted in JSON fixtures with no spaces after the colon, which
looks unnecessarily convoluted.

So now, we use the built-in `json` module to get around this.

For further reading, this issue
<https://github.com/esnme/ultrajson/issues/82> opened on ujson's
repo explains why they are reluctant to support such formatting
due to performance considerations.
2018-02-20 20:10:38 -03:30
Eeshan Garg e80609a3e9 api docs: Test sample fixture for invalid stream name error.
This commit adds a test for the sample fixture for when an invalid
stream name is passed to a query that expects a valid stream name
as an argument. This is the case with almost all of our queries
documented under the sidebar heading "Streams".

EDIT: Actually, I was wrong. This payload is highly specific to
get-stream-id, so it shouldn't be a part of common-error-payloads
at all.
2018-02-20 20:03:55 -03:30
Eeshan Garg 9add5bf6a7 api/delete-queue: Test fixture for BAD_EVENT_QUEUE_ID error.
In templates/zerver/api/delete-queue.md, we have a sample fixture
for when the queue_id passed to client.deregister_queue is not
valid or the event queue in question has already been deleted.
This commit tests that fixture.

Note that this error payload is specific to client.deregister_queue.
2018-02-20 14:54:12 -08:00
Eeshan Garg a0818975fb api/create-user: Test fixture for "email already in use" error.
In templates/zerver/api/create-user.md, we have a sample fixture
for when a client attempts to create a user with the same email
as an existing user. This commit adds a test for that fixture.

Note that this error payload is specific to client.create_user
and this error payload isn't generated anywhere else.
2018-02-20 14:54:12 -08:00
Eeshan Garg be0a04f33e api docs: Test sample fixtures for unauthorized_errors_fatal.
This commit adds tests for the sample fixtures for when
unauthorized_errors_fatal is passed to client.add_subscriptions.
2018-02-20 14:54:12 -08:00
Eeshan Garg 806476173f api/add-subscriptions: Test fixture for "already_subscribed".
In templates/zerver/api/add-subscriptions, we have a sample fixture
for when the user being subscribed is already subscribed to a
stream. This commit tests that fixture against a running server.
2018-02-20 14:54:12 -08:00
Eeshan Garg 06bf47d2f5 api docs: Test sample fixture for user not authorized error.
This commit adds tests for the fixture for when a user is not
authorized (perhaps because the query requires the use of admin
privileges) for a particular query.
2018-02-20 14:52:09 -08:00
Eeshan Garg 124c672a7c api docs: Test sample fixture for message edit permission error.
In templates/zerver/api/update-message.md, we have a sample fixture
for when a zulip.Client does not have the permission to update/edit
a particular message. This commit adds a test for that fixture.
Also, tools/test-api now also uses a non-admin client for this test,
which might come in handy in the future.
2018-02-20 14:52:09 -08:00
Eeshan Garg 1c32163bb9 api docs: Test sample fixture for missing request args error.
This commit adds tests for the sample fixture for when a required
request argument is missing. Also, it moves the sample fixture
to common-error-payloads.md, since this is an error payload that
is common to most requests (except the ones that don't take any
arguments).
2018-02-20 14:52:09 -08:00
Eeshan Garg d1a5bbc2a5 api docs: Test sample fixture for invalid PM recipient error.
In templates/zerver/api/private-message.md, we have a sample fixture
for when the email address of the PM's recipient is invalid. This
commit makes sure that fixture is tested against a running server.
2018-02-20 14:52:09 -08:00
Eeshan Garg 93b44a41c5 api docs: Test sample fixture for nonexistent stream error.
In templates/zerver/api/stream-message.md, we have a sample fixture
for when the target stream does not exist. This commit adds a test
for that sample fixture.
2018-02-20 14:52:09 -08:00
nyan-salmon 03624b78c2 webhooks: Add Facebook integration. 2018-02-19 09:41:49 -08:00
neiljp (Neil Pilgrim) 4efb83ee29 mypy: Amend logic in check_upload_within_quota to avoid None. 2018-02-19 09:20:30 -08:00
Steve Howell a474af8d58 Use newer stream urls in missed message emails.
This changes the missed-message logic to use the new encoding
scheme for stream url fragments that prefixes them with
"{stream_id}-".

For simplicity sake, we just get a Stream object to pass to
the helper functions, and we only call get_display_recipient()
now for the HUDDLE case.  (We were calling it wastefully before
for the PM case.)
2018-02-19 09:03:11 -08:00
Steve Howell 46a49777c4 Add stream ids to urls for stream-related narrows.
This commit prefixes stream names in urls with stream ids,
so that the urls don't break when we rename streams.

strean name: foo bar.com%
before: #narrow/stream/foo.20bar.2Ecom.25
after: #narrow/stream/20-foo-bar.2Ecom.25

For new realms, everything is simple under the new scheme, since
we just parse out the stream id every time to figure out where
to narrow.

For old realms, any old URLs will still work under the new scheme,
assuming the stream hasn't been renamed (and of course old urls
wouldn't have survived stream renaming in the first place).  The one
exception is the hopefully rare case of a stream name starting with
something like "99-" and colliding with another stream whose id is 99.

The way that we enocde the stream name portion of the URL is kind
of unimportant now, since we really only look at the stream id, but
we still want a safe encoding of the name that is mostly human
readable, so we now convert spaces to dashes in the stream name.  Also,
we try to ensure more code on both sides (frontend and backend) calls
common functions to do the encoding.

Fixes #4713
2018-02-19 09:03:11 -08:00
Rhea Parekh b702bbe5a1 slack importer: Allocate ids in a single db query.
We use the command
'select nextval('sequence') from generate_series(1, increment_number)'
which returns a list of allocated values for the ids.

This list is used to assign ids to the to be converted objects.
2018-02-19 08:55:50 -08:00
Rhea Parekh 5dfacfcfca slack importer: Change 'allocate_ids' to return a list of ids.
Update the callers of this function to process the list and add
tests for the same.
2018-02-18 20:47:45 -08:00
Shubham Dhama 9feae472f8 org settings: Add button to deactivate organization.
This adds button under "Organization profile" settings, which
deactivates the organization and sends an "event" to all the
active user and log out them.

Fixes: #8212.
2018-02-18 10:20:38 -08:00
Greg Price 693a9a5e70 push notifs: For group PMs, identify the users in the group. 2018-02-16 16:06:03 -08:00
Shubham Dhama a32e1eb913 markdown: Require double-asterisk around all mentions.
This enforces `**` around all the mentions including "at-all" and
"at-everyone" mentions. Hence this makes `@all` and `@everyone`
invalid mentions, resulting into proper syntax for these mentions as
`@**all**` and `@**everyone**` respectively.

Note from tabbott: This removes an old feature/syntax, which made
sense back when @Tim was also a way to mention a user with Tim as
their first name.  Given how nice typeahead is now, the user part of
the feature was removed a while ago; this should have gone at the same
time.

Fixes: #8143.
2018-02-16 11:45:08 -08:00
Eeshan Garg 6206647641 bugdown/api_code_examples: Add support for multiple languages.
This commit modifies the Markdown extension in bugdown/api_code_examples.py
to support rendering code examples in multiple languages by specifying
the language like so:

{generate_code_example(python)|doc.md|example}

This makes us one step closer towards adding support for testable
JavaScript code examples.
2018-02-16 10:07:13 -08:00
Eeshan Garg 73a1f8aa3e api docs: Add comment about why the event queue example isn't tested.
This commit simply adds a comment in zerver/lib/api_test_helpers.py
explaining why it isn't worth the effort to explicitly test the
code example in api/get-events-from-queue.md.
2018-02-16 10:07:13 -08:00
Shubham Padia a4b686297a api: Return anchor in get_messages when use_first_unread_anchor=True.
This may be helpful for some API clients, since it avoids them needed
to do somewhat messy post-processing on the results (the data was
always available via scanning for the first unread message in the result).

Fixes #6244.
2018-02-16 10:06:20 -08:00
Robert Hönig a19a69bfe3 embedded bots: Log warning when bot quit()s.
External bots may call bot_handler.quit() when
they wish to terminate, e.g. due to a misconfiguration.
Currently, embedded bots ignore calls to quit(), even
though they signal a problem. This commit does the first
step in handling quit() calls by logging a warning.
2018-02-13 14:56:37 -08:00
neiljp (Neil Pilgrim) 756af7518c REQ: Add type parameter to act as hint for typing. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) ec829ea832 mypy: Add 2 Optional[StreamTopicTarget] annotations in actions.py. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 79582a59f1 mypy: Move assertion into similar branch in get_recipient_info. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 3cb12230b2 mypy: Annotate email_gateway_bot in export_files_from_s3(). 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 16b939a964 REQ: Ensure only initial parameter is positional & adjust views/messages.py. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) b0126ca8dc mypy: Amend diff_strings to return str or raise DiffException. 2018-02-13 11:40:51 -08:00
neiljp (Neil Pilgrim) 23cc7e1671 mypy: Allow acting_user of do_change_full_name() to be None.
This is based on usage in bulk_change_user_names.py, and that
the RealmAuditLog acting_user field is Optional[UserProfile].

This could be more meaningfully changed in future, perhaps to
indicate that the command was run by a specific zulip user.
2018-02-13 11:40:51 -08:00
Vishnu Ks 81f18bd69a actions: Create do_create_multiuse_invite_link function. 2018-02-13 09:31:45 -08:00
Greg Price ecbc72b857 push notifs: Add a diagnostic in API of whether push notifs enabled.
When the answer is False, this will allow the mobile app to show a
warning that push notifications will not work and the server admin
should set them up.

Based partly on Kunal's PR #7810.  Provides the necessary backend API
for zulip/zulip-mobile#1507.
2018-02-12 14:34:59 -08:00
Greg Price 8be2dfa81c APNs: Fix a case I broke while working out tests.
I got distracted, came back later to a successful test run in my
terminal, and thought I remembered finishing the change and just
kicking off a final test run to check.

In fact, there was an `assert False` right in the normal case for
production, and I just hadn't finished a test for that path. (m.-)
Definitely the most grateful I've been for our coverage checks,
which highlighted this for me.

Remove the `assert False`, and also finish writing the test it was
there to help me write.  Those lines are covered now.
2018-02-09 18:28:16 -08:00
Umair Khan 0eca2e102d cache: Add ignore_unhashable_lru_cache function.
This is a wrapper over lru_cache function. It adds following features on
top of lru_cache:

    * It will not cache result of functions with unhashable arguments.
    * It will clear cache whenever zerver.lib.cache.KEY_PREFIX changes.
2018-02-09 18:14:08 -08:00
Greg Price 7f6a1714aa APNs: Don't try to send anything when not configured. 2018-02-09 17:16:21 -08:00
Robert Hönig 323284e0b6 backend: Call real endpoint in create_test_bot().
This allows tests to be more realistical and to
directly test payload added to add_bot_backend().
2018-02-09 12:30:24 -08:00
Rhea Parekh fbc11b141b slack importer: Add 'realm_name_in_notifications' field in zerver_userprofile.
This was recently added in b875fe07eb and
hence should be updated here.
2018-02-09 12:17:10 -08:00
Rhea Parekh be05bccb5b slack importer: optimize allocation of id range before import. 2018-02-09 12:17:10 -08:00
Rhea Parekh a0b7976f77 slack importer: Fix output_dir path and remove sys.exit(0).
Also, as the exception is being handled in 'slack_data_to_zulip_data'
remove it from the manage.py command 'convert_slack_data'.
2018-02-09 12:17:10 -08:00
Rhea Parekh c0e30079f6 slack importer: Get user data from a get request to slack users api.
The fresh imported data shows that the users emails are not included
in the data. However, the data received from the older method of slack
(which is using legacy tokens) contains the email data of the users.
2018-02-09 12:17:10 -08:00
Tim Abbott 77addc5456 bugdown: Fix handling of ultra-long renderings.
If some bug in Bugdown results in a rendered message content that is
bigger than twice the message size, we now just throw an exception
from Bugdown.  This is considerably better than the old behavior,
which might result in an enormous message being placed in the database
(potentially, bigger than the 1MB limit to store in memcached), which
would in turn result in tragic consequences.

This fixes #8322, in that it prevents the super bad outcome seen there
(where basically Zulip became unusable for everyone on the stream
where the message is posted).  Now, the failure mode is just the
message failing to send.  Still not ideal (and requires further work
on the URL embed feature), but not a minor problem, not a major one.
2018-02-09 10:57:55 -08:00
Eeshan Garg 3853577486 api tests: Add example/tests for unsubscribing other users.
This commit adds tests (and thus, an extra code example) for
unsubscribing another user from a particular stream by passing in
the `principals` argument to client.remove_subscriptions. The
ability to pass in `principals` was added in the latest release
of the zulip API PyPI package.
2018-02-08 17:58:41 -08:00
Eeshan Garg ac5c355a60 api/delete-queue: Make fixtures/examples testable.
api/delete-queue now uses the Markdown extension in
bugdown/api_code_examples to generate code examples and fixtures
from tests.
2018-02-08 17:58:41 -08:00
Eeshan Garg 344cf2c8ee api_tests: Add tests for client.register_queue. 2018-02-08 17:58:41 -08:00
Eeshan Garg 66ca0c725f api/get-all-users: Make fixtures/examples testable.
api/get-all-users now uses the Markdown extension in
bugdown/api_code_examples to generate code examples and fixtures
from tests.
2018-02-08 17:58:41 -08:00
Eeshan Garg 929724e5e7 api docs: Add page for common error payloads.
We now have a separate page for common error payloads, for example,
the payload for when the client's API key is invalid. All error
payloads that are presented on this page will be tested similarly
to our other non-error sample fixtures.
2018-02-08 17:58:41 -08:00
Rhea Parekh 48640fd28f slack importer: Suppress logger output from the unit tests. 2018-02-08 16:21:35 -08:00
Eeshan Garg 61e4fc3b74 api/remove-subscriptions: Make fixtures/examples testable.
api/remove-subscriptions now uses the Markdown extension in
bugdown/api_code_examples to generate code example and fixtures
from tests.
2018-02-07 15:58:57 -08:00
Eeshan Garg f4cb5e70ed api/add-subscriptions: Make fixtures/examples testable.
api/add-subscriptions now uses the Markdown extension in
bugdown/api_code_examples to generate code example and fixtures
from tests.
2018-02-07 15:58:57 -08:00
Eeshan Garg 09721e9ec3 bugdown/api_code_examples: Support multiple fixtures per doc.
This commit adds support for passing in an argument to the macro
"call" to explicitly specify a fixture to render, like so:

{generate_code_example|doc_name|fixture(stream_message_with_args)}
2018-02-07 15:58:57 -08:00
Eeshan Garg 7ae704d1bb api/get-all-streams: Render multiple examples from single test method.
api/get-all-streams - make use of the api_code_example Markdown extension's
feature of recursively extracting multiple code examples from a single
test method.
2018-02-07 15:58:57 -08:00
Eeshan Garg 0cca39d1fc bugdown/api_code_examples: Support multiple examples in one test.
This commit adds support for recursively extracting multiple code
examples from a single test method.
2018-02-07 15:58:57 -08:00
Eeshan Garg ffe930d50d bugdown/api_generate_examples: Replace 'method' with 'example'.
To generate a code exammple,

{generate_code_example|<api_doc_md>|example} sounds better and
more intuitive than,

{generate_code_example|<api_doc_md>|method}
2018-02-07 15:58:57 -08:00
Ricky 1303248b0b integration: Add statuspage.io webhook integration. 2018-02-07 09:59:12 -08:00
Greg Price c80802ff1e bots: Clean up create_realm_internal_bots.
This code duplicated the code in setup_realm_internal_bots, with some
added logic to avoid trying to create the same bot twice.  That logic
was buggy so that it would never work at all -- it subtracted a set of
UserProfile objects from a set of email strings -- so it looked like
the command might blow up when run after the users already existed.

In fact, the buggy logic wasn't necessary, because the work the
command does after it is idempotent -- in particular `create_users`,
within its subroutine `bulk_create_users`, already filters out users
that already exist.  So just cut the buggy stuff out, deduplicate the
rest with `setup_realm_internal_bots`, and document that invariant on
the latter.

While we're here, in the common case bail early without doing any
per-realm work in Python, since we're running this on every upgrade.
2018-02-06 18:59:57 -08:00
Rhea Parekh 83a7fd84ab slack importer: Import primary owner user first.
According to https://get.slack.help/hc/en-us/articles/201912948-Owners-and-Administrators,
only one Primary owner of a slack organsation exists. This allocates the first id
to the Primary owner and hence makes sure that the primary owner is imported first.
Added tests for the same.
2018-02-06 14:48:30 -08:00
Rhea Parekh 052e3e1540 slack importer: Change organization admin mappings.
Map 'Primary owner', 'owner' and 'admin' to 'organization admin'.
Added tests for the same.
2018-02-06 14:48:30 -08:00
Rhea Parekh b3b6023230 slack importer: Always map 'is_staff' to false in user data.
"staff" is only for server administrators, which doesn't exist in Slack.
Hence, this should always be false.
2018-02-06 14:48:29 -08:00
Roman Godov b875fe07eb settings: Added setting to turn on and off realm name in email subject.
Users having only account in one realm will not be distracted by realm
name in subject lines of every email.  Users who have multiple
accounts in realms can turn this setting on and receive a
corresponding realm name in email's subject.

Tweaked by tabbott to rebase and address a few small issues.

Fixes #5489.
2018-02-05 18:01:54 -08:00
Aditya Bansal ecbb763e79 notifications: Handle an edge case in relative_to_full_url.
For messages where the entire rendered body is a message_inline_image
object, we actually don't display any text and just display the
image. These messages may have links to images which might or might
not be internal to Zulip but in both cases there is a chance of this
links being broken when accessed by an email server like Gmail that
doesn't possess the recipient user's cookies.

We don't want to have ugly looking broken images displayed in email
notifications.  So we patch this by inserting a replacement for the
`message_inline_image` block in which we essentially replace the
content with the textual link.

Edited for clarity by tabbott.
2018-02-05 17:30:55 -08:00
Greg Price ca1129dea3 Revert "Streams: Disallow certain markdown characters in stream names."
This reverts commit dd1b14378b.

Deleted a conflicted test entirely.
2018-02-05 16:55:21 -08:00
Rhea Parekh 811294c6cc slack importer: Add test for checking link in message conversion. 2018-02-05 14:46:39 -08:00
Rhea Parekh eb7a9675a4 slack importer: Add unit tests. 2018-02-05 14:46:39 -08:00
Rhea Parekh 2b11bff9ad slack importer: Refactor script for easier unit tests.
This is to break the bigger functions into smaller
ones and hence, makes unit testing easier.

Also includes renaming the stream's topic name to 'from slack'.
2018-02-05 14:46:39 -08:00
Tim Abbott b84f2223a5 avatars: Always include a working URL for the user's own avatar.
This fixes an issue where the user's own avatar was being sent down
the wire as None.  We could have fixed it, as in #8265, by adding code
in the webapp and mobile apps to compute medium-size gravatar URLs as
well, but that would be messy, and there's little benefit to that
complexity (saving at most 2 URLs from the payload).

Fixes #8253.
2018-02-05 12:45:41 -08:00
rht 9161f8c39b zerver/lib: Remove u prefix from strings. 2018-02-05 12:12:58 -08:00
Privisus 9fdd6f2bd4 integrations: Add Insping integration. 2018-01-31 14:36:54 -08:00
rht 0c4066b0f3 django-2.0: Change parameter name to app_label.
showmigrations command expects a parameter with the name app_label.
2018-01-31 12:07:36 -08:00
Eeshan Garg 162f34f0d0 api/get-profile: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
2018-01-31 07:30:54 -05:00
Eeshan Garg 2bea4b70de api/create-user: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
2018-01-31 07:30:54 -05:00
Eeshan Garg fe98a59f5c bugdown/api_code_examples: Add macro for admin zuliprc lines.
Some of our code examples can only be run with administrator
credentials (such as create-user). Thus, the Markdown extension
for generating code examples should have an option to include
the lines that recommend using an admin zuliprc instead of a
non-admin one.
2018-01-31 07:30:54 -05:00
Eeshan Garg 73a3755120 api/get-all-streams: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
2018-01-31 07:30:54 -05:00
Eeshan Garg dcf3a9cd18 api/get-subscribed-streams: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg 6c634d6847 api/get-stream-id: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg b1878f14a2 api/update-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg cf80587f47 api/private-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg f58ecee2d8 api/stream-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg c158869096 Rename tools/lib/api_tests.py to zerver/lib/api_test_helpers.py.
Now that the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples depended on code in the
tools/lib/* directory, it caused the production tests to fail since
the tools/ directory wouldn't exist in a production environment.
2018-01-31 07:30:54 -05:00
Eeshan Garg c89550cfa0 api docs: Add Markdown extension for generating examples/fixtures.
This commit adds a Markdown extension that allows the following
syntax,

{generate_code_example|<md_file_name>|<fixture or method>}

to generate code examples and fixtures found in tools/lib/api_tests.py
and templates/zerver/api/fixtures.json, respectively.
2018-01-31 07:30:54 -05:00
Greg Price cad4083987 export: Fix an unnecessary Any.
This was introduced a few weeks ago in
ed4054d11 "Import script: Check and add system bots after every import."
2018-01-30 15:34:47 -08:00
rht 9a8d2244ca django-2.0: Shift to resolvers from urlresolvers.
The old name is deprecated.
2018-01-30 10:53:54 -08:00
Vishnu Ks 43a6439b3b upload: Enforce per-realm quota. 2018-01-29 16:06:11 -08:00
Vishnu Ks 035d3c5e97 upload: Remove UserProfile field for old per-user quota. 2018-01-29 16:06:11 -08:00
Greg Price 55cf54c087 upload: Remove old per-user quota feature.
We'll replace this primarily with per-realm quotas (plus the simple
per-file limit of settings.MAX_FILE_UPLOAD_SIZE, 25 MiB by default).

We do want per-user quotas too, but they'll need some more management
apparatus around them so an admin has a practical way to set them
differently for different users.  And the error handling in this
existing code is rather confused.  Just clear this feature out
entirely for now; then we'll build the per-realm version more cleanly,
and then we can later add back per-realm quotas modelled after that.

The migration to actually remove the field is in a subsequent commit.

Based in part on work by Vishnu Ks (hackerkid).
2018-01-29 16:06:11 -08:00
Rohitt Vashishtha e2a42650c9 emails: Fix link to org-admin help docs. 2018-01-27 15:33:36 -08:00
Shubham Dhama dbce49ac53 refactor: Move topic name checks to Addressee class.
In order to get test coverage on topic name checks, we
do them in Addressee, so that we don't hit an assertion
first. The assertion in question is in Addressee.topic(),
and it was added partly to appease mypy.
2018-01-25 06:21:01 -05:00
Vishnu Ks a11b742984 messages: Calculate value of first visible message ID using cron job.
[greg: Fixed buggy time conversion in estimate_recent_messages.]
2018-01-24 17:15:08 -08:00
Patrick Grave f2b322c32d mypy: Use Python 3 syntax for typing in upload.py 2018-01-23 08:49:42 -08:00
Patrick Grave 008e250668 mypy: Use Python 3 syntax for typing in test_runner.py 2018-01-23 08:49:42 -08:00
Rhea Parekh 11b549e566 slack importer: Add regex for mail links.
`<mailto:foo@foo.com>` is changed to `mailto:foo@foo.com`.
2018-01-23 10:01:15 -05:00
Rhea Parekh df1bb5fd27 slack importer: Fix link regex and add test case. 2018-01-23 10:01:15 -05:00
Rhea Parekh 5b1e9f8181 slack importer: correct the implementation of unzipping slack data file.
Slack zip file unzips in the same folder.
2018-01-23 10:01:15 -05:00
Rhea Parekh 845a8d63bf slack importer: Prevent id clashes for import in an active db.
Previously we had a problem of id clashes while importing converted
slack data into an existing zulip instance with realms which are actively
populating the database.

This counts the total objects to be imported and does a db transaction
to increase the SEQUENCE number for that table by that number,
and hence allocates a range of ids for the to be converted slack data
objects.
2018-01-23 10:01:15 -05:00
Weronika Grzybowska 7ac7100a1d messages: Make checking for status message consistent with backend.
Adds a check for newline that was present on backend, but missing in the
frontend markdown implementation. Updating messages uses is_me_message flag
received from server instead of its own partial test. Similarly, rendering
previews uses markdown code.

Fixes #6493.
2018-01-23 09:26:41 -05:00
Robert Hönig 9a15c4e3ff Add bot services to page_params.
This is the first step for allowing users
to edit a bot's service entries, name the
outgoing webhook configuration entries. The
chosen data structures allow for a future
with multiple services per bot; right now,
only one service per bot is supported.
2018-01-23 07:29:00 -05:00
Vishnu Ks 036dc53d20 messages: Rename last_visible_message_id to first_visible_message_id. 2018-01-22 19:53:44 -08:00
YJDave 8905744bb1 stream settings: Add period at the end of error messages. 2018-01-19 13:05:43 -05:00