Commit Graph

3770 Commits

Author SHA1 Message Date
Adam Birds 5855fe3e53 integrations: Add Zabbix webhook.
With a few changes to the documentations by Eeshan Garg.
2018-06-07 08:31:24 -07:00
Aditya Bansal 1205e02c64 provisioning: Don't rebuild DB if running migrations is sufficient.
This results in a significant optimization in the performance of
re-provisioning Zulip if all that you're doing is rebasing onto a
newer version of master (which just adds new migrations).

The change carries some risk of generating unpleasant-to-debug
situations, because if we merge a buggy migration and then later fix
it, some clients may not have a properly migrated database (and also,
this changes how populate_db commutes with migrations).  But it seems
worth it, given how much time is currently wasted by not having this.

Fixes: #9512.
2018-06-06 13:43:59 -07:00
Aditya Bansal 65dc80fe9d test_fixtures: Add run_generate_fixtures_if_required function.
In this commit we are adding run_generate_fixtures_if_required,
a new function which is meant to de-duplicate a bit of code
between test-server and test-backend which is essentially
responsible for rebuilding the test database if that was required.
2018-06-06 13:37:35 -07:00
Aditya Bansal f7c11d1747 test_fixtures: Refactor to have template_database_status API.
In this commit we are essentially just refactoring the function
is_template_database_current to be called template_database_status
and adjusting the return values accordingly.
This is essentially a preparatory commit for the upcoming commits
which will essentially enable us to not throw away entire DB and
rebuild from scratch if only running migrations could do the job.
2018-06-06 13:37:35 -07:00
Vishnu Ks 53237d39aa registration: Allow users to import profile picture. 2018-06-06 13:29:42 -07:00
Vishnu Ks ca87cf4c97 upload: Seperate writing avatars into s3 as function. 2018-06-06 13:29:42 -07:00
Vishnu Ks 7960785261 upload: Separate writing avatars as local files into function. 2018-06-06 13:29:42 -07:00
Yashashvi Dave 5145b24635 users: Replace duplication with generic func to validate user id.
This adds a common function `access_user_by_id` to access user id
within same realm, complete with a full suite of unit tests.

Tweaked by tabbott to make the test much more readable.
2018-06-05 11:13:13 -07:00
Shubham Dhama 01555e8772 streams: Handle guest user ids for stream settings changes' events. 2018-06-04 11:35:37 -07:00
Steve Howell a42492d0ac Fix very recent build error w/url coverage for casper.
Example:
    http://localhost:9991/casper/casper-failure1.png
2018-06-04 12:37:10 -04:00
Tim Abbott ec90322e5d bots: Fix behavior for bots mentioned in PMs.
We've for a long time had the behavior that a bot mentioned in a
stream message receives the notification, regardless of whether the
bot was actually subscribed to the stream.

Apparently, this behavior also triggered if you mentioned a bot in a
private message (i.e. the bot would be delievered the private message
and would probably respond unhelpfully in a new group private message
thread with the PMs original recipients plus the bot).

The fix for this bug is simple: To exclude this feature for private
messages.
2018-06-03 22:29:15 -07:00
Tim Abbott ec2eb0edba storage: Don't double-minify webpack bundles.
What was happening before is that we built the webpack bundles in
tools/minify-js with nicely hashed filenames, and then `manage.py
collectstatic` was extending these filenames with a second hash
through the use of storage.

Removing the first one didn't seem ideal, but would probably have
worked, but seems confusing for people only familiar with webpack
(ideally, we want the Django toolchain piece to be increasingly
invisible as we replace it).

And we can't exclude the webpack bundles from being processed by
storage, since we need these bundles to be included in the manifest.
So, instead, we set the hash function to be a no-op for the bundle
files.

Fixes significant portions #5971.

More work is required to deal with versioning for some of the
image/font assets.
2018-06-03 16:49:59 -07:00
Tim Abbott 714ad67db1 storage: Move library code into zerver/lib.
There's no particularly good reason for this to have been a top-level
file.
2018-06-03 16:02:02 -07:00
Shubham Dhama 4483e33102 digest: Make newly registered users data inaccessible to guest users.
The new can_access_all_realm_members function is meant to act as a
base function for guest users and Zephyr realm users regarding the
accessibility of the information of other users in the realm.
2018-06-03 09:30:59 -07:00
Shubham Padia 8883107a7f streams: Allow cross-realm bots to post to announcement_only streams.
This fixes an issue where if you make #announce (the default
announcement stream) announce-only, then creating a new stream will
throw an exception (because notification-bot can't send there).

Fixes #9636.
2018-06-02 10:36:29 -07:00
Steve Howell 47b4dd6bdb slash commands: Refine /day and /night.
These two slash commands now use zcommand to talk to
the server, so we have no Message overhead, and if you're
on a stream, you no longer spam people by accident.

The commands now also give reasonable messages
if you are already in the mode you ask for.

It should be noted that by moving these commands out of
widget.py, they are no longer behind the ALLOW_SUB_MESSAGES
setting guard.
2018-06-02 09:40:12 -07:00
Rhea Parekh f8e53511ea slack import: Add support for bold-italics formatting.
Fixes #8927
2018-06-02 09:01:55 -07:00
Tim Abbott 18702b0d71 actions: Document validate_user_access_to_subscribers_helper.
This function is important, carefully optimized, and hard to read, so
it deserves a long docstring.
2018-06-02 08:46:58 -07:00
Shubham Dhama 8e2337509d streams: Hide public streams & subscribers from guest users.
The main change here is making can_access_public_streams
2018-06-02 08:39:39 -07:00
Eeshan Garg 52c3c00ea0 test-api: Test against generic fixtures.
In a few commits before this one, we just added de-duplicated
generic fixtures that apply to multiple API tests. The tests
needed to be modified to accommodate that change.
2018-06-01 14:06:26 -07:00
Yago González 9f98002b76 tests: Allow testing our REST API against the OpenAPI docs. 2018-06-01 12:03:31 -07:00
Tim Abbott 0d84eb0a8b mypy: Clean zerver/views/users.py for strict-optional. 2018-06-01 08:48:17 -07:00
Tim Abbott 0a0ae5e703 users: Move add_service to zerver.lib.users so it is reusable. 2018-06-01 08:48:17 -07:00
Eeshan Garg 1df6c22df0 webhooks/pagerduty: Fix small capitalization error in docs.
It is PagerDuty, not Pagerduty!
2018-05-31 23:42:07 -04:00
Sampriti Panda e95f972e24 migrate: Add do_batch_update method for running batch updates.
* Refactor pgroonga_0002 migration to use new method.
2018-05-31 13:00:34 -07:00
Tim Abbott b9b81cf658 export: Rename ALL_ZERVER_TABLES to ALL_ZULIP_TABLES.
They don't all start with zerver, now :).
2018-05-31 10:47:27 -07:00
Tim Abbott 42aea68df3 export: Automate validation of ALL_ZERVER_TABLES.
This should help make it explicit whenever we add a new table to Zulip
that we need to correctly categorize it for whether it will be
included in the data export, or not.
2018-05-31 10:47:27 -07:00
Tim Abbott 328136344a import: Fix typo in zerver_customprofilefieldvalue table name.
Apparently, we were doing this slightly wrong.
2018-05-31 10:47:27 -07:00
Shubham Padia cd1233d3f7 upload: Add test jpg image with no exif data. 2018-05-31 08:08:59 -07:00
Steve Howell 554883e039 submessages: Add submessage_id to submessage event.
It's useful to have the submessage_id so that widgets
can in theory sort all the events (and not have to rely
on them arriving in sequence).
2018-05-30 14:47:36 -07:00
Shubham Padia 214ce1ccca streams: Add create and edit ui for is_announcement_only.
The user can now specify the value while creating a stream.
An admin can later change it via `Change stream permissions`
modal. Add is_announcement_only to subscription type text.
2018-05-30 14:22:15 -07:00
Shubham Padia bb8ad15fa9 streams: Set is_announcement_only when creating streams.
Adds is_announcement_only to create_stream_if_needed and
add_subscriptions_backend.
2018-05-30 14:17:33 -07:00
Steve Howell bfc13ecea4 submessages: Just send content for submessage events.
For some reason in my original version I was sending both
content and data to the client for submessage events,
where data === JSON.parse(content).  There's no reason
to not just let the client parse it, since the client
already does it for data that comes on the original
message, and since we might eventually have non-JSON
payloads.

The server still continues to validate that the payload
is JSON, and the client will blueslip if the server
regressses and sends bad JSON for some reason.
2018-05-30 13:53:11 -07:00
Tim Abbott f8e8d1fb27 upload: Suppress coverage warnings for now. 2018-05-30 09:31:47 -07:00
Tim Abbott bbcdf8ca99 upload: Fix logic for orientation when uploading avatars.
The previous logic was broken for an avatar without exif data.
2018-05-30 09:22:36 -07:00
Shubham Dhama 1d9afc7328 i18n: Pass translation data in page_params.
This is a preliminary step to make translation_data available as a
resource to `i18next.init()`.
2018-05-30 08:57:26 -07:00
Robert Hönig 647c63050f botserver: Add outgoing webhook tokens to botserverrc.
The tokens will be used to authorize the server when sending
messages to the Botserver.
2018-05-30 10:00:19 -04:00
Yashashvi Dave 47aaf4e20a users: Replace duplication with generic func to validate bot id.
This adds a common function `access_bot_by_id` to access bot id within
same realm.  It probably fixes some corner case bugs where we weren't
checking for deactivated bots when regenerating API keys.
2018-05-29 15:47:27 -07:00
Rhea Parekh 1db442acd4 IRC: Add integration documentation. 2018-05-29 10:59:53 -07:00
Rhea Parekh 25e3515d9b Matrix: Add integration documentation. 2018-05-29 10:59:53 -07:00
Shubham Padia 8b8a9be377 upload: Rotate image according to exif data when resizing avatar and emojis.
Fixes the avatar/emoji part of #8177.

Does not address the issue with uploaded images, since we don't do
anything with them.

Also adds 3 images with different orientation exif tags to
test-images.
2018-05-29 10:39:39 -07:00
Tim Abbott bf9012a559 message: Do future-proofing for public streams with private history.
This should have no effect for now, but it'll make things a bit
simpler in case we make future changes to support public streams
without history public to subscribers (and other organization
members).
2018-05-28 12:25:57 -07:00
Shubham Dhama b410aeb897 message: Limit message access for guest users.
Tweaked by tabbott to interact correctly with the
history_public_to_subscribers feature.
2018-05-28 12:20:32 -07:00
Yashashvi Dave e82c879b85 custom fields: Add user type of custom fields.
Fixes #8878
2018-05-27 23:01:21 -07:00
Vishnu Ks c90aac56ba registration: Copy user full_name when importing settings. 2018-05-27 22:39:21 -07:00
Rhea Parekh 468afe4840 export: Support export of Custom emojis.
Export of RealmEmoji should also include the image
file of those emojis.

Here, we export emojis both for local and S3 backend
in a method with is similar to attachments and avatars.

Added tests for the same.
2018-05-27 21:54:20 -07:00
Rhea Parekh 7198cc3899 import: Fix RealmEmoji import bug.
RealmEmoji should be imported after UserProfile,
as the new user_profile ids are not allocated
if we import it before.
2018-05-27 21:54:20 -07:00
Rhea Parekh 1b7b9a7164 import: Fix reaction import bug.
In 'zerver_reaction', the emoji_code should be updated
with the RealmEmoji allocated id when the 'reaction_type'
is 'realm_emoji'. Hence we add an extra field 'reaction_field'
in 're_map_foreign_keys', to process the above mentioned
condition.
2018-05-27 21:54:20 -07:00
Rhea Parekh 7a8b853708 Export: Support export of reactions.
We get the reactions from the messages exported.
2018-05-27 21:54:20 -07:00
Rhea Parekh c79d7f1070 Import: Move zerver_reaction from 'messages-000001.json' to 'realm.json'.
Also change the existing slack conversion script structure, to
include 'zerver_realm' in 'realm.json'.
2018-05-27 21:54:20 -07:00
Yago González f84c9b919b api docs: Read parameters and response fixtures from OpenAPI files. 2018-05-26 22:49:55 +02:00
Yago González 30682241c7 api docs: Move OpenAPI files to zerver/openapi. 2018-05-26 21:48:17 +02:00
Tim Abbott 445932cc86 bugdown: Increase limit for number of digits in tweet IDs.
Apprently, they just reached 19 digits.  We set the limit to 30 to
give plenty of headroom for the future.
2018-05-26 10:12:15 -07:00
Yago González b4084d30d2 emails: Move user role logic for followup_day1 to notifications.py. 2018-05-26 07:29:58 -07:00
Yago González d87f0ccb4e api docs: Show Yes/No in the "Required" field of the args table. 2018-05-25 23:31:30 +02:00
Tim Abbott 4e70c9402a export: Fix path logic for exporting avatars with S3 backend.
Apparently, we missed this when we converted the export format to use
longer path names for avatars.
2018-05-25 12:04:34 -07:00
Robert Hönig 7a8c1ec9dc outgoing webhooks: Send additional useful data.
This adds the fields `trigger` and `service_email`
to each message event dispatched by outgoing webhook bots.
`trigger` will be used by the Botserver to determine if
a bot is mentioned in the message.
`service_email` will be used by the Botserver to determine
by which outgoing webhook bot the message should be handled.
2018-05-25 10:33:40 -07:00
Eeshan Garg 0a1b429d04 settings: Remove PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS.
Now that we have a UI for this feature, we don't need this setting
anymore.
2018-05-24 17:18:14 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Steve Howell 69517f5ac5 Support zform-based widget content in the server.
API users, particularly bots, can now send a field
called "widget_content" that will be turned into
a submessage for the web app to look at.  (Other
clients can still rely on "content" to be there,
although it's up to the bot author to make the
experience good for those clients as well.)

Right now widget_content will be a JSON string that
encodes a "zform" widget with "choices."  Our first
example will be a trivia bot, where users will see
something like this:

    Which fruit is orange in color?

        [A] orange
        [B] blackberry
        [C] strawberry

The letters will be turned into buttons on the webapp
and have canned replies.

This commit has a few parts:
    - receive widget_content in the request (simply
        validating that it's a string)
    - parse the JSON in check_message and deeply
        validate its structure
    - turn it into a submessage in widget.py
2018-05-24 09:30:22 -07:00
Umair Khan e063bb0d50 2FA: Add tests for two-factor auth. 2018-05-23 15:49:35 -07:00
Tim Abbott 9c29592178 mypy: Fix rebase errors in create_user.py. 2018-05-23 11:36:28 -07:00
Vishnu Ks 8369e2b15e registration: Allow users to import settings from other realm.
This should significantly improve the user experience for creating
additional accounts on zulipchat.com.

Currently, disabled in production pending some work on visual styling.
2018-05-23 10:30:00 -07:00
Vishnu Ks 2a66818e2e registration: Add function to copy user settings.
This is intended to support our upcoming feature to support copying a
user's customization settings from an existing account that user owns
in another organization.
2018-05-23 10:12:14 -07:00
Rhea Parekh c24c249b8c export: Support export of Custom Profile Field. 2018-05-23 09:07:26 -07:00
Robert Hönig ac04553d67 outgoing_webhook.py: Fix incorrect variable type. 2018-05-23 08:57:56 -07:00
Aditya Bansal d343f25cc6 create_realm_internal_bots: Refactor to extract main op as a function.
We extract the entire operations of the management command to a
function create_if_missing_realm_internal_bots in the
zerver/lib/onboarding.py. The logic for determining if there are any realm
internal bots which have not been created is extracted to a function
missing_any_realm_internal_bots in actions.py.
2018-05-23 11:53:22 +05:30
Aditya Bansal 528230df41 test_fixtures.py: Add check_settings to is_template_database_current.
In this commit we add a new param (Django settings) on which the
is_template_database_current decision will depend upon.
2018-05-23 11:53:20 +05:30
Eeshan Garg 3ed20589f2 webhooks: Add generic exception for unexpected webhook events.
UnexpectedWebhookEventType is a generic exception that we may
now raise when we encounter a webhook event that is new or one
that we simply aren't aware of.
2018-05-22 08:30:19 -07:00
Michael J. Sullivan d5ee801d60 mypy: Set local_partial_types = True.
The daemon implicitly sets this flag, so set it explicitly and fix all
of the type errors.
2018-05-21 22:41:00 -07:00
Tim Abbott 80a331385e push_notifications: Fix missing mypy annotation. 2018-05-21 11:39:58 -07:00
Tim Abbott cec7686f3d push notifications: Clean up unregistered/bad APNS tokens.
We've had this sort of logic for GCM for a long time; it's worth
adding for APNS as well.

Writing this is a bit of a reminder that I'm not a fan of how our unit
tests for push notifications work.
2018-05-21 11:30:56 -07:00
Tim Abbott c6909fb99c migration_tests: Document the migration test framework. 2018-05-21 09:59:55 -07:00
Aditya Bansal 0767a5b955 test_classes: Add class MigrationsTestCase for writing migration tests.
The MigrationsTestCase class is based on this blog post:
https://www.caktusgroup.com/blog/2016/02/02/writing-unit-tests-django-migrations/

It provides a basic framework for writing unit tests for django
db migrations.
2018-05-21 09:46:37 -07:00
Aditya Bansal a85576149d test_helpers: Add use_db_models decorator.
This is somewhat messy, but it's purpose is to make it easy to write
tests in test_migrations.py.
2018-05-21 09:46:24 -07:00
Tim Abbott 43ec7ed456 narrow: Add is_web_public_compatible check for narrows.
This will be used to limit which narrows will be allowed for
web-public browsing of Zulip.
2018-05-21 08:45:48 -07:00
Tim Abbott 6cbe9890aa embedded bots: Use get_active_user helper to look up users.
This should have no effect, but is part of a larger effort to avoid
using get_user (not get_active_user) unless we actually want inactive
users.
2018-05-20 20:04:16 -07:00
Tim Abbott e7fa77f120 validate_email_for_realm: Clarify errors for deactivated users.
If a user's account has been deactivated, we want to provide a special
error message that makes clear what's going on.

Future work is to provide some administrative controls on whether a
user should be able to re-activate their account.
2018-05-20 19:32:53 -07:00
Tim Abbott 9360af37d4 typing: Remove unnecessary conditional for recipient types.
It wasn't actually possible for `recipient_for_emails` to return a
STREAM regardless, and this makes things read a little clearer.
2018-05-20 18:35:48 -07:00
Tim Abbott ecdc7fb296 typing: Fix unnecessary else clause for recipient validation.
The other cases all return anyway.
2018-05-20 18:27:25 -07:00
Joshua Pan afe09071b9 slash-commands: Implement /day and /night. 2018-05-20 14:42:09 -07:00
Joshua Pan bdba539480 Rename get_fixed_content_for_widget to do_widget_pre_save_actions. 2018-05-20 14:42:09 -07:00
Sampriti Panda d94202c662 notifications: Add consistent order for sending email notifications.
Notification emails are sent in increasing order of the max message-id
of the unread message thread.
2018-05-20 10:19:13 -07:00
Tim Abbott 3006b3f52f url_preview: Fix crash when description has no content.
There's several things we'll want to cleanup with this feature, but
for now we're content to just make this not crash.
2018-05-17 12:40:43 -07:00
jkiely 058ee1ce1e mypy: Enable strict optional on lib/soft_deactivation.
Tweaked by tabbott to add assert statements, rather than new
conditionals.
2018-05-17 12:13:53 -07:00
jkiely 104fdd8bf9 mypy: enable strict optional for lib/message.
Add assert in order to pass checks under strict conditions.
2018-05-17 11:12:08 -07:00
jkiely 04c50cffa7 mypy: Enable strict optional in zerver/lib/bugdown.
Explicitly check for none in optional value and set it to a dict.
2018-05-17 11:12:01 -07:00
jkiely b3d43df498 mypy: Enable strict optional for lib/exceptions.
Change return type of reduce_ex to pass under stricter conditions.
2018-05-17 11:11:58 -07:00
jkiely ca3ce90496 mypy: Enable strict optional in lib/avatar.py.
Add assert to function and modify tests in order to pass under
strict conditions.
2018-05-17 11:11:55 -07:00
jkiely a1b5e7d6d5 mypy: Enable strict optional in lib/events.
Modify one variable assignment to pass under strict conditions.
2018-05-17 11:11:53 -07:00
jkiely ad065fde29 mypy: Enable strict optional for lib/notifications.
Modify fix_emojis function to pass under strict conditions.
2018-05-17 11:11:49 -07:00
Vishnu Ks 372e9740ac events: Add date_joined to user_dict. 2018-05-17 07:49:35 -07:00
Steve Howell 8812dba57e Add server-side widget support for poll/tictactoe. 2018-05-16 15:13:33 -07:00
Steve Howell 1f0ffdc415 Add widget.py to enable widgets on the server side.
The only slash command implemented in this initial
version is an extremely crippled version of a
"/stats" slash command that reports that you are
running 1 server.
2018-05-16 15:13:33 -07:00
Steve Howell de47eeb6f1 Add /submessage endpoint. 2018-05-16 15:13:33 -07:00
Steve Howell 4332fd64f7 Add submessages to message payloads. 2018-05-16 15:13:33 -07:00
Shubham Padia 4d70d032bc populate_db: Make is_web_public default to False.
Explicit mentions of `is_web_public` as False in the stream dict to be
passed to `bulk_create_streams` have been removed.
2018-05-16 13:40:22 -07:00
Shubham Padia 315047a38b populate_db: Make invite_only default to False.
Explicit mentions of `invite_only` as False in the stream dict to be
passed to `bulk_create_streams` have been removed.
2018-05-16 13:40:10 -07:00
Shubham Padia 7b4f6e0058 tests: Make is_announcement_only true for `announce` stream in populate_db.
Makes announce stream `is_announcement_only` for the dev db for easier
manual testing. The default value for `is_announcement_only` in
`bulk_create_streams` is False.
2018-05-16 13:35:45 -07:00
Tim Abbott f0ef335412 models: Remove unused ModelReprMixin class.
It appeared to be used as a base class in various Django migrations,
but because it didn't define any model fields, it wasn't actually.
2018-05-15 19:11:22 -07:00
Tim Abbott 351fab204b str_utils: Move force_bytes into ccache.py.
This is only used there, and so belongs in that bundle of
barely-maintained code.
2018-05-15 19:07:32 -07:00
Tim Abbott c46149efcb migrate: Remove obsolete act_on_message_ranges.
This has a cool structure, but it's written against the long-dead
South API, and we can always pull it out of the Git history if we want
to use this approach in the future.
2018-05-15 17:19:14 -07:00
Tim Abbott 16d807d157 Revert "test_helpers: Fix a nonexistent import."
This reverts commit fa18913b8b.

We fix the typo when doing this, though.
2018-05-15 16:08:29 -07:00
Greg Price fa18913b8b test_helpers: Fix a nonexistent import.
This module doesn't exist, and never did; the name appears to be a
mistaken variant of the module that really does contain ZulipTestCase.
So, fix the import to use the real name.

This would never have worked at runtime, which is why it's in an
`if False:`.  It's also an example of the kind of error that can be
hidden by `ignore_missing_imports`; we'd have caught the issue
immediately if we hadn't had a blanket application of that flag
in place.
2018-05-15 18:14:33 -04:00
Eeshan Garg f29b9f14fd integrations: Update docs for git and codebase.
Doing both as once due to the common dependency on
change-zulip-config-file-indented.md.
2018-05-15 18:02:56 -04:00
Tim Abbott 31ce382e71 coverage: Disable coverage checking for exceptions.py.
We also take the opportunity to slightly improve the errors if
AbstractEnum ever gets used.
2018-05-15 13:57:40 -07:00
Tim Abbott 191e6dc13a create_user: Remove user_profile_id option.
This hasn't been used in years, had no test coverage, and doesn't have
a clear use case.
2018-05-15 13:51:56 -07:00
= e731aeda44 zerver: Remove dead code from do_change_password. 2018-05-15 13:39:53 -07:00
= 9d07faaf0c zerver: Remove dead code from do_create_realm. 2018-05-15 13:39:53 -07:00
= 344d6544da zerver: Remove dead code for accessing subscribers.
These haven't been used in years, and clutter the codebase.
2018-05-15 13:39:39 -07:00
Baron Chandler f59adfa67c actions: Change do_change_is_admin Exception to an Assertion Error.
- do_change_is_admin now raises AssertionError when a non-admin
  permission is given.
- adds test to test_users to ensure admin asserts on invalid
  permission values.
2018-05-15 11:27:01 -07:00
Yago González f9f8d9c578 bugdown: Parse argument JSON files as streams. 2018-05-15 11:05:03 -07:00
RobbieClarken f81b936727 zerver/tests: Require 100% test coverage of zerver/lib/upload.py.
The last line here is impossible to test.

Fixes #4489.
2018-05-15 10:52:20 -07:00
Joshua Pan df84e1d7eb tests: Reach 100% coverage for zerver/liv/management.py.
Cleaned up add_user_list_args(). The "help" and
"all_users_help" have all default values. As noted in
an earlier commit, "all_users_help" is always passed in,
so we can get rid of "all_users_arg". We keep the default
for "all_users_help" so we don't have to change variable order
in function definition.
2018-05-14 10:46:21 -07:00
Joshua Pan ef098d2223 management: Remove the parameter required from add_user_list_args.
We remove an unecessary "required" paramter from this function
because as seen in the get_users() function right below, you have
to pass either -u/--users or -a/--all-users, meaning there should
never be a reason to require --users.
2018-05-14 10:46:21 -07:00
Joshua Pan 231b487bca tests: Cover check_config() in zerver/lib/management.py.
Add a comment to code to clarify what the config check does.
2018-05-14 10:46:21 -07:00
Tim Abbott 726017f682 actions: Remove dead do_change_bot_type function.
This is a tiny fraction of a feature we don't support, so we should
just kill it.
2018-05-13 17:58:49 -07:00
Tim Abbott 8b09118009 actions: Rename pick_color_helper to pick_color.
Now that there's only one function here, it's weird to have the
unnecessary _helper suffix.
2018-05-13 17:52:36 -07:00
Tim Abbott 234b5fa21b actions: Remove dead pick_color function.
We've been using pick_color_helper only for a long time.
2018-05-13 17:52:04 -07:00
Tim Abbott 60cfc210ce address: Remove dead msg_type function.
This was never used; we instead use `.is_stream` and `.is_private`
instead.
2018-05-13 17:47:21 -07:00
Aditya Bansal bd063b86c4 request.pyi: Remove unused import of typing.Text. 2018-05-14 05:16:22 +05:30
Eeshan Garg 5c0d4660c1 models: Add history_public_to_subscribers to Stream.to_dict().
This commit also updates all the relevant parts where this attribute
could be useful, e.g. payloads for user subscriptions.
2018-05-13 09:15:07 -07:00
Shubham Padia 295fcb8536 models: Add `is_announcement_only` to stream_dict in `actions.py`.
Adds `is_announcement_only` to `stream_dict`s in order to access
the property in the frontend.
2018-05-13 09:11:51 -07:00
Shubham Padia 897ed17f0c api: Allow realm_admins to make a stream announcement_only. 2018-05-13 09:11:51 -07:00
Shubham Padia bb8577ba94 stream: Only realm admins can post to an announcement_only streams.
If a non-admin tries to post to an announcement_only stream, error
message will be shown.
2018-05-13 09:11:51 -07:00
Aditya Bansal a68376e2ba zerver/lib: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Aditya Bansal 63dff4549f zerver/lib/test_classes.py: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Aditya Bansal c02011d7f5 zerver/lib/bugdown/__init__.py: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Aditya Bansal 97b7075dd2 zerver/lib/actions.py: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Aditya Bansal 1f9244e060 zerver/lib: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Tim Abbott 0a39eb2a58 export: Convert a bunch of error cases to AssertionError.
This reflects the fact that these are just defensive programming (we
don't expect them to ever happen) and also nicely makes these lines
not show up in our missing test coverage reports.
2018-05-09 20:49:13 -07:00
Tim Abbott 7e2841b358 bulk_create: Mark bulk_create_streams as nocoverage.
It's only used in `populate_db`, so it gets tested that way.
2018-05-09 20:49:12 -07:00
Raymond Akornor f67efd5291 zerver: Migrate away from six. 2018-05-09 18:38:25 +00:00
Tim Abbott fba45bb9d3 archive: Add functions for accessing web-public streams.
These will be helpful for variable upcoming projects to support the
web-public streams feature.
2018-05-08 12:12:15 -07:00
Tim Abbott f24630fd4a events: Include data for whether push notifications are enabled.
This is primarily useful for the mobile app, but could also be used to
control whether we display push-notifications related settings to
users in the web UI.
2018-05-08 11:45:13 -07:00
Tim Abbott 6317e4d63c events: Extract post_process_state.
This is a preparatory refactor for potentially being able to call this
from the web-public archive feature's code path.
2018-05-06 23:21:29 -07:00
Tim Abbott 112805251f fetch_initial_state_data: Use realm local variable.
This is just a simple code cleanup.
2018-05-06 23:20:52 -07:00
Yago González 6837fc5d56 i18n: Add missing strings for custom profile fields and fix capitalization.
The "Short/Long Text" option for custom profile fields wasn't properly
capitalized (i.e. "Text" should have been all lowercase), and also
wasn't properly tagged for translation.

For the sake of consistency, the change to proper capitalization has
also been applied to the models and any tests involving this feature.

Due to a bug in Django, it complained about the models having changed
and thus not being consistent with the migrations. That isn't actually
true (since the database stores the numeric values for each key), but
the migrations have been modified to avoid this error. This does not
affect the migrations' behaviour in any way.
2018-05-06 19:44:36 -07:00
Eeshan Garg 34d1b0ebf1 webhooks: Add helper to extract and validate HTTP event headers.
This is a part of our efforts to close #6213.
2018-05-05 15:48:37 -07:00
Eeshan Garg 19fa73891e actions: Refactor rate-limiting logic out of send_pm_if_empty_stream.
It makes sense to refactor out the last_reminder logic out of
send_pm_if_empty_stream and have a generic function that can send
rate-limited PM notifications to a bot owner and can be used by
methods other than send_pm_if_empty_stream.
2018-05-05 15:46:27 -07:00
Tim Abbott 956bd74905 attachments: Send events for attachment updates.
We send add events on upload, update events when sending a message
referencing it, and delete updates on removal.

This should make it possible to do real-time sync for the attachments
UI.

Based in part on work by Aastha Gupta.
2018-05-04 16:22:27 -07:00
Tim Abbott 69c4645bd2 attachments: Stop fetching attachments in / endpoint.
We only use this data in a rarely-used settings screen, and it can be
large after years of posting screenshots.

So optimize the performance of / by just loading these data when we
actually visit the page.

This saves about 300ms of runtime for loading the home view for my
user account on chat.zulip.org.
2018-05-04 16:22:26 -07:00
Tim Abbott 60185dddfd bots: Fix buggy database queries in loop.
A typo in my reading of 6cc2e8bbff meant
that we were incorrectly doing database queries for each Service
object, just to get the user_profile.id, which we already had.
2018-05-04 15:23:13 -07:00
Tim Abbott 59d3fefc07 bots: Improve performance of fetching core bot data.
This eliminates the need to call user_ids_to_users inside the
get_service_dicts_for_bots code path, saving a database query.

This completes my refactor to fix backend performance issues in this
code path.  Previously, our messy layering of queries that resulted in
Zulip doing work even if none of the bots actually had Services or
config_data.
2018-05-04 13:44:43 -07:00
Tim Abbott 0e81353ce0 bots: Merged get_services_for_bots into its caller.
This function is now only a few lines of simple code called in one
place, and is better eliminated.
2018-05-04 13:41:41 -07:00
Tim Abbott 6cc2e8bbff bots: Remove unnecessary select_related on Service objects.
Given how we're using these, there's no need for the full UserProfile
object to be fetched from the server.
2018-05-04 13:41:29 -07:00
Tim Abbott 186152bfc0 bots: Pass a list of user IDs into get_services_for_bots.
We weren't using the full profile objects.
2018-05-04 13:41:15 -07:00
Tim Abbott 4eb3c72c74 bots: Move get_services_for_bots into actions.py.
It's better to just have this bundle of code all in one place; also,
after some cleanup, we'll be inlining it into
get_service_dicts_for_bots.
2018-05-04 13:36:25 -07:00
Tim Abbott 894a952f6f bots: Compute embedded bots list using bot_dicts. 2018-05-04 13:34:54 -07:00
Tim Abbott 13f1f6a388 bots: Pass the full bot_dicts objects into get_service_dicts_for_bots.
This is preparatory refactoring for not needing to fetch the
bot_profile objects from the database.
2018-05-04 13:33:32 -07:00
Tim Abbott 447f8db8cb get_bot_configs: Adjust API to accept a list of bot user IDs.
This is preparatory refactoring for removing user_profile objects from
the get_service_dicts_for_bots code path.
2018-05-04 13:31:28 -07:00
Tim Abbott fdc1182a76 events: Optimize query for custom profile fields.
Our query for Custom Profile fields was for no good reason passing the
list of all users in the realm (potentially many thousands) into a
database query, rather than letting the database do that join.

Fixing this saves 100ms-200ms in the loading time for / on
chat.zulip.org for all users, since we were previously doing a ton of
work even if the feature wasn't being used.
2018-05-04 12:59:30 -07:00
Vishnu Ks 733da0ac07 settings: Fix 500 when trying to change email to disposable email.
Fixes #9240
2018-05-04 10:52:38 -07:00
Tim Abbott 8b26f912af streams: Limit access to public streams for guest users.
With most of the tests tests written by Shubham Dhama.
2018-05-04 09:47:58 -07:00
Tim Abbott b1ad7593ba validators: Improve mypy type annotations.
The main thing here is writing check_string_fixed_length and
check_capped_string as returning a Validator, but we also fix issues
around passing default=None.
2018-05-03 21:27:43 -07:00
Tim Abbott 51517fa188 request: Add new str_validator validator type.
This is helpful for cases where an argument is supposed to be a normal
string, and we want to use a Zulip validator function to do basic
things like check its length.
2018-05-03 21:11:02 -07:00
Tim Abbott f2e84f25a0 management: Refactor checkconfig code to live in library.
This makes it possible to call this from other management commands.
2018-05-03 21:11:02 -07:00
Tim Abbott ad0cfb3512 validators: Add check_string_fixed_length.
This will be useful in some upcoming work on zilencer.
2018-05-03 14:30:03 -07:00
Tim Abbott 594451707d check_capped_string: Clean up corner case.
We were rejecting strings of length equal to the max.

While we're at it, fix the unnecessary period in the error message,
which doesn't align with similar validators.
2018-05-03 14:30:03 -07:00
Rishi Gupta 42a99e8c1d zilencer: Rename remote_server_unregister_push. 2018-05-03 12:50:29 -07:00
Rishi Gupta 76650f5930 zilencer: Rename remote_server_register_push to register_remote_push_device. 2018-05-03 12:50:29 -07:00
Eeshan Garg d28d08e7da streams: Add get_default_value_for_history_public_to_subscribers().
This commit adds a function that makes it easier to get a default
value for Stream.history_public_to_subscribers when one isn't
explicitly provided.
2018-05-03 10:24:27 -07:00
Aditya Bansal 75d76e4eb3 lib/streams.py: Extract get_stream_by_id as a separate function.
We extract get_stream_by_id function out of the body of
access_stream_by_id function to help us access streams for archives.
2018-05-02 15:23:33 -07:00
Aditya Bansal 9629be689b populate_db: Add a web public stream to dev database.
We flip the Stream "Rome" to be a web public stream. Also we add
attribute is_web_public in various stream dicts and in the
bulk_create_streams function of bulk_create.py responsible for
default stream creation in dev environment.
2018-05-02 15:23:33 -07:00
Aditya Bansal 1f358954be web_public_streams: Add is_web_public to Stream table.
Also add function do_change_stream_web_public in lib/actions.py
to help in changing a streams web public status.
2018-05-02 15:23:33 -07:00
novokrest 0fb13eed2f outgoing_webhook: Extract success response handling to separate method.
Extract success response handling in do_rest_call() method to
separate method process_success_response()
2018-05-02 11:57:26 -07:00
novokrest 036bc120c3 outgoing_webhook: Extend process_success() return value to tuple.
Change return value type of OutgoingWebhookServiceInterface.process_success
to 2-elements tuple as (success_message, failure_message)
2018-05-02 11:57:08 -07:00
Tim Abbott 4df886f36f populate_db: Fix initialization of history_public_to_subscribers.
This was being incorrectly not initialized properly in the test suite,
because we neglected to update the bulk_create code path for creating
streams.
2018-05-02 09:02:57 -07:00
Tim Abbott 866cb38270 test_classes: Compute history_public_to_subscribers correctly.
We apparently missed updating this when we split out this database field.
2018-05-02 09:02:57 -07:00
Rohitt Vashishtha 2f6da2661f push_notifications: Format blockquotes properly in text_output.
New output is of the format:

Hamlet said:
> Polonius said:
> > This is the.
> > Second layer of nesting.
> First layer of nesting.

Fixes #9251.
2018-05-02 08:57:17 -07:00
Tim Abbott c1432d9dfc slack import: Reformat UserProfile to one-line-per-field.
This should be more readable and convenient for future editing.
2018-05-01 09:16:12 -07:00
Tim Abbott a2f49b425b slack import: Fix some erroneous UserProfile field values.
* tutorial_status was the invalid value 'T'; should be the default of 'W'.
* last_reminder can be just the default None
* enable_desktop_notifications was just the model default of True.
2018-05-01 09:14:47 -07:00
Rhea Parekh f00b80058d slack import: Remove unwanted comments. 2018-05-01 09:09:36 -07:00
Rhea Parekh e579bef8fd slack import: Improve how we construct user_profile objects.
Fixes #9260
2018-05-01 09:09:36 -07:00
Angelika Serwa f4f64243dd custom_profile_fields: Support changing the sort order of the fields.
Tweaked by tabbott for variable naming and the URL.

Closes #8879.
2018-04-30 18:17:41 -07:00
Yashashvi Dave 4033f210af custom fields: Add support for custom date field type. 2018-04-30 23:04:25 +05:30
Yashashvi Dave 66759358e2 create stream: Add maxlength restriction on name and description. 2018-04-30 10:11:25 -07:00
Tim Abbott 976e61d687 validators: Improve error messages for check_capped_string. 2018-04-30 10:07:06 -07:00
Yashashvi Dave 7bbe44d7a0 org settings: Remove "Delete streams" administrative tab from settings.
Fixes #9227
2018-04-30 17:47:34 +05:30
Eeshan Garg 057ff9c91e models: Add Stream.history_public_to_subscribers.
This commit adds a new field history_public_to_subscribers to the
Stream model, which serves a similar function to the old
settings.PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS; we still use that
setting as the default value for new streams to avoid breaking
backwards-compatibility for those users before we are ready with an
actual UI for users to choose directly.

This also comes with a migration to set the value of the new field for
existing streams with an algorithm matching that used at runtime.

With significant changes by Tim Abbott.

This is an initial part of our efforts on #9232.
2018-04-28 22:54:04 -07:00
Tim Abbott 7d6bb3dcb4 settings: Remove obsolete default_desktop_notifications setting.
This actually hasn't been hooked up to do anything in years.

While we're at it, we remove the entire "Zulip Labs" settings page.
2018-04-28 13:46:07 -07:00
Tim Abbott 127ac0df54 auth: Remove unnecessary case from validate_email_for_realm.
The removed code path was only needed due to buggy setup code in the
test_cross_realm_scenarios test.  We address that with a less buggy
workaround, and which lets us remove unnecessary complexity from this
important validation function.

Thanks for Umair Waheed for some preliminary work on this.

Fixes #7561.
2018-04-28 11:03:03 -07:00
YJDave 6bef44a9fa org setting: Add time limits for message deleting.
Add realm setting to set time limit for message deleitng.
Set default value of message_content_delete_limit_seconds
to 600 seconds(10 min).

Thanks to Shubham Dhama for rebasing and reworking this.  Some final
edits also done by Tim Abbott.

Fixes #7344.
2018-04-27 19:22:28 -07:00
Eeshan Garg 7d14ce2cb6 pypi packaging: Upgrade to release 0.4.6.
As a part of the upgrade, we had to update our API tests in
zerver/lib/api_test_helpers.
2018-04-27 14:50:25 -07:00
Tim Abbott abef9f203b api: Don't use ujson library unecessarily.
ujson is very fast, but doesn't provide good error messages for
parsing errors.
2018-04-27 14:49:46 -07:00
Eeshan Garg bd2270eecb test-api: Add test helpers that pretty-print JSON output. 2018-04-27 14:49:34 -07:00
Vishnu Ks f140b0e870 emails: Add translation tags to invitation. 2018-04-27 11:59:36 -07:00
Tim Abbott 2cdd367d49 email_mirror: Fix handling of empty topic.
Also fixs some corner cases around pure-whitespace topics, and
migrates from the years-obsolete "no subject".

Fixes #9207.
2018-04-26 10:21:29 -07:00
Umair Khan cf2f6b38dd profile: Add choice field.
Fixes part of #8878
2018-04-26 00:35:51 -07:00
Umair Khan 4ea3e8003a profile: Create mypy types for profile data.
This makes the code more readable.
2018-04-25 23:28:27 -07:00
Tim Abbott b40780d003 mypy: Fix errors in new bugdown module.
I introduced these when making final changes before merging.
2018-04-25 15:56:46 -07:00
Lyla Fischer 0f3cb14aae user docs: Use {settings_tab} macro for more organization settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer dbc573584b user docs: Use the {settings_tab} macro for four organization settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer b2be1a67f8 help docs: Add {settings_tab} for the first three org settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer 68f68bf56d help docs: Use {settings_tab} macro for some user settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer a4ea71ec0f help docs: Use {settings_tab|notifications} macro. 2018-04-25 14:39:30 -07:00
Lyla Fischer d40f246599 help docs: Use {settings_tab|display-settings}. 2018-04-25 13:51:29 -07:00
Lyla Fischer b24659b005 bugdown: Add {settings|my-setting} macro.
Tweaked by tabbott to add a test and fix a super subtle issue with the
relative_settings_link variable having been set once the first time a
/help article was rendered.
2018-04-25 13:41:24 -07:00
Tim Abbott ff9371d63c slack import: Fix issues with Slack empty files.
Fixes #9217.
2018-04-25 10:20:55 -07:00
neiljp (Neil Pilgrim) 9692a8572d mypy: Add assertion in timeout.py. 2018-04-25 08:58:55 -07:00
Rishi Gupta 718a87bd47 emails: Update followup_day1. 2018-04-25 08:52:29 -07:00
Tim Abbott 62fb139af7 Revert "test_fixtures: Add settings files to things that require reprovision."
This reverts commit 2bc51931a8.

See #9210 for the follow-up work needed before we can re-add this.
2018-04-24 11:01:20 -07:00
Preston Hansen efc7967355 slack import: Update `build_zerver_realm` to use `Realm` defaults.
Fixes #9131.
2018-04-23 17:15:51 -07:00
Vishnu Ks 134fdd8fd0 bugdown: Replace vimeo link with video title. 2018-04-23 17:04:24 -07:00
Vishnu Ks 5671cef6d0 bugdown: Cleanup Vimeo preview. 2018-04-23 17:04:24 -07:00
Tim Abbott c4b886d8ae import: Split out import.py into its own module.
This should make it a bit easier to find the code.
2018-04-23 15:21:12 -07:00
Vishnu Ks c9e932a7ce settings: Add support for Hangouts as the video chat provider.
The only thing that's annoying about this feature is that you need to
be a paying G Suite customer to use it.
2018-04-23 09:39:47 -07:00
Tim Abbott 01be6b01b1 validate_domain: Add verification of domain length. 2018-04-23 09:29:03 -07:00
Shubham Dhama 26d2ffa821 populate_db: Add new user of "guest" type.
The purpose of this user is to act as a guest.
(This is a preliminary step in adding the guest type of user
and is a part of #8385.)
2018-04-20 16:20:00 -07:00
Tim Abbott 49f58583a4 models: Introduce can_access_public_streams field.
This is a simple computed field.  It's intended to more clearly
capture the meaning of this restriction for the users in zephyr mirror
realms, and eventually support guest user accounts in normal Zulip
realms.
2018-04-20 12:06:57 -07:00
Tim Abbott dc6d7d0d12 actions: Use presence_disabled for presence zephyr_mirror code path.
This is part of the effort to remove the use of is_zephyr_mirror_realm
across the code path for situations that might be relevant for other
users.  It helps keep the code readable.
2018-04-20 12:06:57 -07:00
Roman Godov d99758129e subs: Notify organization admins when private streams are renamed.
This commit sends the event for renaming of a private stream to
organization admins of the realm, in addition to the obvious list of
subscribers of the private stream.

Normally, admins can manage a private stream (e.g. unsubscribing a
user).  But when the admin tried to unsubscribes a user from a
previously renamed stream, we previously were throwing a JS error, as
the webapp hadn't been notified about the new stream name.

Fixes #9034.
2018-04-19 22:00:18 -07:00
Preston Hansen e168f9938c tests: Refactor use of test and webhook data fixtures. 2018-04-19 21:50:29 -07:00
Preston Hansen 76d6c71595 tests: Move zerver/fixtures to zerver/tests/fixtures for clarity.
Fixes #9153.
2018-04-19 21:50:17 -07:00
Umair Khan 0420b89468 profile: Add hint attribute to custom profile fields.
This is the model for #8876.
2018-04-19 11:32:35 -07:00
Umair Khan 00ffa808da check_capped_string: Return a validator.
This allows us to create custom validators.
2018-04-19 11:21:50 -07:00
Eeshan Garg 19177a4aff webhooks: Move github_webhook/ to github/. 2018-04-19 11:00:55 -07:00
Eeshan Garg 48b8558c02 webhooks: Move github/ to github_legacy/ and remove docs. 2018-04-19 11:00:55 -07:00
Tim Abbott 7dfa0edfa6 slack import: Don't try to import pinned/unpinned items.
There isn't a corresponding Zulip concept, and they don't have a
"text" attribute, so there's no message content to import.
2018-04-18 11:36:31 -07:00
Tim Abbott acd3a364e1 slack import: Refactor handling of dropped messages.
This is a more coherent ordering, because some messages we skip lack a
"text" attribute.
2018-04-18 11:35:59 -07:00
Tim Abbott e759fd9be4 slack import: Improve error handling for invalid messages. 2018-04-18 11:29:40 -07:00
Tim Abbott 1410a1e460 slack import: Remove unnecessary zerver_realm_skeleton.json.
This was stored as a fixture file under zerver/fixtures, which caused
problems, since we don't show that directory under production (as its
part of the test system).

The simplest emergency fix here would be to just move the file, but
when looking at it, it's clear that we don't need or want a fixture
file here; we want a Python object, so we just do that.

A valuable follow-up improvement to this block would be to create an
actual new Realm object (not saved to the database), and dump it the
same code we use in the export tool; that should handle the vast
majority of these correctly.

Fixes #9123.
2018-04-18 10:33:53 -07:00
Tim Abbott 77ca9e7eca hotspots: Tag strings for translation. 2018-04-18 09:33:09 -07:00
Rhea Parekh a2070fb7e5 slack importer: Add comment on size information of avatars.
The size information of an avatar is not required during the import.
Check function 'import_uploads_local' and 'import_uploads_s3'
in 'export.py' for this.
2018-04-16 14:44:57 -07:00
Tim Abbott 0c30a26d81 bulk_create: Remove some long-dead code.
We used to use these in populate_db, but haven't done so in a long
time, and it doesn't seem likely that will change anytime in the
future.
2018-04-16 11:41:42 -07:00
Ben Reeves fdfbd45208 soft_deactivation: Change `<` to `<=` in add_missing_messages.
We should still short-circuit the iteration in
`add_missing_messages` if the unsubscription was the last
thing to happen to the user before unsubscription and
soft deactivation.
2018-04-16 11:28:08 -07:00
Rhea Parekh f6b6aa1e75 slack import: Implement threading as a management command. 2018-04-15 19:53:02 +05:30
Rhea Parekh 7c0c3930a8 slack importer: Thread avatar downloads. 2018-04-15 19:53:01 +05:30
Rhea Parekh ebc2ee28e9 slack importer: Thread emoji downloads. 2018-04-15 19:52:59 +05:30
Rhea Parekh 8a291d0232 slack importer: Thread attachment downloads.
Use Zulip's run_parallel method to run thread downloads.
2018-04-15 19:51:58 +05:30
Eeshan Garg 3e0eb9530c webhooks: Remove the Facebook integration.
Rishi and I decided that it makes sense to get rid of the Facebook
integration for a few reasons, some of which are:

* The setup process is too complicated on Facebook's end. The users
  will surely have to browse Facebook's huge API reference before even
  having a vague idea of what they want.
* Slack chooses not to have a Facebook integration, but relies on
  Zapier for it. Zaps that integrate with Facebook are much more
  streamlined and the setup process isn't as much of a pain. Zapier's
  Facebook Zaps are much more fine-tuned and there are different Zaps
  for different parts of the FB API, a luxury that would likely span
  2K+ lines of code on our end if we were to implement it from
  scratch. So, I think we should relegate integration with Facebook to
  Zapier as well!
* After thoroughly testing the setup process, we concluded that the
  person who submitted the FB integration didn't really test it
  thoroughly because there were some gaping holes in the docs (missing
  steps, user permissions, etc.).
2018-04-14 09:38:22 -07:00
Tim Abbott 5ddf2614f0 uploads: Add new way of querying for mobile uploads endpoint.
This extends the /user_uploads API endpoint to support passing the
authentication credentials via the URL, not the HTTP_AUTHORIZATION
headers.  This is an important workaround for the fact that React
Native's Webview system doesn't support setting HTTP_AUTHORIZATION;
the app will be responsible for rewriting URLs for uploaded files
directly to add this parameter.
2018-04-13 17:51:45 -07:00
Tim Abbott db830c4085 bugdown: Replace link to old pre-open-source Zulip trac. 2018-04-13 08:41:44 -07:00
Rohitt Vashishtha 9e7929417d markdown: Increase rendered_content length limit.
This commit increases the rendered_content limit from 2x to 10x of the
original message length.

Earlier, we had placed a limit of MAX_MESSAGE_LENGTH * 2 for the
rendered content (explained in commit
77addc5456).  That limit was based on
the assumption that in most cases, the rendered content wouldn't cause
a large increase in message length. However, quite prominently in
syntax highlighted codeblocks, that wasn't true and this caused the
limit condition to be hit for long messages composed primarily of code
blocks.

Example: The following message would render close to 10x it's original size.

```py
if:
def:
print("x", var)
x = y
```

Because the syntax highlighted logic is extremely compressible, having
rendered_content reach up to 100KB doesn't create a network
performance problem.
2018-04-13 08:39:51 -07:00
Rhea Parekh f7398cbb09 slack import: Implement custom profile fields.
Add custom profile fields in the slack converted
data 'realm' file.
Added tests for the custom profile fields.

Fixes #8928
2018-04-10 13:28:53 -07:00
Rhea Parekh 852e8516b4 slack import: Add custom profile fields.
Build CustomProfileField and CustomProfileFieldValue
for every user and process the field type after getting an
entire list of the custom fields.
2018-04-10 13:28:53 -07:00
rht a183186672 slack importer: User session.get to recycle previous connections. 2018-04-09 22:02:01 -07:00
Shubham Dhama b650b6b38c markdown: Add @stream as an alias for @all.
Fixes: #8930.
2018-04-09 16:35:14 -07:00
rht 630adb406b Slack importer: Map Slack channel mentions to Zulip stream mentions. 2018-04-09 10:47:39 -07:00
Rhea Parekh 035c440ff3 import script: Support import custom profile fields.
Import of Custom profile fields is only supported for slack
import script for now.
2018-04-09 10:45:35 -07:00
Tim Abbott c41d7ee300 slack import: Write reasonable multi-line JSON.
This is a lot better for debugging.
2018-04-09 10:45:35 -07:00
Rhea Parekh ed7127c8b4 import script: Delete medium sized avatars if it exists.
Deletion of medium sized image is done if it exists before calling the
function 'ensure_medium_avatar_image', to avoid potentially confusing
problems with left-over medium-size avatar images from a previous run
being used when repeatedly importing the same realm in a development
environment..

Fixes #8949.
2018-04-08 07:04:24 -07:00
rht f6feac1316 Slack importer: Map Slack command for mentions to Zulip's all.
Fixes #9003.
2018-04-07 20:02:39 -07:00
Rhea Parekh e037c2f93e import script: Fix upload links.
Rendered content is None for Slack imports, hence it is replaced only
for Zulip->Zulip imports.

Fixes #8959.
2018-04-07 20:01:20 -07:00
Rhea Parekh b3f951d2cf import script: User profile ids should be allocated before allocating bot ids. 2018-04-07 13:28:33 +05:30
Rhea Parekh 2baa9bc16e Import: Add subdomain in the import script.
Also remove user input of subdomain in the slack data
conversion script.
2018-04-06 09:12:56 -07:00
Tim Abbott c06565d909 users: Improve testing for user_ids_to_users. 2018-04-04 16:31:30 -07:00
novokrest 4d2082ab14 actions.py: Obtain bot profiles by using users.user_ids_to_users().
Remove models.get_user_profiles_by_ids() and
obtain user's bots profiles in actions.get_service_dicts_for_bots() by
users.user_ids_to_users() instead of models.get_user_profiles_by_ids().
Fixes #8939
2018-04-04 16:24:55 -07:00
novokrest 807a6ccf2c users.py: Implement user_ids_to_users() by generic_bulk_cached_fetch().
Optimize users.user_ids_to_users() implementation
by using generic_bulk_cached_fetch() to obtain user profiles
2018-04-04 16:24:55 -07:00
Tim Abbott 53e47e6991 messages: Modify access_message for is_history_public_to_subscribers.
This completes the Message side of #2745.
2018-04-04 16:18:47 -07:00
Tim Abbott bec71d7a50 messages: Add a server-level setting to control private stream history.
We don't indend for this server-level setting to exist in the long
term; the purpose of this is just to make it easy to test this code
path for development purposes.

This implements much of the Message side part of #2745.
2018-04-04 16:18:46 -07:00
Tim Abbott 228f41e916 messages: Pass UserProfile to is_public_stream_by_name and rename.
The new name can_access_stream_history_by_name gets to the point of
what this function actually does.  And passing in a user object lets
us define what this does based on the user subscribed.
2018-04-04 15:13:11 -07:00
Rhea Parekh f4ad464d82 import script: Fix broken links to attachments.
The comments explain this pretty well, but basically because we
rewrite the realm ID during the import process, we need to edit all
the message bodies that link to an attachment to instead link to the
post-processed URL where that file will be hosted on the new server.

Fixes #8926.
2018-04-04 10:05:15 -07:00
Rhea Parekh 5a9cea4134 import script: re map foreign key of UserProfile.last_active_message_id. 2018-04-04 08:53:09 -07:00
Rhea Parekh ed36314042 import script: Fix 're_map_foreign_keys' logging error. 2018-04-04 08:53:09 -07:00
Rhea Parekh 877c7760b7 import script: re_map Attachment foreign keys. 2018-04-04 08:53:09 -07:00
novokrest a613f3c0a1 get_owned_bot_dicts: Optimize admin's bot dicts loading from database.
Implement few optimizations for reading admin's bot dicts from database
for a constants number of requests:
- add models.get_user_profiles_by_ids() for reading bots profiles
  by single query from database
- add models.get_services_for_bots() for reading services for bots
  by single query from database
- add bot_config.get_bot_configs() for reading config data for bots
  by single query from database

Fixes #8838
2018-04-03 09:10:32 -07:00
novokrest ad3ea775df actions.py: Rename get_service_dicts_for_bots() to get_service_dicts_for_bot() 2018-04-02 11:18:50 -07:00
Tim Abbott da27461491 bugdown: Rename rewrite_if_relative_link to more clear name.
The name was actually sorta the reverse of what that function did.
2018-04-02 10:37:18 -07:00
Tim Abbott 7d4234b167 bugdown: Only open #narrow hash links inside the app. 2018-04-02 10:34:48 -07:00
Tim Abbott 6744938b6d bugdown: Simplify implementation for target_blank.
The previous version not using startswith() was harder to read.
2018-04-02 10:30:50 -07:00
Tim Abbott 6055a7bceb bugdown: Clean up rewrite_if_relative_link.
This implementation is more obviously correct, and may fix a corner
case where a `/` is missing after the realm URI.
2018-04-02 10:29:32 -07:00
Umair Khan e48f8bdafd check_short_string: Use check_capped_string for impl. 2018-04-02 09:51:09 -07:00
Umair Khan 7fdf043564 profile: Set max length for long text to 500. 2018-04-02 09:51:09 -07:00
Umair Khan 1d4b74482c check_short_string: Change max_length to 50. 2018-04-02 09:51:09 -07:00
Rhea Parekh 1bba6cc4ce slack importer: Support custom emoji reactions. 2018-04-01 23:24:35 -07:00
Rhea Parekh 2c116e46ce slack importer: Get custom emojis. 2018-04-01 23:24:35 -07:00
Rhea Parekh 00c1f25b58 import script: Support custom emojis.
'processing_emojis' check is added in the 'import_uploads'
function, so that the emoji files present in the to be imported
data file can be uploaded.

The procedure of saving emoji files in slack importer is same as
saving attachments and avatars, and the import has the similar
procedure too.
2018-04-01 23:24:35 -07:00
Rhea Parekh c650b8fa3e slack importer: Add zerver_realmemoji. 2018-04-01 23:24:35 -07:00
Rhea Parekh b133d175a7 slack importer: Change 'get_user_data' function implementation.
Change 'get_user_data' function to a more general function
to get data from the slack api using legacy tokens.

Also, change the error handling as upon invalid token,
the response is 200, but the response has an error
field in it.

For eg. Go to the following link with invalid token:
https://slack.com/api/emoji.list?token=xoxp-249056023425
2018-04-01 23:24:35 -07:00
Rhea Parekh 220ad6a386 slack importer: Map standard reactions.
As mentioned in
https://get.slack.help/hc/en-us/articles/202931348-Use-emoji-and-emoticons,
slack supports the standard emoji codes
(https://www.webpagefx.com/tools/emoji-cheat-sheet/)
and majority of them are already supported in Zulip.
2018-04-01 23:24:35 -07:00
Rhea Parekh 6f867fee40 import script: Support import of reactions. 2018-04-01 23:24:33 -07:00
Rhea Parekh 8a028142d8 slack importer: Remove id allocation function and its implementation.
Remove allocation ID function from slack import script. All the IDs
count will start from 0. Hence the ID List returned
by the allocation function is of no use, and we remove its implementation.
(example: get_total_messages_and_attachments function is of no use anymore,
hence we remove it)
2018-04-01 23:10:55 -07:00
Rhea Parekh d147bd25d0 import script: Change file path of the upload in the import script.
In importing avatars, we use the implementation where the 'avatar_path'
is seperately calculated using realm and user ID and then the content
of the path provided in the avatar's 'records.json' are copied to this
'avatar_path'.

Similary, here for the uploads, 's3_file_name' is seperately calculated
using the realm ID and uploaded file name and then the content of the
path provided in upload's 'records.json' are copied to this 's3_file_name'.
2018-04-01 23:04:14 -07:00
Rhea Parekh ff34d07fa0 import script: Add function to update model ids after allocation.
Add function 'update_model_ids' to remove repetitive code.
2018-04-01 22:29:23 -07:00
Rhea Parekh a2ecdeb28d import script: re_map minor foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 078453554e import script: re_map Message foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 93aabcb81c import script: re_map Subscription foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 9ef7870c5a import script: re_map Recipient foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 4537223ba7 import script: re_map UserProfile foreign keys. 2018-04-01 22:29:23 -07:00
Rhea Parekh 1314e7d247 import script: re_map Stream foreign keys.
'recipient_field' is added as a bool variable in the function
'update_id_map' to update the recipient foreign keys.

Recipient Foreign Key is equal to the UserProfile ID, if the
type is 1, and the same is equal to Stream ID, if the type is 2.
Hence a check is added in the 'update_id_map' field for this.
2018-04-01 22:29:23 -07:00
Rhea Parekh 8624ba4132 import script: re_map Realm foreign keys.
All the objects with realm ID as the foreign keys need to
be remapped with updated with the allocated ID.
Also the ID of the realm object itself is updated with the allocated
ID.
2018-04-01 22:29:23 -07:00
Rhea Parekh 2b0ee472af import script: Refactor re_map_foreign_keys.
The 'id_field' bool variable is added to the function just to check
if the field is the ID of that object, and not the foreign key relation.
For foreign key field names, a "_id" has to be added after the field name,
however we don't need that for the ID field of the object.
2018-04-01 22:29:23 -07:00
Rhea Parekh cd0871bae4 Import script: Add id allocation functions. 2018-04-01 22:29:23 -07:00
Shubham Padia 10115491a3 real-time sync: Server sends recipent data with delete_message event.
Fixes #8853.
In certain cases, the browser is not able to look up the message.
Include the recipient data for the message in the delete_message event,
so look up of those attributes by the browser isn't required.
2018-04-01 22:19:08 -07:00
Shubham Dhama b7aae89029 org settings: Add realm level default time format setting. 2018-03-31 16:23:26 -07:00
Puneeth Chaganti 4ce8f2aaa2 upload: Rename upload_message_image to upload_message_file.
Tweaked by tabbott to also fix a Slack import comment.
2018-03-30 13:38:31 -07:00
Rohitt Vashishtha a3ed83f4e2 markdown: Update bugdown emoticon translation logic to match frontend.
This PR solves some of the parity issues in the emoticon translation
logic. I was unable to find a way of matching only one of the
lookaround groups, so we still have some inconsistency (see
testcase). The approach of having another check while converting just
for this seemed like an inefficient way, so I've left that last change
as it is.
2018-03-29 15:56:24 -07:00
Umair Khan 7885dd4408 profile: Send field types to client side. 2018-03-29 13:09:12 -07:00
Umair Khan f6fb88549f profile: Send operation in the event.
This allows us to show a useful message in the handler when the
event is received.
2018-03-29 13:09:11 -07:00
Tim Abbott 656f882a44 bots: Eliminate NEW_USER_BOT.
This bot was basically a duplicate of NOTIFICATION_BOT for some
specific corner cases, and didn't add much value.  It's better to just
eliminate it, which also removes some ugly corner cases around what
happens if the user account doesn't exist.
2018-03-29 12:01:21 -07:00
Tim Abbott 2bc51931a8 test_fixtures: Add settings files to things that require reprovision.
Since the test database is in part controlled by the Zulip settings
files for testing, these settings files should be included in the list
of files that require populate_db to be rerun.

This issue was found due to changes to internal bots.
2018-03-29 11:53:12 -07:00
neiljp (Neil Pilgrim) 704c33331c mypy: Add explicit Optional for default=None parameters in export.py. 2018-03-28 12:31:51 -07:00
neiljp (Neil Pilgrim) 090b47ed19 mypy: Add explicit Optional for default=None parameters in various files. 2018-03-28 12:31:51 -07:00
Balaji2198 e1eabe286a integrations: Add Gocd webhook integration.
Fixes #38.
2018-03-28 11:08:27 -07:00
Tim Abbott c94deff920 mypy: Remove some now-unnecessary type: ignores. 2018-03-28 10:39:05 -07:00
Rohitt Vashishtha 3c96b04763 emoji: Add translate_emoticons flag in bugdown testcases.
Also switches the default behaviour of the code to not translate the
emoticons. Earlier, the code was testing-aware, and used to translate
when there was no user profile data available(assuming that as a testing
environment).
2018-03-27 17:16:55 -07:00
Eeshan Garg 538746fc65 webhooks: Stop raising an exception if stream does not exist.
webhook-errors.log file is cluttered with Stream.DoesNotExist
errors, which hides the errors that we actually need to see. So,
since check_message already sends the bot_owner a PM if the webhook
bot tries to send a message to a non-existent stream, we can ignore
such exceptions.
2018-03-24 13:50:17 -07:00
Ricky 4b8bd0bc3b webhooks: Add Dialogflow integration.
With minor fixes by eeshangarg!

Fixes #6990
2018-03-24 15:03:35 -02:30
Steve Howell 4f05cefb32 Remove obsolete code for stream subscribe buttons.
In 18e43895ff we replaced
stream subscribe buttons with stream links.  The new feature
has been well tested and well received for over a year now,
so it's safe to remove the older feature at this point.

Older sites will have super old messages that still have the
rendered markup; this commit does not attempt to address those
situations.  Most likely, clicking on an old button in the old
message will either do nothing or look like a message reply.
2018-03-24 12:31:36 -04:00
Aayush Agrawal 78a4924ecc markdown: Apply both bold and italic for words like ***text***.
Adds rule 'strong_em' to markdown parser to recognize ***{TEXT}***
Fixes #8155.
2018-03-23 15:01:24 -07:00
Tarun Kumar 86eaa553a6 user-groups: Add backend enforcing for new modification settings.
Add function in user-groups.py for getting member ids
for a group.
Update view to enforce checks for modifying user-groups.
Only admins and user group members can modify user-groups.
2018-03-23 14:44:47 -07:00
Greg Price b42a7b1701 digest: Add a server setting, and disable by default.
This feature isn't really ready yet -- the relevance isn't good, so
the emails aren't a great experience.  More work needed; pending that,
just don't send them.

There's already a per-realm setting, which doesn't have a control in
the org settings UI but does suppress it in the per-user settings UI.
Piggyback on that to suppress that UI control when the feature is
disabled at the server level too.

Also cut a comment that hasn't really made sense since the logic was
changed months ago -- the comment originally explained why we sent
digests on Tuesday, Wednesday, and Thursday, and doesn't correspond to
why we dialled back to weekly on Tuesdays.
2018-03-23 14:12:01 -07:00
neiljp (Neil Pilgrim) 97836c4bac mypy: Remove Optional & Mutable from do_schedule_message.
Removes various mypy errors.
2018-03-23 11:32:00 -07:00
neiljp (Neil Pilgrim) 2079c5e4d7 mypy: Mark Subscription as Optional in lib/streams.py:access_stream*. 2018-03-23 11:32:00 -07:00
neiljp (Neil Pilgrim) d741e0ea01 mypy: Introduce query variable in lib/users.py for clarity.
This also avoids mypy showsing an error.
2018-03-23 11:32:00 -07:00
Tim Abbott 62c0d27d1e push_notifications: Handle errors with token reuse.
If an Android token has been used to connect a given device with
multiple Zulip servers, and then is expired, we would 500 in trying to
remove the Zulip-side registration for it, because the code assumed
there was only one such registration.  If a token is no longer valid,
it's invalid for all servers, so the correct fix is to just remove them all.
2018-03-23 11:02:55 -07:00
Sarah f5c2fb8438 org settings: Create backend api for allow_community_topic_editing.
Adds the code for updating the allow_community_topic_editing
setting.
2018-03-22 16:02:27 -07:00
Jack Weatherilt 8535625341 parse_user_agent: Assert user agent is not None.
This commit asserts that parse_user_agent never returns None. The
RegEx will match any string, so that `match` is never None. This
brings test coverage of lib/user_agent.py to 100%. Changes were also
made in test/test_decorators.py and views/compatibility.py to reflect
that parse_user_agent cannot return None.

Improves: #7089.
Fixes: #8779.
2018-03-22 14:29:29 -07:00
Greg Price fe0f1edddb settings: Fix double negative in LOGGING_NOT_DISABLED.
Saying "not disabled" just makes it more work to read than it needs to
be -- instead say ENABLED.
2018-03-21 18:03:05 -07:00
Greg Price 73559e5320 queue: Suppress error mail from brief rabbitmq downtimes.
Details in comment.  Together with a few previous commits, this should
completely eliminate sending error mail to admins when the RabbitMQ
server is simply restarted and comes back up normally.
2018-03-21 18:03:05 -07:00
Jack Weatherilt d857f26cd3 refactoring: Remove unused assignment on ensure_stream.
There were two instances of `ensure_stream` being called and assigned to
a variable with the variable not being used elsewhere. pyflakes picked
up on this (where it didn't in the previous version likely due to tuple
unpacking), so the the variable assignment has been replaced with a call
to `ensure_stream`.
2018-03-21 16:47:52 -07:00
Jack Weatherilt 3396cfc2ef refactoring: Replaced occurences of create_stream_if_needed.
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.

The changes weren't significant enough to add any tests or do any
additional manual testing.

The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.

Fixes: #2088.
2018-03-21 16:47:36 -07:00
YJDave 11c995b70f custom profile data: Send event to active user on update.
On update of custom profile fields, send an event to all
active users of realm.
2018-03-21 16:08:12 -07:00
Harshit Bansal 64372690f9 emoji: Fix the filtering condition in `check_emoji_admin()`.
This commit fixes an unreported bug which if hadn't been fixed would
have caused errors while deactivating realm emojis in some corner
cases.
2018-03-20 22:24:44 -07:00
Harshit Bansal a49655e0d4 emoji: Migrate realm emoji to be addressed by `id` rather than `name`.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.

Fixes: #6977.
2018-03-20 22:24:44 -07:00
Greg Price 3b3154527f queue: Don't blow up when a connection closes quickly. 2018-03-20 16:49:05 -07:00
Greg Price 9dcc436766 queue: Fix __init__ logic so heartbeat choice works fully.
Because the base class's __init__ calls `_connect`, when we set the
value after that call has already returned, our new value only takes
effect if the first connection fails and we have to reconnect.
Make it take effect from the beginning.
2018-03-20 16:49:05 -07:00
Greg Price 5edc26a0df queue: Cut disused, broken parameter to `_connect`.
This parameter isn't used anywhere.  A good thing, because if it were,
the code would immediately raise an exception -- `self._on_open_cbs`
hasn't been initialized yet when we first call `_connect`, from the
base class's `__init__`.

So, just cut it.  If we later need something like this, it's easy to
add a working version then.
2018-03-20 16:49:05 -07:00
Tim Abbott 51018e12f5 message_edit: Remove obsolete block for diff highlighting.
We disabled the original "colorized HTML edit-history" feature way
back in 2013 in c51056ff8e.

That original feature involved showing what had been edited inline in
message bodies, so one could easily see what had been changed.

That old feature has since been replaced with the "view edit history"
menu option, and we're unlikely to ever want the old feature back.
So, we can just remove its code.  There's a few supporting variables
that were created to help implement this; we can clean those up and
simplify the `update_message` code now that this feature is fully
removed.
2018-03-20 15:22:53 -07:00
Rhea Parekh d0355f52cb slack importer: Show when files are being downloaded. 2018-03-20 14:42:26 -07:00
Rhea Parekh e7291148e8 slack importer: Add '/me' in content for specific subtypes. 2018-03-20 14:42:26 -07:00
Nikhil Kumar Mishra f29a1918f3 hotspots: Add ALWAYS_SEND_ALL_HOTSPOTS in dev_settings.
Replace the local variable SEND_ALL from get_next_hotspots.
Add unit test for the same.
2018-03-19 10:39:43 -07:00
Nikhil Kumar Mishra 3b4ff4f75c markdown: Clean render_markdown.
Remove Message is None and not None case as the
render_message_backend passes a fake Message object.
2018-03-19 10:36:26 -07:00
Harshit Bansal 646700b144 emoji: Start using `get_active_emoji()` in `emoji_name_to_emoji_code()`. 2018-03-18 19:34:02 -07:00
Greg Price 22071a44a7 antispam: Add a sitewide ratelimit on invites by new realms.
This applies only on a server open for anyone to create a realm.
Moreover, if the server admins have granted any given realm a
max_invites greater than the default, that realm is exempt too.
2018-03-16 18:00:11 -07:00
Tim Abbott c147d2e140 notifications: Fix missed-message emails for presence-idle users.
This fixes an unpleasant regression in
f5edeb01ae, where we stopped correctly
filtering users who have an open browser session that's idle.  These
users are tagged as "UserPresence.IDLE" with an current timestamp in
the database, and should be treated as idle for presence purposes.

As a result, if you had an open Zulip browser session, you incorrectly
wouldn't get missed-message emails for PMs and mentions before this fix.
2018-03-16 16:30:23 -07:00
Tim Abbott 34e165c100 webhooks: Fix passing client string to authenticated webhook API views.
This fixes a regression in 93678e89cd
and a4979410f9, where the webhooks using
authenticated_rest_api_view were migrated to a new model that didn't
include setting a custom Client string for the webhook.

When restoring these webhooks' client strings, we also fix places
where the client string was not capitalized the same was as the
product's name.
2018-03-16 15:43:19 -07:00
Shubham Dhama 610f2cbacf notification email: Send followup_day2 email two days later.
This changes the followup_day2 emails delay from one day later to two days
later if it is getting delivered on any working days(i.e. Mon - Fri).
For Thursday it is compromised to next day as it would be too late to
postponed to Monday and for Friday it should be Monday.
At last actually, emails should send one hour before the above calculated so
that user can catch them when they are dealing with these kinds of stuff.
Fixes: #7078.
2018-03-16 13:35:57 -07:00
neiljp (Neil Pilgrim) 5726d26d50 mypy: Use centralized Validator in request.pyi & validator.py.
These changes are in one commit, since the previous typing of check_url
does not match the centralized strict definition (object/Any vs Text),
actually already used elsewhere in validator.py, and also had a different
API.

check_url is updated here to match the API of the other check_* functions,
ie. val is an object (not Text) & returns Optional[str]. It also now checks
the value is text explicitly at run-time, which was only type-checked
previously. Tests are updated accordingly.
2018-03-16 13:30:32 -07:00
YJDave 0281079a39 stream settings: Send all private streams subs to realm admins on load.
Tweaked by tabbott to simplify the conditionals in actions.py.

Fixes #8695.
2018-03-16 12:26:56 -07:00
YJDave 72a440a86d stream settings: Fix error in real time sync in subs add/remove event.
Currently, when other private stream subscriber add realm admin to
stream, new copy private stream is created in realm admin's streams.
Which resulted in error, cause there are two similar stream element
in stream settings.

If new subscriber is added to private stream, we first send them
stream `create` event, cause private stream are not visible until
user don't get subscribed at least once. But realm admins can now
always access private stream, so when realm admin is subscribed to
stream, realm admin get stream `create` event even if stream already
exist in on realm admin client side.

Fix this by extracting realm admins from stream `create` event on
`add` subscription operation and sending private stream `create`
event to all realm admins on stream creation operation.

Fixes #8695
2018-03-16 12:22:06 -07:00
Eeshan Garg 93678e89cd webhooks: Migrate 14 webhooks to use check_send_webhook_message.
These are the straightforward ones.

Note that there is a line in zerver.lib.test_classes.build_webhook_url
that lost test coverage. That's because most of our tests test using
stream messages so the webhook URLs being tested always have a query
parameter. So the line that accounts for there being no query
parameters never gets called, which is fine, but we should still
keep it.
2018-03-16 11:34:20 -07:00
Eeshan Garg af56df7723 webhooks: Enable custom topics and default PM notifications.
This commit adds a generic function called check_send_webhook_message
that does the following:
* If a stream is specified in the webhook URL, it sends a stream
  message, otherwise sends a PM to the owner of the bot.
* In the case of a stream message, if a custom topic is specified
  in the webhook URL, it uses that topic as the subject of the
  stream message.

Also, note that we need not test this anywhere except for the
helloworld webhook. Since helloworld is our default example for
webhooks, it is here to stay and it made sense that tests for a
generic function such as check_send_webhook_message be tested
with an actual generic webhook!

Fixes #8607.
2018-03-16 11:34:20 -07:00
Tim Abbott 707af5ab56 cache: Remove a now-unnecessary TODO.
We solved the problem the TODO raised by using a different type
annotation syntax, and I'm not sure whether that refactor would
actually improve the code.
2018-03-16 11:32:14 -07:00
neiljp (Neil Pilgrim) 966ca7015f mypy: Finalize migration of cache.py to python3 function annotation.
- Use forward declarations of some types from models.py to avoid cycles.
- Remove cache.py from linter rule exclude list to ensure it stays that way.
2018-03-16 11:29:12 -07:00
neiljp (Neil Pilgrim) 005cb6bd03 mypy: Improve [get_]cache_with_key typing & use py3 annotation. 2018-03-16 11:29:12 -07:00
neiljp (Neil Pilgrim) b79c3840da mypy: Introduce strict definition of Validator into types.py. 2018-03-16 11:16:38 -07:00
Tim Abbott 99bae7fa5d slack import: Improve algorithm and formatting for attachments.
The previous system would crash with some files (because for some
reason the comment count was 1 but there was no "initial comment") and
also the file comment and file name were sorta redundant.
2018-03-16 11:12:58 -07:00
Rhea Parekh 06071166c7 slack importer: Handle case where attachment download link is not from slack. 2018-03-16 11:12:58 -07:00
Rhea Parekh b7d6608ba6 slack importer: Clean 'create_converted_data_files' function.
The 'make_new_dir' bool value was used to create a new directory
every time True is passed. Now that avatars and uploads directory
are being created seperately, we don't need this anymore.
2018-03-16 11:12:58 -07:00
Rhea Parekh 4b66a2d0dc slack importer: Add function to fetch and save uploads. 2018-03-16 11:12:58 -07:00
Rhea Parekh e62945eb86 slack importer: Implement changes in script due to zerver_attachment. 2018-03-16 11:12:58 -07:00
Rhea Parekh 8e2d930644 slack importer: Implement changes in script due to user upload object. 2018-03-16 11:12:58 -07:00
Rhea Parekh b0851eb20b slack importer: Add helper functions to build attachment object. 2018-03-16 11:12:58 -07:00
Rhea Parekh 68af6e4b7a slack importer: Add helper functions to build user uploads object. 2018-03-16 11:12:58 -07:00
Rhea Parekh 6d85f8c9ef slack importer: Randomize stream colors. 2018-03-15 23:50:32 -07:00
Rhea Parekh 90a3ffc5c0 slack importer: Include only slack's purpose field in description. 2018-03-15 23:50:32 -07:00
Rhea Parekh 8a4f307c43 slack importer: Change topic for imported content. 2018-03-15 23:50:32 -07:00
Tim Abbott ec21997bfe slack import: Fix importing of bot users from Slack. 2018-03-15 18:35:40 -07:00
Rhea Parekh a5b0957e5d slack importer: Set domain name in 'do_convert_data'.
The domain name is being set in the helper function
'slack_workspace_to_realm', but it should be set in the main function
'do_convert_data', as we need it in other child functions of
'do_convert_data'.
2018-03-15 18:34:51 -07:00
Tim Abbott 14e8ac5675 migrate: Remove dead migration code.
This code was originally written when we were using the old South
system, and hasn't been used in a few years.  It probably doesn't
work, and thus only serves to clutter the codebase.
2018-03-15 17:56:32 -07:00
Vishnu Ks 951b88dd30 models: Make email_allowed_for_realm raise exception. 2018-03-15 14:35:24 -07:00
neiljp (Neil Pilgrim) 17937175ac mypy: Centralize ViewFuncT definition into new file zerver/lib/types.py.
Originally was going to centralize this in zerver/lib/request.pyi, but this
file is not visible at run-time, being only a stub. The matching request.py
file seemed inappropriate, as it doesn't actually use ViewFuncT.
2018-03-15 14:16:40 -07:00
neiljp (Neil Pilgrim) a7bfb09067 Mypy: Use models.py QuerySet annotation approach in stream_subscription.py.
Namely, annotate as best as possible, and add notes to indicate preference,
if QuerySet develops generic typing.

Note that the return values of functions with annotations changed in this
commit are used elsewhere as QuerySets, so the Sequence[T] approach used
for some functions in models.py is not applicable.
2018-03-15 12:54:43 -07:00
Tim Abbott 1a3ae8a376 stream settings: Fix inadvertently merged broken whitespace.
I'm not sure why the pre-commit linter didn't flag this for me; maybe
it did and I just missed it.
2018-03-13 15:12:45 -07:00
YJDave 2031118545 stream settings: Allow realm admins to access all private stream subs.
This will allow realm admins to access subscribers of unsubscribed
private stream.  This is a preparatory commit for letting realm admins
remove those users.
2018-03-13 14:59:09 -07:00
YJDave 37f9d5c193 stream settings: Allow realm admins to update any stream name & description.
This will allow realm admins to update the names and descriptions of
private streams even if they are not subscribed, which fixes the buggy
behavior that previously nobody could(!).
2018-03-13 14:59:09 -07:00
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
neiljp (Neil Pilgrim) 21033669a8 mypy: Migrate lib/bugdown/__init__.py to python3.5 annotations. 2018-03-12 11:23:30 -07:00
neiljp (Neil Pilgrim) bb1585b2c2 mypy: Migrate lib/debug.py to python3.5 annotations. 2018-03-12 11:23:30 -07:00
Vishnu Ks 41f8618c04 email: Use PyPi module for disposable email providers list. 2018-03-11 21:48:56 -07:00
Tim Abbott 25d9731a3c custom profiles: Fix totally broken events.py logic.
Apparently, my manual testing here was in error; the new version
actaully works for delivering custom profile data to the frontend.
2018-03-11 21:25:23 -07:00
Tim Abbott ef92fcbe2b topic history: Fix fetching topic history of public streams.
Apparently, we did essentially all the work to support showing full
topic history to newly subscribed users from a data flow perspective,
but didn't actually enable this feature by having the topic history
endpoint grant access to historical topics.  This fixes that gap.

I'm not altogether happy with how the code and tests read for this
feature; the code itself has more duplication than I'd like, and the
tests do too, but it works.
2018-03-11 20:59:20 -07:00
Ricky 9e52a9f879 webhooks/flock: Add flock integration. 2018-03-11 20:31:13 -07:00
Tim Abbott 02b8453367 custom profiles: Send custom profile data to frontend.
This will fetch the data of custom fields for all users.
2018-03-11 18:08:17 -07:00
Harshit Bansal e41067a833 emoji: Refactor bugdown to use only active realm emojis.
This commit refactors the bugdown to perform a lookup only on active
realm emojis. This was needed because once we migrate realm emojis
to be addressed by `id` rather than name, it will be costly to
perform a lookup on all the realm emojis.
2018-03-11 16:24:44 -07:00
neiljp (Neil Pilgrim) 8d188f50eb bugdown: Clarify walk_tree_with_family via ElementPair NamedTuple. 2018-03-11 08:30:56 +00:00
neiljp (Neil Pilgrim) d4cfab4823 mypy: Use cast in walk_tree_with_family to access inside opaque Element. 2018-03-10 10:04:14 -08:00
YJDave ee87c146ff settings: If message content disabled, flush all missed emails informations.
If user has disabled message content in missed email notifications,
we shouldn't send any informations about missed messages i.e. sender,
stream, message text, etc. to email servers.
We are already hiding this informations in email templates, but we
shoudln't expose any information about message content if user
has disabled this setting.
2018-03-09 21:16:02 -08:00
YJDave c94b21e9ac settings: Add setting to disable message content in missed message emails.
Fixes #6938.
2018-03-09 21:16:02 -08:00
YJDave 382ac24151 create stream: Fix real time sync bug in private stream creation.
If new private stream is created by realm admin without realm admin
subscribed to it, then it doesn't automatically add created stream to
realm admin's stream list. We have to reload the browser to get newly
created stream in stream list. Cause private stream creation event is
only sent to the subscribed users to private stream, so even if realm
admin is acting user, they don't get creation event.

We should send private stream creation event to realm admin users along
with subscribed user to stream, as realm admins can access unsubscribed
private streams.

Tweaked by tabbott to fix various typos and clean up the code.
2018-03-09 18:35:12 -08:00
Harshit Bansal 5aa8629a10 reactions: Migrate `emoji_code` to store `id` for realm emoji.
Till now, we had been storing realm emoji's name in emoji code field
in reactions' model. This commit migrates it to store realm emoji's id.
It is a part of effort to migrate realm emojis to be referenced by their
id and not by name.
2018-03-09 13:46:27 -08:00
Shubham Dhama 777b6de689 org settings: Add setting to prevent users from adding bots.
Fixes: #7908.
2018-03-09 13:21:55 -08:00
Robert Hönig 649e76e932 Display error when creating embedded bot with incorrect config data.
"incorrect" here means rejected by a bot's validate_config() method.
A common scenario for this is validating API keys before the bot is
created. If validate_config() fails, the bot will not be created.
2018-03-08 15:05:42 -08:00
Eeshan Garg 12396dd39b webhooks/gogs: Update text to conform to doc style guide. 2018-03-08 08:19:32 -08:00
Shubham Padia 32e38d36aa bots: Send add/delete event on bot ownership change.
Adds realm_bot delete event. On bot ownership change, add event is
sent to the bot_owner(if not admin) and delete event to the
previous bot owner(if not admin). For admin, update event is sent.
2018-03-08 07:54:19 -08:00
Tim Abbott c3964dff6e i18n: Fix a last few strings mentioning realms. 2018-03-07 17:15:29 -08:00
Tim Abbott e2a6541133 i18n: Fix use of 'realm administrator' in translated strings.
These are user-facing and thus should refer to being an "organization
administrator".
2018-03-07 17:15:29 -08:00
Tim Abbott 8fba40fdd5 emoji: Clean up strings for emoji errors.
The main goal here is to remove the use of the term "realm", but we
also make these strings more consistent and using better English.
2018-03-07 17:15:29 -08:00
Rhea Parekh 0d788823c2 slack importer: realm_id cleanup. 2018-03-07 14:07:24 -08:00
Rhea Parekh 7878cc53a3 slack importer: Cleanup build_subscription. 2018-03-07 14:07:24 -08:00
Rhea Parekh f947194e4c slack importer: Cleanup build_avatar. 2018-03-07 14:07:24 -08:00
Rhea Parekh 5efe05d5b6 slack importer: Cleanup build_zerver_usermessage. 2018-03-07 14:07:24 -08:00
Rhea Parekh 6c4eef16e9 slack importer: Sort messages in the order of their dates.
Sort the list of all messages in the order of their timestamp.
2018-03-06 14:07:09 -08:00
Rhea Parekh c2d4b49bf3 slack importer: Use precomputed value in 'channel_message_to_zerver_message'.
Use the List of all messages in this helper function
instead of using the messages channel-wise.
2018-03-06 14:07:09 -08:00
Rhea Parekh 3e4086d1b8 slack importer: Use precomputed value in 'get_total_messages_and_usermessages'.
Use the List of all messages in this helper function
instead of using the messages channel-wise.
2018-03-06 14:07:09 -08:00
Rhea Parekh 10c73ae577 slack importer: Add function to precompute all the messages.
The messages were first being read and passed to the helper
functions channel wise.
This function makes a list of all the messages in the all the channels
beforehand which would be used to pass in the helper functions.
2018-03-06 14:07:09 -08:00
Vishnu Ks 59b8f85c63 bugdown: Do only image preview if relative URL. 2018-03-06 13:50:02 -08:00
Tim Abbott 3d8e45f1cb cache: Avoid caching /help/ documentation page content.
This should make it a lot less annoying to edit these pages locally,
without regressing the test performance which motivated the cache.
2018-03-05 09:26:58 -08:00
Aditya Bansal 9eeb1c59f6 bugdown: Remove email from rendered content of messages with mentions.
This field has been unused by clients for some time, and isn't great
for our public archive feature plans (where we'll not want to be
including email addresses in messages).
2018-03-04 20:04:27 -08:00
Tim Abbott 9acf30b3d3 display_settings: Fix real-time sync of default language.
We apparently weren't sending to the frontend the data needed to
update the display of the current language.
2018-03-04 19:23:58 -08:00
Yago González b475e74c47 i18n: Fix string interpolation. 2018-03-04 15:44:12 -08:00
Marco Burstein bdb86f1b5e emoji: Add support for translating emoticons.
Add `translate_emoticons` to `prop_types` and `expected_keys`.
Furthermore, create a emoji-translating Markdown inline pattern.

Also use a JavaScript version of `translate_emoticons` and then use
this function during Markdown previews and as a preprocessor. This
is only needed for previews, because usually emoticon translation
happens on the backend after sending.

Add tests for emoticon translation, a settings UI, and a /help/ page
as well.

Tweaked by tabbott to fix various test failurse as well as how this
handles whitespace, requiring emoticons to not have adjacent
characters.

Fixes #1768.
2018-03-04 15:37:24 -08:00
Tim Abbott 038579b840 streams: Fix counts using only one data point.
We had the aggregation query for StreamCount done slightly wrong.
2018-03-04 14:06:29 -08:00
Robert Hönig e69af3011e Add basic test for FeedbackBot.
This sets up a new test class with a simple
test, mostly for increasing coverage. The class
should in the future be extended to properly
verify the handle_feedback() logic.
2018-03-04 13:31:33 -08:00
Archana BS c7650c0d9d streams: send data for recent streams traffic to frontend.
Significantly edited by Tim Abbott and Vishnu Ks.
2018-03-04 13:24:53 -08:00
Robert Hönig 4c7dd971d0 embedded bots: Dispatch all config data on update event.
Previously, when a user updated the config data of an
embedded bot, only the updated fields were dispatched
back to the client. Dispatching all config data fields
makes the client updating code less brittle.
2018-03-04 13:11:23 -08:00
Shubham Padia 13664f1289 uploads: Convert CMYK to RGB when saving avatar/realm icon as png.
Fixes #8546.
PNG does not support CMYK mode. CMYK file is converted to RGB and
then saved as PNG.
2018-03-02 12:57:22 -08:00
Eeshan Garg ce56ee80e7 api_docs: Document the `POST /api/v1/user_uploads` endpoint.
Fixes #1353.
2018-03-02 09:44:17 -08:00