Commit Graph

3319 Commits

Author SHA1 Message Date
Tim Abbott fdc1182a76 events: Optimize query for custom profile fields.
Our query for Custom Profile fields was for no good reason passing the
list of all users in the realm (potentially many thousands) into a
database query, rather than letting the database do that join.

Fixing this saves 100ms-200ms in the loading time for / on
chat.zulip.org for all users, since we were previously doing a ton of
work even if the feature wasn't being used.
2018-05-04 12:59:30 -07:00
Vishnu Ks 733da0ac07 settings: Fix 500 when trying to change email to disposable email.
Fixes #9240
2018-05-04 10:52:38 -07:00
Tim Abbott 8b26f912af streams: Limit access to public streams for guest users.
With most of the tests tests written by Shubham Dhama.
2018-05-04 09:47:58 -07:00
Tim Abbott b1ad7593ba validators: Improve mypy type annotations.
The main thing here is writing check_string_fixed_length and
check_capped_string as returning a Validator, but we also fix issues
around passing default=None.
2018-05-03 21:27:43 -07:00
Tim Abbott 51517fa188 request: Add new str_validator validator type.
This is helpful for cases where an argument is supposed to be a normal
string, and we want to use a Zulip validator function to do basic
things like check its length.
2018-05-03 21:11:02 -07:00
Tim Abbott f2e84f25a0 management: Refactor checkconfig code to live in library.
This makes it possible to call this from other management commands.
2018-05-03 21:11:02 -07:00
Tim Abbott ad0cfb3512 validators: Add check_string_fixed_length.
This will be useful in some upcoming work on zilencer.
2018-05-03 14:30:03 -07:00
Tim Abbott 594451707d check_capped_string: Clean up corner case.
We were rejecting strings of length equal to the max.

While we're at it, fix the unnecessary period in the error message,
which doesn't align with similar validators.
2018-05-03 14:30:03 -07:00
Rishi Gupta 42a99e8c1d zilencer: Rename remote_server_unregister_push. 2018-05-03 12:50:29 -07:00
Rishi Gupta 76650f5930 zilencer: Rename remote_server_register_push to register_remote_push_device. 2018-05-03 12:50:29 -07:00
Eeshan Garg d28d08e7da streams: Add get_default_value_for_history_public_to_subscribers().
This commit adds a function that makes it easier to get a default
value for Stream.history_public_to_subscribers when one isn't
explicitly provided.
2018-05-03 10:24:27 -07:00
Aditya Bansal 75d76e4eb3 lib/streams.py: Extract get_stream_by_id as a separate function.
We extract get_stream_by_id function out of the body of
access_stream_by_id function to help us access streams for archives.
2018-05-02 15:23:33 -07:00
Aditya Bansal 9629be689b populate_db: Add a web public stream to dev database.
We flip the Stream "Rome" to be a web public stream. Also we add
attribute is_web_public in various stream dicts and in the
bulk_create_streams function of bulk_create.py responsible for
default stream creation in dev environment.
2018-05-02 15:23:33 -07:00
Aditya Bansal 1f358954be web_public_streams: Add is_web_public to Stream table.
Also add function do_change_stream_web_public in lib/actions.py
to help in changing a streams web public status.
2018-05-02 15:23:33 -07:00
novokrest 0fb13eed2f outgoing_webhook: Extract success response handling to separate method.
Extract success response handling in do_rest_call() method to
separate method process_success_response()
2018-05-02 11:57:26 -07:00
novokrest 036bc120c3 outgoing_webhook: Extend process_success() return value to tuple.
Change return value type of OutgoingWebhookServiceInterface.process_success
to 2-elements tuple as (success_message, failure_message)
2018-05-02 11:57:08 -07:00
Tim Abbott 4df886f36f populate_db: Fix initialization of history_public_to_subscribers.
This was being incorrectly not initialized properly in the test suite,
because we neglected to update the bulk_create code path for creating
streams.
2018-05-02 09:02:57 -07:00
Tim Abbott 866cb38270 test_classes: Compute history_public_to_subscribers correctly.
We apparently missed updating this when we split out this database field.
2018-05-02 09:02:57 -07:00
Rohitt Vashishtha 2f6da2661f push_notifications: Format blockquotes properly in text_output.
New output is of the format:

Hamlet said:
> Polonius said:
> > This is the.
> > Second layer of nesting.
> First layer of nesting.

Fixes #9251.
2018-05-02 08:57:17 -07:00
Tim Abbott c1432d9dfc slack import: Reformat UserProfile to one-line-per-field.
This should be more readable and convenient for future editing.
2018-05-01 09:16:12 -07:00
Tim Abbott a2f49b425b slack import: Fix some erroneous UserProfile field values.
* tutorial_status was the invalid value 'T'; should be the default of 'W'.
* last_reminder can be just the default None
* enable_desktop_notifications was just the model default of True.
2018-05-01 09:14:47 -07:00
Rhea Parekh f00b80058d slack import: Remove unwanted comments. 2018-05-01 09:09:36 -07:00
Rhea Parekh e579bef8fd slack import: Improve how we construct user_profile objects.
Fixes #9260
2018-05-01 09:09:36 -07:00
Angelika Serwa f4f64243dd custom_profile_fields: Support changing the sort order of the fields.
Tweaked by tabbott for variable naming and the URL.

Closes #8879.
2018-04-30 18:17:41 -07:00
Yashashvi Dave 4033f210af custom fields: Add support for custom date field type. 2018-04-30 23:04:25 +05:30
Yashashvi Dave 66759358e2 create stream: Add maxlength restriction on name and description. 2018-04-30 10:11:25 -07:00
Tim Abbott 976e61d687 validators: Improve error messages for check_capped_string. 2018-04-30 10:07:06 -07:00
Yashashvi Dave 7bbe44d7a0 org settings: Remove "Delete streams" administrative tab from settings.
Fixes #9227
2018-04-30 17:47:34 +05:30
Eeshan Garg 057ff9c91e models: Add Stream.history_public_to_subscribers.
This commit adds a new field history_public_to_subscribers to the
Stream model, which serves a similar function to the old
settings.PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS; we still use that
setting as the default value for new streams to avoid breaking
backwards-compatibility for those users before we are ready with an
actual UI for users to choose directly.

This also comes with a migration to set the value of the new field for
existing streams with an algorithm matching that used at runtime.

With significant changes by Tim Abbott.

This is an initial part of our efforts on #9232.
2018-04-28 22:54:04 -07:00
Tim Abbott 7d6bb3dcb4 settings: Remove obsolete default_desktop_notifications setting.
This actually hasn't been hooked up to do anything in years.

While we're at it, we remove the entire "Zulip Labs" settings page.
2018-04-28 13:46:07 -07:00
Tim Abbott 127ac0df54 auth: Remove unnecessary case from validate_email_for_realm.
The removed code path was only needed due to buggy setup code in the
test_cross_realm_scenarios test.  We address that with a less buggy
workaround, and which lets us remove unnecessary complexity from this
important validation function.

Thanks for Umair Waheed for some preliminary work on this.

Fixes #7561.
2018-04-28 11:03:03 -07:00
YJDave 6bef44a9fa org setting: Add time limits for message deleting.
Add realm setting to set time limit for message deleitng.
Set default value of message_content_delete_limit_seconds
to 600 seconds(10 min).

Thanks to Shubham Dhama for rebasing and reworking this.  Some final
edits also done by Tim Abbott.

Fixes #7344.
2018-04-27 19:22:28 -07:00
Eeshan Garg 7d14ce2cb6 pypi packaging: Upgrade to release 0.4.6.
As a part of the upgrade, we had to update our API tests in
zerver/lib/api_test_helpers.
2018-04-27 14:50:25 -07:00
Tim Abbott abef9f203b api: Don't use ujson library unecessarily.
ujson is very fast, but doesn't provide good error messages for
parsing errors.
2018-04-27 14:49:46 -07:00
Eeshan Garg bd2270eecb test-api: Add test helpers that pretty-print JSON output. 2018-04-27 14:49:34 -07:00
Vishnu Ks f140b0e870 emails: Add translation tags to invitation. 2018-04-27 11:59:36 -07:00
Tim Abbott 2cdd367d49 email_mirror: Fix handling of empty topic.
Also fixs some corner cases around pure-whitespace topics, and
migrates from the years-obsolete "no subject".

Fixes #9207.
2018-04-26 10:21:29 -07:00
Umair Khan cf2f6b38dd profile: Add choice field.
Fixes part of #8878
2018-04-26 00:35:51 -07:00
Umair Khan 4ea3e8003a profile: Create mypy types for profile data.
This makes the code more readable.
2018-04-25 23:28:27 -07:00
Tim Abbott b40780d003 mypy: Fix errors in new bugdown module.
I introduced these when making final changes before merging.
2018-04-25 15:56:46 -07:00
Lyla Fischer 0f3cb14aae user docs: Use {settings_tab} macro for more organization settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer dbc573584b user docs: Use the {settings_tab} macro for four organization settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer b2be1a67f8 help docs: Add {settings_tab} for the first three org settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer 68f68bf56d help docs: Use {settings_tab} macro for some user settings. 2018-04-25 14:39:30 -07:00
Lyla Fischer a4ea71ec0f help docs: Use {settings_tab|notifications} macro. 2018-04-25 14:39:30 -07:00
Lyla Fischer d40f246599 help docs: Use {settings_tab|display-settings}. 2018-04-25 13:51:29 -07:00
Lyla Fischer b24659b005 bugdown: Add {settings|my-setting} macro.
Tweaked by tabbott to add a test and fix a super subtle issue with the
relative_settings_link variable having been set once the first time a
/help article was rendered.
2018-04-25 13:41:24 -07:00
Tim Abbott ff9371d63c slack import: Fix issues with Slack empty files.
Fixes #9217.
2018-04-25 10:20:55 -07:00
neiljp (Neil Pilgrim) 9692a8572d mypy: Add assertion in timeout.py. 2018-04-25 08:58:55 -07:00
Rishi Gupta 718a87bd47 emails: Update followup_day1. 2018-04-25 08:52:29 -07:00
Tim Abbott 62fb139af7 Revert "test_fixtures: Add settings files to things that require reprovision."
This reverts commit 2bc51931a8.

See #9210 for the follow-up work needed before we can re-add this.
2018-04-24 11:01:20 -07:00
Preston Hansen efc7967355 slack import: Update `build_zerver_realm` to use `Realm` defaults.
Fixes #9131.
2018-04-23 17:15:51 -07:00
Vishnu Ks 134fdd8fd0 bugdown: Replace vimeo link with video title. 2018-04-23 17:04:24 -07:00
Vishnu Ks 5671cef6d0 bugdown: Cleanup Vimeo preview. 2018-04-23 17:04:24 -07:00
Tim Abbott c4b886d8ae import: Split out import.py into its own module.
This should make it a bit easier to find the code.
2018-04-23 15:21:12 -07:00
Vishnu Ks c9e932a7ce settings: Add support for Hangouts as the video chat provider.
The only thing that's annoying about this feature is that you need to
be a paying G Suite customer to use it.
2018-04-23 09:39:47 -07:00
Tim Abbott 01be6b01b1 validate_domain: Add verification of domain length. 2018-04-23 09:29:03 -07:00
Shubham Dhama 26d2ffa821 populate_db: Add new user of "guest" type.
The purpose of this user is to act as a guest.
(This is a preliminary step in adding the guest type of user
and is a part of #8385.)
2018-04-20 16:20:00 -07:00
Tim Abbott 49f58583a4 models: Introduce can_access_public_streams field.
This is a simple computed field.  It's intended to more clearly
capture the meaning of this restriction for the users in zephyr mirror
realms, and eventually support guest user accounts in normal Zulip
realms.
2018-04-20 12:06:57 -07:00
Tim Abbott dc6d7d0d12 actions: Use presence_disabled for presence zephyr_mirror code path.
This is part of the effort to remove the use of is_zephyr_mirror_realm
across the code path for situations that might be relevant for other
users.  It helps keep the code readable.
2018-04-20 12:06:57 -07:00
Roman Godov d99758129e subs: Notify organization admins when private streams are renamed.
This commit sends the event for renaming of a private stream to
organization admins of the realm, in addition to the obvious list of
subscribers of the private stream.

Normally, admins can manage a private stream (e.g. unsubscribing a
user).  But when the admin tried to unsubscribes a user from a
previously renamed stream, we previously were throwing a JS error, as
the webapp hadn't been notified about the new stream name.

Fixes #9034.
2018-04-19 22:00:18 -07:00
Preston Hansen e168f9938c tests: Refactor use of test and webhook data fixtures. 2018-04-19 21:50:29 -07:00
Preston Hansen 76d6c71595 tests: Move zerver/fixtures to zerver/tests/fixtures for clarity.
Fixes #9153.
2018-04-19 21:50:17 -07:00
Umair Khan 0420b89468 profile: Add hint attribute to custom profile fields.
This is the model for #8876.
2018-04-19 11:32:35 -07:00
Umair Khan 00ffa808da check_capped_string: Return a validator.
This allows us to create custom validators.
2018-04-19 11:21:50 -07:00
Eeshan Garg 19177a4aff webhooks: Move github_webhook/ to github/. 2018-04-19 11:00:55 -07:00
Eeshan Garg 48b8558c02 webhooks: Move github/ to github_legacy/ and remove docs. 2018-04-19 11:00:55 -07:00
Tim Abbott 7dfa0edfa6 slack import: Don't try to import pinned/unpinned items.
There isn't a corresponding Zulip concept, and they don't have a
"text" attribute, so there's no message content to import.
2018-04-18 11:36:31 -07:00
Tim Abbott acd3a364e1 slack import: Refactor handling of dropped messages.
This is a more coherent ordering, because some messages we skip lack a
"text" attribute.
2018-04-18 11:35:59 -07:00
Tim Abbott e759fd9be4 slack import: Improve error handling for invalid messages. 2018-04-18 11:29:40 -07:00
Tim Abbott 1410a1e460 slack import: Remove unnecessary zerver_realm_skeleton.json.
This was stored as a fixture file under zerver/fixtures, which caused
problems, since we don't show that directory under production (as its
part of the test system).

The simplest emergency fix here would be to just move the file, but
when looking at it, it's clear that we don't need or want a fixture
file here; we want a Python object, so we just do that.

A valuable follow-up improvement to this block would be to create an
actual new Realm object (not saved to the database), and dump it the
same code we use in the export tool; that should handle the vast
majority of these correctly.

Fixes #9123.
2018-04-18 10:33:53 -07:00
Tim Abbott 77ca9e7eca hotspots: Tag strings for translation. 2018-04-18 09:33:09 -07:00
Rhea Parekh a2070fb7e5 slack importer: Add comment on size information of avatars.
The size information of an avatar is not required during the import.
Check function 'import_uploads_local' and 'import_uploads_s3'
in 'export.py' for this.
2018-04-16 14:44:57 -07:00
Tim Abbott 0c30a26d81 bulk_create: Remove some long-dead code.
We used to use these in populate_db, but haven't done so in a long
time, and it doesn't seem likely that will change anytime in the
future.
2018-04-16 11:41:42 -07:00
Ben Reeves fdfbd45208 soft_deactivation: Change `<` to `<=` in add_missing_messages.
We should still short-circuit the iteration in
`add_missing_messages` if the unsubscription was the last
thing to happen to the user before unsubscription and
soft deactivation.
2018-04-16 11:28:08 -07:00
Rhea Parekh f6b6aa1e75 slack import: Implement threading as a management command. 2018-04-15 19:53:02 +05:30
Rhea Parekh 7c0c3930a8 slack importer: Thread avatar downloads. 2018-04-15 19:53:01 +05:30
Rhea Parekh ebc2ee28e9 slack importer: Thread emoji downloads. 2018-04-15 19:52:59 +05:30
Rhea Parekh 8a291d0232 slack importer: Thread attachment downloads.
Use Zulip's run_parallel method to run thread downloads.
2018-04-15 19:51:58 +05:30
Eeshan Garg 3e0eb9530c webhooks: Remove the Facebook integration.
Rishi and I decided that it makes sense to get rid of the Facebook
integration for a few reasons, some of which are:

* The setup process is too complicated on Facebook's end. The users
  will surely have to browse Facebook's huge API reference before even
  having a vague idea of what they want.
* Slack chooses not to have a Facebook integration, but relies on
  Zapier for it. Zaps that integrate with Facebook are much more
  streamlined and the setup process isn't as much of a pain. Zapier's
  Facebook Zaps are much more fine-tuned and there are different Zaps
  for different parts of the FB API, a luxury that would likely span
  2K+ lines of code on our end if we were to implement it from
  scratch. So, I think we should relegate integration with Facebook to
  Zapier as well!
* After thoroughly testing the setup process, we concluded that the
  person who submitted the FB integration didn't really test it
  thoroughly because there were some gaping holes in the docs (missing
  steps, user permissions, etc.).
2018-04-14 09:38:22 -07:00
Tim Abbott 5ddf2614f0 uploads: Add new way of querying for mobile uploads endpoint.
This extends the /user_uploads API endpoint to support passing the
authentication credentials via the URL, not the HTTP_AUTHORIZATION
headers.  This is an important workaround for the fact that React
Native's Webview system doesn't support setting HTTP_AUTHORIZATION;
the app will be responsible for rewriting URLs for uploaded files
directly to add this parameter.
2018-04-13 17:51:45 -07:00
Tim Abbott db830c4085 bugdown: Replace link to old pre-open-source Zulip trac. 2018-04-13 08:41:44 -07:00
Rohitt Vashishtha 9e7929417d markdown: Increase rendered_content length limit.
This commit increases the rendered_content limit from 2x to 10x of the
original message length.

Earlier, we had placed a limit of MAX_MESSAGE_LENGTH * 2 for the
rendered content (explained in commit
77addc5456).  That limit was based on
the assumption that in most cases, the rendered content wouldn't cause
a large increase in message length. However, quite prominently in
syntax highlighted codeblocks, that wasn't true and this caused the
limit condition to be hit for long messages composed primarily of code
blocks.

Example: The following message would render close to 10x it's original size.

```py
if:
def:
print("x", var)
x = y
```

Because the syntax highlighted logic is extremely compressible, having
rendered_content reach up to 100KB doesn't create a network
performance problem.
2018-04-13 08:39:51 -07:00
Rhea Parekh f7398cbb09 slack import: Implement custom profile fields.
Add custom profile fields in the slack converted
data 'realm' file.
Added tests for the custom profile fields.

Fixes #8928
2018-04-10 13:28:53 -07:00
Rhea Parekh 852e8516b4 slack import: Add custom profile fields.
Build CustomProfileField and CustomProfileFieldValue
for every user and process the field type after getting an
entire list of the custom fields.
2018-04-10 13:28:53 -07:00
rht a183186672 slack importer: User session.get to recycle previous connections. 2018-04-09 22:02:01 -07:00
Shubham Dhama b650b6b38c markdown: Add @stream as an alias for @all.
Fixes: #8930.
2018-04-09 16:35:14 -07:00
rht 630adb406b Slack importer: Map Slack channel mentions to Zulip stream mentions. 2018-04-09 10:47:39 -07:00
Rhea Parekh 035c440ff3 import script: Support import custom profile fields.
Import of Custom profile fields is only supported for slack
import script for now.
2018-04-09 10:45:35 -07:00
Tim Abbott c41d7ee300 slack import: Write reasonable multi-line JSON.
This is a lot better for debugging.
2018-04-09 10:45:35 -07:00
Rhea Parekh ed7127c8b4 import script: Delete medium sized avatars if it exists.
Deletion of medium sized image is done if it exists before calling the
function 'ensure_medium_avatar_image', to avoid potentially confusing
problems with left-over medium-size avatar images from a previous run
being used when repeatedly importing the same realm in a development
environment..

Fixes #8949.
2018-04-08 07:04:24 -07:00
rht f6feac1316 Slack importer: Map Slack command for mentions to Zulip's all.
Fixes #9003.
2018-04-07 20:02:39 -07:00
Rhea Parekh e037c2f93e import script: Fix upload links.
Rendered content is None for Slack imports, hence it is replaced only
for Zulip->Zulip imports.

Fixes #8959.
2018-04-07 20:01:20 -07:00
Rhea Parekh b3f951d2cf import script: User profile ids should be allocated before allocating bot ids. 2018-04-07 13:28:33 +05:30
Rhea Parekh 2baa9bc16e Import: Add subdomain in the import script.
Also remove user input of subdomain in the slack data
conversion script.
2018-04-06 09:12:56 -07:00
Tim Abbott c06565d909 users: Improve testing for user_ids_to_users. 2018-04-04 16:31:30 -07:00
novokrest 4d2082ab14 actions.py: Obtain bot profiles by using users.user_ids_to_users().
Remove models.get_user_profiles_by_ids() and
obtain user's bots profiles in actions.get_service_dicts_for_bots() by
users.user_ids_to_users() instead of models.get_user_profiles_by_ids().
Fixes #8939
2018-04-04 16:24:55 -07:00
novokrest 807a6ccf2c users.py: Implement user_ids_to_users() by generic_bulk_cached_fetch().
Optimize users.user_ids_to_users() implementation
by using generic_bulk_cached_fetch() to obtain user profiles
2018-04-04 16:24:55 -07:00
Tim Abbott 53e47e6991 messages: Modify access_message for is_history_public_to_subscribers.
This completes the Message side of #2745.
2018-04-04 16:18:47 -07:00
Tim Abbott bec71d7a50 messages: Add a server-level setting to control private stream history.
We don't indend for this server-level setting to exist in the long
term; the purpose of this is just to make it easy to test this code
path for development purposes.

This implements much of the Message side part of #2745.
2018-04-04 16:18:46 -07:00