Commit Graph

8173 Commits

Author SHA1 Message Date
Rhea Parekh af20ef4789 slack import: Save messages within convert_slack_workspace_messages.
Previously, the messages where being stored in a output file from
outside the function 'convert_slack_workspace_messages', but
now we store it from the inside the mentioned function.
This will help in processing and saving the messages batch-wise
so as to avoid a memory error.

Reactions are returned separately from 'convert_slack_workspace_messages'
rather than 'message_json'.

Also updated test for 'convert_slack_workspace_messages' and an additional
test for reactions is added.
2018-07-01 07:08:13 -07:00
Rhea Parekh 2f88ca7446 slack import: Import skype and phone data of users. 2018-07-01 07:05:40 -07:00
Eeshan Garg 061e760d1e webhooks: Add Clubhouse integration. 2018-07-01 04:01:17 -07:00
Tim Abbott 5f7691b74e upload: Remove unnecessary use of has_request_variables.
All the parameters for this function are parsed in urls.py.
2018-07-01 01:47:03 -07:00
Eeshan Garg e1e4d523aa Replace remaining user-facing occurences of the term "invite-only".
Fixes #9611.
2018-06-27 12:34:29 -07:00
Steve Howell a2a695dfa7 test flake: Prevent flaky test with less common id.
This fixes a test flake introduced here:

    317a2fff2a

We need a higher bogus bot owner id to prevent
flakes where our userid sequence gets to 100.  (Tests
aren't completely deterministic in what data you
use, since sequences don't get rolled back when
you roll back transactions.)
2018-06-26 16:56:24 -04:00
Rishi Gupta 2de9029a69 stream settings: Add tests for weekly stream traffic calculation. 2018-06-23 12:48:15 -07:00
Rishi Gupta dc7cfd3eb7 stream settings: Fix stream_age calculation in weekly traffic calculation. 2018-06-23 12:48:15 -07:00
Rishi Gupta 3054b175c9 stream setting: Fix mypy error in get_average_weekly_stream_traffic.
Not sure why mypy wasn't catching this.
2018-06-23 12:48:15 -07:00
Rishi Gupta 15b236a21f stream settings: Show non-zero stream traffic if traffic is greater than 0.
An estimated traffic of 0 suggests a stream is dead, and has pretty
different semantics from any non-zero value. So we should round up any
number between 0 and 1 to 1.
2018-06-23 12:48:15 -07:00
Rishi Gupta 5eccabc3c5 stream settings: Set average_weekly_traffic to -1 if stream is new.
We don't ever use this value, but it's confusing to have the incorrect
calculation in the code.

Ideally we would set this to "None", but I don't know the code well enough
to be confident nothing would break.
2018-06-23 12:48:15 -07:00
Rishi Gupta f63bcb99d7 stream settings: Refactor out STREAM_TRAFFIC_CALCULATION_MIN_AGE_DAYS. 2018-06-23 12:48:15 -07:00
Rishi Gupta ccf5f1ac89 stream settings: Fix weekly stream traffic computation.
Previously, if stream_age were 13, this number would be off by almost a
factor of 2.
2018-06-23 12:48:15 -07:00
Yashashvi Dave 317a2fff2a bots settings: Improve API for editing bot owners to refer to users by ID.
Fixes #9502
2018-06-23 12:47:50 -07:00
Vishnu Ks 6e6e2e03da management: Use tokenized no-reply address in send_test_email. 2018-06-23 12:03:30 -07:00
Vishnu Ks 1b179ca530 signup: Prevent users from signing up with email containing +. 2018-06-23 12:03:30 -07:00
Vishnu Ks a88e44e254 registration: Use tokenized noreply address in email change. 2018-06-23 12:03:30 -07:00
Vishnu Ks e34fcf982f registration: Use tokenized noreply address in user invite. 2018-06-23 12:03:30 -07:00
Vishnu Ks 994e1a2154 registration: Use tokenized noreply address in password reset. 2018-06-23 12:03:30 -07:00
Vishnu Ks 2b1424e51a registration: Use tokenized noreply address in signup confirmation. 2018-06-23 12:03:30 -07:00
Vishnu Ks 74351fb250 send_email: Add function to generate tokenized noreply in FromAddress. 2018-06-23 12:03:30 -07:00
Yago González 51f107f5bc api docs: Fix indentation. 2018-06-23 12:42:11 +02:00
Yago González 8b420da653 api docs: Remove unnecessary imports. 2018-06-23 12:42:11 +02:00
Yago González 8e50564717 api docs: Document the /users/me/<stream_id>/topics endpoint. 2018-06-23 12:42:11 +02:00
Raymond Akornor af5de189d8 tests_users: Use assert_json_success instead of assertTrue. 2018-06-22 09:06:43 -04:00
Marco Burstein 6dad6f3dae emoji: Add some edge-casey ordering tests for emoji.
Add 3 new Markdown emoji tests for newlines, emphasis, and links. The
goal of these tests is to ensure that Markdown operations concerning
emoji are preformed in proper order, with emoji being added correctly
based on other Markdown operations.

See suggestion here: https://git.io/flF5W.
2018-06-22 08:59:08 -04:00
Rhea Parekh d066c11b47 zcommand: Strip the slash in the backend.
The slash in command is stripped in the backend,
rather than in the client to make the client code
cleaner.
This would make client code cleaner in the slash
commands which include parameters.
2018-06-20 16:49:29 -04:00
Rhea Parekh f4bbd4f1a3 tests: Use Set instead of List in import tests. 2018-06-20 10:22:34 -07:00
Marco Burstein 02489e5fb8 emoji: Fix a bug where emoticon translation is not done after a newline.
This bug is caused by the conversion of newlines to `<br>` statements,
since `>` is not allowed as a character around an emoticon during
translation.

Also, add a new test case for preventing this bug from occurring in the
future.

Fix #9763.
2018-06-20 08:09:18 -07:00
Rishi Gupta 4f8c44a019 notifications: Simplify notification messages for new subscriptions.
We're adding more stream types, e.g. splitting private streams into
with/without shared history, adding publicly-archived streams, adding
announce-only streams, etc. So maintaining this text is going to get more
complicated over time.

Also, the right place to explain this stuff is in the stream header, or near
the z-in-a-circle.

This commit also adds translation tags to the messages.
2018-06-20 07:53:10 -04:00
Vishnu Ks 403f254557 signup: Create get_accounts_for_email function. 2018-06-19 11:25:23 -07:00
Shubham Dhama 8e032376f9 guest: Restrict guest access to user group creation and updation. 2018-06-19 11:04:14 -07:00
Rhea Parekh 200c381c91 tests: Add tests for importing files from S3 backend. 2018-06-19 10:50:11 -07:00
neiljp (Neil Pilgrim) 88a70c5f5d mypy: test_embedded_bot_system.py: add asserts & remove from mypy.ini. 2018-06-19 10:48:38 -07:00
neiljp (Neil Pilgrim) e4821875f7 mypy: Improve typing of oembed data, to Dict[str, Any]. 2018-06-19 10:48:38 -07:00
neiljp (Neil Pilgrim) b5aa705137 mypy: test_link_embed.py: add assert & remove from mypy.ini. 2018-06-19 10:48:38 -07:00
Rhea Parekh d1ba6bae03 import: 'processing_emojis' and 'processing_avatars' should now be True together.
Raise an exception when the fields
'processing_emojis' and 'processing_avatars' are
True at the same time. Also add test for the same.
2018-06-18 23:06:09 +05:30
Rhea Parekh a2a74d9271 export: The records.json IDs should be integer.
In records the IDs like the realm_id and user_profile_id
of 'records.json' should be integers. This was missing in the
S3 backend and this commit fixes that.

Added tests for this as well.
2018-06-18 23:06:09 +05:30
Rhea Parekh 4d21f7f747 import: 'attachment_path' should be saved with the 's3_path' of the record.
For the S3 backend uploads, 'attachment_path' should be
saved with the 's3_path' of the record, as the original
'path' is changed while exporting files from s3. (See
function 'export_files_from_s3' in export.py for reference.)
2018-06-18 23:06:01 +05:30
Rhea Parekh 0730087111 import: Add elif condition for 'processing_emojis' in 'import_uploads_s3'.
'processing_emojis' should have an 'elif' condition here as
we want the function to work for avatars, emojis or uploads
one at a time.
2018-06-18 23:04:18 +05:30
Rhea Parekh f2b5f5a8f9 import: Fix processing_avatars bug in 'import_uploads_s3'.
All the avatars should be processed later on to run the
'ensure_medium_avatar_image' function. This is similar to
'import_uploads_local'.
2018-06-18 22:37:34 +05:30
Rhea Parekh f66ca9a5c3 import: Pass 'processing_emojis' in 'import_uploads_s3'.
'import_uploads_s3' should be passed with the parameter
'processing_emojis' from 'import_uploads'.
2018-06-18 22:35:36 +05:30
Rhea Parekh b2e971b9b1 tests: Add tests for the export file's records.
For the emojis, In 'records.json', the record should contain
the attribute 'file_name', which was missing in the S3 backend.
This commit adds this attribute, as well as tests for the
records of uploads, avatars and emojis in both local and S3 backend.
2018-06-18 09:19:24 -07:00
Rhea Parekh 5cbad76445 tests: Add tests for importing files from local. 2018-06-18 09:19:24 -07:00
Rhea Parekh edb506a605 tests: Add a function to setup export of files.
Add the function '_setup_export_files' to setup the files to
be exported.
2018-06-18 09:19:24 -07:00
Rhea Parekh f735fae777 tests: Add tests for a basic import logic. 2018-06-18 09:19:24 -07:00
Rhea Parekh 0e8f0e2393 tests: Rename 'message.json' to 'messages-000001.json' in export tests.
The export script produces messages in the format 'messages-%06d.json'.
Hence we rename it to 'messages-000001.json'.
2018-06-18 09:19:24 -07:00
Yashashvi Dave 8909cb1d15 custom fields: Allow list of users in user type of custom fields.
Allow user to add more than one user-value in user type of custom
fields.

Tweaked by tabbott to improve the models.py code and type annotations.
2018-06-16 09:37:49 -07:00
Eeshan Garg 943a77105b integrations/irc: Capitalize the integration name properly.
It is IRC, not Irc, which is what the default code in Integration
renders.
2018-06-16 08:52:49 -07:00
Vishnu Ks 4da61e2a17 signup: Copy enter_sends value when importing settings. 2018-06-16 08:50:49 -07:00
Vishnu Ks b639778d0f signup: Copy hotspots when importing settings. 2018-06-16 08:50:49 -07:00
Rhea Parekh d0bc8d0736 zcommands: Add zcommand module and separate test module.
Move the zcommands from '/views/messages.py' to
'/lib/zcommand'.

Also, move the zcommand tests from '/tests/test_messages.py'
to '/tests/test_zcommand'.
2018-06-16 10:32:42 -04:00
Shubham Dhama b0bfb6b153 streams: Hide create stream UI from guest users. 2018-06-16 06:33:14 -07:00
Shubham Dhama d40c2bb166 emoji: Make uploading new realm emoji inaccessible for guest users. 2018-06-16 06:33:14 -07:00
Shubham Dhama 596ce5b60f invite: Make inviting new users inaccessible for guest users. 2018-06-16 06:33:13 -07:00
Tim Abbott 425b4a3866 invite: Use require_realm_admin decorator. 2018-06-16 06:33:13 -07:00
Neil Pilgrim ba55d22fdb mypy: Improve MessageOutput typing in export.py.
See the comments above for why this is the correct list of options.
2018-06-14 15:22:56 -07:00
Eeshan Garg 05b752bd3f webhooks/trello: Mention that Trello script can be run from anywhere.
This is one of those weird webhooks where the
download-python-bindings.md macro doesn't work, because the user
only needs the bindings to run the one-time Trello script to register
the webhook and that script can be run from anywhere and doesn't need
to be "hosted" anywhere.
2018-06-11 07:46:16 -04:00
Tim Abbott 93b5e0df61 custom profile fields: Extract a separate various for whether enabled.
This will make it possible to do this on a per-organization basis.
2018-06-07 11:35:04 -07:00
Roman Godov 0b9d4fbb06 auth: Added time conversion in new login emails.
This commit adds code for date and time conversion in
"new login" emails according to user's timezone.

Fixes #9336.
2018-06-07 08:40:22 -07:00
Yashashvi Dave f7f039e772 custom fields: Add endpoint to delete value of custom profile data.
Add delete endpoint at `users/me/profile_data` to delete value of
custom profile data.

Fixes #9669
2018-06-07 08:35:04 -07:00
Adam Birds 5855fe3e53 integrations: Add Zabbix webhook.
With a few changes to the documentations by Eeshan Garg.
2018-06-07 08:31:24 -07:00
Eeshan Garg 4446b76427 pypi: Upgrade to release 0.5.0. 2018-06-07 08:31:21 -07:00
MaTaM673 d27c230982 webhooks/teamcity: Include branch name in build notifications. 2018-06-07 08:15:30 -07:00
Tim Abbott 4481c74a71 GitHubAuthBackendTest: Remove token_data_dict argument.
This was always the same, and there's not much reason to customize it.
2018-06-06 15:22:52 -07:00
Aditya Bansal 1205e02c64 provisioning: Don't rebuild DB if running migrations is sufficient.
This results in a significant optimization in the performance of
re-provisioning Zulip if all that you're doing is rebasing onto a
newer version of master (which just adds new migrations).

The change carries some risk of generating unpleasant-to-debug
situations, because if we merge a buggy migration and then later fix
it, some clients may not have a properly migrated database (and also,
this changes how populate_db commutes with migrations).  But it seems
worth it, given how much time is currently wasted by not having this.

Fixes: #9512.
2018-06-06 13:43:59 -07:00
Aditya Bansal 65dc80fe9d test_fixtures: Add run_generate_fixtures_if_required function.
In this commit we are adding run_generate_fixtures_if_required,
a new function which is meant to de-duplicate a bit of code
between test-server and test-backend which is essentially
responsible for rebuilding the test database if that was required.
2018-06-06 13:37:35 -07:00
Aditya Bansal f7c11d1747 test_fixtures: Refactor to have template_database_status API.
In this commit we are essentially just refactoring the function
is_template_database_current to be called template_database_status
and adjusting the return values accordingly.
This is essentially a preparatory commit for the upcoming commits
which will essentially enable us to not throw away entire DB and
rebuild from scratch if only running migrations could do the job.
2018-06-06 13:37:35 -07:00
Vishnu Ks 53237d39aa registration: Allow users to import profile picture. 2018-06-06 13:29:42 -07:00
Vishnu Ks ca87cf4c97 upload: Seperate writing avatars into s3 as function. 2018-06-06 13:29:42 -07:00
Vishnu Ks 7960785261 upload: Separate writing avatars as local files into function. 2018-06-06 13:29:42 -07:00
Tim Abbott 7d250cb2f9 attachment: Fix handling of stream history.
This fixes two issues:

* Our guest users feature gave guest users access to public stream
  attachments even if they couldn't access the public stream.

* After a user joins a private stream with our new shared history
  feature, they couldn't see images uploaded before they joined.

The tests need to check for a few types of issues:
* The actual access control permissions.
* How many database queries are used in the various
  cases for that second model, especially with multiple messages
  referencing an attachment.  This function gets called a lot, and we
  want to keep it fast.

Fixes #9372.
2018-06-06 09:40:22 -07:00
Tim Abbott 541ccfeb7f validate_attachment_request: Simplify execution flow.
This makes the code more readable, by narrowing the try/except block
and only querying `messages` when we're going to use it.
2018-06-06 09:13:51 -07:00
Tim Abbott 35c4a9f1d2 auth: Rewrite our social auth integration to use pipeline.
This new implementation model is a lot cleaner and should extend
better to the non-oauth backend supported by python-social-auth (since
we're not relying on monkey-patching `do_auth` in the OAuth backend
base class).
2018-06-05 23:24:48 -07:00
Yashashvi Dave 5145b24635 users: Replace duplication with generic func to validate user id.
This adds a common function `access_user_by_id` to access user id
within same realm, complete with a full suite of unit tests.

Tweaked by tabbott to make the test much more readable.
2018-06-05 11:13:13 -07:00
Umair Khan 9b698dec08 2FA: Use patch.object for monkey patching.
Ref #9521
2018-06-05 09:25:40 -07:00
Tim Abbott 238ba2d486 auth: Move some simple GitHub tests to the long-term test class. 2018-06-04 23:57:09 -07:00
Tim Abbott 869d75b02f AuthBackendTest: Fix typos in error message checks.
Previously, these checks did nothing.
2018-06-04 23:45:13 -07:00
Raymond Akornor 0335da7e05 python: Use python3 style super() in 2FA code paths. 2018-06-04 20:55:42 -07:00
Shubham Dhama 01555e8772 streams: Handle guest user ids for stream settings changes' events. 2018-06-04 11:35:37 -07:00
Steve Howell a42492d0ac Fix very recent build error w/url coverage for casper.
Example:
    http://localhost:9991/casper/casper-failure1.png
2018-06-04 12:37:10 -04:00
Tim Abbott ec90322e5d bots: Fix behavior for bots mentioned in PMs.
We've for a long time had the behavior that a bot mentioned in a
stream message receives the notification, regardless of whether the
bot was actually subscribed to the stream.

Apparently, this behavior also triggered if you mentioned a bot in a
private message (i.e. the bot would be delievered the private message
and would probably respond unhelpfully in a new group private message
thread with the PMs original recipients plus the bot).

The fix for this bug is simple: To exclude this feature for private
messages.
2018-06-03 22:29:15 -07:00
Tim Abbott ec2eb0edba storage: Don't double-minify webpack bundles.
What was happening before is that we built the webpack bundles in
tools/minify-js with nicely hashed filenames, and then `manage.py
collectstatic` was extending these filenames with a second hash
through the use of storage.

Removing the first one didn't seem ideal, but would probably have
worked, but seems confusing for people only familiar with webpack
(ideally, we want the Django toolchain piece to be increasingly
invisible as we replace it).

And we can't exclude the webpack bundles from being processed by
storage, since we need these bundles to be included in the manifest.
So, instead, we set the hash function to be a no-op for the bundle
files.

Fixes significant portions #5971.

More work is required to deal with versioning for some of the
image/font assets.
2018-06-03 16:49:59 -07:00
Tim Abbott 714ad67db1 storage: Move library code into zerver/lib.
There's no particularly good reason for this to have been a top-level
file.
2018-06-03 16:02:02 -07:00
Rishi Gupta 182215d125 portico: Update text of why-zulip.
With edits from tabbott.
2018-06-03 09:37:00 -07:00
Shubham Dhama 4483e33102 digest: Make newly registered users data inaccessible to guest users.
The new can_access_all_realm_members function is meant to act as a
base function for guest users and Zephyr realm users regarding the
accessibility of the information of other users in the realm.
2018-06-03 09:30:59 -07:00
Shubham Padia 8883107a7f streams: Allow cross-realm bots to post to announcement_only streams.
This fixes an issue where if you make #announce (the default
announcement stream) announce-only, then creating a new stream will
throw an exception (because notification-bot can't send there).

Fixes #9636.
2018-06-02 10:36:29 -07:00
Steve Howell 47b4dd6bdb slash commands: Refine /day and /night.
These two slash commands now use zcommand to talk to
the server, so we have no Message overhead, and if you're
on a stream, you no longer spam people by accident.

The commands now also give reasonable messages
if you are already in the mode you ask for.

It should be noted that by moving these commands out of
widget.py, they are no longer behind the ALLOW_SUB_MESSAGES
setting guard.
2018-06-02 09:40:12 -07:00
Steve Howell 4b2e8b83c4 slash commands: Add /ping command (via zcommand).
This adds a /ping command that will be useful for users
to see what the round trip to the Zulip server is (including
only a tiny bit of actual server time to basically give a
200).

It also introduce the "/zcommand" endpoint and zcommand.js
module.
2018-06-02 09:40:12 -07:00
Rhea Parekh f8e53511ea slack import: Add support for bold-italics formatting.
Fixes #8927
2018-06-02 09:01:55 -07:00
Tim Abbott 18702b0d71 actions: Document validate_user_access_to_subscribers_helper.
This function is important, carefully optimized, and hard to read, so
it deserves a long docstring.
2018-06-02 08:46:58 -07:00
Shubham Dhama 8e2337509d streams: Hide public streams & subscribers from guest users.
The main change here is making can_access_public_streams
2018-06-02 08:39:39 -07:00
Shubham Dhama ca02143074 test_subs: Fix typos in test_previously_subscribed_private_streams. 2018-06-02 08:06:45 -07:00
Tim Abbott 751c602a0b update-prod-static: Don't copy/minify templates and styles.
This is a performance optimization: Rather than copying these files
into the `prod-static` directory and then deleting them, we just don't
copy them over in the first place.

For styles, it might have once been the case that this did something,
but we've moved them all to being managed by webpack some time ago.

For the js directory, I think it was never useful to copy and then
delete them; these files were always compiled via tools/minify-js,
and the raw JS files weren't needed, anyway.
2018-06-01 16:39:35 -07:00
Tim Abbott 2a52b0c1e8 compilemessages: Use json.dump with sort_keys option.
This should help make these files stable over time.  Mostly useful for
diffing release tarballs.
2018-06-01 16:39:03 -07:00
Eeshan Garg 52c3c00ea0 test-api: Test against generic fixtures.
In a few commits before this one, we just added de-duplicated
generic fixtures that apply to multiple API tests. The tests
needed to be modified to accommodate that change.
2018-06-01 14:06:26 -07:00
Yago González 9f98002b76 tests: Allow testing our REST API against the OpenAPI docs. 2018-06-01 12:03:31 -07:00
Tim Abbott c37fe998d2 templates: Make zulip_default_context available in API endpoints.
This is mostly useful for advertising support_email, but some other
fields may by helpful over time.
2018-06-01 11:32:18 -07:00
Tim Abbott 0d84eb0a8b mypy: Clean zerver/views/users.py for strict-optional. 2018-06-01 08:48:17 -07:00
Tim Abbott 0a0ae5e703 users: Move add_service to zerver.lib.users so it is reusable. 2018-06-01 08:48:17 -07:00